Anonymization processing method of pedestrian image dataset based on stable diffusion model

Abstract

This invention discloses a method for processing pedestrian image datasets based on a stable diffusion model. First, corresponding prompt text is generated for each pedestrian image in the pedestrian image dataset to be anonymized. A stable diffusion model is constructed and pre-trained according to actual needs. The model is then fine-tuned using the pedestrian image dataset. New prompt text, different from the prompt text in the pedestrian image dataset, is generated and fed multiple times into the fine-tuned stable diffusion model to generate pedestrian images. All pedestrian images generated based on the same new prompt text are assigned the same pedestrian identity ID, thereby forming an anonymous pedestrian image dataset. This method effectively anonymizes pedestrian image datasets and effectively protects the privacy of pedestrian images without sacrificing dataset performance.

Description

Technical Field

[0001] The present invention belongs to the technical field of pedestrian image data anonymity processing, and more specifically, relates to a pedestrian image data set anonymity processing method based on a stable diffusion model. Background Art

[0002] With the rapid development of artificial intelligence (AI) technology, the protection of personal privacy has also received increasing attention. Some institutions and research projects have taken corresponding measures to mitigate the risk of privacy leaks. For example, to avoid the public disclosure of identity information, well-known datasets such as DukeMTMC and Tiny ImageNet have been modified or withdrawn in subsequent versions. The ImageNet dataset also blurs images containing facial features. Technology companies such as Meta have also banned the use of facial recognition systems for privacy reasons. These measures highlight the inherent contradiction between the use of personal images for AI research and the protection of privacy. However, in some areas, such as person re-identification, the collection of person re-identification data involves personal images and identity information, and still faces significant privacy challenges.

[0003] Person re-identification (PRI) is a key subtask in computer vision. It involves detecting and tracking the same person based on image features from different camera views. Major PRI datasets, such as Market-1501, DukeMTMC-reID, CUHK03, and MSMT17, are mostly collected in public locations like shopping malls and campuses. However, questions remain about whether these datasets adequately consider the privacy of participants. For example, simply by obtaining photos of these participants from social media, attackers could attempt to retrieve more private information from public datasets to conduct criminal activities. For example, DukeMTMC-reID contained a large number of unauthorized student images, resulting in user privacy leaks and ultimately the dataset's withdrawal. While these datasets have made significant contributions to the development of PRI research, their insufficient privacy protection has hindered the technology's practical application. Therefore, how to effectively utilize PRI datasets while preserving privacy is crucial for the feasibility and sustainability of PRI technology.

[0004] To address this issue, some existing person re-identification privacy protection studies have attempted to reconstruct datasets through image encryption or adversarial attacks. Image encryption-based methods aim to selectively blur sensitive areas, add noise, or encrypt the dataset before releasing it. However, these encrypted images require a complex decryption process before being used by the person re-identification model, which limits the scope of application of the encryption method. Adversarial attack methods reduce their identifiability during the retrieval process by adding carefully designed perturbations to the samples. Although these methods can effectively reduce the probability of a certain identity being retrieved by the person re-identification model, since the original personal identity is still retained, the image cannot be anonymized, and the true identity may still be restored through decryption or manual observation. Therefore, how to achieve image anonymization while maintaining person re-identification performance and prevent the original identity from being retrieved is an important issue that needs to be urgently addressed in the current research field. Summary of the Invention

[0005] The purpose of the present invention is to overcome the shortcomings of the existing technology and provide a pedestrian image dataset anonymization method based on a stable diffusion model, which can effectively anonymize the pedestrian image dataset and effectively protect the privacy of pedestrian images without sacrificing the performance of the dataset.

[0006] In order to achieve the above-mentioned object of the invention, the pedestrian image dataset anonymization processing method based on the stable diffusion model of the present invention includes the following steps:

[0007] S1: For the pedestrian image dataset X to be anonymized, each pedestrian image is denoted as x i , the corresponding identity ID is ID i , i=1,2,…,N, N represents the size of the data set X; determine M pedestrian attributes according to actual needs, and record the number of values ​​of each pedestrian attribute as K m , m=1,2,…,M, for the kth value of the mth pedestrian attribute, a description text t is generated m,k , k=1,2,…,K m ; Then use the pre-trained image-text retrieval model to extract K m Description text t m,k Each pedestrian image x is retrieved from i The corresponding description text, thereby determining the value y of the mth pedestrian attribute i,m ; Set the image prompt text template containing M pedestrian attributes according to the actual situation, and convert each pedestrian image x i The corresponding M pedestrian attribute values ​​y i,m Fill in the corresponding position of the image prompt text template to obtain the pedestrian image x i Corresponding prompt text txt i ;

[0008] S2: Build a stable diffusion model and perform pre-training according to actual needs, and then use the pedestrian image dataset X to fine-tune it. The specific method of fine-tuning training is as follows:

[0009] First, the pedestrian image dataset X is used to perform the first stage fine-tuning training on the stable diffusion model; then all pedestrian images in the pedestrian image dataset X are divided according to the same prompt text to obtain Q pedestrian image data subsets X q , q=1,2,…,Q, Q represents the number of prompt texts, for each pedestrian image data subset X q , from which pedestrian images belonging to the same identity are selected as the pedestrian image data subset Use Q pedestrian image data subsets Perform the second stage of fine-tuning training on the stable diffusion model;

[0010] S3: Generate all value combinations of M pedestrian attributes, then delete the existing value combinations of the prompt text of the original pedestrian image dataset X, and fill the pedestrian attribute values ​​in each remaining value combination into the corresponding position of the image prompt text template to generate a new prompt text;

[0011] S4: All new prompt texts obtained in step S3 are input multiple times into the stable diffusion model fine-tuned and trained in step S2 to generate pedestrian images. The same pedestrian identity ID is assigned to all pedestrian images generated based on the same new prompt text to form an anonymous pedestrian image dataset X′.

[0012] The present invention is based on a pedestrian image dataset processing method of a stable diffusion model. First, corresponding prompt text is generated for each pedestrian image in the pedestrian image dataset to be anonymized, a stable diffusion model is constructed and pre-trained according to actual needs, and then fine-tuned and trained using the pedestrian image dataset. Then, new prompt text different from the prompt text in the pedestrian image dataset is generated, and the fine-tuned stable diffusion model is input multiple times to generate pedestrian images. The same pedestrian identity ID is assigned to all pedestrian images generated according to the same new prompt text to form an anonymous pedestrian image dataset.

[0013] The present invention has the following beneficial effects:

[0014] 1) This paper uses a stable diffusion model to generate an anonymous pedestrian image dataset with a new identity, which can prevent the original identity from being retrieved through neural networks or manual observation, effectively strengthening privacy protection;

[0015] 2) The present invention can effectively remove the identity association between real data and generated data, and excels in preventing private identities from being retrieved from public datasets, while maintaining high performance in specific application areas (such as pedestrian re-identification). BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 This is a flowchart of a specific implementation of the method for anonymizing a pedestrian image dataset based on a stable diffusion model of the present invention;

[0017] Figure 2 This is an example diagram of generating image text in this embodiment. DETAILED DESCRIPTION

[0018] The following describes the specific embodiments of the present invention in conjunction with the accompanying drawings so that those skilled in the art can better understand the present invention. It should be noted that in the following description, when detailed descriptions of known functions and designs may dilute the main content of the present invention, such descriptions will be omitted here.

[0019] Example

[0020] Figure 1 This is a flowchart of a specific implementation of the pedestrian image dataset anonymization method based on the stable diffusion model of the present invention. Figure 1 As shown, the specific steps of the pedestrian image dataset anonymization processing method based on the stable diffusion model of the present invention include:

[0021] S101: Generate image text.

[0022] For the pedestrian image dataset X to be anonymized, each pedestrian image is denoted as x i , the corresponding identity ID is ID i , i=1,2,…,N, N represents the size of the data set X. Determine M pedestrian attributes according to actual needs, and record the number of values ​​of each pedestrian attribute as K m , m=1,2,…,M, for the kth value of the mth pedestrian attribute, generate a description text t m,k , k=1,2,…,K m Then a pre-trained image-text retrieval model is used to extract K m Description text t m,k Each pedestrian image x is retrieved from i The corresponding description text, thereby determining the value y of the mth pedestrian attribute i,m According to the actual situation, set the image prompt text template containing M pedestrian attributes, and i The corresponding M pedestrian attribute values ​​y i,mFill in the corresponding position of the image prompt text template to obtain the pedestrian image x i Corresponding prompt text txt i .

[0023] In this embodiment, pedestrian attributes are set to include gender, hair, top color, top type, trouser color and trouser type. A description text is generated for each attribute in the format of "a picture of a [attribute] person". The image prompt text template is set to "a person of <hair> <gender> wearing <top color><top type> and <trouser color><trouser type>". Figure 2 This is an example diagram of generating image text in this embodiment. Figure 2 As shown in the figure, an image of a short-haired male pedestrian wearing a green T-shirt and black shorts is retrieved through image-text retrieval to obtain the values ​​of its various pedestrian attributes. The resulting prompt text filled into the image prompt text template is "A short-haired man wearing a green T-shirt and black shorts".

[0024] The image-text retrieval model can be set according to actual needs. In this embodiment, a multimodal pre-trained neural network CLIP (Contrastive Language-Image Pre-Training) model is adopted. CLIP is an effective and scalable method for learning from natural language supervision. The model learns to perform a wide range of tasks during pre-training, including OCR, geolocation, action recognition, and is more computationally efficient while outperforming the best publicly available ImageNet pre-trained model. The core idea of ​​the CLIP model is to use a large amount of paired data of images and texts for pre-training to learn the alignment relationship between images and texts. The CLIP model has two modalities, one is textual and the other is visual, and includes two main parts: 1. Text Encoder: used to convert text into a low-dimensional vector representation - Embedding. 2. Image Encoder: used to convert images into similar vector representations - Embedding. In the prediction stage, the CLIP model generates a predicted image-text matching degree by calculating the cosine similarity between the text and image vectors, thereby achieving retrieval.

[0025] S102: Fine-tuning the stable diffusion model:

[0026] A stable diffusion model is constructed and pre-trained based on actual needs, and then fine-tuned using the pedestrian image dataset X. The specific structure of the stable diffusion model can be set according to actual needs. In this embodiment, the SDv1-5 model is used. The fine-tuning training process of the stable diffusion model in the present invention is divided into two stages:

[0027] The goal of the first phase is to fine-tune the style of the generated images to match those in the Pedestrian Image Dataset X and develop a preliminary ability to generate realistic and recognizable pedestrian images based on prompts. Therefore, the Stable Diffusion Model is first fine-tuned using the Pedestrian Image Dataset X. The specific training method and loss function are determined based on the specific architecture of the Stable Diffusion Model. Through this fine-tuning training, the Stable Diffusion Model learns the relationship between the prompt text and the visual attributes and features of the people in the image, enabling it to generate images that match the prompt text provided by the Pedestrian Image Dataset X.

[0028] The goal of the second stage is to more accurately map the cue words to pedestrian features to better control the generation process of the stable diffusion model. After the first stage, the generated images match the style of the original pedestrian image dataset X. However, due to the overlapping attributes between identities in the entire dataset, the stable diffusion model cannot accurately interpret the cue words. In addition, the cue words have difficulty accurately describing the appearance and texture of the image. For example, when faced with the attribute term "black coat", the stable diffusion model does not know which identity "black coat" refers to, because images of multiple identities in the original pedestrian image dataset X can all be described by "black coat". Therefore, the stable diffusion model may generate a coat for pedestrian A in the first image and a coat for pedestrian B in the second image, resulting in poor consistency within the same identity. Therefore, the second stage of the present invention fine-tunes the stable diffusion model by selecting a subset with better consistency based on the first stage to address this problem, thereby improving consistency within the identity and making the features more consistent with the cue words.

[0029] The specific method of the second stage is: divide all pedestrian images in the pedestrian image dataset X according to the same prompt text to obtain Q pedestrian image data subsets X q , q=1,2,…,Q, Q represents the number of prompt texts. For each pedestrian image data subset X q , from which pedestrian images belonging to the same identity are selected as the pedestrian image data subset Use Q pedestrian image data subsets The second stage of fine-tuning training is performed on the stable diffusion model. In practical applications, in order to make the fine-tuned stable diffusion model not overly focus on a certain attribute, the pedestrian image data subset The number of pedestrian images N in q The following requirements should be met:

[0030]

[0031] Among them, λ∈(0,1) represents the preset ratio.

[0032] Through fine-tuning in the second stage, we can effectively prevent different identities from being described by repeated attributes, allowing the generative diffusion model to learn a direct mapping from prompt text to pedestrian appearance, further improving the generation quality and consistency of images generated by the stable diffusion model.

[0033] S103: Generate new prompt text:

[0034] After step S102, the stable diffusion model has the ability to generate behavioral images through prompt text to form a new dataset. In order to differentiate the original pedestrian image dataset X from the new anonymous pedestrian image dataset X′ and better achieve pedestrian image anonymization, the present invention needs to generate new prompt text that is different from the prompt text in the original pedestrian image dataset X. The specific method is as follows:

[0035] Generate all value combinations of M pedestrian attributes, then delete the existing value combinations of the prompt text of the original pedestrian image dataset X, and fill the pedestrian attribute values ​​in each remaining value combination into the corresponding position of the image prompt text template to generate a new prompt text.

[0036] S104: Generate anonymous pedestrian image dataset:

[0037] All new prompt texts obtained in step S103 are fed multiple times into the fine-tuned stable diffusion model trained in step S102 to generate pedestrian images. All pedestrian images generated based on the same new prompt text are assigned the same pedestrian ID, forming an anonymous pedestrian image dataset X′. As can be seen, because the new prompt text is different from all prompt texts in the original pedestrian image dataset X, the pedestrian information in the generated pedestrian images is also different from that in the original pedestrian images, thus achieving pedestrian anonymity.

[0038] In practical applications, in order to improve the application performance of the anonymous pedestrian image dataset X′ in the target model and increase the stability of the same identity in the new dataset, the low-quality samples of the anonymous pedestrian image dataset X′ can also be filtered. The specific method is: the pedestrian images corresponding to the same pedestrian identity ID in the anonymous pedestrian image dataset X′ are recorded as {x1′, x′2,…, x′ B}, where x b ′ represents the bth pedestrian image corresponding to the pedestrian ID, b=1,2,…,B, B represents the number of pedestrian images corresponding to the pedestrian ID. The target model is used to extract each pedestrian image x b ′’s characteristic f b , then according to the feature f bThe outlier samples in the B pedestrian images are searched and removed to filter out low-quality samples. The target model is determined based on the application scenario of the pedestrian image dataset. For example, in the field of pedestrian re-identification, the target model is the pedestrian re-identification model.

[0039] In order to better illustrate the technical effects of the present invention, specific examples are used to experimentally verify the present invention.

[0040] Example 1

[0041] The experimental conditions in this embodiment are set as follows: system: Ubuntu 20.04, software: Python 3.9, processor: Intel(R) Xeon(R) CPU E5-2678 v3@2.50GHz×2, memory: 256GB, graphics processor: NVIDIA A100x4.

[0042] This example conducts person re-identification tests on the Market-1501, DukeMTMC-reID, and CUHK-SYSU datasets, and the evaluation index uses the generalization index T g , availability index T u and privacy index T p .

[0043] Table 1 is a statistical table of indicators of the present invention on three data sets in this embodiment.

[0044]

[0045] Table 1

[0046] As shown in Table 1, in order to gain a deeper understanding of the privacy protection differences between datasets, the present invention achieves a high generalization index T on all three datasets. g and availability index T u , while achieving a lower privacy index T p , which illustrates the stronger privacy protection and higher data utilization of the present invention.

[0047] In addition, this embodiment defines an indicator of identity separation ISD to measure the difference between the new dataset and the original dataset. Specifically, the real pedestrian image dataset contains G pedestrians, and their corresponding IDs are respectively denoted as ID g , g=1,2,…,G. The pedestrian image of the g-th pedestrian is denoted as x g,b ,b=1,2,…,B g , B g Indicates the number of pedestrian images of the g-th pedestrian. Use the target model to transform each pedestrian image x g,b The characteristic f g,b, and then use the following formula to calculate the pedestrian's central feature c g :

[0048]

[0049] The identity separation degree ISD is calculated using the following formula:

[0050]

[0051] Among them, F j Represents the features of generating the jth pedestrian image in the pedestrian image dataset,

[0052] The greater the identity separation, the lower the similarity between the real pedestrian image dataset and the generated pedestrian image dataset, indicating a higher degree of privacy protection. Table 2 is a comparison table of identity separation before and after processing the dataset using the present invention in this embodiment.

[0053] Dataset Original ISD ISD of the present invention promote Market-1501 0.15 0.50 2.33 DukeMTMC-reID 0.11 0.53 3.82 CUHK-SYSU 0.13 0.63 3.85

[0054] Table 2

[0055] The original ISD in Table 2 is calculated by using the real pedestrian image dataset as the generated pedestrian image dataset. As shown in Table 2, the distance between the images in the anonymous pedestrian image dataset generated by the present invention and the centroid of the real identity is significantly larger. This means that the characteristics of the new identity of the generated anonymous pedestrian image are significantly different from the real identity, thereby achieving better privacy protection. It should be noted that compared with other datasets, the present invention obtains better results on CUHK-SYSU, which has more identities than other datasets. This shows that the present invention may benefit from a large number of training identities in the real dataset, because more real identities lead to better diversity in the generated data.

[0056] Furthermore, to demonstrate the technical advantages of the present invention, six existing technologies were used as comparison methods to compare their performance with the present invention on the Market-1501 and DukeMTMC-reID datasets. The comparison methods include:

[0057] UMDL (Unsupervised Domain Adaptation Method), for details, see the reference "Peng P, Xiang T, Wang Y, et al. Unsupervised cross-dataset transfer learning for person re-identification [C]. Proceedings of the IEEE conference on computer vision and pattern recognition. 2016: 1306-1315.";

[0058] CamStyle. For details, see the reference "Zhong Z, Zheng L, Zheng Z, et al. Camstyle: A novel data augmentation method for person re-identification[J]. IEEE Transactions on Image Processing, 2018, 28(3): 1176-1190.";

[0059] HHL, for details, see the reference “Zhong Z, Zheng L, Li S, et al. Generalizing a person retrieval model hetero-and homogeneously[C]. Proceedings of the European conference on computer vision (ECCV). 2018: 172-188.”;

[0060] PersonX Sub1, PersonX Sub4. For details, see the reference "Sun X, Zheng L. Dissecting person re-identification from the viewpoint of viewpoint[C]. Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. 2019: 608-617."

[0061] TRS, for details, see the reference "Lin Y, Guo X, Wang Z, et al. Privacy-protected personre-identification via virtual samples[J]. IEEE Transactions on Information Forensics and Security, 2023.".

[0062] Table 3 shows the generalization index T of the present invention and the comparative method in this embodiment. g Comparison table.

[0063]

[0064] Table 3

[0065] As shown in Table 3, for the generalization index T g , the present invention achieved better results on both datasets. Higher retrieval results, or mAP, indicate that person re-ID models trained on data generated by the present invention perform better on real-world data. Compared to existing comparison methods, the present invention can generate more realistic data that is applicable to real-world application scenarios.

[0066] Although the above describes the illustrative specific embodiments of the present invention to facilitate understanding of the present invention by those skilled in the art, it should be clear that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the appended claims, these changes are obvious, and all inventions and creations using the concepts of the present invention are protected.

Claims

1. A pedestrian image dataset anonymization method based on a stable diffusion model, characterized by: The following steps are involved: S1: For the pedestrian image dataset X to be anonymized, each pedestrian image is denoted as x i , the corresponding identity ID is ID i , i=1,2,…,N, N represents the size of the data set X; Determine M pedestrian attributes according to actual needs, and record the number of values ​​of each pedestrian attribute as K m , m=1,2,…,M, for the kth value of the mth pedestrian attribute, a description text t is generated m,k , k=1,2,…,K m ; Then use the pre-trained image-text retrieval model to extract K m Description text t m,k Each pedestrian image x is retrieved from i The corresponding description text, thereby determining the value y of the mth pedestrian attribute i,m ; Set the image prompt text template containing M pedestrian attributes according to the actual situation, and convert each pedestrian image x i The corresponding M pedestrian attribute values ​​y i,m Fill in the corresponding position of the image prompt text template to obtain the pedestrian image x i Corresponding prompt text txt i ; S2: Build a stable diffusion model and perform pre-training according to actual needs, and then use the pedestrian image dataset X to fine-tune it. The specific method of fine-tuning training is as follows: First, the pedestrian image dataset X is used to perform the first stage fine-tuning training on the stable diffusion model; then all pedestrian images in the pedestrian image dataset X are divided according to the same prompt text to obtain Q pedestrian image data subsets X q , q=1,2,…,Q, for each pedestrian image data subset X q , from which pedestrian images belonging to the same identity are selected as the pedestrian image data subset Use Q pedestrian image data subsets Perform the second stage of fine-tuning training on the stable diffusion model; S3: Generate all value combinations of M pedestrian attributes, then delete the existing value combinations of the prompt text of the original pedestrian image dataset X, and fill the pedestrian attribute values ​​in each remaining value combination into the corresponding position of the image prompt text template to generate a new prompt text; S4: All new prompt texts obtained in step S3 are input multiple times into the stable diffusion model fine-tuned and trained in step S2 to generate pedestrian images. The same pedestrian identity ID is assigned to all pedestrian images generated based on the same new prompt text to form an anonymous pedestrian image dataset X′.

2. The pedestrian image dataset anonymization processing method according to claim 1, characterized in that: The pedestrian attributes in step S1 include gender, hair, top color, top type, trouser color and trouser type.

3. The pedestrian image dataset anonymization processing method according to claim 1, characterized in that: The image-text retrieval model in step S1 adopts a multimodal pre-trained neural network CLIP model.

4. The pedestrian image dataset anonymization processing method according to claim 1, characterized in that: The stable diffusion model adopts the SDv1-5 model.

5. The pedestrian image dataset anonymization processing method according to claim 1, characterized in that: The pedestrian image data subset in step S2 The number of pedestrian images N in q The following requirements should be met: Among them, λ∈(0,1) represents the preset ratio.

6. The pedestrian image dataset anonymization processing method according to claim 1, characterized in that: The step S4 also includes filtering low-quality samples of the anonymous pedestrian image dataset X′. The specific method is: the pedestrian images corresponding to the same pedestrian ID in the anonymous pedestrian image dataset X′ are recorded as {x1′, x′2,…, x′ B }, where x b ′ represents the b-th pedestrian image corresponding to the pedestrian ID, b=1,2,…,B, B represents the number of pedestrian images corresponding to the pedestrian ID; Use the target model to extract each pedestrian image x b ′’s characteristic f b , then according to the feature f b The outlier samples in B pedestrian images are searched and deleted to filter out low-quality samples.

Images