A model inversion method based on a traffic sign classification system

Abstract

The application discloses a model inversion method based on a traffic sign classification system, which is divided into auxiliary data set enhancement, generator pre-training and privacy data reconstruction. In the generator pre-training, a WGAN model with gradient penalty is trained on unlabeled traffic sign data, including a generator and a discriminator. In the privacy data reconstruction, the discriminator trained in the generator pre-training process is replaced with a traffic sign target classifier, the generation result of the generator is input into the target classifier to obtain a classification confidence output, the generator is fine-tuned through gradient descent, and finally the generator can generate reconstructed traffic sign data from random noise. According to the reconstructed data, the related information of the original training data corresponding to the traffic sign classification system is inferred, so that the model inversion for the traffic sign classification system is completed. The application enables the model inversion to obtain good inversion results without relying on high-quality auxiliary data sets.

Description

Technical Field

[0001] This invention belongs to the field of traffic information security, and in particular relates to a model inversion method based on a traffic sign classification system. Background Technology

[0002] In the field of intelligent transportation, traffic sign classification systems are a crucial component, helping drivers and other road users better understand and comply with traffic rules. These systems typically employ advanced image recognition and artificial intelligence technologies to identify and classify traffic signs, providing drivers with relevant navigation and information services. In recent years, deep learning technology has been widely applied to traffic sign classification tasks. However, the over-memorization of models by deep learning has been shown to pose a risk of privacy data leakage. Model inversion is a technique for obtaining training data information from a model, such as inverting the traffic sign images used for training from a traffic sign classification system. Research shows that generative adversarial networks (GANs)-based model inversion methods can invert good face training data from face classifiers. However, the performance of existing GAN-based model inversion methods depends on the quality of the auxiliary dataset; the closer the auxiliary dataset is to the real private training data, the better the model inversion effect. In the transportation field, obtaining high-quality auxiliary datasets is very difficult; therefore, existing model inversion methods perform poorly for traffic sign classification systems. Furthermore, existing research often focuses on high-semantic face data, and the final step of model inversion methods often optimizes input noise. For low-semantic traffic industry data, this method struggles to achieve good inversion results. Summary of the Invention

[0003] The purpose of this invention is to address the shortcomings of existing technologies by providing a model inversion method based on a traffic sign classification system. This invention proposes a pre-training-fine-tuning mechanism based on a Large Language Model (LLM) and Generative Adversarial Networks (GANs) to complete the model inversion of the traffic sign classification system.

[0004] The objective of this invention is achieved through the following technical solution:

[0005] A model inversion method based on a traffic sign classification system, comprising the following steps:

[0006] Step 1: Use a large language model to query relevant domain knowledge on traffic sign classification and introductions to different categories of traffic signs to obtain descriptive prompts for different categories of traffic signs; combine the prompts with existing unlabeled traffic sign images, and use a text-to-image generation model to generate more unlabeled traffic sign images to achieve data augmentation.

[0007] Step 2: Train a generative adversarial network (GAN) model with gradient penalty using unlabeled traffic sign images. The GAN model includes a generator and a discriminator. The generator is a deconvolutional neural network consisting of one linear layer and three deconvolutional layers. The discriminator is a convolutional neural network consisting of five convolutional layers and one fully connected layer. The generator's input is noisy samples, and its output is fake samples generated based on the noise. The discriminator's input is the unlabeled traffic sign image and the fake samples output by the generator. The initial training of the generator and discriminator is completed by constructing the loss functions for the generator and discriminator.

[0008] Step 3: Replace the already trained discriminator with a traffic sign target classifier. The input of the traffic sign target classifier is the generation result of the trained generator, and the output is the classification confidence. Calculate the classification confidence of the output and the negative log-likelihood loss of the one-hot encoding of the corresponding category. Continue to train the generator a second time through gradient descent.

[0009] Step 4: Input a piece of random noise into the generator after secondary training. The generator outputs the reconstructed traffic sign image, completing the model inversion of the traffic sign classification system.

[0010] Furthermore, step one includes the following sub-steps:

[0011] Step 1.1: Use GPT4 to search for relevant knowledge on traffic sign classification and introductions to different categories of traffic signs;

[0012] Step 1.2: According to the description of GPT4, select a similar image from the existing unlabeled data as the initial image; if there is no similar image, select an image without too many features as the initial image.

[0013] Step 1.3: Write prompts for Stable Diffusion, including: first, describe the input image; second, explain the parts that need to be added or modified based on the input image; third, describe the features of the parts in detail; and finally, explain that the added or modified parts should be organically integrated with the original image.

[0014] Step 1.4: Adjust the relevant parameters of Stable Diffusion and enable Stable Diffusion based on the initial image and prompt words;

[0015] Step 1.5: Repeat steps 1.3-1.4 based on the results generated by Stable Diffusion until the results match the description of the traffic sign.

[0016] Furthermore, step two includes the following sub-steps:

[0017] Step 2.1: Initialize the parameters of the generator G and the discriminator D;

[0018] Step 2.2: From the real unlabeled data distribution p data A batch of real samples x are sampled from (x); from the noise distribution p z A noise z is sampled from (z), and a fake sample is generated by generator (G). in, and real sample x and fake sample All inputs are into the discriminator D;

[0019] Step 2.3: Construct the loss function L of the discriminator as follows: D And update the discriminator's parameters to minimize L. D :

[0020]

[0021] Where λ is the coefficient of the gradient penalty. It is obtained by random interpolation between the real sample x and the fake sample g(z), and ∈ is a random number uniformly distributed in [0,1]. express The gradient of , where ∥∥2 represents the L2 norm. Indicates the distribution of real data p data The expectation obtained from sampling in (x); The representation represents the noise distribution p. z The expectation obtained from sampling in (z); Let represent a distribution between real samples and fake samples. The expected value obtained from sampling;

[0022] Step 2.4: Construct the generator's loss function L G And update the generator's parameters to minimize L. G ;

[0023]

[0024] Step 2.5: Repeat steps 2.2-2.4 until the training stopping condition is met, thus completing the initial training of the generator and discriminator.

[0025] Furthermore, step three includes the following sub-steps:

[0026] Step 3.1: From the noise distribution p z A noise z is sampled from (z), and a fake sample G′(z) is generated by the generator G′ after initial training;

[0027] Step 3.2: Select the data category c to be reconstructed and obtain the one-hot vector y corresponding to the category. c Where c∈[0,N-1], and N is the category dimension of traffic signs;

[0028] Step 3.3: Input the fake sample G′(z) into the traffic sign classifier T to obtain the classification confidence output T(G′(z)); the classification confidence output has completed the calculation of the softmax function;

[0029] Step 3.4: Calculate the classification confidence output T(G′(z)) and the one-hot vector y corresponding to the class. c The negative log-likelihood loss is applied, and the generator parameters are updated to minimize the loss function L. T :

[0030] L T =-y c (log e T(G′(z)))

[0031] Step 3.5: Repeat steps 3.3-3.4 until the training stopping condition is met, thus completing the retraining of the generator;

[0032] Step 3.6: Repeat steps 3.2-3.5 to obtain the reconstructed samples of all data categories by the retrained generator, thus completing the model inversion for all data categories.

[0033] An electronic device, comprising:

[0034] One or more processors;

[0035] A storage device for storing one or more programs, which, when executed by the electronic device, enable the electronic device to implement a model inversion method based on a traffic sign classification system.

[0036] A computer-readable storage medium having a program stored thereon that, when executed by a processor, implements a model inversion method based on a traffic sign classification system.

[0037] The beneficial effects of this invention are:

[0038] The model inversion method based on a traffic sign classification system provided by this invention has the following advantages over existing technologies:

[0039] 1. The method proposed in this invention utilizes the prior knowledge of a large model to augment the auxiliary dataset, enabling the model inversion to obtain good inversion results without relying on high-quality auxiliary datasets.

[0040] 2. This invention performs inversion by fine-tuning the generator parameters. Compared with the conventional model inversion method that only optimizes noise, it improves the performance of model inversion and provides a defensive approach to enhance the safety of traffic sign classification. Attached Figure Description

[0041] Figure 1 This is a schematic diagram of the model inversion based on the traffic sign classification system.

[0042] Figure 2 This is a schematic diagram of the various categories of data in the training set used by the traffic sign classification system.

[0043] Figure 3 This is a schematic diagram of the unlabeled data used for pre-training a generative adversarial network with gradient penalty.

[0044] Figure 4 This is a schematic diagram showing the results of model inversion for a traffic sign classification system. Detailed Implementation

[0045] The model inversion method for the traffic sign classification system proposed in this invention will be further described in detail below with reference to specific implementation methods.

[0046] like Figure 1 As shown, the model inversion method based on the traffic sign classification system of this invention consists of three steps: auxiliary dataset augmentation, generator pre-training, and privacy data reconstruction. In the large model data augmentation process, a large language model is used to understand domain knowledge about intelligent transportation and traffic sign classification systems, and descriptive prompts for different categories of traffic signs are generated with the assistance of the large language model. Then, combining the prompts and existing unlabeled data, a Stable Diffusion model is used to generate more unlabeled traffic sign data. In the generator pre-training process, a WGAN model with gradient penalty is trained on the unlabeled traffic sign data, including a generator and a discriminator. In the privacy data reconstruction process, the discriminator, which has already been trained in the generator pre-training process, is first replaced with a traffic sign target classifier, and the generator's output is input into the target classifier to obtain a classification confidence output. Then, the negative log-likelihood (NLL) loss of the confidence output and the one-hot encoding of the corresponding category is calculated. The generator is further fine-tuned using gradient descent, and finally, the generator can generate reconstructed traffic sign data from a piece of random noise. Based on the reconstructed data, relevant information of the original training data corresponding to the traffic sign classification system is inferred, thus completing the model inversion for the traffic sign classification system. Figure 1The model structure of the present invention and the relationship between the three processes are illustrated.

[0047] Step 1: Augmentation of auxiliary datasets.

[0048] Step 1.1: Use a large language model to query relevant domain knowledge on traffic sign classification and introductions to different categories of traffic signs.

[0049] Step 1.2: Input the initial image for each category into the large language model. In this implementation case, GPT4 is selected. Based on the description of GPT4, select a similar image from the existing unlabeled data. If no similar image is found, select an image with few features.

[0050] Step 1.3: Write the prompt for Stable Diffusion. First, describe the input image; second, explain the parts that need to be added or modified based on the input image; third, describe the features of that part in detail; and finally, emphasize that the added or modified parts should be organically integrated with the original image.

[0051] Step 1.4: Adjust the relevant parameters of Stable Diffusion, including image redraw ratio, prompt word guidance coefficient, etc., and then start the model.

[0052] Step 1.5: Repeat steps 1.3-1.4 based on the generated results until the results match the description of the traffic sign.

[0053] Step 2: Generator pre-training:

[0054] Step 2.1: Initialize the parameters of the generator G and discriminator D. The generator is a deconvolutional neural network consisting of one linear layer and three deconvolutional layers, and the discriminator is a convolutional neural network consisting of five convolutional layers and one fully connected layer.

[0055] Step 2.2: From the real unlabeled data distribution p data A batch of real samples x are sampled from (x); from the noise distribution p z A noise z is sampled from (z), and a fake sample is generated by generator (G). in, and real sample x and fake sample All inputs are to the discriminator D. The noise distribution here is Gaussian noise.

[0056] Step 2.3: Construct the loss function L of the discriminator as follows: D And update the discriminator's parameters to minimize L. D .

[0057]

[0058] Where λ is the coefficient of the gradient penalty. It is obtained by random interpolation between the real sample x and the fake sample G(z), and ∈ is a random number uniformly distributed in [0,1]. express The gradient of ||||2 represents the L2 norm. Indicates the distribution of real data p data The expectation obtained from sampling in (x); The representation represents the noise distribution p. z The expectation obtained from sampling in (z); Let represent a distribution between real samples and fake samples. The expected value obtained from sampling;

[0059] Step 2.4: Construct the generator's loss function L G And update the generator's parameters to minimize L. G ;

[0060]

[0061] Step 2.5: Repeat steps 2.2-2.4 until the preset number of training epochs is reached or other stopping conditions are met, thus completing the initial training of the generator.

[0062] Step 3: Reconstructing privacy data.

[0063] Step 3.1: From the noise distribution p z A noise z is sampled from (z), and a fake sample G′(z) is generated by the generator G′ after initial training.

[0064] Step 3.2: Select a data category c that you want to reconstruct, and obtain the one-hot vector y corresponding to the category. c Where c∈[0,N-1], and N is the category dimension of traffic signs.

[0065] Step 3.3: Input the fake sample G′(z) into the traffic sign classifier T to obtain the classification confidence output T(G′(z)). The classification confidence output here has already completed the softmax function calculation.

[0066] Step 3.4: Calculate the classification confidence output T(G′(z)) and the one-hot vector y corresponding to the class. c The negative log-likelihood loss is applied, and the generator parameters are updated to minimize the loss function L. T

[0067] L T =-y c (log e T(G′(z)))

[0068] Step 3.5: Repeat steps 3.3-3.4 until the preset number of training epochs is reached or other stopping conditions are met, thereby completing the retraining of the generator.

[0069] Step 3.6: Repeat steps 3.2-3.5 to obtain the reconstructed samples of all data categories by the retrained generator, thus completing the model inversion for all data categories.

[0070] This invention also provides an electronic device, including one or more processors and a storage device; the storage device is used to store one or more programs, which, when executed by the electronic device, enable the electronic device to implement a model inversion method based on a traffic sign classification system.

[0071] This invention also provides a computer-readable storage medium storing a program that, when executed by a processor, implements a model inversion method based on a traffic sign classification system.

[0072] The following example, a traffic sign classification dataset, illustrates the performance of the model inversion method based on a traffic sign classification system. The TSRD (Traffic Sign Retrieval Dataset) is a dataset containing a large number of traffic sign images. The data originates from cameras or Baidu Street View in natural scenes, thus including traffic sign images under different lighting conditions, angles, and occlusions, making the data more closely resemble real-world road scenarios. The TSRD dataset contains 6164 traffic sign images across 58 sign categories. These images are divided into two sub-databases: a training database and a test database. In this case, the test database serves as unlabeled data for pre-training. Meanwhile, 14 classes with more than 100 samples each from the training database are extracted to train a traffic sign classifier using a ResNet18 architecture. Figure 2 and Figure 3 The training set data and unlabeled data are shown separately. Since the traffic sign images in the TSRD dataset are not uniform in size, all images were first scaled to 64*64 pixels. To reduce model computation and facilitate implementation, all images were converted to grayscale. Figure 4 The results of the model inversion are shown. As can be seen from the figure, for most classes, the model inversion results basically restore the original training images and show a lot of information from the original training images.

Claims

1. A model inversion method based on a traffic sign classification system, characterized in that, The method includes the following steps: Step 1: Use a large language model to query relevant domain knowledge on traffic sign classification and introductions to different categories of traffic signs to obtain descriptive prompts for different categories of traffic signs; combine the prompts with existing unlabeled traffic sign images, and use a text-to-image generation model to generate more unlabeled traffic sign images to achieve data augmentation. Step 2: Train a generative adversarial network (GAN) model with gradient penalty using unlabeled traffic sign images. The GAN model includes a generator and a discriminator. The generator is a deconvolutional neural network consisting of one linear layer and three deconvolutional layers. The discriminator is a convolutional neural network consisting of five convolutional layers and one fully connected layer. The generator's input is noisy samples, and its output is fake samples generated based on the noise. The discriminator's input is the unlabeled traffic sign image and the fake samples output by the generator. The initial training of the generator and discriminator is completed by constructing the loss functions for the generator and discriminator. Step 3: Replace the already trained discriminator with a traffic sign target classifier. The input of the traffic sign target classifier is the generation result of the trained generator, and the output is the classification confidence. Calculate the classification confidence of the output and the negative log-likelihood loss of the one-hot encoding of the corresponding category. Continue to train the generator a second time through gradient descent. Step 4: Input a piece of random noise into the generator after secondary training. The generator outputs the reconstructed traffic sign image, completing the model inversion of the traffic sign classification system.

2. The model inversion method based on a traffic sign classification system according to claim 1, characterized in that, Step one includes the following sub-steps: Step 1.1: Use GPT4 to search for relevant knowledge on traffic sign classification and introductions to different categories of traffic signs; Step 1.2: According to the description of GPT4, select a similar image from the existing unlabeled data as the initial image; if there is no similar image, select an image without too many features as the initial image. Step 1.3: Write prompts for Stable Diffusion, including: first, describe the input image; second, explain the parts that need to be added or modified based on the input image; third, describe the features of the parts in detail; and finally, explain that the added or modified parts should be organically integrated with the original image. Step 1.4: Adjust the relevant parameters of Stable Diffusion and enable Stable Diffusion based on the initial image and prompt words; Step 1.5: Repeat steps 1.3-1.4 based on the results generated by Stable Diffusion until the results match the description of the traffic sign.

3. The model inversion method based on a traffic sign classification system according to claim 1, characterized in that, Step two includes the following sub-steps: Step 2.1: Initialize the parameters of the generator G and the discriminator D; Step 2.2: From the real unlabeled data distribution p data A batch of real samples x is sampled from (x); From noise distribution p z A noise z is sampled from (z), and a fake sample is generated by generator (G). in, and real sample x and fake sample All inputs are into the discriminator D; Step 2.3: Construct the loss function L of the discriminator as follows: D And update the discriminator's parameters to minimize L. D : Where λ is the coefficient of the gradient penalty. It is obtained by random interpolation between the real sample x and the fake sample G(z), and ∈ is a random number uniformly distributed in [0,1]. express The gradient of , where |||2 represents the L2 norm; Indicates the distribution of real data p data The expectation obtained from sampling in (x); The representation represents the noise distribution p. z The expectation obtained from sampling in (z); Let represent a distribution between real samples and fake samples. The expected value obtained from sampling; Step 2.4: Construct the generator's loss function L G And update the generator's parameters to minimize L. G ; Step 2.5: Repeat steps 2.2-2.4 until the training stopping condition is met, thus completing the initial training of the generator and discriminator.

4. The model inversion method based on a traffic sign classification system according to claim 1, characterized in that, Step three includes the following sub-steps: Step 3.1: From the noise distribution p z A noise z is sampled from (z), and a fake sample G′(z) is generated by the generator G′ after initial training; Step 3.2: Select the data category c to be reconstructed and obtain the one-hot vector y corresponding to the category. c Where c∈[0,N-1], and N is the category dimension of traffic signs; Step 3.3: Input the fake sample G′(z) into the traffic sign classifier T to obtain the classification confidence output T(G′(z)); the classification confidence output has completed the calculation of the softmax function; Step 3.4: Calculate the classification confidence output T(G′(z)) and the one-hot vector y corresponding to the class. c The negative log-likelihood loss is applied, and the generator parameters are updated to minimize the loss function L. T : L T =-y c (log e T(G′(z))) Step 3.5: Repeat steps 3.3-3.4 until the training stopping condition is met, thus completing the retraining of the generator; Step 3.6: Repeat steps 3.2-3.5 to obtain the reconstructed samples of all data categories by the retrained generator, thus completing the model inversion for all data categories.

5. An electronic device, characterized in that, include: One or more processors; A storage device for storing one or more programs, which, when executed by the electronic device, cause the electronic device to implement the model inversion method based on the traffic sign classification system as described in any one of claims 1 to 4.

6. A computer-readable storage medium, characterized in that, It stores a program that, when executed by a processor, implements the model inversion method based on the traffic sign classification system as described in any one of claims 1 to 4.

Images