A method and system for generating camouflaged speed limit traffic sign samples based on StyleGAN2 and CLIP

Abstract

This invention discloses a method for generating disguised speed limit traffic sign samples based on StyleGAN2 and CLIP. It combines the StyleGAN2 generative adversarial network and the CLIP model, and introduces a multi-objective loss optimization framework to achieve efficient generation of disguised speed limit traffic sign samples. StyleGAN2 provides powerful image generation capabilities, generating high-quality speed limit traffic sign samples through its latent variable space. The CLIP model provides guidance through its semantic understanding capabilities, helping the StyleGAN2 generative adversarial network optimize the image generation process, ensuring that the generated disguised samples visually match the key features of the speed limit traffic sign and conform to the target semantic description text s, thereby ensuring the semantic consistency of the generated disguised samples. The disguised sample generation method proposed in this invention includes three core modules: semantic constraint, latent variable optimization, and multi-loss fusion. The semantic constraint module uses the CLIP model to define the semantic similarity between the generated disguised samples and the target semantic description text s.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and generative adversarial network technology, and more specifically, relates to a method and system for generating camouflaged speed limit traffic sign samples based on StyleGAN2 and Contrastive Language-Image Pre-training (CLIP). Background Technology

[0002] With the rapid development of autonomous driving technology, attack testing methods targeting the perception modules of autonomous driving systems have become a research hotspot in recent years to improve the safety of these systems. Traffic Sign Recognition (TSR) systems are a key component of the perception systems of autonomous vehicles. Accurately recognizing traffic signs, especially speed limit signs, is crucial for ensuring safe driving in complex traffic environments. However, the perception modules of autonomous driving systems are vulnerable to maliciously designed inputs, especially in complex and ever-changing real-world traffic environments. This vulnerability can lead to errors in the system's recognition of critical targets.

[0003] Existing attack methods targeting the perception modules of autonomous driving systems, especially traffic sign recognition modules, mainly fall into the following two categories:

[0004] 1. Adversarial examples are attacks that involve making small, carefully designed perturbations to an input image, causing a deep learning model to make incorrect predictions without significantly altering the image's appearance. For example, an attacker could make small perturbations to an image of a speed limit traffic sign, causing the traffic sign recognition system to misidentify the sign as a stop sign or a warning sign. The generation of adversarial examples typically relies on gradient descent algorithms, which optimize the loss function to minimize the perturbation, thereby confusing the perception model.

[0005] 2. Physical world attacks refer to attacks where attackers interfere with the perception of autonomous driving systems by modifying real-world traffic signs or placing distracting objects. These attacks typically involve covering traffic signs with reflective materials or altering their shape, color, and size to confuse the perception module. For example, an attacker might add a light-reflective material to a traffic sign, preventing cameras or other sensors from correctly acquiring the sign's features and causing the autonomous driving system to misidentify it.

[0006] However, the two attack methods for traffic sign recognition modules mentioned above still have some significant flaws:

[0007] First, adversarial attack methods lack precise control over changes in the category features of the image or the target model. The attack effect is highly dependent on the parameters of the target model, resulting in poor generalization ability.

[0008] Second, physical world attack methods often overlook the stealth of attack samples, and the generated samples are easily detected or identified by autonomous driving systems, making it impossible to effectively test the system's anti-interference capabilities.

[0009] Third, the physical world attack methods are too simplistic and lack comprehensive consideration of the key features and diverse styles of traffic signs, making them difficult to adapt to complex real-world traffic environments. Summary of the Invention

[0010] To address the aforementioned deficiencies or improvement needs of existing technologies, this invention provides a method and system for generating camouflaged traffic sign samples by combining StyleGAN2 and CLIP models. Its purpose is to solve the technical problems of existing attack methods targeting traffic sign recognition modules, which suffer from poor generalization ability due to a lack of precise control over image category features or changes in the target model, resulting in attack effectiveness highly dependent on the parameters of the target model; the technical problems of existing attack methods neglecting the concealment of attack samples, making the generated samples easily detectable or recognized by autonomous driving systems, thus failing to effectively test the system's anti-interference capabilities; and the technical problems of existing attack methods being too simplistic in form, lacking comprehensive consideration of key features and diverse styles of traffic signs, making them difficult to adapt to complex real-world traffic environments.

[0011] To achieve the above objectives, according to one aspect of the present invention, a method for generating camouflaged speed limit traffic sign samples based on StyleGAN2 and CLIP is provided, comprising the following steps:

[0012] (1) Load the CLIP model's text encoder to obtain the target semantic description text s input by the user, and use the text encoder to parse the target semantic description text s to obtain the semantic feature vector v. text .

[0013] (2) Load the pre-trained StyleGAN2 generative adversarial network model and initialize the loaded StyleGAN2 generative adversarial network potential vector to obtain the initialized potential vector z0.

[0014] (3) Calculate the loss function using the CLIP model and the StyleGAN2 generator in the pre-trained StyleGAN2 generative adversarial network model loaded in step (2), and use this loss function to iteratively optimize the latent vector z obtained in step (2) (in the first iteration, the latent vector z is the initialized latent vector z0 obtained in step (2)) until the iteration termination condition is met, so as to obtain the final optimized latent vector z. final ;

[0015] (4) The final optimized potential vector z obtained in step (3) final Input the StyleGAN2 generator from the pre-trained StyleGAN2 generative adversarial network model to obtain spoofed sample images.

[0016] Preferably, step (1) specifically involves first loading the text encoder of the CLIP model, then obtaining the target semantic description text s input by the user, inputting the target semantic description text s into the text processing module of the CLIP model for standardization processing to obtain a standardized token matrix; then, inputting the standardized token matrix into the previously loaded text encoder of the CLIP model for encoding to obtain the semantic feature vector v of the target semantic description text s. text .

[0017] Preferably, step (2) specifically involves first calling the pre-trained StyleGAN2 generative adversarial network model, loading the corresponding StyleGAN2 generative adversarial network structure and weight parameters, and switching the StyleGAN2 generator in the StyleGAN2 generative adversarial network to evaluation mode to obtain the prepared StyleGAN2 generator; then, sampling a set of latent vectors z from the standard normal distribution, and performing parameterization and initialization processing on the sampled latent vectors z to ensure that the dimension of the latent vector z is consistent with the input dimension of the StyleGAN2 generator, thereby obtaining the initialized latent vector z0.

[0018] Preferably, step (3) specifically includes the following sub-steps:

[0019] (3-1) Input the latent vector z and the target semantic description text s into the StyleGAN2 generator G to generate a disguised sample image x. Camouflaged =G(z,s);

[0020] (3-2) Input the camouflaged sample image generated in step (3-1) into the CLIP model to extract the image embedding vector v corresponding to the camouflaged sample image. image Based on the semantic feature vector v obtained in step (1) textCalculate CLIP similarity loss L CLIP Based on the camouflaged sample image x obtained in step (3-1) Camouflaged Calculate the L2 regularization loss L L2 ) and content loss L content And based on CLIP similarity loss L CLIP L2 regularization loss L L2 and content loss L content Obtain the total loss L:

[0021] L=λ1L CLIP +λ2L L2 +λ3L content

[0022] Where λ1, λ2, and λ3 are loss weight coefficients, their values ​​range from 0 to 1, and the sum of the three is equal to 1;

[0023] (3-3) Based on the total loss obtained in step (3-2), the latent vector z is optimized using the backpropagation method, and the Adam optimizer is used to iteratively update the optimized latent vector z until the predetermined number of iterations is reached, thereby obtaining the final optimized latent vector z. final .

[0024] Preferably, CLIP similarity loss L CLIP It is calculated using the following formula:

[0025] L CLIP =1-cos(v text v image )

[0026] L2 regularization loss L L2 It is calculated using the following formula:

[0027]

[0028] Where, x Baseline =G(z0, s) is the baseline image generated by inputting the initialized latent vector z0 and the target semantic description text s into the StyleGAN2 generator.

[0029] Content loss L content It is calculated using the following formula:

[0030]

[0031] Where φ l φ represents the feature extraction function of the l-th layer in the VGG network model downloaded from the public model library. l (x Camouflaged ) represents the input camouflaged sample image xCamouflaged The features extracted from the l-th layer in the VGG network model, φ l (x Baseline ) represents the input reference image x Baseline The features extracted in the l-th layer of the VGG network, where l∈{relu1_2, relu2_2, relu3_3}, where relu1_2 represents the second layer of the first convolutional block in the VGG network, used to extract low-level features in the image, relu2_2 represents the second layer of the second convolutional block in the VGG network, used to extract high-level image features, and relu3_3 represents the third layer of the third convolutional block in the VGG network, used to extract abstract and complex image features.

[0032] Preferably, the StyleGAN2 generative adversarial network model includes a StyleGAN2 generator, a StyleGAN2 discriminator, and a semantic constraint module.

[0033] The specific structure of the StyleGAN2 generator is as follows:

[0034] The first layer is a mapping network, whose input is the latent variable z. This mapping network maps the latent vector z to the latent space w through 8 fully connected layers, and the final output is the latent representation of the latent space w.

[0035] The second layer is a synthesis network, whose input is the latent representation of the latent space w output by the first layer. The synthesis network first upsamples the latent representation layer by layer to generate a high-resolution image; then it extracts image features from the high-resolution image through multiple convolutional layers, and then uses the latent space w to perform stylization processing on the extracted image features. Finally, it injects noise into the stylized image features, and the final output is the image with injected noise.

[0036] The semantic constraint module is embedded in a layer of the synthesis network of the StyleGAN2 generator, and its specific structure is as follows:

[0037] The first layer is a text encoder, whose input is the target semantic description text s. This text encoder uses the CLIP model to encode the input target semantic description text s, and the final output is a semantic feature vector v. text ;

[0038] The second layer is the image feature extractor, which takes as input the image features output from a layer in the synthesis network of the StyleGAN2 generator. This image feature extractor normalizes the input image features and then inputs the normalized image features into the CLIP model for encoding. The final output is the image embedding vector v. image ;

[0039] The third layer is the similarity calculation module, whose input is the semantic feature vector v. text and image embedding vector v image This similarity calculation module calculates the semantic feature vector v using the CLIP model. text and image embedding vector v image The similarity score is used as the final output.

[0040] Preferably, the StyleGAN2 discriminator employs a multi-scale feature extraction structure, the specific structure of which is as follows:

[0041] The first layer is the convolutional feature extraction layer. Its input is the image with injected noise output by the StyleGAN2 generator. This convolutional feature extraction layer extracts low-level to high-level features from the image with injected noise layer by layer through multiple convolutional layers. These features are mapped to the corresponding feature maps through convolution operations, and the spatial resolution of these feature maps is reduced through max pooling operations. Finally, multiple feature maps are output to capture image features at different scales.

[0042] The second layer is a fully connected layer, whose input is multiple feature maps output by the convolutional feature extraction layer. This fully connected layer uses the fully connected network to calculate the real or fake probability corresponding to each feature map, which is used to determine the authenticity of the image after noise injection output by the StyleGAN2 generator, and finally outputs the discrimination result.

[0043] Preferably, the StyleGAN2 generative adversarial network model is trained through the following steps:

[0044] (a) Obtain the speed limit traffic sign image dataset, preprocess the speed limit traffic sign image dataset, and divide the preprocessed speed limit traffic sign image dataset to obtain the training set, validation set and test set.

[0045] (b) Randomly initialize the parameters in the StyleGAN2 generator G and StyleGAN2 discriminator D of the StyleGAN2 generative adversarial network model to obtain the initialized StyleGAN2 generative adversarial network.

[0046] (c) Obtain multiple speed limit traffic sign images in batches from the training set obtained in step (a) as input x real .

[0047] (d) Randomly sample the latent vector z and input it into the StyleGAN2 generator G in the StyleGAN2 generative adversarial network initialized in step (b) to generate fake images x. fake =G(z).

[0048] (e) The input x obtained in step (c) realThe fake image x obtained in step (d) fake Input the StyleGAN2 discriminator D to obtain the loss function L:

[0049] (f) Based on the loss function obtained in step (e), the StyleGAN2 generative adversarial network model is iteratively trained using the backpropagation method until the StyleGAN2 generative adversarial network model converges, thereby obtaining the initially trained StyleGAN2 generative adversarial network model.

[0050] (g) Use the test set obtained in step (a) to validate the StyleGAN2 generative adversarial network model initially trained in step (f) until the performance of the StyleGAN2 generative adversarial network model is stable on the test set, so as to obtain the trained StyleGAN2 generative adversarial network model.

[0051] Preferably, step (a) specifically involves first acquiring a traffic sign image dataset, and then selecting multiple speed limit traffic sign images from it to form a speed limit traffic sign image dataset. Next, the acquired speed limit traffic sign image dataset undergoes preprocessing operations including size normalization, color space conversion, and noise removal to obtain a preprocessed speed limit traffic sign image dataset. The preprocessed speed limit traffic sign image dataset is then divided into a training set, a validation set, and a test set in a 7:2:1 ratio.

[0052] Step (b) involves randomly initializing the parameters of the StyleGAN2 generator G and the StyleGAN2 discriminator D. The mapping network and synthesis network of the StyleGAN2 generator G, as well as the convolutional layer and fully connected layer of the StyleGAN2 discriminator D, are initialized using the standard initialization method to obtain the initialized StyleGAN2 generative adversarial network.

[0053] Step (e) specifically uses the following formula:

[0054] L=λ1L adv +λ2L L2 +λ3L content

[0055] Where λ1, λ2, and λ3 represent loss weight coefficients, their values ​​range from 0 to 1, and the sum of the three is 1. adv Indicating resistance to loss, L content Indicates content loss, L L2 Indicates the L2 regularization loss;

[0056] Combat loss L adv It is calculated using the following formula:

[0057]

[0058] in This indicates that the distribution p is from the real data distribution data Medium-sampled real image x real Calculate the expected value; D(x) real ) represents the discriminator's judgment of the real image x. real The output measures the probability of it being a real image; Indicates the distribution p from the potential space z The expected value of the sampled latent vector z is calculated; D(x) fake ) indicates that the discriminator evaluates the generated image x. fake The output measures the probability that it is a fake image.

[0059] L2 regularization loss L L2 It is calculated using the following formula:

[0060]

[0061] Content loss L content It is calculated using the following formula:

[0062]

[0063] According to another aspect of the present invention, a speed limit traffic sign camouflage sample generation system based on StyleGAN2 and CLIP is provided, comprising:

[0064] The first module loads the text encoder of the CLIP model, obtains the target semantic description text s input by the user, and uses the text encoder to parse and process the target semantic description text s to obtain the semantic feature vector v. text .

[0065] The second module is used to load the pre-trained StyleGAN2 generative adversarial network model and initialize the loaded StyleGAN2 generative adversarial network latent vectors to obtain the initialized latent vector z0.

[0066] The third module is used to calculate the loss function using the CLIP model and the pre-trained StyleGAN2 generator in the StyleGAN2 generative adversarial network model loaded in the second module. This loss function is then used to iteratively optimize the latent vector z obtained in the second module (in the first iteration, this latent vector z is the initialized latent vector z0 obtained in the second module) until the iteration termination condition is met, resulting in the final optimized latent vector z. final .

[0067] The fourth module is used to process the final optimized latent vector z obtained in the third module. final Input the StyleGAN2 generator from the pre-trained StyleGAN2 generative adversarial network model to obtain spoofed sample images.

[0068] In summary, compared with the prior art, the above-described technical solutions conceived by this invention can achieve the following beneficial effects:

[0069] 1. Since the present invention adopts steps (1) to (3), it can accurately control the key features of the generated samples and improve the generalization ability of the generated samples by introducing semantic constraints and optimizing the generation process of potential vectors. Therefore, it can solve the technical problem that the attack effect of existing methods is highly dependent on the parameters of the target model, resulting in poor generalization ability.

[0070] 2. By employing steps (3-1) to (3-3), this invention improves the concealment of generated samples by designing highly concealed camouflage samples and comprehensively optimizing the loss function, making them visually highly similar to real traffic signs, but difficult to be recognized by autonomous driving systems. Therefore, it can solve the technical problem that existing methods are easily detected or recognized by autonomous driving systems and cannot effectively test the anti-interference ability of the system.

[0071] 3. Because the present invention adopts steps (3) to (4), it can accurately simulate the key features of the target category and introduce diversity by increasing pattern changes to meet the needs of complex traffic scenarios. Therefore, it can solve the technical problem that the existing methods are monotonous, lack comprehensive consideration of the key features and diverse styles of traffic signs, and are difficult to adapt to complex actual traffic environments. Attached Figure Description

[0072] Figure 1 This is a flowchart of the method for generating disguised speed limit traffic sign samples based on StyleGAN2 and CLIP according to the present invention;

[0073] Figure 2 This is a schematic diagram of the structure of the StyleGAN2 generative adversarial network model used in this invention;

[0074] Figure 3 This is a flowchart of the speed limit traffic sign camouflage sample generation model based on StyleGAN2 and CLIP of the present invention. Detailed Implementation

[0075] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0076] The basic idea of ​​this invention is to combine the StyleGAN2 generative adversarial network and the CLIP model, and introduce a multi-objective loss optimization framework to achieve efficient generation of disguised speed limit traffic sign samples. StyleGAN2 provides powerful image generation capabilities, and can generate high-quality speed limit traffic sign samples through its latent variable space. The CLIP model provides guidance through its semantic understanding capabilities, helping the StyleGAN2 generative adversarial network optimize the image generation process, so that the generated disguised samples are visually consistent with the key features of the speed limit traffic sign, and conform to the target semantic description text s, thereby ensuring the semantic consistency of the generated disguised samples. The disguised sample generation method proposed in this invention includes three core modules: semantic constraint, latent variable optimization, and multi-loss fusion. The semantic constraint module uses the CLIP model to define the semantic similarity between the generated disguised samples and the target semantic description text s, and calculates the similarity in the embedding space to bring the disguised samples closer to the target semantic description text s. The latent variable optimization, by limiting the adjustment range of latent variables, avoids deviation from the basic features of the speed limit traffic sign when generating disguised samples, while retaining certain semantic camouflage characteristics. The multi-loss fusion optimization framework combines CLIP loss, L2 regularization loss, and content loss to balance the semantic similarity, visual consistency, and generation quality of samples. This invention's camouflaged sample generation method aims to conduct robustness testing on the perception module of an autonomous driving system, revealing its potential recognition vulnerabilities and providing a basis for improving the safety of autonomous driving systems.

[0077] Experimental results demonstrate that this invention exhibits significant advantages in terms of semantic consistency, diversity, and attack effectiveness in generating camouflaged samples. In the experiments conducted using a large number of standard speed limit sign images, two types of camouflaged samples were successfully generated: one based on semantic similarity and the other based on pattern transformation. Testing on these generated camouflaged samples revealed that they could successfully mislead autonomous driving systems into recognizing the camouflaged samples as speed limit traffic signs. This indicates that the method of this invention can effectively interfere with existing traffic sign recognition systems, revealing potential security vulnerabilities in autonomous driving perception modules. In conclusion, the camouflaged sample generation method of this invention provides a novel tool for the safety assessment of intelligent driving systems.

[0078] This invention proposes a method and system for generating camouflaged speed limit traffic sign samples by combining StyleGAN2 and CLIP models. Camouflaged samples represent a novel visual attack scheme, a type of targeted attack, where the system identifies the camouflaged sample as a specified target category. The principle behind the camouflaged sample design is to incorporate some features of the target category into the sample using certain image generation models, while the human eye will not recognize it as the target category. When the target category is a speed limit traffic sign, the camouflaged sample induces the system to misidentify an image of a non-speed limit traffic sign as a speed limit traffic sign. Currently, methods for generating camouflaged samples mainly fall into two categories: the first is based on semantic similarity, which generates camouflaged samples highly similar to the target category image by deforming or replacing only the key semantic features of the target category; the second is based on pattern transformation, which retains only the basic visual features of the target category image and transforms some patterns in the image to create abstract samples.

[0079] like Figure 1 As shown, this invention provides a method for generating camouflaged speed limit traffic sign samples based on StyleGAN2 and CLIP, comprising the following steps:

[0080] (1) Load the CLIP model's text encoder to obtain the target semantic description text s input by the user, and use the text encoder to parse the target semantic description text s to obtain the semantic feature vector v. text (It is used to guide the generation of spoofed samples).

[0081] Specifically, this step involves first loading the text encoder of the CLIP model, then obtaining the target semantic description text s input by the user, and inputting the target semantic description text s into the text processing module of the CLIP model for standardization to obtain a standardized token matrix. Next, the standardized token matrix is ​​input into the previously loaded CLIP model's text encoder for encoding, thereby obtaining the semantic feature vector v of the target semantic description text s. text .

[0082] In this step, for the spoofed samples based on semantic similarity, the semantic feature vector v obtained through parsing is... text The core features of the target semantic description text s will be strictly matched to ensure that the generated camouflaged sample closely resembles the speed limit traffic sign in overall visual effect and structure, but appropriate transformations will be made in details to achieve the camouflage effect; for camouflaged samples based on pattern transformation, the semantic feature vector v obtained by parsing will be... text Allowing for a wider range of semantic matching, StyleGAN2 generative adversarial networks are guided to generate camouflaged samples that are similar in overall structure but differ significantly in detail, in order to introduce more diverse and creative designs.

[0083] The advantage of this step is that by using the CLIP model's text encoder to finely process the target semantic description text s, it can accurately extract semantic feature vectors that meet different needs, corresponding to camouflaged samples based on semantic similarity and camouflaged samples based on pattern transformation. This greatly expands the flexibility and adaptability of camouflaged sample generation and meets the needs of diverse application scenarios.

[0084] (2) Load the pre-trained StyleGAN2 generative adversarial network model and initialize the loaded StyleGAN2 generative adversarial network potential vector to obtain the initialized potential vector z0.

[0085] Specifically, this step involves first calling a pre-trained StyleGAN2 generative adversarial network model, loading the corresponding StyleGAN2 network structure and weight parameters, and switching the StyleGAN2 generator in the StyleGAN2 generative adversarial network to evaluation mode (to ensure that the randomness during training is avoided during the generation process) to obtain the prepared StyleGAN2 generator. Then, a set of latent vectors z is sampled from a standard normal distribution. The sampled latent vectors z are then parameterized (representing the randomly sampled latent vectors z as an optimizable model parameter so that it can accept gradient updates in the optimization step of the subsequent generation process) and initialized (when generating latent vectors, they are initialized to random values ​​that conform to a standard normal distribution to ensure that the latent vectors have uniform random distribution characteristics and avoid the bias that may be caused by manually setting initial values) to ensure that the dimension of the latent vector z is consistent with the input dimension of the StyleGAN2 generator, thereby obtaining the initialized latent vector z0.

[0086] (3) Calculate the loss function using the CLIP model and the StyleGAN2 generator in the pre-trained StyleGAN2 generative adversarial network model loaded in step (2), and use this loss function to iteratively optimize the latent vector z obtained in step (2) (in the first iteration, the latent vector z is the initialized latent vector z0 obtained in step (2)) until the iteration termination condition is met, so as to obtain the final optimized latent vector z. final ;

[0087] This step specifically includes the following sub-steps:

[0088] (3-1) Input the latent vector z and the target semantic description text s into the StyleGAN2 generator G to generate a disguised sample image x. Camouflaged =G(z,s);

[0089] (3-2) Input the camouflaged sample image generated in step (3-1) into the CLIP model to extract the image embedding vector v corresponding to the camouflaged sample image. image Based on the semantic feature vector v obtained in step (1) text Calculate CLIP similarity loss L CLIP (It is used to measure the degree of matching between the camouflaged sample and the target semantic description text s), based on the camouflaged sample image x obtained in step (3-1). Camouflaged Calculate the L2 regularization loss L L2 (It is used to calculate the difference between the generated image and the original image in pixel space, limiting excessive deviation between the generated image and the original image, and maintaining visual consistency) and content loss L content (It extracts high-level features of images by using a VGG network model downloaded from a public model library to preserve the overall structure of the image), and calculates the CLIP similarity loss L. CLIP L2 regularization loss L L2 and content loss L content Obtain the total loss L:

[0090] L=λ1L CLIP +λ2L L2 +λ3L content

[0091] Where λ1, λ2, and λ3 are loss weight coefficients, their values ​​range from 0 to 1, and the sum of the three is equal to 1. Preferably, λ1 = 0.6, λ2 = 0.2, and λ3 = 0.2.

[0092] CLIP similarity loss L CLIP It is calculated using the following formula:

[0093] L CLIP =1-cos(v text v image )

[0094] L2 regularization loss L L2 It is calculated using the following formula:

[0095]

[0096] Where, x Baseline =G(z0, s) is the baseline image generated by inputting the initialized latent vector z0 and the target semantic description text s into the StyleGAN2 generator.

[0097] Content loss L content It is calculated using the following formula:

[0098]

[0099] Where, φ l φ represents the feature extraction function of the l-th layer in the VGG network model downloaded from the public model library. l (x Camouflaged ) represents the input camouflaged sample image x Camouflaged The features extracted from the l-th layer in the VGG network model, φ l (x Baseline ) represents the input reference image x Baseline The features extracted in the l-th layer of the VGG network, where l∈{relu1_2, relu2_2, relu3_3}, where relu1_2 represents the second layer of the first convolutional block in the VGG network, used to extract low-level features in the image, relu2_2 represents the second layer of the second convolutional block in the VGG network, used to extract high-level image features, and relu3_3 represents the third layer of the third convolutional block in the VGG network, used to extract abstract and complex image features.

[0100] (3-3) Based on the total loss obtained in step (3-2), the latent vector z is optimized using the backpropagation method, and the Adam optimizer is used to iteratively update the optimized latent vector z until the predetermined number of iterations is reached, thereby obtaining the final optimized latent vector z. final .

[0101] Specifically, the predetermined number of iterations in this invention is 200. In each iteration, the potential vector z is updated based on the loss, thereby gradually optimizing the potential vector z.

[0102] The advantage of the sub-steps (3-1) to (3-3) above is that CLIP loss, as a key semantic constraint module, improves the semantic feature vector v. text The generated image is matched to ensure it meets predetermined semantic requirements, thus ensuring that the generated camouflaged sample not only has visual consistency but also accurately captures the features of the target semantic description text s. Furthermore, by fusing multi-target losses, the authenticity, semantic consistency, and visual consistency of the image are comprehensively considered, ensuring that the generated camouflaged sample is of high quality and strong effectiveness.

[0103] (4) The final optimized potential vector z obtained in step (3) final Input the StyleGAN2 generator from the pre-trained StyleGAN2 generative adversarial network model to obtain spoofed sample images.

[0104] like Figure 2 As shown, the StyleGAN2 generative adversarial network model used in this invention includes a StyleGAN2 generator, a StyleGAN2 discriminator, and a semantic constraint module.

[0105] The specific structure of the StyleGAN2 generator is as follows:

[0106] The first layer is a mapping network, whose input is the latent variable z. This mapping network maps the latent vector z to the latent space w through 8 fully connected layers, and the final output is the latent representation of the latent space w.

[0107] The second layer is a synthesis network, whose input is the latent representation of the latent space w output by the first layer. The synthesis network first upsamples the latent representation layer by layer to generate a high-resolution image; then it extracts image features from the high-resolution image through multiple convolutional layers, and then uses the latent space w to perform stylization processing on the extracted image features. Finally, it injects noise into the stylized image features (the purpose of which is to make the generated image more natural), and the final output is the image with injected noise.

[0108] The semantic constraint module is embedded in a layer of the synthesis network of the StyleGAN2 generator, and its specific structure is as follows:

[0109] The first layer is a text encoder, whose input is the target semantic description text s. This text encoder uses the CLIP model to encode the input target semantic description text s, and the final output is a semantic feature vector v. text ;

[0110] The second layer is the image feature extractor, which takes as input the image features output from a layer in the synthesis network of the StyleGAN2 generator. This feature extractor normalizes the input image features (by calculating the mean and standard deviation of the image features, adjusting the feature values ​​to a fixed range to balance the numerical distribution of different image features). The normalized image features are then input into the CLIP model for encoding, and the final output is the image embedding vector v. image ;

[0111] The third layer is the similarity calculation module, whose input is the semantic feature vector v. text and image embedding vector v image This similarity calculation module calculates the semantic feature vector v using the CLIP model. text and image embedding vector v image The similarity score (used for optimizing the StyleGAN2 generator loss function) is used as the final output.

[0112] The StyleGAN2 discriminator employs a multi-scale feature extraction structure, the specific structure of which is as follows:

[0113] The first layer is the convolutional feature extraction layer, whose input is the image with injected noise output from the StyleGAN2 generator. This convolutional feature extraction layer extracts low-level to high-level features from the image with injected noise layer by layer through multiple convolutional layers. These features are mapped to the corresponding feature maps through convolution operations, and the spatial resolution of these feature maps is reduced through max pooling operations (the purpose of which is to reduce computational complexity). Finally, multiple feature maps are output to capture image features at different scales.

[0114] The second layer is a fully connected layer, whose input is multiple feature maps output by the convolutional feature extraction layer. This fully connected layer uses the fully connected network to calculate the real or fake probability corresponding to each feature map, which is used to determine the authenticity of the image after noise injection output by the StyleGAN2 generator, and finally outputs the discrimination result.

[0115] like Figure 3 As shown, the StyleGAN2 generative adversarial network model of the present invention is trained through the following steps:

[0116] (a) Obtain the speed limit traffic sign image dataset, preprocess the speed limit traffic sign image dataset, and divide the preprocessed speed limit traffic sign image dataset to obtain the training set, validation set and test set.

[0117] Specifically, this step first acquires a traffic sign image dataset, from which multiple speed limit traffic sign images (containing different speed limit values, such as "20", "30", "50", "60", etc., and distributed under different shooting conditions, lighting, and background environments) are selected to form a speed limit traffic sign image dataset. Then, the acquired speed limit traffic sign image dataset undergoes preprocessing operations including size normalization (e.g., adjusting to 256×256 pixels), color space conversion (specifically, converting the speed limit traffic sign images from BGR format to RGB format), and noise removal (to ensure the quality and consistency of the speed limit traffic sign images) to obtain a preprocessed speed limit traffic sign image dataset. The preprocessed speed limit traffic sign image dataset is then divided into a training set, a validation set, and a test set in a 7:2:1 ratio.

[0118] More specifically, the training set in this invention is used to adjust parameters such as trainable weights and biases in the StyleGAN2 generative adversarial network model, while the validation set is used to adjust hyperparameters such as the learning rate of the StyleGAN2 generative adversarial network model. The test set does not participate in the training of the model and is used to test the final generation effect of the StyleGAN2 generative adversarial network model.

[0119] (b) Randomly initialize the parameters in the StyleGAN2 generator G and StyleGAN2 discriminator D of the StyleGAN2 generative adversarial network model to obtain the initialized StyleGAN2 generative adversarial network.

[0120] Specifically, this step involves randomly initializing the parameters of the StyleGAN2 generator G and the StyleGAN2 discriminator D. The mapping and synthesis networks of the StyleGAN2 generator G, as well as the convolutional and fully connected layers of the StyleGAN2 discriminator D, are initialized using standard methods to obtain the initialized StyleGAN2 generative adversarial network.

[0121] (c) Obtain multiple speed limit traffic sign images in batches from the training set obtained in step (a) as input x real .

[0122] Specifically, this step involves acquiring multiple speed limit traffic sign images from the training set in batches, with each batch containing 16 images.

[0123] (d) Randomly sample the latent vector z and input it into the StyleGAN2 generator G in the StyleGAN2 generative adversarial network initialized in step (b) to generate fake images x. fake =G(z).

[0124] (e) The input x obtained in step (c) real The fake image x obtained in step (d) fake Input the StyleGAN2 discriminator D to obtain the loss function L:

[0125] L=λ1L adv +λ2L L2 +λ3L content

[0126] Where λ1, λ2, and λ3 represent loss weight coefficients, their values ​​range from 0 to 1, and the sum of the three is 1. Preferably, λ1 = 0.7, λ2 = 0.2, λ3 = 0.1, and L adv Indicating resistance to loss, L content Indicates content loss, L L2 Indicates the L2 regularization loss;

[0127] Combat loss L adv It is calculated using the following formula:

[0128]

[0129] in This indicates that the distribution p is from the real data distribution dataMedium-sampled real image x real Calculate the expected value; D(x) real ) represents the discriminator's judgment of the real image x. real The output measures the probability of it being a real image; Indicates the distribution p from the potential space z The expected value of the sampled latent vector z is calculated; D(x) fake ) indicates that the discriminator evaluates the generated image x. fake The output measures the probability that it is a fake image.

[0130] L2 regularization loss L L2 It is calculated using the following formula:

[0131]

[0132] Content loss L content It is calculated using the following formula:

[0133]

[0134] The advantage of this step is that by cleverly designing the loss function, comprehensively considering adversarial loss, L2 regularization loss, and content loss, the difference between the generated fake images and real images is measured from multiple dimensions, providing accurate feedback signals to the discriminator, prompting the generator to continuously optimize the generated fake images, so that the generated fake images are increasingly closer to the real images.

[0135] (f) Based on the loss function obtained in step (e), the StyleGAN2 generative adversarial network model is iteratively trained using the backpropagation method until the StyleGAN2 generative adversarial network model converges, thereby obtaining the initially trained StyleGAN2 generative adversarial network model.

[0136] (g) Use the test set obtained in step (a) to validate the StyleGAN2 generative adversarial network model initially trained in step (f) until the performance of the StyleGAN2 generative adversarial network model is stable on the test set, so as to obtain the trained StyleGAN2 generative adversarial network model.

[0137] Specifically, the pre-trained StyleGAN2 generative adversarial network model can generate high-quality speed limit traffic sign images, providing a foundation for the subsequent generation of camouflage samples.

[0138] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for generating camouflaged speed limit traffic sign samples based on StyleGAN2 and CLIP, characterized in that, Includes the following steps: (1) Load the text encoder of the CLIP model to obtain the target semantic description text 𝑠 input by the user, and use the text encoder to parse the target semantic description text 𝑠 to obtain the semantic feature vector. ; (2) Load the pre-trained StyleGAN2 generative adversarial network model and initialize the loaded StyleGAN2 generative adversarial network latent vectors to obtain the initialized latent vectors. ; (3) Calculate the loss function using the CLIP model and the StyleGAN2 generator in the pre-trained StyleGAN2 generative adversarial network model loaded in step (2), and use this loss function to iteratively optimize the latent vector 𝑧 obtained in step (2) until the iteration termination condition is met, so as to obtain the final optimized latent vector. ; (4) The final optimized latent vector obtained in step (3) Input the StyleGAN2 generator from the pre-trained StyleGAN2 generative adversarial network model to obtain spoofed sample images.

2. The method for generating camouflaged speed limit traffic sign samples based on StyleGAN2 and CLIP according to claim 1, characterized in that, Step (1) specifically involves first loading the text encoder of the CLIP model, then obtaining the target semantic description text 𝑠 input by the user, and inputting the target semantic description text 𝑠 into the text processing module of the CLIP model for standardization processing to obtain a standardized token matrix; then, inputting the standardized token matrix into the previously loaded text encoder of the CLIP model for encoding to obtain the semantic feature vector of the target semantic description text 𝑠. .

3. The method for generating camouflaged speed limit traffic sign samples based on StyleGAN2 and CLIP according to claim 1 or 2, characterized in that, Step (2) specifically involves: first, calling the pre-trained StyleGAN2 generative adversarial network model, loading the corresponding StyleGAN2 generative adversarial network structure and weight parameters, and switching the StyleGAN2 generator in the StyleGAN2 generative adversarial network to evaluation mode to obtain the prepared StyleGAN2 generator; then, sampling a set of latent vectors 𝑧 from the standard normal distribution, and performing parameterization and initialization processing on the sampled latent vectors 𝑧 to ensure that the dimension of the latent vectors 𝑧 is consistent with the input dimension of the StyleGAN2 generator, thereby obtaining the initialized latent vectors. .

4. The method for generating camouflaged speed limit traffic sign samples based on StyleGAN2 and CLIP according to claim 3, characterized in that, Step (3) specifically includes the following sub-steps: (3-1) Input the latent vector 𝑧 and the target semantic description text 𝑠 into the StyleGAN2 generator 𝐺 to generate camouflaged sample images. ; (3-2) Input the camouflaged sample image generated in step (3-1) into the CLIP model to extract the image embedding vector corresponding to the camouflaged sample image. Based on the semantic feature vector obtained in step (1) Calculate CLIP similarity loss Based on the camouflaged sample image obtained in step (3-1) Calculate L2 regularization loss ) and content loss And based on CLIP similarity loss L2 regularization loss and content loss Obtain total loss : ， in , , These are all loss weighting coefficients, with values ​​ranging from 0 to 1, and the sum of the three is equal to 1; (3-3) Based on the total loss obtained in step (3-2), the latent vector 𝑧 is optimized using the backpropagation method, and the Adam optimizer is used to iteratively update the optimized latent vector 𝑧 until the predetermined number of iterations is reached, thereby obtaining the final optimized latent vector. .

5. The method for generating camouflaged speed limit traffic sign samples based on StyleGAN2 and CLIP according to claim 4, characterized in that, CLIP Similarity Loss It is calculated using the following formula: ， L2 regularization loss It is calculated using the following formula: ， in, It is the initialized latent vector The baseline image generated after inputting the target semantic description text 𝑠 into the StyleGAN2 generator; Content loss It is calculated using the following formula: ， in This represents the feature extraction function of the l-th layer in the VGG network model downloaded from the public model library. This represents the spoofed sample image of the input. Features extracted from the l-th layer in the VGG network model Represents the input reference image The features extracted in the l-th layer of the VGG network, where l∈{relu1_2, relu2_2, relu3_3}, where relu1_2 represents the second layer of the first convolutional block in the VGG network, used to extract low-level features in the image, relu2_2 represents the second layer of the second convolutional block in the VGG network, used to extract high-level image features, and relu3_3 represents the third layer of the third convolutional block in the VGG network, used to extract abstract and complex image features.

6. The method for generating camouflaged speed limit traffic sign samples based on StyleGAN2 and CLIP according to claim 5, characterized in that, The StyleGAN2 generative adversarial network model includes a StyleGAN2 generator, a StyleGAN2 discriminator, and a semantic constraint module. The specific structure of the StyleGAN2 generator is as follows: The first layer is a mapping network, whose input is the latent variable 𝑧. This mapping network maps the latent vector 𝑧 to the latent space 𝑤 through 8 fully connected layers, and the final output is the latent representation of the latent space 𝑤. The second layer is a synthetic network, whose input is the latent representation of the latent space 𝑤 output by the first layer. The synthetic network first upsamples the latent representation layer by layer to generate a high-resolution image; then it extracts the image features from the high-resolution image through multiple convolutional layers, and then uses the latent space 𝑤 to perform stylization processing on the extracted image features. Finally, it injects noise into the stylized image features, and the final output is the image with injected noise. The semantic constraint module is embedded in a layer of the synthesis network of the StyleGAN2 generator, and its specific structure is as follows: The first layer is a text encoder, whose input is the target semantic description text *i*. This text encoder uses the CLIP model to encode the input target semantic description text *i*, and the final output is a semantic feature vector. ; The second layer is the image feature extractor, which takes as input the image features output from a layer in the synthesis network of the StyleGAN2 generator. This image feature extractor normalizes the input image features and then inputs the normalized image features into the CLIP model for encoding, finally outputting an image embedding vector. ; The third layer is the similarity calculation module, whose input is... and image embedding vector This similarity calculation module calculates using the CLIP model. and image embedding vector The similarity score is used as the final output.

7. The method for generating camouflaged speed limit traffic sign samples based on StyleGAN2 and CLIP according to claim 6, characterized in that, The StyleGAN2 discriminator employs a multi-scale feature extraction structure, the specific structure of which is as follows: The first layer is the convolutional feature extraction layer. Its input is the image with injected noise output by the StyleGAN2 generator. This convolutional feature extraction layer extracts low-level to high-level features from the image with injected noise layer by layer through multiple convolutional layers. These features are mapped to the corresponding feature maps through convolution operations, and the spatial resolution of these feature maps is reduced through max pooling operations. Finally, multiple feature maps are output to capture image features at different scales. The second layer is a fully connected layer, whose input is multiple feature maps output by the convolutional feature extraction layer. This fully connected layer uses the fully connected network to calculate the real or fake probability corresponding to each feature map, which is used to determine the authenticity of the image after noise injection output by the StyleGAN2 generator, and finally outputs the discrimination result.

8. The method for generating camouflaged speed limit traffic sign samples based on StyleGAN2 and CLIP according to claim 1, characterized in that, The StyleGAN2 generative adversarial network model is trained through the following steps: (a) Obtain the speed limit traffic sign image dataset, preprocess the speed limit traffic sign image dataset, and divide the preprocessed speed limit traffic sign image dataset to obtain the training set, validation set and test set; (b) Randomly initialize the StyleGAN2 generator 𝐺 and StyleGAN2 discriminator of the StyleGAN2 generative adversarial network model. The parameters in the code are used to obtain the initialized StyleGAN2 generative adversarial network; (c) Obtain multiple speed limit traffic sign images in batches from the training set obtained in step (a) as input. ; (d) Randomly sample the latent vector 𝑧 and input it into the StyleGAN2 generator 𝐺 in the StyleGAN2 generative adversarial network initialized in step (b) to generate fake images. ; (e) The input obtained in step (c) The fake image obtained in step (d) Input StyleGAN2 discriminator To obtain the loss function L: (f) Based on the loss function obtained in step (e), the StyleGAN2 generative adversarial network model is iteratively trained using the backpropagation method until the StyleGAN2 generative adversarial network model converges, thereby obtaining the initially trained StyleGAN2 generative adversarial network model. (g) Use the test set obtained in step (a) to validate the StyleGAN2 generative adversarial network model initially trained in step (f) until the performance of the StyleGAN2 generative adversarial network model is stable on the test set, so as to obtain the trained StyleGAN2 generative adversarial network model.

9. The method for generating camouflaged speed limit traffic sign samples based on StyleGAN2 and CLIP according to claim 8, characterized in that, Step (a) specifically involves first acquiring a traffic sign image dataset, then selecting multiple speed limit traffic sign images from it to form a speed limit traffic sign image dataset; then, performing preprocessing operations, including size normalization, color space conversion, and noise removal, on the acquired speed limit traffic sign image dataset to obtain a preprocessed speed limit traffic sign image dataset; and finally, dividing the preprocessed speed limit traffic sign image dataset into a training set, a validation set, and a test set in a 7:2:1 ratio. Step (b) involves modifying the StyleGAN2 generator 𝐺 and the StyleGAN2 discriminator. The parameters are randomly initialized, including the mapping network and synthesis network of the StyleGAN2 generator, and the StyleGAN2 discriminator. Both the convolutional and fully connected layers are initialized using standard methods to obtain the initialized StyleGAN2 generative adversarial network. Step (e) specifically uses the following formula: ， in , , This represents the loss weighting coefficient, which ranges from 0 to 1, and the sum of the three values ​​is 1. Indicating resistance to loss, Indicates content loss. Indicates the L2 regularization loss; Combating losses It is calculated using the following formula: ， in This indicates the distribution of real data. Medium-sampled real images Calculate the expected value; This indicates that the discriminator recognizes the real image. The output measures the probability of it being a real image; Indicates the distribution from the potential space Mid-sampled latent vector Calculate the expected value; This indicates that the discriminator evaluates the generated image. The output measures the probability that it is a fake image; L2 regularization loss It is calculated using the following formula: ， Content loss It is calculated using the following formula: 。 10. A system for generating camouflaged speed limit traffic sign samples based on StyleGAN2 and CLIP, characterized in that, include: The first module loads the text encoder of the CLIP model, obtains the target semantic description text 𝑠 input by the user, and uses the text encoder to parse and process the target semantic description text 𝑠 to obtain semantic feature vectors. ; The second module is used to load the pre-trained StyleGAN2 generative adversarial network model and initialize the loaded StyleGAN2 generative adversarial network's latent vectors to obtain the initialized latent vectors. ; The third module is used to calculate the loss function using the CLIP model and the pre-trained StyleGAN2 generator in the StyleGAN2 generative adversarial network model loaded in the second module. This loss function is then used to iteratively optimize the latent vector 𝑧 obtained in the second module until the iteration termination condition is met, resulting in the final optimized latent vector. ; The fourth module is used to process the final optimized latent vector obtained in the third module. Input the StyleGAN2 generator from the pre-trained StyleGAN2 generative adversarial network model to obtain spoofed sample images.

Images