A fine-tuning CLIP-based illegal outdoor advertisement identification method

Abstract

This invention relates to the field of multimodal computing and discloses a method for identifying illegal outdoor advertisements based on fine-tuned CLIP. The method involves acquiring image data of illegal outdoor advertisements, creating a dataset and dividing it into training sets A and B, and a validation set; constructing a fine-tuned CLIP network model; inputting training set A into the constructed network model to learn text features for each advertisement category, and saving the model's state dictionary at the end of training, naming it prompt.pth; inputting training set B into the constructed network model, comparing the learned text features with the image features in training set B to fine-tune the CLIP model's image encoder, and saving the model parameters with the highest accuracy during training, naming its file best.pth; inputting the image of the outdoor advertisement to be detected into the fine-tuned CLIP network model, loading two weight files into the model for inference detection, and determining whether it is illegal. Compared with existing technologies, this invention effectively improves the model's ability to classify images of illegal outdoor advertisements.

Description

Technical Field

[0001] This invention belongs to the field of multimodal technology, specifically a method for identifying illegal outdoor advertisements based on fine-tuned CLIP. Background Technology

[0002] With the rapid development of the internet, outdoor advertising has become an important means of commercial promotion and advertising, but it has also brought about problems such as the abuse of outdoor advertising. The existence of illegal outdoor advertisements can have a negative impact on the urban environment, damaging the aesthetics and cleanliness of public spaces. They can also lead to false advertising, misleading consumers, and privacy violations, harming the rights and interests of the public. Therefore, an effective mechanism for identifying illegal outdoor advertisements can be established to promptly detect and address such violations, maintaining the city's aesthetics and protecting public rights and safety.

[0003] To address the aforementioned problems, previous methods for identifying illegal outdoor advertisements have combined visual and content features of text. While this approach effectively identifies the textual content of illegal outdoor advertisements, it neglects the image content, which is clearly insufficient. CLIP, based on the multimodal domain, can simultaneously understand images and text and connect them. It can understand the content of images and generate relevant textual descriptions, and it can also understand the meaning of images based on textual descriptions. This cross-modal capability makes CLIP perform exceptionally well in image classification, image retrieval, and text classification. However, because the original CLIP model lacks precise text labels for illegal outdoor advertisements and the ability to handle fine-grained advertisement image classification, a fine-tuned CLIP method is used to address this issue, effectively improving the recognition capabilities for both images and text. Summary of the Invention

[0004] The purpose of this invention is to address the current incomplete identification of illegal outdoor advertising content by proposing a method for identifying illegal outdoor advertising based on fine-tuned CLIP. This method not only solves the problem of identifying both the images and text content of illegal outdoor advertisements, but also effectively improves the model's ability to classify images of illegal outdoor advertisements.

[0005] Technical Solution: This invention proposes a method for identifying illegal outdoor advertising based on fine-tuned CLIP, specifically including the following steps:

[0006] Step 1: Obtain image data of illegal outdoor advertisements through a web spider, manually clean the image data, and create a dataset. Divide the dataset into a training set and a validation set. At the same time, use different data augmentation methods to augment the training set and generate training set A and training set B for use in the training phase.

[0007] Step 2: Build the fine-tuned CLIP network model. The fine-tuned CLIP network model uses ViT-B / 16 as the backbone network, adds a cue learner, and performs two-stage training simultaneously.

[0008] Step 3: First training phase: Read training set A from the illegal outdoor advertising dataset into the network model built in step 2. During the model training process, learn the text features of each advertising category. When the training ends, save the model's state dictionary and name it prompt.pth for use when reloading the model.

[0009] Step 4: Second training phase: Read training set B from the illegal outdoor advertising dataset into the network model built in step 2. Use the text features learned in step 3 and the image features in training set B to compare similarity, thereby fine-tuning the image encoder of the CLIP model. Save the model parameters with the highest accuracy during training and name the file best.pth.

[0010] Step 5: Read the image of the outdoor advertisement to be detected into the fine-tuned CLIP network model, load the weight files prompt.pth and best.pth from Step 3 and Step 4 into the model for inference detection, and determine whether there is a violation.

[0011] Furthermore, the specific method of step 1 is as follows:

[0012] Step 1.1: Use PyCharm programming code to collect illegal outdoor advertising image data from Baidu Images website online and store the image data in a local directory file;

[0013] Step 1.2: Manually filter the image data to remove irrelevant and low-quality image data;

[0014] Step 1.3: Divide the image data into training and validation sets in an 8:2 ratio for model training. Use different data augmentation methods to augment the training set, generating training set A and training set B respectively. Both sets use image resizing, normalization, and tensor conversion operations. Training set B adds random horizontal flipping, padding, random cropping, and random erasing operations.

[0015] Furthermore, the specific method for step 2 is as follows:

[0016] Step 2.1: The fine-tuned CLIP network model uses ViT-B / 16 as the backbone network, and downloads the corresponding pre-trained weight file. Two linear layers are added to linearly transform the input features, and two batch normalization layers are added for the input features. The model is specified to use batch normalization as the neck structure, and cross-entropy is selected as the loss function, as shown in the following formula:

[0017]

[0018] Where N is the number of images, y n p is the label of image n. n It is one of the element values ​​in the network output, and y n They are corresponding;

[0019] Using the calculated image resolution, model name ViT-B / 16, and stride size as parameters, the CLIP model is called, and the image encoder and text encoder components of the CLIP model are applied to the fine-tuned network model, while a cue learner is added.

[0020] Step 2.2: Build a cue learner. Provide a learnable context to initialize the text, and perform tokenization and initialization using random feature vectors from cue learning. Optimize the value of the feature vectors through backpropagation. Select "set cue before" in backpropagation for subsequent model training.

[0021] Step 2.3: In the first training phase, a contrastive loss function is used, and in the second training phase, a cross-entropy loss function is used. Adam is used as the optimizer in both phases. By adding a regularization term to the loss function, the magnitude of the parameters is limited. The derivation formula for Adam is shown below:

[0022]

[0023] m t =β1m t-1 +(1-β1)g t (3)

[0024]

[0025]

[0026]

[0027]

[0028] Wherein, the first gradient g t The loss function For θ t Find the partial derivative, θ t For the optimized parameters, the second term m t For time t, the first moment estimate of the gradient in momentum form, the third term u t The fourth term is the second moment estimate of the gradient in momentum form. This is the first-order moment estimate after bias correction, where β1 t It is beta 1 raised to the power of t, β1 and β2 are the hyperparameters for attenuating the first-order and second-order matrices of the gradient, respectively, and the fifth term. This is the second-order moment estimate after bias correction. The last term η and ε represent the learning rate and a constant added to prevent the denominator from being zero, respectively. Substituting these into the formula updates the parameters to obtain θ. t+1 .

[0029] Furthermore, the specific method of step 3 is as follows:

[0030] In the first training phase, the training set images from the illegal outdoor advertising dataset are loaded into the fine-tuned network model. The model is trained using a GPU, and the network model parameters are set as follows: 120 training epochs, 64 image samples in each training batch, an initial learning rate of 0.00035, a minimum learning rate of 0.000006, a weight decay of 0.0004 for the bias term, and Adam as the optimizer.

[0031] During model training, the text features of illegal outdoor advertisements are continuously optimized and learned. At the end of the first training stage, the model's state dictionary is saved and named prompt.pth for use when reloading the model. Since the entire program has not yet ended, the learned text features are still retained in memory for use in the next training stage.

[0032] Furthermore, the specific method of step 4 is as follows:

[0033] In the second training phase, the training set images from the illegal outdoor advertising dataset are loaded into the fine-tuned network model. For preprocessing the images in the training set, GPU training is used, and the network model parameters are set as follows: 120 training epochs, 64 image samples in each training batch, an initial learning rate of 0.000005, a linear learning rate warm-up method, 10 warm-up iterations, a warm-up factor of 0.1, a weight decay (L2 regularization) parameter of 0.0001, a bias term weight decay parameter of 0.0001, a bias term learning rate factor of 2, a learning rate adjustment step of [30, 50], and a learning rate adjustment multiplication factor of 0.1.

[0034] During model training, the image encoder of the model is fine-tuned by comparing the similarity between the images and the text features learned in step 3. Finally, the model parameters with the best accuracy are saved and named best.pth.

[0035]

[0036] In this context, TP represents true positives, TN represents true negatives, FP represents false positives, and FN represents false negatives.

[0037] Furthermore, the specific method of step 5 is as follows:

[0038] Step 5.1: Use precision as the evaluation metric for the model algorithm:

[0039]

[0040] Where TP represents the number of samples correctly predicted as positive, and FP represents the number of samples incorrectly predicted as positive.

[0041] Beneficial effects:

[0042] 1. The present invention combines text and images to learn and adjust the CLIP model through the above steps, and divides the training process into two stages. The advantage is that the text features learned in the first stage using the cue learning method can be compared with the image features in the second stage, thus providing additional contextual information for the second stage. This helps the model better understand semantics and background knowledge, and also enhances the model's generalization ability, enabling the model to better handle new and unseen samples.

[0043] 2. The benefit of introducing cue learning in the first stage of this invention is to enhance the model's understanding of the semantics and context of illegal outdoor advertising, while simultaneously addressing the problem of sample scarcity in illegal outdoor advertising and reducing its data requirements. This improves the model's generalization ability and performance. (See attached...) Figure 2 Analysis of the loss value variation curve shows that as the loss value decreases, the lower the loss value, the closer the model's prediction is to the true value. Therefore, a lower loss value indicates that the model's understanding and prediction ability of the input data is more accurate and precise.

[0044] 3. In the second stage, the text features learned in the first stage are used as contextual information to jointly learn and adjust the model with image features. This allows the model to more accurately match image features with text features, improving the correlation between visual and semantic information. Furthermore, text features can provide richer and more abstract semantic information because text can describe the attributes, relationships, and concepts of things. Using these text features as contextual information for the image encoder can better capture the semantic content in the image, as well as the concepts and semantic information related to the image. (See appendix...) Figure 3 Analysis of the loss value change curve shows that the loss value between text features and image features continuously decreases, indicating that the model gradually learns better visual and semantic correlations during fine-tuning. (See attached...) Figure 4 The analysis of the accuracy change curve shows that the continuously improving accuracy indicates that the model has been able to learn and capture the patterns and features in the training data well, and can accurately map the input to the correct output.

[0045] 4. Because the CLIP model possesses multimodal learning capabilities, it has a certain ability to classify both images and text. The fine-tuned CLIP model was trained using the methods described above, and combined with experimental data analysis, the fine-tuned CLIP model achieved high performance in the task of identifying illegal outdoor advertisements. It also solved the problem of previous methods only recognizing the text content of illegal outdoor advertisements. This provides a new method for identifying illegal outdoor advertisements in public places. Attached Figure Description

[0046] Figure 1 This is a flowchart of a method for identifying illegal outdoor advertisements based on fine-tuning CLIP, according to the present invention.

[0047] Figure 2 This is a graph showing the change in the loss function during the first stage of training the CLIP model for fine-tuning according to the present invention.

[0048] Figure 3 This is a graph showing the change in the loss function during the second stage of training the CLIP model in this invention.

[0049] Figure 4 This is a graph showing the accuracy changes during the second stage of training the CLIP model, which is fine-tuned according to the present invention. Detailed Implementation

[0050] To make the objectives, features, and advantages of this invention clearer, a method for identifying illegal outdoor advertising based on fine-tuned CLIP is described clearly and completely below with reference to specific examples. Furthermore, the specific examples described in this invention are for illustrative purposes only and are not intended to limit the invention.

[0051] The purpose of this invention is to address the incompleteness of current methods for identifying illegal outdoor advertising content. It proposes a method for identifying illegal outdoor advertising based on fine-tuned CLIP, which not only solves the identification of both the images and text content of illegal outdoor advertisements, but also addresses the lack of accurate text descriptions of illegal outdoor advertisements in the CLIP model by introducing cue learning. This effectively improves the model's fine-grained classification ability for illegal outdoor advertising images. This contributes to better maintaining the aesthetics of the city and protecting public rights and safety. The specific steps include:

[0052] Step 1: Obtain image data of illegal outdoor advertisements through a web spider, manually clean the image data, and create a dataset;

[0053] Step 1.1: First, examine the HTML source code of the Baidu Images website to determine the location and URL of the images. Then, write a web crawler using PyCharm, employing the requests library to send HTTP requests to retrieve the JSON data of the search results and extract the image URLs from it. Finally, send another HTTP request to download the images, naming them according to the outdoor advertising category name plus the sequence number of images crawled, and save them to a local file.

[0054] Step 1.2: By manually screening the image data, duplicate images, low-quality images, images with restricted copyright, and irrelevant or ambiguous images in the local files are removed. After screening, 100 images are retained for each outdoor advertising category.

[0055] Step 1.3: Using PyCharm, divide the image data into training and validation sets in an 8:2 ratio for model training and evaluation. In addition, use different data augmentation methods to augment the training set, generating training set A and training set B respectively. Both use image resizing, normalization, and tensor conversion operations. The difference is that training set B adds random horizontal flipping, padding, random cropping, and random erasing operations.

[0056] Step 2: Build the fine-tuned CLIP network model. The fine-tuned CLIP network model uses ViT-B / 16 as the backbone network, adds a cue learner, and performs two-stage training simultaneously.

[0057] Step 2.1: Before fine-tuning, the CLIP model pre-trained by OpenAI is used as the base model. The CLIP model uses ViT-B / 16 as the backbone network, and the corresponding pre-trained weight file is downloaded. In this implementation, the number of input channels is set to 768, and the number of input channels for the projection layer is set to 512 for subsequent calculations. Two new linear layers are added to the model in this fine-tuning process for the output of the classification task. These linear layers linearly transform the input features and generate output features that match the number of classes for the final classification prediction. Two batch normalization layers are also added, and their bias terms are set to non-trainable to reduce redundant parameters and increase model stability. These are applied to the input features to improve the model's training effect and generalization ability. Batch normalization is specified as the neck structure to further process and adjust the features extracted by the backbone network to improve the model's performance and expressive power. Finally, cross-entropy is chosen as the loss function, which can directly measure classification error, provide rich gradient information, and handle multi-class and soft targets in image classification tasks. Its formula is shown below:

[0058]

[0059] Where N is the number of images, y n p is the label of image n. n It is one of the element values ​​in the network output, and y n They are corresponding.

[0060] The input size of the model is determined by calculating the height and width resolution of the image. Specifically, the input image size is subtracted from the edge size, then divided by the stride size and incremented by one. Next, the CLIP model is invoked using the calculated resolution, the model name (ViT-B / 16), and the stride size as parameters. The fine-tuned model obtains the image encoder from the loaded CLIP model and creates a TextEncoder object as a text encoder using the CLIP model. The TextEncoder is used to convert the text into the input representation of the CLIP model. Finally, a cue learner is added for joint learning tasks between vision and text.

[0061] Step 2.2: Build the aforementioned cue learner. First, provide a context initialization text template, such as "An illegal outdoor advertising image of X XXX". Then, perform tokenization on this text. This process segments the initial context text into words or subwords and assigns a corresponding token to each word or subword. The tokenized result can be converted into a tensor and used in the model. Initialize the model using random feature vectors from cue learning to represent the context of each category. Set these as learnable parameters and optimize their values ​​through backpropagation. In backpropagation, set a pre-cue, which is a tensor formed by concatenating the prefix context of each category with the corresponding category feature vector, used for subsequent model training.

[0062] Step 2.3: In the first training phase, specifically Step 3 below, a contrastive loss function is used to learn feature representations by maximizing the similarity between matched sample pairs. The formula for the contrastive loss function is shown below:

[0063]

[0064] Where d = ||a n -b n ||2 represents the Euclidean distance between the features of the two samples, y is the label indicating whether the two samples match, y=1 means the two samples are similar or match, y=0 means they do not match, and margin is the set threshold.

[0065] In the second training phase, specifically step 4, cross-entropy is used as the loss function, as shown in formula (1) above. Adam is used as the optimizer in both phases: an adaptive learning rate adjustment method is employed, automatically adjusting the learning rate based on the gradient of each parameter. The concept of momentum is also introduced, considering previous gradient information to accelerate convergence and escape local optima when updating parameters. Furthermore, weight decay is used to control the magnitude of the model parameters. Adding a regularization term to the loss function limits the magnitude of the parameters, helping to prevent overfitting. The derivation formula for Adam is shown below:

[0066]

[0067] m t =β1m t-1 +(1-β1)g t (4)

[0068]

[0069]

[0070]

[0071]

[0072] Wherein, the first gradient g t The loss function For θ t Find the partial derivative, θ t For the optimized parameters, the second term m t For time t, the first moment estimate of the gradient in momentum form, the third term u t The fourth term is the second moment estimate of the gradient in momentum form. This is the first-order moment estimate after bias correction, where β1 t It is beta 1 raised to the power of t, β1 and β2 are the hyperparameters for attenuating the first-order and second-order matrices of the gradient, respectively, and the fifth term. This is the second-order moment estimate after bias correction. The last term η and ε represent the learning rate and a constant added to prevent the denominator from being zero, respectively. Substituting these into the formula updates the parameters to obtain θ. t+1 .

[0073] In the first training phase, specifically step 3, the model's parameters are iterated through, and parameters whose names contain "prompt_learner" are added to the optimizer's parameter list. In training phase 2, specifically step 4, the model's parameters are iterated through again, and if a parameter's name contains both "text_encoder" and "prompt_learner," its gradient calculation is frozen, and no optimization is performed.

[0074] Step 3: Enter the first training phase, read the training set A from the illegal outdoor advertising dataset into the network model built in Step 2. During the model training process, learn the text features of each advertising category. When the training ends, save the model's state dictionary and name it prompt.pth for use when reloading the model.

[0075] Step 3.1: Training set A undergoes the following operations: the input image size is adjusted to the specified size [256, 128], the image data is converted to tensor format, and standardized (mean subtracted and standard deviation divided) is performed to improve the model's training stability and convergence speed. Training set A is loaded into the model for GPU training. A learning rate scheduler is created, specifying the optimizer as Adam, the total number of training epochs as 120, the minimum learning rate as 0.000006, the initial learning rate for the warm-up phase as 0.000001, and the number of warm-up epochs as 5. The bias term's weight decay is 0.0004.

[0076] First, by iteratively training each batch in dataset A, image features are acquired using the image encoder of the model from step 2 and paired with the corresponding target data, then saved. The image features and target data are stacked in batches. The model is then trained through iterative training epochs and batches. In each training batch, the similarity loss between image features and text features is calculated based on randomly selected samples, and backpropagation and parameter updates are performed. At the end of training phase 1, the model's state dictionary is saved and named prompt.pth for use when reloading the model. In deep learning, model parameters are typically stored in the memory of the computing device, such as GPU memory or CPU memory. These parameters are used for the model's forward propagation and inference processes. Since the program is not yet finished, the learned text features of the illegal outdoor advertisements are still stored in memory for use in training phase 2.

[0077] Step 4: Enter the second training phase, read the training set B from the illegal outdoor advertising dataset into the network model built in step 2, and use the text features learned in step 3 and the image features in training set B to compare similarity, so as to fine-tune the image encoder of the CLIP model, save the model parameters with the highest accuracy during the training process, and name the file best.pth.

[0078] Step 4.1: Training set B undergoes the following operations: The input image size is adjusted to the specified size [256, 128]. Operations such as resizing, horizontal flipping, padding, cropping, converting to tensors, and standardization increase data diversity, robustness, and trainability. Finally, random erasure further increases data diversity and prevents the model from overfitting the training data. Training set B is then loaded into the model for GPU training. A learning rate scheduler is created, specifying the optimizer as Adam, the number of training epochs as 120, the number of image samples in each training batch as 64, the initial learning rate as 0.000005, the learning rate warm-up method as linear, the number of warm-up iterations as 10, the warm-up factor as 0.1, the weight decay (L2 regularization) parameter as 0.0001, the bias term weight decay parameter as 0.0001, the bias term learning rate factor as 2, the learning rate adjustment steps as [30, 50], and the learning rate adjustment multiplication factor as 0.1.

[0079] The text features learned in step 3 and the image features in training set B are compared to fine-tune the CLIP image encoder. The model parameters with the highest accuracy during training are saved and the file is named best.pth.

[0080] Step 5: Input the image to be detected containing illegal outdoor advertisements into the fine-tuned CLIP network model, load the weight files prompt.pth and best.pth from steps 3 and 4 into the model for inference detection, and determine whether it is illegal.

[0081] Step 5.1: To evaluate the performance of the fine-tuned model, precision is used as the metric for assessing the model algorithm. Precision ranges from 0 to 1; a higher value indicates more true positive samples in the model's predictions, signifying better model performance. In multi-class classification problems, the precision for each class can be calculated, and the average can be taken as the overall precision. Precision can be calculated using the following formula:

[0082]

[0083] Where TP represents the number of samples correctly predicted as positive, and FP represents the number of samples incorrectly predicted as positive.

[0084] The finely tuned CLIP model is applied to public scenes such as city streets, public transportation stations, or parks. By acquiring images of outdoor advertisements, the images are input into the finely tuned CLIP model, and the weight files prompt.pth and best.pth are loaded into the detection model for inference and detection to determine whether the outdoor advertisements are in violation of regulations.

[0085] This invention is based on a method for identifying illegal outdoor advertising using a fine-tuned CLIP. The experimental environment used an NVIDIA GeForce RTX 3050 graphics card and the PyTorch deep learning framework. The specific configuration is shown in Table 1 below:

[0086] Table 1 Environment Configuration Table

[0087] CPU Intel(R)Core(TM)i5-11260H@2.60GHz GPU NVIDIA GeForce RTX 3050 CUDA version CUDA 12.2 Programming language version Python 3.10 Deep learning framework PyTorch 2.0.1

[0088] The data used in the experiment consisted of images of illegal outdoor advertisements collected via web crawler, categorized into eight types. Each category was further subdivided into several similar types of images to maintain sample diversity. These eight types of image content were used as the key components for detecting illegal outdoor advertisement images.

[0089] from Figure 2 and Figure 3 As can be seen, when the loss value decreases during the two-stage training, it indicates that the model's optimization algorithm gradually optimizes the model parameters based on feedback from the training data, causing the model's predictions to gradually approach the true values. This can be explained as the model gradually improving its matching ability to training samples by learning patterns and features between samples.

[0090] from Figure 4 The continuously improving accuracy indicates that the model is more accurately predicting the category of samples in classification tasks. It also shows that the model has a deeper understanding of features and data, and is better able to distinguish between samples of different categories.

[0091] The above embodiments are only for illustrating the technical concept and features of the present invention, and are intended to enable those skilled in the art to understand the content of the present invention and implement it accordingly. They should not be construed as limiting the scope of protection of the present invention. All equivalent transformations or modifications made in accordance with the spirit and essence of the present invention should be covered within the scope of protection of the present invention.

Claims

1. A method for identifying illegal outdoor advertising based on fine-tuned CLIP, characterized in that, Specifically, it includes the following steps: Step 1: Obtain image data of illegal outdoor advertisements through a web spider, manually clean the image data, and create a dataset. Divide the dataset into a training set and a validation set. At the same time, use different data augmentation methods to augment the training set and generate training set A and training set B for use in the training phase. Step 2: Build the fine-tuned CLIP network model. The fine-tuned CLIP network model uses ViT-B / 16 as the backbone network, adds a cue learner, and performs two-stage training simultaneously. Step 2.1: The fine-tuned CLIP network model uses ViT-B / 16 as the backbone network, and downloads the corresponding pre-trained weight file. Two linear layers are added to linearly transform the input features, and two batch normalization layers are added for the input features. The model is specified to use batch normalization as the neck structure, and cross-entropy is selected as the loss function, as shown in the following formula: (1)； Where N is the number of images, y n p is the label of image n. n It is one of the element values ​​in the network output, and y n They are corresponding; Using the calculated image resolution, model name ViT-B / 16, and stride size as parameters, the CLIP model is called, and the image encoder and text encoder components of the CLIP model are applied to the fine-tuned network model, while a cue learner is added. Step 2.2: Build a cue learner. Provide a learnable context to initialize the text, and perform tokenization and initialization using random feature vectors from cue learning. Optimize the value of the feature vectors through backpropagation. Select "set cue before" in backpropagation for subsequent model training. Step 2.3: In the first training phase, a contrastive loss function is used, and in the second training phase, a cross-entropy loss function is used. Adam is used as the optimizer in both phases. By adding a regularization term to the loss function, the magnitude of the parameters is limited. The derivation formula of Adam is shown below: (2)； (3)； (4)； (5)； (6)； (7)； Wherein, the first gradient g t The loss function right Find the partial derivative. For the optimized parameters, the second term m t For time t, the first moment estimate of the gradient in momentum form, the third term u t The fourth term is the second moment estimate of the gradient in momentum form. This is the first-order moment estimate after bias correction, where yes power of t and These are the hyperparameters that decay the first-order and second-order matrices of the gradient, respectively; the fifth term... The last term is the second-order moment estimate after bias correction. and Let these represent the learning rate and the constant added to prevent the denominator from being zero, respectively. Substituting these values ​​into the formula updates the parameters. ; Step 3: First training phase: Read training set A from the illegal outdoor advertising dataset into the network model built in step 2. During the model training process, learn the text features of each advertising category. When the training ends, save the model's state dictionary and name it prompt.pth for use when reloading the model. Step 4: Second training phase: Read training set B from the illegal outdoor advertising dataset into the network model built in step 2. Use the text features learned in step 3 and the image features in training set B to compare similarity, thereby fine-tuning the image encoder of the CLIP model. Save the model parameters with the highest accuracy during training and name the file best.pth. Step 5: Read the image of the outdoor advertisement to be detected into the fine-tuned CLIP network model, load the weight files prompt.pth and best.pth from Step 3 and Step 4 into the model for inference detection, and determine whether there is a violation.

2. The method for identifying illegal outdoor advertising based on fine-tuned CLIP according to claim 1, characterized in that, The specific method for step 1 is as follows: Step 1.1: Use PyCharm programming code to collect illegal outdoor advertising image data from Baidu Images website online and store the image data in a local directory file; Step 1.2: Manually filter the image data to remove irrelevant and low-quality image data; Step 1.3: Divide the image data into training and validation sets in an 8:2 ratio for model training. Use different data augmentation methods to augment the training set, generating training set A and training set B respectively. Both sets use image resizing, normalization, and tensor conversion operations. Training set B adds random horizontal flipping, padding, random cropping, and random erasing operations.

3. The method for identifying illegal outdoor advertising based on fine-tuned CLIP according to claim 1, characterized in that, The specific method for step 3 is as follows: In the first training phase, the training set images from the illegal outdoor advertising dataset are loaded into the fine-tuned network model. The model is trained using a GPU, and the network model parameters are set as follows: 120 training epochs, 64 image samples in each training batch, an initial learning rate of 0.00035, a minimum learning rate of 0.000006, a weight decay of 0.0004 for the bias term, and Adam as the optimizer. During model training, the text features of illegal outdoor advertisements are continuously optimized and learned. At the end of the first training stage, the model's state dictionary is saved and named prompt.pth for use when reloading the model. Since the entire program has not yet ended, the learned text features are still retained in memory for use in the next training stage.

4. The method for identifying illegal outdoor advertising based on fine-tuned CLIP according to claim 1, characterized in that, The specific method for step 4 is as follows: In the second training phase, the training set images from the illegal outdoor advertising dataset are loaded into the fine-tuned network model. For preprocessing the training set images, GPU training is used, and the network model parameters are set as follows: 120 training epochs, 64 image samples in each training batch, an initial learning rate of 0.000005, a linear learning rate warm-up method, 10 warm-up iterations, a warm-up factor of 0.1, an L2 regularization weight decay parameter of 0.0001, a bias term weight decay parameter of 0.0001, a bias term learning rate factor of 2, and a learning rate adjustment step of [30, 50] with a learning rate adjustment multiplication factor of 0.

1. During model training, the image encoder of the model is fine-tuned by comparing the similarity between the images and the text features learned in step 3. Finally, the model parameters with the best accuracy are saved and named best.pth. (2)； In this context, TP represents true positives, TN represents true negatives, FP represents false positives, and FN represents false negatives.

5. The method for identifying illegal outdoor advertising based on fine-tuned CLIP according to claim 1, characterized in that, The specific method for step 5 is as follows: Step 5.1: Use precision as the evaluation metric for the model algorithm: (3)； Where TP represents the number of samples correctly predicted as positive, and FP represents the number of samples incorrectly predicted as positive.

Images