Advertisement detection method based on large model data enhancement and deep learning

Abstract

The invention relates to the technical field of advertisement detection, in particular to an advertisement detection method based on large model data enhancement and deep learning, which adopts a stratified sampling strategy to sample initial annotation data and finely adjust a large model based on a sampling data set to improve the classification capability of the large model on an advertisement text. According to the method, advertisement text data with obvious semantic features are screened by using a screening rule, the advertisement text data meeting conditions are pre-labeled by using the semantic understanding ability of a large model after fine tuning, and then manual proofreading is performed to ensure the accuracy and consistency of the data. The BERT model is trained by utilizing the data after manual proofreading, and the advertisement detection effect and application value are remarkably improved by classifying and identifying large-scale data.

Description

Technical Field

[0001] The present invention relates to advertisement detection technology, and in particular to an advertisement detection method based on large model data enhancement and deep learning. Background Art

[0002] With the widespread use of instant messaging tools, the way online advertising is disseminated has changed significantly, and advertising information is spread at a very high frequency on social platforms. Many businesses and individuals publish advertising content through online social platforms. These advertisements have the characteristics of large quantity, high similarity, low intention information entropy, and low data signal-to-noise ratio, which are often not conducive to effective semantic analysis or public opinion monitoring. Therefore, how to accurately and efficiently detect and filter advertising texts in massive social platform messages has become a key issue that needs to be solved urgently.

[0003] Existing technologies mainly use the following methods to detect advertisements:

[0004] 1Keyword matching: By establishing an advertising keyword library, specific keywords appearing in the text are used to identify advertising content.

[0005] 2. Text similarity analysis: By calculating the similarity between texts, similar advertising texts can be identified and duplicate content can be filtered out.

[0006] 3. Machine learning algorithm: Use traditional machine learning algorithms such as support vector machines and decision trees, use feature engineering to extract text features, and train models to classify advertising texts.

[0007] 4 Graph model algorithm: Messages in social networks are regarded as graph structures, and advertisement detection is performed through methods such as graph convolutional networks to capture the relationships and propagation paths between messages.

[0008] Existing ad detection technologies have the following technical problems:

[0009] (1) Insufficient accuracy: Existing methods often rely on fixed keywords or features and are easily affected by text variations and noise, resulting in insufficient recognition of new advertising content and difficulty in adapting to the rapid changes in advertising formats, thus causing false positives or negatives;

[0010] (2) High cost of obtaining annotated data: Many methods require a large amount of annotated data for training. In practical applications, obtaining high-quality annotated advertising text data requires a lot of labor and time costs, which limits the promotion and adaptability of the model.

[0011] (3) Low processing efficiency: Some complex deep learning models and graph model algorithms have high requirements for computing resources and time, resulting in suboptimal performance when detecting advertising content in real time and difficulty in meeting the response speed requirements of large-scale data;

[0012] (4) High model complexity: Although some algorithms, such as graph models, can capture the relationship between messages, their construction and computational complexity are high, and their processing speed is slow. They are not suitable for advertising detection on massive data, and they have high requirements for the design of graph structures, which may lead to unstable performance. Summary of the invention

[0013] In order to solve the above technical problems, the present invention provides an advertisement detection method based on large model data enhancement and deep learning, which improves the existing advertisement monitoring technology and improves the recognition accuracy and processing efficiency.

[0014] To achieve the above object, the technical solution adopted by the present invention is: to provide an advertisement detection method based on large model data enhancement and deep learning, comprising the following steps:

[0015] S1 fine-tuning large model to assist data annotation:

[0016] S1.1 manually annotate the initial data according to the data type and label to obtain the initial annotated data set;

[0017] S1.2 adopts a stratified sampling strategy, stratifies the initial annotated data set according to label distribution and text features, and extracts samples from each layer in proportion to obtain a sampled data set D = where x i Represents the input data, y i Representation labels, as input to the large model;

[0018] S1.3 fine-tune the large model based on the sample data set D to obtain the large model M;

[0019] S2 Data Enhancement:

[0020] S2.1 Filter out the advertisement text data with obvious semantic features based on the screening rules, input them into the large model M for annotation, and output the label: L(x)=M(x;θ), where θ represents the parameters of the large model;

[0021] S2.2 manually proofread the label L(x) and obtain the result: L'(x) = A(L(x));

[0022] S3 fine-tuning and training BERT model:

[0023] S3.1 processes the data obtained in step S2.2 using resampling technology and data enhancement technology, and uses it as the input of the BERT model;

[0024] S3.2 uses a step-by-step fine-tuning strategy to further train the BERT model. The training process includes the following operations:

[0025] The cross entropy loss function is used to evaluate the classification error of the model:

[0026]

[0027] Among them, y i is the true label, for the positive class y i =1, for negative class y i =0, The probability that the model predicts that the sample is a positive class;

[0028] Use batch training strategy to optimize the model, set training batches, and calculate gradients for each batch.

[0029]

[0030] Among them, θ t is the current model parameter, η is the learning rate, B is the batch size, Represents the cross entropy loss function The gradient of x i ,y i for samples and labels within a batch;

[0031] The hyperparameters are tuned by grid search. The objective function of grid search for optimizing hyperparameters is defined as:

[0032]

[0033] Among them, η * ,B * ,E * is the optimal hyperparameter combination, η is the learning rate, B is the batch size, and E is the number of training rounds.

[0034] Preferably, the process of extracting samples in proportion in step S1.2 is as follows:

[0035] Let the label distribution be D i , the size of the dataset corresponding to each label is n i , then the ratio is:

[0036]

[0037] Among them, k is the number of label types and N is the total number of samples.

[0038] Preferably, the sample data set D described in S1.2 is preprocessed before being input into the large model, including data cleaning, label encoding and format conversion.

[0039] Preferably, the fine-tuning process in step S1.3 is as follows:

[0040] Use the cross entropy loss function to update parameters:

[0041]

[0042] Where θ is the large model parameter, f(x i ; θ) is the large model output.

[0043] Preferably, the screening rules in step S2.1 include: keyword matching and text pattern recognition.

[0044] Preferably, step S2.2 further includes the following steps after manual proofreading:

[0045] An iterative process is designed, and a feedback mechanism is used to feed the manual proofreading result L'(x) back to the large model for optimization training. The optimization training process can be expressed as:

[0046] θ'(x)=T(θ,L'(x))

[0047] Wherein, T(θ,L'(x)) represents the optimization of the large model parameters based on the manual proofreading results.

[0048] Preferably, the resampling technique in step S3.1 includes: for small sample categories, using oversampling SMOTE technology to generate more samples; for large sample categories, using undersampling to reduce the number of samples;

[0049] The data enhancement technology includes: adopting a synonym replacement strategy to randomly replace words in a sentence with their synonyms, introducing diverse language expressions while maintaining semantic consistency; adopting a random deletion and insertion strategy to randomly delete non-core words in a sentence, or randomly inserting words in a sentence that are consistent with the context semantics.

[0050] Preferably, the data processed by the resampling technology and the data enhancement technology in step S3.1 needs to be formatted before being input into the BERT model, including word segmentation, tagging and encoding of the text, specifically:

[0051] BERT's native tokenizer is used to transform each text sample, truncating the text that exceeds the maximum length L_{max}, and padding the text that is less than L_{max} to ensure the consistency of the length of the input sequence, which is expressed as:

[0052] Tokenized Input=[CLS]+Tokenizer(Text)+[SEP],

[0053] Among them, [CLS] and [SEP] represent special markers at the beginning and end of each sentence.

[0054] Preferably, the training process described in step S3.2 also includes:

[0055] Set the initial learning rate η 0 , gradually reduce the learning rate through linear decay, the learning rate η of different layers i =η 0 ×γ i , where γ is the attenuation factor and i is the number of layers, to avoid overfitting and oscillation during training.

[0056] The above technical solution has the following advantages or beneficial effects:

[0057] (1) Improve recognition accuracy: By combining data enhancement and deep learning algorithms, the large model’s ability to classify a variety of advertising formats is enhanced, thereby improving recognition accuracy and reducing false positives and negatives.

[0058] (2) Reduce the cost of obtaining labeled data: Use the semantic understanding ability of large models for labeling, reduce reliance on manual labeling, and thus reduce the time and economic costs of the labeling process.

[0059] (3) Improve processing efficiency: By optimizing data processing flow and model reasoning, reduce processing time, quickly respond to ad detection needs, and improve overall system efficiency.

[0060] (4) Reduce model complexity: By using enhanced datasets to train the BERT model, the reliance on complex model structures is reduced, thereby reducing the complexity of model training and reasoning and improving operability.

[0061] The above summary is for illustrative purposes only and is not intended to be limiting in any way. In addition to the illustrative aspects, embodiments and features described above, further aspects, embodiments and features of the present invention will be readily apparent by reference to the accompanying drawings and the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] In the accompanying drawings, unless otherwise specified, the same reference numerals throughout the multiple drawings represent the same or similar parts or elements. These drawings are not necessarily drawn to scale. It should be understood that these drawings only depict some embodiments disclosed in the present invention and should not be regarded as limiting the scope of the present invention.

[0063] Figure 1 A flowchart of an advertisement detection method based on large model data enhancement and deep learning provided by the present invention;

[0064] Figure 2 A flow chart of the method for fine-tuning large models to assist in data annotation provided by the present invention;

[0065] Figure 3 BERT model training flow chart provided by the present invention; DETAILED DESCRIPTION

[0066] In the following, only some exemplary embodiments are briefly described. As those skilled in the art will appreciate, the described embodiments may be modified in various ways without departing from the spirit or scope of the present invention. Therefore, the drawings and descriptions are considered to be exemplary and non-restrictive in nature.

[0067] The present invention first provides an advertisement detection method based on large model data enhancement and deep learning, the process is as follows: Figure 1 As shown, the following steps are included:

[0068] S1 fine-tuning the large model to assist data annotation, the process is as follows Figure 2 As shown:

[0069] S1.1 Manually annotate the initial data according to data types and labels. During the annotation process, fully consider the diversity of the data to obtain the initial annotated data set as the basis of the entire process; through quality checks, ensure the high quality and consistency of the initial annotated data.

[0070] S1.2 adopts a stratified sampling strategy to stratify the initial annotated dataset according to label distribution and text features, and extract samples from each layer in proportion. The process is as follows:

[0071] Let the label distribution be D i , the size of the dataset corresponding to each label is n i , then the ratio is:

[0072]

[0073] Among them, k is the number of label types, and N is the total number of samples;

[0074] Get the sample data set where x i Represents the input data, y i Represent labels; ensure the representativeness and balance of sampled data, so that the large model can learn the characteristics of various types of data, thereby improving its generalization ability.

[0075] S1.3 Before the sampled data set D is input into the large model, it is preprocessed, including data cleaning, label encoding and format conversion, to ensure that the data input into the large model can match the requirements of fine-tuning; the large model is fine-tuned based on the sampled data set D to obtain the large model M. The process is as follows:

[0076] Use the cross entropy loss function to update parameters:

[0077]

[0078] Where θ is the large model parameter, f(x i ; θ) is the large model output.

[0079] During the fine-tuning process, hyperparameters such as the learning rate η and batch size B are adjusted; a single-card NVIDIA 3090 GPU is used for accelerated computing to improve training efficiency; and parallel processing and hardware optimization techniques are used to improve the model's labeling efficiency. Specifically, the processing time for labeling a piece of data is only 0.2 seconds, which significantly improves the overall efficiency of the labeling task, and is especially suitable for the rapid labeling of large-scale data.

[0080] After fine-tuning is completed, data outside the sampling data set is selected from the initial labeled data as the test set, which is input into the large model M to generate labels and evaluate the labeling performance of the large model.

[0081] Evaluation metrics include:

[0082] Precision:

[0083]

[0084] Among them, TP is the true positive example, that is, the number of samples that are actually positive and the model predicts as positive; FP is the false positive example, that is, the number of samples that are actually negative but the model predicts as positive;

[0085] Recall:

[0086]

[0087] FN is the number of false negative examples, that is, the number of samples that are actually positive but predicted by the model as negative;

[0088] F1 score:

[0089]

[0090] That is, the harmonic mean of precision and recall.

[0091] After evaluation, all indicators reached a level above 90%, indicating that the performance of the large model in the labeling task has reached the expected goal. In addition, if it is found during the application process that the model has inaccurate labeling in certain categories or scenarios, the error type can be further analyzed, and the large model can be optimized by increasing the initial labeling data or fine-tuning the large model parameters until the labeling performance of the large model is stable and meets the expected standards.

[0092] S2 Data Enhancement:

[0093] S2.1 Screen out advertising text data with obvious semantic features based on screening rules (including: keyword matching, text pattern recognition), ensure that the advertising text data has a certain degree of representativeness and diversity, input the large model M for preliminary semantic understanding and pre-labeling, and the output label is: L(x) = M(x; θ), where θ represents the parameters of the large model;

[0094] S2.2 In order to further ensure the accuracy and consistency of the annotation, manual proofreading is also required to review and fine-tune the large model output label L(x) to correct the possible deviations and errors of the large model and improve the overall annotation accuracy. The adjusted result can be expressed as: L'(x) = A(L(x));

[0095] In order to improve efficiency and reduce the workload of manual proofreading, an iterative process is designed, and a feedback mechanism is used to feed the manual proofreading result L'(x) back to the large model for optimization training. The optimization training process can be expressed as:

[0096] θ'(x)=T(θ,L'(x)),

[0097] Wherein, T(θ,L'(x)) represents the optimization of the parameters of the large model based on the manual proofreading results, so that the large model gradually improves its annotation performance.

[0098] The entire data enhancement process not only improves the quality of advertising text data, but also provides richer and more accurate data support for subsequent model training.

[0099] S3 fine-tuning and training the BERT model, the process is as follows Figure 3 As shown:

[0100] The present invention uses pre-trained BERT models (such as bert-base-chinese or bert-large-chinese), which have completed unsupervised pre-training on large-scale corpora and have strong language representation capabilities. By loading these pre-trained models, the cold start problem is effectively solved, avoiding the high cost and low efficiency of training from scratch. Since BERT has mastered rich language features, the tasks in the fine-tuning stage can converge quickly based on the existing models.

[0101] S3.1 Since the data obtained in step S2.2 has problems such as imbalanced category ratios, the data needs to be further adjusted to ensure that the BERT model can fully learn the characteristics of each category during training.

[0102] In response to data imbalance, resampling technology is used to adjust the training data to ensure a more balanced data distribution. Data enhancement technology is used to increase data diversity and expand data coverage.

[0103] The resampling techniques include: for small sample categories, using oversampling SMOTE technology to generate more samples; for large sample categories, using undersampling to reduce the number of samples;

[0104] The data enhancement technology includes: using a synonym replacement strategy to randomly replace words in a sentence with their synonyms, introducing diverse language expressions while maintaining semantic consistency, and increasing data diversity. Using a random deletion and insertion strategy to randomly delete non-core words in a sentence, or randomly insert words that are consistent with the context semantics in a sentence to enhance the language variability of the data.

[0105] Before the processed data is input into the BERT model, it needs to be formatted according to the input requirements of the BERT model, including word segmentation, tagging and encoding of the text. Specifically:

[0106] In order to be compatible with BERT's input format, BERT's native tokenizer is used to transform each text sample, truncating the text that exceeds the maximum length L_{max}, and padding the text that is less than L_{max} to ensure the consistency of the length of the input sequence, which is expressed as:

[0107] Tokenized Input=[CLS]+Tokenizer(Text)+[SEP],

[0108] Among them, [CLS] and [SEP] represent special markers at the beginning and end of each sentence.

[0109] S3.2 To further optimize the training process in the cold start phase, a gradual fine-tuning strategy (Layerwise Learning Rate Decay, LLRD) is adopted. This strategy can avoid large-scale updates to all parameters in the early stages of fine-tuning and reduce the instability of the model. First, fine-tune the top layer of the BERT model close to the output layer, and then gradually adjust the underlying parameters of the model downward;

[0110] Further training of the BERT model includes the following operations:

[0111] Set the initial learning rate η 0 , gradually reduce the learning rate through linear decay, the learning rate η of different layers i =η 0 ×γ i , where γ is the attenuation factor and i is the number of layers, to avoid overfitting and oscillation during training.

[0112] The cross entropy loss function is used to evaluate the classification error of the model:

[0113]

[0114] Among them, y i is the true label, for the positive class y i =1, for negative class y i =0, The probability that the model predicts that the sample is a positive class.

[0115] The parameter update uses the AdamW optimizer, combined with the learning rate scheduling mechanism, and the initial learning rate is set to η 0 =2×10 -5 .

[0116] In order to speed up the training process and improve the training efficiency, the present invention adopts a batch training strategy to optimize the model, sets the training batch to 32, and performs gradient calculation for each batch.

[0117]

[0118] Among them, θ t is the current model parameter, η is the learning rate, B is the batch size, Represents the cross entropy loss function The gradient of x i ,y i are the samples and labels within a batch.

[0119] To prevent the BERT model from overfitting during training, the present invention introduces a validation set for real-time monitoring and uses early stopping to terminate training when performance stagnates.

[0120] During the training process, a series of key hyperparameters, including learning rate, batch size, and number of training rounds, are relied upon to control the training process. In order to obtain the best model performance, these hyperparameters are tuned through grid search. The objective function of grid search optimization hyperparameters is defined as:

[0121]

[0122] Among them, η * ,B * ,E * is the optimal hyperparameter combination, η is the learning rate, B is the batch size, and E is the number of training rounds.

[0123] After the training is completed, it enters the evaluation and verification phase. In order to comprehensively evaluate the performance of the BERT model in the labeling task, the present invention adopts a variety of performance evaluation indicators, including accuracy, recall, and F1 score. By inputting an independent test set into the BERT model, the output label is compared with the true label, and the value of each evaluation indicator is calculated to reach more than 95%, indicating that the labeling accuracy and robustness of the BERT model have reached the standard of practical application.

[0124] According to the above method, this implementation provides an example, taking the detection of advertisements on a social networking platform as an example, the steps are as follows:

[0125] S1 uses keyword matching to filter out online social platform messages containing advertising terms and preliminarily screens out suspected advertising content.

[0126] S2 uses a large model to perform semantic analysis on the filtered messages and pre-labels advertising categories based on contextual understanding.

[0127] S3 introduces a manual proofreading step to review the labeling results of the large model to ensure the accuracy and consistency of advertising categories.

[0128] S4 uses manually proofread annotated data as training data and uses the BERT model to further optimize the advertising detection effect.

[0129] In actual applications, S5 deployed the fine-tuned BERT model in an advertising detection system and verified through multiple rounds of testing that its accuracy on large-scale data reached over 95%.

[0130] The above technical solution has the following advantages or beneficial effects: the stage of fine-tuning the large model described in the present invention improves the classification ability of the large model for advertising text data. This targeted training significantly improves the recognition accuracy of the model's advertising text, reduces misjudgments, and ensures the effectiveness of the system in diverse advertising scenarios. By screening advertising text data with obvious semantic features and pre-labeling it through the semantic understanding ability of the large model, the quality of the training data is enhanced. The manual proofreading process ensures the accuracy and consistency of the annotations, provides high-quality training data for the BERT model, and further improves the classification effect. The data that has been enhanced is used to train the BERT model, so that the model can perform effective advertising detection and recognition in large-scale data. This method significantly improves the efficiency and accuracy of detection, not only reduces manual intervention, but also greatly improves the application value of advertising content recognition, and promotes the advancement of advertising detection technology.

[0131] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of various changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. An advertisement detection method based on large model data enhancement and deep learning, characterized in that the steps include: S1 fine-tuning large model to assist data annotation: S1.1 manually annotate the initial data according to the data type and label to obtain the initial annotated data set; S1.2 adopts a stratified sampling strategy to stratify the initial annotated dataset according to label distribution and text features, and extracts samples proportionally from each layer to obtain a sampled dataset. where x i Represents input data, y i Representation labels, as input to the large model; S1.3 fine-tune the large model based on the sample data set D to obtain the large model M; S2 Data Enhancement: S2.1 Filter out the advertising text data with obvious semantic features based on the screening rules, input them into the large model M for annotation, and output the label: L(x)=M(x;θ), where θ represents the parameters of the large model; S2.2 manually proofread the label L(x) and obtain the result: L'(x) = A(L(x)); S3 fine-tuning and training BERT model: S3.1 processes the data obtained in step S2.2 using resampling technology and data enhancement technology, and uses it as the input of the BERT model; S3.2 uses a step-by-step fine-tuning strategy to further train the BERT model. The training process includes the following operations: The cross entropy loss function is used to evaluate the classification error of the model: Among them, y i is the true label, for the positive class y i =1, for negative class y i =0, The probability that the model predicts that the sample is a positive class; Use batch training strategy to optimize the model, set training batches, and calculate gradients for each batch. Among them, θ t is the current model parameter, η is the learning rate, B is the batch size, Represents the cross entropy loss function The gradient of x i ,y i for samples and labels within a batch; The hyperparameters are tuned by grid search. The objective function of grid search for optimizing hyperparameters is defined as: Among them, η * ,B * ,E * is the optimal hyperparameter combination, η is the learning rate, B is the batch size, and E is the number of training rounds.

2. The advertisement detection method based on large model data enhancement and deep learning as claimed in claim 1, characterized in that: The process of sampling in proportion described in step S1.2 is as follows: Let the label distribution be D i , the size of the dataset corresponding to each label is n i , then the ratio is: Among them, k is the number of label types and N is the total number of samples.

3. The advertisement detection method based on large model data enhancement and deep learning as claimed in claim 1, characterized in that: The sample data set D described in S1.2 is preprocessed before being input into the large model, including data cleaning, label encoding and format conversion.

4. The advertisement detection method based on large model data enhancement and deep learning as claimed in claim 1, characterized in that: The fine-tuning process described in step S1.3 is as follows: Use the cross entropy loss function to update parameters: Where θ is the large model parameter, f(x i ; θ) is the large model output.

5. The advertisement detection method based on large model data enhancement and deep learning as claimed in claim 1, characterized in that: The screening rules described in step S2.1 include: keyword matching and text pattern recognition.

6. The advertisement detection method based on large model data enhancement and deep learning as claimed in claim 1, characterized in that: Step S2.2 also includes the following steps after manual proofreading: An iterative process is designed, and a feedback mechanism is used to feed the manual proofreading result L'(x) back to the large model for optimization training. The optimization training process can be expressed as: θ'(x)=T(θ,L'(x)) Wherein, T(θ,L'(x)) represents the optimization of the large model parameters based on the manual proofreading results.

7. The advertisement detection method based on large model data enhancement and deep learning as claimed in claim 1, characterized in that: The resampling technique described in step S3.1 includes: for small sample categories, using oversampling SMOTE technology to generate more samples; for large sample categories, using undersampling to reduce the number of samples; The data enhancement technology includes: adopting a synonym replacement strategy to randomly replace words in a sentence with their synonyms, introducing diverse language expressions while maintaining semantic consistency; adopting a random deletion and insertion strategy to randomly delete non-core words in a sentence, or randomly inserting words in a sentence that are consistent with the context semantics.

8. The advertisement detection method based on large model data enhancement and deep learning as claimed in claim 1, characterized in that: The data processed by the resampling technology and data enhancement technology in step S3.1 needs to be formatted before being input into the BERT model, including word segmentation, tagging and encoding of the text. Specifically: BERT's native tokenizer is used to transform each text sample, truncating the text that exceeds the maximum length L_{max}, and padding the text that is less than L_{max} to ensure the consistency of the length of the input sequence, which is expressed as: TokenizedInput=[CLS]+Tokenizer(Text)+[SEP], Among them, [CLS] and [SEP] represent special markers at the beginning and end of each sentence.

9. The advertisement detection method based on large model data enhancement and deep learning as claimed in claim 1, characterized in that: The training process described in step S3.2 also includes: Set the initial learning rate η0, and gradually reduce the learning rate through linear decay. The learning rate η of different layers i =η0×γ i , where γ is the attenuation factor and i is the number of layers, to avoid overfitting and oscillation during training.

Images