Large model text auditing optimization method based on clustering preprocessing

Abstract

The invention discloses a large model text auditing optimization method based on clustering preprocessing, which relates to the technical field of natural language processing, and comprises the following steps: S1, preprocessing input original text data, including text cleaning, word segmentation, stop word and punctuation mark removal and standardized text vector generation, during preprocessing, a TF-IDF algorithm is adopted to calculate a text feature weight, and a high-dimensional feature vector matrix is generated. According to the large model text auditing optimization method provided by the invention, the text is divided into clusters with similar semantics or theme contents through clustering preprocessing, and a special auditing model is constructed for each cluster, so that the models can more accurately understand the text contents, the auditing accuracy is improved, and meanwhile, the auditing efficiency is improved. Due to the fact that the data size needing to be processed is reduced through clustering, the calculation efficiency of the model is remarkably improved, the clustering center vector and the cluster division rule are continuously adjusted, and closed-loop optimization of the model is achieved.

Description

Technical Field

[0001] The present invention relates to the technical field of natural language processing, and in particular to a large model text audit optimization method based on clustering preprocessing. Background Art

[0002] With the exponential growth of the scale of Internet text data, traditional text review methods face the bottlenecks of low efficiency in processing massive data and insufficient semantic understanding accuracy. The current mainstream technology uses a unified large model to review the entire text. Although there have been breakthroughs in general semantic recognition, there are defects such as redundant model parameters and excessive consumption of computing resources. Especially in mixed text review scenarios with multiple topics and multiple scenarios, due to the lack of pre-analysis of text feature distribution, it is difficult for the model to specifically process content areas with different semantic densities, resulting in a significant negative correlation between review accuracy and processing speed. Some studies have attempted to reduce the processing volume through pre-filtering of rule engines, but the maintenance cost of the manual rule base is high and it is difficult to adapt to the dynamically changing characteristics of network language.

[0003] There are three key problems in the existing technology: First, the dimension of text feature extraction is single, and traditional methods such as TF-IDF cannot effectively capture contextual semantic associations, resulting in deviations between clustering results and true semantic distribution; second, the adaptability of the audit model to text clusters is insufficient, and a dynamic model selection mechanism based on cluster features has not been established. In similar studies, model fine-tuning only uses simple parameter adjustments, and fails to combine optimization methods such as adversarial training and knowledge distillation to improve professional field recognition capabilities; third, there is a lack of a closed-loop optimization system, and most systems adopt a static audit strategy and fail to feed back misjudged samples to the feature extraction and clustering links for iterative optimization, resulting in an error accumulation effect. In response to this, we propose a large-model text audit optimization method based on clustering preprocessing. Summary of the invention

[0004] In order to solve the above technical problems, a large model text review optimization method based on clustering preprocessing is provided. This technical solution solves the above problems.

[0005] In order to achieve the above purpose, the technical solution adopted by the present invention is: A large model text review optimization method based on clustering preprocessing includes the following steps: S1, preprocessing the input raw text data, the preprocessing includes: text cleaning, word segmentation, removal of stop words and punctuation marks, generating a standardized text vector, and using the TF-IDF algorithm to calculate text feature weights during preprocessing to generate a high-dimensional feature vector matrix; S2. Perform cluster analysis on the standardized text vector based on the K-means clustering algorithm, determine the optimal clustering number K through the silhouette coefficient, and divide the text into K clusters, each cluster corresponding to similar semantics or subject content; S3. Build an audit model for each cluster and use a pre-trained language model to fine-tune the text within the cluster to optimize the audit accuracy; S4, introduce a dynamic feedback mechanism to feed back the misjudgment results of the audit module as negative samples to the clustering module, and adjust the cluster center vector and cluster division rules; S5. Periodically iterate and optimize clustering algorithm parameters and audit model weights until the audit accuracy converges to the preset threshold.

[0006] Preferably, the pretreatment in step S1 specifically includes: Use regular expressions to match non-text characters and remove non-text characters from a string; The word form of the word segmentation results is normalized through the stem extraction algorithm; Filter low-frequency words whose frequency is less than 0.1% of the total number of times; Use principal component analysis to reduce the dimension of high-dimensional vectors and retain the principal components with variance contribution exceeding 95%. The dimension reduction formula is: ; In the formula, represents the data matrix after dimensionality reduction, represents the original data matrix, Indicates that The projection matrix composed of the eigenvectors corresponding to the largest eigenvalues; The formula for calculating text feature weights using the TF-IDF algorithm is: ; In the formula, Indicates a term, Represents a document, Indicates the total number of documents. Indicates that it contains a term The number of documents.

[0007] Preferably, the K-means clustering algorithm improvement in step S2 includes: The k-means++ algorithm is used to initialize the cluster center, and the sample points that maximize the initial center distance are selected; The cosine similarity is used to calculate the distance between the sample and the centroid, where the calculation formula for the distance between the sample and the centroid is: ; In the formula, represents the vector dot product, Represents the magnitude of a vector; Iterative update, dynamically adjust the number of clusters during the iteration process , when the average silhouette coefficient within a cluster is lower than 0.5, the split operation is triggered to generate a new cluster.

[0008] Preferably, the specific method for determining the optimal cluster number K by the silhouette coefficient is: For each sample, calculate its silhouette coefficient. The overall silhouette coefficient is the mean of all samples. Select the K value corresponding to the maximum mean. The calculation formula of the silhouette coefficient is: ; In the formula, Representation sample The silhouette coefficient, Representation sample The intra-cluster compactness of Representation sample The inter-cluster separation degree, The calculation formula is: ; In the formula, Representation sample Cluster The number of samples, Representation sample With the same cluster samples distance; The calculation formula is: ; In the formula, Representation Cluster The number of samples, Indicates taking and sampling Other clusters that are in different clusters, Representation sample With cluster Medium Sample distance; Draw the K-contour coefficient curve and select the K value corresponding to the peak value of the contour coefficient; K-means was run multiple times, and the mode of K values ​​and the variance of silhouette coefficient were counted to verify stability.

[0009] Preferably, building the audit model in step S3 specifically includes: Based on the label distribution of the text in the cluster, oversampling is used to balance the training data; Choose BERT as the base model architecture; Add a bidirectional LSTM network in the fine-tuning layer to capture contextual dependencies; Adopt adversarial training strategy to inject perturbation noise into input text to improve model robustness; The cross entropy loss function is used for fine-tuning, and the formula is: ; In the formula, Represents the cross entropy loss value, which is used to measure the difference between the model prediction result and the true label. represents the number of classification categories, represents the true label, Predict probabilities for the model.

[0010] Preferably, the adversarial training strategy specifically includes: Adding Gaussian noise to the word embedding layer , noise intensity It decays linearly from 0.1 to 0.01 with the training rounds; By constraining the perturbation amplitude through the gradient penalty term, the loss function is expanded to: ; In the formula, represents the loss function value after adversarial training expansion, represents the original cross entropy loss value, Represents the penalty coefficient, which ranges from 0.5 to 1.0 and is used to control the weight of the gradient penalty term. Represents the loss function About input The square of the norm of the gradient is used to constrain the perturbation amplitude.

[0011] Preferably, the dynamic feedback mechanism in step S4 is specifically: Add samples that are judged as "misjudged" by the audit module to the cluster training set and recalculate the TF-IDF weight; According to the distance between the misjudged sample and the original cluster center, the cluster center coordinates are adjusted, and the update formula is: ; In the formula, Indicates the updated cluster center vectors, Indicates the number before the update Cluster center vectors, represents the learning rate, Represents the misjudged sample vector; When the misjudgment rate decreases by less than 2% in three consecutive iterations, the clustering parameters are reset.

[0012] Preferably, the iterative optimization in step S5 includes: The simulated annealing algorithm is used to optimize the initial center selection for the clustering module; The knowledge distillation technology is used for the audit module, and the output distribution of the teacher model is used as soft labels to train the lightweight student model; The model parameters are automatically updated every 24 hours based on the newly added data, and a sliding window mechanism is used during the update to retain the data for the last 30 days.

[0013] Preferably, the steps of implementing the knowledge distillation technology include: The teacher model uses a 12-layer Transformer, and the student model uses a 6-layer Transformer; Distillation temperature parameters Gradually reduce from 5.0 to 2.0; The loss function is the weighted sum of the teacher-student output KL divergence and the student model cross entropy, with a weight ratio of 3:1.

[0014] Preferably, an effect evaluation module is included, which is implemented as follows: The accuracy, recall and F1 value are used as core indicators, and the formula is: ; In the formula, Represents the harmonic mean of precision and recall, which is used to comprehensively evaluate the performance of the model; It represents the accuracy, that is, the proportion of samples predicted by the model to be positive that are actually positive. Recall is the ratio of samples that are actually positive examples to those that are predicted as positive examples by the model. Deploy an A / B testing framework to compare the difference in false positive rates between clustering preprocessing and direct review; The clustering results and decision boundaries were visualized through t-SNE, and controversial samples were manually reviewed.

[0015] Compared with the prior art, the present invention has the following beneficial effects: The present invention proposes a large-model text audit optimization method that divides the text into clusters with similar semantics or subject content through clustering preprocessing, and constructs a special audit model for each cluster, which enables the model to understand the text content more accurately, thereby improving the accuracy of the audit. At the same time, since clustering reduces the amount of data to be processed, the computational efficiency of the model is also significantly improved. By introducing a dynamic feedback mechanism, the misjudgment results of the audit module are fed back to the clustering module as negative samples, and the cluster center vector and cluster division rules are continuously adjusted to achieve closed-loop optimization of the model. An adversarial training strategy is adopted to inject disturbance noise into the input text, which further improves the robustness of the model and enables it to better adapt to dynamically changing network language characteristics. Through the knowledge distillation technology, the output distribution of the teacher model is used as a soft label to train a lightweight student model, which achieves model compression and acceleration, reduces computing resource consumption, and adopts a sliding window mechanism to regularly update model parameters to ensure the timeliness and accuracy of the model, reducing manual intervention and maintenance costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 The figure is a flow chart of the method of the present invention. DETAILED DESCRIPTION

[0017] The following description is used to disclose the present invention so that those skilled in the art can implement the present invention. The preferred embodiments described below are only examples, and those skilled in the art may think of other obvious variations.

[0018] Reference Figure 1 As shown, a large model text review optimization method based on clustering preprocessing includes: S1: Preprocess text data to generate a high-dimensional feature vector matrix; S2: Clustering text vectors based on K-means clustering algorithm; S3: Build and fine-tune the audit model for each cluster; S4: Introduce dynamic feedback mechanism to adjust clustering and partitioning rules; S5: Iteratively optimize clustering parameters and model weights until convergence.

[0019] In step S1, we preprocess the input raw text data comprehensively. This process includes text cleaning to remove irrelevant characters and noise; word segmentation to divide the text into meaningful vocabulary units; and removal of stop words and punctuation to reduce data sparsity and noise interference. On this basis, we use the TF-IDF algorithm to calculate the text feature weights and generate a high-dimensional feature vector matrix. This step provides a solid data foundation for subsequent clustering analysis.

[0020] Next, in step S2, we performed cluster analysis on the standardized text vectors based on the K-means clustering algorithm. In order to obtain better clustering results, we used the k-means++ algorithm for initialization to ensure that the distribution of the initial cluster centers is more uniform. At the same time, we used cosine similarity to calculate the distance between the sample and the centroid, and iteratively updated the cluster center until convergence was reached. During the iteration process, we also dynamically adjusted the number of clusters. When the average silhouette coefficient within the cluster was lower than 0.5, the split operation was triggered to generate a new cluster. This step enables the text data to be divided into multiple clusters with similar semantics or thematic content, providing strong support for the construction of subsequent review models.

[0021] In step S3, we build and fine-tune the audit model for each cluster. Based on the label distribution of the text in the cluster, we use an oversampling strategy to balance the training data to reduce the impact of category imbalance on model performance. At the same time, we choose BERT as the basic model architecture and add a bidirectional LSTM network in the fine-tuning layer to capture text context dependencies. In addition, we also adopt an adversarial training strategy to improve the robustness and generalization ability of the model by injecting perturbation noise into the input text. During the fine-tuning process, we use the cross entropy loss function to measure the difference between the model prediction results and the true label, and continuously adjust the model parameters through the optimization algorithm to obtain higher audit accuracy.

[0022] In step S4, we introduced a dynamic feedback mechanism to further improve the effect of text review. We added samples that the review module judged as "misjudged" to the cluster training set, recalculated the TF-IDF weight, and adjusted the cluster center coordinates according to the distance between the misjudged samples and the original cluster center. When the misjudgment rate drops by less than 2% in three consecutive iterations, we trigger the cluster parameter reset to avoid the error accumulation effect. This step enables the cluster analysis and review model to promote and optimize each other, thereby improving the overall review performance.

[0023] Finally, in step S5, we performed periodic iterative optimization to continuously improve the performance of clustering algorithm parameters and audit model weights. We used simulated annealing algorithm to optimize the initial center selection for the clustering module to obtain better clustering effect; we used knowledge distillation technology for the audit module to use the output distribution of the teacher model as soft labels to train the lightweight student model to reduce computing resource consumption. At the same time, we automatically updated the model parameters based on the newly added data every 24 hours, and used a sliding window mechanism to retain the data of the last 30 days to ensure that the model can adapt to dynamically changing text features.

[0024] In summary, the large-model text audit optimization method based on clustering preprocessing of the present invention achieves a dual improvement in text audit efficiency and accuracy through a series of carefully designed steps. The method has broad application prospects and important practical value, and can provide powerful technical support and solutions for the field of Internet text audit.

[0025] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions only describe the principles of the present invention. The present invention may be subject to various changes and improvements without departing from the spirit and scope of the present invention. These changes and improvements fall within the scope of the present invention. The scope of protection claimed by the present invention is defined by the attached claims and their equivalents.

Claims

1. A large model text audit optimization method based on clustering preprocessing, characterized in that: The following steps are involved: S1, preprocessing the input raw text data, the preprocessing includes: text cleaning, word segmentation, removal of stop words and punctuation marks, generating a standardized text vector, and using the TF-IDF algorithm to calculate text feature weights during preprocessing to generate a high-dimensional feature vector matrix; S2. Perform cluster analysis on the standardized text vector based on the K-means clustering algorithm, determine the optimal clustering number K through the silhouette coefficient, and divide the text into K clusters, each cluster corresponding to similar semantics or subject content; S3. Build an audit model for each cluster and use a pre-trained language model to fine-tune the text within the cluster to optimize the audit accuracy; S4, introduce a dynamic feedback mechanism to feed back the misjudgment results of the audit module as negative samples to the clustering module, and adjust the cluster center vector and cluster division rules; S5. Periodically iterate and optimize clustering algorithm parameters and audit model weights until the audit accuracy converges to the preset threshold.

2. According to claim 1, a large model text audit optimization method based on clustering preprocessing is characterized in that: The pretreatment in step S1 specifically includes: Use regular expressions to match non-text characters and remove non-text characters from a string; The word form of the word segmentation results is normalized through the stem extraction algorithm; Filter low-frequency words whose frequency is less than 0.1% of the total number of times; Use principal component analysis to reduce the dimension of high-dimensional vectors and retain the principal components with variance contribution exceeding 95%. The dimension reduction formula is: ； In the formula, represents the data matrix after dimensionality reduction, represents the original data matrix, Indicates that The projection matrix composed of the eigenvectors corresponding to the largest eigenvalues; The formula for calculating the text feature weight using the TF-IDF algorithm is: ； In the formula, Indicates a term, Represents a document, Indicates the total number of documents. Indicates that it contains a term The number of documents.

3. The large model text audit optimization method based on clustering preprocessing according to claim 1 is characterized in that: The improvements to the K-means clustering algorithm described in step S2 include: The k-means++ algorithm is used to initialize the cluster center, and the sample points that maximize the initial center distance are selected; The cosine similarity is used to calculate the distance between the sample and the centroid, where the calculation formula for the distance between the sample and the centroid is: ； In the formula, represents the vector dot product, Represents the magnitude of a vector; Iterative update, dynamically adjust the number of clusters during the iteration process , when the average silhouette coefficient within a cluster is lower than 0.5, the split operation is triggered to generate a new cluster.

4. The large model text audit optimization method based on clustering preprocessing according to claim 1 is characterized in that: The specific method for determining the optimal number of clusters K by the silhouette coefficient is: For each sample, calculate its silhouette coefficient. The overall silhouette coefficient is the mean of all samples. Select the K value corresponding to the maximum mean. The calculation formula of the silhouette coefficient is: ； In the formula, Representation sample The silhouette coefficient, Representation sample The intra-cluster compactness of Representation sample The inter-cluster separation degree, The calculation formula is: ； In the formula, Representation sample Cluster The number of samples, Representation sample With the same cluster samples distance; The calculation formula is: ； In the formula, Representation Cluster The number of samples, Indicates taking and sampling Other clusters that are in different clusters, Representation sample With cluster Medium Sample distance; Draw the K-contour coefficient curve and select the K value corresponding to the peak value of the contour coefficient; K-means was run multiple times, and the mode of K values ​​and the variance of silhouette coefficient were counted to verify stability.

5. The large model text audit optimization method based on clustering preprocessing according to claim 1 is characterized in that: The construction of the audit model in step S3 specifically includes: Based on the label distribution of the text in the cluster, oversampling is used to balance the training data; Choose BERT as the base model architecture; Add a bidirectional LSTM network in the fine-tuning layer to capture contextual dependencies; Adopt adversarial training strategy to inject perturbation noise into input text to improve model robustness; The cross entropy loss function is used for fine-tuning, and the formula is: ； In the formula, Represents the cross entropy loss value, which is used to measure the difference between the model prediction result and the true label. represents the number of classification categories, represents the true label, Predict probabilities for the model.

6. The large model text audit optimization method based on clustering preprocessing according to claim 5 is characterized in that: The adversarial training strategy specifically includes: Adding Gaussian noise to the word embedding layer , noise intensity It decays linearly from 0.1 to 0.01 with the training rounds; By constraining the perturbation amplitude through the gradient penalty term, the loss function is expanded to: ； In the formula, represents the loss function value after adversarial training expansion, represents the original cross entropy loss value, Represents the penalty coefficient, which ranges from 0.5 to 1.0 and is used to control the weight of the gradient penalty term. Represents the loss function About input The square of the norm of the gradient is used to constrain the perturbation amplitude.

7. The large model text audit optimization method based on clustering preprocessing according to claim 1 is characterized in that: The dynamic feedback mechanism in step S4 is as follows: Add samples that are judged as "misjudged" by the audit module to the cluster training set and recalculate the TF-IDF weight; According to the distance between the misjudged sample and the original cluster center, the cluster center coordinates are adjusted, and the update formula is: ； In the formula, Indicates the updated Cluster center vectors, Indicates the number before the update cluster center vectors, represents the learning rate, Represents the misjudged sample vector; When the misjudgment rate decreases by less than 2% in three consecutive iterations, the clustering parameters are reset.

8. The large model text audit optimization method based on clustering preprocessing according to claim 1 is characterized in that: The iterative optimization in step S5 includes: The simulated annealing algorithm is used to optimize the initial center selection for the clustering module; The knowledge distillation technology is used for the audit module, and the output distribution of the teacher model is used as soft labels to train the lightweight student model; The model parameters are automatically updated every 24 hours based on the newly added data, and a sliding window mechanism is used during the update to retain the data for the last 30 days.

9. The large model text audit optimization method based on clustering preprocessing according to claim 8 is characterized in that: The implementation steps of the knowledge distillation technology include: The teacher model uses a 12-layer Transformer, and the student model uses a 6-layer Transformer; Distillation temperature parameters Gradually reduce from 5.0 to 2.0; The loss function is the weighted sum of the teacher-student output KL divergence and the student model cross entropy, with a weight ratio of 3:

1.

10. The large model text audit optimization method based on clustering preprocessing according to claim 1 is characterized in that: Including effect evaluation module, which is implemented as follows: The accuracy, recall and F1 value are used as core indicators, and the formula is: ； In the formula, Represents the harmonic mean of precision and recall, which is used to comprehensively evaluate the performance of the model; It represents the accuracy, that is, the proportion of samples predicted by the model to be positive that are actually positive. Recall is the ratio of samples that are actually positive examples to those that are predicted as positive examples by the model. Deploy an A / B testing framework to compare the difference in false positive rates between clustering preprocessing and direct review; The clustering results and decision boundaries were visualized through t-SNE, and controversial samples were manually reviewed.

Images