A similarity text mining method and system based on RoBERTa knowledge distillation

Abstract

The present invention discloses a similarity text mining method and system based on RoBERTa knowledge distillation, including: mapping sentence embedding to probability distribution space to convert into probability distribution, using KL divergence to measure the semantic difference between sentence embeddings, adjusting the parameters of the embedding vector, reducing the KL divergence of similar sentence pairs while increasing the KL divergence of dissimilar sentences; performing back propagation based on the loss function optimized by KL divergence, updating the word vector parameters, saving the optimized sentence vector, and generating the word vector space; training the knowledge distillation embedding model based on RoBERTa, migrating the rich semantic information in the deep model to the lightweight student model; and using the trained student model to perform similarity text mining tasks. The present invention can better capture the subtle semantic differences between sentences and significantly improve the accuracy and efficiency of semantic similarity calculation.

Description

Technical Field

[0001] The present invention belongs to the technical field of natural language processing, and specifically relates to a similarity text mining method and system based on RoBERTa knowledge distillation. Background Art

[0002] Semantic Textual Similarity (STS) is an important task in natural language processing (NLP), which aims to evaluate the semantic similarity between two text fragments. This task is widely used in fields such as search engine optimization, information retrieval, machine translation, text summarization, and dialogue systems. However, traditional word vector models, such as Word2Vec, GloVe, and FastText, although they can effectively generate distributed representations of vocabulary, have limitations in capturing contextual dependencies and the semantic expression of polysemous words. In addition, the simple word vector averaging method is difficult to fully reflect the complex semantic structure in the sentence, and often performs insufficiently when faced with complex semantic dependencies, resulting in limited accuracy in semantic similarity assessment. Traditional embedding methods often only support a single similarity calculation method, such as cosine similarity. This single similarity measure may not provide sufficient flexibility when processing certain complex texts, and is often limited when extended to other application scenarios.

[0003] In recent years, deep learning models such as BERT and RoBERTa have made significant progress in semantic expression. The full name of RoBERTa is "A Robustly Optimized BERT Pretraining Approach". It is an improved version of the BERT model. Its main goal is to improve the performance of the model on various natural language processing tasks by performing a series of optimizations on the BERT pretraining process. For example, the invention with the publication number CN117807235B discloses a text classification method based on internal model feature distillation. First, the microblog text dataset is preprocessed, and the text is converted into the features required by the model using a tokenizer; secondly, the features are respectively passed to the student model and the teacher model to extract the prediction results; then the internal features in the teacher model are distilled as soft labels, and the loss loss1 is calculated with the internal features in the student model, and the loss loss3 is calculated using the prediction results of the student model and the teacher model, and the loss loss2 is calculated using the prediction results of the student model and the label of the student model; finally, the three losses are summed up, back-propagated in the network, and the network is optimized. This invention compresses the model's memory, improves the model's performance, and better balances the model size and model performance in text classification tasks. However, the high performance of such models is usually accompanied by huge computing costs and complex hardware requirements. It is difficult to achieve real-time reasoning and large-scale deployment in practical applications, and it is also difficult to adapt to real-time processing needs. In addition, traditional methods usually perform well in specific fields or specific data sets, but when migrating to other tasks or fields, the model often needs to be retrained. In addition, facing the needs of semantic analysis in a big data environment, how to balance the model's computational efficiency and semantic expression capabilities has become one of the key challenges of research.

[0004] The invention with the publication number CN116738955A discloses an unsupervised phrase extraction method based on pre-trained model prompt fine-tuning, which uses a custom embedding layer to embed the template, and then splices the two results to obtain a vector with text information and template information; uses the attention structure in the pre-trained language model to generate a vector sequence, and then reduces the dimension of the vector sequence; calculates the KL divergence of the global semantic template vector and the local semantic template vector, and uses the sentence length to constrain the score to avoid the local optimal solution; obtains the background and global scores of each word; uses the continuous positions where the difference between the global score and the background score is greater than a certain threshold to consider the segment as a key phrase for phrase extraction. The invention is able to effectively separate the background semantic information and global semantic information in the text, and uses the score difference of the word in the background information and semantic information to locate the key phrase unsupervisedly. However, the KL divergence loss function of the invention mainly focuses on the optimization of the proximity of similar sentence pairs, and the model's ability to distinguish in the semantic space is still insufficient; and it performs poorly on similarity text mining tasks. Summary of the invention

[0005] The present invention discloses a similarity text mining method and system based on RoBERTa knowledge distillation. By introducing KL divergence as a metric, the subtle semantic differences between sentences can be better captured. The KL divergence optimization of the present invention makes the distance between similar sentences in the semantic space closer, and the distance between dissimilar sentences is effectively widened. On this basis, combined with RoBERTa knowledge distillation, the accuracy and efficiency of semantic similarity calculation are significantly improved, which is particularly suitable for online application scenarios that require fast response.

[0006] In a first aspect, the present invention discloses a similarity text mining method based on RoBERTa knowledge distillation, the method comprising the following steps:

[0007] S1, map sentence embeddings to probability distribution space to convert them into probability distributions, use KL divergence to measure the semantic differences between sentence embeddings, adjust the parameters of the embedding vectors, reduce the KL divergence of similar sentence pairs while increasing the KL divergence of dissimilar sentences; perform back propagation based on the loss function optimized by KL divergence, update the word vector parameters, save the optimized sentence vectors, and generate the word vector space;

[0008] S2, based on the generated word vector space, train the knowledge distillation embedding model based on RoBERTa to transfer the rich semantic information in the deep model to the lightweight student model; during the training process, regularly verify the performance indicators of the student model on the semantic similarity task until the performance indicators tend to be stable and meet the preset requirements, and the training is completed;

[0009] S3, uses the trained student model to perform similarity text mining tasks.

[0010] Step S1 further comprises:

[0011] S11, load the Word2Vec pre-trained model to generate word vectors; pre-process the input sentence, and generate the initial sentence embedding by averaging all the word vectors in the sentence or using TF-IDF weighted average. The initial sentence embedding is used to reflect the semantic features of the sentence;

[0012] S12, using the softmax function to map sentence embeddings to the probability distribution space to convert them into probability distributions. The probability distributions generated by each pair of sentence embeddings represent the location of the sentences in the semantic space.

[0013] S13, adjust the parameters of each sentence embedding vector to reduce the KL divergence of similar sentence pairs while increasing the KL divergence of dissimilar sentences;

[0014] S14, update the word vector parameters through the back-propagation algorithm to minimize the loss function after KL divergence optimization; calculate the overall loss after each iteration until the model converges and generates sentence embeddings with stronger semantic distinction ability; save the optimized sentence vectors and generate the word vector space.

[0015] As a preferred example, in step S11, the process of preprocessing the input sentence includes the following steps:

[0016] Filter invalid or blank sentences; for the remaining sentences, remove stop words, punctuation marks, and other meaningless characters.

[0017] Step S2 further comprises:

[0018] S21, input sentence pairs to the pre-trained RoBERTa teacher model, obtain its hidden layer output, and use the probability distribution of the calculated output as the soft label; the soft label of each sentence contains rich semantic information, which is used to guide the training of the student model;

[0019] S22, initialize the structure of the student model, set the hyperparameters of the student model including the number of layers, number of nodes and activation function, so that the size of its embedding layer is consistent with the dimension of sentence embedding; use TensorDataset to combine the sentence embedding and the soft labels of the RoBERTa teacher model into a training dataset;

[0020] S23, using the mean square error loss function, aligns the prediction results of the student model with the soft labels generated by the teacher model, and uses the training data set to train the student model; during the training process, load the data batch by batch, and use the Adam optimizer to optimize the parameters of the student model so that its output tends to be consistent with the performance of the teacher model;

[0021] S24, regularly evaluate the performance indicators of the student model on the semantic similarity task until the performance indicators tend to be stable and meet the preset requirements, complete the training, and save the weight of the trained student model.

[0022] As a preferred example, in step S24, the performance indicator includes one or more of the loss value, Pearson correlation coefficient, Spearman correlation coefficient and mean absolute error of the training process.

[0023] As a preferred example, in step S3, the process of performing the similarity text mining task includes:

[0024] Design a cosine similarity calculation module to obtain the similarity value between sentences by calculating the dot product of sentence embeddings and normalizing them;

[0025] Implement TF-IDF weighted cosine similarity to perform TF-IDF weighted processing on word vectors to improve the calculation accuracy when the importance of keywords in long texts or sentences is different;

[0026] Smoothed inverse frequency similarity is introduced to remove the principal components in sentence embedding to reduce the interference of high-frequency words.

[0027] In a second aspect, the present invention discloses a similarity text mining system based on RoBERTa knowledge distillation, the system comprising a data input module, a sentence embedding generation module, a divergence calculation module, a knowledge distillation module, an output and visualization module, and a monitoring and evaluation module;

[0028] The data input module is used to clean and preprocess input data in various text formats, and perform word segmentation on the preprocessed text;

[0029] The sentence embedding generation module is used to generate high-dimensional sentence embedding using a pre-trained word vector model;

[0030] The divergence calculation module is used to map sentence embeddings to a probability distribution space to convert them into probability distributions, use KL divergence to measure the semantic differences between sentence embeddings, adjust the parameters of the embedding vectors, reduce the KL divergence of similar sentence pairs, and increase the KL divergence of dissimilar sentences; perform back propagation based on the loss function optimized by KL divergence, update the word vector parameters, save the optimized sentence vectors, and generate a word vector space;

[0031] The knowledge distillation module is used to train the knowledge distillation embedding model based on RoBERTa based on the generated word vector space, and migrate the rich semantic information in the deep model to the lightweight student model; during the training process, the performance indicators of the student model on the semantic similarity task are regularly verified until the performance indicators tend to be stable and meet the preset requirements, and the training is completed;

[0032] The output and visualization module is used to intuitively display the similarity calculation results and support the generation of analysis reports.

[0033] The monitoring and evaluation module is used to monitor the system performance in real time and regularly verify the performance indicators of the student model on the semantic similarity task.

[0034] As a preferred example, the output and visualization module presents the similarity calculation results in a visualization form including tables and graphs.

[0035] As a preferred example, the monitoring and evaluation module collects system operation data including response time and computing resource utilization to evaluate the system operation efficiency.

[0036] The beneficial effects of the present invention are as follows:

[0037] First, the similarity text mining method and system based on RoBERTa knowledge distillation of the present invention have more accurate semantic capture capabilities. Traditional sentence embedding generation methods, such as Word2Vec and GloVe, mainly generate sentence embeddings by simple averaging or weighted averaging of word vectors. Although they can capture the basic semantic information of sentences in some scenarios, they often perform insufficiently when faced with complex semantic dependencies. Especially when dealing with long sentences or polysemous words, these methods are prone to losing contextual information, resulting in a decrease in the accuracy of sentence similarity assessment. The present invention can better capture the subtle semantic differences between sentences by introducing KL divergence as a metric. KL divergence optimization makes the distance between similar sentences in the semantic space closer, and the distance between dissimilar sentences is effectively widened, thereby improving the accuracy of semantic similarity calculation.

[0038] Second, the similarity text mining method and system based on RoBERTa knowledge distillation of the present invention significantly improves the computing efficiency. Although the embedding generation methods based on deep learning, such as BERT and RoBERTa, have excellent performance in semantic expression, these models usually contain a large number of parameters, resulting in the training and reasoning process requiring huge computing resources and time, and it is difficult to adapt to real-time processing requirements. The present invention uses knowledge distillation technology to migrate the deep semantic information of large pre-trained models such as RoBERTa to a lightweight student model, thereby greatly reducing the computing overhead. The performance of the distilled student model in semantic similarity tasks is almost the same as that of the teacher model, but its computing efficiency is significantly improved, which is suitable for online application scenarios that require fast response, such as dialogue systems and search engines.

[0039] Third, the similarity text mining method and system based on RoBERTa knowledge distillation of the present invention has stronger model generalization ability. Traditional methods usually perform well in specific fields or specific data sets, but when migrating to other tasks or fields, the model often needs to be retrained. The sentence embedding generation method based on KL divergence optimization and knowledge distillation combines the deep semantic expression ability of the pre-trained model and the efficient characteristics after KL divergence optimization, and has stronger generalization ability. Through the iterative optimization of KL divergence, the model can better adapt to different task scenarios when capturing complex semantic relationships, reduce dependence on data set characteristics, and thus perform well in different natural language processing tasks.

[0040] Fourth, the similarity text mining method and system based on RoBERTa knowledge distillation of the present invention can support multiple similarity measurement methods. Traditional embedding methods often only support a single similarity calculation method, such as cosine similarity. This single similarity measurement may not provide sufficient flexibility when processing certain complex texts. The present invention not only supports cosine similarity, but also integrates multiple measurement methods such as TF-IDF weighted cosine similarity and smoothed inverse frequency (SIF) similarity. By introducing TF-IDF weights, this method can more accurately evaluate the similarity between sentences when processing long texts and texts with large differences in vocabulary importance. In addition, SIF similarity can effectively eliminate the interference of high-frequency words and improve the ability to recognize semantic differences in sentences in complex contexts.

[0041] Fifth, the similarity text mining method and system based on RoBERTa knowledge distillation of the present invention can adapt to the expansion needs of multiple scenarios and multiple tasks. Although traditional sentence embedding methods can perform well in certain specific tasks, they are often limited when extended to other application scenarios. The sentence embedding generation method of the present invention is highly flexible and adaptable, and can be applied to a variety of natural language processing tasks, such as text classification, sentiment analysis, information retrieval, machine translation, and dialogue systems. Through the combination of KL divergence and knowledge distillation, the model not only performs well in STS (semantic text similarity) tasks, but also maintains high precision and computational efficiency in these tasks, providing high-quality semantic embedding for various application scenarios.

[0042] Sixth, the similarity text mining method and system based on RoBERTa knowledge distillation of the present invention has real-time reasoning and large-scale deployment capabilities. As the scale of natural language processing applications continues to expand, the real-time reasoning capability and deployment performance of the model have become crucial. Large pre-trained models such as BERT often require high-performance hardware support when deployed in actual systems due to their huge number of parameters, and the reasoning speed is slow. Through knowledge distillation, the present invention effectively transfers the semantic capabilities of the teacher model to a smaller student model, reduces the number of parameters of the model, and significantly improves the reasoning speed. At the same time, the lightweight model is suitable for real-time reasoning in a variety of hardware environments, supports large-scale deployment, and meets application scenarios with high real-time requirements, such as online customer service systems, intelligent search engines, etc. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 This is a flow chart of a similarity text mining method based on RoBERTa knowledge distillation according to an embodiment of the present invention;

[0044] Figure 2 This is a schematic diagram of the knowledge distillation principle of an embodiment of the present invention;

[0045] Figure 3 A schematic diagram of the similarity between two related sentences before KL divergence optimization according to an embodiment of the present invention;

[0046] Figure 4 A schematic diagram of the similarity between two related sentences after KL divergence optimization according to an embodiment of the present invention;

[0047] Figure 5 A schematic diagram of the similarity between two unrelated sentences before KL divergence optimization according to an embodiment of the present invention;

[0048] Figure 6 A schematic diagram of the similarity between two unrelated sentences after KL divergence optimization according to an embodiment of the present invention;

[0049] Figure 7A graph showing the loss and correlation coefficients of multiple rounds of training of the model according to an embodiment of the present invention;

[0050] Figure 8 Schematic diagram of the results of calculating the similarity of the SemEval-2016 dataset using the SIF method. DETAILED DESCRIPTION

[0051] The following examples will enable those skilled in the art to more fully understand the present invention, but are not intended to limit the present invention in any way.

[0052] The present invention discloses a similarity text mining method based on RoBERTa knowledge distillation, the method comprising the following steps:

[0053] S1, map sentence embedding to probability distribution space to convert it into probability distribution. KL divergence (Kullback-Leibler Divergence) is an asymmetric measure used to measure the difference between two probability distributions. KL divergence is used to measure the semantic difference between sentence embeddings, adjust the parameters of the embedding vector, reduce the KL divergence of similar sentence pairs, and increase the KL divergence of dissimilar sentences; perform back propagation based on the loss function optimized by KL divergence, update the word vector parameters, save the optimized sentence vector, and generate the word vector space;

[0054] S2, based on the generated word vector space, train the knowledge distillation embedding model based on RoBERTa to transfer the rich semantic information in the deep model to the lightweight student model; during the training process, regularly verify the performance indicators of the student model on the semantic similarity task until the performance indicators tend to be stable and meet the preset requirements, and the training is completed;

[0055] S3, uses the trained student model to perform similarity text mining tasks.

[0056] See also Figure 1 The similarity text mining method of the present invention specifically includes the following steps:

[0057] 1. KL divergence optimization sentence vector

[0058] 1.1, Word vector generation and initialization: Word2Vec is a method for generating word embeddings, which can convert words in a text into numerical vectors. In this step, first load the pre-trained word vector model Word2Vec. This method captures the semantic relationship between words in the vector space, making the distance between semantically similar words in the vector space closer. Figure 1As shown in the figure, the model can be downloaded from the open source library and imported into the project. Next, the input sentences are preprocessed, including removing stop words, punctuation marks, and other meaningless characters to ensure the neatness of the text. TF-IDF is a statistical method used to evaluate the importance of a word to a document set or one of the documents in a corpus. Its main idea is that if a word appears frequently in an article (Term Frequency, TF) and rarely appears in other articles (Inverse Document Frequency, IDF), it is considered that the word has good category discrimination ability and has a good indication of the content of the article. The TF-IDF value is obtained by multiplying the TF and IDF parts. After processing, each sentence will generate its sentence embedding by averaging or TF-IDF weighted averaging. The generation of sentence embedding ensures that it can reflect the semantic characteristics of the sentence. In addition, for invalid or blank sentences, the system will filter them to avoid including these unqualified sentences in the model training. The output of this stage is a high-dimensional embedding representation of each valid sentence, which serves as the input for the subsequent steps.

[0059] 1.2, Probability distribution transformation of sentence vectors: After completing the generation of sentence embeddings, the next step is to transform these embeddings into probability distributions that meet the requirements of KL divergence calculation. To this end, the softmax function is used to process each sentence embedding and map it to a probability distribution space. This transformation ensures that each sentence embedding has a clear positioning in the semantic space, so that the subsequent KL divergence calculation can proceed smoothly. In specific implementation, the embedding vector needs to be input into the softmax function, and the output obtained is the probability distribution of the sentence in a given context. Ensure that in this process, the calculated values ​​are stable and conform to the properties of the probability distribution (that is, the sum of all values ​​is 1). This step lays the foundation for the subsequent KL divergence calculation.

[0060] 1.3, calculation and optimization of KL divergence: At this stage, the focus is on calculating the KL divergence between each pair of sentence embeddings and quantifying their semantic differences. The present invention not only focuses on reducing the KL divergence between similar sentence pairs, but also focuses on increasing the divergence between dissimilar sentence pairs. This dual-objective optimization strategy makes the model more discriminating in the semantic space. KL divergence provides an asymmetric metric that can measure the difference between two probability distributions P and Q. By processing similar sentence pairs, the system will reduce their KL divergence through an optimization algorithm, making these sentences closer in the embedding space. For dissimilar sentence pairs, the system will deliberately increase their KL divergence to distance them in the embedding space. This optimization process is achieved by adjusting the parameters of the embedding vector to ensure that the model can effectively distinguish the subtle differences between different semantics during the training process. The final result of this step is an optimized sentence embedding with higher semantic discrimination ability.

[0061] 1.4, Iterative optimization and model convergence: In the final stage of model training, the word vector parameters are updated through the back-propagation algorithm to minimize the loss function after KL divergence optimization. At each iteration, the overall loss is calculated and recorded to ensure that the loss gradually decreases with the number of iterations. The iteration process will continue until the model converges, which means that the loss no longer decreases significantly. Through this mechanism, the system can generate more accurate sentence embeddings and improve its performance in semantic similarity tasks. After the model converges, the optimized sentence vectors will be saved to a file for subsequent task calls and applications. This step ensures that the final model can effectively utilize the learned knowledge and has strong generalization ability and practicality. For example, first select a pair of related sentences and a pair of unrelated sentences, calculate and draw their similarity heat maps before and after optimization, the sentence pairs with high similarity are "A man puts three pieces of meat into a pan." and "A man is putting meat in a pan.", and the sentence pairs with low similarity are "Yes, there is a rule against this." and "There's no rule against it.". Figures 3 to 6 The heat map shows the cosine similarity between words in each sentence, with darker colors indicating higher similarity and lighter colors indicating lower similarity. Figure 3 Although the sentences show a certain degree of similarity as a whole, the similarity of some word pairs is low, indicating that there are semantic differences in the details of the sentences. After KL divergence optimization, Figure 4It shows more consistent high similarity areas, indicating that the optimization successfully reduced the semantic differences between sentences, making the representation of related sentence pairs more concentrated in the semantic space. In Figure 5, the word pair similarity of unrelated sentence pairs is relatively high, which may reflect that the word vector space fails to effectively distinguish the semantics of unrelated sentences. After KL divergence optimization, Figure 6 shows a more dispersed similarity distribution, especially the similarity of most word pairs is significantly reduced, indicating that the optimization effectively increases the semantic distance of unrelated sentence pairs.

[0062] 2. Knowledge distillation embedding model training based on RoBERTa

[0063] 2.1, Generate soft labels for the teacher model: Figure 2 As shown in the figure, in the first step of model training, the sentence pairs are input to the pre-trained RoBERTa teacher model and the output of its hidden layer is extracted. After these hidden layer outputs are processed by softmax, probability distributions can be generated as soft labels. These soft labels contain rich semantic information and can effectively guide the training process of the student model. The selection of the teacher model is crucial because its performance directly affects the learning effect of the student model. During the processing, it is necessary to ensure that the teacher model can adapt to the input sentence pairs and generate high-quality soft labels. The core of this step is to use the information generated by the teacher model to ensure that the student model can receive the best guidance during training.

[0064] 2.2. Initialization of the student model and preparation of training data: Next, initialize the structure of the student model to ensure that the size of its embedding layer is consistent with the sentence embedding dimension. This process includes setting hyperparameters such as the number of layers, number of nodes, and activation functions of the model to optimize model performance. When preparing training data, use TensorDataset to combine sentence embeddings with soft labels generated by the teacher model to form a training data set. This data set will be used in subsequent training processes to ensure that the student model can effectively learn under diverse inputs. When preparing data, it is also necessary to consider the data segmentation strategy so that the model can be effectively evaluated in the subsequent verification and testing phases.

[0065] 2.3, Knowledge distillation training process: During the training process, the mean square error (MSE) is used as the loss function to measure the difference between the output of the student model and the soft label of the teacher model. The training adopts batch processing, loading data batch by batch for training to improve training efficiency. The processing of each batch of data involves the optimization of model parameters, which are updated using the Adam optimizer, and the learning rate is set to 0.001 to ensure the stability of the training process. Through continuous iterations, the student model gradually learns the knowledge contained in the teacher model. At the same time, the training process needs to be evaluated regularly to ensure that the student model gradually approaches the performance of the teacher model. The success of this stage is directly related to the effectiveness and practical application value of the student model.

[0066] 2.4, Model training and verification: During the training process, the performance of the student model on the semantic similarity task is regularly evaluated and verified using the SemEval-2016 dataset. At this stage, it is necessary to calculate the performance indicators of the student model, such as the Pearson correlation coefficient (τ), the Spearman correlation coefficient (ρ), and the mean absolute error (MAE), to fully understand the performance of the model. If the model overfits or underfits during training, it is necessary to adjust the model parameters or introduce regularization techniques in a timely manner to improve the generalization ability of the model. By comparing the performance of the training set and the validation set, the stability and reliability of the model can be better evaluated, providing a basis for subsequent model adjustments. Figure 7 The comprehensive changes of loss value (Loss), Pearson correlation coefficient (τ), Spearman correlation coefficient (ρ) and mean absolute error (MAE) during the training process of this model are shown. The model was rapidly optimized in the first 200 rounds of training, and then all indicators tended to stabilize, showing good convergence. As the training deepened, the correlation indicators remained at a high level and the mean absolute error gradually decreased. Although the model still had slight fluctuations in the later stage of training, the overall performance has reached a relatively ideal state, reflecting that the model has good generalization ability.

[0067] 2.5. Model saving and loading: Once the student model training is completed, the weights of the student model need to be saved to a file for subsequent reasoning and deployment. The saved model weights include the parameters of all layers and the state of the optimizer, so that they can be restored to the state when training is completed during subsequent loading. This step ensures the repeatability and usability of the model in practical applications. After the model is saved, the system should have the ability to load the model quickly to support real-time reasoning requirements. Ultimately, the optimized and verified model will have strong semantic understanding capabilities and be able to perform well in different NLP tasks.

[0068] 3. Multiple similarity measures and application scenario expansion

[0069] Support multiple similarity measurement methods: When implementing multiple similarity measurement methods, you first need to design a cosine similarity calculation module to obtain the similarity value between sentences by calculating the dot product of sentence embeddings and normalizing them. Then, implement TF-IDF weighted cosine similarity to perform TF-IDF weighting on word vectors to improve the calculation accuracy in cases where the importance of keywords in long texts or sentences is different. SIF (Smooth Inverse Frequency) is a variant of TF-IDF that is used to calculate text feature weights. When calculating IDF, SIF does not simply take the logarithm, but uses a smooth function to avoid division by zero, and can control the weight difference of different words. Finally, introduce smooth inverse frequency (SIF) similarity to reduce the interference of high-frequency words by removing the principal component in sentence embeddings. The introduction of this series of methods can effectively improve the generalization ability of the model in complex contexts and lay a good foundation for a variety of application scenarios.

[0070] Application of semantic text similarity task: In the semantic text similarity task, the system will test the SemEval-2016 dataset and calculate the performance of different models. By evaluating indicators such as the Pearson correlation coefficient (τ), the Spearman correlation coefficient (ρ), and the mean absolute error (MAE), we can fully understand the similarity calculation effect of the model. Figure 8 The results of calculating cosine similarity using the SIF method are shown. The model of the present invention performs well in τ, ρ and MAE, reaching 0.518, 0.517 and 1.287 respectively. Compared with other models, the model of the present invention shows stronger robustness and accuracy in the smoothed inverse frequency method. Experiments show that sentence vectors optimized by KL divergence perform well in multiple indicators compared with traditional word vector models (such as Word2Vec, FastText) and pre-trained language models (such as BERT), showing significant performance improvement. The experimental data at this stage will provide strong support for subsequent optimization and promotion and application.

[0071] Expanded application scenarios: Through optimized sentence embedding, the system can be widely used in scenarios such as text classification, information retrieval, dialogue systems, and sentiment analysis. In text classification, sentence embedding is used to improve the accuracy and robustness of the classification model; in information retrieval, similarity evaluation based on sentence embedding can achieve efficient matching of user queries and document content. For dialogue systems, sentence embedding can enhance the contextual understanding ability of multi-round dialogues, while in sentiment analysis, the system can achieve accurate judgment of user emotions. The expanded application of various scenarios will help improve the practicality of the system.

[0072] Real-time reasoning and deployment of models: In order to achieve real-time reasoning of models, the system needs to support efficient operation on different hardware platforms (such as GPU and CPU). This process first involves the loading of the model and the design of the reasoning process. By optimizing the reasoning process, it can quickly respond to different natural language processing task requirements, such as intelligent customer service systems or virtual assistants. During real-time reasoning, the input text data will undergo necessary preprocessing and converted into the input format required by the model. Subsequently, the trained model is used for reasoning to obtain the corresponding output. This process needs to minimize latency to ensure user experience. In addition, the system must have the ability to reason in batches to support large-scale deployment and ensure the stability and efficiency of the model under high concurrency.

[0073] Performance optimization and resource management: In the process of system performance optimization, the use of batch processing technology can significantly improve reasoning efficiency and reduce the consumption of computing resources. This process first requires analyzing the performance bottleneck of the system, identifying the links with low resource utilization, and then optimizing according to the actual situation. For example, the batch size can be adjusted to find the best computing resource utilization while reducing the idle time of the GPU / CPU. In addition, it is also crucial to dynamically adjust the computing resources of the model to cope with different load conditions. This mechanism can be achieved by monitoring system performance indicators (such as response time, CPU / GPU load, etc.) to ensure that the model can still run stably under high load conditions and ensure the user's access experience.

[0074] Based on the above method, the embodiment of the present invention also proposes a system module component of a similarity text mining method based on RoBERTa knowledge distillation, which specifically includes:

[0075] Data input module: The data input module is responsible for supporting input in multiple text formats, including text files, CSV, JSON, etc. The system will clean and preprocess the input data, remove useless characters, standardize the text format, and perform word segmentation. The goal of data preprocessing is to ensure the quality of input data and lay a solid foundation for subsequent sentence embedding generation. By achieving compatibility with multiple data formats, the system can more flexibly access data from different sources, ensuring its efficiency and usability in practical applications.

[0076] Sentence embedding generation module: This module uses a pre-trained word vector model to generate high-dimensional sentence embedding representations. First, the pre-trained model is called on the word segmentation results of the input sentence to generate the word vector for each word, and then these word vectors are averaged or weighted averaged to obtain the overall embedding representation of the sentence. In the process of generating embeddings, the system needs to ensure that invalid sentences are filtered to ensure that the final generated sentence embedding is of high quality and accuracy. This module is the core part of the system, and its performance directly affects the subsequent similarity calculation and text mining tasks.

[0077] Divergence calculation module: The divergence calculation module calculates the divergence between sentence embeddings using the KL divergence method. This module first receives the generated sentence embeddings and then converts them into probability distributions using the softmax function. Subsequently, the similarity or difference between sentences is evaluated by calculating the KL divergence. The results of the similarity calculation will be used for subsequent text mining tasks such as information retrieval or text classification. In this process, the system needs to ensure efficient calculations to cope with the processing needs of large-scale text data.

[0078] Knowledge distillation module: The main task of the knowledge distillation module is to optimize the training process of the student model. In this process, the system inputs sentence pairs to the RoBERTa teacher model, obtains its hidden layer output, and generates soft labels. These soft labels provide rich semantic information for the training of the student model, ensuring that the student model can gradually learn the knowledge of the teacher model during the training process. The module implements an effective training data processing and model parameter update mechanism to ensure the efficiency and accuracy of the knowledge distillation process.

[0079] Output and visualization module: This module is responsible for intuitively displaying the similarity calculation results and supports the generation of analysis reports. The output results can be presented in tables, graphs, or other visual forms, allowing users to quickly understand the performance of the model and the similarity between texts. In addition, it supports exporting the results to multiple formats (such as PDF, Excel, etc.) for further analysis and sharing. The design of the visualization module needs to take into account the user experience, ensure that the information display is clear and concise, and improve the usability of the system.

[0080] Monitoring and evaluation module: The monitoring and evaluation module is responsible for real-time monitoring of system performance and regular evaluation of the performance and effect of the model. The system's operating efficiency is evaluated by collecting system operation data, such as response time, computing resource utilization, etc. In addition, this module is also required to implement regular review of the model's prediction results to ensure the stability and accuracy of the model in practical applications. Through continuous monitoring and evaluation, the system can promptly identify potential problems and make corresponding optimizations and adjustments to ensure the efficiency and reliability of the overall operation.

[0081] During the experimental deployment process, you first need to prepare the hardware environment to ensure that the system can run smoothly. Choose a server with at least a 16-core processor and a high-performance GPU (such as NVIDIA A100 or RTX 3090) to accelerate the training and reasoning of deep learning models. At the same time, ensure that the memory is at least 64GB, and for storage, choose an SSD of more than 1TB to increase data reading and writing speed.

[0082] Next, install the operating system. Ubuntu 20.04 or CentOS 7 is recommended. During the installation process, be sure to update the system to the latest stable version for optimal performance and security, and set up a firewall to protect the system. Then, install Python 3.8 or higher, and use pip or conda to install the required deep learning frameworks (such as TensorFlow or PyTorch), data processing libraries (such as NumPy and Pandas), similarity calculation libraries (such as Scikit-learn), and visualization tools (such as Matplotlib and Seaborn).

[0083] When configuring the network, ensure a Gigabit Ethernet connection for fast data processing and transmission, set up firewall rules, and use VPN technology to protect remote access. For data storage, choose a suitable database management system (such as PostgreSQL or MongoDB) to store processed data and model output results. For large-scale data sets, consider using a distributed file system (such as HDFS) to ensure efficient access and management.

[0084] In terms of the configuration of monitoring and management tools, Prometheus and Grafana are installed for system performance monitoring, real-time tracking of CPU, GPU utilization and memory usage. In addition, the ELK (Elasticsearch, Logstash, Kibana) stack can be used for log management and visualization to help troubleshoot and optimize performance.

[0085] Next, set up an independent test environment and use standard data sets (such as the SICK data set) to evaluate the model. Verify the model effect by calculating indicators such as the Pearson correlation coefficient and the Spearman correlation coefficient. Train the model in the local environment, monitor performance indicators during training, such as loss value and accuracy, and adjust hyperparameters as needed to optimize model performance.

[0086] After training is complete, the model is packaged as a Docker image, including writing a Dockerfile and configuring environment variables to ensure the portability of the image in different environments. Use Docker Compose or Kubernetes for container orchestration to facilitate the management of multiple services. Subsequently, a comprehensive test is performed in the production environment to ensure the stability and responsiveness of the system, and the performance of the model is verified by simulating user requests.

[0087] Finally, monitoring tools are used to observe system performance in real time, and computing resources and model parameters are dynamically adjusted according to actual load to ensure efficient and stable operation of the system.

[0088] The above are only preferred embodiments of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should be regarded as the protection scope of the present invention.

Claims

1. A similarity text mining method based on RoBERTa knowledge distillation, characterized in that: The method comprises the following steps: S1, map sentence embeddings to probability distribution space to convert them into probability distributions, use KL divergence to measure the semantic differences between sentence embeddings, adjust the parameters of the embedding vectors, reduce the KL divergence of similar sentence pairs while increasing the KL divergence of dissimilar sentences; perform back propagation based on the loss function optimized by KL divergence, update the word vector parameters, save the optimized sentence vectors, and generate the word vector space; S2, based on the generated word vector space, train the knowledge distillation embedding model based on RoBERTa to transfer the rich semantic information in the deep model to the lightweight student model; during the training process, regularly verify the performance indicators of the student model on the semantic similarity task until the performance indicators tend to be stable and meet the preset requirements, and the training is completed; S3, uses the trained student model to perform similarity text mining tasks.

2. The similarity text mining method based on RoBERTa knowledge distillation according to claim 1, characterized in that: Step S1 further comprises: S11, load the Word2Vec pre-trained model to generate word vectors; pre-process the input sentence, and generate the initial sentence embedding by averaging all the word vectors in the sentence or using TF-IDF weighted average. The initial sentence embedding is used to reflect the semantic features of the sentence; S12, using the softmax function to map sentence embeddings to the probability distribution space to convert them into probability distributions. The probability distributions generated by each pair of sentence embeddings represent the location of the sentences in the semantic space. S13, adjust the parameters of each sentence embedding vector to reduce the KL divergence of similar sentence pairs while increasing the KL divergence of dissimilar sentences; S14, update the word vector parameters through the back-propagation algorithm to minimize the loss function after KL divergence optimization; calculate the overall loss after each iteration until the model converges and generates sentence embedding; save the optimized sentence vector and generate the word vector space.

3. The similarity text mining method based on RoBERTa knowledge distillation according to claim 2 is characterized in that: In step S11, the process of preprocessing the input sentence includes the following steps: Filter invalid or blank sentences; for the remaining sentences, remove stop words, punctuation marks, and other meaningless characters.

4. The similarity text mining method based on RoBERTa knowledge distillation according to claim 1, characterized in that: Step S2 further comprises: S21, input sentence pairs to the pre-trained RoBERTa teacher model, obtain its hidden layer output, and use the probability distribution of the calculated output as the soft label; the soft label of each sentence contains rich semantic information, which is used to guide the training of the student model; S22, initialize the structure of the student model, set the hyperparameters of the student model including the number of layers, number of nodes and activation function, so that the size of its embedding layer is consistent with the dimension of sentence embedding; use TensorDataset to combine the sentence embedding and the soft labels of the RoBERTa teacher model into a training dataset; S23, using the mean square error loss function, aligns the prediction results of the student model with the soft labels generated by the teacher model, and uses the training data set to train the student model; during the training process, load the data batch by batch, and use the Adam optimizer to optimize the parameters of the student model so that its output tends to be consistent with the performance of the teacher model; S24, regularly evaluate the performance indicators of the student model on the semantic similarity task until the performance indicators tend to be stable and meet the preset requirements, complete the training, and save the weight of the trained student model.

5. The similarity text mining method based on RoBERTa knowledge distillation according to claim 4 is characterized in that: In step S24, the performance indicators include one or more of the loss value, Pearson correlation coefficient, Spearman correlation coefficient and mean absolute error of the training process.

6. The similarity text mining method based on RoBERTa knowledge distillation according to claim 1, characterized in that: In step S3, the process of performing the similarity text mining task includes: Design a cosine similarity calculation module to obtain the similarity value between sentences by calculating the dot product of sentence embeddings and normalizing them; Implement TF-IDF weighted cosine similarity to perform TF-IDF weighted processing on word vectors to improve the calculation accuracy when the importance of keywords in long texts or sentences is different; Smoothed inverse frequency similarity is introduced to remove the principal components in sentence embedding to reduce the interference of high-frequency words.

7. A similarity text mining system based on RoBERTa knowledge distillation based on the method according to any one of claims 1 to 6, characterized in that: The system includes a data input module, a sentence embedding generation module, a divergence calculation module, a knowledge distillation module, an output and visualization module, and a monitoring and evaluation module; The data input module is used to clean and preprocess input data in various text formats, and perform word segmentation on the preprocessed text; The sentence embedding generation module is used to generate high-dimensional sentence embedding using a pre-trained word vector model; The divergence calculation module is used to map sentence embeddings to a probability distribution space to convert them into probability distributions, use KL divergence to measure the semantic differences between sentence embeddings, adjust the parameters of the embedding vectors, reduce the KL divergence of similar sentence pairs, and increase the KL divergence of dissimilar sentences; perform back propagation based on the loss function optimized by KL divergence, update the word vector parameters, save the optimized sentence vectors, and generate a word vector space; The knowledge distillation module is used to train the knowledge distillation embedding model based on RoBERTa based on the generated word vector space, and migrate the rich semantic information in the deep model to the lightweight student model; during the training process, the performance indicators of the student model on the semantic similarity task are regularly verified until the performance indicators tend to be stable and meet the preset requirements, and the training is completed; The output and visualization module is used to intuitively display the similarity calculation results and support the generation of analysis reports; The monitoring and evaluation module is used to monitor the system performance in real time and regularly verify the performance indicators of the student model on the semantic similarity task.

8. The similarity text mining system based on RoBERTa knowledge distillation according to claim 7, characterized in that: The output and visualization module presents the similarity calculation results in a visualization form including tables and graphs.

9. The similarity text mining system based on RoBERTa knowledge distillation according to claim 7, characterized in that: The monitoring and evaluation module collects system operation data including response time and computing resource utilization, and evaluates the system operation efficiency.

Images