Sentence representation method for dynamic course oriented to contrastive learning
By calculating the difficulty of positive samples and conducting dynamic course learning, the problem of unstable quality of positive samples in contrastive learning is solved, achieving higher quality sentence representation and improved model performance.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- HEFEI UNIV OF TECH
- Filing Date
- 2023-03-06
- Publication Date
- 2026-04-21
AI Technical Summary
In existing contrastive learning methods, the quality of positive samples generated through data augmentation is unstable, leading to inaccurate sentence representations and limiting model performance.
By calculating the difficulty of positive samples and performing dynamic course learning, a set of sentence pairs is generated from easy to difficult. Self-supervised learning is used to optimize the model training process, and the sample order is dynamically updated to improve the quality of sentence representation.
It improves the accuracy of sentence representation and model performance, and generates higher quality sentence representations.
Smart Images
Figure CN116341515B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of natural language processing, specifically to a sentence representation method for dynamic courses oriented towards contrastive learning. Background Technology
[0002] With the rapid development of internet technology, language, as a fundamental human ability, makes machine processing of natural language an inevitable requirement for productivity development. Therefore, research in natural language processing (NLP) is of significant value and importance. Sentence representation, as one of the main research directions in the field of NLP, is applied to specific downstream tasks through pre-training and fine-tuning.
[0003] Early sentence representation methods, such as one-hot encoding and TF-IDF based on bag-of-words models, resulted in sparse data. Later methods utilized Word2Vec, employing a sliding window to specify a fixed-size context, predicting the context from the current word or vice versa. However, because the relationship between words and vectors is one-to-one, it cannot address the problem of polysemy. BERT pre-training aims to learn general sentence representations and transfer them to multiple downstream applications. Directly using BERT to obtain sentence representations is not very effective because the semantic vector distribution generated directly by the language model suffers from non-linearity and anisotropy, leading to high similarity between any two sentences and collapsing the semantic space into a narrow area.
[0004] Self-supervised learning trains networks by mining supervisory information from large-scale unsupervised data using auxiliary tasks, thereby learning representations valuable for downstream tasks. Contrastive learning, a type of self-supervised learning, alleviates the anisotropy problem caused by BERT sentence representations by bringing similar data closer together and separating dissimilar data during model training, thus obtaining effective data representations. In contrastive learning, different positive samples are typically obtained through data augmentation. In the same batch of training data, samples other than anchor samples are used as negative samples to better learn effective information between data. The core of contrastive learning lies in how to construct the set of positive and negative samples. In computer vision, data augmentation is performed by rotating, cropping, flipping, and changing the color of images. In natural language processing (NLP), data augmentation methods often construct positive samples through back-translation, random deletion, character insertion, and synonym replacement. However, due to the highly abstract nature of natural language, these data augmentation methods cannot maintain the quality of the generated positive samples. For example, if the generated positive samples are too easy or too difficult, ignoring the quality of the samples and treating all generated positive samples equally will lead to inaccurate sentence representations, thus limiting the performance in contrastive learning. Therefore, ensuring the quality of positive samples plays a crucial role in sentence representation for contrastive learning. Summary of the Invention
[0005] This invention addresses the shortcomings of existing positive sample quality in contrastive learning by proposing a sentence representation method for dynamic courses based on contrastive learning. The aim is to more fully mine the information in positive sample representations, achieve more accurate modeling of sentence representations, and thus improve the performance of sentence representation.
[0006] The present invention adopts the following technical solution to solve the technical problem:
[0007] The sentence representation method for dynamic courses oriented towards contrastive learning, as described in this invention, is characterized by the following steps:
[0008] Step 1: Preprocessing of text data:
[0009] Step 1.1: Obtain n sentences S = {s1, ..., s2} i ,…,s n After data augmentation, the augmented sentence dataset is obtained, denoted as} Among them, s i Let represent the i-th anchor sentence, and n represent the total number of sentences; Suppose the i-th anchor sentence s i The enhanced set of sentences is used as the i-th positive sample set, and Suppose the i-th anchor sentence s i The enhanced j-th sentence is taken as a positive sample, and m represents the total number of enhanced sentences for each sentence;
[0010] Let S′ denote the set of preprocessed sentence pairs, and
[0011] Step 2: Construct a contrastive learning model, including: an encoding module and a projection module; wherein the projection module consists of a single MLP linear layer;
[0012] Step 2.1: The encoding module encodes each sentence in the sentence set S′ to obtain a representation. Among them, h i Suppose the i-th anchor sentence s i The representation vector, This represents the enhanced j-th positive sample sentence. The representation vector;
[0013] Step 2.2: The projection module represents the sentence vector h. i and Mapping to a low-dimensional semantic vector space, we obtain the sentence vector representation as follows: Among them, z i Suppose the i-th anchor sentence s iThe representation vector h i The low-dimensional sentence representation vector; This represents the j-th sentence. representation vector The low-dimensional sentence representation vector;
[0014] Step 3, Calculation with h i The cosine similarity distance in the semantic space is used as the difficulty d of the j-th sentence in the i-th positive sample set. i,j Thus, we obtain the difficulty set D = {d} of n×m positive samples. 1,1 ,…,d i,j ,…d n,m The difficulty set D is sorted in descending order to obtain a sorted difficulty set. Then, based on this sorted difficulty set, the n×m positive samples are sorted to obtain a sorted set of positive sample sentence pairs. Finally, the sorted set of positive sample sentence pairs is paired with their corresponding anchor sentences to obtain a sorted set of sentence pairs. Among them, S i This represents the i-th anchor sentence after sorting. S represents the i-th anchor sentence after sorting. i The j-th positive sample sentence in the corresponding set of positive sample sentences;
[0015] Step 4: Select the difficulty of positive samples. Let the total number of training iterations of the contrastive learning model be T, the current iteration number be t, and initialize t = 1.
[0016] Step 4.1: Calculate the capability c(t) of the contrastive learning model in the current t-th iteration using equation (1), where c(t)∈(0,1]:
[0017]
[0018] In equation (1), c0 is the initial capability of the contrastive learning model and is initialized to a fixed value;
[0019] Step 4.2: Calculate the number of positive samples P(t) of the contrastive learning model in the current iteration t using equation (2):
[0020] P(t)=n×m×c(t) (2)
[0021] Step 4.3: Let win be the size of the sliding window, and let the training interval for positive samples be [P(t)-win, P(t)], where P(t) ≥ win; then, partition the sorted sentence set S″ according to the training interval [P(t)-win, P(t)] to obtain R from easy to difficult in the current t-th iteration. t training samples in batches; let r be... tFor R t Any r-th batch t One training sentence pair;
[0022] Step 5: Construct R for the t-th iteration using equation (3). t The loss function L for each batch t ;
[0023]
[0024] In equation (3), λ represents the adjustment parameter, and τ represents the temperature parameter. Represents the r-th order after sorting. t Anchor sentence The low-dimensional sentence representation vector of the representation vector. Represents the r-th order after sorting. t Anchor sentence The j-th positive sample sentence in the corresponding set of positive sample sentences The low-dimensional sentence representation vector of the representation vector, where sim represents the cosine similarity;
[0025] Step 6: In the current t-th iteration, R t Training samples from each training batch are sequentially input into the contrastive learning model, and the model is iteratively trained using gradient descent, while simultaneously calculating the loss function L. t To update the model parameters until the loss function L is reached. t The process continues until convergence, thus obtaining the optimal contrastive learning model for the current t-th iteration;
[0026] Step 7: After assigning t+1 to t, determine whether t>T holds true. If it does, use the best contrastive learning model in the Tth iteration as the model with the best sentence representation performance to achieve higher quality sentence representation; otherwise, return to step 4.1 and execute sequentially.
[0027] The present invention provides an electronic device, including a memory and a processor, wherein the memory is used to store a program that supports the processor in executing the sentence representation method, and the processor is configured to execute the program stored in the memory.
[0028] The present invention discloses a computer-readable storage medium on which a computer program is stored, wherein the computer program is executed by a processor to perform the steps of the sentence representation method.
[0029] Compared with the prior art, the beneficial effects of the present invention are reflected in:
[0030] 1. This invention addresses the issue that the random sampling of positive samples generated through data augmentation in existing contrastive learning sentence representation methods can negatively impact the performance of contrastive learning models. Therefore, it proposes a dynamic curriculum-based sentence representation method for contrastive learning. Steps 2 and 3 calculate the similarity between the anchor sentence and the augmented positive sample sentence to determine the difficulty of the current augmented sentence; higher similarity indicates lower difficulty. The augmented sentence pairs are then sorted in descending order of difficulty, resulting in a sequence of sentence pairs from easy to difficult. This alleviates the performance impact of varying positive sample quality on contrastive learning models, thereby achieving higher-quality sentence representations.
[0031] 2. This invention considers different levels of difficulty for each generated sample in different models, and even for the same model, the difficulty of samples considered varies at different learning stages. The dynamic course comparison learning in steps 5 and 6 dynamically sorts samples based on the current model's difficulty, thereby more accurately updating the ranking of positive sample quality. This improves the performance of the comparison learning model and generates higher-quality sentence representations.
[0032] 3. This invention utilizes contrastive learning for pre-training via self-supervision. Positive and negative samples are constructed as supervision signals, and during model training, the objective of contrastive learning is better optimized by iteratively reordering positive samples. Specifically, this invention not only learns the semantic information of sentence representations in contrastive learning but also captures the semantic information of positive sample sentences more precisely and with finer granularity. Parameter tuning can lead to high-quality sentence representations in the model. Attached Figure Description
[0033] Figure 1 This is a flowchart of a dynamic course sentence representation method for contrastive learning according to the present invention. Detailed Implementation
[0034] In this embodiment, a dynamic curriculum-based sentence representation method for contrastive learning considers that positive samples generated by existing data augmentation methods can affect the performance of contrastive learning models during random sampling. By calculating the difficulty of generating positive samples and introducing the concept of curriculum learning, these samples are ranked to train sentences of varying quality. Furthermore, since the difficulty of the samples is relative to the model, and the model's performance changes during training, the model re-represents sentence pairs and recalculates sentence difficulty. The order of training samples is dynamically updated after each training iteration, enabling dynamic curriculum learning and thus improving the performance of the contrastive learning model and generating higher-quality sentence representations. Specifically, as... Figure 1 As shown, the method is performed according to the following steps.
[0035] Step 1: Preprocessing of text data:
[0036] Step 1.1: Randomly sample n sentences S = {s1, ..., s2} from the English-Chinese Wikipedia dataset. i ,…,s n After data augmentation, the augmented sentence dataset is obtained, denoted as} Among them, s i Let represent the i-th anchor sentence, and n represent the total number of sentences; Suppose the i-th anchor sentence s i The enhanced set of sentences is used as the i-th positive sample set, and Suppose the i-th anchor sentence s i The enhanced j-th sentence is taken as a positive sample, and m represents the total number of enhanced sentences for each sentence;
[0037] Let S′ denote the set of preprocessed sentence pairs, and
[0038] Step 2: Construct a contrastive learning model, including an encoding module and a projection module; wherein the projection module consists of a single MLP linear layer.
[0039] Step 2.1: The encoding module encodes each sentence in the sentence set S′ using BERT-base to obtain a representation. Among them, h i Suppose the i-th anchor sentence s i The representation vector, This represents the enhanced j-th positive sample sentence. The representation vector;
[0040] Step 2.2: The projection module represents the sentence vector h. i and Mapping to a low-dimensional semantic vector space, we obtain the sentence vector representation as follows: Among them, z i Suppose the i-th anchor sentence s i The representation vector h i The low-dimensional sentence representation vector; This represents the j-th sentence. representation vector The low-dimensional sentence representation vector;
[0041] Step 3, Calculation with h i The cosine similarity distance in the semantic space is used as the difficulty d of the j-th sentence in the i-th positive sample set. i,j Thus, we obtain the difficulty set D = {d} of n×m positive samples. 1,1 ,…,d i,j,…d n,m The difficulty set D is sorted in descending order to obtain a sorted difficulty set. Then, based on this sorted difficulty set, the n×m positive samples are sorted to obtain a sorted set of positive sample sentence pairs. Finally, the sorted set of positive sample sentence pairs is paired with their corresponding anchor sentences to obtain a sorted set of sentence pairs. Among them, S i This represents the i-th anchor sentence after sorting. S represents the i-th anchor sentence after sorting. i The j-th positive sample sentence in the corresponding set of positive sample sentences;
[0042] Step 4: Selecting the difficulty of positive samples. To avoid overfitting due to repeated training on simple samples, and considering the performance of the current model, we determine the size of the sliding window by considering the capabilities of the current model:
[0043] Let the total number of training iterations of the contrastive learning model be T, the current iteration number be t, and initialize t = 1;
[0044] Step 4.1: Calculate the capability c(t) of the contrastive learning model in the current t-th iteration using equation (1), where c(t)∈(0,1]:
[0045]
[0046] In equation (1), c0 is the initial capability of the contrastive learning model and is initialized to a fixed value;
[0047] Step 4.2: Calculate the number of positive samples P(t) of the contrastive learning model in the current iteration t using equation (2):
[0048] P(t)=n×m×c(t) (2)
[0049] Step 4.3: Let win be the size of the sliding window, and let the training interval for positive samples be [P(t)-win, P(t)], where P(t) ≥ win; then, partition the sorted sentence set S″ according to the training interval [P(t)-win, P(t)] to obtain R from easy to difficult in the current t-th iteration. t training samples in batches; let r be... t For R t Any r-th batch t One training sentence pair;
[0050] Step 5: Construct R for the t-th iteration using equation (3). t The loss function L for each batch t ;
[0051]
[0052] In equation (3), λ represents the adjustment parameter, and τ represents the temperature parameter. Represents the r-th order after sorting. t Anchor sentence The low-dimensional sentence representation vector of the representation vector. Represents the r-th order after sorting. t Anchor sentence The j-th positive sample sentence in the corresponding set of positive sample sentences The low-dimensional sentence representation vector of the representation vector, where sim represents the cosine similarity;
[0053] Step 6: In the current t-th iteration, R t Training samples from each training batch are sequentially input into the contrastive learning model, and the model is iteratively trained using gradient descent, while simultaneously calculating the loss function L. t Update the model parameters until the loss function L is reached. t The process continues until convergence, thus obtaining the optimal contrastive learning model for the current t-th iteration;
[0054] Step 7: After assigning t+1 to t, determine whether t>T holds true. If true, use the optimal contrastive learning model in the Tth iteration as the model with the best sentence representation performance to achieve higher quality sentence representation; otherwise, return to step 4.1 and execute sequentially. In this embodiment, the model converges and achieves optimal results when t=4.
[0055] In this embodiment, an electronic device includes a memory and a processor. The memory stores a program that supports the processor in executing the methods described above, and the processor is configured to execute the program stored in the memory.
[0056] In this embodiment, a computer-readable storage medium stores a computer program, which is executed by a processor to perform the steps of the above method.
[0057] Experimental examples:
[0058] To verify the effectiveness of this method, this invention compares several competitive unsupervised semantic representation models, including GloVe embeddings (average), BERT-base (first-last average), BERT-flow, BERT-whitening, IS-BERT, ConSERT, SimCSE, ESimCSE, and DiffCSE. The models are evaluated using unsupervised data from seven semantic text similarity (STS) and seven transfer tasks (TR) in the SentEval toolkit, with the Spearman coefficient as the evaluation metric. The STS tasks include STS 2012-2016, STS Benchmark, and SICK-Relatedness. In these datasets, each sample consists of two sentences, with similarity scores ranging from 0 to 5 to indicate the degree of semantic similarity.
[0059] Table 1. Performance of different sentence embedding models on the STS task (Spearman's correlation)
[0060]
[0061]
[0062] Table 2 Results of different sentence embedding models on the transfer task
[0063]
[0064]
[0065] Tables 1 and 2 show the model's performance on seven semantic text similarity (STS) and transfer tasks (TR), respectively. They also compare the recommendation performance of the proposed method with contrasting methods. The experimental results show that the proposed method outperforms existing methods on Spearman's correlation metric. Furthermore, the results verify that this method improves sentence representation quality by using dynamic curriculum learning that incorporates model performance to rank the quality of positive samples and thus obtains more semantically distinct representations.
Claims
1. A sentence representation method for dynamic courses oriented towards contrastive learning, characterized in that, The procedure is as follows: Step 1: Preprocessing of text data: Step 1.1: Obtain n sentences S = {s1, ..., s2} i ,…,s n After data augmentation, the augmented sentence dataset is obtained, denoted as} Among them, s i Let represent the i-th anchor sentence, and n represent the total number of sentences; Suppose the i-th anchor sentence s i The enhanced set of sentences is used as the i-th positive sample set, and Suppose the i-th anchor sentence s i The enhanced j-th sentence is taken as a positive sample, and m represents the total number of enhanced sentences for each sentence; Let S′ denote the set of preprocessed sentence pairs, and Step 2: Construct a contrastive learning model, including: an encoding module and a projection module; wherein the projection module consists of a single MLP linear layer; Step 2.1: The encoding module encodes each sentence in the sentence set S′ to obtain a representation. Among them, h i Suppose the i-th anchor sentence s i The representation vector, This represents the enhanced j-th positive sample sentence. The representation vector; Step 2.2: The projection module represents the sentence vector h. i and Mapping to a low-dimensional semantic vector space, we obtain the sentence vector representation as follows: Among them, z i Suppose the i-th anchor sentence s i The representation vector h i The low-dimensional sentence representation vector; This represents the j-th sentence. representation vector The low-dimensional sentence representation vector; Step 3, Calculation with h i The cosine similarity distance in the semantic space is used as the difficulty d of the j-th sentence in the i-th positive sample set. i,j Thus, we obtain the difficulty set D = {d} of n×m positive samples. 1,1 ,…,d i,j ,…d n,m The difficulty set D is sorted in descending order to obtain a sorted difficulty set. Then, based on this sorted difficulty set, the n×m positive samples are sorted to obtain a sorted set of positive sample sentence pairs. Finally, the sorted set of positive sample sentence pairs is paired with their corresponding anchor sentences to obtain a sorted set of sentence pairs. Among them, S i This represents the i-th anchor sentence after sorting. S represents the i-th anchor sentence after sorting. i The j-th positive sample sentence in the corresponding set of positive sample sentences; Step 4: Select the difficulty of positive samples. Let the total number of training iterations of the contrastive learning model be T, the current iteration number be t, and initialize t = 1. Step 4.1: Calculate the capability c(t) of the contrastive learning model in the current t-th iteration using equation (1), where c(t)∈(0,1]: In equation (1), c0 is the initial capability of the contrastive learning model and is initialized to a fixed value; Step 4.2: Calculate the number of positive samples P(t) of the contrastive learning model in the current iteration t using equation (2): P(t)=n×m×c(t) (2) Step 4.3: Let win be the size of the sliding window, and let the training interval for positive samples be [P(t)-win, P(t)], where P(t) ≥ win; then, partition the sorted sentence pair set S″ according to the training interval [P(t)-win, P(t)] to obtain R from easy to difficult in the current t-th iteration. t training samples in batches; let r be... t For R t Any r-th batch t One training sentence pair; Step 5: Construct R for the t-th iteration using equation (3). t The loss function L for each batch t ; In equation (3), λ represents the adjustment parameter, and τ represents the temperature parameter. Represents the r-th order after sorting. t Anchor sentence The low-dimensional sentence representation vector of the representation vector. Represents the r-th order after sorting. t Anchor sentence The j-th positive sample sentence in the corresponding set of positive sample sentences The low-dimensional sentence representation vector of the representation vector, where sim represents the cosine similarity; Step 6: In the current t-th iteration, R t Training samples from each training batch are sequentially input into the contrastive learning model, and the model is iteratively trained using gradient descent, while simultaneously calculating the loss function L. t To update the model parameters until the loss function L is reached. t The process continues until convergence, thus obtaining the optimal contrastive learning model for the current t-th iteration; Step 7: After assigning t+1 to t, determine whether t>T holds true. If it does, use the best contrastive learning model in the Tth iteration as the model with the best sentence representation performance to achieve higher quality sentence representation; otherwise, return to step 4.1 and execute sequentially.
2. An electronic device, comprising a memory and a processor, characterized in that, The memory is used to store a program that supports the processor in executing the sentence representation method of claim 1, and the processor is configured to execute the program stored in the memory.
3. A computer-readable storage medium storing a computer program, characterized in that, The computer program is executed by the processor to perform the steps of the sentence representation method of claim 1.