Unsupervised Chinese contrast learning method based on prompt learning and mutual information
By constructing positive and negative sample pairs and denoising them based on cue learning and mutual information, the unsupervised Chinese contrastive learning model is optimized, which solves the problems of data augmentation noise and insufficient generalization ability, and improves the robustness and feature representation alignment of the model.
Patent Information
- Application Number
- CN202411136219.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-19
- Publication Date
- 2026-03-03
AI Technical Summary
Existing techniques in unsupervised Chinese contrastive learning suffer from problems such as data augmentation strategies introducing noise that leads to unstable representations, and poor generalization ability on small datasets.
We employ a cue-based learning and mutual information approach. We construct positive and negative sample pairs by manually designing cue templates, encode them using a RoBERTa pre-trained model, and denoise at the representation level. We optimize the joint loss function by maximizing the distributional similarity of the positive attention tensors through mutual information.
It improves the robustness of the model and its generalization ability on small datasets, reduces the impact of noise, and enhances the alignment of feature representations between pairs.
Smart Images
Figure CN121598091A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of semantic text similarity technology, specifically to an unsupervised Chinese contrastive learning method based on cue learning and mutual information. Background Technology
[0002] Semantic Textual Similarity (STS) is an important task in Natural Language Processing (NLP), aiming to evaluate the semantic similarity between two texts. Traditional methods mainly rely on the Bag-of-Words (BOW) model and TF-IDF, but these methods suffer from drawbacks such as ignoring word order, failing to capture contextual information, and high-dimensional sparsity, and have been gradually abandoned. Word2Vec and GloVe are also outdated due to their neglect of global information, only considering local context and inability to handle polysemous words. With the advent of pre-trained language models, methods based on pre-trained language models have improved upon the problem of ignoring global information. Models such as BERT and RoBERTa can simultaneously consider the contextual information on both sides of a word, generating richer and more accurate word vector representations. However, the vectors generated by models like BERT and RoBERTa are biased by the different data distributions of different datasets, leading to poor performance in STS tasks. Improving this problem remains a challenging task.
[0003] Contrastive learning is a widely used unsupervised learning method in Natural Language Processing (NLP). It constructs a large number of positive and negative sample pairs using various data augmentation techniques, effectively utilizing unlabeled data and mitigating the bias issues in vector generation by models like BERT and RoBERTa. The core idea of contrastive learning is to construct positive and negative sample pairs using data augmentation techniques and then use a contrastive loss function to maximize the similarity between positive sample pair representations while minimizing the similarity between negative sample pair representations. However, while constructing positive and negative samples using different data augmentation strategies increases data diversity, it can also introduce unnecessary noise, leading to unstable or inaccurate learned representations. In terms of generalization ability, contrastive learning performs well on large-scale unlabeled datasets but performs poorly on small-scale datasets. After constructing positive and negative pairs, contrastive learning learns feature representations by pushing apart different samples (negative pairs) or bringing similar samples (positive pairs) closer together. Traditional contrastive learning methods often rely heavily on the selection of negative samples. However, optimizing the selection strategy to pick "harder" samples can improve learning performance. This approach may lead to insufficient data utilization on small datasets and poor generalization of the learned representations.
[0004] Therefore, an unsupervised Chinese contrastive learning method based on cue learning and mutual information is invented to narrow the gap between the pre-trained model and the training target by introducing cue learning as a data augmentation method; to remove noise introduced by data augmentation at the representation level by using a cue template denoising method; and to enhance the similarity of the distribution of attention values in the opposite augmented views by maximizing mutual information on the attention tensor, thereby maximizing the alignment of the augmented view structure and better narrowing the distance between opposite views. Summary of the Invention
[0005] In view of the problems existing in the above and / or existing unsupervised Chinese contrastive learning methods based on prompting and mutual information, this invention is proposed.
[0006] Therefore, the purpose of this invention is to provide an unsupervised Chinese contrastive learning method based on cue learning and mutual information, which can solve the aforementioned problems.
[0007] To address the aforementioned technical problems, according to one aspect of the present invention, the present invention provides the following technical solution:
[0008] An unsupervised Chinese contrastive learning method based on cue learning and mutual information includes the following specific steps:
[0009] S1: Construct positive and negative sample pairs for contrastive learning by embedding the same sentence into two different prompt templates;
[0010] S2: Construct template sentences for noise reduction based on sentence length;
[0011] S3: The RoBERTa pre-trained model is used to encode the constructed sample pairs and the corresponding denoised template sentences, and the sentence embedding is obtained by manually designing prompts. Finally, the representation of the sample, the representation of the denoised template sentence, and the attention tensor of the positive sample pair are output.
[0012] S4: Add denoising processing: Subtract the representation of the corresponding denoised template sentence from the representation of the sample to remove the noise introduced by using the prompt template at the representation level, and use the denoised representation for optimization of the joint loss function;
[0013] S5: Perform four steps—slicing, sampling, logarithmicizing, and aggregating—on the attention tensor of positive sample pairs to calculate mutual information;
[0014] S6: Optimize the model using a joint loss function to obtain better sentence embeddings.
[0015] As a preferred embodiment of the unsupervised Chinese contrastive learning method based on cue learning and mutual information described in this invention, in step S1, a data augmentation method based on hand-designed cueing is used to construct the positive and negative samples required for contrastive learning. The specific steps are as follows:
[0016] S11: Manually design two different prompt templates, a and b, with the two prompt templates being "[X], which means [MASK]" and "[X], which means [MASK]" respectively;
[0017] S12: Take a sentence and replace the [X] replacement symbol in the prompt template to form two samples. This allows the same sentence to be represented as different views using different prompt templates. The two samples generated by embedding the same sentence into two different prompt templates are considered as positive sample pairs, while the samples generated by embedding different sentences into prompt templates in the same batch are considered as negative sample pairs.
[0018] As a preferred embodiment of the unsupervised Chinese contrastive learning method based on cue learning and mutual information described in this invention, the specific steps of S2 are as follows: Taking the template "[X], which means [MASK]" as an example, assuming there is a sentence s with a length of 5, the [X] placeholder in the template is copied 5 times to obtain the denoised sentence "[X][X][X][X][X], which means [MASK]" of template a, so as to ensure that the position ID of the denoised template sentence does not deviate from the template position ID of the cue enhancement sample during encoding.
[0019] As a preferred embodiment of the unsupervised Chinese contrastive learning method based on prompting and mutual information described in this invention, step S3 uses a method based on hand-designed prompts to obtain sentence embeddings and encodes them using a pre-trained model RoBERTa-base. The specific steps are as follows.
[0020] S31: Input the cue enhancement samples constructed in S1 into the RoBERTa pre-trained model for encoding;
[0021] S32: After encoding, the hidden vector of the [MASK] tag in the cue enhancement sample is used as the representation of the entire sentence.
[0022] As a preferred embodiment of the unsupervised Chinese contrastive learning method based on prompting and mutual information described in this invention, the RoBERTa-base learns language representations through a bidirectional Transformer architecture, including a 12-layer Transformer encoder with 12 self-attention heads per layer, totaling 125M parameters. The method of obtaining sentence embeddings based on hand-designed prompts reduces the gap between the pre-trained model and the training objective by reformulating the sentence embedding task into a masked language task, enabling more effective use of the original BERT or RoBERTa layers. Its task differs from text classification and question answering tasks; the output in sentence embedding is not the label predicted by the MLM classification head, but a vector representing the sentence.
[0023] As a preferred embodiment of the unsupervised Chinese contrastive learning method based on cue learning and mutual information described in this invention, wherein: the sample representation output in S3 is denoised in step S4, and the specific steps are as follows:
[0024] S41: Take the denoised template sentence generated in S2 and obtain its representation in the same way as the sentence embedding method in S3;
[0025] S42: Subtract the representation of the denoised template sentence from the representation of the cue-enhanced sample to eliminate the bias introduced by using the cue template at the representation level.
[0026] As a preferred embodiment of the unsupervised Chinese contrastive learning method based on cue learning and mutual information described in this invention, the specific steps of S5 are as follows:
[0027] S51: Tiling the attention tensor. After tiling, the dimension of the attention tensor will become (L×H, n, n). After tiling, the attention tensor is divided into R blocks. The final dimension of the attention tensor is (R, n, n).
[0028] S52: Taking an augmented sample s as an example, first calculate the number m (1≤m≤n) of non-PAD-padded tokens in the augmented sample s. Then, sample from the R blocks of s, following the following sampling rule: take m from the attention value pool of each block. 2 Each attention value participates in the sampling, in m 2 The probability of each attention value being sampled is 1 / m. 2 The rest of n 2 -m 2The sampling probability of each attention value is 0. After sampling, the index set of each block sampling is obtained. When sampling the positive pair of the enhanced sample s, the above steps do not need to be repeated. The index set of s is used directly for sampling to ensure that the sampling positions of the same block of the positive sample pair are the same, which is beneficial to the alignment of the positive sample pair view structure.
[0029] S53: The process of calculating logarithmic mutual information between blocks is as follows: First, the attention of the blocks is logarithmically converted, and the formula is:
[0030] t = log(w)
[0031] Then, the mutual information of the logarithmically transformed attention tensor is calculated using the following formula:
[0032]
[0033] Where w is the attention tensor, ρ is the correlation coefficient calculated based on t1 and t2, the log function converts the normal random variable into a log-normal random variable, Cov is the covariance, and σ is the variance of the log-normal distribution.
[0034] S54: Aggregate the mutual information of the blocks and use it to optimize the joint loss. The aggregation formula is as follows:
[0035]
[0036] Where n equals the number of sentences in a batch, R equals the number of attention slices, and I... i，j The mutual information between attention slices j of the positive sample pairs generated by sentence i.
[0037] As a preferred embodiment of the unsupervised Chinese contrastive learning method based on cue learning and mutual information described in this invention, wherein: the joint loss in S6 consists of two parts, the first part being the optimization of the sentence representation, as shown in the following formula:
[0038]
[0039] Where, r a -r pa It is a sample representation enhanced with template cueing (a) after denoising. b -r pb It is a sample representation enhanced with b template hints after denoising. The value of n is equal to the number of sentences in a batch. τ is the temperature parameter. The numerator represents the distance between positive sample pairs, and the denominator represents the distance between all pairs of samples in a batch.
[0040] The final joint loss function is:
[0041] L=αLC +λL M
[0042] Where α and λ are adjustable hyperparameters.
[0043] Compared with existing technologies:
[0044] First, by introducing cue learning as a data augmentation method and then using a method based on hand-designed cues to obtain sentence embeddings, the sentence embedding task is reformulated as a masked language task. This narrows the gap between the pre-trained model and the training objective, allowing for more effective use of the original BERT or RoBERTa layers and improving model performance. Second, by using cue template denoising, noise introduced by data augmentation is removed at the representation level, reducing its impact on the model and effectively mitigating the bias problem in vector generation by models such as BERT and RoBERTa, thus improving the model's robustness. Finally, by maximizing mutual information on the attention tensors of opposite views, the similarity of attention value distributions in opposite augmented views is enhanced, thereby maximizing the alignment of augmented view structures, better bridging the distance between opposite views, and better learning of sentence feature representations. Attached Figure Description
[0045] Figure 1 This is a schematic diagram of the process of the present invention;
[0046] Figure 2 This is a schematic diagram of the overall model framework of the present invention;
[0047] Figure 3 This invention provides a schematic diagram illustrating the construction of enhanced samples and the acquisition of sentence embeddings.
[0048] Figure 4 This is a schematic diagram of the sentence construction for the noise reduction template of the present invention;
[0049] Figure 5 This is a schematic diagram of the sliced attention tensor of the present invention. Detailed Implementation
[0050] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be described in further detail below with reference to the accompanying drawings.
[0051] This invention provides an unsupervised Chinese contrastive learning method based on prompting and mutual information. Please refer to [link / reference]. Figures 1-5 The specific steps are as follows:
[0052] S1: Use a data augmentation method based on hand-designed prompts to construct the positive and negative samples needed for contrastive learning. The specific process is as follows: Figure 3 ( Figure 3The process is illustrated using template a as an example. First, we manually designed two different prompt templates, a and b, which are "[X] means [MASK]." and "[X] means [MASK]." respectively. Next, we replaced the [X] symbol in each template with a sentence, creating two samples. The idea behind constructing these samples is to use different templates to represent the same sentence as different views. The two samples generated by embedding the same sentence into two different templates are considered positive sample pairs, while samples generated by embedding different sentences into templates within the same batch are considered negative sample pairs.
[0053] S2: Generate template sentences for noise reduction based on the sentence length. The specific process is as follows: Figure 4 As shown, taking the template "[X], which means [MASK]." as an example, assuming there is a sentence s with a length of 5, the [X] placeholder in the template is copied 5 times to obtain the denoised sentence "[X][X][X][X][X], which means [MASK]." of template a. The reason for this operation is to ensure that the position ID of the denoised template sentence does not deviate from the position ID of the cue enhancement sample template during encoding.
[0054] S3: Use a method based on hand-designed prompts to obtain sentence embeddings and encode them using the pre-trained model RoBERTa-base. The specific method for obtaining sentence embeddings based on hand-designed prompts is as follows: Figure 3 As shown, by reformulating the sentence embedding task as a masked language task, the gap between the pre-trained model and the training objective is narrowed, allowing for more effective use of the original BERT or RoBERTa layers. This task differs from text classification and question answering tasks; the output of sentence embedding is not the label predicted by the MLM classification head, but rather a vector representing the sentence. The specific process is as follows: the cue enhancement samples constructed in S1 are input into the RoBERTa pre-trained model for encoding. After encoding, the hidden vector of the [MASK] label in the cue enhancement samples is used as the representation of the entire sentence.
[0055] S4: Denoise the sample representation output in S3. The specific process is as follows: First, the denoised template sentence generated in S2 is processed using the same method as obtaining the sentence embedding in S3 to obtain the representation of the denoised template sentence. Then, the representation of the denoised template sentence is subtracted from the representation of the cue-enhanced sample to eliminate the bias caused by using the cue template at the representation level.
[0056] S5: The mutual information is calculated by performing four steps: slicing, sampling, logarithmicizing, and aggregating the attention tensor of positive sample pairs.
[0057] S51: Since the positive samples are generated from the original sentence through cue enhancement, the impact of this enhancement on the attention distribution depends on the depth (layer) and position (head) in the Transformer stack. Therefore, we choose to slice the attention tensor. Slicing the attention tensor has several advantages. On the one hand, it preserves the locality of distribution variations. On the other hand, limiting the spatial range of attention blocks allows us to use simpler distribution models, such as bivariate distributions, rather than complex mixed distribution models over the entire stack. The specific process is as follows: the slicing operation of the attention tensor is as follows... Figure 5 As shown, since the RoBERTa-base model uses a multi-head attention mechanism (the output attention tensor has dimensions of (L, H, n, n), where L represents the number of layers in the Transformer stack and H represents the number of heads in the multi-head attention mechanism), this is not conducive to the slicing operation. Therefore, the attention tensor needs to be flattened first. After the flattening operation, the dimension of the attention tensor becomes (L×H, n, n). After the flattening is completed, the attention tensor is divided into R blocks, and the final dimension of the attention tensor is (R, n, n).
[0058] S52: Since different sentences in a batch typically have different lengths, they need to be padded to achieve length consistency. This is usually done by adding [PAD] markers to the end of the sequence, making all sequences of equal length for efficient batch computation on the GPU. However, when analyzing the statistical relationships of attention, attention values caused by [PAD] markers must be discarded to ensure the accuracy of the results. Therefore, to accommodate different lengths of the marked sequences, a sampling step needs to be performed on the attention tensor after each slice. The specific sampling steps are as follows: Sampling is performed on the blocks in S51. For ease of understanding, an augmented sample s is used as an example. First, the number m (1≤m≤n) of non-[PAD] padded tokens in the augmented sample s is calculated. Then, sampling is performed on the R blocks of s respectively, with the following sampling rule: m is taken from the attention value pool of each block. 2 Each attention value participates in the sampling, in m 2 The probability of each attention value being sampled is 1 / m. 2 The rest of n 2 -m 2 The sampling probability of each attention value is 0. After sampling, the index set of each block sampling is obtained. Note that when sampling the positive pair of the enhanced sample s, it is not necessary to repeat the above steps. The index set of s can be used directly for sampling. This can ensure that the sampling position of the same block of the positive sample pair is the same, which is beneficial to the alignment of the positive sample pair view structure.
[0059] S53: Because directly calculating the mutual information of the attention tensor is too complex, the attention tensor can be treated as a log-normal distribution for calculation. This allows the mutual information to be reformulated as a correlation function, thus enabling implementation on a native GPU. The process of calculating the log-mutual information between blocks is as follows: First, the attention of each block is logarithmically converted, as shown in the formula:
[0060] t = log(w)
[0061] Then, the mutual information of the logarithmically transformed attention tensor is calculated using the following formula:
[0062]
[0063] Where w is the attention tensor, ρ is the correlation coefficient calculated based on t1 and t2, the log function converts the normal random variable into a log-normal random variable, Cov is the covariance, and σ is the variance of the log-normal distribution.
[0064] S54: Aggregate the mutual information of the blocks and use it to optimize the joint loss. The aggregation formula is as follows:
[0065]
[0066] Where n equals the number of sentences in a batch, R equals the number of attention slices, and I... i，j The mutual information between attention slices j of the positive sample pairs generated by sentence i.
[0067] S6: Optimize the model using a joint loss function. The joint loss consists of two parts. The first part is the optimization of the sentence representation, as shown in the following formula:
[0068]
[0069] Where, r a -r pa It is a sample representation enhanced with template cueing (a) after denoising. b -r pb This is a sample representation enhanced with b-template hints after denoising. The value of n equals the number of sentences in a batch, and τ is the temperature parameter. The numerator represents the distance between positive sample pairs, and the denominator represents the pairwise distance between all samples in a batch.
[0070] The final joint loss function is:
[0071] L=αL C +λL M
[0072] Where α and λ are adjustable hyperparameters.
[0073] Although the present invention has been described above with reference to embodiments, various modifications can be made and components can be replaced with equivalents without departing from the scope of the invention. In particular, as long as there is no structural conflict, the features in the disclosed embodiments can be combined with each other in any manner. The lack of an exhaustive description of these combinations in this specification is merely for the sake of brevity and resource conservation. Therefore, the present invention is not limited to the specific embodiments disclosed herein, but includes all technical solutions falling within the scope of the claims.
Claims
1. An unsupervised Chinese contrastive learning method based on cue-based learning and mutual information, characterized in that, The specific steps are as follows: S1: Construct positive and negative sample pairs for contrastive learning by embedding the same sentence into two different prompt templates; S2: Construct template sentences for noise reduction based on sentence length; S3: The RoBERTa pre-trained model is used to encode the constructed sample pairs and the corresponding denoised template sentences, and the sentence embedding is obtained by manually designing prompts. Finally, the representation of the sample, the representation of the denoised template sentence, and the attention tensor of the positive sample pair are output. S4: Add denoising processing: Subtract the representation of the corresponding denoised template sentence from the representation of the sample to remove the noise introduced by using the prompt template at the representation level, and use the denoised representation for optimization of the joint loss function; S5: Perform four steps—slicing, sampling, logarithmicizing, and aggregating—on the attention tensor of positive sample pairs to calculate mutual information; S6: Optimize the model using a joint loss function to obtain better sentence embeddings.
2. The unsupervised Chinese contrastive learning method based on cue learning and mutual information according to claim 1, characterized in that, In step S1, a data augmentation method based on hand-designed prompts is used to construct the positive and negative samples required for contrastive learning. The specific steps are as follows: S11: Manually design two different prompt templates, a and b, with the two prompt templates being "[X], which means [MASK]" and "[X], which means [MASK]" respectively; S12: Take a sentence and replace the [X] replacement symbol in the prompt template to form two samples. This allows the same sentence to be represented as different views using different prompt templates. The two samples generated by embedding the same sentence into two different prompt templates are considered as positive sample pairs, while the samples generated by embedding different sentences into prompt templates in the same batch are considered as negative sample pairs.
3. The unsupervised Chinese contrastive learning method based on cue learning and mutual information according to claim 1, characterized in that, The specific steps of S2 are as follows: Taking the template "[X], which means [MASK]" as an example, assuming there is a sentence s with a length of 5, the [X] placeholder in the template is copied 5 times to obtain the denoised sentence "[X][X][X][X][X], which means [MASK]" of template a, so as to ensure that the position ID of the denoised template sentence does not deviate from the template position ID of the cue enhancement sample during encoding.
4. The unsupervised Chinese contrastive learning method based on cue learning and mutual information according to claim 1, characterized in that, In step S3, a method based on hand-designed prompts is used to obtain sentence embeddings and they are encoded using a pre-trained model RoBERTa-base. The specific steps are as follows: S31: Input the cue enhancement samples constructed in S1 into the RoBERTa pre-trained model for encoding; S32: After encoding, the hidden vector of the [MASK] tag in the cue enhancement sample is used as the representation of the entire sentence.
5. The unsupervised Chinese contrastive learning method based on cue learning and mutual information according to claim 4, characterized in that, The RoBERTa-base learns language representations through a bidirectional Transformer architecture, including a 12-layer Transformer encoder with 12 self-attention heads per layer, totaling 125M parameters. The method of obtaining sentence embeddings based on hand-designed prompts reduces the gap between the pre-trained model and the training objective by reformulating the sentence embedding task into a masked language task. It can more effectively use the original BERT or RoBERTa layers. Its task differs from text classification and question answering tasks. The output of the sentence embedding is not the label predicted by the MLM classification head, but a vector representing the sentence.
6. The unsupervised Chinese contrastive learning method based on cue learning and mutual information according to claim 1, characterized in that, In step S4, the sample representation output from S3 is denoised. The specific steps are as follows: S41: Take the denoised template sentence generated in S2 and obtain its representation in the same way as the sentence embedding method in S3; S42: Subtract the representation of the denoised template sentence from the representation of the cue-enhanced sample to eliminate the bias introduced by using the cue template at the representation level.
7. The unsupervised Chinese contrastive learning method based on cue learning and mutual information according to claim 1, characterized in that, The specific steps of S5 are as follows: S51: Tiling the attention tensor. After tiling, the dimension of the attention tensor will become (L×H, n, n). After tiling, the attention tensor is divided into R blocks. The final dimension of the attention tensor is (R, n, n). S52: Taking an augmented sample s as an example, first calculate the number m (1≤m≤n) of non-PAD-padded tokens in the augmented sample s. Then, sample from the R blocks of s, following the following sampling rule: take m from the attention value pool of each block. 2 Each attention value participates in the sampling, in m 2 The probability of each attention value being sampled is 1 / m. 2 The rest of n 2 -m 2 The sampling probability of each attention value is 0. After sampling, the index set of each block sampling is obtained. When sampling the positive pair of the enhanced sample s, the above steps do not need to be repeated. The index set of s is used directly for sampling to ensure that the sampling positions of the same block of the positive sample pair are the same, which is beneficial to the alignment of the positive sample pair view structure. S53: The process of calculating logarithmic mutual information between blocks is as follows: First, the attention of the blocks is logarithmically converted, and the formula is: t = log(w) Then, the mutual information of the logarithmically transformed attention tensor is calculated using the following formula: Where w is the attention tensor, ρ is the correlation coefficient calculated based on t1 and t2, the log function converts the normal random variable into a log-normal random variable, Cov is the covariance, and σ is the variance of the log-normal distribution. S54: Aggregate the mutual information of the blocks and use it to optimize the joint loss. The aggregation formula is as follows: Where n equals the number of sentences in a batch, R equals the number of attention slices, and I... i，j The mutual information between attention slices j of the positive sample pairs generated by sentence i.
8. The unsupervised Chinese contrastive learning method based on cue learning and mutual information according to claim 1, characterized in that, The joint loss in S6 consists of two parts. The first part is an optimization of the sentence representation, as shown in the following formula: Where, r a -r pa It is a sample representation enhanced with template cueing (a) after denoising. b -r pb It is a sample representation enhanced with b template hints after denoising. The value of n is equal to the number of sentences in a batch. τ is the temperature parameter. The numerator represents the distance between positive sample pairs, and the denominator represents the distance between all pairs of samples in a batch. The final joint loss function is: L=αL C +λL M Where α and λ are adjustable hyperparameters.