An unsupervised sentence representation learning method for removing negative samples

Abstract

This invention discloses an unsupervised sentence representation learning method for negative sample removal. Its key feature is the use of a prediction task to filter redundant information in the pre-training dataset, thereby improving the performance of the SiamCSE model. Randomly selected words in a sentence are repeated and then fed into the encoder to obtain positive samples of varying lengths without losing semantic information. This achieves sentence representation learning by removing negative samples. Specifically, the method includes: setting the mapping layer of a multilayer perceptron, constraining the similarity of intermediate hidden vectors with an objective function, removing redundant information in the prediction task, generating cross-correlation and decorrelation matrices, and training the model. Compared with existing technologies, this invention demonstrates superior performance in sentence representation learning tasks, fills the gaps in current sentence representation models, improves the evaluation scheme for negative sample removal sentence representation learning models, and helps promote the widespread application of negative sample removal sentence representation learning models in real-world scenarios.

Description

Technical Field

[0001] This invention relates to the field of unsupervised sentence representation learning technology based on contrastive learning, and in particular to an unsupervised sentence representation learning method that removes negative samples. Background Technology

[0002] In recent years, contrastive learning has shone brightly in the field of representation learning due to its easy-to-understand approach, relatively wide range of applications, and surprisingly excellent results. These research efforts have gradually expanded from initial visual representation learning to later text representation learning. A review of these works reveals that the vast majority achieve the desired results by using both positive and negative samples for training. However, in reality, the use of negative samples can bring a series of problems. For example, the SimCSE model coarsely treats other sentences in the same training mini-batch as negative samples, which can easily lead to the problem of false negative samples (i.e., negative samples with high semantic similarity to the input samples).

[0003] From another perspective, to achieve ideal results, contrastive learning methods typically require a certain number of negative samples to compare with positive samples, and generally, the more negative samples, the better. However, increasing the number of negative samples also increases the computational cost of training typical contrastive learning models, resulting in larger model sizes that are difficult to implement on smaller devices such as mobile phones. The use of negative samples can introduce the spurious negative sample problem, significantly impacting model accuracy. Furthermore, negative samples make the model more bloated and complex, increasing computational costs and ultimately hindering the practical application of the research technique.

[0004] Therefore, researching how to improve the performance of sentence representation learning models without using negative samples, to the point where it reaches or even surpasses the performance of models using negative samples, is a highly noteworthy issue. Currently, there are relatively few sentence representation learning models that do not use negative samples, and a complete system has not yet been formed. Specifically, a few studies are independent in content, briefly exploring model architecture and data augmentation methods separately, but failing to connect the two; in terms of effectiveness, the performance of the models proposed in these studies not only differs significantly from the performance of current models using negative samples, but also lacks a fair, effective, and intuitive comparison between different models.

[0005] Current sentence representation learning models that remove negative samples generally ignore redundant information in pre-training corpora, resulting in poor model performance and significantly limiting their widespread application. Summary of the Invention

[0006] The purpose of this invention is to provide an unsupervised sentence representation learning method for removing negative samples, addressing the shortcomings of existing technologies. This method employs design from the perspectives of pre-training corpus denoising, model architecture, and positive sample construction to achieve sentence representation learning without negative samples. Pre-training corpus denoising uses a prediction task to filter redundant information in the pre-training dataset. Positive sample construction involves randomly selecting words from a sentence, repeating them, and then feeding them into the encoder. This results in positive samples that do not lose semantic information and have varying lengths, preventing the model from using length as a feature of positive samples. Furthermore, quantitative analysis of key components in the model is performed, improving the evaluation scheme of the negative sample removal sentence representation learning model and significantly enhancing its performance. The method is simple, effective, and effectively solves the problem of negative sample removal sentence representation learning models generally ignoring redundant information in the pre-training corpus, greatly contributing to improved model performance. The model's performance is not only comparable to current models using negative samples but also helps promote the universality of negative sample removal sentence representation learning models in real-world applications, demonstrating promising application prospects.

[0007] The objective of this invention is achieved as follows: an unsupervised sentence representation learning method for removing negative samples, characterized by employing the SiamCSE unsupervised sentence representation learning model for removing negative samples. This method filters redundant information from the pre-training dataset to improve model performance. It repeats randomly selected words from a sentence and then feeds them into the encoder. The resulting positive samples do not lose semantic information and are also of varying lengths, thus preventing the model from using length as a feature of positive samples. Quantitative analysis is performed on important components of the model, and a model evaluation scheme is integrated.

[0008] The main contents of the SiamCSE model include the following:

[0009] (1) Set the multilayer perceptron as the mapping layer.

[0010] Adding a multilayer perceptron to the encoder output helps improve the quality of the input representation. Inspired by this, the SiamCSE model uses different temporary backoff values ​​p for the same input x. large and p small The input is to the encoder, and the output of the encoder is mapped by the multilayer perceptron to obtain the intermediate hidden vectors z1 and z2 of the upper and lower branches.

[0011] (2) Using the objective function L C To constrain the similarity between the hidden vectors of the upper and lower branches

[0012] SiamCSE model design objective function L CTo constrain the similarity between the hidden vectors z1 and z2 of the upper and lower branches, the negative cosine similarity between z1 and z2 is defined by the following equation (b):

[0013]

[0014] Where ||·||2 represents the L2 norm; D is the negative cosine similarity between z1 and z2.

[0015] The objective function L C Defined by the following equation (a):

[0016]

[0017] The objective function L C It is calculated for each input, and the loss value of a mini-batch is the average of all input loss values.

[0018] (3) Removing redundant information from the hidden vector prediction task based on upper and lower branches

[0019] While existing works such as BSL and SCD have investigated negative sample removal sentence representation learning, they lack consideration for the redundant information contained in the pre-training data itself. This redundant information is detrimental to the performance of sentence representations on downstream tasks. Therefore, this invention proposes a hidden vector prediction task based on upper and lower branches to filter out redundant information.

[0020] The SiamCSE model uses the intermediate hidden vector z1 of the upper branch to predict the intermediate hidden vector z2 of the lower branch. This step is expressed by the optimization objective function L in equation (c) below. P To achieve:

[0021]

[0022] (4) Generation of cross-correlation matrix based on mapping function

[0023] To prevent the learned sentence representation from collapsing, the SiamCSE model maps the output vector of the multilayer perceptron to a higher-dimensional space P using the following equation (d):

[0024] p = H(z) (d).

[0025] Where p represents the vector obtained after mapping z to space P; H is the mapping function.

[0026] The mapping function used in this invention consists of three linear layers, each containing 4096 neurons, with ReLU as the activation function, and BatchNorm is used between each layer. In space P, the SiamCSE model expresses the hidden representations of the left and right branches as a cross-correlation matrix, and uses element C... ij The calculation is expressed by the following equation (e):

[0027]

[0028] Where b is the index number of the sample; i and j are different dimensions of the neural network output vector; C is a square matrix with element values ​​between -1 and 1.

[0029] (5) Generate the decorrelation matrix

[0030] The relevance matrix is ​​optimized using an optimization strategy of "being as similar as possible in the same dimension and as different as possible in different dimensions" to remove noise and redundant information contained in the input representation, ensuring that the learned sentence representation does not collapse.

[0031] This invention uses the objective function L of the optimized cross-correlation matrix, expressed by the following equation (f). D :

[0032]

[0033] The optimized matrix is ​​called the decorrelation matrix. The diagonal elements of this matrix are close to 1, while the off-diagonal elements are close to zero.

[0034] (6) Based on objective function L total Training the SiamCSE model

[0035] The above L C L P and L D The three objective functions are combined to form the overall objective function L. total Based on the objective function L total The SiamCSE model is trained to obtain effective sentence representations, achieving sentence representation learning with negative samples removed. The overall training objective function L... total It can be expressed by the following formula (g):

[0036] L total =L C +αL P +βL D (g)

[0037] Here, α and β are hyperparameters.

[0038] Compared with existing technologies, this invention is simple in method and greatly improves the performance of the model. It effectively solves the problem that sentence representation learning models for removing negative samples generally ignore redundant information in pre-training corpora. Based on the shortcomings of existing sentence representation learning models and the difficulty in implementing them in small-scale devices, this invention breaks the awkward situation of existing research on sentence representation learning for removing negative samples being independent and incomplete. It makes a more detailed analysis from various perspectives such as redundant information, model architecture, and data augmentation, providing a reference for the research on sentence representation learning models for removing negative samples, helping to promote the universality of sentence representation learning models for removing negative samples in reality, and has good application prospects. Attached Figure Description

[0039] Figure 1 This is a schematic diagram of the SiamCSE model structure. Detailed Implementation

[0040] This invention enables the SiamCS model to learn high-quality sentence representations without using negative samples. The model employs methodological design from the perspectives of pre-training corpus denoising, model architecture improvement, and positive sample construction to achieve sentence representation learning without negative samples. The innovations of this invention are: 1) Addressing the current situation where sentence representation learning models generally ignore redundant information in the pre-training corpus, this invention uses a prediction task to filter redundant information in the pre-training dataset, thereby improving model performance; 2) Addressing the shortcoming of current positive sample construction methods that tend to treat sentences of the same length as a pair of positive samples, this invention repeats randomly selected words from the sentence before feeding them into the encoder. This results in positive samples that do not lose semantic information and are of varying lengths, preventing the model from using length as a feature of positive samples; 3) Quantitative analysis of key components in the model and integration of model evaluation schemes.

[0041] See Figure 1 The specific implementation steps of this invention are as follows:

[0042] (1) Set the multilayer perceptron as the mapping layer.

[0043] The same input x is given with different timeout values ​​p. large and p small The input is to the encoder, and the output of the encoder is mapped by the multilayer perceptron to obtain the intermediate hidden vectors z1 and z2 of the upper and lower branches.

[0044] (2) Using the objective function L C To constrain the similarity between the hidden vectors of the upper and lower branches

[0045] Design the objective function L CTo constrain the similarity between the hidden vectors z1 and z2 of the upper and lower branches, the negative cosine similarity between z1 and z2 is defined by the following equation (b):

[0046]

[0047] Where ||·||2 represents the L2 norm; D is the negative cosine similarity between z1 and z2.

[0048] The objective function L C Defined by the following equation (a):

[0049]

[0050] The objective function L C It is calculated for each input, and the loss value of a mini-batch is the average of all input loss values.

[0051] (3) Removing redundant information in hidden vector prediction task based on upper and lower branches

[0052] The intermediate hidden vector z1 of the upper branch is used to predict the intermediate hidden vector z2 of the lower branch. The optimization objective function used is L. P Defined by the following equation (c):

[0053]

[0054] (4) Generation of cross-correlation matrix based on mapping function

[0055] To prevent the learned sentence representation from collapsing, the SiamCSE model maps the output vector of the multilayer perceptron to a higher-dimensional space P using the following equation (d):

[0056] p = H(z) (d).

[0057] Where p represents the vector obtained after mapping z to space P; H is the mapping function.

[0058] The mapping function used in this invention consists of three linear layers, each containing 4096 neurons, with ReLU as the activation function, and BatchNorm is used between each layer. In space P, the model expresses the hidden representations of the left and right branches as a cross-correlation matrix, using elements C. ij The calculation is expressed by the following equation (e):

[0059]

[0060] Where b is the index number of the sample; i and j are different dimensions of the neural network output vector; C is a square matrix with element values ​​between -1 and 1.

[0061] (5) Generate the decorrelation matrix

[0062] The relevance matrix is ​​optimized using an optimization strategy of "being as similar as possible in the same dimension and as different as possible in different dimensions" to remove noise and redundant information contained in the input representation, ensuring that the learned sentence representation does not collapse.

[0063] This invention uses the objective function L of the optimized cross-correlation matrix, expressed by the following equation (f). D :

[0064]

[0065] The optimized matrix is ​​called the decorrelation matrix. The diagonal elements of this matrix are close to 1, while the off-diagonal elements are close to zero.

[0066] (6) Based on objective function L total Model training

[0067] The above L C L P and L D The three objective functions are combined to form the overall objective function L. total Based on the objective function L total The SiamCSE model is trained to obtain effective sentence representations, achieving sentence representation learning with negative samples removed. The overall training objective function L... total It can be expressed by the following formula (g):

[0068] L total =L C +αL P +βL D (g)

[0069] Here, α and β are hyperparameters.

[0070] Experimental analysis shows that the method designed in this invention exhibits excellent performance in sentence representation learning tasks, filling the gaps in current sentence representation models and improving the evaluation scheme for negative sample removal sentence representation learning models. This invention can help promote the widespread application of negative sample removal sentence representation learning models in real-world scenarios. The above embodiments are merely illustrative of this invention and are not intended to limit the scope of this patent. All equivalent implementations of this invention should be included within the scope of the claims of this patent.

Claims

1. An unsupervised sentence representation learning method for removing negative samples, including the SiamCSE model, characterized in that, To improve the performance of the SiamCSE model, a method is adopted to filter redundant information in the pre-training dataset using a prediction task. Randomly selected words in a sentence are repeated and then fed into the encoder to obtain positive samples of varying lengths without losing semantic information. This achieves sentence representation learning by removing negative samples. The specific steps include: (1) Set up a mapping layer based on a multilayer perceptron The same input x is given with different timeout values ​​p. large and p small The input encoder is mapped by a multilayer perceptron to obtain the intermediate hidden vectors z1 and z2 of the upper and lower branches; (2) Using the objective function L C Constrain the similarity between the intermediate hidden vectors z1 and z2 The objective function L is defined using the following equation (a). C Constrain the similarity between the intermediate hidden vectors z1 and z2: Where D is the negative cosine similarity as defined in equation (b) below: Where ||·||2 is the L2 norm; (3) Removing redundant information based on intermediate hidden vectors z1 and z2 in the prediction task The hidden vector z1 is used to predict the intermediate hidden vector z2 of the lower branch, and the objective function L is optimized. P It can be expressed by the following formula (c): (4) Generation of cross-correlation matrix based on mapping function Using SiamCSE, the output vector of the multilayer perceptron is mapped to a higher-dimensional space P by the following equation (d): p = H(z) (d); Where p represents the vector obtained by mapping z to space P; H is the mapping function; The mapping function H consists of three linear layers, each containing 4096 neurons, using ReLU as the activation function, and using BatchNorm between each layer; The model on space P represents the intermediate hidden vectors z1 and z2 of the upper and lower branches by the element C of the following equation (e). ij Cross-correlation matrix: Where b is the index number of the sample; i and j are the different dimensions of the neural network output vector; C is a square matrix with element values ​​between -1 and 1; (5) Generate the decorrelation matrix The objective function L is expressed by the following equation (f). D Optimize the cross-correlation matrix to a decorrelation matrix: The diagonal elements of the decorrelation matrix approach 1, while the off-diagonal elements approach zero. (6) Based on objective function L total Model training L C L P and L D The three objective functions are combined to form the overall objective function L. total Train the SiamCSE model to obtain effective sentence representations and achieve sentence representation learning with negative samples removed. The overall objective function L... total It can be expressed by the following formula (g): L total =L C +αL P +βL D (g); Here, α and β are hyperparameters.

Images