An unsupervised sentence learning method based on redundancy information reduction

Abstract

The present application relates to a kind of unsupervised sentence learning method based on redundancy information reduction, belong to natural language processing technical field.The present application focuses on the redundancy information identification and reduction problem in sentence embedding Sentence Embedding technology, aims at optimizing the quality of sentence semantic representation by the way of unsupervised learning.This method is applicable to text classification, information retrieval, semantic similarity calculation and other tasks, and provides an efficient solution for low-resource language and cross-language application scenarios.The technical scheme of the present application combines high-frequency vocabulary analysis, dynamic dimension screening and contrast learning regularization strategy, which can effectively alleviate the over-smoothing problem and redundant information representation problem existing in the sentence embedding process of pre-training model.By constructing, screening and separating redundant information from Token level, this method significantly improves the distinguishing ability of model in semantic feature capture, and provides a new research idea and technical means for sentence representation learning under unsupervised learning framework.

Description

Technical Field

[0001] This invention relates to the field of natural language processing technology, and in particular to an unsupervised sentence learning method based on redundant information reduction. Background Technology

[0002] Currently, efficiently understanding and processing massive amounts of text information has become one of the core challenges in the field of Natural Language Processing (NLP). Sentence embedding, as an important tool for representing semantic information of sentences, plays a crucial role in tasks such as text classification, information retrieval, and semantic similarity calculation.

[0003] Traditional sentence embedding methods typically rely on supervised learning, requiring large amounts of labeled data to train the model and obtain high-quality sentence representations. However, acquiring large-scale labeled data is not only costly, but also, for certain domains or low-resource languages, the scarcity of labeled data becomes a major bottleneck restricting model performance. Therefore, unsupervised sentence embedding techniques have gradually become a research hotspot.

[0004] The greatest advantage of unsupervised sentence embedding lies in its ability to effectively address the problem of data scarcity. Unlike supervised methods, unsupervised sentence embedding relies solely on large-scale monolingual text data, capturing semantic information of sentences through self-supervised learning or pre-trained models such as BERT and GPT. This approach not only reduces reliance on labeled data but also allows for wide applicability across various languages ​​and domains, greatly expanding the applicability of sentence embedding. However, unsupervised sentence embedding technology also faces some pressing technical challenges:

[0005] 1. Oversmoothing problem: In unsupervised sentence embedding, models often map different sentences to similar vector spaces, leading to reduced discriminative power between sentence embeddings. This phenomenon is called "oversmoothing," making it difficult for models to accurately capture semantic differences between sentences.

[0006] 2. Redundant Information Representation Problem: Due to the large number of high-frequency words and common expressions in natural language, these words often exhibit high similarity in the embedding space. For example, ... Figure 1 As shown, two sentences with similar meanings but different wording are selected for analysis:

[0007] Sentence A: She was born and raised in the north, but after years of living in the south, she considers it her second home. Sentence B: She was born in the south but has lived in the north for several years.

[0008] Figure 1 This demonstrates the cosine similarity matrix between the tokens in the embedding space of these two sentences, revealing the existence of token-level redundant information:

[0009] The orange heatmap on the left represents the pairwise cosine similarity between all tokens in sentence A, such as "she", "was", "born", "raised", "north", and "south".

[0010] Of particular note is that the two semantically distinct verbs, “was” and “born”, have a cosine similarity of 0.79 in their embedding vectors, which is located in the second row and third column. This indicates that they are very close in the embedding space, reflecting the phenomenon of “semantic drift” or “redundant representation” of function words / high-frequency words.

[0011] The blue heatmap on the right represents the pairwise cosine similarity between all tokens in sentence B.

[0012] Similarly, it can be seen that the similarity between "was" and "born" is still 0.75, in the second row and third column, further verifying that this phenomenon is not an isolated case, but rather a common occurrence in the pre-trained embedding space.

[0013] also, Figure 1 The 0.86 marked below refers to the cosine similarity between the overall embedding vectors of sentence A and sentence B. This high value indicates that although the sentences have different wording, their semantics are still relatively close; however, at the same time, the high similarity at the token level within the sentences masks local semantic differences, such as "was" and "born," which is a micro-manifestation of "oversmoothing."

[0014] In conclusion, Figure 1 This intuitively reveals that even tokens that differ at the semantic level may exhibit high similarity in the embedding space due to high-frequency co-occurrence or similar grammatical roles, thus forming redundant encodings, exacerbating the over-smoothing problem of sentence embeddings, and affecting the model's ability to capture subtle semantic differences.

[0015] To address these issues, existing technologies attempt to optimize sentence embedding quality by introducing methods such as contrastive learning and regularization strategies. However, these methods typically fail to analyze the sources of redundant information and their impact on sentence embedding at the token level, resulting in limited effectiveness in reducing redundant information. Summary of the Invention

[0016] To address the aforementioned issues, this invention proposes an unsupervised sentence learning method based on redundant information reduction. This method aims to identify, filter, and separate redundant information at the token level, thereby effectively alleviating the over-smoothing problem in sentence embedding, improving the model's discriminative ability in semantic representation, providing a new approach for the development of unsupervised sentence embedding technology, and laying a solid technical foundation for semantic understanding tasks in practical application scenarios.

[0017] The technical solution adopted in this invention is as follows:

[0018] An unsupervised sentence learning method based on redundant information reduction includes the following steps:

[0019] Step 1: Redundancy Representation Construction: Construct global and local redundancy information representations using two methods: high-frequency word pool and batch data.

[0020] Step 2: Redundant Dimension Filtering: Based on the constructed global and local redundancy information representations, static and dynamic methods are used to accurately identify redundant dimensions that contribute little to semantic distinction.

[0021] Step 3: Redundancy Representation Separation: Using the direct subtraction method and the negative example push method, redundant information is effectively removed from the selected redundant dimensions, which alleviates the excessive smoothing in sentence embedding and improves semantic representation capabilities.

[0022] Furthermore, in step 1, the process of constructing a token-level redundant representation based on high-frequency words is as follows:

[0023] First, the unsupervised dataset is segmented using the NLTK word segmentation tool;

[0024] Then, the word frequencies are counted, and the words with the highest frequencies are selected as the redundant representation pool;

[0025] Then, in each training batch, a number of tokens are randomly sampled from the candidate pool and represented by k;

[0026] Finally, the mean of the token embedding vector is calculated as a redundant representation for each batch.

[0027] Constructing a token-level redundant representation based on high-frequency words provides the global perspective in step 1.

[0028] Furthermore, in step 1, the process of constructing the token-level redundancy representation based on the batch data is as follows:

[0029] First, the sentences in each training batch are segmented into words;

[0030] Then, sort them in descending order of word frequency and select the top few tokens;

[0031] Finally, the mean of the token embedding vector is calculated as a redundant representation for each batch.

[0032] The token-level redundancy representation based on the batch data is constructed as the local perspective in step 1.

[0033] Furthermore, the mean of the token embedding vector is calculated using the following formula:

[0034] ;

[0035] In the formula, The redundant representation of this batch is a vector; Indicates the number of tokens selected from the candidate pool or batch data; Indicates the first Embedding vectors of each token; This indicates that all selected items... Element-wise summation of each token embedding vector; This indicates that the sum vector is averaged.

[0036] Furthermore, in step 2, the process of the static redundant dimension filtering method is as follows:

[0037] First, select the tokens with the highest word frequency from the high-frequency word list;

[0038] Then, the standard deviation of the embedding vector of the most frequent token is calculated in each dimension;

[0039] Finally, the threshold was determined based on the experiment. Standard deviation less than the threshold The dimension is marked as a redundant dimension;

[0040] The static redundancy dimension filtering method is the global static filtering in step 2.

[0041] Furthermore, in step 2, the process of the dynamic redundant dimension filtering method is as follows:

[0042] First, the sentences in each training batch are segmented into words, sorted in descending order of word frequency, and the top few tokens are selected.

[0043] Then, the standard deviation of the token embedding vector in each dimension is calculated;

[0044] Finally, the threshold was determined based on the experiment. Standard deviation below the threshold The dimension is marked as a redundant dimension in this batch;

[0045] The dynamic redundancy dimension filtering method is the local dynamic filtering in step 2.

[0046] Furthermore, the standard deviation of the token embedding vector across each dimension is calculated using the following formula:

[0047] ;

[0048] In the formula, Indicates the first Standard deviation of dimension; Indicates the number of selected token embedding vectors; Indicates the first The Token embedding vector of the _th ... Dimensional value; Indicates the first The mean of the dimension; Indicates the first The token embedding vector at the th in the th The squared deviation between the value on the dimension and its mean; Indicates all The token embedding vector at the th in the th Summing the squared deviations over the dimension; This indicates that the total dispersion has been normalized. The standard deviation is obtained by taking the square root of the average dispersion.

[0049] Furthermore, in step 3, the direct subtraction method is performed as follows:

[0050] First, in each training batch, the difference between the embedded representation and the redundant representation of the data within the batch is calculated on the redundant dimension;

[0051] Then, contrastive learning optimization is performed using the corrected embedding representation, with the following loss function:

[0052] ;

[0053] In the formula, Indicates the first The loss value for each sample; Indicates the first The corrected embedding representation of each sample; Indicates the first The corrected positive example embedding representation of each sample; Indicates the first The corrected positive example embedding representation of each sample; A similarity function representing the similarity between two embedded vectors; Indicates temperature parameter; Represents an exponential function; Indicates all Summing the positive example embeddings of each sample; This represents taking the negative logarithm of the normalized similarity score.

[0054] Furthermore, in step 3, the process of using the negative example method is as follows:

[0055] First, redundant representations are added as additional negative examples to the contrastive learning framework;

[0056] Then, during the comparative fine-tuning process, the loss function is optimized to simultaneously narrow the distance to positive examples and widen the distance to negative examples:

[0057] ;

[0058] In the formula, Indicates the first The loss value for each sample; Indicates the first Embedded representation of each sample; Indicates the first Positive example embeddings of each sample; Indicates the first Positive example embeddings of each sample; Indicates the first Negative example embedding representation of each sample; Indicates the number of selected token embedding vectors; Indicates the number of redundant negative examples; This represents a similarity function between two embedded vectors; Indicates temperature parameter; Represents an exponential function; Indicates all The positive and negative example embeddings of each sample are summed. This represents taking the negative logarithm of the normalized similarity score.

[0059] The beneficial effects of this invention are:

[0060] This invention proposes an unsupervised sentence learning method based on redundant information reduction. By identifying, filtering, and removing redundant information at the token level, it significantly improves the quality of sentence embeddings and demonstrates excellent performance in multiple application scenarios. Specifically, this invention has the following beneficial effects:

[0061] 1. Enhanced Discrimination of Sentence Embeddings: By constructing redundant representations and separating redundant information, the excessive similarity between different sentence embeddings is effectively reduced, thus alleviating the over-smoothing problem in sentence embeddings. After removing redundant information, the model can focus more on capturing the core semantic features of sentences, improving the performance of sentence embeddings in semantic similarity tasks.

[0062] 2. Improved performance of unsupervised learning: This method is entirely based on an unsupervised learning framework, relying solely on large-scale monolingual text data, thus reducing dependence on expensive labeled data and making it suitable for low-resource languages ​​and domains. Experimental results show that this method achieves significant performance improvements on various pre-trained models such as BERT and RoBERTa, validating its good versatility and adaptability.

[0063] 3. Accurate Identification and Removal of Redundant Information: Redundant representations are constructed using both high-frequency word pools and intra-batch data, taking into account both global and local redundancy information to ensure comprehensive redundancy identification. Static and dynamic methods are employed to filter redundant dimensions, capturing global redundancy patterns while adapting to variations in different batches of data, thus improving the accuracy of redundant dimension identification. Two regularization strategies—direct subtraction and negative example shifting—remove redundant information from the perspectives of vector space and contrastive learning, respectively, providing diverse solutions.

[0064] 4. Broad Application Prospects: This method can be widely applied to various languages ​​and fields, greatly expanding the applicability of sentence embedding and providing technical support for cross-language communication and knowledge dissemination in a globalized context. The optimized sentence embedding can be directly applied to tasks such as text classification, information retrieval, machine translation, and dialogue systems, providing a more efficient and flexible semantic representation method for various application scenarios in the field of natural language processing.

[0065] 6. Value of Technological Innovation: This invention provides a deep analysis of the sources of redundant information and their impact on sentence embedding at the token level, offering new research ideas for the development of unsupervised sentence embedding technology. In the context of globalization and the rapid development of information technology, the advancements in this method will further promote cross-language and cross-cultural communication and cooperation, providing strong technical support for information sharing and knowledge dissemination globally.

[0066] In summary, this invention, by accurately identifying and effectively removing redundant information at the token level, not only solves the problem of excessive smoothing in sentence embedding but also significantly improves the model's performance in semantic similarity tasks. Its efficiency, versatility, and wide applicability make it of significant academic value and practical application in the field of natural language processing, laying a solid foundation for future research and practice. Attached Figure Description

[0067] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this invention. For those skilled in the art, other drawings can be obtained based on these drawings.

[0068] Figure 1 It is a heatmap of token embedding similarity;

[0069] Figure 2 This is a flowchart of the unsupervised sentence learning method based on redundant information reduction of the present invention. Detailed Implementation

[0070] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0071] Therefore, the following detailed description of the embodiments of the present invention is not intended to limit the scope of the claimed invention, but merely illustrates some embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention. It should be noted that, in the absence of conflict, the embodiments and features and technical solutions in the embodiments of the present invention can be combined with each other. It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0072] To address the issues of oversmoothing and redundant information representation in sentence embedding of pre-trained models, this embodiment provides an unsupervised sentence learning method based on redundant information reduction. This method aims to identify, filter, and separate redundant information at the token level, thereby effectively alleviating the oversmoothing problem in sentence embedding, improving the model's discriminative ability in semantic representation, providing a new approach for the development of unsupervised sentence embedding technology, and laying a solid technical foundation for semantic understanding tasks in practical application scenarios.

[0073] Specifically, such as Figure 2 As shown, this unsupervised sentence learning method based on redundant information reduction includes the following steps:

[0074] Step 1: Redundancy Representation Construction: Global and local redundancy representations are constructed using both high-frequency word pools and batch-specific data. The purpose of redundancy representation construction is to identify and construct token representations that may contain redundant information, providing a foundation for subsequent redundancy dimension filtering and removal.

[0075] Step 1.1, Constructing a token-level redundant representation based on high-frequency words:

[0076] Because natural language contains a large number of high-frequency words, such as function words and stop words, these words often exhibit high similarity in the embedding space and are the main source of redundant information; therefore, a token-level redundant representation based on high-frequency words is constructed, and the implementation process is as follows:

[0077] First, the NLTK word segmentation tool is used to segment unsupervised datasets, such as Wiki.

[0078] Then, the word frequencies are counted, and the 1000 words with the highest frequency are selected as the redundant representation pool.

[0079] Then, in each training batch, the model randomly samples several tokens from the candidate pool and represents them as k;

[0080] Finally, the mean of the token embedding vector is calculated and used as a redundant representation for each batch during training.

[0081] The mean of the token embedding vector is calculated using the following formula:

[0082] ;

[0083] In the formula, The redundancy representation for this batch is a vector, which is obtained by averaging the selected token embedding vectors and is used to capture redundant information in the current batch. This indicates the number of tokens selected from the candidate pool or batch data; for example, in a method based on a high-frequency word pool, It is the number of tokens randomly selected from the high-frequency word pool; in the batch-based data method, It refers to the number of the top few high-frequency tokens selected, such as 100. Indicates the first The embedding vector of each token; after each token is encoded by a pre-trained model such as BERT or RoBERTa, a fixed-dimensional vector representation is generated. That is, the vector. This indicates that all selected items... The embedding vectors of each token are summed element by element; this operation merges the embedding vectors of multiple tokens into a single sum vector. This indicates that the sum vector is averaged; by dividing by This normalizes the sum vector into a mean vector, making the redundant representation more stable and unaffected by the number of tokens.

[0084] This formula constructs a redundant representation that reflects the common characteristics of a set of high-frequency or highly correlated token embedding vectors. This redundant representation can capture the redundancy patterns in high-frequency word pools or batch data, providing basic data support for subsequent redundant dimension filtering and removal.

[0085] Step 1.1 The token-level redundancy representation based on high-frequency words is constructed as a global perspective, which is suitable for capturing general redundancy information in the entire corpus.

[0086] Step 1.2, Constructing a token-level redundant representation based on in-batch data:

[0087] Since data from different training batches may exhibit specific redundancy patterns, analysis requires the integration of local information. Therefore, a token-level redundancy representation based on intra-batch data is constructed, and the implementation process is as follows:

[0088] First, the sentences in each training batch are segmented into words.

[0089] Then, the words are sorted in descending order according to word frequency, and considering the limitations of computing resources, the top 100 tokens are selected.

[0090] Finally, the mean of the token embedding vector is calculated as a redundant representation for each batch.

[0091] The mean of the token embedding vector is calculated using the following formula:

[0092] ;

[0093] Step 1.2 constructs a token-level redundancy representation based on the batch data as a local perspective, which can capture dynamic redundancy information within the training batch.

[0094] In the redundant representation construction of step 1, two methods are used to capture redundant information at both the global and local levels. The high-frequency word pooling method captures general redundant information globally; by statistically analyzing high-frequency words in the unsupervised corpus, it identifies the high similarity of these words in the embedding space, thus quantifying their redundant characteristics. The intra-batch data method captures dynamic redundant information locally; for each training batch of data, high-frequency tokens are selected based on word frequency, and the mean of their embedding vectors is dynamically calculated as the redundant representation to adapt to changes in different batches of data. Through these two methods, the redundant representation construction in step 1 not only provides a quantitative representation of redundant information, laying the foundation for subsequent redundant dimension filtering and removal, but also takes into account both global and local perspectives, comprehensively capturing the characteristics of redundant patterns.

[0095] Step 2: Redundant Dimension Filtering. Based on the constructed global and local redundancy information representations, static and dynamic methods are used to accurately identify redundant dimensions that contribute little to semantic distinction. The purpose of redundant dimension filtering is to identify redundant dimensions from the high-dimensional space of token embeddings and mark them as targets that need to be removed.

[0096] Step 2.1, Static Redundancy Dimension Filtering Method:

[0097] Since the embedding vectors of high-frequency words may have low dispersion in certain dimensions (i.e., small standard deviation), these dimensions contribute little to distinguishing different tokens and can be considered redundant dimensions. Therefore, static redundant dimension filtering is implemented, and the process is as follows:

[0098] First, select the 100 tokens with the highest word frequency from the high-frequency word list.

[0099] Then, the standard deviation of the embedding vector of the most frequent token is calculated in each dimension;

[0100] The standard deviation of the token embedding vector across each dimension is calculated using the following formula:

[0101] ;

[0102] In the formula, Indicates the first The standard deviation of a dimension measures the dispersion of a set of data along a particular dimension. A smaller standard deviation indicates a more concentrated distribution of data along that dimension, potentially contributing less to distinguishing different embedding vectors. This indicates the number of selected token embedding vectors; for example, in a static filtering method, This refers to the number of 100 words selected from the high-frequency word list; in the dynamic filtering method, It represents the number of the top few high-frequency tokens selected in the current batch. Indicates the first The Token embedding vector of the _th ... Dimensionality; Each token, after being encoded by a pre-trained model, generates a fixed-dimensional embedding vector. Is this vector at the th The specific value in the dimension. Indicates the first The mean of the dimension; it is obtained by applying the average of all dimensions. The token embedding vector at the th in the th The average of the values ​​over the dimension is used to describe the 6th dimension. Central tendency of data in the dimension. Indicates the first The token embedding vector at the th in the th The squared deviation of a value on a dimension from its mean; this operation is used to quantify the degree of deviation of a single data point from the mean. Indicates all The token embedding vector at the th in the th The squared deviations over the dimension are summed; this step calculates the summation of all data points in the dimensional dimension. The total degree of dispersion in the dimension. This indicates that the total dispersion has been normalized; by dividing by This transforms the total dispersion into the average dispersion, making the standard deviation calculation independent of the data size. The standard deviation is obtained by taking the square root of the average dispersion. Taking the square root is to restore the squared deviation to the same units as the original data.

[0103] The purpose of this formula is to calculate the standard deviation of the embedding vectors along a certain dimension, thereby evaluating the data distribution characteristics along that dimension. A small standard deviation indicates that the data distribution along that dimension is relatively concentrated, potentially contributing less to distinguishing different embedding vectors, and thus can be considered a redundant dimension. Conversely, a large standard deviation indicates that the data distribution along that dimension is relatively dispersed, potentially playing an important role in semantic distinction, and should be retained. This formula allows for precise identification of which dimensions in the embedding space are redundant, providing a basis for subsequent redundant information removal.

[0104] Finally, the threshold was determined based on the experiment. Standard deviation less than the threshold The dimension is marked as a redundant dimension.

[0105] Step 2.1, the static redundant dimension filtering method is a global static filtering method, which is applicable to the entire model training process.

[0106] Step 2.2, Dynamic Redundancy Dimension Filtering Method:

[0107] Data from different training batches may exhibit different redundancy patterns, therefore it is necessary to dynamically adjust the selection of redundancy dimensions. The implementation process for static redundancy dimension selection is as follows:

[0108] First, the sentences in each training batch are segmented into words, sorted in descending order of word frequency, and the top 100 tokens are selected.

[0109] Then, the standard deviation of the token embedding vector in each dimension is calculated;

[0110] The standard deviation of the token embedding vector across each dimension is calculated using the following formula:

[0111]

[0112] Finally, the threshold was determined based on the experiment. Standard deviation below the threshold The dimension is marked as a redundant dimension for this batch.

[0113] Step 2.2 The dynamic redundancy dimension filtering method is a local dynamic filtering method that can adapt to changes in different batches of data.

[0114] In step 2, redundant dimension screening accurately identifies redundant dimensions using both static and dynamic methods. The static method calculates the standard deviation of high-frequency word embedding vectors across all dimensions based on global data from the entire training process. Dimensions with standard deviations below a set threshold are marked as redundant, thus identifying dimensions that contribute less to distinguishing different embeddings. The dynamic method dynamically calculates the standard deviation of high-frequency token embedding vectors across all dimensions based on local data from each training batch. Similarly, dimensions with standard deviations below a threshold are marked as redundant for that batch, adapting to changes in different batches of data. Through these two methods, step 2's redundant dimension screening provides a clear direction for removing target redundant information. The static method is suitable for identifying global redundancy patterns and has high stability; the dynamic method can flexibly respond to changes in different batches of data, ensuring the accuracy and adaptability of redundant dimension screening.

[0115] Step 3: Redundancy Representation Separation: Using direct subtraction and negative example propagation, redundant information is effectively removed from the selected redundant dimensions, alleviating excessive smoothing in sentence embedding and improving semantic representation capabilities. The purpose of redundancy representation separation is to remove or reduce the impact of redundant information on sentence embedding through regularization strategies, thereby mitigating the excessive smoothing problem.

[0116] Step 3.1, direct subtraction method:

[0117] The direct subtraction method, during the comparison fine-tuning stage, directly subtracts the influence of redundant representations on the redundancy dimension from the sentence embeddings. The implementation process of the direct subtraction method is as follows:

[0118] First, in each training batch, the difference between the embedded representation and the redundant representation of the data within the batch is calculated on the redundant dimension;

[0119] Then, contrastive learning optimization is performed using the corrected embedding representation, with the following loss function:

[0120] ;

[0121] In the formula, Indicates the first The loss value for each sample; this is the optimization objective in contrastive learning, used to measure whether the similarity between positive pairs is maximized, while the similarity between negative pairs is minimized. Indicates the first The corrected embedding representation of each sample; the corrected embedding representation is obtained by subtracting from the redundant dimension or pushing away the negative examples. Indicates the first The positive example embedding representation after sample correction; in contrastive learning, a positive example usually refers to a sentence that is semantically similar to the current sample, such as different data-enhanced versions of the same sentence. Indicates the first The positive example embedding representation of each sample after correction; when calculating the denominator, the positive example embedding representation of all samples needs to be considered. This represents the similarity function between two embedded vectors; the cosine similarity function is defined as follows: ,in, It is the vector dot product. and They are vectors and The length of the module. This represents the temperature parameter; the temperature parameter controls the distribution range of the similarity score. Smaller... Values ​​amplify differences between similarity scores, while larger values... The value will make the score smoother. This represents an exponential function; the exponential function is used to map similarity scores to a positive range and amplify the effect of high similarity scores. Indicates all The positive embeddings of each sample are summed; the denominator is used for normalization to ensure that the value of the loss function is within a reasonable range. This represents taking the negative logarithm of the normalized similarity score; the role of the negative logarithm is to transform the optimization objective into a minimization problem: when the similarity score of positive pairs is high, the loss value is low; conversely, the loss value is high.

[0122] The purpose of this formula is to optimize the model's embedding representation through contrastive learning, maximizing the similarity between semantically similar positive pairs and minimizing the similarity between semantically unrelated negative pairs. The numerator measures the similarity between the current sample and its positive counterparts; the denominator normalizes the similarity scores of all positive pairs; the ultimate goal is to minimize the loss value, enabling the model to better capture the semantic relationships between sentences. This loss function, combined with the modified embedding representation, effectively removes the influence of redundant information on sentence embeddings, thereby improving the model's performance in semantic similarity tasks.

[0123] Step 3.1 The direct subtraction method is simple and efficient, and can directly remove redundant information.

[0124] Step 3.2, Extending the Negative Example Method:

[0125] The negative example push-out method leverages the optimization objective of negative example push-out in contrastive learning, treating redundant representations as additional negative examples and reducing redundant encodings in sentence embeddings by pushing these negative examples out. The implementation process of the negative example push-out method is as follows:

[0126] First, redundant representations are added as additional negative examples to the contrastive learning framework;

[0127] Then, during the comparative fine-tuning process, the loss function is optimized to simultaneously narrow the distance to positive examples and widen the distance to negative examples:

[0128] ;

[0129] In the formula, Indicates the first An embedding representation of a sample; it is a sentence embedding vector generated by the model to represent the semantic features of the current sentence. Indicates the first The positive example embedding representation of a sample; in contrastive learning, a positive example is usually a sentence that is semantically similar to the current sample, such as different data-enhanced versions of the same sentence. Indicates the first The positive example embeddings of each sample are considered when calculating the denominator. Indicates the first The negative example embedding representation of each sample; in contrastive learning, negative examples usually refer to sentences that are semantically unrelated to the current sample. Redundancy representation is a vector of redundant information constructed from a high-frequency word pool or batch data, used to capture redundant codes in sentence embeddings. This indicates the number of selected token embedding vectors; when calculating the denominator, it is necessary to iterate through all positive and negative examples of the samples. Indicates the number of redundant negative examples; redundant negative examples are introduced as additional negative examples to further reduce redundant information in the sentence embedding. This represents the similarity function between two embedded vectors. This represents an exponential function; the exponential function is used to map similarity scores to a positive range and amplify the effect of high similarity scores.

[0130] Step 3.2 The push-away negative example method makes full use of the contrastive learning framework, thereby indirectly reducing redundant information.

[0131] Based on the above steps, this embodiment proposes an unsupervised sentence learning method based on redundancy information reduction. This method effectively solves the over-smoothing problem in sentence embedding by constructing, filtering, and separating token-level redundant representations. The method demonstrates good performance improvement in both unsupervised and supervised learning scenarios, with particularly outstanding performance in unsupervised scenarios. This method not only promotes the development of natural language processing technology but also provides strong technical support for practical applications.

[0132] Furthermore, to verify the effectiveness of this unsupervised sentence learning method based on redundant information reduction, the following experiments were conducted in this embodiment:

[0133] Experimental platform: The experiment was conducted on an NVIDIA RTX 3080 GPU platform.

[0134] Hyperparameter configuration: Strictly follow the hyperparameter configuration scheme in the original study to ensure the reliability of experimental results.

[0135] Comparison method: Data augmentation using dropout for comparative learning;

[0136] The proposed method includes two strategies for reducing redundant information:

[0137] The direct subtraction method (sub) removes redundant information directly from the embedded representation.

[0138] Neg: Redundant representations are treated as additional negative examples, and these negative examples are pushed away by comparative learning to optimize the target.

[0139] The experimental results are shown in Tables 1, 2, and 3:

[0140] Table 1. Comparison of sentence embedding performance on STS tasks

[0141]

[0142] Table 2 Comparison of Redundancy Representation Construction Methods

[0143]

[0144] Table 3 Comparison of Redundant Dimension Filtering Methods

[0145]

[0146] In Tables 1, 2, and 3, STS12, STS13, STS14, STS15, and STS16 refer to the SemEval semantic text similarity STS task datasets from 2012 to 2016, used to evaluate the semantic similarity calculation capabilities of sentence embedding models in different years. STS-B refers to the SemEval 2017 benchmark semantic text similarity STS Benchmark dataset, widely used to test the performance of sentence embedding models. SICK-R refers to the relevance scoring task of the Sentences Involving Compositional Knowledge dataset, used to evaluate the ability of sentence embedding models to capture sentence relevance.

[0147] Analysis of Experimental Results

[0148] 1. Unsupervised learning scenarios:

[0149] The experimental results are shown in the upper part of Table 1, demonstrating that the unsupervised sentence learning method based on redundant information reduction proposed in this embodiment has a significant performance improvement in unsupervised learning scenarios:

[0150] SimCSE-BERTbase model: The original model has an average performance of 75.21; after using the "pull-out negative example method", the performance is improved to 75.97; after using the "direct subtraction method", the performance is further improved to 77.34.

[0151] RoBERTabase model: The original model has an average performance of 76.35; after using the "push-away negative example method", the performance is improved to 76.65; after using the "direct subtraction method", the performance is further improved to 77.29.

[0152] Conclusion: The direct subtraction method outperforms the negative example push method, effectively removing redundant encoded information and thus achieving superior performance. Therefore, the unsupervised sentence learning method based on redundant information reduction proposed in this embodiment demonstrates good versatility and adaptability on different base models, BERT and RoBERTa.

[0153] 2. Supervised learning scenarios:

[0154] The experimental results are shown in the lower half of Table 1, indicating that the unsupervised sentence learning method based on redundant information reduction proposed in this embodiment is also effective in supervised learning scenarios, but the performance improvement is relatively small.

[0155] SimCSE-BERTbase model: The original model has an average performance of 81.00; after using the "push-away negative example method", the performance is improved to 81.54; after using the "direct subtraction method", the performance is further improved to 81.76.

[0156] Cause analysis: In supervised learning frameworks, the complexity of semantic distribution is higher, which may lead to the accidental deletion of some key semantic information during the process of reducing redundant representations, thereby affecting the final performance of the model.

[0157] 3. Comparative experiment on redundancy representation construction methods:

[0158] The in-batch data method is a redundancy reduction approach based on the current training batch data. It segments the sentences in each training batch, sorts them in descending order of word frequency, selects high-frequency tokens, calculates the mean of the embedding vectors of these tokens as the redundancy representation for that batch, and directly subtracts the redundancy representation from the sentence embeddings on the redundancy dimension.

[0159] The high-frequency word pooling method is a redundancy reduction method based on a global high-frequency word pool. It constructs a redundancy representation pool by statistically analyzing high-frequency words in an unsupervised corpus, randomly samples several tokens from it, calculates the mean of their embedding vectors as the redundancy representation, and directly subtracts the redundancy representation from the sentence embedding on the redundancy dimension.

[0160] Table 2 shows a performance comparison of the two redundant representation construction methods; among them, the high-frequency word pool method (high sub) has an average performance of 77.34, and the batch data method (batch sub) has an average performance of 77.01.

[0161] Conclusion: The high-frequency word pool method performs slightly better than the batch data method, indicating that a global perspective is better able to capture redundant information in certain scenarios.

[0162] 4. Comparative experiment on redundant dimension screening methods:

[0163] The static screening method is a redundant dimension screening method based on a global high-frequency vocabulary pool. It performs statistical analysis on the embedding vectors of high-frequency words, calculates the standard deviation of each dimension, and marks dimensions with a standard deviation below a threshold as redundant dimensions.

[0164] Dynamic filtering is a method for filtering redundant dimensions based on the current training batch data. It segments the sentences in each training batch, sorts them in descending order of word frequency, selects high-frequency tokens, calculates the standard deviation of the embedding vectors of these tokens in each dimension, and marks the dimensions with standard deviations below a threshold as redundant dimensions of that batch.

[0165] Table 3 shows a performance comparison of the two redundant dimension filtering methods; among them, the static filtering method has an average performance of 76.80, and the dynamic filtering method has an average performance of 77.34.

[0166] Conclusion: The dynamic screening method outperforms the static screening method, indicating that the dynamic method is more adaptable to changes in different batches of data, thus playing a greater role in the stage of reducing redundant information.

[0167] Based on the above experiments, the unsupervised sentence learning method based on redundant information reduction proposed in this invention shows significant performance improvements in both unsupervised and supervised learning scenarios. Especially in unsupervised scenarios, the direct subtraction method exhibits the best performance, verifying its efficiency in removing redundant information. Among the redundant representation construction methods, the high-frequency word pool method is slightly better than the batch data method; among the redundant dimension filtering methods, the dynamic filtering method is better than the static filtering method, demonstrating the flexibility and adaptability of the dynamic method.

[0168] In supervised learning scenarios, the high complexity of semantic distribution means that redundant information reduction may lead to the accidental deletion of key semantic information, thus limiting the extent of performance improvement. Future research could focus on further optimizing the identification and reduction mechanisms for redundant information to minimize accidental deletion of key semantic information; and exploring the application potential in multilingual support and low-resource language scenarios.

[0169] The above examples are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above examples. Any modifications, alterations, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. An unsupervised sentence learning method based on redundant information reduction, characterized in that, Includes the following steps: Step 1: Redundancy Representation Construction: Construct global and local redundancy information representations using two methods: high-frequency word pool and batch data. Step 2: Redundant Dimension Filtering: Based on the constructed global and local redundancy information representations, static and dynamic methods are used to accurately identify redundant dimensions that contribute little to semantic distinction. Step 3: Redundancy Representation Separation: Using the direct subtraction method and the negative example push method, redundant information is effectively removed from the selected redundant dimensions, which alleviates the excessive smoothing in sentence embedding and improves semantic representation ability. In step 1, the process of constructing a token-level redundant representation based on high-frequency words is as follows: First, the unsupervised dataset is segmented using the NLTK word segmentation tool; Then, the word frequencies are counted, and the words with the highest frequencies are selected as the redundant representation pool; Then, in each training batch, a number of tokens are randomly sampled from the candidate pool and represented by k; Finally, the mean of the token embedding vector is calculated as a redundant representation for each batch. Constructing a token-level redundant representation based on high-frequency words is the global perspective in step 1; In step 1, the process of constructing the token-level redundancy representation based on the batch data is as follows: First, the sentences in each training batch are segmented into words; Then, sort them in descending order of word frequency and select the top few tokens; Finally, the mean of the token embedding vector is calculated as a redundant representation for each batch. The token-level redundancy representation based on the batch data is constructed as the local perspective in step 1; In step 2, the static redundant dimension filtering method proceeds as follows: First, select the tokens with the highest word frequency from the high-frequency word list; Then, the standard deviation of the embedding vector of the most frequent token is calculated in each dimension; Finally, the threshold was determined based on the experiment. Standard deviation less than the threshold The dimension is marked as a redundant dimension; The static redundant dimension filtering method is the global static filtering in step 2; In step 2, the process of the dynamic redundant dimension filtering method is as follows: First, the sentences in each training batch are segmented into words, sorted in descending order of word frequency, and the top few tokens are selected. Then, the standard deviation of the token embedding vector in each dimension is calculated; Finally, the threshold was determined based on the experiment. Standard deviation below the threshold The dimension is marked as a redundant dimension in this batch; The dynamic redundancy dimension filtering method is the local dynamic filtering in step 2.

2. The unsupervised sentence learning method based on redundant information reduction according to claim 1, characterized in that: The mean of the token embedding vector is calculated using the following formula: ； In the formula, The redundant representation of this batch is a vector; Indicates the number of tokens selected from the candidate pool or batch data; Indicates the first Embedding vectors of each token; This indicates that all selected items... Element-wise summation of each token embedding vector; This indicates that the sum vector is averaged.

3. The unsupervised sentence learning method based on redundant information reduction according to claim 1, characterized in that: The standard deviation of the token embedding vector across each dimension is calculated using the following formula: ； In the formula, Indicates the first Standard deviation of dimension; Indicates the number of selected token embedding vectors; Indicates the first The Token embedding vector of the _th ... Dimensional value; Indicates the first The mean of the dimension; Indicates the first The token embedding vector at the th in the th The squared deviation between the value on the dimension and its mean; Indicates all The token embedding vector at the th in the th Summing the squared deviations over the dimensions; This indicates that the total dispersion has been normalized. The standard deviation is obtained by taking the square root of the average dispersion.

4. The unsupervised sentence learning method based on redundant information reduction according to claim 1, characterized in that: In step 3, the direct subtraction method is performed as follows: First, in each training batch, the difference between the embedded representation and the redundant representation of the data within the batch is calculated on the redundant dimension; Then, contrastive learning optimization is performed using the corrected embedding representation, with the following loss function: ； In the formula, Indicates the first The loss value for each sample; Indicates the first The corrected embedding representation of each sample; Indicates the first The corrected positive example embedding representation of each sample; Indicates the first The corrected positive example embedding representation of each sample; This represents a similarity function between two embedded vectors; Indicates temperature parameter; Represents an exponential function; Indicates all Summing the positive example embeddings of each sample; This represents taking the negative logarithm of the normalized similarity score.

5. The unsupervised sentence learning method based on redundant information reduction according to claim 1, characterized in that: In step 3, the process of pushing the negative example further is as follows: First, redundant representations are added as additional negative examples to the contrastive learning framework; Then, during the comparative fine-tuning process, the loss function is optimized to simultaneously narrow the distance to positive examples and widen the distance to negative examples: ； In the formula, Indicates the first The loss value for each sample; Indicates the first Embedded representation of each sample; Indicates the first Positive example embeddings of each sample; Indicates the first Positive example embeddings of each sample; Indicates the first Negative example embedding representation of each sample; Indicates the number of selected token embedding vectors; Indicates the number of redundant negative examples; This represents a similarity function between two embedded vectors; Indicates temperature parameter; Represents an exponential function; , Respectively represent all Summing is performed on the positive examples of each sample and summing is performed on the negative example embeddings; This represents taking the negative logarithm of the normalized similarity score.

Images