Sentence representation contrast learning method using artificial intelligence feedback

Abstract

The invention provides a sentence representation comparative learning method using artificial intelligence feedback, which is used for obtaining corresponding sentence representation according to a sentence to be recognized, and is characterized by comprising the following steps: S1, constructing a training set according to a large language model LLMs and existing data; s2, training an existing language model according to the training set, updating the language model according to the loss function, and taking the trained language model as a sentence encoder; and S3, inputting the sentence to be recognized into a sentence encoder to obtain a corresponding sentence representation. In a word, the method can generate high-quality sentence representation.

Description

Technical Field

[0001] The present invention belongs to the field of deep learning and natural language processing, and specifically relates to a sentence representation contrast learning method using artificial intelligence feedback. Background Art

[0002] In natural language processing tasks, learning sentence representations with rich semantics is a basic task and is also crucial for downstream tasks such as semantic matching and information retrieval. Pre-trained language models such as BERT, RoBERTa, T5, etc. provide a convenient way to obtain sentence representations - that is, directly using pre-trained language models to encode text sentences. However, the sentence representations generated by pre-trained language models perform poorly on semantic text similarity tasks. It is generally believed that this is due to the representation degradation problem introduced by the pre-training process. Therefore, how to further improve the pre-trained model to produce better sentence representations has become an important challenge in natural language processing.

[0003] On the other hand, due to the shortage of labeled data, research in recent years has mainly focused on unsupervised methods, such as contrastive learning. Contrastive learning can be divided into two categories: supervised contrastive learning and unsupervised contrastive learning. The contrastive learning method used depends on whether additional label information is used to construct positive and negative sample pairs. Generally speaking, the quality of positive and negative sample pairs in unsupervised contrastive learning of natural language is difficult to guarantee. Recent studies have also shown that data augmentation strategies in unsupervised contrastive learning may introduce some biases, such as length information and inappropriate negative samples. Existing supervised contrastive learning methods also have some problems. Although they can use label information to generate more accurate sample pairs, such as supervised datasets using natural language inference, they can only provide coarse-grained labels, lack fine-grained supervisory signals, and face high labor costs.

[0004] In order to further improve the pre-trained model to learn better sentence representation, with the emergence of large pre-trained language models (LLMs), technicians in this field consider using these powerful large models to help train other artificial intelligence models. One of the methods is to use large models to generate data sets for zero-shot learning. However, this method often uses predefined labels and task descriptions to generate training inputs instead of using artificial intelligence feedback as a supervisory signal. Therefore, existing large model-based methods are not suitable for tasks where the labels are continuous values, and may result in a lack of diversity in the generated training samples, which in turn makes the sentence representation unable to capture richer semantic knowledge, affecting the quality of the sentence representation. Summary of the invention

[0005] The present invention is made to solve the above-mentioned problems, and aims to provide a sentence representation contrast learning method using artificial intelligence feedback.

[0006] The present invention provides a sentence representation contrast learning method using artificial intelligence feedback, which is used to obtain corresponding sentence representations according to a sentence to be recognized, and has the following characteristics, including the following steps: step S1, constructing a training set according to large language models (LLMs) and existing data; step S2, training the existing language model according to the training set, and updating the language model according to a loss function, and using the trained language model as a sentence encoder; step S3, inputting the sentence to be recognized into the sentence encoder to obtain a corresponding sentence representation, wherein the existing data is a plurality of unpaired single sentences or an existing data set with label information.

[0007] In the sentence representation contrast learning method using artificial intelligence feedback provided by the present invention, it can also have the following characteristics: wherein, step S1 includes the following sub-steps: step S1-1, selecting a sentence from multiple single sentences as the original sentence, and masking at least one word in the original sentence by a mask symbol as the sentence to be input; step S1-2, inputting the sentence to be input into the large language model LLMs, generating a new sentence according to the set generation prompt, and taking the original sentence and the corresponding new sentence as a sentence pair; step S1-3, according to the set marking prompt, through the artificial intelligence feedback of the large language model LLMs, marking the semantic similarity score of the sentence pair; step S1-4, repeatedly executing steps S1-1 to S1-3 to obtain multiple sentence pairs and corresponding semantic similarity scores as a training set.

[0008] The sentence representation contrast learning method using artificial intelligence feedback provided by the present invention may also have the following feature: wherein, in step S1-1, when multiple mask symbols in the original sentence are adjacent, the multiple mask symbols are merged into one mask symbol.

[0009] In the sentence representation contrastive learning method using artificial intelligence feedback provided by the present invention, there may also be such a feature: wherein, in step S2, the loss function is expressed as follows: Where N is the batch size during training, h i and h' i For the i-th sentence pair (x i ,x' i ) is the sentence representation after being encoded by the language model, x i is the original sentence of the ith sentence pair, x' i is the new sentence of the i-th sentence pair, y i is the semantic similarity score of the i-th sentence pair, and cos is the calculation of Cosine similarity.

[0010] In the sentence representation contrast learning method using artificial intelligence feedback provided by the present invention, it can also have the following characteristics: wherein, step S1 includes the following sub-steps: step S1-1, constructing positive and negative sentence sample pairs according to label information; step S1-2, marking semantic similarity scores for positive and negative sentence sample pairs according to artificial intelligence feedback of large language models LLMs; step S1-3, repeatedly executing steps S1-1 to S1-2 to obtain multiple positive and negative sentence sample pairs and corresponding semantic similarity scores as training sets.

[0011] In the sentence representation contrastive learning method using artificial intelligence feedback provided by the present invention, there may also be such a feature: wherein, in step S2, the loss function is expressed as follows: Where N is the batch size during training, h i , are the original sentences x in the i-th positive and negative sentence sample pair respectively. i and the corresponding original sentence x i Positive sample sentences The sentence representation encoded by the language model, and are the original sentences x in the jth positive and negative sentence sample pairs respectively. j Positive sample sentences And negative sample sentences The sentence representation after encoding by the language model, y i is the i-th sentence pair The semantic similarity score is , τ is the temperature coefficient, and cos is the calculation of Cosine similarity.

[0012] Functions and Effects of the Invention

[0013] According to the sentence representation contrastive learning method using artificial intelligence feedback involved in the present invention, because high-quality sentence pairs and semantic similarity scores are generated as training data through large language models LLMs, the trained language model can generate better sentence representations, and the language model can be applied to standard semantic text similarity tasks and sentence representation transfer learning tasks to achieve better results. Therefore, the sentence representation contrastive learning method using artificial intelligence feedback of the present invention can generate high-quality sentence representations. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 is a schematic diagram of a process of comparative learning from artificial intelligence feedback in an embodiment of the present invention;

[0015] Figure 2 is a schematic diagram of generating a new sentence from an original sentence in an embodiment of the present invention;

[0016] Figure 3 is a schematic diagram of marking semantic similarity scores for sentence pairs in an embodiment of the present invention;

[0017] Figure 4 It is a flowchart of comparative learning from human feedback and artificial intelligence feedback in an embodiment of the present invention. DETAILED DESCRIPTION

[0018] In order to make the technical means, creative features, objectives and effects achieved by the present invention easy to understand, the following embodiments and accompanying drawings specifically illustrate the sentence representation comparison learning method using artificial intelligence feedback of the present invention.

[0019] <Example 1>

[0020] Figure 1 It is a flowchart of comparative learning from artificial intelligence feedback in an embodiment of the present invention.

[0021] like Figure 1 As shown, the sentence representation contrast learning method using artificial intelligence feedback in this embodiment is used to obtain the corresponding sentence representation according to the sentence to be recognized, including the following steps:

[0022] Step S1, constructing a training set according to a large language model LLMs and existing data, where the existing data is a plurality of unpaired single sentences. In this embodiment, the large language model LLMs is a GPT-3 language model.

[0023] Wherein, step S1 includes the following sub-steps:

[0024] Step S1-1, select a sentence from multiple single sentences as the original sentence, and use a mask symbol to mask at least one word in the original sentence as the sentence to be input. In this embodiment, the mask symbol is <mask>Symbols, by masking the words to remove some information in the original sentence.

[0025] When multiple mask symbols in the original sentence are adjacent, the multiple mask symbols are merged into one mask symbol. In this embodiment, 50% of the adjacent multiple mask symbols in the masked sentences are merged into one to generate different sentence lengths.

[0026] Step S1-2, input the sentence to be input into the large language model LLMs, generate a new sentence according to the set generation prompt, and use the original sentence and the corresponding new sentence as a sentence pair.

[0027] Figure 2 It is a schematic diagram of generating a new sentence from an original sentence in an embodiment of the present invention.

[0028] like Figure 2 As shown, the original sentence "a man is playing a flute." is transformed into <mask>The symbols are used to mask the words, and the three sentences to be input are: "a man <mask>playing a flute.”、"a man is playing <mask> <mask>.” and "a <mask>playing <mask> <mask>.", and then input these three sentences to be input into the GPT-3 language model. The GPT-3 language model generates three corresponding new sentences according to the task description prompts: "a man was playing a flute.", "a man is playing a gultar." and "a boy and girl playing hopsoctch together.".

[0029] Step S1-3, according to the set marking prompts, through the artificial intelligence feedback of the large language model LLMs, the sentence pairs are marked with semantic similarity scores. In this embodiment, the marking prompts are used to make the range of the semantic similarity score from 0 to 1. When the semantic similarity score is 1, it indicates that the semantics of the two sentences in the corresponding sentence pair are exactly the same. When the semantic similarity score is 0, it indicates that the semantics of the two sentences in the corresponding sentence pair are completely different. The diversity of the semantic similarity scores is guaranteed by the constructed diversity of sentence pairs.

[0030] Figure 3 It is a schematic diagram of marking semantic similarity scores for sentence pairs in an embodiment of the present invention.

[0031] like Figure 3 As shown in the figure, three sentence pairs ("aman is playing a flute.","aman was playing aflute."), ("a man is playing a flute.","a man is playing a gultar.") and ("a manis playing a flute.","a boy and girl playing hopsoctch together.") are respectively input into the GPT-3 language model, and according to the marking prompts, the corresponding semantic similarity scores are obtained, namely 0.89, 0.20 and 0.00.

[0032] Step S1-4, repeatedly executing steps S1-1 to S1-3, to obtain a plurality of sentence pairs and corresponding semantic similarity scores as a training set.

[0033] In this embodiment, because the sentence pairs have fine-grained semantic similarity scores, there is no need for explicitly constructed positive and negative sample pairs in the training set.

[0034] Step S2: train the existing language model according to the training set, and update the language model according to the loss function, and use the trained language model as a sentence encoder. In this embodiment, the language model is a BERT language model or a RoBERTa language model.

[0035] Among them, the loss function is expressed as follows:

[0036]

[0037] Where N is the batch size during training, h i and h' i For the i-th sentence pair (x i ,x' i ) is the sentence representation after being encoded by the language model, x i is the original sentence of the ith sentence pair, x' i is the new sentence of the i-th sentence pair, y i is the semantic similarity score of the i-th sentence pair, and cos is the calculation of Cosine similarity.

[0038] In this embodiment, the cosine similarity of the sentence representation is cos(h i ,h' i ) as the similarity score of the language model for the sentence pair, and use the mean square error loss to fit the similarity score of the sentence pair with the semantic similarity score to train the language model.

[0039] Step S3: input the sentence to be recognized into a sentence encoder to obtain a corresponding sentence representation.

[0040] In this embodiment, the comparative learning method from artificial intelligence feedback of the present invention, that is, the first method of the present invention, is compared with the existing BERT-flow method, BERT-whitening method, Prompt based BERT method, ConSERT method, SimCSE method, DiffCSE method, PromptBERT method and Dino GPT-3 Methods The BERT language model was trained to obtain sentence encoders, and sentence representations were output on the existing STS12, STS13, STS14, STS15, STS16, STS-B, and SICK-R datasets. The Spearman correlation coefficient was calculated based on the output sentence representations. The calculation results are shown in the following table:

[0041]

[0042]

[0043] The first column in the above table is the method name, the second to eighth columns are the calculation results of the Spearman correlation coefficient of the BERT language model trained by each method on the STS12 dataset, STS13 dataset, STS14 dataset, STS15 dataset, STS16 dataset, STS-B dataset and SICK-R dataset, respectively, and the ninth column is the average value of the calculation results of the Spearman correlation coefficient of the BERT language model trained by each method on each dataset. For example, the cell in the second row and the second column indicates that the calculation result of the Spearman correlation coefficient of the BERT language model trained by the BERT-flow method on the STS12 dataset is 58.40. It can be seen that the comparative learning method from artificial intelligence feedback of the present invention has the highest average Spearman correlation coefficient on the above datasets compared with other methods, and has the highest Spearman correlation coefficient on the STS15 dataset, STS16 dataset, STS-B dataset and SICK-R dataset, indicating that the sentence representation generated by the BERT language model trained by this method has a higher correlation.

[0044] In this embodiment, the comparative learning method from artificial intelligence feedback of the present invention, that is, the first method of the present invention, is compared with the existing BERT-whitening method, SimCSE method, DiffCSE method, PromptRoBERTa method, Dino method and Dino GPT-3 Methods The RoBERTa language model was trained to obtain sentence encoders, and sentence representations were output on the existing STS12, STS13, STS14, STS15, STS16, STS-B, and SICK-R datasets. The Spearman correlation coefficient was calculated based on the output sentence representations. The calculation results are shown in the following table:

[0045]

[0046] The first column in the above table is the method name, the second to eighth columns are the Spearman correlation coefficient calculation results of the BERT language model trained by each method on the STS12 dataset, STS13 dataset, STS14 dataset, STS15 dataset, STS16 dataset, STS-B dataset and SICK-R dataset, respectively, and the ninth column is the average value of the Spearman correlation coefficient calculation results of the BERT language model trained by each method on each dataset. For example, the cell in the second row and the second column indicates that the Spearman correlation coefficient calculation result of the RoBERTa language model trained by the BERT-whitening method on the STS12 dataset is 46.99. It can be seen that the comparative learning method from artificial intelligence feedback of the present invention has the highest average Spearman correlation coefficient on the above datasets compared with other methods, and has the highest Spearman correlation coefficient on the STS15 dataset, STS16 dataset, STS-B dataset and SICK-R dataset, indicating that the sentence representation generated by the BERT language model trained by this method has a higher correlation.

[0047] <Example 2>

[0048] For the convenience of expression, in this embodiment, the same symbols are given to the same structures as those in the first embodiment, and the same descriptions are omitted.

[0049] Figure 4 It is a flowchart of comparative learning from human feedback and artificial intelligence feedback in an embodiment of the present invention.

[0050] like Figure 4 As shown, the sentence representation comparison learning method using artificial intelligence feedback in the embodiment is used to obtain the corresponding sentence representation according to the sentence to be recognized, including the following steps:

[0051] Step S1, constructing a training set according to the large language model LLMs and existing data, where the existing data is an existing data set with label information.

[0052] Wherein, step S1 includes the following sub-steps:

[0053] Step S1-1, construct positive and negative sentence sample pairs according to label information.

[0054] Step S1-2, according to the set marking prompt, through the artificial intelligence feedback of the large language model LLMs, the semantic similarity scores of the positive and negative sentence sample pairs are marked. In this embodiment, the semantic similarity score is marked for each positive sample pair.

[0055] Step S1-3, repeatedly executing steps S1-1 to S1-2, to obtain a plurality of positive and negative sentence sample pairs and corresponding semantic similarity scores as a training set.

[0056] Step S2: train the existing language model according to the training set, update the language model according to the loss function, and use the trained language model as a sentence encoder.

[0057] Among them, in step S2, the loss function is expressed as follows:

[0058]

[0059]

[0060] Where N is the batch size during training, h i , are the original sentences x in the i-th positive and negative sentence sample pair respectively. i and the corresponding original sentence x i Positive sample sentences The sentence representation encoded by the language model, and are the original sentences x in the jth positive and negative sentence sample pairs respectively. j Positive sample sentences And negative sample sentences The sentence representation after encoding by the language model, y i is the i-th sentence pair The semantic similarity score, τ is the temperature coefficient, cos is the calculation of Cosine similarity, in this embodiment, the positive sample sentence is a sentence that is semantically similar to the corresponding original sentence, and the negative sample sentence is a sentence that is not semantically similar to the corresponding original sentence.

[0061] In this embodiment, the semantic similarity score is used to replace the unique hot label used in the InfoNCE loss function, providing a more fine-grained supervision signal for contrastive learning.

[0062] Step S3: input the sentence to be recognized into a sentence encoder to obtain a corresponding sentence representation.

[0063] In this embodiment, the comparative learning method from human feedback and artificial intelligence feedback of the present invention, i.e., the second method of the present invention, is used together with the existing SBERT method, SBERT-flow method, SBERT-whitening method, ConSERT method, SimCSE method, SimCSE+CLHAIF method, PromptBERT method, and PromptBERT+CLHAIF method to train the BERT language model to obtain a sentence encoder, CLHAIF is supervised comparative learning, and sentence representations are output on the existing STS12 dataset, STS13 dataset, STS14 dataset, STS15 dataset, STS16 dataset, STS-B dataset, and SICK-R dataset, respectively, and the Spearman correlation coefficient is calculated based on the output sentence representation. The calculation results are shown in the following table:

[0064]

[0065]

[0066] The first column in the above table is the method name, the second to eighth columns are the calculation results of the Spearman correlation coefficient of the BERT language model trained by each method on the STS12 dataset, STS13 dataset, STS14 dataset, STS15 dataset, STS16 dataset, STS-B dataset and SICK-R dataset, respectively, and the ninth column is the average value of the calculation results of the Spearman correlation coefficient of the BERT language model trained by each method on each dataset. For example, the cell in the second row and the second column indicates that the calculation result of the Spearman correlation coefficient of the BERT language model trained by the SBERT method on the STS12 dataset is 70.97. It can be seen that the comparative learning method from artificial intelligence feedback of the present invention has the highest average Spearman correlation coefficient on the above datasets compared with other methods, and has the highest Spearman correlation coefficient on the STS15 dataset, STS16 dataset and STS-B dataset, indicating that the sentence representation generated by the BERT language model trained by this method has a higher correlation.

[0067] In this embodiment, the comparative learning method from human feedback and artificial intelligence feedback of the present invention, i.e., the second method of the present invention, is used to train the RoBERTa language model with the existing SRoBERTa method, SRoBERTa-whitening method, SimCSE method, SimCSE+CLHAIF method, PromptRoBERTa method, and PromptRoBERTa+CLHAIF method to obtain sentence encoders, and sentence representations are output on the existing STS12 dataset, STS13 dataset, STS14 dataset, STS15 dataset, STS16 dataset, STS-B dataset, and SICK-R dataset, respectively. The Spearman correlation coefficient is then calculated based on the output sentence representations, and the calculation results are shown in the following table:

[0068]

[0069] The first column in the above table is the method name, the second to eighth columns are the Spearman correlation coefficient calculation results of the RoBERTa language models trained by each method on the STS12 dataset, STS13 dataset, STS14 dataset, STS15 dataset, STS16 dataset, STS-B dataset and SICK-R dataset, respectively, and the ninth column is the average value of the Spearman correlation coefficient calculation results of the RoBERTa language models trained by each method on each dataset. For example, the cell in the second row and the second column indicates that the Spearman correlation coefficient calculation result of the RoBERTa language model trained by the SRoBERTa method on the STS12 dataset is 71.54. It can be seen that the comparative learning method from artificial intelligence feedback of the present invention has the highest Spearman correlation coefficient on the STS15 dataset, STS16 dataset and STS-B dataset compared with other methods, indicating that the sentence representation generated by the RoBERTa language model trained by this method has good correlation.

[0070] Functions and Effects of the Embodiments

[0071] According to the sentence representation contrastive learning method using artificial intelligence feedback involved in this embodiment, high-quality sentence pairs and semantic similarity scores are generated as training data through large language models LLMs, so that the trained language model can generate better sentence representations, and the language model can be applied to standard semantic text similarity tasks and sentence representation transfer learning tasks to achieve good results. In short, this method can generate high-quality sentence representations.

[0072] The above-mentioned embodiments are preferred examples of the present invention and are not intended to limit the protection scope of the present invention.< / mask> < / mask> < / mask> < / mask> < / mask> < / mask> < / mask> < / mask>

Claims

1. A sentence representation contrast learning method using artificial intelligence feedback, for obtaining corresponding sentence representations according to a sentence to be recognized, characterized in that: The following steps are involved: Step S1, constructing a training set based on large language models LLMs and existing data; Step S2, training an existing language model according to the training set, and updating the language model according to a loss function, and using the trained language model as a sentence encoder; Step S3, inputting the sentence to be recognized into the sentence encoder to obtain the corresponding sentence representation, The existing data is a plurality of unpaired single sentences or an existing data set with label information.

2. The sentence representation contrast learning method using artificial intelligence feedback according to claim 1, Features: Wherein, the step S1 includes the following sub-steps: Step S1-1, selecting a sentence from the plurality of single sentences as an original sentence, and masking at least one word in the original sentence by a mask symbol as a sentence to be input; Step S1-2, inputting the sentence to be input into the large language model LLMs, generating a new sentence according to the set generation prompt, and taking the original sentence and the corresponding new sentence as a sentence pair; Step S1-3, according to the set tag prompt, through the artificial intelligence feedback of the large language model LLMs, the sentence pair is marked with a semantic similarity score; Step S1-4, repeatedly executing steps S1-1 to S1-3 to obtain a plurality of the sentence pairs and the corresponding semantic similarity scores as the training set.

3. The sentence representation contrast learning method using artificial intelligence feedback according to claim 2, characterized in that: in, In the step S1 - 1 , when multiple mask symbols in the original sentence are adjacent, the multiple mask symbols are merged into one mask symbol.

4. The sentence representation contrast learning method using artificial intelligence feedback according to claim 2, characterized in that: in, In step S2, the loss function is expressed as follows: Where N is the batch size during training, h i and h' i For the i-th sentence pair (x i ,x' i ) is the sentence representation after being encoded by the language model, x i is the original sentence of the i-th sentence pair, x' i is the new sentence of the i-th sentence pair, y i is the semantic similarity score of the i-th sentence pair, and cos is the calculation of Cosine similarity.

5. The sentence representation contrast learning method using artificial intelligence feedback according to claim 1, characterized in that: in, The step S1 comprises the following sub-steps: Step S1-1, constructing positive and negative sentence sample pairs according to the label information; Step S1-2, according to the set marking prompt, through the artificial intelligence feedback of the large language model LLMs, marking the semantic similarity score of the positive and negative sentence sample pairs; Step S1-3, repeatedly executing step S1-1 to step S1-2, to obtain a plurality of positive and negative sentence sample pairs and the corresponding semantic similarity scores as the training set.

6. The sentence representation contrast learning method using artificial intelligence feedback according to claim 5, characterized in that: in, In step S2, the loss function is expressed as follows: Where N is the batch size during training, h i , are the original sentences x in the i-th positive and negative sentence sample pair respectively. i and the corresponding original sentence x i Positive sample sentences The sentence representation encoded by the language model, and are the original sentences x in the jth positive and negative sentence sample pairs respectively. j Positive sample sentences And negative sample sentences The sentence representation after encoding by the language model, y i is the i-th sentence pair The semantic similarity score is , τ is the temperature coefficient, and cos is the calculation of Cosine similarity.

Images