Chinese sentence simplification method and system based on contrastive learning

Abstract

The present application proposes a Chinese sentence simplification method and system based on contrastive learning, which includes: mining multiple semantically similar complex sentence-simple sentence pairs based on unsupervised learning; calculating the supervision signal of each sentence pair; adding the supervision signal to the starting position of the complex sentence in the sentence pair in the form of a string, generating a data set of complex sentence-simple sentence pairs with supervision signals, and dividing the data set into a training set, a validation set, and a test set; pruning the preset encoder-decoder based multilingual pre-training model to obtain a Chinese monolingual pre-training model; introducing contrastive learning loss to fine-tune the Chinese monolingual pre-training model, and jointly training a Chinese sentence simplification model; inputting the complex sentences in the test set into the simplified model to generate predicted simplified sentences, and evaluating the effect of the Chinese sentence simplification model. This method can control the generated simplified sentences according to actual needs and improve the fidelity of the generated simplified sentences.

Description

Technical Field

[0001] The present application relates to the technical field of natural language processing, and in particular to a Chinese sentence simplification method and system based on contrastive learning. Background Art

[0002] With the explosive growth of information today, sentence simplification has attracted attention in various fields. Sentence simplification aims to modify the content and structure of the original sentence to reduce the complexity of the sentence, but at the same time requires retaining its main idea and being close to its original semantics so that it is easier to read and understand. It is especially valuable for groups with low literacy, such as children, hearing-impaired people, second language learners, and people with low literacy. In addition, sentence simplification can also improve the performance of other natural language processing (NLP) tasks, such as text summarization, information extraction, semantic role labeling, syntactic analysis, and machine translation.

[0003] In the related technologies, most sentence simplification focuses on the English language. There is little research on Chinese sentence simplification, and there is no public data set. Therefore, a mature and available sentence simplification method is urgently needed. Early sentence simplification methods adopted rule-based methods, mainly considering vocabulary simplification or syntactic simplification. There are two main research methods at present: one is to regard sentence simplification as a machine translation task between monolinguals; the other is to regard sentence simplification as a sentence editing task, that is, deletion, insertion, retention, etc.

[0004] However, in some specific application scenarios, the length of the simplified content of the above sentence simplification method is uncontrollable, and the simplified content is prone to being less faithful to the original sentence and missing key points. Summary of the invention

[0005] The present application aims to solve one of the technical problems in the related art at least to some extent.

[0006] To this end, the first purpose of the present application is to propose a Chinese sentence simplification method based on contrastive learning. The method regards the sentence simplification task as an end-to-end conditional generation task. First, an unsupervised method is used to mine semantically similar complex sentence-simple sentence pairs, and then the editing signal and the lexical and syntactic complexity signal are calculated as supervised signals. Contrastive learning is introduced on the basis of the pre-trained model of the encoder-decoder for fine-tuning, so that users can not only control the generated simplified sentences according to actual needs, but also improve the fidelity of the generated simplified sentences.

[0007] The second objective of this application is to propose a Chinese sentence simplification system based on contrastive learning.

[0008] A third object of the present application is to provide a non-transitory computer-readable storage medium.

[0009] To achieve the above-mentioned purpose, the first embodiment of the present application proposes a Chinese sentence simplification method based on contrastive learning, comprising the following steps:

[0010] Mining multiple semantically similar complex sentence-simple sentence pairs based on unsupervised learning;

[0011] Calculating a supervision signal for each of the complex sentence-simple sentence pairs;

[0012] Adding each of the supervisory signals in the form of a character string to the starting position of the complex sentence in the corresponding sentence pair, generating a data set of complex sentence-simple sentence pairs with supervisory signals, and dividing the data set of complex sentence-simple sentence pairs with supervisory signals into a training set, a validation set, and a test set according to a preset ratio;

[0013] Prune the preset encoder-decoder based multilingual pre-trained model to obtain a Chinese monolingual pre-trained model;

[0014] Based on the training set and the validation set, contrastive learning loss is introduced to fine-tune the Chinese monolingual pre-training model, and a Chinese sentence simplification model is jointly trained;

[0015] The complex sentences in the test set are input into the Chinese sentence simplification model, predicted simplified sentences are generated by the Chinese sentence simplification model, and the simplification effect of the Chinese sentence simplification model is evaluated. When the simplification effect is greater than a preset threshold, the Chinese sentence to be simplified is simplified by the Chinese sentence simplification model.

[0016] Optionally, in one embodiment of the present application, multiple semantically similar complex sentence-simple sentence pairs are mined based on an unsupervised learning method, including: obtaining a large amount of Chinese sentences from a preset resource library; obtaining the vector of each sentence through a language tool library and creating an index, and mining multiple similar candidate sentences corresponding to each sentence; conditionally filtering the candidate sentences corresponding to each sentence, determining the target sentence corresponding to each sentence, and generating the multiple semantically similar complex sentence-simple sentence pairs.

[0017] Optionally, in one embodiment of the present application, the supervisory signal includes a sentence length ratio, an edit distance ratio, a lexical complexity ratio and a syntactic tree depth ratio, and the calculation of the supervisory signal for each complex sentence-simple sentence pair includes: calculating the ratio of the simple sentence length to the complex sentence length in each sentence pair to obtain the sentence length ratio; calculating the Levenshtein distance between the complex sentence and the simple sentence, and calculating the Levenshtein distance ratio of each editing operation to obtain the editing distance ratio, the editing operations including: deletion, insertion and replacement; calculating the ratio of the lexical complexity of the complex sentence to the lexical complexity of the simple sentence to obtain the lexical complexity ratio, wherein the lexical complexity is represented by the word frequency of the vocabulary; obtaining the syntactic tree depths of the complex sentence and the simple sentence respectively through a natural language text processing library, and calculating the ratio of the syntactic tree depth of the complex sentence to the syntactic tree depth of the simple sentence to obtain the syntactic tree depth ratio.

[0018] Optionally, in one embodiment of the present application, each of the supervisory signals is added in the form of a character string to the starting position of the initial complex sentence in the corresponding sentence pair, including: each of the supervisory signals of the sentence pair is added in sequence before the starting position of the corresponding complex sentence in the form of setting the supervisory signal ratio after the supervisory signal name.

[0019] Optionally, in one embodiment of the present application, model pruning is performed on a preset encoder-decoder based multilingual pre-trained model, including: selecting punctuation marks, numbers, English letters and high-frequency Chinese words commonly used in Chinese sentences as a new vocabulary; replacing the original vocabulary of the multilingual pre-trained model with the new vocabulary, and updating the representation parameters of the input vector and output vector of the multilingual pre-trained model to update the multilingual pre-trained model; and saving the new vocabulary and the updated pre-trained model.

[0020] Optionally, in one embodiment of the present application, evaluating the simplification effect of the Chinese sentence simplification model includes: comparing the predicted simplified sentence with a standard reference simplified sentence; evaluating the simplification effect of the Chinese sentence simplification model through a plurality of preset evaluation indicators, the plurality of evaluation indicators including: BLEU-4 indicator, Rouge-L indicator and SARI indicator.

[0021] To achieve the above objectives, the second embodiment of the present application proposes a Chinese sentence simplification system based on contrastive learning, comprising the following modules:

[0022] A mining module, which is used to mine multiple semantically similar complex sentence-simple sentence pairs based on unsupervised learning;

[0023] A calculation module, used for calculating the supervision signal of each complex sentence-simple sentence pair;

[0024] A first generating module is used to add each of the supervisory signals in the form of a character string to the starting position of the complex sentence in the corresponding sentence pair, generate a data set of complex sentence-simple sentence pairs with supervisory signals, and divide the data set of complex sentence-simple sentence pairs with supervisory signals into a training set, a validation set and a test set according to a preset ratio;

[0025] The second generation module is used to perform model pruning on the preset encoder-decoder based multilingual pre-training model to obtain a Chinese monolingual pre-training model;

[0026] A training module, used to introduce contrastive learning loss to fine-tune the Chinese monolingual pre-training model based on the training set and the validation set, and jointly train a Chinese sentence simplification model;

[0027] The third generation module is used to input the complex sentences in the test set into the Chinese sentence simplification model, generate predicted simplified sentences through the Chinese sentence simplification model, evaluate the simplification effect of the Chinese sentence simplification model, and when the simplification effect is greater than a preset threshold, simplify the Chinese sentence to be simplified through the Chinese sentence simplification model.

[0028] Optionally, in one embodiment of the present application, the mining module is specifically used to: obtain a large amount of Chinese sentences from a preset resource library; obtain the vector of each sentence through a language tool library and create an index, and mine multiple similar candidate sentences corresponding to each sentence; perform conditional filtering on the candidate sentences corresponding to each sentence, determine the target sentence corresponding to each sentence, and generate the multiple semantically similar complex sentence-simple sentence pairs.

[0029] Optionally, in one embodiment of the present application, the supervisory signal includes a sentence length ratio, an edit distance ratio, a lexical complexity ratio and a syntactic tree depth ratio, and the calculation module is specifically used to: calculate the ratio of the simple sentence length to the complex sentence length in each of the sentence pairs to obtain the sentence length ratio; calculate the Levenshtein distance between the complex sentence and the simple sentence, and calculate the Levenshtein distance ratio of each editing operation to obtain the editing distance ratio, the editing operations include: deletion, insertion and replacement; calculate the ratio of the lexical complexity of the complex sentence to the lexical complexity of the simple sentence to obtain the lexical complexity ratio, wherein the lexical complexity is represented by the word frequency of the vocabulary; obtain the syntactic tree depths of the complex sentence and the simple sentence respectively through the natural language text processing library, and calculate the ratio of the syntactic tree depth of the complex sentence to the syntactic tree depth of the simple sentence to obtain the syntactic tree depth ratio.

[0030] In order to implement the above-mentioned embodiments, the third aspect of the present application also proposes a non-temporary computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the Chinese sentence simplification method based on contrastive learning in the above-mentioned embodiments is implemented.

[0031] The technical solution provided by the embodiments of the present application brings at least the following beneficial effects: the present application regards the sentence simplification task as an end-to-end conditional generation task, firstly adopts an unsupervised method to mine semantically similar complex sentence-simple sentence pairs, and then uses the editing signal and lexical syntactic complexity signal between the sentence pairs as supervised signals, and uses the encoder-decoder pre-training model to fine-tune, so that the user can control the generated simplified sentences according to different needs. In addition, the present application introduces contrastive learning loss on the basis of the cross entropy loss function to increase the distance between positive and negative samples, thereby improving the fidelity of the generated simplified sentences. The present application not only makes the generation results controllable according to different needs but also can improve the fidelity of the target simplified sentences.

[0032] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, wherein

[0034] Figure 1 A flowchart of a Chinese sentence simplification method based on contrastive learning proposed in an embodiment of the present application;

[0035] Figure 2 A flowchart of a specific method for mining semantically similar sentence pairs proposed in an embodiment of the present application;

[0036] Figure 3 A flowchart of a method for calculating a supervisory signal proposed in an embodiment of the present application;

[0037] Figure 4 A schematic diagram of the structure of a Chinese sentence simplification system based on contrastive learning proposed in an embodiment of the present application. DETAILED DESCRIPTION

[0038] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and should not be construed as limiting the present invention.

[0039] A Chinese sentence simplification method and system based on contrastive learning proposed in an embodiment of the present invention will be described below with reference to the accompanying drawings.

[0040] Figure 1 A flowchart of a Chinese sentence simplification method based on contrastive learning proposed in an embodiment of the present application is shown in FIG. Figure 1 The method comprises the following steps:

[0041] Step S101 , mining a plurality of semantically similar complex sentence-simple sentence pairs based on an unsupervised learning method.

[0042] Among them, unsupervised learning is a machine learning method for pattern recognition based on unlabeled training samples, and there are no labels in the training samples. It can be applied to the present application to mine semantically similar complex sentence-simple sentence pairs in the absence of prior knowledge.

[0043] Among them, the complex sentence-simple sentence pair is a sentence pair consisting of a longer complex sentence and a simple sentence with similar semantics but shorter length, for example, "Today, the construction of Line A was completed" - "Line A was completed today."

[0044] Specifically, an unsupervised method is used to mine semantically similar complex sentence-simple sentence pairs. In order to more clearly illustrate the specific implementation process of mining complex sentence-simple sentence pairs in this application, a semantically similar sentence pair mining method proposed in an embodiment of this application is used as an example to explain in detail. Figure 2 As shown, the method comprises the following steps:

[0045] Step S201: obtaining a large amount of Chinese sentences from a preset resource library.

[0046] Specifically, the resource library is a database pre-stored with a large number of Chinese sentences, which can be various websites and cloud platform servers that can read Chinese data. In this embodiment, a resource library is pre-selected according to actual needs, and a large number of processed Chinese sentences are downloaded from the resource library. For example, a large number of Chinese sentences that have been crawled and pre-processed from the CC-100 website are downloaded.

[0047] Step S202: Obtain the vector of each sentence through the language tool library and create an index, and mine multiple similar candidate sentences corresponding to each sentence.

[0048] Among them, the language tool library is an open source language toolkit that includes multiple languages ​​and supports similarity vector retrieval. For example, the language tool library includes the LASER (Language-Agnostic SEntence Representations) library and the faiss (Facebook AI Similarity Search) library. Among them, the LASER library is a toolkit that can explore multilingual sentence representations. It can write multiple languages ​​through different strings, embed more types of languages ​​into an independent shared space, and achieve cross-language migration without modification. The Faiss library is a library for clustering and similarity search, which provides efficient similarity search and clustering for dense vectors to achieve approximate search.

[0049] In this embodiment, first, each language tool library is used in combination to perform its own search and mining functions, obtain the vector representation of each sentence mined in the above steps, and create an index for each sentence. Then, multiple similar candidate sentences corresponding to each sentence are mined using the Euclidean distance. For example, eight candidate sentences with semantic similarity to each sentence are mined by calculating the Euclidean distance between sentences.

[0050] Step S203: Conditionally filter the candidate sentences corresponding to each sentence, determine the target sentence corresponding to each sentence, and generate a plurality of complex sentence-simple sentence pairs with similar semantics.

[0051] In this embodiment, for each sentence, further conditional filtering is performed on multiple candidate sentences, including filtering out sentences with incorrect content and filtering out sentences with higher semantic similarity, and determining the target sentence corresponding to the current sentence from the multiple candidate sentences. The target sentence refers to the sentence corresponding to the current sentence in the sentence pair. For example, if the current sentence is a complex sentence, the target sentence is the target simple sentence corresponding to the complex sentence.

[0052] Thus, a corresponding semantically similar target simple sentence or target complex sentence is matched for each sentence, and each sentence is paired with the corresponding target sentence to generate multiple semantically similar complex sentence-simple sentence pairs.

[0053] Step S102, calculating the supervisory signal of each complex sentence-simple sentence pair.

[0054] In one embodiment of the present application, the supervisory signal includes information such as sentence length ratio, edit distance ratio, vocabulary complexity ratio and syntax tree depth ratio.

[0055] The present application calculates each type of supervisory signal in a different manner. In order to more clearly illustrate the specific implementation process of calculating each supervisory signal in the present application, a supervisory signal calculation method proposed in an embodiment of the present application is used as an example to explain in detail. Figure 3 As shown, the method comprises the following steps:

[0056] Step S301: Calculate the ratio of the simple sentence length to the complex sentence length in each sentence pair to obtain the sentence length ratio.

[0057] Specifically, the length of the simple sentence and the length of the complex sentence in each sentence pair are first determined, and then the ratio of the length of the simple sentence to the length of the complex sentence is calculated, thereby calculating the sentence length ratio of each sentence pair.

[0058] Step S302: Calculate the Levenshtein distance between the complex sentence and the simple sentence, and calculate the Levenshtein distance ratio of each editing operation to obtain the editing distance ratio, where the editing operations include: deletion, insertion, and replacement.

[0059] Among them, the Levenshtein distance refers to the minimum number of editing operations required to convert one string into another, and can be used to measure the similarity between two strings.

[0060] In this embodiment, when calculating the edit distance ratio, the Levenshtein distance between the complex sentence and the simple sentence is first calculated, and then the Levenshtein ratios of various editing operations such as deletion, insertion and replacement are calculated respectively, that is, this application calculates a total of 4 edit distance ratios.

[0061] Step S303: Calculate the ratio of the lexical complexity of the complex sentence to the lexical complexity of the simple sentence to obtain a lexical complexity ratio, wherein the lexical complexity is represented by the frequency of the vocabulary.

[0062] In this embodiment, the vocabulary complexity is represented by the word frequency of the vocabulary. For each sentence pair, the complexity of each word in the complex sentence and the simple sentence is calculated in turn, and the vocabulary complexity of the complex sentence and the vocabulary complexity of the simple sentence are determined by cumulative calculation. Then, the ratio of the vocabulary complexity of the complex sentence to the vocabulary complexity of the simple sentence is calculated to obtain the vocabulary complexity ratio of each sentence pair.

[0063] Step S304: Obtain the syntactic tree depths of the complex sentence and the simple sentence respectively through the natural language text processing library, and calculate the ratio of the syntactic tree depth of the complex sentence to the syntactic tree depth of the simple sentence to obtain the syntactic tree depth ratio.

[0064] In this embodiment, the natural language text processing library can be a natural language processing open source library such as the spaCy library. The spaCy library can be applied to natural language processing in Python. This application uses the spacy library to obtain the depth of the Chinese dependency syntax tree of complex sentences and simple sentences respectively, and then calculates the ratio of the two to obtain the syntax tree depth ratio.

[0065] Step S103, adding each supervisory signal in the form of a character string to the starting position of the complex sentence in the corresponding sentence pair, generating a data set of complex sentence-simple sentence pairs with supervisory signals, and dividing the data set of complex sentence-simple sentence pairs with supervisory signals into a training set, a validation set and a test set according to a preset ratio.

[0066] In one embodiment of the present application, for the supervisory signals calculated in the embodiment of step S102, each supervisory signal is added to the starting position of the initial complex sentence in the corresponding sentence pair in the form of a character string, including adding each supervisory signal of each sentence pair to the starting position of the corresponding complex sentence in the form of setting the supervisory signal ratio after the supervisory signal name. Among them, the value of the supervisory signal corresponding to the supervisory signal after the supervisory signal name can be added to the starting position of the complex sentence in sequence according to the order of calculating the supervisory signals or setting weights for each supervisory signal and sorting them from large to small.

[0067] For example, the supervision signal of a complex sentence-simple sentence pair is added to the starting position of the original complex sentence in the sentence pair as <supervision signal name_supervision signal ratio>, and a complex sentence-simple sentence dataset with supervision signals is formed on the basis of the original sentence pair. Each row of data is in the form of: <supervision signal name 1_supervision signal ratio 1>…<supervision signal name n_supervision signal ratio n> complex sentence, simple sentence. Then save it in a file format such as csv.

[0068] Furthermore, the dataset of complex sentence-simple sentence pairs with supervisory signals is divided into training set, validation set and test set according to a preset ratio. For example, the dataset of complex sentence-simple sentence pairs with supervisory signals is divided into training set, validation set and test set according to a ratio of 8:1:1, so as to facilitate the subsequent training of the model and verify the performance of the model.

[0069] Step S104, performing model pruning on the preset encoder-decoder based multilingual pre-training model to obtain a Chinese monolingual pre-training model.

[0070] It should be noted that most of the pre-training models for sentence simplification in the related art are multilingual pre-training models, which are implemented based on the encoder-decoder structure. Therefore, the encoder-decoder-based pre-training model containing Chinese usually also contains multiple other languages. As for the Chinese sentence simplification performed in this application, it contains too much useless redundant information and consumes too many resources. The vector representation parameters of other languages ​​may account for more than half of the model parameters, and these useless information will waste computing resources. Therefore, this application can prune redundant vector representations and transform the multilingual pre-training model into a single language pre-training model suitable for Chinese.

[0071] In the specific implementation, the present application pre-selects a multilingual pre-training model in the related technology as a simplified pre-training model for Chinese, and then performs model pruning on the preset encoder-decoder based multilingual pre-training model, prunes redundant vector representations, and converts the multilingual pre-training model into a Chinese monolingual pre-training model.

[0072] As one possible implementation method, punctuation marks, numbers, English letters and high-frequency Chinese words commonly used in Chinese sentences are first selected as a new vocabulary. The preset multilingual pre-training model has an original vocabulary, which may include words in other languages. In this embodiment, punctuation marks, numbers, English letters and high-frequency Chinese words commonly used in Chinese sentences are selected as a new vocabulary to replace the original vocabulary.

[0073] Then, the original vocabulary of the multilingual pre-trained model is replaced with the new vocabulary, and the representation parameters of the input vector and output vector of the multilingual pre-trained model are updated to update the multilingual pre-trained model. In specific implementation, the vocabulary can be directly replaced, or the words contained in the original vocabulary that are not in the new vocabulary can be deleted, and the new words in the new vocabulary can be imported into the original vocabulary. Then, the vector representation of the input and output of the model is updated, thereby replacing the parameters of the input and output vector representation in the multilingual pre-trained model to update the neural network.

[0074] Finally, save the newly generated vocabulary and updated pre-trained model, complete the pruning of the multilingual pre-trained model, and obtain the Chinese monolingual pre-trained model.

[0075] Step S105, based on the training set and the validation set, introduce contrastive learning loss to fine-tune the Chinese monolingual pre-training model, and jointly train a Chinese sentence simplification model.

[0076] Among them, contrastive learning is a type of self-supervised learning. Contrastive learning learns the feature representation of samples by comparing data with positive samples and negative samples in feature space.

[0077] Specifically, the present application adds contrastive learning loss to the original cross entropy loss of the pre-trained model to connect the training optimization model, uses the data in the training set and validation set after the above division as training data, and fine-tunes the pre-trained model after pruning in step S104, and trains to obtain a Chinese simplification model, so as to facilitate the subsequent simplification of Chinese sentences through the Chinese simplification model.

[0078] It should be noted that, when performing contrastive learning, the present application trains the pruned pre-trained model by adding contrastive positive and negative samples, using the vector representation of the simple sentence of the learning target, wherein the simple sentence in the original complex sentence-simple sentence pair mined in the above embodiment is used as the simple sentence of the learning target, and by learning an encoder, the features output by the model are more similar to the features of the positive sample, and at the same time, are less similar to the features of the other remaining negative samples. The specific implementation process of contrastive learning can refer to the methods in the relevant technology, which will not be repeated here.

[0079] Step S106, input the complex sentences in the test set into the Chinese sentence simplification model, generate predicted simplified sentences through the Chinese sentence simplification model, and evaluate the simplification effect of the Chinese sentence simplification model. When the simplification effect is greater than a preset threshold, simplify the Chinese sentence to be simplified through the Chinese sentence simplification model.

[0080] Specifically, the complex sentences in the test set are input into the trained Chinese sentence simplification model, the best simplified sentences are predicted and generated by the fine-tuned simplified model, and then the predicted simplified sentences output by the model are obtained. The Chinese sentence simplification effect of the model is evaluated based on the prediction results.

[0081] In one embodiment of the present application, the simplification effect of the Chinese sentence simplification model is evaluated, including first comparing the predicted simplified sentence output by the model with the standard reference simplified sentence, and then evaluating the simplification effect of the Chinese sentence simplification model through a plurality of preset evaluation indicators, wherein the plurality of evaluation indicators include: BLEU-4 indicator, Rouge-L indicator and SARI indicator.

[0082] During specific implementation, the standard reference simplified sentences may include the original simple sentences in the mined sentence pairs, or the reference sentences after the complex sentences in the test set are manually annotated. The predicted simplified sentences output by the Chinese sentence simplification model are first compared with the standard reference simplified sentences, and the models are evaluated in turn using three evaluation indicators: BLEU-4, Rouge-L, and SARI. Among them, BLEU is mainly used to evaluate the quality of machine translation, and the quality of translation is mainly measured according to the precision rate (Precision). Rouge-L is a commonly used evaluation indicator for machine translation and article abstracts, which measures the quality of translation according to the recall rate (Recall), and it mainly considers the longest common subsequence between the machine translation and the reference translation. The SARI indicator is considered from the aspect of vocabulary simplification, and is used to measure the "good or bad" degree of adding, deleting and retaining words by the simplified model. In an embodiment of the present application, the evaluation value of the Chinese sentence simplification model corresponding to each evaluation indicator can be calculated in the above manner to obtain the simplification effect of the Chinese sentence simplification model.

[0083] Furthermore, after evaluating the simplification effect of the Chinese sentence simplification model, when the simplification effect is greater than a preset threshold, the Chinese sentence to be simplified is simplified by the Chinese sentence simplification model. The preset threshold is a threshold indicating that the simplification effect of the Chinese sentence simplification model meets the minimum requirement for sentence simplification. It can be understood that when the simplification effect is greater than the preset threshold, it indicates that the Chinese sentence simplification model is applicable to the Chinese sentence simplification task, and then the actual Chinese sentence simplification task is performed by the Chinese sentence simplification model, and the Chinese sentence to be simplified is input into the Chinese sentence simplification model for simplification.

[0084] As an example, according to the current application scenario of sentence simplification, the minimum score thresholds of the BLEU-4 index, Rouge-L index and SARI index are preset. For example, when applied to text summary generation, the values ​​of each preset threshold are set higher, and when applied to a human-computer interaction system, the values ​​of each preset threshold are set slightly lower. Then, when evaluating the values ​​of each indicator of the current Chinese sentence simplification model, each indicator value is compared with the corresponding preset threshold. If they are all greater than the corresponding threshold, the Chinese sentence simplification task in the corresponding scenario can be performed through the current Chinese sentence simplification model.

[0085] Therefore, this method not only controls the simplification results in terms of length, lexical and syntactic complexity, but also improves the fidelity of the simplified sentences on the target end.

[0086] In summary, the Chinese sentence simplification method based on contrastive learning in the embodiment of the present application regards the sentence simplification task as an end-to-end conditional generation task. First, an unsupervised method is used to mine semantically similar complex sentence-simple sentence pairs, and then the editing signals and lexical syntactic complexity signals between the sentence pairs are used as supervised signals, and the encoder-decoder pre-trained model is used for fine-tuning, so that the user can control the generated simplified sentences according to different needs. In addition, the method introduces contrastive learning loss on the basis of the cross entropy loss function to increase the distance between positive and negative samples, thereby improving the fidelity of the generated simplified sentences. This method not only makes the generation results controllable according to different needs but also can improve the fidelity of the target simplified sentences.

[0087] In order to implement the above embodiment, the present application also proposes a Chinese sentence simplification system based on contrastive learning.

[0088] Figure 4 A schematic diagram of the structure of a Chinese sentence simplification system based on contrastive learning proposed in an embodiment of the present application.

[0089] like Figure 4As shown, the Chinese sentence simplification system based on contrastive learning includes a mining module 100 , a calculation module 200 , a first generation module 300 , a second generation module 400 , a training module 500 and a third generation module 600 .

[0090] The mining module 100 is used to mine multiple semantically similar complex sentence-simple sentence pairs based on an unsupervised learning method.

[0091] The calculation module 200 is used to calculate the supervision signal of each complex sentence-simple sentence pair.

[0092] The first generation module 300 is used to add each supervisory signal in the form of a character string to the starting position of the complex sentence in the corresponding sentence pair, generate a data set of complex sentence-simple sentence pairs with supervisory signals, and divide the data set containing complex sentence-simple sentence pairs with supervisory signals into a training set, a validation set and a test set according to a preset ratio.

[0093] The second generation module 400 is used to perform model pruning on the preset encoder-decoder based multilingual pre-training model to obtain a Chinese monolingual pre-training model.

[0094] The training module 500 is used to fine-tune the Chinese monolingual pre-training model based on the training set and the validation set by introducing contrastive learning loss, and jointly train a Chinese sentence simplification model.

[0095] The third generation module 600 is used to input the complex sentences in the test set into the Chinese sentence simplification model, generate predicted simplified sentences through the Chinese sentence simplification model, evaluate the simplification effect of the Chinese sentence simplification model, and when the simplification effect is greater than a preset threshold, simplify the Chinese sentence to be simplified through the Chinese sentence simplification model.

[0096] Optionally, in one embodiment of the present application, the mining module 100 is specifically used to: obtain a large amount of Chinese sentences from a preset resource library; obtain the vector of each sentence through the language tool library and create an index, and mine multiple similar candidate sentences corresponding to each sentence; perform conditional filtering on the candidate sentences corresponding to each sentence, determine the target sentence corresponding to each sentence, and generate multiple semantically similar complex sentence-simple sentence pairs.

[0097] In one embodiment of the present application, the supervisory signal includes a sentence length ratio, an edit distance ratio, a lexical complexity ratio and a syntactic tree depth ratio, and the calculation module 200 is specifically used to: calculate the ratio of the simple sentence length to the complex sentence length in each sentence pair to obtain the sentence length ratio; calculate the Levenshtein distance between the complex sentence and the simple sentence, and calculate the Levenshtein distance ratio of each editing operation to obtain the editing distance ratio, and the editing operations include: deletion, insertion and replacement; calculate the ratio of the lexical complexity of the complex sentence to the lexical complexity of the simple sentence to obtain the lexical complexity ratio, wherein the lexical complexity is represented by the word frequency of the vocabulary; obtain the syntactic tree depths of the complex sentences and the simple sentences respectively through the natural language text processing library, and calculate the ratio of the syntactic tree depth of the complex sentence to the syntactic tree depth of the simple sentence to obtain the syntactic tree depth ratio.

[0098] Optionally, in one embodiment of the present application, the first generation module 300 is specifically used to: sequentially add each supervisory signal of each sentence pair to the start position of the corresponding complex sentence in the form of setting a supervisory signal ratio after the supervisory signal name.

[0099] Optionally, in one embodiment of the present application, the second generation module 400 is specifically used to: select punctuation marks, numbers, English letters and high-frequency Chinese words commonly used in Chinese sentences as a new vocabulary; replace the original vocabulary of the multilingual pre-trained model with the new vocabulary, and update the representation parameters of the input vector and output vector of the multilingual pre-trained model to update the multilingual pre-trained model; save the new vocabulary and the updated pre-trained model.

[0100] Optionally, in one embodiment of the present application, the third generation module 600 is specifically used to: compare the predicted simplified sentence with the standard reference simplified sentence; evaluate the simplification effect of the Chinese sentence simplification model through multiple preset evaluation indicators, and the multiple evaluation indicators include: BLEU-4 indicator, Rouge-L indicator and SARI indicator.

[0101] In summary, the Chinese sentence simplification system based on contrastive learning in the embodiment of the present application regards the sentence simplification task as an end-to-end conditional generation task. First, an unsupervised method is used to mine semantically similar complex sentence-simple sentence pairs, and then the editing signals and lexical syntactic complexity signals between the sentence pairs are used as supervised signals, and the encoder-decoder pre-training model is used for fine-tuning, so that users can control the generated simplified sentences according to different needs. In addition, the system introduces contrastive learning loss on the basis of the cross entropy loss function to increase the distance between positive and negative samples, thereby improving the fidelity of the generated simplified sentences. The system not only makes the generation results controllable according to different needs but also can improve the fidelity of the target simplified sentences.

[0102] In order to implement the above embodiments, the present invention also proposes a non-temporary computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, it implements the Chinese sentence simplification method based on contrastive learning described in the embodiment of the first aspect of the present application.

[0103] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, if schematic expressions of the above terms are used in multiple embodiments or examples, it does not mean that these embodiments or examples are the same. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, unless they are contradictory.

[0104] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0105] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0106] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic devices), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways if necessary, and then stored in a computer memory.

[0107] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0108] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0109] In addition, each functional unit in each embodiment of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0110] The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present application. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A Chinese sentence simplification method based on contrastive learning, characterized in that: The following steps are involved: Mining multiple semantically similar complex sentence-simple sentence pairs based on unsupervised learning; Calculating a supervision signal for each of the complex sentence-simple sentence pairs; Adding each of the supervisory signals in the form of a character string to the starting position of the complex sentence in the corresponding sentence pair, generating a data set of complex sentence-simple sentence pairs with supervisory signals, and dividing the data set of complex sentence-simple sentence pairs with supervisory signals into a training set, a validation set, and a test set according to a preset ratio; Prune the preset encoder-decoder based multilingual pre-trained model to obtain a Chinese monolingual pre-trained model; Based on the training set and the validation set, contrastive learning loss is introduced to fine-tune the Chinese monolingual pre-training model, and a Chinese sentence simplification model is jointly trained; Inputting the complex sentences in the test set into the Chinese sentence simplification model, generating predicted simplified sentences through the Chinese sentence simplification model, and evaluating the simplification effect of the Chinese sentence simplification model, and simplifying the Chinese sentence to be simplified through the Chinese sentence simplification model when the simplification effect is greater than a preset threshold; The step of pruning the preset encoder-decoder based multilingual pre-trained model includes: Select punctuation marks, numbers, English letters and high-frequency Chinese words commonly used in Chinese sentences as new vocabulary; Replacing an original vocabulary of the multilingual pre-trained model with the new vocabulary, and updating representation parameters of input vectors and output vectors of the multilingual pre-trained model to update the multilingual pre-trained model; The new vocabulary and updated pre-trained model are saved.

2. The streamlining method according to claim 1, characterized in that: The method of mining multiple semantically similar complex sentence-simple sentence pairs based on unsupervised learning includes: Get a large amount of Chinese sentences from the preset resource library; Obtain the vector of each sentence through the language tool library and create an index to mine multiple similar candidate sentences corresponding to each sentence; Conditional filtering is performed on the candidate sentences corresponding to each sentence to determine the target sentence corresponding to each sentence, and the plurality of complex sentence-simple sentence pairs with similar semantics are generated.

3. The streamlining method according to claim 1 or 2, characterized in that: The supervisory signal includes a sentence length ratio, an edit distance ratio, a vocabulary complexity ratio, and a syntactic tree depth ratio. The supervisory signal of each complex sentence-simple sentence pair is calculated, including: Calculating the ratio of the simple sentence length to the complex sentence length in each of the sentence pairs to obtain the sentence length ratio; Calculating the Levenshtein distance between the complex sentence and the simple sentence, and calculating the Levenshtein distance ratio of each editing operation to obtain the editing distance ratio, wherein the editing operation includes: deletion, insertion and replacement; Calculating the ratio of the lexical complexity of the complex sentence to the lexical complexity of the simple sentence to obtain the lexical complexity ratio, wherein the lexical complexity is represented by the frequency of words; The syntactic tree depths of the complex sentence and the simple sentence are respectively obtained through a natural language text processing library, and the ratio of the syntactic tree depth of the complex sentence to the syntactic tree depth of the simple sentence is calculated to obtain the syntactic tree depth ratio.

4. The streamlining method according to claim 1, characterized in that: The step of adding each of the supervisory signals in the form of a character string to the starting position of the initial complex sentence in the corresponding sentence pair comprises: The supervisory signals of each of the sentence pairs are sequentially added before the starting position of the corresponding complex sentence in the form of setting the supervisory signal ratio after the supervisory signal name.

5. The streamlining method according to claim 1, characterized in that: The evaluating the simplification effect of the Chinese sentence simplification model includes: comparing the predicted condensed sentences with a standard reference condensed sentence; The simplification effect of the Chinese sentence simplification model is evaluated by using a plurality of preset evaluation indicators, wherein the plurality of evaluation indicators include: a BLEU-4 indicator, a Rouge-L indicator, and a SARI indicator.

6. A Chinese sentence simplification system based on contrastive learning, characterized in that: include: A mining module, which is used to mine multiple semantically similar complex sentence-simple sentence pairs based on unsupervised learning; A calculation module, used for calculating the supervision signal of each complex sentence-simple sentence pair; A first generating module is used to add each of the supervisory signals in the form of a character string to the starting position of the complex sentence in the corresponding sentence pair, generate a data set of complex sentence-simple sentence pairs with supervisory signals, and divide the data set of complex sentence-simple sentence pairs with supervisory signals into a training set, a validation set and a test set according to a preset ratio; The second generation module is used to perform model pruning on the preset encoder-decoder based multilingual pre-training model to obtain a Chinese monolingual pre-training model, wherein punctuation marks, numbers, English letters and high-frequency Chinese words commonly used in Chinese sentences are selected as a new vocabulary; the original vocabulary of the multilingual pre-training model is replaced with the new vocabulary, and the representation parameters of the input vector and the output vector of the multilingual pre-training model are updated to update the multilingual pre-training model; and the new vocabulary and the updated pre-training model are saved; A training module, used to introduce contrastive learning loss to fine-tune the Chinese monolingual pre-training model based on the training set and the validation set, and jointly train a Chinese sentence simplification model; The third generation module is used to input the complex sentences in the test set into the Chinese sentence simplification model, generate predicted simplified sentences through the Chinese sentence simplification model, evaluate the simplification effect of the Chinese sentence simplification model, and when the simplification effect is greater than a preset threshold, simplify the Chinese sentence to be simplified through the Chinese sentence simplification model.

7. The system according to claim 6, characterized in that The mining module is specifically used for: Get a large amount of Chinese sentences from the preset resource library; Obtain the vector of each sentence through the language tool library and create an index to mine multiple similar candidate sentences corresponding to each sentence; Conditional filtering is performed on the candidate sentences corresponding to each sentence to determine the target sentence corresponding to each sentence, and the plurality of complex sentence-simple sentence pairs with similar semantics are generated.

8. The system according to claim 6 or 7, characterized in that: The supervisory signal includes a sentence length ratio, an edit distance ratio, a vocabulary complexity ratio, and a syntax tree depth ratio. The calculation module is specifically used for: Calculating the ratio of the simple sentence length to the complex sentence length in each of the sentence pairs to obtain the sentence length ratio; Calculating the Levenshtein distance between the complex sentence and the simple sentence, and calculating the Levenshtein distance ratio of each editing operation to obtain the editing distance ratio, wherein the editing operation includes: deletion, insertion and replacement; Calculating the ratio of the lexical complexity of the complex sentence to the lexical complexity of the simple sentence to obtain the lexical complexity ratio, wherein the lexical complexity is represented by the frequency of words; The syntactic tree depths of the complex sentence and the simple sentence are respectively obtained through a natural language text processing library, and the ratio of the syntactic tree depth of the complex sentence to the syntactic tree depth of the simple sentence is calculated to obtain the syntactic tree depth ratio.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for simplification of Chinese sentences based on contrastive learning as described in any one of claims 1 to 5 is implemented.

Images