Text summarization extraction method and system

Abstract

The present disclosure relates to a text summary extraction method and system. The method comprises: selecting M sentences from a given document containing L sentences to construct N candidate summaries; concatenating each candidate summary with the given document and inputting into a PLM to obtain N output vectors; inputting the N output vectors into a text paraphrase ranking model to obtain N paraphrase probabilities; selecting the candidate summary corresponding to the highest probability from the N paraphrase probabilities as the extracted text summary of the given document. The present invention converts the summary extraction task into a text paraphrase problem between the candidate summary and the source text, narrows the training gap between the summary extraction task and the PLM, and can better tap the knowledge of the PLM to improve the model performance. Further, relevant knowledge is learned from the existing text paraphrase rich training dataset by using knowledge transfer, which assists the model to identify the candidate summary that can better paraphrase the core semantics of the document, and makes up for the problem of missing training supervision signal caused by small-scale dataset.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the field of deep learning, and in particular, to a text summarization extraction method and system. BACKGROUND

[0002] The Internet produces a large amount of data every moment, making the information overload problem increasingly serious, so it is necessary to extract the main points of various texts, and obtaining a summary of the text is an important means. Text summarization can be divided into extractive summary and generative summary according to the output type. Extractive summary needs to extract key sentences from the source document to form a summary, and the summary information is all from the original text.

[0003] In recent years, the development of pre-trained language models (PLMs) has greatly improved the performance of text summarization extraction. However, the pre-trained language model-based summary extraction requires a large amount of training data and a very long running time to obtain satisfactory performance. This is not feasible in terms of cost and time in most application scenarios.

[0004] Therefore, an improved text summarization extraction scheme is needed. SUMMARY

[0005] One of the technical problems to be solved by the present disclosure is to provide a text summarization extraction method and system. The method converts the text summarization extraction task into a text paraphrasing problem between the candidate summary and the source text, thereby narrowing the training gap between the text summarization extraction task and the pre-trained language model, so as to better exploit the knowledge already existing in the PLM to improve the model performance. Further, relevant knowledge can be learned from the existing text paraphrasing task rich training dataset, and through knowledge transfer, the model can identify the candidate summary that can better paraphrase the core semantic information of the document, thereby making up for the lack of training supervision signal caused by the small-scale dataset.

[0006] According to a first aspect of the present disclosure, a text summarization extraction method is provided, comprising: selecting M sentences from a given document containing L sentences to construct N candidate summaries, wherein M < L, L ≤ N; concatenating each candidate summary with the given document and inputting into a pre-trained language model (PLM) to obtain N output vectors; inputting the N output vectors into a text paraphrasing ranking model to obtain N paraphrasing probabilities; and selecting the candidate summary corresponding to the highest probability from the N paraphrasing probabilities as the extractive text summary of the given document.

[0007] Optionally, in the training phase of the text summarization extraction task using the text summarization extraction task dataset, each given document corresponds to a reference summary.

[0008] Optionally, the method further comprises: in the training phase, calculating a similarity score of each of the N candidate summaries with the reference summary, and setting the label of the one of the N candidate summaries with the highest similarity score with the reference summary as 1, and setting the labels of the remaining candidate summaries as 0, and defining a first loss function as a cross-entropy loss based on the labels and the obtained probability of the paraphrase; and fine-tuning the PLM and training the text paraphrase ranking model based on the first loss function.

[0009] Optionally, the method further comprises: in the training phase, calculating a similarity score of each of the N candidate summaries with the reference summary, and sorting the candidate summaries in descending order based on the similarity scores; taking the candidate summaries sorted in the front based on the similarity scores as positive samples, and taking the candidate summaries sorted in the rear as negative samples, to define a second loss function as a contrastive loss; and fine-tuning the PLM and training the text paraphrase ranking model based on the second loss function.

[0010] Optionally, the method further comprises: in the training phase, concatenating the reference summary with the given document and inputting the same into the PLM to obtain a reference summary output vector; inputting the reference summary output vector into the text paraphrase ranking model; defining a third loss function for generating a loss when the paraphrase probability of the reference summary output vector is not higher than the paraphrase probabilities of the output vectors of all candidate summaries by the ranking model; and fine-tuning the PLM and training the text paraphrase ranking model based on the third loss function.

[0011] Optionally, the method further comprises: training using a text paraphrase dataset to fine-tune the PLM and train the text paraphrase ranking model; and using the PLM and the text paraphrase ranking model trained based on the text paraphrase dataset as initial models for training the text summary extraction task using a text summary extraction task dataset.

[0012] Optionally, the method further comprises: in the training using the text paraphrase dataset, defining the probability of paraphrase p g as a positive sample, and 1-p g as a negative sample, to define a contrastive loss for fine-tuning the PLM and training the text paraphrase ranking model.

[0013] Optionally, the method further comprises: in the training phase of the text summary extraction task using the text summary extraction task dataset, using the loss function having a corresponding form when training based on the text paraphrase dataset to fine-tune the PLM and train the text paraphrase ranking model.

[0014] Optionally, selecting M sentences from the given document comprising L sentences to construct N candidate summaries comprises: selecting K sentences from the L sentences based on relevance; and selecting M sentences from the K sentences to construct N candidate summaries, wherein M < K < L.

[0015] Optionally, selecting the K sentences that are rank-significant from the L sentences comprises: feeding the L sentences into the PLM and obtaining corresponding L sentence vectors; and feeding the sentence vectors into a relevance scoring model and selecting the top K scored sentences as the K rank-significant sentences, wherein the relevance scoring model is trained in a training phase using a text summarization extraction task dataset.

[0016] According to a second aspect of the disclosure, there is provided a text summarization extraction system comprising: a storage server configured to store a pre-trained language model; and a text summarization extraction server configured to perform the method of the first aspect.

[0017] According to a third aspect of the disclosure, there is provided a computing device comprising: a processor; and a memory having stored thereon executable code which, when executed by the processor, causes the processor to perform the method of the first aspect.

[0018] According to a fourth aspect of the disclosure, there is provided a computer program product comprising executable code which, when executed by a processor of an electronic device, causes the processor to perform the method of the first aspect.

[0019] According to a fifth aspect of the disclosure, there is provided a non-transitory machine-readable storage medium having stored thereon executable code which, when executed by a processor of an electronic device, causes the processor to perform the method of the first aspect.

[0020] Thus, the present disclosure reformulates the text summarization extraction as text interpretation, which can utilize the knowledge of the PLM to facilitate the summarization extraction task. Further, the present disclosure can also relax the data requirement for achieving good performance by means of transfer learning, thereby achieving good performance of the text summarization extraction task under extremely low resources. BRIEF DESCRIPTION OF DRAWINGS

[0021] The above and other objects, features and advantages of the present disclosure will become more apparent from the following detailed description when taken in conjunction with the accompanying drawings in which like reference characters refer to like parts throughout the figures, and in which:

[0022] Figure 1 A schematic flowchart of a text summarization extraction method according to an embodiment of the present disclosure is shown.

[0023] Figure 2 The overall architecture of the text summarization extraction and the process of knowledge transfer of the present application are shown.

[0024] Figure 3 The composition diagram of the text summarization extraction system according to an embodiment of the present application is shown.

[0025] Figure 4 The structural diagram of the computing device according to an embodiment of the present application for implementing the above-mentioned text summarization extraction method is shown. DETAILED DESCRIPTION

[0026] The preferred embodiments of the present disclosure will be described in detail with reference to the drawings. Although the preferred embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to make the present disclosure more thorough and complete, and to fully convey the scope of the present disclosure to those skilled in the art.

[0027] Text summarization extraction (Extractive Summarization, hereinafter also referred to as "extractive text summary") aims to select a subset of text fragments from a document to summarize the core content of the document to construct an abstract, for example, to extract the most appropriate M sentences from a given document with L sentences and arrange them in order into an extractive text summary. It has strong practical value in a wide range of industrial applications. For example, extracting key content from massive conversations to reduce the reading pressure of customer service; extracting the core content of meetings or emails to improve reading speed, etc.

[0028] The development of large-scale pre-trained language models (PLM) has greatly improved the ability of context representation learning of vocabulary, and has achieved great success in various fields of NLP (Natural Language Processing), changing the process of model engineering. First, on a large amount of general corpus, a general PLM is obtained through some unsupervised proxy tasks. Then, in the downstream task, the general PLM fine-tunes the parameters on the supervised data, i.e., the language knowledge in the general corpus can be utilized. Such a two-stage model paradigm has been widely adopted in many practical language services. The existing advanced text summarization extraction method is also based on the above-mentioned downstream task fine-tuning paradigm based on general PLM.

[0029] However, the PLM-based text summarization method needs large-scale high-quality training data and requires a very long training time to achieve advanced performance. For example, the current mainstream text summarization dataset has about 20,000 training data sets. Most actual application scenarios do not have such a large-scale high-quality text summarization dataset, and it is not feasible to build a large-scale high-quality text summarization dataset for different application fields. Therefore, in the prior art, it is difficult to achieve satisfactory text summarization performance by directly applying the existing text summarization extraction method to a low-resource application scenario.

[0030] Therefore, the present application proposes a text summarization extraction method, which is particularly suitable for low-resource application scenarios (for example, the training data set only includes a few hundred pairs of source text and reference summary). Since only a small amount of training data is available, the present application discards the traditional sentence evaluation text summarization method and the autoregressive text summarization method, which have a higher demand for the size of the training data, and instead attempts to transform the form of text summarization.

[0031] Considering that the PLM itself is a good small sample learner, the present application redefines the text summarization extraction method to minimize the training gap with the PLM itself, thereby utilizing the knowledge inherently contained in the PLM to facilitate the summarization extraction task (it can be considered that a large amount of text knowledge is contained in the parameters of the PLM). The text summarization extraction method proposed by the present application creatively redefines text summarization as text paraphrasing. This method constructs a set of candidate summaries from the source document and distinguishes the most significant summary as the output.

[0032] In order to facilitate the understanding of the principles of the present application, the text paraphrasing method will be described first. Text paraphrasing is to express the same meaning using one sentence with another sentence. The text paraphrasing method using PLM as an encoder generally follows the way in which the PLM processes sentence pairs during the pre-training stage. The following will describe the text paraphrasing training using PLM as an encoder with mathematical expressions.

[0033] When performing text paraphrasing task training, a set of text paraphrasing sentence pairs is given wherein represents the t-th sentence pair, y t is the corresponding binary label, which is used to represent whether the two sentences are semantically equivalent, for example, y t = 1 when they are equivalent, and y t = 0 when they are not equivalent. When using BERT as the PLM, the sentence pair can be concatenated in the following way: In BERT, [CLS] and [SEP] are special markers. The [CLS] marker is placed at the beginning of the first sentence, and the representation vector C obtained by BERT can be used for subsequent classification tasks. The [SEP] marker is used to separate two input sentences. For example, if the input sentences are A and B, the [SEP] marker is added after sentences A and B to mark the beginning and end of the sentences, respectively. x in As input and x in This is mapped to a series of hidden vector representations. The hidden vector at position [CLS] is used as the context representation of the input sequence, i.e., at this position... H [s] The sentences are fed into a ranking model to compute representations of the two sentences. and The probability p of whether they are mutual interpretations eq The ranking model typically consists of a single-layer feedforward network (FFN) and a sigmoid function. The calculation process is as follows:

[0034]

[0035]

[0036] Where d represents the dimension of the PLM output representation, and W and b are learnable parameters of FFN.

[0037] This invention reformulates text summarization as a series of text interpretations between a source document and its corresponding candidate summaries. First, multiple candidate summaries are constructed. Then, the effectiveness of different candidate summaries in explaining the core meaning of the document is calculated. Finally, the candidate summary that best explains the document is selected as the extractive text summary.

[0038] Figure 1 A schematic flowchart illustrating a text summarization extraction method according to an embodiment of the present invention is shown. In the text summarization extraction method of the present invention, a PLM (e.g., BERT) and an additionally trained text paraphrase ranking model are used for text summarization extraction.

[0039] In step S110, M sentences are selected from a given document containing L sentences to construct N candidate summaries, where M... <L,L≤N。

[0040] As mentioned earlier, extractive summarization extracts key sentences from the source document to form a summary, and all summary information comes from the original text. Therefore, given a document D = {s} i |i=1...L}, where document D has L sentences, s i Let C represent the i-th sentence of D. Construct N candidate summaries of the source document D, i.e., Ci. D ={c j{ j = 1...N}. Candidate summaries consists of M sentences from D, where represents the k-th sentence in c j . In c j , the sentences are ordered by their positions in the source document D. In other words, the order of the sentences in a candidate summary will not be rearranged. M is a hyper-parameter and varies with different datasets. Obviously, M needs to be smaller than L.

[0041] For a document D with L sentences, if there is no restriction on the selected sentences, the number of all candidate summaries is equal to When M is large, is usually much larger than L. For some embodiments of the present application, for example, when M is small (e.g., a typical value of M is 1 or 2), since the number of all candidate summaries is not large, we can construct candidate summaries (in this case, ). In other embodiments, especially when M is large, it is time-consuming and computationally intensive to calculate all candidate summary-document pairs (i.e., L candidate summary-document pairs) for summary extraction. At this time, most trivial candidate summaries can be excluded in advance to reduce the magnitude of the candidate summary set C D . At this time, the step S110 of selecting M sentences from a given document containing L sentences to construct N candidate summaries can include a salient sentence extraction step, for example, can include: selecting K sentences ranked as salient from the L sentences; and selecting M sentences from the K sentences to construct N candidate summaries, where M < K < L. At this time, the number of candidate summaries changes from to The so-called salient sentence can be a sentence ranked as salient in terms of relevance to the document. In this way, the magnitude of the candidate summary set C D can be reduced by filtering out low-relevance sentences, thereby improving the efficiency of summary extraction. The selection of the K sentences ranked as salient will be described in detail in the application examples section below.

[0042] In step S120, each candidate summary is concatenated with the given document and fed into a pre-trained language model (PLM) to obtain N output vectors.

[0043] In the present application, the BERT (Bidirectional Encoder Representation from Transformers) model is also used as an example of the PLM. However, it should be understood that the scheme of the present application can also be applied to other PLMs that have been proposed or have not yet been proposed.

[0044] In BERT, concatenating two texts requires the use of special markers. Similar to the concatenation method used in the text definition task above, in this invention, each candidate summary c... j Concatenating with the source document D yields:

[0045]

[0046] Then, Feed to PLM The generated sequence is represented and the hidden vector of [CLS] is used (i.e., PLM for input). The output vector) is used as a candidate summary The representation of, that is Since there are N candidate summaries, step S120 can obtain N hidden vectors. It should be understood that the vector output by BERT is conventionally referred to as a "hidden vector" or "hidden representation" in the art, but when using the transformed vector of the PLM output cascade representation in this invention, it is still preferred to use "output vector" for description.

[0047] In step S130, the N hidden vectors are fed into the text definition ranking model. In step S140, the text definition ranking model is used to sort the vectors according to their definition probabilities, and the highest-ranked candidate summary is selected as the extractive text summary.

[0048] Here, the text paraphrasing ranking model is used for computation. The model for calculating the interpretation probability can also be composed of a single-layer feedforward network (FFN) and a sigmoid function. Here, the model for calculating the interpretation probability is called the "text interpretation" ranking model to illustrate that this model is based on N output vectors X. D Each vector in The candidate abstract referred to c j A model capable of evaluating the textual interpretation of source document D. Although this model is used for the summary extraction task of this invention, it actually performs a ranking task based on textual interpretation. The N output vectors... After being fed into the ranking model, candidate summaries c can be calculated using the same ranking model as the text paraphrasing. j The ranking score, i.e. Rank Score It can directly correspond to the interpretation probability calculated by the ranking model, or it can be another form of representation of that interpretation probability. Therefore, in C... D The ranking score of the candidate abstract is obtained, i.e. Then, simply select the highest-ranked abstract as the extractable text summary.

[0049] Figure 1 The limited text summary extraction method realizes summary extraction through the text paraphrase paradigm. In an embodiment, steps S110-S140 can be regarded as obtaining ranking scores of N candidate summaries using the fine-tuned PLM and the trained text paraphrase ranking model and selecting the candidate summary with the highest score as the inference process description of the extracted text summary obtained from the given document. In another embodiment, steps S110-S140 can also be regarded as the training process of the text summary extraction task. At this time, the text paraphrase ranking model obtains the ranking scores of N candidate summaries which can be used to construct a loss function, and by adjusting the parameters based on the loss function, the parameters of the PLM are fine-tuned and the text paraphrase ranking model is trained.

[0050] Before using the fine-tuned PLM and the trained text paraphrase ranking model for inference, the model needs to be trained for the text summary extraction task. It should be understood that fine-tuning the PLM means directly training the pre-trained model according to the requirements of the downstream task. When training the downstream task, the PLM itself has been pre-trained and contains a large amount of text knowledge in its parameters. In other words, the text summary extraction task training phase mentioned below is the training for the text summary extraction task as a downstream task, not the training for the PLM, but the text summary extraction task training still needs to fine-tune the parameters of the PLM. In addition, since an additional ranking model is needed to calculate the paraphrase probability, the parameters contained in the ranking model (for example, the learnable parameters W and b of the FFN) need to be trained during the text summary extraction task training phase.

[0051] When training the text summary extraction task, since the PLM needs to be fine-tuned and the text paraphrase ranking model needs to be trained, a loss function needs to be defined to update the parameters based on the backpropagation algorithm.

[0052] In the present application, the text summary extraction task is trained using a text summary extraction task dataset. The text summary extraction task dataset includes multiple groups of data, and each group of data includes a given document and a reference summary. By converting the text summary extraction task into a text paraphrase task, the present application can better utilize the knowledge contained in the PLM itself, so that good text extraction performance can still be achieved on a small text summary extraction task dataset.

[0053] Since the reference summary exists in the training stage, the similarity scores of the N candidate summaries with the reference summary can be calculated. A loss function can be constructed so that the ranking model can rank the candidate summaries closest to the reference summary the highest. In one embodiment, the ROUGE indicator can be used as an evaluation indicator of the text summary, i.e., the ROUGE indicator is used to give the similarity score of the candidate summary with the reference summary. The ROUGE indicator is based on recall calculation, which can measure the similarity of two texts from the perspective of words and phrases, and thus can be used to measure the matching degree of the generated result and the standard result.

[0054] In one embodiment, in order to force the model to assign most of the probability mass to the candidate summary with the highest ROUGE score, i.e., c1, the binary cross-entropy loss (BCE) can be used and the true label of the candidate summary c1 is set to 1 and the labels of the remaining candidate summaries are set to 0, as follows:

[0055]

[0056]

[0057] wherein l j represents the true label of c j .

[0058] Therefore, the text summary extraction method of the present application can further include: in the training stage, calculating the similarity scores of the N candidate summaries with the reference summary and setting the label of the candidate summary with the highest similarity score with the reference summary among the N candidate summaries to 1 and setting the labels of the remaining candidate summaries to 0, and constructing a first loss function as a cross-entropy loss based on the above label definition and the probability interpretation, and the PLM can be fine-tuned and the text interpretation ranking model can be trained based on the first loss function. At this time, the first loss function can be implemented as the cross-entropy loss L CE .

[0059] The text summary extraction method of the present application is particularly suitable for low-resource datasets. Further, when the resources are extremely low, the supervised BCE information can not be sufficient to train the model to distinguish the significant summaries among all candidate summaries. For this reason, in order to be able to apply the present application to extremely low-resource scenarios and to improve the performance of the model, it is attempted to introduce contrastive learning for self-supervision.

[0060] The contrastive learning paradigm that tries to make negative samples far away from positive samples has flexibility in defining positive and negative samples and has achieved great success in visual-lingual cross-modal model pre-training. Since the candidate summaries have been ranked, for C DAny two of the summaries can be directly utilized by the contrastive learning to train the model to always prefer the relatively salient summary by leveraging the ranking information. To this end, in a preferred embodiment, the contrastive loss can be utilized as a supplement to the BCE loss.

[0061] At this time, after the ROUGE score is calculated, the C D The candidate summaries in C

[0062] R(c m , R D ) > R(c n , R D ), C n ∈ C D , if m < n

[0063] where R(·) is the abbreviation of the ROUGE score.

[0064] Given a pair of candidate summaries c m , c n ∈ C D , where m < n, the summary c m with the higher ROUGE score can be defined as the positive sample, and the summary c n with the lower ROUGE score can be defined as the negative sample. Then, the contrastive loss of all pairs of candidate summaries in C D is calculated, i.e.:

[0065]

[0066] Here, α1 is a hyperparameter, which can be set to 0 in an embodiment.

[0067] Therefore, the text summary extraction method of the present application can further include: in the training phase, calculating the similarity scores of the N candidate summaries with the reference summary respectively and performing descending order sorting based on the high and low of the similarity scores; taking the candidate summaries sorted in the front based on the similarity scores as the positive samples and the candidate summaries sorted in the back as the negative samples to define a second loss function as the contrastive loss; and fine-tuning the PLM and training the text summary ranking model based on the second loss function. At this time, the second loss function can be implemented as the contrastive loss as above

[0068] Thus, the present application forces the model to generate a higher ranking summary probability for the candidate summary with a higher ROUGE score by introducing contrastive learning, thereby greatly improving the model performance when training with a small sample.

[0069] In a preferred embodiment, the reference summary R DIn addition to the calculation of the candidate summary ROUGE indicator and the construction of the cross-entropy loss L CE and the contrastive loss , the reference summary R D itself can also be further used to force the model to assign a higher ranking score to the reference summary R D . At this time, according to the calculation of the candidate summary ranking score, the score of the reference summary R D , i.e., s ref , is calculated. That is, the paraphrase probability of the reference summary R D is also calculated, and the loss function is constructed so that the ranking of the reference summary should be higher than that of all candidate summaries as follows:

[0070]

[0071] Here, α1 is a hyperparameter, which can be set to 0 in an embodiment.

[0072] At this time, the text summary extraction method of the present application can further include: in the training stage, concatenating the reference summary with the given document and feeding it into the PLM to obtain a reference summary output vector; feeding the reference summary output vector into the text paraphrase ranking model; defining a third loss function for generating a loss when the paraphrase probability (i.e., s ref ) of the reference summary output vector by the ranking model is not higher than the paraphrase probability of all candidate summary output vectors; and based on the third loss function, fine-tuning the PLM and training the text paraphrase ranking model. At this time, the second loss function can be implemented as the loss L ref above.

[0073] In a preferred embodiment, the training of the text summary extraction task can be performed based on the first, second and third loss functions above. At this time, the loss function L ext of the text summary extraction can be written as:

[0074]

[0075] As above, the present application narrows the training gap between the text summary extraction method and the pre-trained language model by converting the text summary extraction task into a text paraphrase task, thereby better mining relevant knowledge from the pre-trained language model to assist the text summary task. Further, the present application can also simplify the structure of the text summary model and perform model-based transfer learning to enable the model to learn to identify the candidate summary that best paraphrases the core semantics of the document from the rich training data of the text paraphrase task related to the text summary, thereby improving the performance of the text summary.

[0076] Therefore, the text summarization extraction method of the present invention may also include "pre-training" by directly using an existing and resource-rich text summarization dataset to fine-tune the PLM and train the text summarization ranking model for the text summarization task, before fine-tuning the PLM and training the text summarization ranking model on the paraphrased text summarization training set.

[0077] At this point, the text summarization extraction method of the present invention further includes: training with a text paraphrase dataset (e.g., the QQP dataset) to fine-tune the PLM and train the text paraphrase ranking model; and using the PLM and the text paraphrase ranking model trained based on the text paraphrase dataset as an initial model for training the text summarization extraction task using the text summarization extraction task dataset. It should be understood that the text paraphrase dataset includes multiple sets of text paraphrase sentence pairs, each set of text paraphrase sentence pairs may be as described above. in Let y represent the t-th sentence pair. t These are the corresponding binary tags used to indicate whether the two sentences are semantically equivalent. For example, if they are equivalent, then y t =1, y is not equivalent t =0. When training with a text definition dataset to fine-tune the PLM and train the text definition ranking model, sentence pairs can be used. It can be cascaded as follows: [SEP]. PLM x in As input and x in This is mapped to a series of hidden vector representations. The hidden vector at position [CLS] is used as the context representation of the input sequence, i.e., at this position... h [CLS] The sentences are fed into a ranking model to compute representations of the two sentences. and The probability p of whether they are mutual interpretations eq Ranking models typically consist of a single-layer feedforward network (FFN) and a sigmoid function.

[0078] Standard text parsing methods only use binary cross-entropy loss during the training phase. To better transfer knowledge from text parsing to low-resource text summarization extraction, the training paradigm for parsing needs to be consistent with that for summarization extraction. Therefore, the text summarization extraction method of this invention may further include: when training using a text parsing dataset, defining the parsing probability of the true label as p... g As positive samples, 1-p gThe PLM is fine-tuned and the text entailment ranking model is trained by defining the contrastive loss. Thus, the text summarization extraction method of the present application can further comprise: in the training phase of the text summarization extraction task using the text summarization extraction task dataset, fine-tuning the PLM and training the text entailment ranking model using the loss function with corresponding form when training based on the text entailment dataset.

[0079] Specifically, the calculation probability of the real label of the entailment dataset can be represented as p g , and p g is defined as a positive sample, and 1-p g is defined as a negative sample. Then the contrastive loss is used to maximize the real label probability. The contrastive loss L CL of the text entailment, BCE the BCE loss L para and the final loss function L CL can be defined as:

[0080] L g =max(0,(1-p g )-p BCE +α2)

[0081] L t =-l eq log p t -(1-l eq )log(l-p para )

[0082] L CL =L BCE

[0083] wherein l t represents the real label of the QQP dataset. Thus, the model trained by QQP is repeatedly used for low-resource summarization extraction to obtain better initialization parameters, in other words, the knowledge from text entailment is used to improve the performance of low-resource text summarization extraction. Then, the model is fine-tuned by the extraction loss L ext using the text summarization dataset.

[0084] As mentioned before, for a document D with L sentences, if there is no restriction on the selected sentences, the number of all candidate summaries is equal to When the value of M is large, which is usually much larger than L, so that it is time-consuming and computationally intensive to calculate all candidate summary-document pairs (i.e. candidate summary-document pairs) for summarization extraction. At this time, most trivial candidate summaries need to be excluded in advance to narrow down the candidate summary set C Dthe order of magnitude of 10,000. In one embodiment, the present application employs a simple yet effective heuristic technique that utilizes a ranking-based approach to pre-select the top K ranking significant sentences from a document and list all combinations of the K sentences according to a hyper-parameter M to form candidate summaries of the document D. Specifically, the selection of the K sentences can be based on the BERTExtSum method. This method utilizes the special token of BERT to link the sentences in a document as the input of BERT in the following form: [CLS] Sentence1 [SEP] Sentence2 [SEP] SentenceL, where Sentence1, Sentence2, …, SentenceL are the sentences in the document. The output vector of the [CLS] token is then taken as the sentence vector of the document. A feed-forward neural network is then utilized as a scoring model to score each sentence. At this time, the top K sentences with the highest scores among the L sentences can be selected as the pre-selected top K ranking significant sentences, and these K sentences are used to generate candidate summaries. At this time, the selection of the top K ranking significant sentences from the L sentences includes: feeding the L sentences into the PLM to obtain the corresponding L sentence vectors; and feeding the sentence vectors into a relevance scoring model to select the top K sentences with the highest scores as the top K ranking significant sentences, where the relevance scoring model is trained in a training phase of a text summarization extraction task using a text summarization extraction task dataset. In a low-resource setting, the feed-forward neural network for sentence screening can be trained using the same text summarization extraction dataset.

[0085] For the convenience of understanding, Figure 2 The overall architecture of the text summarization extraction and the process of knowledge transfer of the present application are shown. On the left side of Figure 2 , a typical text paraphrasing paradigm using a PLM is shown. When text paraphrasing is performed using a PLM, text A and text B are fed into the PLM in the form of [CLS] text A [SEP] text B [SEP], and the output vector h [] at the [CLS] position is obtained. It can be considered that the output vector h [] contains the relevant information of the two texts (i.e., text A and text B). The output vector h [] is then fed into a ranking model, and a paraphrasing probability p eq indicating whether text A and text B are paraphrased with each other is obtained. The value of the paraphrasing probability p eq can be regarded as an evaluation criterion for whether text A and text B are paraphrased with each other.

[0086] On the right side of Figure 2 , it is shown how the present application performs text summarization extraction based on the text paraphrasing paradigm. Similarly as on the left side, when text summarization extraction in the form of text paraphrasing is performed using a PLM, the constructed multiple candidate summaries C and the source document D are fed into the PLM, for example, also in the form of [CLS] text C [SEP] document D [SEP]. Since N candidate summaries are constructed, N output vectors at the [CLS] position can be obtained, which are denoted as h [CLS] in the figure. The N output vectors can be fed into the ranking model, and the paraphrase probability indicating whether each of the N candidate summaries C and the source document D are paraphrased with each other can be obtained (also in the form of ranking scores). The ranking scores can be regarded as the paraphrase degree of each of the N candidate summaries C to the source document D. In the reasoning process of the text summary extraction task of the present application, the candidate summary C with the highest value in the ranking scores can be directly selected as the extracted text summary.

[0087] In the training process, the present application preferably includes two training stages. The first stage is to directly use the text paraphrase database, such as QQP, to fine-tune the PLM and preliminarily train the ranking model. At this time, Figure 2 the left side can be regarded as the training process based on the text paraphrase database. After the preliminary training based on the text paraphrase database, the text summary extraction task training can be performed based on the fine-tuned PLM and the preliminarily trained ranking model (i.e., the parameters of the PLM and the ranking model are the parameters obtained after the text paraphrase training) as shown on the right side.

[0088] In one embodiment, the cross-entropy loss function L CE can be constructed based on the candidate summary with the highest ROUGE indicator and the paraphrase probability thereof. In the preferred embodiment, the contrast loss function L as described above can also be constructed as a supplement to L CE . The loss function L D that assigns a higher ranking score to the reference summary R ref can also be used, and the total loss function L ext is constructed therefrom. When only the cross-entropy loss function L CE is used for text summary extraction training, the text paraphrase task training performed in advance can also use the cross-entropy function. When the contrast loss function L is constructed as a supplement to L CE for text summary extraction training, the text paraphrase contrast loss L CL performed in advance can also be used as a supplement to the BCE loss L BCE . Thereby, the paraphrase training paradigm for knowledge transfer is consistent with the summary extraction training paradigm.

[0089] Figure 3A composition diagram of a text summarization extraction system according to an embodiment of the present application is shown. As shown in the diagram, the system 300 includes a storage server 310 and a text summarization extraction server 320. The storage server 310 is configured to store a PLM, which is implemented using BERT. The text summarization extraction server 320 is configured to implement the text summarization extraction method of the present application.

[0090] In one embodiment, the text summarization extraction server 320 can include a text summarization extraction task training server 321 and a text summarization extraction task inference server 322. The text summarization extraction task training server 321 is configured to train a model, for example, a PLM can be trained based on a loss function L para The PLM is fine-tuned and a ranking model is pre-trained. Subsequently, using a low-resource text summarization extraction dataset, further fine-tuning of the PLM and training of the ranking model are performed based on the fine-tuned PLM and the pre-trained ranking model. The text summarization extraction task inference server 322 is configured to implement model inference. The PLM and the ranking model obtained after the text summarization extraction task training are loaded on the text summarization extraction task inference server 322, and the text summarization extraction task inference server 322 is configured to perform summarization extraction on an input document. The document used for summarization extraction can be interactive information obtained from a user, and through summarization extraction, the subsequent operation of an information processing personnel or a worker is facilitated.

[0091] Figure 4 A composition diagram of a computing device according to an embodiment of the present application is shown. The computing device can be used to implement the text summarization extraction method described above.

[0092] Referring to Figure 4 , the computing device 400 includes a memory 410 and a processor 420.

[0093] The processor 420 can be a multi-core processor, or can include multiple processors. In some embodiments, the processor 420 can include a general-purpose main processor and one or more special-purpose coprocessors, such as a graphics processing unit (GPU), a digital signal processor (DSP), and the like. In some embodiments, the processor 420 can be implemented using a custom circuit, such as an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA).

[0094] The memory 410 can include various types of storage units, such as a system memory, a read-only memory (ROM), and a permanent storage device. Among them, the ROM can store static data or instructions required by the processor 420 or other modules of the computer. The permanent storage device can be a read and write storage device. The permanent storage device can be a non-volatile storage device that does not lose stored instructions and data even after the computer is powered off. In some embodiments, the permanent storage device employs a mass storage device (such as a magnetic or optical disk, a flash memory) as a permanent storage device. In some other embodiments, the permanent storage device can be a removable storage device (such as a floppy disk, an optical drive). The system memory can be a read and write storage device or a volatile read and write storage device, such as a dynamic random access memory. The system memory can store some or all of the instructions and data required by the processor during runtime. In addition, the memory 410 can include a combination of any computer readable storage media, including various types of semiconductor memory chips (DRAM, SRAM, SDRAM, flash memory, programmable read-only memory), magnetic disks and / or optical disks. In some embodiments, the memory 410 can include a read and / or write removable storage device, such as a compact disc (CD), a read-only digital versatile disc (such as DVD-ROM, double-layer DVD-ROM), a read-only Blu-ray disc, an ultra-density optical disc, a flash memory card (such as an SD card, a min SD card, a Micro-SD card, etc.), a magnetic floppy disk, etc. The computer readable storage medium does not include a carrier wave and an instantaneous electronic signal transmitted by wireless or wired transmission.

[0095] The memory 410 stores executable code, which, when processed by the processor 420, can cause the processor 420 to perform the text summarization extraction method described above.

[0096] The text summarization extraction and system according to the present application has been described in detail above with reference to the drawings. The present application proposes a new text summarization extraction method, which generalizes the generation of extractive text summaries to low-resource scenarios. The inventors found that the rich knowledge stored in PLMs can be better explored through specific patterns to facilitate downstream NLP tasks. In addition, transfer learning relaxes the assumption that training data should be independent and identical to test data, prompting the present application to use transfer learning to solve the problem of low-resource datasets. In the present application, it is attempted to reformulate text summarization extraction according to two rules: a) minimize the training paradigm gap between text summarization extraction and PLMs in order to explore the vast knowledge embedded in PLMs to improve summaries, b) the backbone architecture of the model should be simple enough to facilitate network-based transfer learning, which makes it easy to reuse models pre-trained with a large amount of training data from related fields to relax the requirements for training data and training time and achieve satisfactory performance. Textual entailment for determining whether a given pair of sentences is mutually entailed has a potential connection with text summarization extraction, which aims to form summaries that entail the gist of a document. To this end, the present application proposes a text summarization extraction method that creatively reformulates text summarization extraction as textual entailment. Given a set of candidate summaries from a source document, the method must distinguish the most salient summaries as output. Experiments show that the present application outperforms traditional ranking or autoregressive-based methods in low-resource environments.

[0097] It should be noted that the user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards of countries and regions, and provide corresponding operation portal for user to choose authorization or refusal.

[0098] In addition, the method according to the present application can also be implemented as a computer program or computer program product, which includes computer program code instructions for executing the above steps defined in the above method of the present application.

[0099] Alternatively, the present application can also be implemented as a non-transitory machine readable storage medium (or computer readable storage medium, or machine readable storage medium), which stores executable code (or computer program, or computer instruction code), when the executable code (or computer program, or computer instruction code) is executed by the processor of the electronic device (or computing device, server, etc.), the processor executes each step of the above method according to the present application.

[0100] Those skilled in the art will further appreciate that the various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the disclosure herein can be implemented as electronic hardware, computer software, or combinations of both.

[0101] The flow diagrams and block diagrams in the drawings are presented to illustrate potentially implementable systems and methods in accordance with embodiments of the application. In this regard, each block in the flow diagrams and block diagrams can represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that in some alternative implementations, the functions noted in the box can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and / or flow diagrams, and combinations thereof, can be implemented by special purpose hardware-based systems that perform the specified functions or operations, or combinations of special purpose hardware and

[0102] Embodiments of the application have been described above with the understanding that the explanations are illustrative and are not exhaustive, and are not limited to the disclosed embodiments. Many modifications and changes can occur to those skilled in the art without departing from the scope and spirit of the described embodiments. The selection of terms herein is intended to best explain the principles of the embodiments, the practical application, or the improvement over the technology in the art, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method of text summarization extraction, comprising: selecting a number of sentences from a given document comprising L a number of sentences from the given document to construct a number of candidate summaries, wherein M N M L L < N ;​​​​ concatenating each candidate summary with the given document and feeding into a pre-trained language model, PLM, to obtain N an output vector; The output vector is input into a text paraphrase ranking model to obtain a paraphrase probability. N N ​​ From the above N The candidate summary with the highest probability among the various interpretation probabilities is selected as the extractive text summary of the given document. further comprising: training using a text entailment dataset to fine-tune the PLM and train the text entailment ranking model; and using the PLM and the text entailment ranking model trained based on the text entailment dataset as initial models for training using a text summarization extraction task dataset for the text summarization extraction task; In training using the text entailment dataset, the probability of the real label is defined 1- is defined as a positive sample is defined as a negative sample, to define a contrastive loss to fine-tune the PLM and train the text entailment ranking model; during the training phase using a text summarization extraction task dataset for the text summarization extraction task, fine-tuning the PLM and training the text entailment ranking model using a loss function having a corresponding form as when trained based on a text entailment dataset.

2. The method of claim 1, during the training phase using a text summarization extraction task dataset for the text summarization extraction task, each given document corresponding to a reference summary, and the method further comprising: In the training phase, the similarity scores of the candidate summaries with the reference summary are calculated N The labels of the candidate summaries are set to 1 and 0, and a first loss function as a cross-entropy loss is defined based on the labels and the obtained paraphrase probability. N The label of the candidate summary with the highest similarity score with the reference summary is set to 1, and the labels of the remaining candidate summaries are set to 0, and a first loss function as a cross-entropy loss is defined based on the labels and the obtained paraphrase probability. based on the first loss function, fine-tuning the PLM and training the text entailment ranking model.

3. The method of claim 1, during the training phase using a text summarization extraction task dataset for the text summarization extraction task, each given document corresponding to a reference summary, and the method further comprising: In the training phase, the similarity scores of the candidate summaries to the reference summary are calculated N and ranked in descending order based on the similarity scores. defining a second loss function as a contrastive loss using candidate summaries ranked higher based on the similarity score as positive samples and candidate summaries ranked lower as negative samples; and based on the second loss function, fine-tuning the PLM and training the text entailment ranking model.

4. The method of claim 1, during the training phase using a text summarization extraction task dataset for the text summarization extraction task, each given document corresponding to a reference summary, and the method further comprising: during the training phase, concatenating the reference summary with the given document and feeding into the PLM to obtain a reference summary output vector; feeding the reference summary output vector into the text entailment ranking model; defining a third loss function to incur a loss when the ranking model’s probability of entailment for the reference summary output vector is not higher than the probability of entailment for output vectors of all candidate summaries; and based on the third loss function, fine-tuning the PLM and training the text entailment ranking model.

5. The method of claim 1, wherein, From containing L Select from a given document containing sentences M A sentence to construct N The candidate abstracts include: based on the relevance, select L one sentence from K one sentence; and From K M N M K L .​​​​​ 6. The method of claim 5, wherein, Selecting a top-ranked sentence from a plurality of sentences L includes: K ranking the plurality of sentences The L sentence is sent into the PLM, and the corresponding L sentence vector is obtained; and The sentence vectors are fed into a relevance scoring model and the top K scored sentences are selected as the ranked salient sentences K wherein the relevance scoring model is trained in a training phase using a text summarization extraction task dataset.

7. A text summarization extraction system, comprising: a storage server for storing a pre-trained language model; and a text summarization extraction server for performing the method of any one of claims 1-6.

8. A computing device, comprising: a processor; and a memory having stored thereon executable code that, when executed by the processor, causes the processor to perform the method of any one of claims 1-6.

9. A computer program product comprising executable code that, when executed by a processor of an electronic device, causes the processor to perform the method of any one of claims 1-6.

10. A non-transitory machine-readable storage medium having stored thereon executable code that, when executed by a processor of an electronic device, causes the processor to perform the method of any one of claims 1-6.

Images