Text summarization extraction method based on joint training and corresponding device

Abstract

The application provides a text summary extraction method based on joint training and a corresponding device, which are used to improve the problem of insufficient semantic correctness of the extracted summary text. The method comprises the following steps: obtaining a to-be-processed text, and performing sentence segmentation on the to-be-processed text to obtain a plurality of to-be-processed sentences; using a vector extraction layer in a summary extraction model to perform vectorization representation on the plurality of to-be-processed sentences to obtain word vectors and sentence vectors corresponding to the plurality of to-be-processed sentences; using a feature extraction layer in the summary extraction model to perform feature extraction on the word vectors and the sentence vectors corresponding to the plurality of to-be-processed sentences to obtain core feature vectors and similar feature vectors; and using a sentence extraction layer in the summary extraction model to extract the plurality of to-be-processed sentences according to the core feature vectors and the similar feature vectors to obtain a summary text corresponding to the to-be-processed text.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of natural language processing, in particular to a text summarization extraction method based on joint training and a corresponding device. BACKGROUND

[0002] Summary extraction refers to compressing a long text into a shorter text, and ensuring that the main information of the text before compression is not lost in the compressed text, so as to improve the reading efficiency and experience of users.

[0003] At present, the existing summary sentence extraction technology is mostly to regard the summary extraction task as a sequence labeling task, that is, to select appropriate sentences from a long text to join a short text set, and to take the short text set as the result of summary extraction. However, it is found in specific practice that there are many semantic problems in the summary text extracted by using the existing summary sentence extraction technology, such as: there are some unimportant summary sentences in the summary text, or the sentences in the summary text are all important sentences, but there are two or more than two semantically repeated sentences, etc. Therefore, the semantic correctness of the summary text extracted by using the existing summary sentence extraction technology is not enough. SUMMARY

[0004] The purpose of the embodiments of the present application is to provide a text summarization extraction method based on joint training and a corresponding device, which is used to improve the problem that the semantic correctness of the extracted summary text is not enough.

[0005] The embodiment of the present application provides a text summary extraction method based on joint training, comprising: obtaining a to-be-processed text, and performing sentence division on the to-be-processed text to obtain a plurality of to-be-processed sentences; performing vectorization representation on the plurality of to-be-processed sentences by using a vector extraction layer in a summary extraction model to obtain word vectors and sentence vectors corresponding to the plurality of to-be-processed sentences, wherein the summary extraction model is obtained through joint training of a sentence recognition task of importance degree and a sentence matching task of similarity degree; performing feature extraction on the word vectors and the sentence vectors corresponding to the plurality of to-be-processed sentences by using a feature extraction layer in the summary extraction model to obtain core feature vectors and similar feature vectors, wherein the core feature vectors represent the importance degree of the to-be-processed sentences, and the similar feature vectors represent the similarity degree between the plurality of to-be-processed sentences; and extracting the plurality of to-be-processed sentences according to the core feature vectors and the similar feature vectors by using a sentence extraction layer in the summary extraction model to obtain a summary text corresponding to the to-be-processed text. In the above implementation process, the summary extraction model obtained through the joint training of the sentence recognition task of importance degree and the sentence matching task of similarity degree is used to extract the plurality of to-be-processed sentences, which can effectively extract the core feature vectors representing the importance degree of the to-be-processed sentences and the similar feature vectors representing the similarity degree between the plurality of to-be-processed sentences, and extract the plurality of to-be-processed sentences according to the core feature vectors and the similar feature vectors, which greatly reduces the probability of unimportant summary sentences and similar summary sentences in the summary text, thereby effectively improving the semantic correctness of the extracted summary text.

[0006] Optionally, in the embodiment of the present application, the vectorization representation of the plurality of to-be-processed sentences comprises: performing word segmentation and vectorization on each to-be-processed sentence in the plurality of to-be-processed sentences to obtain a plurality of word vectors corresponding to each to-be-processed sentence; and determining a preset word vector in the plurality of word vectors corresponding to each to-be-processed sentence as a sentence vector of each to-be-processed sentence. In the above implementation process, by determining the preset word vector in the plurality of word vectors corresponding to each to-be-processed sentence as the sentence vector of each to-be-processed sentence, each to-be-processed sentence can be distinguished by the preset word vector, the importance degree of each to-be-processed sentence and the similarity degree between sentences can be effectively identified, the probability of unimportant summary sentences and similar summary sentences in the summary text is greatly reduced, and thus the semantic correctness of the extracted summary text is effectively improved.

[0007] Optionally, in the embodiment of the present application, the feature extraction layer in the summary extraction model is used to perform feature extraction on the word vectors and sentence vectors corresponding to the plurality of to-be-processed sentences to obtain core feature vectors and similar feature vectors, including: performing feature extraction on the word vectors and sentence vectors corresponding to each to-be-processed sentence in the plurality of to-be-processed sentences to obtain a core feature vector of each to-be-processed sentence; calculating the similarity between each to-be-processed sentence in the plurality of to-be-processed sentences and all to-be-processed sentences in the plurality of to-be-processed sentences to obtain a similar feature vector of each to-be-processed sentence. In the above implementation process, the core feature vector and the similar feature vector are obtained through feature extraction, and the plurality of to-be-processed sentences are extracted according to the core feature vector and the similar feature vector, which greatly reduces the probability of unimportant summary sentences and similar summary sentences in the summary text, thereby effectively improving the semantic correctness of the extracted summary text.

[0008] Optionally, in the embodiment of the present application, the sentence extraction layer includes: a Transformer layer and a full connection layer; and the plurality of to-be-processed sentences are extracted according to the core feature vector and the similar feature vector, including: performing attention calculation on the core feature vector and the similar feature vector using the Transformer layer to obtain an attention feature vector, and extracting the plurality of to-be-processed sentences according to the attention feature vector using the full connection layer; or performing attention calculation on the core feature vector using the Transformer layer to obtain an attention feature vector, and extracting the plurality of to-be-processed sentences according to the attention feature vector and the similar feature vector using the full connection layer. In the above implementation process, the Transformer layer is used for attention calculation, and the full connection layer is used for summary extraction, so that the summary extraction model pays more attention to reducing the probability of unimportant summary sentences and similar summary sentences in the summary text, thereby effectively improving the semantic correctness of the extracted summary text.

[0009] Optionally, in the embodiment of the present application, before the plurality of to-be-processed sentences are vectorized using the vector extraction layer in the summary extraction model, the method further includes: obtaining a plurality of sample texts, a plurality of core summary labels, and a plurality of similarity labels, wherein the core summary label represents the importance of each sample sentence in the sample text in the summary text corresponding to the sample text, and the similarity label represents the similarity of each sample sentence in the sample text to all sample sentences in the sample text; and performing multi-task joint training on the summary extraction network using the plurality of sample texts, the plurality of core summary labels, and the plurality of similarity labels to obtain the summary extraction model. In the above implementation process, the summary extraction network is trained using the plurality of sample texts, the plurality of core summary labels, and the plurality of similarity labels, and the multi-task includes a sentence recognition task of importance and a sentence matching task of similarity, thereby greatly reducing the probability of unimportant summary sentences and similar summary sentences in the summary text, and effectively improving the semantic correctness of the extracted summary text.

[0010] Optionally, in the embodiment of the present application, the multi-task joint training of the summary extraction network using the plurality of sample texts, the plurality of core summary labels, and the plurality of similarity labels includes: using the vector extraction layer in the summary extraction network to vectorize the plurality of sample sentences corresponding to the sample texts to obtain word vectors and sentence vectors corresponding to the plurality of sample sentences; using the feature extraction layer in the summary extraction network to extract features from the word vectors and the sentence vectors corresponding to the plurality of sample sentences to obtain core prediction vectors and similarity prediction vectors; calculating a core cross-entropy loss value between the core prediction vectors and the core summary labels, and calculating a similarity distribution loss value between the similarity prediction vectors and the similarity labels; calculating a total loss value according to the core cross-entropy loss value and the similarity distribution loss value; and performing joint training of the summary extraction network according to the total loss value for the sentence recognition task of importance and the sentence matching task of similarity. In the above implementation process, the total loss value is calculated according to the core cross-entropy loss value and the similarity distribution loss value, and the summary extraction network is trained according to the total loss value for the sentence recognition task of importance and the sentence matching task of similarity, thereby greatly reducing the probability of unimportant summary sentences and similar summary sentences in the summary text, and effectively improving the semantic correctness of the extracted summary text.

[0011] Optionally, in the embodiment of the present application, the plurality of sample sentences comprises: a first sample sentence and a second sample sentence; the similarity distribution loss value between the similarity prediction vector and the similarity label is calculated, comprising: for each similarity prediction vector in the plurality of similarity prediction vectors, calculating the cosine similarity between each similarity prediction vector and all similarity prediction vectors in the plurality of similarity prediction vectors to obtain a similarity matrix composed of a plurality of cosine similarities; if it is determined according to the similarity label that the first sample sentence and the second sample sentence are similar, then the first sample loss value is calculated according to the formula If it is determined according to the similarity label that the first sample sentence and the second sample sentence are not similar, then the second sample loss value is calculated according to the formula If it is determined according to the similarity label that the first sample sentence and the second sample sentence are not similar, then the second sample loss value is calculated according to the formula The first sample loss value, the second sample loss value, the number of similar sample sentences in the similarity label and the number of dissimilar sample sentences in the similarity label are statistically calculated to obtain the similarity distribution loss value; wherein Ew represents the cosine similarity in the similarity matrix, m represents the hyperparameter value, Loss2 represents the similarity distribution loss value, L + Loss1 represents the first sample loss value, L - Loss2 represents the second sample loss value, sum(S + ) represents the number of similar sample sentences in the similarity label, and sum(S - ) represents the number of dissimilar sample sentences in the similarity label.

[0012] The embodiment of the present application also provides a text summary extraction device based on joint training, comprising: a processed text acquisition module for acquiring a to-be-processed text and performing sentence division on the to-be-processed text to obtain a plurality of to-be-processed sentences; a sentence vector representation module for using a vector extraction layer in a summary extraction model to perform vectorization representation on the plurality of to-be-processed sentences to obtain word vectors and sentence vectors corresponding to the plurality of to-be-processed sentences, the summary extraction model being obtained through joint training of a sentence importance recognition task and a sentence similarity matching task; a feature vector acquisition module for using a feature extraction layer in the summary extraction model to perform feature extraction on the word vectors and the sentence vectors corresponding to the plurality of to-be-processed sentences to obtain core feature vectors and similarity feature vectors, the core feature vectors representing the importance of the to-be-processed sentences, and the similarity feature vectors representing the similarity between the plurality of to-be-processed sentences; and a summary text acquisition module for using a sentence extraction layer in the summary extraction model to extract the plurality of to-be-processed sentences according to the core feature vectors and the similarity feature vectors to obtain a summary text corresponding to the to-be-processed text.

[0013] Optionally, in the embodiment of the present application, the sentence vector representation module comprises: a word vector obtaining module, configured to perform word segmentation and vectorization on each of the plurality of to-be-processed sentences to obtain a plurality of word vectors corresponding to each to-be-processed sentence; and a sentence vector determining module, configured to determine a preset word vector in the plurality of word vectors corresponding to each to-be-processed sentence as a sentence vector of each to-be-processed sentence.

[0014] Optionally, in the embodiment of the present application, the feature vector obtaining module comprises: a core feature obtaining module, configured to perform feature extraction on the word vector and the sentence vector corresponding to each of the plurality of to-be-processed sentences to obtain a core feature vector of each to-be-processed sentence; and a similar feature obtaining module, configured to calculate a similarity between each of the plurality of to-be-processed sentences and all of the plurality of to-be-processed sentences to obtain a similar feature vector of each to-be-processed sentence.

[0015] Optionally, in the embodiment of the present application, the sentence extraction layer comprises: a Transformer layer and a fully connected layer; and the abstract text obtaining module comprises: a first abstract extraction module, configured to perform attention calculation on the core feature vector and the similar feature vector using the Transformer layer to obtain an attention feature vector, and extract the plurality of to-be-processed sentences according to the attention feature vector using the fully connected layer; or a second abstract extraction module, configured to perform attention calculation on the core feature vector using the Transformer layer to obtain an attention feature vector, and extract the plurality of to-be-processed sentences according to the attention feature vector and the similar feature vector using the fully connected layer.

[0016] Optionally, in the embodiment of the present application, the text abstract extraction device based on joint training further comprises: a text label obtaining module, configured to obtain a plurality of sample texts, a plurality of core abstract labels and a plurality of similarity labels, the core abstract label representing an importance degree of each sample sentence corresponding to the sample text in the abstract text corresponding to the sample text, and the similarity label representing a similarity degree between each sample sentence in the sample text and all sample sentences in the sample text; and a model joint training module, configured to perform multi-task joint training on the abstract extraction network using the plurality of sample texts, the plurality of core abstract labels and the plurality of similarity labels to obtain the abstract extraction model.

[0017] Optionally, in the embodiment of the present application, the model joint training module comprises: a word and sentence vector obtaining module, configured to use a vector extraction layer in the summary extraction network to perform vector representation on a plurality of sample sentences corresponding to a sample text, and obtain word vectors and sentence vectors corresponding to the plurality of sample sentences; a prediction vector obtaining module, configured to use a feature extraction layer in the summary extraction network to perform feature extraction on the word vectors and the sentence vectors corresponding to the plurality of sample sentences, and obtain core prediction vectors and similarity prediction vectors; a model loss calculation module, configured to calculate a core cross-entropy loss value between the core prediction vectors and core summary labels, and calculate a similarity distribution loss value between the similarity prediction vectors and similarity labels; a total loss value calculation module, configured to calculate a total loss value according to the core cross-entropy loss value and the similarity distribution loss value; and a task joint training module, configured to perform joint training of the summary extraction network according to the total loss value for the sentence importance recognition task and the sentence similarity matching task.

[0018] The embodiment of the present application also provides an electronic device, comprising a processor and a memory, the memory storing machine readable instructions executable by the processor, and the machine readable instructions are executed by the processor to perform the method as described above.

[0019] The embodiment of the present application also provides a computer readable storage medium, the computer readable storage medium storing a computer program, and the computer program is executed by the processor to perform the method as described above. BRIEF DESCRIPTION OF DRAWINGS

[0020] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments of the present application. It should be understood that the following drawings only show some embodiments of the present application, and therefore should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can also be obtained without creative labor on the basis of these drawings.

[0021] Figure 1 The embodiment of the present application provides a flowchart of the training phase of the summary extraction model;

[0022] Figure 2 The embodiment of the present application provides a schematic diagram of the training process and network structure of the summary extraction network;

[0023] Figure 3 The embodiment of the present application provides a flowchart of the text summary extraction method based on joint training in the inference phase of the summary extraction model;

[0024] Figure 4 The embodiment of the present application provides a network structure schematic diagram of the first inference process of the summary extraction model;

[0025] Figure 5 A network structure schematic diagram of a second inference process of the abstract extraction model provided by the embodiment of the application is shown.

[0026] Figure 6 A structure schematic diagram of the text abstract extraction device based on joint training provided by the embodiment of the application is shown. DETAILED DESCRIPTION

[0027] The technical solutions in the embodiments of the application will be described clearly and completely below with reference to the drawings in the embodiments of the application.

[0028] Before introducing the text abstract extraction method based on joint training provided by the embodiments of the application, some concepts involved in the embodiments of the application are introduced first.

[0029] Joint training (Joint Training), also known as joint learning (Joint Learning), refers to joint training of neural network models such as feature extraction networks, detection networks and classification attention networks using a multi-task learning framework, that is, training or simultaneously training the neural network models such as feature extraction networks, detection networks and classification attention networks using a multi-task learning framework in sequence; wherein, the multi-task learning framework that can be used includes but is not limited to: Multi-gate Mixture-of-Experts (MMoE) framework and the like.

[0030] Bidirectional Encoder Representations from Transformers (BERT), also translated as Transformer-based bidirectional encoder representation, or translated as bidirectional encoder representation from Transformer, also known as bidirectional self-attention layer encoding and decoding, is usually simply referred to as BERT. BERT is a language representation model, and the design purpose of the model is to obtain the semantic representation of text containing rich semantic information by training a large amount of unlabeled corpus, and then fine-tune the semantic representation of text in a specific natural language processing task, and finally apply it to the natural language processing task. When using bidirectional encoding representation encoding layer for operation, that is, using BERT model for encoding, decoding, self-attention mechanism and transformer operation, etc.

[0031] It should be noted that the text summary extraction method based on joint training provided by the embodiments of the present application can be executed by an electronic device, and the electronic device refers to a device terminal or a server having a computer program execution function, for example, a smart phone, a personal computer (PC), a tablet computer, a personal digital assistant (PDA), a mobile Internet device (MID), or the like; and the server, for example, an x86 server and a non-x86 server, the non-x86 server including a mainframe, a minicomputer, and a UNIX server.

[0032] The application scenario to which the text summary extraction method based on joint training is applicable will be introduced below, and the application scenario includes but is not limited to: using the text summary extraction method based on joint training to extract summaries from the to-be-processed text, which can effectively extract the core feature vector representing the importance of the to-be-processed sentence and the similarity feature vector representing the similarity between multiple to-be-processed sentences, thereby reducing the probability of unimportant summary sentences and similar summary sentences in the summary text, and effectively improving the semantic correctness of the extracted summary text.

[0033] The main idea of the text summary extraction method based on joint training provided by the embodiments of the present application is that the summary extraction model obtained through the joint training of the sentence importance recognition task and the sentence similarity matching task is used to extract multiple to-be-processed sentences, which can effectively extract the core feature vector representing the importance of the to-be-processed sentence and the similarity feature vector representing the similarity between multiple to-be-processed sentences, and extract multiple to-be-processed sentences according to the core feature vector and the similarity feature vector, thereby greatly reducing the probability of unimportant summary sentences and similar summary sentences in the summary text, and effectively improving the semantic correctness of the extracted summary text.

[0034] Please refer to Figure 1 The flowchart of the summary extraction model training stage provided by the embodiments of the present application is shown; it can be understood that the text summary extraction method based on joint training can include a summary extraction model training stage and a summary extraction model inference stage, which will be introduced below, and first, the summary extraction model training stage will be introduced, and the training stage of the summary extraction model can include:

[0035] Step S110: obtaining multiple sample texts, multiple core summary labels, and multiple similarity labels.

[0036] The core summary label refers to a label representing the importance of each sample sentence in the sample text in the summary text corresponding to the sample text. For example, if a sample sentence in the sample text is "Today the weather is really good", and the sample sentence is not included in the summary text corresponding to the sample text, the core summary label of the sample text can be set to 0%. If a sample sentence in the sample text is "Today I did something happy", and the sample sentence is included in the summary text corresponding to the sample text, the core summary label of the sample text can be set to 99%.

[0037] The similarity label refers to a label representing the similarity between each sample sentence in the sample text and all sample sentences in the sample text. For example, if two sample sentences in the sample text are "Today the weather is really good" and "Today the weather is not bad", the similarity label between the two sample sentences can be set to 99% because the two sample sentences are very similar in semantics. If two sample sentences in the sample text are "Today it rained and the weather was terrible" and "Today I did something happy", the similarity label between the two sample sentences can be set to 0% because the two sample sentences are not similar in semantics.

[0038] It should be noted that the plurality of sample texts, the plurality of core summary labels and the plurality of similarity labels can be obtained separately. For example, a plurality of sample texts can be collected manually, and each sample text can be segmented according to punctuation marks or pre-set symbols to obtain a plurality of sample sentences. Then, each sample sentence in the plurality of sample sentences can be labeled according to the importance (i.e., whether it is in the summary text or the probability of being selected as a summary sentence) to obtain the core summary label of each sample sentence in the plurality of sample sentences. The semantic similarity between any two sample sentences in the plurality of sample sentences can be calculated to obtain the plurality of similarity labels corresponding to each sample text. Thus, the plurality of core summary labels and the plurality of similarity labels can be obtained. Of course, the plurality of sample texts, the plurality of core summary labels and the plurality of similarity labels can be packaged as a training data set to obtain them together. Here, the training data set is obtained together as an example.

[0039] There are many implementations of the above step S110, including but not limited to: a first implementation, receiving a training data set sent by another terminal device, and storing the training data set in a file system, a database or a mobile storage device; a second implementation, obtaining a pre-stored training data set, for example, obtaining the training data set from the file system, or obtaining the training data set from the database, or obtaining the training data set from the mobile storage device; a third implementation, using a browser or other software to obtain the training data set on the Internet, or using other application programs to access the Internet to obtain the training data set.

[0040] After step S110, step S120 is performed: multi-task joint training of the summary extraction network is performed using the plurality of sample texts, the plurality of core summary labels and the plurality of similarity labels, to obtain a summary extraction model.

[0041] Please refer to Figure 2 The training process and network structure of the summary extraction network provided by the embodiments of the present application are shown in the figure; it should be noted that the solid line in the figure represents the network structure of the neural network, and the network structure of the summary extraction network can include: a vector extraction layer, a feature extraction layer and a sentence extraction layer; however, the dashed line in the figure represents the data required by the neural network, the data generated by the network or the data generated in the intermediate process; wherein the training data set required includes: sample texts, core summary labels and similarity labels; the data generated by the summary extraction network includes: core predicted vectors, similarity predicted vectors and predicted summary texts; the data generated in the intermediate process includes: core cross-entropy loss values, similarity distribution loss values and total loss values. The summary extraction network refers to a neural network that has not been trained for summary extraction of text content; the summary extraction model refers to a neural network that has been trained for summary extraction of text content. The implementation of the above step S120 can include:

[0042] Step S121: using the vector extraction layer in the summary extraction network to vectorize the plurality of sample sentences corresponding to the sample texts, to obtain word vectors and sentence vectors corresponding to the plurality of sample sentences.

[0043] The implementation of the above step S121, for example: using a generative pre-training (GPT) model, a GPT-2 model, a BERT model or an ELMo model and the like language preprocessing model as the vector extraction layer in the summary extraction network, and using the vector extraction layer to vectorize the plurality of sample sentences corresponding to the sample texts, to obtain word vectors and sentence vectors corresponding to the plurality of sample sentences.

[0044] Step S122: using the feature extraction layer in the summary extraction network to extract features from the word vectors and sentence vectors corresponding to the plurality of sample sentences, to obtain core predicted vectors and similarity predicted vectors.

[0045] The implementation of the above step S122, for example: using a bidirectional encoding representation encoder BERT model or a RoBERTa model as the feature extraction layer in the summary extraction network, and using the feature extraction layer to extract features from the word vectors and sentence vectors corresponding to the plurality of sample sentences, to obtain core predicted vectors and similarity predicted vectors.

[0046] Step S123: calculate the core cross-entropy loss value between the core prediction vector and the core summary label, and calculate the similarity distribution loss value between the similarity prediction vector and the similarity label.

[0047] The implementation of calculating the core cross-entropy loss value between the core prediction vector and the core summary label in the above step S123 is, for example, according to the formula The core cross-entropy loss value between the core prediction vector and the core summary label is calculated; wherein Loss1 represents the core cross-entropy loss value; N is the number of multiple sample sentences; y i The core summary label value representing the i-th sample sentence; p i The core prediction vector representing the semantic importance of the i-th sample sentence.

[0048] The implementation of calculating the similarity distribution loss value between the similarity prediction vector and the similarity label in the above step S123 is, for example, for each similarity prediction vector in the multiple similarity prediction vectors, according to the formula Calculate the cosine similarity between each similarity prediction vector and all similarity prediction vectors in the multiple similarity prediction vectors to obtain a similarity matrix composed of multiple cosine similarities; wherein cos(θ) represents the cosine similarity between each similarity prediction vector and all similarity prediction vectors in the multiple similarity prediction vectors, a represents one of the multiple similarity prediction vectors, and b represents another of the multiple similarity prediction vectors. In the process of calculating the similarity matrix, only the upper triangular matrix needs to be calculated, for example, assuming that there are 4 sample sentences, which are represented as S1, S2, S3 and S4 respectively, then only the cosine similarities between S1 and S2, S3, S4, between S2 and S3, S4, and between S3 and S4 need to be calculated.

[0049] Assuming that the multiple sample sentences include a first sample sentence and a second sample sentence, the first sample sentence and the second sample sentence may be similar or not similar, and the loss values in the two cases are discussed respectively. After obtaining the similarity matrix, the similarity distribution loss value is calculated according to the similarity label (the similarity label includes similar sample sentences and dissimilar sample sentences); for example, if it is determined according to the similarity label that the first sample sentence and the second sample sentence are similar, then according to the formula The first sample loss value is calculated; similarly, if it is determined according to the similarity label that the first sample sentence and the second sample sentence are not similar, then according to the formula The second sample loss value is calculated; wherein L + represents the first sample loss value, L -Loss2 represents the second sample loss value, Ew represents the cosine similarity in the similarity matrix, m represents a hyperparameter value, and the value range of the hyperparameter value can be set to 0.35-0.8. According to the formula The first sample loss value, the second sample loss value, the number of similar sample sentences in the similarity label, and the number of dissimilar sample sentences are calculated to obtain a similarity distribution loss value; wherein Loss2 represents the similarity distribution loss value, L + Loss1 represents the first sample loss value, L - Loss2 represents the second sample loss value, sum(S + ) represents the number of similar sample sentences in the similarity label, and sum(S - ) represents the number of dissimilar sample sentences in the similarity label.

[0050] Step S124: using the sentence extraction layer in the summary extraction network, the core prediction vector and the similarity prediction vector are used to extract a plurality of sample sentences to obtain a predicted summary text corresponding to the sample text.

[0051] The sentence extraction layer in the summary extraction network can include a Transformer layer and a fully connected layer, and the Transformer layer and the fully connected layer have different calculation orders for the core prediction vector and the similarity prediction vector, and have different summary extraction methods.

[0052] There are many implementation manners of the above step S124, including but not limited to: a first implementation manner, in which attention calculation is performed on the core feature vector and the similarity feature vector first, and then summary extraction is performed, which can include: using the Transformer layer to perform attention calculation on the core feature vector and the similarity feature vector to obtain an attention feature vector; using the fully connected layer to extract a plurality of to-be-processed sentences according to the attention feature vector to obtain a summary text corresponding to the to-be-processed text. A second implementation manner, in which attention calculation is performed on the core feature vector and the similarity feature vector first, and then summary extraction is performed, which can include: using the Transformer layer to perform attention calculation on the core feature vector to obtain an attention feature vector. Using the fully connected layer to extract a plurality of to-be-processed sentences according to the attention feature vector and the similarity feature vector to obtain a summary text corresponding to the to-be-processed text.

[0053] Step S125: calculating a total loss value between the predicted summary text and the true summary text of the sample text according to the core cross-entropy loss value and the similarity distribution loss value.

[0054] An implementation of the step S125 is, for example, calculating the core cross-entropy loss value and the similarity distribution loss value according to the formula Loss = Loss1 + £ x Loss2 to obtain the total loss value between the predicted summary text and the true summary text of the sample text; wherein Loss represents the total loss value between the predicted summary text and the true summary text of the sample text, Loss1 represents the core cross-entropy loss value, Loss2 represents the similarity distribution loss value, and £ represents a hyperparameter that adjusts the weight between the core cross-entropy loss and the similarity distribution loss.

[0055] Step S126: Joint training of the sentence importance recognition task and the sentence similarity matching task of the summary extraction network according to the total loss value to obtain a summary extraction model.

[0056] An implementation of the step S126 is, for example, using a multi-task learning framework to perform joint training (Joint Training) of the sentence importance recognition task and the sentence similarity matching task of the summary extraction network according to the total loss value to obtain a summary extraction model; wherein the multi-task learning framework that can be used includes but is not limited to a Multi-gate Mixture-of-Experts (MMoE) framework and the like.

[0057] In the above implementation process, first, the vector extraction layer in the summary extraction model is used to vectorize the multiple to-be-processed sentences to obtain the word vectors and sentence vectors corresponding to the multiple to-be-processed sentences, and the feature extraction layer in the summary extraction model is used to extract features from the word vectors and sentence vectors corresponding to the multiple to-be-processed sentences to obtain core feature vectors and similarity feature vectors, and then the multiple to-be-processed sentences are extracted according to the core feature vectors and the similarity feature vectors to obtain the summary text corresponding to the to-be-processed text. That is, the summary extraction model obtained by joint training of the sentence importance recognition task and the sentence similarity matching task is used to extract the multiple to-be-processed sentences, which can effectively extract core feature vectors representing the importance of the to-be-processed sentences and similarity feature vectors representing the similarity between the multiple to-be-processed sentences, and extract the multiple to-be-processed sentences according to the core feature vectors and the similarity feature vectors, greatly reducing the probability of unimportant summary sentences and similar summary sentences in the summary text, thereby effectively improving the semantic correctness of the extracted summary text.

[0058] See Figure 3The flowchart of the joint training based text abstract extraction method in the inference stage of the abstract extraction model provided by the embodiments of the application is shown. The joint training based text abstract extraction method in the inference stage of the abstract extraction model is introduced as follows, which can include:

[0059] Step S210: obtaining the to-be-processed text and performing sentence segmentation on the to-be-processed text to obtain a plurality of to-be-processed sentences.

[0060] There are many ways to obtain the to-be-processed text in step S210, including but not limited to: a first obtaining method, receiving the to-be-processed text sent by other terminal devices and storing the to-be-processed text in a file system, a database or a mobile storage device; a second obtaining method, obtaining the pre-stored to-be-processed text, for example, obtaining the to-be-processed text from the file system, or obtaining the to-be-processed text from the database, or obtaining the to-be-processed text from the mobile storage device; a third obtaining method, using a browser or other software to obtain the to-be-processed text on the Internet, or using other application programs to access the Internet to obtain the to-be-processed text.

[0061] After step S210, step S220 is performed: using the vector extraction layer in the abstract extraction model to perform vector representation on the plurality of to-be-processed sentences to obtain the word vectors and sentence vectors corresponding to the plurality of to-be-processed sentences, and the abstract extraction model is obtained through the joint training of the sentence recognition task of importance and the sentence matching task of similarity.

[0062] The implementation of step S220 can include:

[0063] Step S221: performing word segmentation and vectorization on each to-be-processed sentence in the plurality of to-be-processed sentences to obtain a plurality of word vectors corresponding to each to-be-processed sentence.

[0064] The implementation of the above step S221 is, for example, Tokenization of each of the plurality of to-be-processed sentences using a mechanical word segmentation method (i.e., based on a dictionary), a statistical-based word segmentation method, or a grammar and rule-based word segmentation method to obtain a plurality of words of each to-be-processed sentence; wherein the mechanical word segmentation method is, for example, a dictionary-based forward maximum matching method, a reverse maximum matching method, and a minimum segmentation method, and the statistical-based method is, for example, a Term Frequency-Inverse Document Frequency (TF-IDF) tokenizer, a Hidden Markov Model (HMM) method, an N-gram method, and a conditional random field method, etc. Then, a language preprocessing model such as a Generative Pre-Training (GPT) model, a GPT-2 model, a BERT model, or an ELMo model, etc. is used as a vector extraction layer in the abstract extraction network, and the vector extraction layer is used to vectorize the plurality of words of each to-be-processed sentence to obtain Token Embeddings vectors, position embedding vectors, and / or Reverse-Position Embedding vectors of each word. Finally, the Token Embeddings vectors, the position embedding vectors, and / or the Reverse-Position Embedding vectors of each word are added and summed or concatenated and fused to obtain word vectors of each word. Each to-be-processed sentence is processed as described above, and a plurality of word vectors corresponding to each to-be-processed sentence can be obtained.

[0065] Step S222: determining a preset word vector in the plurality of word vectors corresponding to each to-be-processed sentence as a sentence vector of each to-be-processed sentence.

[0066] An implementation example of the above-mentioned step S222 is as follows: in the above-mentioned process of sentence segmentation of the text to be processed, the text to be processed may also be preprocessed, and the preprocessing here may be to use [CLS] and [SEP] to distinguish each sentence to be processed; for example: assuming that the sentence to be processed is "It is thundering and raining today, and the weather is very bad;", then the text to be processed after preprocessing using [CLS] and [SEP] is "[CLS]It is thundering and raining today, [SEP][CLS]The weather is very bad; [SEP]". Then, the preprocessed text to be processed is tokenized and vectorized to obtain a token embedding vector of "[CLS]" in each sentence; finally, the token embedding vector of "[CLS]" in the multiple word vectors corresponding to each sentence to be processed is determined as the sentence vector of each sentence to be processed.

[0067] After step S220, execute step S230: use the feature extraction layer in the summary extraction model to perform feature extraction on the word vectors and sentence vectors corresponding to the multiple sentences to be processed to obtain core feature vectors and similar feature vectors. The core feature vector represents the importance of the sentence to be processed, and the similar feature vector represents the similarity between the multiple sentences to be processed.

[0068] Among them, the above-mentioned feature extraction layer can adopt a pre-trained language model such as a bidirectional encoding representation encoder BERT model or a RoBERTa model.

[0069] The implementation method of the above-mentioned step S230 may include: using a pre-trained language model such as a bidirectional encoding representation encoder BERT model or a RoBERTa model as a feature extraction layer in the summary extraction model, and using the feature extraction layer to extract features from the word vector and sentence vector corresponding to each of the multiple sentences to be processed, to obtain the core feature vector of each sentence to be processed. Then, using a neural network model such as a VGG model, a Resnet model, a Wide Resnet model and an Inception model to extract the feature vector of each of the multiple sentences to be processed, and calculating the similarity between the feature vector of each of the multiple sentences to be processed and the feature vector of all the sentences to be processed in the multiple sentences to be processed based on a quantitative similarity index, to obtain a similar feature vector of each sentence to be processed; wherein, the quantitative similarity index that can be used is, for example, cosine distance, Euclidean distance, Hamming distance or information entropy, etc.

[0070] After step S230, step S240 is performed: using a sentence extraction layer in the summary extraction model, the plurality of to-be-processed sentences are extracted according to the core feature vector and the similar feature vector, and a summary text corresponding to the to-be-processed text is obtained.

[0071] The sentence extraction layer can include a Transformer layer and a full connection layer. The Transformer layer can include one or two Transformer layers.

[0072] There are many embodiments of step S240, including but not limited to the following embodiments:

[0073] The first embodiment is shown in Figure 4 The first inference process of the summary extraction model provided by the embodiment of the application is shown in the network structure diagram. The core feature vector and the similar feature vector are first calculated by attention, and then the summary is extracted. This embodiment can include: using the Transformer layer to calculate the core feature vector and the similar feature vector by attention to obtain an attention feature vector. Using the full connection layer to extract the plurality of to-be-processed sentences according to the attention feature vector to obtain the summary text corresponding to the to-be-processed text.

[0074] The second embodiment is shown in Figure 5 The second inference process of the summary extraction model provided by the embodiment of the application is shown in the network structure diagram. The core feature vector and the similar feature vector are first calculated by attention, and then the summary is extracted. This embodiment can include: using the Transformer layer to calculate the core feature vector by attention to obtain an attention feature vector. Using the full connection layer to extract the plurality of to-be-processed sentences according to the attention feature vector and the similar feature vector to obtain the summary text corresponding to the to-be-processed text.

[0075] The second embodiment is shown in Figure 6 The structure of the text summary extraction device based on joint training provided by the embodiment of the application is shown in the structure diagram. The embodiment of the application provides a text summary extraction device 300 based on joint training, which includes:

[0076] The processing text acquisition module 310 is configured to acquire a to-be-processed text, and divide the to-be-processed text into a plurality of to-be-processed sentences.

[0077] The sentence vector representation module 320 is configured to use a vector extraction layer in the summary extraction model to vectorize the plurality of to-be-processed sentences, and obtain word vectors and sentence vectors corresponding to the plurality of to-be-processed sentences. The summary extraction model is obtained through joint training of a sentence recognition task according to importance and a sentence matching task according to similarity.

[0078] The feature vector obtaining module 330 is configured to perform feature extraction on the word vectors and the sentence vectors corresponding to the plurality of to-be-processed sentences by using a feature extraction layer in the summary extraction model, to obtain core feature vectors and similarity feature vectors, the core feature vectors representing the importance of the to-be-processed sentences, and the similarity feature vectors representing the similarity between the plurality of to-be-processed sentences.

[0079] The summary text obtaining module 340 is configured to perform extraction on the plurality of to-be-processed sentences according to the core feature vectors and the similarity feature vectors by using a sentence extraction layer in the summary extraction model, to obtain a summary text corresponding to the to-be-processed text.

[0080] Optionally, in the embodiment of the present application, the sentence vector representing module comprises:

[0081] The word vector obtaining module is configured to perform word segmentation and vectorization on each to-be-processed sentence in the plurality of to-be-processed sentences, to obtain a plurality of word vectors corresponding to each to-be-processed sentence.

[0082] The sentence vector determining module is configured to determine a preset word vector in the plurality of word vectors corresponding to each to-be-processed sentence as a sentence vector of each to-be-processed sentence.

[0083] Optionally, in the embodiment of the present application, the feature vector obtaining module comprises:

[0084] The core feature obtaining module is configured to perform feature extraction on the word vectors and the sentence vectors corresponding to each to-be-processed sentence in the plurality of to-be-processed sentences, to obtain a core feature vector of each to-be-processed sentence.

[0085] The similarity feature obtaining module is configured to calculate the similarity between each to-be-processed sentence in the plurality of to-be-processed sentences and all to-be-processed sentences in the plurality of to-be-processed sentences, to obtain a similarity feature vector of each to-be-processed sentence.

[0086] Optionally, in the embodiment of the present application, the sentence extraction layer comprises a Transformer layer and a fully connected layer; and the summary text obtaining module comprises:

[0087] The first summary extraction module is configured to perform attention calculation on the core feature vectors and the similarity feature vectors by using the Transformer layer, to obtain attention feature vectors, and to perform extraction on the plurality of to-be-processed sentences according to the attention feature vectors by using the fully connected layer.

[0088] Alternatively, the second summary extraction module is configured to perform attention calculation on the core feature vectors by using the Transformer layer, to obtain attention feature vectors, and to perform extraction on the plurality of to-be-processed sentences according to the attention feature vectors and the similarity feature vectors by using the fully connected layer.

[0089] Optionally, in the embodiment of the present application, the text summary extraction device based on joint training further comprises:

[0090] a text label obtaining module, configured to obtain a plurality of sample texts, a plurality of core summary labels and a plurality of similarity labels, wherein the core summary label represents the importance of each sample sentence in the sample text in the summary text corresponding to the sample text, and the similarity label represents the similarity of each sample sentence in the sample text to all sample sentences in the sample text.

[0091] a model joint training module, configured to perform multi-task joint training on the summary extraction network using the plurality of sample texts, the plurality of core summary labels and the plurality of similarity labels, and obtain a summary extraction model.

[0092] Optionally, in the embodiment of the present application, the model joint training module comprises:

[0093] a word and sentence vector obtaining module, configured to use a vector extraction layer in the summary extraction network to perform vector representation on the plurality of sample sentences corresponding to the sample text, and obtain word vectors and sentence vectors corresponding to the plurality of sample sentences.

[0094] a prediction vector obtaining module, configured to use a feature extraction layer in the summary extraction network to perform feature extraction on the word vectors and the sentence vectors corresponding to the plurality of sample sentences, and obtain core prediction vectors and similarity prediction vectors.

[0095] a model loss calculation module, configured to calculate a core cross-entropy loss value between the core prediction vectors and the core summary labels, and calculate a similarity distribution loss value between the similarity prediction vectors and the similarity labels.

[0096] a total loss value calculation module, configured to calculate a total loss value according to the core cross-entropy loss value and the similarity distribution loss value.

[0097] a task joint training module, configured to perform joint training on the summary extraction network according to the total loss value, in the sentence recognition task of the importance and the sentence matching task of the similarity.

[0098] It should be understood that the device corresponds to the above-mentioned embodiment of the text summary extraction method based on joint training, and can perform each step involved in the above-mentioned method embodiments. The specific functions of the device can be referred to the description in the above, and the detailed description is appropriately omitted here to avoid repetition. The device includes at least one software function module stored in the memory in the form of software or firmware or solidified in the operating system (OS) of the device.

[0099] The electronic device provided in the embodiments of the present application comprises a processor and a memory, the memory stores machine readable instructions executable by the processor, and the machine readable instructions are executed by the processor to perform the method as above.

[0100] The embodiments of the present application further provide a computer readable storage medium, and the computer readable storage medium stores a computer program, and the computer program is executed by a processor to perform the method as above.

[0101] The computer readable storage medium can be implemented by any type of volatile or nonvolatile storage device or a combination thereof, such as a static random access memory (SRAM), an electrically erasable programmable read-only memory (EEPROM), an erasable programmable read-only memory (EPROM), a programmable read-only memory (PROM), a read-only memory (ROM), a magnetic storage, a flash memory, a magnetic disk or an optical disk.

[0102] In several embodiments provided by the embodiments of the present application, it should be understood that the disclosed apparatus and method can also be implemented by other ways. The apparatus embodiments described above are only schematic, for example, the flowcharts and block diagrams in the drawings show the possible implementation architectures, functions and operation of the apparatus, method and computer program product according to the embodiments of the present application. In this regard, each block in the flowcharts or block diagrams can represent a module, a program segment or a part of code, which contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions noted in the blocks can be performed in a different order from that noted in the drawings. For example, two consecutive blocks can actually be executed substantially in parallel, and sometimes they can be executed in reverse order, depending on the functions involved.

[0103] In addition, each functional module in each embodiment of the embodiments of the present application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0104] In this document, relational terms such as first and second and the like can be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions.

[0105] The above description is merely exemplary of the application embodiments and is not intended to limit the scope of the application embodiments. Any modification or equivalent arrangement within the spirit and scope of the application embodiments should be included within the scope of the application embodiments.

Claims

1. A joint training based text summarization extraction method, characterized in that, The method comprises: acquiring a to-be-processed text, and segmenting the to-be-processed text to obtain a plurality of to-be-processed sentences; using a vector extraction layer in an abstract extraction model to perform vectorization representation on the plurality of to-be-processed sentences to obtain word vectors and sentence vectors corresponding to the plurality of to-be-processed sentences, the abstract extraction model being obtained through joint training of a sentence importance recognition task and a sentence similarity matching task; using a feature extraction layer in the abstract extraction model to perform feature extraction on the word vectors and the sentence vectors corresponding to the plurality of to-be-processed sentences to obtain core feature vectors and similarity feature vectors, the core feature vectors representing the importance of the to-be-processed sentences, and the similarity feature vectors representing the similarity between the plurality of to-be-processed sentences; using a sentence extraction layer in the abstract extraction model to extract the plurality of to-be-processed sentences according to the core feature vectors and the similarity feature vectors to obtain an abstract text corresponding to the to-be-processed text.

2. The method of claim 1, wherein, The vectorization representation on the plurality of to-be-processed sentences comprises: segmenting and vectorizing each to-be-processed sentence in the plurality of to-be-processed sentences to obtain a plurality of word vectors corresponding to the each to-be-processed sentence; determining a preset word vector in the plurality of word vectors corresponding to the each to-be-processed sentence as a sentence vector of the each to-be-processed sentence.

3. The method of claim 1, wherein, The feature extraction on the word vectors and the sentence vectors corresponding to the plurality of to-be-processed sentences by using the feature extraction layer in the abstract extraction model to obtain the core feature vectors and the similarity feature vectors comprises: performing feature extraction on the word vectors and the sentence vectors corresponding to each to-be-processed sentence in the plurality of to-be-processed sentences to obtain a core feature vector of the each to-be-processed sentence; calculating the similarity between each to-be-processed sentence in the plurality of to-be-processed sentences and all to-be-processed sentences in the plurality of to-be-processed sentences to obtain a similarity feature vector of the each to-be-processed sentence.

4. The method of claim 1, wherein, The sentence extraction layer comprises a Transformer layer and a fully connected layer, and the extraction of the plurality of to-be-processed sentences according to the core feature vectors and the similarity feature vectors comprises: performing attention calculation on the core feature vectors and the similarity feature vectors by using the Transformer layer to obtain attention feature vectors, and extracting the plurality of to-be-processed sentences according to the attention feature vectors by using the fully connected layer; or, performing attention calculation on the core feature vectors by using the Transformer layer to obtain attention feature vectors, and extracting the plurality of to-be-processed sentences according to the attention feature vectors and the similarity feature vectors by using the fully connected layer.

5. The method according to any of claims 1 to 4, characterized in that, Before the vectorization representation on the plurality of to-be-processed sentences by using the vector extraction layer in the abstract extraction model, the method further comprises: obtaining a plurality of sample texts, a plurality of core abstract labels and a plurality of similarity labels, wherein the core abstract labels represent the importance of each sample sentence in the sample texts in the abstract texts corresponding to the sample texts, and the similarity labels represent the similarity between each sample sentence in the sample texts and all sample sentences in the sample texts; using the plurality of sample texts, the plurality of core abstract labels and the plurality of similarity labels to jointly train the abstract extraction network in multiple tasks, and obtaining the abstract extraction model.

6. The method of claim 5, wherein, The method for jointly training the abstract extraction network in multiple tasks using the plurality of sample texts, the plurality of core abstract labels and the plurality of similarity labels comprises: using a vector extraction layer in the abstract extraction network to vectorize the plurality of sample sentences corresponding to the sample texts, and obtaining word vectors and sentence vectors corresponding to the plurality of sample sentences; using a feature extraction layer in the abstract extraction network to extract features from the word vectors and sentence vectors corresponding to the plurality of sample sentences, and obtaining core prediction vectors and similarity prediction vectors; calculating the core cross-entropy loss value between the core prediction vectors and the core abstract labels, and calculating the similarity distribution loss value between the similarity prediction vectors and the similarity labels; calculating the total loss value according to the core cross-entropy loss value and the similarity distribution loss value; jointly training the abstract extraction network according to the total loss value in the sentence recognition task of the importance and the sentence matching task of the similarity.

7. The method of claim 6, wherein, The plurality of sample sentences comprises a first sample sentence and a second sample sentence, and the method for calculating the similarity distribution loss value between the similarity prediction vectors and the similarity labels comprises: for each similarity prediction vector in the plurality of similarity prediction vectors, calculating the cosine similarity between the each similarity prediction vector and all similarity prediction vectors in the plurality of similarity prediction vectors, and obtaining a similarity matrix composed of a plurality of cosine similarities; If it is determined according to the similarity label that the first sample sentence and the second sample sentence are similar, a first sample loss value is calculated according to a formula ​ If it is determined according to the similarity label that the first sample sentence and the second sample sentence are not similar, a second sample loss value is calculated according to a formula ​ According to the formula The first sample loss value, the second sample loss value, the number of similar sample sentences and the number of dissimilar sample sentences in the similarity label are statistically calculated to obtain the similarity distribution loss value. wherein Ew represents a cosine similarity in the similarity matrix, m represents a hyperparameter value, Loss2 represents the similarity distribution loss value, L + represents the first sample loss value, L - represents the second sample loss value, sum(S + ) represents a number of similar sample sentences in the similarity label, sum(S - ) represents a number of dissimilar sample sentences in the similarity label.

8. A joint training based text summarization extraction apparatus, characterized by, comprising: a text processing module configured to obtain a text to be processed, and divide the text to be processed into a plurality of sentences to obtain a plurality of sentences to be processed; a sentence vector representation module configured to use a vector extraction layer in an abstract extraction model to vectorize the plurality of sentences to be processed, and obtain word vectors and sentence vectors corresponding to the plurality of sentences to be processed, wherein the abstract extraction model is obtained by jointly training a sentence recognition task of importance and a sentence matching task of similarity; a feature vector obtaining module configured to use a feature extraction layer in the abstract extraction model to extract features from the word vectors and sentence vectors corresponding to the plurality of sentences to be processed, and obtain core feature vectors and similarity feature vectors, wherein the core feature vectors represent the importance of the sentences to be processed, and the similarity feature vectors represent the similarity between the plurality of sentences to be processed; an abstract text obtaining module configured to use a sentence extraction layer in the abstract extraction model to extract the plurality of sentences to be processed according to the core feature vectors and the similarity feature vectors, and obtain an abstract text corresponding to the text to be processed.

9. An electronic device, comprising: comprising: A processor and a memory, the memory storing machine readable instructions executable by the processor, the machine readable instructions, when executed by the processor, performing the method of any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, A computer readable storage medium storing a computer program, the computer program, when executed by a processor, performing the method of any one of claims 1 to 7.

Images