Text summary model production method and its device, equipment and medium

By using core statement keyword sequences for self-supervised training and noise processing in the text summary model, the problem of poor quality of generated abstracts in the existing technology is solved, the model's understanding and extraction ability of deep semantic information is improved, and the generation is improved Quality and training efficiency.

CN115129819BActive Publication Date: 2024-11-22GUANGZHOU HUANJU SHIDAI INFORMATION TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210833616.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-07-14
Publication Date
2024-11-22
Estimated Expiration
2042-07-14

AI Technical Summary

Technical Problem

Existing neural network models are of poor quality when generating text summary, making it difficult to effectively extract deep semantic information from the original text.

Method used

Self-supervised training by extracting the core statement keyword sequence in the paragraph text, generating equivalent statements, and pre-training the text summary model with light noise and severe noise, forcing the model to understand the paragraph text more deeply and accurately extract deep semantic features.

Benefits of technology

It improves the generation quality of the text summary model, shortens training time, saves training costs, and achieves rapid convergence state.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN115129819B_ABST
    Figure CN115129819B_ABST
Patent Text Reader

Abstract

The present application relates to a method for producing a text summary model in the field of computer technology and its device, equipment, and medium. The method comprises: obtaining a training set, wherein the training set comprises a plurality of sample data, wherein the sample data comprises a paragraph text and its corresponding summary text; using a keyword sequence of a non-core sentence in the paragraph text in the sample data as a training sample, using the non-core sentence as a supervision label, and implementing self-supervisory training on a preset generator; replacing the corresponding non-core sentence in the paragraph text in the sample data with an equivalent sentence and covering part of the core sentence in the paragraph text as a training sample, using the core sentence as a supervision label, and implementing pre-training on the text summary model until convergence; using the paragraph text as a training sample, using the summary text corresponding to the paragraph text as a supervision label, and implementing fine-tuning training on the text summary model until convergence. The present application can generate high-quality text summaries.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a text summary model production method and its corresponding device, computer equipment, and computer-readable storage medium. Background Art

[0002] In order to cope with the phenomenon of information overload, information processing technology that can quickly screen and use effective information was born, and it has developed in two directions: extractive summarization and generative summarization. Extractive summarization is to sort the importance of words and / or sentences in the original text, and extract the words and / or sentences with the highest importance from the original text as the summary; generative summarization is to understand the semantic information of the original text through a machine, and paraphrase the refined and accurate words and / or sentences as the summary.

[0003] The neural network model used for generative summarization usually uses a noise function to destroy the original text input into the neural network model during pre-training, so that the model can repair the damaged original text, so that the model can more accurately extract the semantic information of the original text after pre-training to convergence, so as to help generate a highly readable and accurate summary in the future. In traditional technology, the noise function usually uses token masking, which can force the neural network model to repair the masked tokens in the original text by understanding the contextual semantic information corresponding to the masked tokens in the original text during the pre-training stage, thereby improving the model's ability to extract the semantic information of the original text to a certain extent.

[0004] In practice, it is found that the neural network model implemented under traditional technology still has poor generation quality when generating summaries. Therefore, this application hopes to explore a better solution in a different way. Summary of the invention

[0005] The primary purpose of the present application is to solve at least one of the above problems and provide a text summary model production method and its corresponding device, computer equipment, and computer-readable storage medium.

[0006] In order to meet the various objectives of this application, this application adopts the following technical solutions:

[0007] A text summary model production method provided for one of the purposes of this application includes the following steps:

[0008] Acquire a training set, wherein the training set includes a plurality of sample data, wherein the sample data includes a paragraph text and a corresponding summary text, wherein the paragraph text includes a core sentence and a non-core sentence, wherein the core sentence represents the key semantics of the paragraph text;

[0009] Using the keyword sequence of the non-core sentence in the paragraph text in the sample data as a training sample and the non-core sentence as a supervision label, a preset generator is subjected to self-supervisory training, so that the generator trained to convergence is suitable for generating equivalent sentences of the non-core sentence according to the keyword sequence;

[0010] Replacing corresponding non-core sentences in the paragraph text in the sample data with equivalent sentences and covering some core sentences in the paragraph text as training samples, using the core sentences as supervision labels, and pre-training the text summarization model until convergence;

[0011] The paragraph text in the sample data is used as a training sample, and the summary text corresponding to the paragraph text is used as a supervision label, and the text summary model is fine-tuned and trained until convergence.

[0012] In a preferred embodiment, the step of covering some core sentences in the paragraph text as training samples includes the following steps:

[0013] Encoding the sentences in the paragraph text using a text encoding model that has been pre-trained to convergence to obtain encoding vectors corresponding to each sentence;

[0014] Calculate the average value of the encoding vectors corresponding to each sentence to obtain a central vector;

[0015] Determining the core sentences in the paragraph text according to the vector distance between the encoding vectors corresponding to the respective sentences and the central vector;

[0016] Some core sentences in the paragraph text are selected according to a preset ratio and replaced with specific masking marks.

[0017] In an extended embodiment, the step of replacing the corresponding non-core sentences in the paragraph text in the sample data with equivalent sentences includes the following steps:

[0018] Selecting a non-core sentence corresponding to the core sentence context in the paragraph text, extracting a target keyword text in the non-core sentence, obtaining the corresponding position information in the sentence to construct a keyword sequence, wherein the target keyword text includes a noun text, a verb text, a subject clause keyword text, and an attributive clause keyword text;

[0019] According to the keyword sequence, a generator of an adversarial neural network that has been pre-trained to convergence is used to generate equivalent sentences of non-core sentences, which are used to replace the corresponding non-core sentences in the paragraph text in the sample data.

[0020] In a further embodiment, the training process of the generator of the adversarial neural network comprises the following steps:

[0021] Connect the preset generator to the discriminator that has been pre-trained to convergence to construct an adversarial neural network and freeze the weight of the discriminator;

[0022] Calling a single training sample and inputting it into the generator to generate an equivalent sentence corresponding to the training sample, wherein the training sample is a keyword sequence of a non-core sentence corresponding to the core sentence context in the paragraph text in the sample data;

[0023] Calling the discriminator to perform classification mapping on the equivalent sentence, mapping it to a binary classification space, and obtaining the classification probability corresponding to the mapping to the positive class space or the negative class space;

[0024] Assuming that the supervisory label of the called training sample is a positive sample, the loss value is calculated. When the loss value reaches a preset threshold, the generator is determined to have converged and the training is terminated. Otherwise, another training sample is called to continue iterative training of the generator.

[0025] In a further embodiment, the preset generator is connected to the discriminator that has been pre-trained to convergence to construct an adversarial neural network, and before freezing the weight of the discriminator, the following steps are also included:

[0026] Using the keyword sequence of the non-core sentence in the paragraph text in the sample data as the training sample, using the non-core sentence as the supervision label, and supervising the training of the preset generator until convergence;

[0027] Calling a generator to generate equivalent sentences corresponding to non-core sentences according to the keyword sequence of the training sample as training samples for training a discriminator, and using the corresponding non-core sentences as supervision labels;

[0028] Call the discriminator to perform classification mapping on the training samples trained on it, map them to the binary classification space, and obtain the classification probability corresponding to the mapping to the positive class space or the negative class space;

[0029] According to the supervisory label corresponding to the training sample of the discriminator, the loss value of the corresponding classification probability is calculated. When the loss value reaches a preset threshold, the discriminator is determined to have converged and the training is terminated. Otherwise, the next training sample is called to continue the iterative training.

[0030] In a further embodiment, pre-training the text summarization model until convergence includes the following steps:

[0031] Calling the encoder of the text summarization model to extract the deep semantic features corresponding to the partially covered core sentences in the single training sample, and encoding a first text feature vector;

[0032] Calling the decoder of the text summary model to decode the first text feature vector, and obtaining the probability distribution corresponding to the mapping of each word in the first decoded text to the preset dictionary;

[0033] According to the supervision label corresponding to the training sample, the loss value corresponding to the probability distribution is calculated. When the loss value reaches a preset threshold, the model is determined to have converged and the pre-training is terminated. Otherwise, another training sample is called to continue iterative training of the model.

[0034] In a further embodiment, fine-tuning the text summarization model until convergence includes the following steps:

[0035] Calling the encoder of the text summarization model to extract the deep semantic features corresponding to the paragraph text of the single training sample, and encoding a second text feature vector;

[0036] Calling the decoder of the text summary model to decode the second text feature vector, and obtaining the probability distribution corresponding to the mapping of each word unit in the second decoded text to the preset dictionary;

[0037] According to the supervision label corresponding to the training sample, the loss value corresponding to the probability distribution is calculated. When the loss value reaches a preset threshold, the model is determined to have converged and the fine-tuning training is terminated. Otherwise, another training sample is called to continue iterative training of the model.

[0038] On the other hand, a text summary model production device provided to meet one of the purposes of the present application includes a data acquisition module, a generator training module, a pre-training module and a fine-tuning training module, wherein: the data acquisition module is used to acquire a training set, the training set includes multiple sample data, the sample data includes a paragraph text and its corresponding summary text, the paragraph text includes a core sentence and a non-core sentence, and the core sentence represents the key semantics of the paragraph text; the generator training module is used to use the keyword sequence of the non-core sentence in the paragraph text in the sample data as a training sample, and the non-core sentence as a supervision label to perform self-supervised training on a preset generator, so that the generator trained to convergence is suitable for generating equivalent sentences of the non-core sentence according to the keyword sequence; the pre-training module is used to replace the corresponding non-core sentence in the paragraph text in the sample data with an equivalent sentence and cover part of the core sentence in the paragraph text as a training sample, and use the core sentence as a supervision label to perform pre-training on the text summary model until convergence; the fine-tuning training module is used to use the paragraph text in the sample data as a training sample, and use the summary text corresponding to the paragraph text as a supervision label to perform fine-tuning training on the text summary model until convergence.

[0039] In a preferred embodiment, the pre-training module includes: a sentence encoding submodule, which is used to encode the sentences in the paragraph text using a text encoding model that has been pre-trained to convergence to obtain an encoding vector corresponding to each sentence; a center vector submodule, which is used to calculate the average value of the encoding vectors corresponding to the sentences to obtain a center vector; a core determination submodule, which is used to determine the core sentences in the paragraph text based on the vector distance between the encoding vectors corresponding to the sentences and the center vector; and a text replacement submodule, which is used to select some core sentences in the paragraph text according to a preset ratio and replace them with specific masking marks.

[0040] In an extended embodiment, the pre-training module includes: a text extraction submodule, which is used to select non-core sentences corresponding to the core sentence context in the paragraph text, extract the target keyword text in the non-core sentence, obtain its corresponding position information in the sentence to construct a keyword sequence, and the target keyword text includes noun text, verb text, subject clause keyword text, and attributive clause keyword text; a sentence generation submodule, which is used to generate equivalent sentences of non-core sentences according to the keyword sequence using a generator of an adversarial neural network pre-trained to convergence, and replace the corresponding non-core sentences in the paragraph text in the sample data with them.

[0041] In a further embodiment, the training process of the generator of the adversarial neural network includes: a network construction module, which is used to connect a preset generator to a discriminator that has been pre-trained to convergence to construct an adversarial neural network and freeze the weight of the discriminator; a first sentence fabrication module, which is used to call a single training sample and input it into the generator to generate an equivalent sentence corresponding to the training sample, wherein the training sample is a keyword sequence of a non-core sentence corresponding to the core sentence context in the paragraph text in the sample data; a first classification mapping module, which is used to call the discriminator to perform classification mapping on the equivalent sentence, map it to a binary classification space, and obtain the classification probability corresponding to the mapping to the positive class space or the negative class space; a first iterative training module, which is used to assume that the supervisory label of the called training sample is a positive class sample to calculate the loss value, and when the loss value reaches a preset threshold, it is determined that the generator has converged and the training is terminated, otherwise another training sample is called to continue to perform iterative training on the generator.

[0042] In a further embodiment, before the network construction module, it also includes: a preset generator training module, which is used to use the keyword sequence of non-core sentences in the paragraph text in the sample data as training samples, and the non-core sentences as supervision labels, to supervise the training of the preset generator until convergence; a second sentence fabrication module, which is used to call the generator to generate equivalent sentences corresponding to non-core sentences according to the keyword sequence of the training samples as training samples for training the discriminator, and use the corresponding non-core sentences as supervision labels; a second classification mapping submodule, which is used to call the discriminator to perform classification mapping on the training samples for training it, map it to the binary classification space, and obtain the classification probability corresponding to the mapping to the positive class space or the negative class space; a second iterative training module, which is used to calculate the loss value of the corresponding classification probability according to the supervision label corresponding to the training sample of the discriminator, and when the loss value reaches the preset threshold, it is determined that the discriminator has converged and the training is terminated, otherwise the next training sample is called to continue the iterative training.

[0043] In a further embodiment, the pre-training module includes: a first encoding submodule, which is used to call the encoder of the text summary model to extract the deep semantic features corresponding to the partially covered core sentences in the single training sample, and encode the first text feature vector; a first encoding submodule, which is used to call the decoder of the text summary model to decode the first text feature vector, and obtain the probability distribution corresponding to the mapping of each word in the first decoded text to a preset dictionary; a first iterative training submodule, which is used to calculate the loss value corresponding to the probability distribution according to the supervision label corresponding to the training sample, and when the loss value reaches a preset threshold, it is determined that the model converges and the pre-training is terminated, otherwise another training sample is called to continue iterative training of the model.

[0044] In a further embodiment, the fine-tuning training module includes: a second encoding submodule, which is used to call the encoder of the text summary model to extract the deep semantic features corresponding to the paragraph text of a single training sample, and encode a second text feature vector; a second decoding submodule, which is used to call the decoder of the text summary model to decode the second text feature vector, and obtain the probability distribution corresponding to the mapping of each word in the second decoded text to a preset dictionary; a second iterative training submodule, which is used to calculate the loss value corresponding to the probability distribution according to the supervision label corresponding to the training sample, and when the loss value reaches a preset threshold, it is determined that the model has converged and the fine-tuning training is terminated, otherwise another training sample is called to continue iterative training of the model.

[0045] On the other hand, a computer device is provided to meet one of the purposes of the present application, including a central processing unit and a memory, wherein the central processing unit is used to call and run a computer program stored in the memory to execute the steps of the text summary model production method described in the present application.

[0046] On the other hand, a computer-readable storage medium is provided to meet another purpose of the present application, which stores a computer program implemented according to the text summary model production method in the form of computer-readable instructions, and when the computer program is called and executed by a computer, the steps included in the method are executed.

[0047] The technical solution of this application has many advantages, including but not limited to the following aspects:

[0048] The present application first extracts keyword sequences from core sentences in paragraph texts, and uses the keyword sequences to train a generator suitable for generating equivalent sentences of core sentences. Then, the generator is used to generate equivalent sentences for training samples of a text summary model to replace non-core sentences of the paragraph text, and a small amount of core sentences in the paragraph text are covered, so as to implement the training of the text summary model. For the training samples of the text summary model, the equivalent sentences generated by the generator constitute slight noise, while the covered core sentences constitute its heavy noise. The training samples of the text summary model are constructed by combining slight noise and heavy noise, which can reasonably generalize the features of the training samples. When applying them to the pre-training of the text summary model, the model can be forced to have a deeper understanding of the paragraph text and accurately extract the corresponding deep semantic features, so that when generating the summary text of the paragraph text, high-quality summary text can be generated according to the accurate deep semantic features. In addition, since the training samples are pre-processed by combining different types of noise, the training speed of the text summary model can be improved, so that it can quickly reach a convergence state, saving training costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0050] Figure 1 A flowchart of a typical embodiment of the text summary model production method of the present application;

[0051] Figure 2 A schematic diagram of a process flow for covering core statements in an embodiment of the present application;

[0052] Figure 3 A schematic diagram of a process for replacing non-core statements in an embodiment of the present application;

[0053] Figure 4 A schematic diagram of a flow chart of generator training of an adversarial neural network in an embodiment of the present application;

[0054] Figure 5 A schematic diagram of a process for training a discriminator against a neural network in an embodiment of the present application;

[0055] Figure 6 A schematic diagram of the process of pre-training a text summarization model in an embodiment of the present application;

[0056] Figure 7 A schematic diagram of a process for fine-tuning a training text summarization model in an embodiment of the present application;

[0057] Figure 8 This is a block diagram of the text summary model production device of this application;

[0058] Fig. 9 A schematic diagram of the structure of a computer device used in this application. DETAILED DESCRIPTION

[0059] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be interpreted as limiting the present application.

[0060] It will be understood by those skilled in the art that, unless expressly stated, the singular forms "one", "said", and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present application refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be an intermediate element. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The term "and / or" used herein includes all or any unit and all combinations of one or more associated listed items.

[0061] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by those skilled in the art to which this application belongs. It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with those in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless specifically defined as here.

[0062] It will be understood by those skilled in the art that the "client", "terminal" and "terminal device" used herein include both devices with wireless signal receivers, which are devices with only wireless signal receivers without transmission capabilities, and devices with receiving and transmitting hardware, which are devices with receiving and transmitting hardware capable of two-way communication on a two-way communication link. Such devices may include: cellular or other communication devices such as personal computers, tablet computers, which have single-line displays or multi-line displays or cellular or other communication devices without multi-line displays; PCS (Personal Communications Service, personal communication system), which can combine voice, data processing, fax and / or data communication capabilities; PDA (Personal Digital Assistant, personal digital assistant), which may include a radio frequency receiver, pager, Internet / intranet access, web browser, notepad, calendar and / or GPS (Global Positioning System, global positioning system) receiver; conventional laptop and / or palmtop computers or other devices, which have and / or include a conventional laptop and / or palmtop computer or other device with and / or including a radio frequency receiver. The "client", "terminal" and "terminal device" used herein may be portable, transportable, installed in a vehicle (air, sea and / or land), or suitable for and / or configured to run locally, and / or in a distributed form, at any other location on the earth and / or in space. The "client", "terminal" and "terminal device" used herein may also be a communication terminal, an Internet terminal, a music / video playing terminal, for example, a PDA, a MID (Mobile Internet Device) and / or a mobile phone with a music / video playing function, or a smart TV, a set-top box and other devices.

[0063] The hardware referred to by the names such as "server", "client", and "service node" in this application is essentially an electronic device with the equivalent capabilities of a personal computer. It is a hardware device with the necessary components revealed by the von Neumann principle, such as a central processing unit (including an arithmetic unit and a controller), a memory, an input device, and an output device. The computer program is stored in its memory, and the central processing unit calls the program stored in the external memory into the internal memory for execution, executes the instructions in the program, and interacts with the input and output devices to complete specific functions.

[0064] It should be pointed out that the concept of "server" referred to in this application can also be extended to the case of server clusters. According to the network deployment principle understood by those skilled in the art, the servers should be logically divided. In physical space, these servers can be independent of each other but can be called through interfaces, or integrated into a physical computer or a set of computer clusters. Those skilled in the art should understand this flexibility, and should not use it to restrict the implementation of the network deployment method of this application.

[0065] Unless expressly specified, one or more technical features of the present application can be deployed on a server for implementation and accessed by a client through a remote call to obtain an online service interface provided by the server, or can be directly deployed and run on a client for access.

[0066] The neural network models referenced or may be referenced in this application, unless expressly specified, can be deployed on a remote server and remotely called on the client, or can be deployed and directly called on a client with sufficient device capabilities. In some embodiments, when it runs on the client, its corresponding intelligence can be obtained through transfer learning to reduce the requirements for the client's hardware operating resources and avoid excessive occupation of the client's hardware operating resources.

[0067] Unless explicitly specified, the various data involved in this application can be stored remotely on a server or on a local terminal device, as long as it is suitable for being called by the technical solution of this application.

[0068] Those skilled in the art should be aware that, although the various methods of the present application are described based on the same concept and thus present commonality to each other, unless otherwise specified, these methods can be independently executed. Similarly, for each embodiment disclosed in the present application, they are all proposed based on the same inventive concept, therefore, concepts with the same expression, and concepts that are appropriately changed for convenience despite different expressions, should be understood as equivalent.

[0069] Unless the mutually exclusive relationship between the embodiments to be disclosed in this application is explicitly stated, the relevant technical features involved in each embodiment can be cross-combined to flexibly construct a new embodiment, as long as such combination does not deviate from the creative spirit of this application and can meet the needs of the prior art or solve certain deficiencies in the prior art. Those skilled in the art should be aware of this flexibility.

[0070] A text summary model production method of the present application can be programmed as a computer program product and deployed in a client or server for execution. For example, in the exemplary application scenario of the present application, it can be deployed and implemented in a server of an e-commerce platform, thereby accessing an interface opened after the computer program product is running and performing human-computer interaction with the process of the computer program product through a graphical user interface to execute the method.

[0071] See also Figure 1 The text summary model production method of the present application, in its typical embodiment, comprises the following steps:

[0072] Step S1100, obtaining a training set, wherein the training set includes a plurality of sample data, wherein the sample data includes a paragraph text and a corresponding summary text, wherein the paragraph text includes a core sentence and a non-core sentence, wherein the core sentence represents the key semantics of the paragraph text;

[0073] In one embodiment, an open source data set can be used as the training set, that is, the paragraph text of each data in the data set and its corresponding summary text are used as the sample data in the training set. For Chinese, the data set can use any one or more of LCSTS, CNewSum, and QBSUM. For English, the data set can use any one or more of DUC, TAC, TREC, TREC, CNN / DailyMail, XSum, Wikiasp, FacetSum, and WikiHow.

[0074] In another embodiment, text content posted by users corresponding to multiple fields can be collected from a third-party forum client, and the fields include but are not limited to entertainment, news, sports, education, finance, technology, and the Internet. The text content includes but is not limited to titles and texts. Then, any one or more operations such as filtering invalid characters such as disabled characters, emoticon characters, and redundant punctuation marks, cleaning template text, and extracting valid text information can be adopted to format the collected text content to obtain formatted text content. The correlation between the text and the title can be scored by manual evaluation, and the corresponding correlation score range can be [1,5]. The higher the score, the stronger the correlation. Therefore, according to the correlation score corresponding to each formatted text content, each text content is sorted in order of high to low scores, so as to select multiple text contents with the highest sorting, and use the text of each text content as the paragraph text, and the title corresponding to each text content as the summary text corresponding to its paragraph text, so as to construct the sample data and obtain the training set.

[0075] It is not difficult to understand that the abstract text is generally a brief summary of the main idea of ​​its corresponding paragraph text.

[0076] Step S1200: using a keyword sequence of a non-core sentence in a paragraph text in the sample data as a training sample and the non-core sentence as a supervision label, performing self-supervisory training on a preset generator, so that the generator trained to convergence is suitable for generating equivalent sentences of the non-core sentence according to the keyword sequence;

[0077] In one embodiment, multiple sentences that are similar to the rest of the text in the paragraph text can be found according to the ROUGE index, and each sentence can be determined as the core sentence in the paragraph text. Specifically, the ROUGE calculates the similarity between two sentences by using n-gram overlap using scores from 0 to 100 (ROUGE-1, ROUGE-2 and ROUGE-L are three common variants). Based on this, ROUGE-2 can be selected to calculate the n-gram overlap to express the similarity between each sentence in the paragraph text and all other sentences, and then the sentences can be sorted in order from high to low according to the similarity corresponding to each sentence, and then the top-ranked sentences are determined as the core sentences in the paragraph text.

[0078] In another embodiment, a text encoding model pre-trained to convergence can be used to encode each sentence in the paragraph text, realize vectorized representation of each sentence, and obtain the corresponding encoding vector. Further, a vector similarity calculation function such as a cosine similarity calculation function can be used to calculate the vector distance between the encoding vectors corresponding to each sentence, so as to characterize the similarity between each sentence, calculate the sum of similarities between each sentence and other sentences, and sort each sentence according to the order of the sum of similarities corresponding to each sentence from high to low, and then determine the top multiple sentences as the core sentences in the paragraph text. The text encoding model can be any one of Word2Vec, CBOW, and Bert.

[0079] A plurality of non-core sentences in the paragraph text in the sample data are randomly selected, and the plurality of non-core sentences can be selected according to a certain proportion to obtain, for example, 20% of the total number of non-core sentences in the paragraph text, and a target keyword text is extracted therefrom, and the target keyword text includes any one or more of a noun text, a verb text, a subject clause keyword text, and an attributive clause keyword text. The subject clause keyword text can be a subordinate conjunction such as that, whether, this / that, etc., a connecting pronoun such as which, what, whose, who, she / he / it, this / that, her / his / its, etc., and any one or more of a connecting adverb such as why, when, where, how, why, how, when, how, where, etc. The attributive clause keyword can be a relational word such as that, who, whose, as, she / he / it, this / that, the same as..., she / he / its, etc. In addition, the position of each target keyword text corresponding to the non-core sentence can be encoded to obtain corresponding position information. The encoding can be implemented using any one of sine-cosine encoding, learnable encoding, and one-hot encoding. Technical personnel in the field can flexibly select and implement it as needed.

[0080] Further, the target keyword text mapping can be associated with its corresponding position information to construct a keyword sequence as a training sample, and the corresponding non-core sentence can be used as a supervision label to perform self-supervision training on the preset generator. Specifically, the encoder of the generator encodes the keyword text vector corresponding to the target keyword text in the keyword sequence of the training sample, splices it with the corresponding position information in the keyword sequence, obtains the spliced ​​coding vector, and then uses the keyword sequence as a hard constraint condition, that is, the sentence generated by the encoder encoding of the generator must contain the target keyword text in the corresponding position order. Based on this, the encoder of the generator decodes and generates the equivalent sentence corresponding to the training sample in an autoregressive manner represented by a unidirectional feature according to the spliced ​​coding vector. According to the supervision label corresponding to the training sample, that is, the non-core sentence, the preset cross entropy loss calculation function is applied to calculate the cross entropy loss value corresponding to the equivalent sentence corresponding to the training sample generated by the generator. When the loss value reaches the preset threshold, it is determined that the generator converges and terminates the training, otherwise another training sample is called to continue to implement iterative training on the generator. The generator can be a Gpt series model, and the cross entropy loss calculation function and the preset threshold can be flexibly set by those skilled in the art according to prior knowledge.

[0081] It can be understood that under the self-supervised training, the generator continuously fits the gap between the equivalent sentences generated by itself and the true labels, i.e., the non-core sentences, so that after the generator is trained to convergence, it can generate equivalent sentences with similar semantics to non-core sentences based on the keyword sequence.

[0082] Step S1300: replacing corresponding non-core sentences in the paragraph text in the sample data with equivalent sentences and covering some core sentences in the paragraph text as training samples, using the core sentences as supervision labels, and performing pre-training on the text summarization model until convergence;

[0083] The generator trained to convergence can generate equivalent sentences based on the keyword sequence corresponding to the non-core sentences to replace the corresponding non-core sentences in the paragraph text in the sample data. On this basis, some core sentences in the paragraph text are randomly selected and replaced with specific masking marks, which can be [mask] characters. Based on this, the preprocessed paragraph text is used as a training sample.

[0084] Further, the training sample and the part of the core sentences covered therein can be used as supervisory labels to implement pre-training to convergence on the text summary model. Specifically, the encoder of the text summary model performs bidirectional feature representation encoding, calculation and feature extraction on the paragraph text, and then the decoder of the text summary model uses Cross_Attention (multi-head attention aggregation operation) and the hidden state result of the last layer of the encoder to perform attention aggregation calculation, and then generates the core sentence replaced by the specific masking mark in the paragraph text of the training sample in an autoregressive manner of unidirectional feature representation. According to the supervisory label corresponding to the training sample, i.e., the core sentence, a preset cross entropy loss calculation function is applied to calculate the cross entropy loss value corresponding to the core sentence of the training sample generated by the text summary model. When the loss value reaches a preset threshold, the model is determined to converge and the training is terminated, otherwise another training sample is called to continue iterative pre-training of the model. The text summary model can be any one of the Transformer model, Longformer model, T5 model, and Bart model. Those skilled in the art can select and implement it as needed. The cross entropy loss calculation function and the preset threshold can be flexibly set by those skilled in the art according to prior knowledge and / or experimental data.

[0085] Step S1400: Using the paragraph text in the sample data as a training sample and the summary text corresponding to the paragraph text as a supervision label, fine-tuning the text summary model until convergence.

[0086] Following step S1300, the text summary model that is self-supervised pre-trained to convergence learns to generate the core sentences masked in the paragraph text according to the core semantics of the extracted paragraph text. It can be understood that the output of the model is quite close to the summary text of the paragraph text. Those skilled in the art should know that for the text summary model that is pre-trained to convergence, the fine-tuning training of the downstream task corresponding to the text summary generation can use a small amount of training samples and their corresponding supervision labels, and implement self-supervised fine-tuning training to obtain considerable model performance. Therefore, the paragraph text in the sample data can be used as a training sample, and the summary text corresponding to the paragraph text can be used as a supervision label to implement fine-tuning training on the text summary model until convergence. Specifically, the encoder of the text summary model performs bidirectional feature representation encoding, calculation and feature extraction on the paragraph text, and then the decoder of the text summary model uses Cross_Attention (multi-head attention aggregation operation) and the hidden state result of the last layer of the encoder to perform attention aggregation calculation, and then generates the summary text corresponding to the paragraph text of the training sample in an autoregressive manner of unidirectional feature representation. According to the supervision label corresponding to the training sample, that is, the summary text, a preset cross entropy loss calculation function is applied to calculate the cross entropy loss value corresponding to the summary text corresponding to the training sample generated by the text summary model. When the loss value reaches a preset threshold, the model is determined to converge and the training is terminated, otherwise another training sample is called to continue iterative pre-training of the model. The cross entropy loss calculation function and the preset threshold can be flexibly set by those skilled in the art based on prior knowledge and / or experimental data.

[0087] It can be known from the typical embodiments of the present application that the technical solution of the present application has many advantages, including but not limited to the following aspects:

[0088] The present application first extracts keyword sequences from core sentences in paragraph texts, and uses the keyword sequences to train a generator suitable for generating equivalent sentences of core sentences. Then, the generator is used to generate equivalent sentences for training samples of a text summary model to replace non-core sentences of the paragraph text, and a small amount of core sentences in the paragraph text are covered, so as to implement the training of the text summary model. For the training samples of the text summary model, the equivalent sentences generated by the generator constitute slight noise, while the covered core sentences constitute its heavy noise. The training samples of the text summary model are constructed by combining slight noise and heavy noise, which can reasonably generalize the features of the training samples. When applying them to the pre-training of the text summary model, the model can be forced to have a deeper understanding of the paragraph text and accurately extract the corresponding deep semantic features, so that when generating the summary text of the paragraph text, high-quality summary text can be generated according to the accurate deep semantic features. In addition, since the training samples are pre-processed by combining different types of noise, the training speed of the text summary model can be improved, so that it can quickly reach a convergence state, saving training costs.

[0089] See also Figure 2 In a preferred embodiment, step S1300, the step of covering some core sentences in the paragraph text as training samples, includes the following steps:

[0090] Step S1301: Encode the sentences in the paragraph text using a text encoding model that has been pre-trained to convergence, and obtain encoding vectors corresponding to each sentence;

[0091] The text encoding model pre-trained to convergence may be an open source Bert model, or a model pre-trained to convergence and suitable for text encoding tasks implemented by technicians in this field.

[0092] The Bert model that has been pre-trained to convergence is used to encode the sentences in the paragraph text, and the deep semantic features corresponding to each sentence are extracted to obtain the encoding vector that vectorizes and represents each sentence.

[0093] Step S1302, calculating the average value of the encoding vectors corresponding to the respective sentences to obtain a central vector;

[0094] The encoding vectors corresponding to the sentences are added and divided by the total number of sentences in the paragraph text, and the corresponding average value is calculated and used as the central vector. It can be understood that the central vector can represent the key semantics of the paragraph text to a certain extent.

[0095] Step S1303, determining the core sentences in the paragraph text according to the vector distance between the encoding vector corresponding to each sentence and the central vector;

[0096] A vector similarity calculation function is used to calculate the vector distance between the encoding vector corresponding to each sentence and the central vector. The vector similarity calculation function can be a cosine similarity algorithm, an Euclidean distance algorithm, a Pearson correlation coefficient algorithm, a Minkowski distance algorithm, a Mahalanobis distance algorithm, a Jaccard coefficient algorithm, etc. A person skilled in the art can select any one of them for implementation, as long as the vector distance can be calculated. In one embodiment, the cosine similarity algorithm is called to calculate the vector distance. The closer the vector distance is to 1, the closer the semantics of the corresponding sentence is to the key semantics of the paragraph text. The closer the vector distance is to 0, the greater the gap between the semantics of the corresponding sentence and the key semantics of the paragraph text. Accordingly, the sentence corresponding to the vector distance that meets the preset threshold can be determined as the core sentence in the paragraph text. The preset threshold can be flexibly set by a person skilled in the art based on prior knowledge and / or experimental data.

[0097] Step S1304: select some core sentences in the paragraph text according to a preset ratio and replace them with specific masking marks.

[0098] It can be understood that the selected core sentences should not be too many or too few, and should be selected according to a certain ratio to avoid the subsequent model pre-training, because the core sentences in the paragraph text are too covered and cannot be fitted, or too few core sentences are covered and affect the performance of the model. The recommended preset ratio is 20%. Of course, those skilled in the art can also set it flexibly according to experimental data. In one embodiment, 20% of the core sentences in the paragraph text are randomly selected and replaced with a specific masking mark, and the specific masking mark can be a [mask] character, thereby realizing the masking of the core sentences in the paragraph text.

[0099] In this embodiment, by simply and scientifically calculating the central vector representing the key semantics of the paragraph text, the core sentences in the paragraph text can be effectively determined based on the vector distance between the encoding vector corresponding to each sentence and the central vector.

[0100] See also Figure 3 In an extended embodiment, the step of replacing the corresponding non-core sentences in the paragraph text in the sample data with equivalent sentences includes the following steps:

[0101] Step S1310', selecting a non-core sentence corresponding to the core sentence context in the paragraph text, extracting a target keyword text in the non-core sentence, obtaining its corresponding position information in the sentence to construct a keyword sequence, wherein the target keyword text includes a noun text, a verb text, a subject clause keyword text, and an attributive clause keyword text;

[0102] Select the non-core sentences corresponding to the core sentence context in the paragraph text, that is, the non-core sentences closest to the core sentence. Jieba segmentation can be used to segment these non-core sentences to obtain corresponding segmentation texts. Then, according to the preset dictionary, the segmentation text is matched with the word text in the dictionary in an accurate matching manner to obtain the part of speech corresponding to the matched word text. The part of speech includes but is not limited to nouns, quantifiers, pronouns, relative words, adverbs, subordinate conjunctions, connecting pronouns, and connecting adverbs. Then, according to the part of speech corresponding to each segmentation text, the segmentation text belonging to the part of speech corresponding to the target keyword text can be determined. Based on this, the target keyword text is extracted from the non-core sentence. In addition, the position of each target keyword text corresponding to the non-core sentence can be encoded to obtain the corresponding position information. The encoding can be implemented by any one of sine-cosine encoding, learnable encoding, and unique hot encoding. The technical personnel in this field can flexibly select and implement as needed. Further, the target keyword text can be mapped and associated with its corresponding position information to construct a keyword sequence.

[0103] The subject clause keyword text may be subordinate conjunctions such as that, whether, this / that, etc., connecting pronouns such as which, what, whose, who, she / he / it, this / that, she / his / its, etc., any one or more of connecting adverbs such as why, when, where, how, why, how, when, how, where, etc., and the attributive clause keywords may be relative words such as that, who, whose, as, she / he / it, this / that, the same as..., she / his / its, etc.

[0104] Step S1320': according to the keyword sequence, use a generator of an adversarial neural network that has been pre-trained to convergence to generate equivalent sentences of non-core sentences, and replace the equivalent sentences in the paragraph text in the sample data with the equivalent sentences.

[0105] The generator of the adversarial neural network can be a Gpt series model, and the corresponding discriminator can be flexibly implemented by those skilled in the art, or implemented with reference to the subsequent embodiments. The training process of the generator of the adversarial neural network is further revealed by the subsequent embodiments, and this step is temporarily not shown.

[0106] In one embodiment, the encoder of the generator encodes a keyword text vector corresponding to the target keyword text in the keyword sequence, concatenates it with the corresponding position information in the keyword sequence to obtain a concatenated coding vector, and then uses the keyword sequence as a hard constraint condition, that is, the sentence generated by encoding with the encoder of the generator must contain the target keyword text in the corresponding position order. Based on this, the encoder of the generator decodes the concatenated coding vector in an autoregressive manner represented by a unidirectional feature to generate an equivalent sentence corresponding to the training sample.

[0107] In this embodiment, firstly, a rule matching method is adopted to quickly and accurately determine the target keyword text in the non-core sentence. Secondly, an adversarial neural network generator is used to generate equivalent sentences of the non-core sentence, which effectively improves the quality of the generated equivalent sentences. In addition, the non-core sentence corresponding to the core sentence context is selected and replaced with the equivalent sentence generated by the generator, which effectively improves the difficulty of understanding the paragraph text, so that when it is subsequently applied to the pre-training of the text summary model, the generalization ability of the model is improved, and the deep semantic features of the masked core sentence can be accurately extracted, which deepens the model's understanding of the paragraph text, and is conducive to improving the quality of the summary text corresponding to the generated paragraph text.

[0108] See also Figure 4 In a further embodiment, step S1320', the training process of the generator of the adversarial neural network, comprises the following steps:

[0109] Step S1325', connect the preset generator to the discriminator that has been pre-trained to convergence to construct an adversarial neural network, and freeze the weight of the discriminator;

[0110] When constructing the adversarial neural network of the present application, the preset generator is connected to the discriminator pre-trained to convergence. In addition, the weight of the discriminator is frozen by setting the operation so that it does not participate in the weight update of the subsequent training process. The preset generator can be a Gpt series model. The training of the discriminator is further revealed by the subsequent embodiments, and this step is temporarily ignored.

[0111] Step S1326', calling a single training sample to input into the generator to generate an equivalent sentence corresponding to the training sample, wherein the training sample is a keyword sequence of a non-core sentence corresponding to the core sentence context in the paragraph text in the sample data;

[0112] Regarding the core sentences in the paragraph text and the keyword sequence, reference may be made to some embodiments of the present application, and this step will not be repeated here.

[0113] In one embodiment, the encoder of the generator encodes a keyword text vector corresponding to the target keyword text in the keyword sequence, concatenates it with the corresponding position information in the keyword sequence to obtain a concatenated coding vector, and then uses the keyword sequence as a hard constraint condition, that is, the sentence generated by encoding with the encoder of the generator must contain the target keyword text in the corresponding position order. Based on this, the encoder of the generator decodes the concatenated coding vector in an autoregressive manner represented by a unidirectional feature to generate an equivalent sentence corresponding to the training sample.

[0114] Step S1327', calling the discriminator to perform classification mapping on the equivalent sentence, mapping it to a binary classification space, and obtaining the classification probability corresponding to mapping to the positive class space or the negative class space;

[0115] The encoder of the discriminator encodes the equivalent sentence to obtain a corresponding equivalent encoding vector, performs classification mapping on the equivalent sentence through full connection, maps the equivalent encoding vector to the binary classification space, and predicts the classification probability corresponding to the positive class space or the negative class space.

[0116] Step S1328', assuming that the supervisory label of the called training sample is a positive sample, calculate the loss value, and when the loss value reaches a preset threshold, determine that the generator has converged and terminate the training, otherwise call another training sample to continue iterative training of the generator.

[0117] The preset threshold value can be flexibly set by those skilled in the art based on prior knowledge and / or experimental data.

[0118] In each iterative training, the corresponding classification probability can be obtained for a single training sample. At this time, it can be assumed that the supervised label of the called training sample is a positive sample, and the corresponding classification probability mapped to the positive class space is 1. The cross entropy loss value corresponding to the classification probability is calculated, and then the generator is judged whether it has converged according to the loss value. When the generator has not converged, the generator is back-propagated according to the loss value to implement gradient update, and then the next training sample is called to implement iterative training. When the generator is judged to have converged according to the loss value, the training can be terminated.

[0119] After the above process, the training of the generator is completed in the generative adversarial network. In this process, the discriminator acts as a referee and assumes that the generator should produce qualified positive sample results. If the quality of the equivalent sentence produced by the generator does not meet expectations, under the influence of the assumed supervision label, the weight parameters of the generator will be corrected to a corresponding extent, so that the actual ability of the generator will be continuously and effectively improved, and finally reach a state of competing with the discriminator.

[0120] In addition, those skilled in the art should be aware that the training of the generator and the discriminator of the adversarial neural network are both for one party. When the trained party is trained, the weight of the other party is frozen, and the other party guides the training of the trained party according to the input of the trained party until convergence. Based on this, the generator and the discriminator are continuously iterated until the discriminator repeatedly judges that the probability of the generator's generation of a non-core sentence is 0.5, and Nash equilibrium is reached. At this time, the generator can be output as the generator of the adversarial neural network of this application.

[0121] In this embodiment, on the one hand, the training process of the generator of the adversarial neural network is disclosed. The generator is trained under the weight of the discriminator that has been pre-trained to convergence, so that after the generator is trained to convergence, it learns the ability to generate equivalent sentences corresponding to non-core sentences according to the keyword sequence of the non-core sentence. Subsequent calls to the generator can quickly and accurately generate equivalent sentences. On the other hand, it can be understood that the equivalent sentences generated by the generator are judged by the discriminator as a referee, which effectively improves the quality of the equivalent sentences generated by the generator and ensures the semantic accuracy of the equivalent sentences.

[0122] See also Figure 5 In a further embodiment, step S1320', connecting the preset generator to the discriminator pre-trained to convergence to construct an adversarial neural network, and freezing the weight of the discriminator, further includes the following steps:

[0123] Step S1321', using the keyword sequence of the non-core sentence in the paragraph text in the sample data as a training sample, using the non-core sentence as a supervision label, and supervising the training of the preset generator until convergence;

[0124] For the specific implementation of this step, please refer to step S1200, and this step will not be described in detail.

[0125] Step S1322', calling the generator to generate equivalent sentences corresponding to the non-core sentences according to the keyword sequence of the training sample as training samples for training the discriminator, and using the corresponding non-core sentences as supervision labels;

[0126] Regarding the specific implementation of the calling generator generating equivalent sentences corresponding to the non-core sentences according to the keyword sequence of the training sample, please refer to step S1200, and this step will not be repeated here.

[0127] The equivalent sentences corresponding to the non-core sentences generated by the generator are used as training samples for training the discriminator, and the corresponding non-core sentences are used as supervision labels.

[0128] Step S1323', calling the discriminator to perform classification mapping on the training samples trained on it, mapping them to the binary classification space, and obtaining the classification probability corresponding to the mapping to the positive class space or the negative class space;

[0129] The encoder of the discriminator encodes the equivalent sentences of the training samples for training it, obtains the corresponding equivalent coding vector, performs classification mapping on it through full connection, maps it to the binary classification space, and predicts the classification probability corresponding to the positive class space or the negative class space.

[0130] Step S1324', according to the supervisory label corresponding to the training sample of the discriminator, calculate the loss value of the corresponding classification probability, and when the loss value reaches a preset threshold, determine that the discriminator has converged and terminate the training, otherwise call the next training sample to continue iterative training.

[0131] The preset threshold value can be flexibly set by those skilled in the art based on prior knowledge and / or experimental data.

[0132] In each iterative training, the corresponding classification probability can be obtained for a single training sample. At this time, according to the supervisory label corresponding to the training sample of the discriminator, the supervised label is that the classification probability of the positive sample mapped to the positive class space is 1, and the cross entropy loss value corresponding to the classification probability is calculated. Then, according to the loss value, it is judged whether the discriminator has converged. When the discriminator has not converged, the discriminator is back-propagated according to the loss value to implement gradient update, and then the next training sample is called to implement iterative training. When the discriminator is judged to have converged according to the loss value, the training can be terminated.

[0133] After the above process, the training of the discriminator is completed. Subsequently, the discriminator can be connected to the generator to construct an adversarial neural network. The discriminator acts as the referee of the generator in the network to judge the generation of the generator.

[0134] In this embodiment, the training process of the discriminator is disclosed. After the discriminator is trained to convergence, it learns the ability to distinguish whether the equivalent sentence generated by the generator is a non-core sentence, so that after being subsequently connected to the generator, it can quickly distinguish the generation of the generator.

[0135] See also Figure 6 In a further embodiment, step S1300, pre-training the text summarization model until convergence, comprises the following steps:

[0136] Step S1310, calling the encoder of the text summarization model to extract the deep semantic features corresponding to the partially covered core sentences in the single training sample, and encoding a first text feature vector;

[0137] The encoder of the text summarization model is called to perform bidirectional feature representation encoding on the paragraph text of the training sample, perform corresponding attention calculation, extract the deep semantic features corresponding to the masked core sentences, and obtain the first text feature vector.

[0138] Step S1320, calling the decoder of the text summary model to decode the first text feature vector, and obtaining the probability distribution corresponding to the mapping of each word in the first decoded text to the preset dictionary;

[0139] Furthermore, the decoder of the text summarization model is called to perform attention aggregation calculation using Cross_Attention (multi-head attention aggregation operation) and the first text feature vector output by the hidden state of the last layer of the encoder, and then obtains the probability distribution corresponding to the mapping of each word in the first decoded text to the preset dictionary in an autoregressive manner of unidirectional feature representation. The preset dictionary can be flexibly set by those skilled in the art based on prior knowledge or experimental data.

[0140] Step S1330: Calculate the loss value corresponding to the probability distribution according to the supervision label corresponding to the training sample. When the loss value reaches a preset threshold, the model is determined to have converged and the pre-training is terminated. Otherwise, another training sample is called to continue iterative training of the model.

[0141] According to the supervisory label corresponding to the training sample, that is, the core sentence covered in the training sample, a preset cross entropy loss calculation function is applied to calculate the cross entropy loss value corresponding to the probability distribution. When the loss value reaches the preset threshold, the model is judged to have converged and the training is terminated, otherwise another training sample is called to continue iterative pre-training of the model. When the model is judged to have converged according to the loss value, the pre-training of the model can be terminated. The cross entropy loss calculation function and the preset threshold can be flexibly set by those skilled in the art based on prior knowledge and / or experimental data.

[0142] In this embodiment, the process of pre-training to convergence of the text summary model is disclosed. The paragraph text of the training sample used for pre-training of the text summary model is processed by covering the core sentences and replacing the non-core sentences. During the pre-training process, the model deepens its understanding of the paragraph text and can accurately extract the deep semantic features of the covered core sentences, which is conducive to improving the quality of the summary text corresponding to the generated paragraph text.

[0143] See also Figure 7 In a further embodiment, step S1400, fine-tuning the text summarization model until convergence, comprises the following steps:

[0144] Step S1410, calling the encoder of the text summarization model to extract the deep semantic features corresponding to the paragraph text of a single training sample, and encoding a second text feature vector;

[0145] The encoder of the text summarization model is called to perform bidirectional feature representation encoding on the paragraph text of the training sample, perform corresponding attention calculations, extract deep semantic features corresponding to the paragraph text, and obtain a second text feature vector.

[0146] Step S1420, calling the decoder of the text summary model to decode the second text feature vector, and obtaining the probability distribution corresponding to the mapping of each word in the second decoded text to the preset dictionary;

[0147] Furthermore, the decoder of the text summarization model is called to perform attention aggregation calculation using Cross_Attention (multi-head attention aggregation operation) and the second text feature vector output by the hidden state of the last layer of the encoder, and then obtains the probability distribution corresponding to the mapping of each word in the second decoded text to the preset dictionary in an autoregressive manner of unidirectional feature representation. The preset dictionary can be flexibly set by those skilled in the art based on prior knowledge or experimental data.

[0148] Step S1430, according to the supervision label corresponding to the training sample, calculate the loss value corresponding to the probability distribution, and when the loss value reaches a preset threshold, determine that the model has converged and terminate the fine-tuning training, otherwise call another training sample to continue iterative training of the model.

[0149] According to the supervisory label corresponding to the training sample, that is, the summary text corresponding to the paragraph text, a preset cross entropy loss calculation function is applied to calculate the cross entropy loss value corresponding to the probability distribution. When the loss value reaches a preset threshold, the model is judged to have converged and the training is terminated, otherwise another training sample is called to continue iterative training of the model. When the model is judged to have converged according to the loss value, the fine-tuning training of the model can be terminated. The cross entropy loss calculation function and the preset threshold can be flexibly set by those skilled in the art based on prior knowledge and / or experimental data.

[0150] In this embodiment, the process of fine-tuning the text summary model until convergence is revealed, so that the text summary model learns the ability to generate corresponding summary text based on the accurate extraction of deep semantic features of paragraph text. The text summary model can be subsequently applied to actual business scenarios to generate high-quality summary text for long texts.

[0151] See also Figure 8, a text summary model production device provided to meet one of the purposes of the present application is a functional embodiment of the text summary model production method of the present application, the device includes a data acquisition module, a generator training module, a pre-training module and a fine-tuning training module, wherein: the data acquisition module is used to obtain a training set, the training set includes a plurality of sample data, the sample data includes a paragraph text and its corresponding summary text, the paragraph text includes core sentences and non-core sentences, the core sentences represent the key semantics of the paragraph text; the generator training module is used to use the keyword sequence of the non-core sentences in the paragraph text in the sample data as the training sample, and the pre-training module and the fine-tuning training module are used to obtain the training set, the training set includes a plurality of sample data, the sample data includes a paragraph text and its corresponding summary text, the paragraph text includes a core sentence and a non-core sentence, the core sentence represents the key semantics of the paragraph text; the generator training module is used to use the keyword sequence of the non-core sentence in the paragraph text in the sample data as the training sample, and the pre-training module and the fine-tuning training module are used to obtain the training set, the training set includes a plurality of sample data, the sample data includes a paragraph text and its corresponding summary text, the paragraph text includes a core sentence and a non-core sentence, the core sentence represents the key semantics of the paragraph text; The non-core sentences are used as supervision labels, and self-supervised training is performed on a preset generator, so that the generator trained to convergence is suitable for generating equivalent sentences of the non-core sentences according to the keyword sequence; a pre-training module is used to replace the corresponding non-core sentences in the paragraph text in the sample data with equivalent sentences and cover part of the core sentences in the paragraph text as training samples, and use the core sentences as supervision labels to perform pre-training on the text summary model until convergence; a fine-tuning training module is used to use the paragraph text in the sample data as training samples, and use the summary text corresponding to the paragraph text as supervision labels, and perform fine-tuning training on the text summary model until convergence.

[0152] In a preferred embodiment, the pre-training module includes: a sentence encoding submodule, which is used to encode the sentences in the paragraph text using a text encoding model that has been pre-trained to convergence to obtain an encoding vector corresponding to each sentence; a center vector submodule, which is used to calculate the average value of the encoding vectors corresponding to the sentences to obtain a center vector; a core determination submodule, which is used to determine the core sentences in the paragraph text based on the vector distance between the encoding vectors corresponding to the sentences and the center vector; and a text replacement submodule, which is used to select some core sentences in the paragraph text according to a preset ratio and replace them with specific masking marks.

[0153] In an extended embodiment, the pre-training module includes: a text extraction submodule, which is used to select non-core sentences corresponding to the core sentence context in the paragraph text, extract the target keyword text in the non-core sentence, obtain its corresponding position information in the sentence to construct a keyword sequence, and the target keyword text includes noun text, verb text, subject clause keyword text, and attributive clause keyword text; a sentence generation submodule, which is used to generate equivalent sentences of non-core sentences according to the keyword sequence using a generator of an adversarial neural network pre-trained to convergence, and replace the corresponding non-core sentences in the paragraph text in the sample data with them.

[0154] In a further embodiment, the training process of the generator of the adversarial neural network includes: a network construction module, which is used to connect a preset generator to a discriminator that has been pre-trained to convergence to construct an adversarial neural network and freeze the weight of the discriminator; a first sentence fabrication module, which is used to call a single training sample and input it into the generator to generate an equivalent sentence corresponding to the training sample, wherein the training sample is a keyword sequence of a non-core sentence corresponding to the core sentence context in the paragraph text in the sample data; a first classification mapping module, which is used to call the discriminator to perform classification mapping on the equivalent sentence, map it to a binary classification space, and obtain the classification probability corresponding to the mapping to the positive class space or the negative class space; a first iterative training module, which is used to assume that the supervisory label of the called training sample is a positive class sample to calculate the loss value, and when the loss value reaches a preset threshold, it is determined that the generator has converged and the training is terminated, otherwise another training sample is called to continue to perform iterative training on the generator.

[0155] In a further embodiment, before the network construction module, it also includes: a preset generator training module, which is used to use the keyword sequence of non-core sentences in the paragraph text in the sample data as training samples, and the non-core sentences as supervision labels, to supervise the training of the preset generator until convergence; a second sentence fabrication module, which is used to call the generator to generate equivalent sentences corresponding to non-core sentences according to the keyword sequence of the training samples as training samples for training the discriminator, and use the corresponding non-core sentences as supervision labels; a second classification mapping submodule, which is used to call the discriminator to perform classification mapping on the training samples for training it, map it to the binary classification space, and obtain the classification probability corresponding to the mapping to the positive class space or the negative class space; a second iterative training module, which is used to calculate the loss value of the corresponding classification probability according to the supervision label corresponding to the training sample of the discriminator, and when the loss value reaches the preset threshold, it is determined that the discriminator has converged and the training is terminated, otherwise the next training sample is called to continue the iterative training.

[0156] In a further embodiment, the pre-training module includes: a first encoding submodule, which is used to call the encoder of the text summary model to extract the deep semantic features corresponding to the partially covered core sentences in the single training sample, and encode the first text feature vector; a first encoding submodule, which is used to call the decoder of the text summary model to decode the first text feature vector, and obtain the probability distribution corresponding to the mapping of each word in the first decoded text to a preset dictionary; a first iterative training submodule, which is used to calculate the loss value corresponding to the probability distribution according to the supervision label corresponding to the training sample, and when the loss value reaches a preset threshold, it is determined that the model converges and the pre-training is terminated, otherwise another training sample is called to continue iterative training of the model.

[0157] In a further embodiment, the fine-tuning training module includes: a second encoding submodule, which is used to call the encoder of the text summary model to extract the deep semantic features corresponding to the paragraph text of a single training sample, and encode a second text feature vector; a second decoding submodule, which is used to call the decoder of the text summary model to decode the second text feature vector, and obtain the probability distribution corresponding to the mapping of each word in the second decoded text to a preset dictionary; a second iterative training submodule, which is used to calculate the loss value corresponding to the probability distribution according to the supervision label corresponding to the training sample, and when the loss value reaches a preset threshold, it is determined that the model has converged and the fine-tuning training is terminated, otherwise another training sample is called to continue iterative training of the model.

[0158] In order to solve the above technical problems, the present application also provides a computer device. Fig. 9 As shown, a schematic diagram of the internal structure of a computer device. The computer device includes a processor, a computer-readable storage medium, a memory, and a network interface connected via a system bus. Among them, the computer-readable storage medium of the computer device stores an operating system, a database, and computer-readable instructions. The database may store a control information sequence. When the computer-readable instructions are executed by the processor, the processor can implement a text summary model production method. The processor of the computer device is used to provide computing and control capabilities to support the operation of the entire computer device. The memory of the computer device may store computer-readable instructions. When the computer-readable instructions are executed by the processor, the processor can execute the text summary model production method of the present application. The network interface of the computer device is used to connect and communicate with a terminal. Those skilled in the art can understand that Fig. 9 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0159] In this embodiment, the processor is used to execute Figure 8 The memory stores the program code and various data required to execute the above modules or submodules. The network interface is used to transmit data between user terminals or servers. The memory in this embodiment stores the program code and data required to execute all modules / submodules in the text summary model production device of this application, and the server can call the program code and data of the server to execute the functions of all submodules.

[0160] The present application also provides a storage medium storing computer-readable instructions. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the text summary model production method of any embodiment of the present application.

[0161] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments of the present application can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a computer-readable storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0162] To summarize, this application combines the preprocessing methods of sentence-level noise processing of core sentences and noise processing of non-core sentence replacement in the context of core sentences, and uses damaged paragraph text as training samples to pre-train the text summarization model, which can effectively ensure the performance of the text summarization model and generate high-quality summary text.

[0163] Those skilled in the art will appreciate that the various operations, methods, steps, measures, and schemes in the processes discussed in this application may be alternated, altered, combined, or deleted. Further, other steps, measures, and schemes in the various operations, methods, and processes discussed in this application may also be alternated, altered, rearranged, decomposed, combined, or deleted. Further, the steps, measures, and schemes in the prior art that are similar to those disclosed in this application may also be alternated, altered, rearranged, decomposed, combined, or deleted.

[0164] The above description is only a partial implementation method of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A method for producing a text summarization model, characterized in that: The steps include: Acquire a training set, wherein the training set includes a plurality of sample data, wherein the sample data includes a paragraph text and a corresponding summary text, wherein the paragraph text includes a core sentence and a non-core sentence, wherein the core sentence represents the key semantics of the paragraph text; Using the keyword sequence of the non-core sentence in the paragraph text in the sample data as a training sample and the non-core sentence as a supervision label, a preset generator is subjected to self-supervisory training, so that the generator trained to convergence is suitable for generating equivalent sentences of the non-core sentence according to the keyword sequence; Replacing corresponding non-core sentences in the paragraph text in the sample data with equivalent sentences and covering some core sentences in the paragraph text as training samples, using the core sentences as supervision labels, and pre-training the text summarization model until convergence; Using the paragraph text in the sample data as a training sample and the summary text corresponding to the paragraph text as a supervision label, fine-tuning the text summary model until convergence; The method uses the keyword sequence of the non-core sentence in the paragraph text in the sample data as a training sample, uses the non-core sentence as a supervision label, and performs self-supervisory training on a preset generator, so that the generator trained to convergence is suitable for generating equivalent sentences of the non-core sentence according to the keyword sequence, including: Associating each target keyword text in the non-core sentence with its corresponding position information to construct a keyword sequence as a training sample, using the non-core sentence as a supervision label, and performing self-supervisory training on a preset generator, wherein the position information corresponding to the target keyword text is obtained by encoding the position of the target keyword text in the non-core sentence; For the self-supervised training, the encoder of the generator encodes the keyword text vector corresponding to the target keyword text in the keyword sequence of the training sample, splices it with the corresponding position information in the keyword sequence to obtain a spliced ​​coding vector, and then uses the keyword sequence as a hard constraint condition. The encoder of the generator decodes the spliced ​​coding vector in an autoregressive manner represented by a unidirectional feature to generate an equivalent sentence corresponding to the training sample, applies a preset cross entropy loss calculation function, and calculates the cross entropy loss value corresponding to the equivalent sentence according to the supervision label corresponding to the training sample. When the loss value reaches a preset threshold, it is determined that the generator has converged and the training is terminated, otherwise another training sample is called to continue iterative training of the generator.

2. The text summarization model production method according to claim 1, characterized in that: The step of covering some core sentences in the paragraph text as training samples includes the following steps: Encoding the sentences in the paragraph text using a text encoding model that has been pre-trained to convergence to obtain encoding vectors corresponding to each sentence; Calculate the average value of the encoding vectors corresponding to each sentence to obtain a central vector; Determining the core sentences in the paragraph text according to the vector distance between the encoding vectors corresponding to the respective sentences and the central vector; Some core sentences in the paragraph text are selected according to a preset ratio and replaced with specific masking marks.

3. The text summarization model production method according to claim 1, characterized in that: The step of replacing the corresponding non-core sentences in the paragraph text in the sample data with equivalent sentences includes the following steps: Selecting a non-core sentence corresponding to the core sentence context in the paragraph text, extracting a target keyword text in the non-core sentence, obtaining the corresponding position information in the sentence to construct a keyword sequence, wherein the target keyword text includes a noun text, a verb text, a subject clause keyword text, and an attributive clause keyword text; According to the keyword sequence, a generator of an adversarial neural network that has been pre-trained to convergence is used to generate equivalent sentences of non-core sentences, which are used to replace the corresponding non-core sentences in the paragraph text in the sample data.

4. The text summarization model production method according to claim 3, characterized in that: The training process of the generator of the adversarial neural network includes the following steps: Connect the preset generator to the discriminator that has been pre-trained to convergence to construct an adversarial neural network and freeze the weight of the discriminator; Calling a single training sample and inputting it into the generator to generate an equivalent sentence corresponding to the training sample, wherein the training sample is a keyword sequence of a non-core sentence corresponding to the core sentence context in the paragraph text in the sample data; Calling the discriminator to perform classification mapping on the equivalent sentence, mapping it to a binary classification space, and obtaining the classification probability corresponding to the mapping to the positive class space or the negative class space; Assuming that the supervisory label of the called training sample is a positive sample, the loss value is calculated. When the loss value reaches a preset threshold, the generator is determined to have converged and the training is terminated. Otherwise, another training sample is called to continue iterative training of the generator.

5. The method for producing a text summarization model according to claim 4, characterized in that: The preset generator is connected to the discriminator that has been pre-trained to convergence to construct an adversarial neural network. Before freezing the weight of the discriminator, the following steps are also included: Using the keyword sequence of the non-core sentence in the paragraph text in the sample data as the training sample, using the non-core sentence as the supervision label, and supervising the training of the preset generator until convergence; Calling a generator to generate equivalent sentences corresponding to non-core sentences according to the keyword sequence of the training sample as training samples for training a discriminator, and using the corresponding non-core sentences as supervision labels; Call the discriminator to perform classification mapping on the training samples trained on it, map them to the binary classification space, and obtain the classification probability corresponding to the mapping to the positive class space or the negative class space; According to the supervisory label corresponding to the training sample of the discriminator, the loss value of the corresponding classification probability is calculated. When the loss value reaches a preset threshold, the discriminator is determined to have converged and the training is terminated. Otherwise, the next training sample is called to continue the iterative training.

6. The method for producing a text summarization model according to claim 1, characterized in that: Pre-training the text summarization model until convergence includes the following steps: Calling the encoder of the text summarization model to extract the deep semantic features corresponding to the partially covered core sentences in the single training sample, and encoding a first text feature vector; Calling the decoder of the text summary model to decode the first text feature vector, and obtaining the probability distribution corresponding to the mapping of each word in the first decoded text to the preset dictionary; According to the supervision label corresponding to the training sample, the loss value corresponding to the probability distribution is calculated. When the loss value reaches a preset threshold, the model is determined to have converged and the pre-training is terminated. Otherwise, another training sample is called to continue iterative training of the model.

7. The text summarization model production method according to claim 1, characterized in that: Fine-tune the text summarization model until convergence, including the following steps: Calling the encoder of the text summarization model to extract the deep semantic features corresponding to the paragraph text of the single training sample, and encoding a second text feature vector; Calling the decoder of the text summary model to decode the second text feature vector, and obtaining the probability distribution corresponding to the mapping of each word unit in the second decoded text to the preset dictionary; According to the supervision label corresponding to the training sample, the loss value corresponding to the probability distribution is calculated. When the loss value reaches a preset threshold, the model is determined to have converged and the fine-tuning training is terminated. Otherwise, another training sample is called to continue iterative training of the model.

8. A text summary model production device, characterized in that: include: A data acquisition module is used to acquire a training set, wherein the training set includes a plurality of sample data, wherein the sample data includes a paragraph text and a corresponding summary text, wherein the paragraph text includes a core sentence and a non-core sentence, wherein the core sentence represents the key semantics of the paragraph text; A generator training module, used to use a keyword sequence of a non-core sentence in a paragraph text in the sample data as a training sample and the non-core sentence as a supervision label to perform self-supervisory training on a preset generator, so that the generator trained to convergence is suitable for generating equivalent sentences of the non-core sentence according to the keyword sequence; A pre-training module, used to replace the corresponding non-core sentences in the paragraph text in the sample data with equivalent sentences and cover some core sentences in the paragraph text as training samples, use the core sentences as supervision labels, and perform pre-training on the text summarization model until convergence; A fine-tuning training module is used to use the paragraph text in the sample data as a training sample and the summary text corresponding to the paragraph text as a supervision label to perform fine-tuning training on the text summary model until convergence; The method uses the keyword sequence of the non-core sentence in the paragraph text in the sample data as a training sample, uses the non-core sentence as a supervision label, and performs self-supervisory training on a preset generator, so that the generator trained to convergence is suitable for generating equivalent sentences of the non-core sentence according to the keyword sequence, including: Associating each target keyword text in the non-core sentence with its corresponding position information to construct a keyword sequence as a training sample, using the non-core sentence as a supervision label, and performing self-supervisory training on a preset generator, wherein the position information corresponding to the target keyword text is obtained by encoding the position of the target keyword text in the non-core sentence; For the self-supervised training, the encoder of the generator encodes the keyword text vector corresponding to the target keyword text in the keyword sequence of the training sample, splices it with the corresponding position information in the keyword sequence to obtain a spliced ​​coding vector, and then uses the keyword sequence as a hard constraint condition. The encoder of the generator decodes the spliced ​​coding vector in an autoregressive manner represented by a unidirectional feature to generate an equivalent sentence corresponding to the training sample, applies a preset cross entropy loss calculation function, and calculates the cross entropy loss value corresponding to the equivalent sentence according to the supervision label corresponding to the training sample. When the loss value reaches a preset threshold, it is determined that the generator has converged and the training is terminated, otherwise another training sample is called to continue iterative training of the generator.

9. A computer device comprising a central processing unit and a memory, characterized in that: The central processing unit is used to call and run the computer program stored in the memory to execute the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: It stores a computer program implemented according to the method described in any one of claims 1 to 7 in the form of computer-readable instructions, and when the computer program is called and executed by a computer, the steps included in the corresponding method are executed.