Text abstract model training method and device, electronic equipment and storage medium
By training the text summary model, using the character generation model and the copy network to generate predictive digests, and training the model based on the real digest and predictive digests, the digest bias problem caused by uncertainty in the generative text model is solved, and more accurate summary generation is achieved.
Patent Information
- Application Number
- CN202411855259.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-16
- Publication Date
- 2025-03-28
AI Technical Summary
Due to uncertainty, the existing generative text model is difficult to accurately reflect the correlation between user demands and the original text, resulting in bias in the generated summary.
By obtaining the training sample set, including multiple text sequences and their real digests, input the text sequence into the initial text summary model including the character generation model and the copy network, generate a predictive summary, and train the model based on the real summary and predictive summary to obtain the trained text summary model.
It slows down the uncertainty problem of the generative text model, enhances the correlation between input text and output digest, and improves the accuracy of digest generation.
Smart Images
Figure CN119719362A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of computer technology, and in particular to a training method, device, electronic device and storage medium for a text summarization model. Background Art
[0002] Text summarization technology is one of the most widely used technologies in natural language processing scenarios. For articles or paragraphs, text information can be extracted or summarized based on text summarization technology.
[0003] Currently, a summary of a text can be determined by a generative text model. Specifically, an encoder in the generative text model can encode the text into a vector of a fixed size, and then a decoder in the generative text model decodes the vector to output a summary of the text.
[0004] However, in the above method, due to the uncertainty of the generative text model, it is difficult for the model to accurately reflect the correlation between user demands and the original text, resulting in deviations in the generated summary. Summary of the invention
[0005] The present application provides a training method, device, electronic device and storage medium for a text summary model, which solves the technical problem in the related art that the generated summary is biased due to the uncertainty of the generative text model.
[0006] In a first aspect, the present application provides a training method for a text summary model, comprising: obtaining a training sample set, the training sample set comprising multiple text sequences and a real text summary of each text sequence; inputting the text sequence into an initial text summary model to obtain a predicted text summary of the text sequence, the initial text summary model comprising a character generation model and a replication network, the character generation model being used to determine a generation probability of each character in a character library, the replication network being used to determine a replication probability of each character, the generation probability of each character and the replication probability of each character being used to determine a predicted text summary of the text sequence; based on the real text summary of the text sequence and the predicted text summary of the text sequence, the initial text summary model is trained to obtain a trained text summary model.
[0007] Optionally, the above-mentioned inputting the text sequence into the initial text summary model to obtain the predicted text summary of the text sequence specifically includes: inputting the text sequence into the initial text summary model to obtain the output result of each character in the character generation model, the output result of each character in the replication network, and the probability of each character being replicated in the text sequence; determining the predicted probability of each character based on the output result of each character in the character generation model, the output result of each character in the replication network, and the probability of each character being replicated in the text sequence; and obtaining the predicted text summary of the text sequence based on the predicted probability of each character.
[0008] Optionally, the character generation model includes an encoder and a decoder, and the text sequence is input into the initial text summary model to obtain the probability of each character being copied in the text sequence specifically includes:
[0009] The text sequence is input into the initial text summary model to obtain the last hidden state of the encoder, the average attention weight of the decoder, and the output result of each character in the character generation model; based on the last hidden state of the encoder and the average attention weight of the decoder, the context information of the text sequence is determined; the target feature is input into the replication network to obtain the probability of each character being replicated in the text sequence, and the target feature contains the context information of the text sequence and the output result of each character in the character generation model.
[0010] Optionally, the above-mentioned determination of the predicted probability of each character based on the output result of each character in the character generation model, the output result of each character in the replication network and the probability of each character being replicated in the text sequence specifically includes: normalizing the output result of each character in the character generation model to obtain the generation probability of each character in the character generation model; normalizing the output result of each character in the replication network to obtain the replication probability of each character in the replication network; determining the predicted probability of each character based on the generation probability of each character in the character generation model, the replication probability of each character in the replication network and the probability of each character being replicated in the text sequence.
[0011] Optionally, the above-mentioned normalizing the output results of each character in the replication network to obtain the replication probability of each character in the replication network specifically includes: performing sparse processing on the output results of each character in the replication network to determine M characters, where the M characters are the M characters in the character library with the largest output results in the replication network, and M is a positive integer; normalizing the output results of the M characters in the replication network to obtain the replication probability of each character in the replication network.
[0012] Optionally, the method further includes: obtaining a text sequence to be recognized; inputting the text sequence to be recognized into the trained text summary model to obtain a text summary of the text sequence to be recognized.
[0013] In a second aspect, the present application provides a training device for a text summary model, comprising: an acquisition module and a processing module; the acquisition module is used to acquire a training sample set, the training sample set comprising multiple text sequences and a real text summary of each text sequence; the processing module is used to input the text sequence into an initial text summary model to obtain a predicted text summary of the text sequence, the initial text summary model comprises a character generation model and a replication network, the character generation model is used to determine the generation probability of each character in a character library, the replication network is used to determine the replication probability of each character, the generation probability of each character and the replication probability of each character are used to determine the predicted text summary of the text sequence; the processing module is also used to train the initial text summary model based on the real text summary of the text sequence and the predicted text summary of the text sequence to obtain a trained text summary model.
[0014] In a third aspect, the present application provides an electronic device comprising: a processor and a memory configured to store processor executable instructions; wherein the processor is configured to execute the instructions to implement any one of the optional text summary model training methods in the first aspect above.
[0015] In a fourth aspect, the present application provides a computer-readable storage medium having instructions stored thereon. When the instructions in the computer-readable storage medium are executed by an electronic device, the electronic device is enabled to execute any one of the optional text summarization model training methods in the first aspect described above.
[0016] In a fifth aspect, the present application provides a computer program product, including a computer program or instructions, which, when executed on an electronic device, enables the electronic device to execute any one of the optional text summary model training methods described in the first aspect.
[0017] The training method, device, electronic device and storage medium of the text summary model provided in the present application, after the electronic device obtains the training sample set (including multiple text sequences and the real text summary of each text sequence), the text sequence can be input into the initial text summary model to obtain the predicted text summary of the text sequence. Since the initial text summary model includes a character generation model and a replication network, and the character generation model is used to determine the generation probability of each character in the character library, and the replication network is used to determine the replication probability of each character, the electronic device can determine the predicted text summary of the text sequence based on the generation probability of each character and the replication probability of each character. Finally, the electronic device can train the initial text summary model based on the real text summary and the predicted text summary to obtain a trained text summary model. In the present application, by integrating multiple NLP (including character generation models and replication networks) technologies, the complexity, noise and uncertainty problems of generative text models can be greatly alleviated. In addition, by introducing a replication network, the characters in the input text sequence can be copied to the output of the decoder, which can alleviate the OOV problem to a certain extent and strengthen the correlation between the input text and the output summary. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art are briefly introduced below.
[0019] Figure 1 A flowchart of a method for training a text summarization model provided in an embodiment of the present application;
[0020] Figure 2 A flowchart of another text summarization model training method provided in an embodiment of the present application;
[0021] Figure 3 A flowchart of another text summarization model training method provided in an embodiment of the present application;
[0022] Figure 4 A flowchart of another text summarization model training method provided in an embodiment of the present application;
[0023] Figure 5 A schematic diagram of the structure of a training device for a text summarization model provided in an embodiment of the present application;
[0024] Figure 6 A schematic diagram of the structure of another text summarization model training device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0025] The following will describe in detail the training method, device, electronic device and storage medium of the text summarization model provided in the embodiments of the present application in conjunction with the accompanying drawings.
[0026] In addition, the terms "including" and "having" and any variations thereof mentioned in the description of the present application are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device comprising a series of steps or units is not limited to the listed steps or units, but may optionally include other steps or units that are not listed, or may optionally include other steps or units that are inherent to these processes, methods, products or devices.
[0027] It should be noted that in the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.
[0028] In the description of the present application, unless otherwise specified, “plurality” means two or more.
[0029] The following explains some concepts involved in a training method, device, electronic device and storage medium for a text summarization model provided in an embodiment of the present application.
[0030] Text summarization, also known as abstracting, is the process of converting a long article or text into a shorter, concise summary designed to convey the key message of the text. An abstract usually contains the main idea, key facts, and supporting evidence of the text.
[0031] The T5 model, Text-to-Text Transfer Transformer, is a natural language processing model based on the Transformer structure. The T5 model treats all natural language processing (NLP) tasks as a task that inputs text and outputs text, namely the "Text-to-Text" framework, making the model adaptable to a variety of NLP tasks.
[0032] A copy network, also known as a copy network, is a neural network structure. This network structure is mainly used to copy elements in a sequence or a group of sequences, such as copying a word or phrase in a text sequence to another position. A copy network usually consists of two subnetworks: one for selecting the position of the element to be copied, and the other for copying the element to a specified position.
[0033] Logits, in deep learning, logits usually refer to the raw, unprocessed scores or scores of the output layer of a model, especially in classification problems. Logits are not converted to probability distributions, and are the output of the neural network model at the last layer (fully connected layer or output layer). Specifically, logits are the linear output of the model for each category, which has not been processed by an activation function (such as softmax or sigmoid). These raw scores can be viewed as a measure of the model's confidence or probability for each category, but they do not directly represent probabilities. When performing classification tasks, an activation function is generally applied to the logits to convert them into a probability distribution, so that the score of each category is between 0 and 1, and the sum of the probabilities of all categories is 1. The processed output can be more intuitively interpreted as the probability of each category.
[0034] The last hidden state of the encoder refers to the hidden state of the last time step generated after the encoder processes the entire input sequence in the Seq2Seq model. This state incorporates the information of all input sequences and is usually used as the initial hidden state of the decoder so that the decoder can generate the output sequence based on this information. In the Seq2Seq model, the encoder is responsible for reading the input sequence and encoding it into a fixed-length vector (usually the last hidden state). The decoder uses this vector as the initial state and gradually generates the output sequence. This mechanism enables the model to perform well when processing variable-length inputs and outputs.
[0035] Sparsification is an optimization technique in model training that aims to reduce the number of parameters in the model, thereby improving the computational efficiency of the model and reducing the required storage space. Sparsification is mainly achieved by setting most of the parameters in the model to zero, which are considered redundant after the model training is completed. Sparsification is the process of removing most of the elements in a matrix or vector to make it sparse (i.e., with fewer non-zero elements). In machine learning and data analysis, sparsification can be used to reduce the dimensionality of data and improve the efficiency and accuracy of the algorithm. Common application scenarios include feature selection, image processing, and natural language processing.
[0036] Currently, the core work of operators includes improving customer service quality and enhancing user experience. Analyzing and summarizing the conversations between customer agents and users is an effective way to improve user demands, discover product problems, and improve resolution efficiency. Since there are many problems with the transcribed text of the call (including high complexity, high noise, long length, serious colloquialism, and a lot of redundant information), it is very difficult to manually read and summarize the call content. In addition, the amount of call content generated every day is huge, and it is difficult to achieve the text summary generation process by manual methods alone.
[0037] Text summarization technology is one of the most widely used technologies in natural language processing scenarios and is constantly used in our daily production and life. For articles or paragraphs, text information can be extracted or summarized based on text summarization technology.
[0038] From the perspective of technical implementation, the current mainstream text summarization technology can be divided into two types: one is the extractive text summarization algorithm, and the other is the generative text summarization algorithm.
[0039] Specifically, extractive text summarization algorithms include unsupervised and supervised. Unsupervised extraction does not require parallel corpus for training, and omits the tedious work of manually labeling corpus. And this method is based on statistics, that is, it can maximize the representation ability of the summary sentence to the original document. The supervised extraction method regards text summarization as a binary classification problem, and uses a neural network model to learn the correspondence between sentences and their labels. However, the extractive text summarization algorithm lacks coherence and global perspective, is difficult to process long texts, and easily introduces redundant information, thus affecting the quality of the text summary.
[0040] Generative text summarization algorithms (also called generative text models) usually use a seq2seq framework. The encoder is responsible for encoding the input text into a fixed-size vector that represents (or captures) the semantic information of the input text. The decoder decodes this vector into output text (i.e., a summary of the input text). This algorithm can adaptively select effective contextual information and generate a coherent, consistent text summary word by word. However, the generation results of the generative text summary algorithm are uncontrollable, and there may be a situation where the generated text summary is inconsistent with the summary corresponding to the original text. However, in the agent call scenario, the user's demands and the generated text summary must remain correct.
[0041] Based on the description in the background technology, due to the uncertainty of the generative text model in the related art, it is difficult for the model to accurately reflect the correlation between the user's demands and the original text, resulting in deviations in the generated summary. Based on this, the embodiment of the present application provides a training method, device, electronic device and storage medium for a text summary model, which can greatly alleviate the complexity, noise and uncertainty problems of the generative text model by integrating multiple NLP (including character generation models and replication networks) technologies. In addition, by introducing a replication network, the characters in the input text sequence can be copied to the output of the decoder, which can alleviate the OOV problem to a certain extent and strengthen the correlation between the input text and the output summary.
[0042] Exemplarily, the electronic device that executes the training method of the text summary model provided in the embodiment of the present application can be a mobile phone, a tablet computer, a desktop, a laptop, a handheld computer, a notebook computer, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook, and a cellular phone, a personal digital assistant (personal digital assistant, PDA), an augmented reality (augmented reality, AR)\virtual reality (virtual reality, VR) device, and the embodiment of the present application does not impose any special restrictions on the specific form of the electronic device. It can interact with the user through one or more methods such as a keyboard, a touchpad, a touch screen, a remote control, voice interaction or a handwriting device.
[0043] Optionally, the above-mentioned electronic device can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, network acceleration services (content delivery network, CDN), as well as big data and artificial intelligence platforms.
[0044] The training method, device, electronic device and storage medium of the text summary model provided in the embodiment of the present application are applied to the agent call scenario, specifically the multi-round call scenario between the customer agent and the user. After the electronic device obtains multiple call texts (including the text sequence to be recognized) between the customer agent and the user, it can obtain the text summary of each of the multiple call texts based on the text summary model provided in the embodiment of the present application, and then the electronic device can identify the real needs of the user based on these text summaries, thereby discovering related problems and improving the user's product experience.
[0045] like Figure 1 As shown, the training method of the text summarization model provided in the embodiment of the present application may include S101-S103.
[0046] S101: Obtain a training sample set.
[0047] The training sample set includes multiple text sequences and a real text summary of each text sequence.
[0048] It should be understood that the real text summary of a text sequence is the label of the text sequence, which can represent the real demands of the user. The real demands of the user (ie, the real text summary) need to be consistent with the semantic information represented by the text sequence.
[0049] It is understandable that in the scenario of an agent call, what the electronic device can directly obtain is the telephone recording in this scenario. Due to the complex environment and randomness of the telephone recording, the text transcribed by automatic voice transcription is long and there is a problem of mixed text lines, which makes the subsequent text semantic understanding and summary generation very difficult. In this way, after the electronic device performs word segmentation on the call text, it needs to use regular expressions and part-of-speech analysis to clean and process the call text, clean a large number of invalid words and modal particles, and re-glue erroneous short sentences to form a highly available text segment. In this way, the text sequence in the embodiment of the present application can be obtained.
[0050] S102: Input the text sequence into an initial text summary model to obtain a predicted text summary of the text sequence.
[0051] Among them, the initial text summary model includes a character generation model and a replication network. The character generation model is used to determine the generation probability of each character in the character library, and the replication network is used to determine the replication probability of each character. The generation probability of each character and the replication probability of each character are used to determine the predicted text summary of the text sequence.
[0052] It should be understood that the character library can also be called a word library or a dictionary, and the character library includes multiple characters (tokens). A token refers to the smallest text unit or the smallest semantic unit, and can also be called a word, a subsequence, etc.
[0053] Exemplarily, the above-mentioned character generation model may be a T5 model.
[0054] S103: Based on the real text summary of the text sequence and the predicted text summary of the text sequence, the initial text summary model is trained to obtain a trained text summary model.
[0055] It can be understood that the electronic device can obtain the loss of the initial text summary model based on the real text summary of the text sequence and the predicted text summary of the text sequence in combination with the cross-entropy loss function (the loss is used to characterize the degree of inconsistency between the real text summary and the predicted text summary), and perform gradient descent training through the loss, specifically updating the parameters in the initial text summary model based on the loss to obtain a trained text summary model.
[0056] The technical solution provided by the above embodiment can at least bring the following beneficial effects: It can be known from S101-S103 that after the electronic device obtains the training sample set (including multiple text sequences and the real text summary of each text sequence), the text sequence can be input into the initial text summary model to obtain the predicted text summary of the text sequence. Since the initial text summary model includes a character generation model and a replication network, and the character generation model is used to determine the generation probability of each character in the character library, and the replication network is used to determine the replication probability of each character, the electronic device can determine the predicted text summary of the text sequence based on the generation probability of each character and the replication probability of each character. Finally, the electronic device can train the initial text summary model based on the real text summary and the predicted text summary to obtain a trained text summary model. In the embodiment of the present application, by integrating multiple NLP (including character generation models and replication networks) technologies, the complexity, noise and uncertainty problems of generative text models can be greatly alleviated. Moreover, by introducing the copy network, the characters in the input text sequence can be copied to the output of the decoder, which can alleviate the out of vocabulary (OOV) problem to a certain extent and strengthen the correlation between the input text and the output summary.
[0057] Combination Figure 1 ,like Figure 2 As shown, the above-mentioned inputting the text sequence into the initial text summary model to obtain the predicted text summary of the text sequence specifically includes S1021-S1023.
[0058] S1021. Input the text sequence into the initial text summary model to obtain the output result of each character in the character generation model, the output result of each character in the replication network, and the probability of each character being replicated in the text sequence.
[0059] It should be understood that the output result of a character in the character generation model is the logical value (Logits) of the character in the character generation model, and the output result of a character in the replication network is the logical value (Logits) of the character in the replication network. These output results (or logical values) have not been processed by the activation function, and the corresponding probability distribution can be obtained after processing.
[0060] S1022: Determine the prediction probability of each character based on the output result of each character in the character generation model, the output result of each character in the replication network, and the probability of each character being replicated in the text sequence.
[0061] S1023. Obtain a predicted text summary of the text sequence based on the prediction probability of each character.
[0062] Specifically, after determining the prediction probability of each character, the electronic device will generate (or obtain) a character with the highest prediction probability, and use this character as a character included in the predicted text summary. At the same time, the electronic device will concatenate this character with the previous input (i.e., the text sequence), and after the concatenation is completed, it will be input into the initial text summary model, and then another character will be generated (also selecting the character with the highest prediction probability) until the termination symbol is output. Finally, the electronic device combines these generated characters together to obtain the predicted text summary of the text sequence.
[0063] In the embodiment of the present application, since the output result of each character in the character generation model can represent the generation probability of each character, the output result of each character in the replication network can represent the replication probability of each character. In this way, the predicted probability of each character determined by the electronic device can cover the generation probability of each character, the replication probability of each character, and the probability of each character being replicated in the text sequence, so that the predicted probability of each character in the character library can accurately and effectively reflect the probabilities of these characters from the generation perspective and the replication perspective, and ultimately improve the accuracy of text summary generation.
[0064] Combination Figure 2 ,like Figure 3 As shown, the above-mentioned determination of the prediction probability of each character based on the output result of each character in the character generation model, the output result of each character in the replication network and the probability of each character being replicated in the text sequence may specifically include S1022a-S1022c.
[0065] S1022a, normalizing the output result of each character in the character generation model to obtain the generation probability of each character in the character generation model.
[0066] It should be understood that the process of normalizing the output result of each character in the character generation model by the electronic device is the process of activating the output result (or logical value) of each character in the character generation model through an activation function, and the activation function can specifically be a normalized exponential function (softmax function).
[0067] It can be understood that the generation probability of each character in the character model is the generation probability of each character in the above embodiment.
[0068] Specifically, the generation probability of each of the above characters in the character generation model can satisfy the following formula:
[0069]
[0070] in, represents the generation probability of the i-th character in the character generation model, ti It represents the output result of the i-th character in the character generation model, e is a natural constant, j represents the number of characters included in the character library, 1≤i≤j, i and j are positive integers.
[0071] S1022b, normalizing the output result of each character in the replication network to obtain the replication probability of each character in the replication network.
[0072] In combination with the description of the above embodiment, the process of the electronic device normalizing the output result of each character in the replication network is the process of activating the output result of each character in the replication network through an activation function. The replication probability of each character in the replication network is the replication probability of each character mentioned above.
[0073] S1022c, determining the prediction probability of each character based on the generation probability of each character in the character generation model, the replication probability of each character in the replication network, and the probability of each character being replicated in the text sequence.
[0074] Optionally, for any character in the character library, the electronic device can determine a first value and a second value, and determine the sum of the first value and the second value as the predicted probability of the character. The first value is the product of the probability of the character being generated in the character generation model and the probability of the character being copied in the text sequence, and the second value is the product of the probability of the character being copied in the copy network and the probability of the character being copied in the text sequence.
[0075] In combination with the description of the above embodiment, the prediction probability of each character can satisfy the following formula:
[0076]
[0077] Among them, p i represents the predicted probability of the i-th character, represents the output result of the i-th character in the character generation model, Represents the output result of the i-th character in the replication network, represents the probability that the i-th character is duplicated in the text sequence, represents the generation probability of the i-th character in the character generation model, represents the replication probability of the i-th character in the replication network, where i is a positive integer.
[0078] Combination Figure 3 ,like Figure 4 As shown, the above-mentioned normalization processing is performed on the output result of each character in the replication network to obtain the replication probability of each character in the replication network, which may specifically include S1022b1-S1022b2.
[0079] S1022b1. Perform sparse processing on the output results of each character in the replication network to determine M characters.
[0080] The M characters are the M characters with the largest output results in the replication network in the above character library, and M is a positive integer.
[0081] S1022b2. Normalize the output results of the M characters in the replication network to obtain the replication probability of each character in the replication network.
[0082] In the embodiment of the present application, the electronic device performs sparse processing on the output results of each character in the replication network and normalizes the output results of M characters in the replication network, which can be understood as using a sparse softmax activation function in the replication network. In the process of determining the replication probability and the prediction probability, only the first M characters are retained, and the other characters are set to zero, which can avoid over-learning of the softmax function and increase the replication ability of user demands (i.e., text summaries) during the model training process.
[0083] Based on this, the replication probability of each character in the replication network can satisfy the following formula:
[0084]
[0085] in, represents the replication probability of the i-th character in the replication network, represents the output result of the i-th character in the replication network, e is a natural constant, j represents the number of characters included in the character library, 1≤i≤j, i and j are positive integers, Ω M is the set of output results of the above M characters in the replication network.
[0086] In this way, the predicted probability of each character above can also satisfy the following formula:
[0087]
[0088] Among them, p i represents the predicted probability of the i-th character, represents the output result of the i-th character in the character generation model, Represents the output result of the i-th character in the replication network, represents the probability that the i-th character is duplicated in the text sequence, represents the generation probability of the i-th character in the character generation model, It represents the replication probability of the i-th character in the replication network after the sparse processing, where i is a positive integer.
[0089] It is understandable that the above formula uses This is to implement a gating mechanism that determines whether the text summarization model tends to generate new words (using the generation logic of the character generation model) or to copy words in the text sequence (using the copy logic in the copy network) during the character generation process. When it is high (specifically close to 1), Close to 0, indicating that the text summary model is more inclined to generate new words, which means will hardly be considered. When it is low (specifically close to 0), Close to 1, indicating that the text summary model tends to copy words in the text sequence, which means will be fully considered.
[0090] In one implementation of the embodiment of the present application, the above-mentioned character generation model includes an encoder and a decoder, and the text sequence is input into the initial text summary model to obtain the probability of each character being copied in the text sequence, which may specifically include steps A to C.
[0091] Step A: Input the text sequence into the initial text summary model to obtain the last hidden state of the encoder, the average attention weight of the decoder, and the output result of each character in the character generation model.
[0092] In the embodiment of the present application, the last hidden state of the encoder can be understood as the output result of the encoder, and can also be understood as the initial hidden state of the decoder, which will be input into the decoder as a feature vector, and then the decoder determines the output result of each character in the character generation model. Therefore, the output result of each character in the character generation model can also be understood as the output of the decoder.
[0093] It should be understood that the average attention weight of the decoder is used to characterize the degree of attention that the decoder pays to each position in the text sequence when generating each character.
[0094] Step B: Determine the contextual information of the text sequence based on the last hidden state of the encoder and the average attention weight of the decoder.
[0095] It should be understood that the context information of the text sequence can represent the semantic information of the text sequence.
[0096] Optionally, the electronic device may determine the product of the last hidden state and the average attention weight as the context information of the text sequence.
[0097] Step C: Input the target feature into the replication network to obtain the probability of each character being replicated in the text sequence.
[0098] The target feature includes the context information of the text sequence and the output result of each character in the character model.
[0099] Specifically, the electronic device can concatenate the context information of the text sequence and the output result of each character in the character model (i.e., the output of the decoder) to obtain the target feature. The electronic device then inputs the target feature into the replication network, and after processing with an activation function (e.g., a sigmoid function), the probability of each character being replicated in the text sequence can be obtained.
[0100] In an optional implementation, after the electronic device obtains the trained text summary model (or the initial text summary model is trained), the electronic device can implement the text summary generation process based on the trained text summary model. Specifically, it can be implemented by the following steps D-E.
[0101] Step D: Obtain the text sequence to be recognized.
[0102] Step E: input the text sequence to be recognized into the trained text summary model to obtain the text summary of the text sequence to be recognized.
[0103] In the embodiment of the present application, since a replication network is used in the trained text summary model to improve the ability to replicate characters contained in a text sequence, the uncontrollability of the generative model output can be reduced, and the correlation between user needs (i.e., text summaries) and the original text (i.e., text sequences) can be increased, thereby ensuring the accuracy of the text summary.
[0104] The embodiment of the present application can divide the functional modules of the electronic device etc. according to the above method example. For example, each functional module can be divided according to each function, or two or more functions can be integrated into one processing module. The above integrated module can be implemented in the form of hardware or in the form of software functional modules. It should be noted that the division of modules in the embodiment of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation.
[0105] In the case of dividing each functional module into corresponding functional modules, Figure 5 A possible structural diagram of a training device for a text summary model involved in the above embodiment is shown. Figure 5 As shown, the training device 20 for the text summary model may include: an acquisition module 201 and a processing module 202.
[0106] The acquisition module 201 is used to acquire a training sample set, where the training sample set includes multiple text sequences and a real text summary of each text sequence.
[0107] The processing module 202 is used to input the text sequence into the initial text summary model to obtain a predicted text summary of the text sequence. The initial text summary model includes a character generation model and a replication network. The character generation model is used to determine the generation probability of each character in the character library, and the replication network is used to determine the replication probability of each character. The generation probability of each character and the replication probability of each character are used to determine the predicted text summary of the text sequence.
[0108] The processing module 202 is further used to train the initial text summary model based on the real text summary of the text sequence and the predicted text summary of the text sequence to obtain a trained text summary model.
[0109] Optionally, the above device 20 further includes a determination module 203 .
[0110] The processing module 202 is specifically used to input the text sequence into the initial text summary model to obtain the output result of each character in the character generation model, the output result of each character in the replication network, and the probability of each character being replicated in the text sequence.
[0111] The determination module 203 is used to determine the prediction probability of each character based on the output result of each character in the character generation model, the output result of each character in the replication network, and the probability of each character being replicated in the text sequence.
[0112] The processing module 202 is further specifically configured to obtain a predicted text summary of the text sequence based on the predicted probability of each character.
[0113] Optionally, the character generation model includes an encoder and a decoder, and the text sequence is input into the initial text summary model to obtain the probability of each character being copied in the text sequence, including:
[0114] The processing module 202 is further specifically used to input the text sequence into the initial text summary model to obtain the last hidden state of the encoder, the average attention weight of the decoder, and the output result of each character in the character generation model.
[0115] The determination module 203 is further used to determine the context information of the text sequence based on the last hidden state of the encoder and the average attention weight of the decoder.
[0116] The processing module 202 is also specifically used to input the target feature into the replication network to obtain the probability of each character being replicated in the text sequence. The target feature includes the context information of the text sequence and the output result of each character in the character generation model.
[0117] Optionally, the processing module 202 is further configured to normalize the output result of each character in the character generation model to obtain a generation probability of each character in the character generation model.
[0118] The processing module 202 is further used to normalize the output result of each character in the replication network to obtain the replication probability of each character in the replication network.
[0119] The determination module 203 is specifically used to determine the prediction probability of each character based on the generation probability of each character in the character generation model, the replication probability of each character in the replication network, and the probability of each character being replicated in the text sequence.
[0120] Optionally, the determination module 203 is further used to perform a sparse processing on the output result of each character in the replication network to determine M characters, where the M characters are the M characters in the character library with the largest output results in the replication network, and M is a positive integer.
[0121] The processing module 202 is further specifically configured to normalize the output results of the M characters in the replication network to obtain the replication probability of each character in the replication network.
[0122] Optionally, the acquisition module 201 is further configured to acquire a text sequence to be recognized.
[0123] The processing module 202 is further configured to input the to-be-recognized text sequence into the trained text summary model to obtain a text summary of the to-be-recognized text sequence.
[0124] In the case of an integrated unit, Figure 6 A possible structural diagram of a training device for a text summary model involved in the above embodiment is shown. Figure 6 As shown, the training device 30 for the text summary model may include: a processing module 301 and a communication module 302. The processing module 301 may be used to control and manage the actions of the training device 30 for the text summary model. The communication module 302 may be used to support the communication between the training device 30 for the text summary model and other entities. Figure 6 As shown, the text summary model training device 30 may further include a storage module 303 for storing program codes and data of the text summary model training device 30 .
[0125] The processing module 301 may be a processor or a controller. The communication module 302 may be a transceiver, a transceiver circuit or a communication interface, etc. The storage module 303 may be a memory.
[0126] When the processing module 301 is a processor, the communication module 302 is a transceiver, and the storage module 303 is a memory, the processor, the transceiver, and the memory may be connected via a bus. The bus may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc.
[0127] It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.
[0128] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.
[0129] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.
[0130] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0131] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using a software program, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When loading and executing a computer program instruction on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, a computer, a server or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (Digital Subscriber Line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that a computer can access or may contain one or more servers, data centers and other data storage devices that can be integrated with a medium. The available medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium (eg, a DVD), or a semiconductor medium (eg, a solid state disk (SSD)).
[0132] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.
Claims
1. A training method for a text summarization model, characterized in that: include: Obtain a training sample set, wherein the training sample set includes multiple text sequences and a real text summary of each text sequence; Inputting the text sequence into an initial text summary model to obtain a predicted text summary of the text sequence, wherein the initial text summary model includes a character generation model and a replication network, wherein the character generation model is used to determine a generation probability of each character in a character library, and the replication network is used to determine a replication probability of each character, and the generation probability of each character and the replication probability of each character are used to determine a predicted text summary of the text sequence; Based on the real text summary of the text sequence and the predicted text summary of the text sequence, the initial text summary model is trained to obtain a trained text summary model.
2. The method according to claim 1, characterized in that The step of inputting the text sequence into an initial text summary model to obtain a predicted text summary of the text sequence includes: Inputting the text sequence into the initial text summarization model, obtaining the output result of each character in the character generation model, the output result of each character in the replication network, and the probability of each character being replicated in the text sequence; Determine the prediction probability of each character based on the output result of each character in the character generation model, the output result of each character in the replication network, and the probability of each character being replicated in the text sequence; Based on the prediction probability of each character, a predicted text summary of the text sequence is obtained.
3. The method according to claim 2, characterized in that The character generation model includes an encoder and a decoder, and the text sequence is input into the initial text summary model to obtain the probability of each character being copied in the text sequence, including: Input the text sequence into the initial text summarization model to obtain the last hidden state of the encoder, the average attention weight of the decoder, and the output result of each character in the character generation model; Determining context information of the text sequence based on a last hidden state of the encoder and an average attention weight of the decoder; The target feature is input into the replication network to obtain the probability of each character being replicated in the text sequence, wherein the target feature includes context information of the text sequence and the output result of each character in the character generation model.
4. The method according to claim 2, characterized in that: Determining the prediction probability of each character based on the output result of each character in the character generation model, the output result of each character in the replication network, and the probability of each character being replicated in the text sequence includes: Normalizing the output results of each character in the character generation model to obtain the generation probability of each character in the character generation model; Normalizing the output results of each character in the replication network to obtain the replication probability of each character in the replication network; The prediction probability of each character is determined based on the generation probability of each character in the character generation model, the replication probability of each character in the replication network, and the probability of each character being replicated in the text sequence.
5. The method according to claim 4, characterized in that The normalizing the output result of each character in the replication network to obtain the replication probability of each character in the replication network includes: Performing a sparse processing on the output result of each character in the replication network to determine M characters, where the M characters are the M characters in the character library with the largest output results in the replication network, and M is a positive integer; The output results of the M characters in the replication network are normalized to obtain the replication probability of each character in the replication network.
6. The method according to any one of claims 1 to 5, characterized in that The method further comprises: Get the text sequence to be recognized; The text sequence to be recognized is input into the trained text summary model to obtain a text summary of the text sequence to be recognized.
7. A training device for a text summarization model, characterized in that: include: Acquisition module and processing module; The acquisition module is used to acquire a training sample set, wherein the training sample set includes a plurality of text sequences and a real text summary of each text sequence; The processing module is used to input the text sequence into an initial text summary model to obtain a predicted text summary of the text sequence, wherein the initial text summary model includes a character generation model and a replication network, wherein the character generation model is used to determine a generation probability of each character in a character library, and the replication network is used to determine a replication probability of each character, and the generation probability of each character and the replication probability of each character are used to determine the predicted text summary of the text sequence; The processing module is further used to train the initial text summary model based on the real text summary of the text sequence and the predicted text summary of the text sequence to obtain a trained text summary model.
8. An electronic device, characterized in that: The electronic device comprises: processor; a memory configured to store instructions executable by the processor; The processor is configured to execute the instructions to implement the training method of the text summarization model as described in any one of claims 1-6.
9. A computer-readable storage medium having instructions stored thereon, characterized in that: When the instructions in the computer-readable storage medium are executed by an electronic device, the electronic device is enabled to execute the training method for a text summarization model as described in any one of claims 1 to 6.
10. A computer program product comprising instructions, characterized in that When the instruction is executed on an electronic device, the electronic device executes the training method for a text summarization model as described in any one of claims 1 to 6.