Training method, device, electronic device and storage medium for text summarization model

Abstract

The present application provides a training method, device, electronic device, and storage medium for a text summarization model, comprising: obtaining a sample set including multiple training samples; encoding the sample text, positive sample summary, and negative sample summary in the training sample using a first encoder of the text summarization model to obtain a first sample text vector, a positive sample summary vector, and a negative sample summary vector; decoding the first sample text vector using a decoder of the text summarization model to obtain a target summary; determining a first loss of the text summarization model based on the target summary and the positive sample summary, and determining a second loss of the text summarization model based on the first sample text vector, the positive sample summary vector, and the negative sample summary vector; and updating the model parameters of the text summarization model by combining the first loss and the second loss. In this way, the model's ability to understand the differences between different summaries can be enhanced, thereby improving the performance of the model.

Description

Technical Field

[0001] The present application relates to Internet technology, and in particular to a training method, device, electronic device, and storage medium for a text summarization model. Background Art

[0002] Automatically summarizing text has crucial applications in many NLP (Natural Language Processing) fields, such as news headline generation, meeting minutes, topic generation for short social media posts, and intelligent customer service tasks. Text summarization methods are generally categorized as extractive and generative. Extractive summarization typically involves selecting keywords and key sentences from the original text to form a summary. Generative summarization allows for the inclusion of new words or phrases, offering greater flexibility. Its implementation is generally based on a sequence-to-sequence (Seq2Seq) model.

[0003] When training a generative text summarization model, the model parameters are typically optimized using the maximum likelihood estimation training objective (maximizing the probability of generating the target summary) between a source document and its corresponding target summary. However, with this training process, the text summarization model only learns how to generate the correct summary, but fails to discern the advantages and disadvantages of different summaries. Summary of the Invention

[0004] The embodiments of the present application provide a training method, apparatus, electronic device, computer-readable storage medium, and computer program product for a text summarization model, which can enhance the model's ability to understand the differences between different summaries, thereby improving the performance of the model.

[0005] The technical solution of the embodiment of the present application is implemented as follows:

[0006] An embodiment of the present application provides a method for training a text summarization model, wherein the text summarization model includes a first encoder and a decoder, and the method includes:

[0007] Acquire a sample set including a plurality of training samples, wherein the training samples include sample text, a positive sample summary corresponding to the sample text, and a negative sample summary corresponding to the sample text;

[0008] Encoding the sample text, the positive sample summary, and the negative sample summary in the training sample by the first encoder respectively to obtain a first sample text vector, a positive sample summary vector, and a negative sample summary vector;

[0009] Decoding the first sample text vector using the decoder to obtain a target summary;

[0010] Determining a first loss of the text summarization model based on the target summary and the positive sample summary, and determining a second loss of the text summarization model based on the first sample text vector, the positive sample summary vector, and the negative sample summary vector;

[0011] The model parameters of the text summarization model are updated in combination with the first loss and the second loss.

[0012] An embodiment of the present application provides a training device for a text summarization model, wherein the text summarization model includes a first encoder and a decoder, and the device includes:

[0013] An acquisition module, configured to acquire a sample set comprising a plurality of training samples, wherein the training samples include sample text, a positive sample summary corresponding to the sample text, and a negative sample summary corresponding to the sample text;

[0014] an encoding module, configured to encode the sample text in the training sample, the positive sample summary, and the negative sample summary respectively through the first encoder to obtain a first sample text vector, a positive sample summary vector, and a negative sample summary vector;

[0015] a decoding module, configured to decode the first sample text vector through the decoder to obtain a target summary;

[0016] a determination module, configured to determine a first loss of the text summarization model based on the target summary and the positive sample summary, and determine a second loss of the text summarization model based on the first sample text vector, the positive sample summary vector, and the negative sample summary vector;

[0017] An updating module is configured to update model parameters of the text summarization model based on the first loss and the second loss.

[0018] In the above scheme, the update module is further used to determine a corresponding first error signal based on the first loss, backpropagate the first error signal in the text summary model, and update the model parameters of the decoder and the first encoder during the propagation process; determine a corresponding second error signal based on the second loss, backpropagate the second error signal in the text summary model, fix the model parameters of the decoder during the propagation process, and update the model parameters of the first encoder.

[0019] In the above scheme, the update module is further used to obtain the weights corresponding to the first loss and the second loss respectively; based on the weights corresponding to the first loss and the second loss, perform weighted summation of the first loss and the second loss to obtain a target loss; and update the model parameters of the text summary model based on the target loss.

[0020] In the above scheme, the text summary model also includes a second encoder, and the encoding module is further used to encode the sample text through the second encoder to obtain a second sample text vector; obtain the weights of the first encoder and the second encoder respectively; based on the weights of the first encoder and the second encoder, perform weighted summation on the first sample text vector and the second sample text vector to obtain a target sample text vector; the decoding module is further used to decode the target sample text vector through the decoder to obtain a target summary.

[0021] In the above scheme, the determination module is further used to determine the first cosine similarity between the first sample text vector and the positive sample summary vector, and determine the second cosine similarity between the first sample text vector and the negative sample summary vector; obtain the similarity coefficient between the positive sample summary and the negative sample summary; wherein the similarity coefficient is used for the similarity between the positive sample summary and the negative sample summary; based on the first cosine similarity, the second cosine similarity and the obtained similarity coefficient, determine the second loss of the text summary model.

[0022] In the above scheme, the determination module is further used to determine the difference between the target summary and the positive sample summary; based on the difference, determine the probability of generating the positive sample summary based on the sample text; based on the probability, determine the first loss of the text summarization model.

[0023] In the above scheme, the device also includes a pre-training module, which is used to obtain a pre-training sample set including multiple pre-training samples, wherein the pre-training samples include pre-training sample text and corresponding pre-training sample summaries; encode the pre-training sample text through the first encoder to obtain a corresponding pre-training sample text vector; decode the pre-training sample text vector through the decoder to obtain a predicted summary; obtain the difference between the predicted summary and the pre-training sample summary, and update the model parameters of the text summary model based on the difference.

[0024] In the above scheme, the acquisition module is also used to construct multiple positive samples, and the positive samples include: the sample text and the positive sample summary corresponding to the sample text; for each positive sample, select another positive sample, and use the positive sample summary of the other selected positive sample as the negative sample summary, and form the training sample together with the positive sample; obtain multiple training samples corresponding to the positive samples to obtain the sample set.

[0025] In the above scheme, the device also includes an application module, which is used to encode the target text through the first encoder to obtain a corresponding text vector; and decode the text vector through the decoder to obtain a text summary corresponding to the target text.

[0026] An embodiment of the present application provides an electronic device, including:

[0027] a memory for storing executable instructions;

[0028] The processor is configured to implement the text summarization model training method provided in the embodiment of the present application when executing the executable instructions stored in the memory.

[0029] An embodiment of the present application provides a computer-readable storage medium storing executable instructions for causing a processor to execute instructions to implement the text summarization model training method provided in the embodiment of the present application.

[0030] The present invention provides a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the electronic device to perform the text summarization model training method provided in the present invention.

[0031] The embodiments of the present application have the following beneficial effects:

[0032] In the process of training the text summarization model, the training samples are divided into sample text, positive sample summaries corresponding to the sample text, and negative sample summaries corresponding to the sample text. By training the training samples containing the sample text, the positive sample summaries, and the negative sample summaries, a first loss and a second loss are obtained, and then the model parameters are updated based on the first loss and the second loss. In this way, the text summary model's ability to understand the differences between different summaries is enhanced, so that the text summary generation model can learn to obtain better text representations from the training samples, thereby making the text summary model have better performance and improving the accuracy of text summary extraction. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 1 is an optional architectural diagram of a text summarization model training system 100 provided in an embodiment of the present application;

[0034] Figure 2 is a schematic structural diagram of an electronic device provided in an embodiment of the present application;

[0035] Figure 3Schematic diagram of the training method of the text summarization model provided in the embodiment of the present application;

[0036] Figure 4 This is an optional flowchart of the training method of the text summarization model provided in the embodiment of the present application;

[0037] Figure 5 1 is a flow chart of determining the first loss by the text summarization model provided in an embodiment of the present application;

[0038] Figure 6 1 is a flow chart of determining the second loss by the text summarization model provided in an embodiment of the present application;

[0039] Figure 7 This is an optional schematic diagram of updating the model parameters of the text summarization model by combining the first loss and the second loss provided in an embodiment of the present application;

[0040] Figure 8 This is an optional schematic diagram of updating the model parameters of the text summarization model by combining the first loss and the second loss provided in an embodiment of the present application;

[0041] Figure 9 This is an optional flowchart of the pre-training of the text summarization model provided in the embodiment of the present application;

[0042] Figure 10 This is an optional flowchart of the text summary extraction process provided in the embodiment of the present application;

[0043] Figure 11 This is an optional flowchart of the training method of the text summarization model provided in the embodiment of the present application;

[0044] Figure 12 This is a comparison chart of the effects of the traditional text summarization model provided in the embodiments of this application and the text summarization model provided in this application. DETAILED DESCRIPTION

[0045] In order to make the purpose, technical solutions and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limiting this application. All other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0046] In the following description, reference is made to “some embodiments”, which describes a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0047] In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0048] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0049] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0050] 1) Contrastive learning: A training method that learns the feature representation of a sample by comparing the data with positive and negative samples in the feature space.

[0051] 2) Text summarization: Text processing technology that converts text into a short summary containing key information.

[0052] During the implementation of the embodiments of this application, the inventors discovered that the related technologies have the following problems:

[0053] Existing text summarization technologies can be mainly divided into two categories: extractive and generative. Extractive text summarization technology selects keywords and key sentences from the original text to form a summary. Extractive text summarization technology has certain guarantees in terms of grammar and syntax, but it also faces certain problems, such as incorrect content selection, poor coherence, and poor flexibility. With the development of deep neural network technology, generative text summarization technology has also achieved good results in this task. Generative text summarization technology allows new words or phrases to be included in the summary and has high flexibility. Among generative text summarization technologies, sequence-to-sequence (Seq2Seq) models are widely used in generative summary tasks and have achieved certain results.

[0054] Specifically, existing generative text summarization techniques are mostly trained using supervised training data. This training data typically consists of a large number of source document and target summary text pairs. Sequence-to-sequence (Seq2Seq) models are trained using MLE (maximum likelihood estimation) training objectives on such datasets. During prediction, only the document to be summarized needs to be input to generate the corresponding target summary.

[0055] Existing text summarization models typically train by optimizing their parameters using maximum likelihood estimation (maximizing the probability of generating the target summary) between a source document and its corresponding target summary. This training process only allows the model to see the target summary corresponding to the source document. This means the model only learns how to generate the correct summary during training, but fails to discern the relative merits of different summaries.

[0056] Based on this, this embodiment provides a training method, device, electronic device, computer-readable storage medium and computer program product for a text summarization model. Incorporating the differences in quality between different summaries during the training process can enable the model to better understand the input text and obtain better text representation, thereby making the text summarization model have better performance.

[0057] The following describes an exemplary application of the training device for the text summarization model provided in the embodiments of the present application. The training device for the text summarization model provided in the embodiments of the present application can be implemented as various types of user terminals such as laptops, tablet computers, desktop computers, set-top boxes, mobile devices (for example, mobile phones, portable music players, personal digital assistants, dedicated messaging devices, portable gaming devices), voice interaction devices, smart home appliances, and vehicle-mounted terminals, and can also be implemented as a server.

[0058] See also Figure 1 , Figure 1 This is a schematic diagram of an optional architecture of a text summarization model training system 100 provided in an embodiment of the present application. A terminal (terminal 400 is shown as an example) is connected to a server 200 via a network 300. Network 300 can be a wide area network (WAN), a local area network (LAN), or a combination of the two. Terminal 400 and server 200 are interconnected via a wired or wireless network.

[0059] The terminal 400 is configured to obtain and send a sample set including a plurality of training samples to the server 200, wherein the training samples include a sample text, a positive sample summary corresponding to the sample text, and a negative sample summary corresponding to the sample text;

[0060] The server 200 is configured to receive a sample set sent by the terminal 400; encode the sample text, the positive sample summary, and the negative sample summary in the training sample using a first encoder included in the text summary model to obtain a first sample text vector, a positive sample summary vector, and a negative sample summary vector; decode the first sample text vector using a decoder included in the text summary model to obtain a target summary; determine a first loss of the text summary model based on the target summary and the positive sample summary, and determine a second loss of the text summary model based on the first sample text vector, the positive sample summary vector, and the negative sample summary vector; and update model parameters of the text summary model based on the first loss and the second loss.

[0061] In some embodiments, the terminal 400 may also be provided with a text summary client 400-1, and the client 400-1 sends a request for obtaining a target text summary to the server 200. The server 200 then obtains the target text and encodes the target text through the first encoder of the text summary model to obtain a corresponding text vector; then the text vector is decoded through the decoder of the text summary model to obtain a text summary corresponding to the target text; and finally, the extracted target text summary is sent to the client 400-1.

[0062] In some embodiments, the server 200 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The terminal 400 can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a set-top box, and a mobile device (for example, a mobile phone, a portable music player, a personal digital assistant, a dedicated messaging device, a portable gaming device, an intelligent speaker, and a smart watch), etc., but is not limited thereto. The terminal device and the server can be directly or indirectly connected by wired or wireless communication, which is not limited in the embodiments of the present application.

[0063] See also Figure 2 , Figure 2 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. In practical applications, the electronic device can be Figure 1 The server 200 or terminal 400 shown, see Figure 2 , Figure 2 The electronic device shown includes: at least one processor 410, a memory 450, at least one network interface 420 and a user interface 430. The various components in the terminal 400 are coupled together via a bus system 440. It is understood that the bus system 440 is used to achieve connection and communication between these components. In addition to including a data bus, the bus system 440 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, the bus system 440 is not shown in FIG. Figure 2 Various buses are labeled as bus system 440 .

[0064] The processor 410 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0065] The user interface 430 includes one or more output devices 431 that enable presentation of media content, including one or more speakers and / or one or more visual display screens. The user interface 430 also includes one or more input devices 432, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0066] The memory 450 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard drives, optical drives, etc. The memory 450 may optionally include one or more storage devices that are physically remote from the processor 410.

[0067] The memory 450 includes volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 450 described in the embodiments of the present application is intended to include any suitable type of memory.

[0068] In some embodiments, the memory 450 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplified below.

[0069] Operating system 451, including system programs for processing various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, and driver layer, which are used to implement various basic services and process hardware-based tasks;

[0070] A network communication module 452 for reaching other computing devices via one or more (wired or wireless) network interfaces 420 , exemplary network interfaces 420 including Bluetooth, WiFi, and USB;

[0071] a presentation module 453 for enabling presentation of information via one or more output devices 431 (e.g., a display screen, a speaker, etc.) associated with the user interface 430 (e.g., a user interface for operating peripheral devices and displaying content and information);

[0072] The input processing module 454 is configured to detect one or more user inputs or interactions from one of the one or more input devices 432 and to translate the detected inputs or interactions.

[0073] In some embodiments, the training device for the text summarization model provided in the embodiments of the present application can be implemented in software. Figure 2 A training device 455 of a text summary model stored in a memory 450 is shown, which can be software in the form of a program and plug-in, etc., including the following software modules: an acquisition module 4551, an encoding module 4552, a decoding module 4553, a determination module 4554 and an update module 4555. These modules are logical and can therefore be arbitrarily combined or further split according to the functions implemented.

[0074] In other embodiments, the training device of the text summarization model provided in the embodiments of the present application can be implemented in hardware. As an example, the training device of the text summarization model provided in the embodiments of the present application can be a processor in the form of a hardware decoding processor, which is programmed to execute the training method of the text summarization model provided in the embodiments of the present application. For example, the processor in the form of a hardware decoding processor can adopt one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0075] In some embodiments, the terminal or server can implement the training method of the text summary model provided in the embodiment of the present application by running a computer program. For example, the computer program can be a native program or software module in the operating system; it can be a local (Native) application (APP, Application), that is, a program that needs to be installed in the operating system to run, such as an instant messaging APP, a web browser APP; it can also be a small program, that is, a program that can be run only by downloading it into a browser environment; it can also be a small program that can be embedded in any APP. In short, the above-mentioned computer program can be an application, module or plug-in in any form.

[0076] Based on the above description of the training system and electronic device of the text summary model provided by the embodiment of the present application, the following describes the training method of the text summary model provided by the embodiment of the present application. In actual implementation, the training method of the text summary model provided by the embodiment of the present application can be implemented by the terminal or the server alone, or by the terminal and the server in collaboration, so that Figure 1The server 200 in the embodiment of the present application alone performs the training method of the text summary model as an example for explanation. Figure 3 , Figure 3 This is a flow chart of the training method of the text summarization model provided in the embodiment of the present application, which will be combined with Figure 3 The steps shown are explained.

[0077] In step 101 , the server obtains a sample set including a plurality of training samples, wherein the training samples include sample text, a positive sample summary corresponding to the sample text, and a negative sample summary corresponding to the sample text.

[0078] In actual implementation, you can first obtain multiple web page data from the Internet, then perform data cleaning on the multiple web page data to obtain cleaned data text, and then construct a training sample set for training the model based on the cleaned text data.

[0079] It should be noted that data cleaning of web page data is the process of removing invalid information from the web page data. Web page data can, for example, be encyclopedia pages on a webpage, novel pages on a novel website, or news pages on a news website. Here, data cleaning of web page data to obtain cleaned text data includes removing URLs and tags from the web page data to obtain cleaned text data. For example, URLs in web page data can include URLs and hyperlinks, and tags in web page data can include HTML tags. These are symbols not required for training a text summarization model and are therefore not useful for training the model. Therefore, these symbols need to be removed to achieve data cleaning. In some embodiments, other irrelevant symbolic content from the web page data can also be removed. Here, data cleaning of web page data to obtain cleaned text data includes obtaining the subject content of the web page data; and based on the subject content of the web page data, removing symbolic content from the web page data that is irrelevant to the subject content to obtain cleaned text data. For example, the subject content of a web page data may be social news, but the web page data also includes advertising information. This advertising information is irrelevant to the subject content and also needs to be removed.

[0080] In actual implementation, after obtaining the cleaned text data, a training sample set for training the model is constructed based on the cleaned text data. The process of constructing a training sample set for training the model based on the cleaned text data specifically includes constructing multiple positive samples, each of which includes: sample text and a positive sample summary corresponding to the sample text; selecting one other positive sample for each positive sample, using the positive sample summary of the selected other positive sample as a negative sample summary, and forming a training sample with the positive sample; and obtaining training samples corresponding to the multiple positive samples to obtain a sample set.

[0081] As an example, the cleaned text data can be multiple news articles included in a web page data. Each news article includes a news title and a news body. The news body will be used as the sample text, the news title will be used as the positive sample summary of the corresponding news body, and other news titles will be the negative sample summaries corresponding to the news body.

[0082] In some embodiments, the process of constructing a training sample set for training a model based on the cleaned text data can also be to construct multiple positive samples, where the positive samples include: sample text, and a positive sample summary corresponding to the sample text; for each positive sample, randomly select a sentence from the sample text as a negative sample summary, which constitutes a training sample together with the positive sample; obtain training samples corresponding to multiple positive samples to obtain a sample set.

[0083] Continuing with the above example, the news body will be used as the sample text, the news title will be used as the positive sample summary of the corresponding news body, and then a sentence randomly selected from the news body can be the negative sample summary corresponding to the news body.

[0084] In this way, a sample set including multiple training samples is automatically generated through data cleaning, thereby greatly improving the efficiency of sample set generation and reducing the labor cost generated by manual setting.

[0085] Step 102: Encode the sample text, positive sample summary, and negative sample summary in the training sample respectively through a first encoder to obtain a first sample text vector, a positive sample summary vector, and a negative sample summary vector.

[0086] In actual implementation, encoding the sample text, positive sample summary and negative sample summary in the training sample by the first encoder includes: first, segmenting the sample text, positive sample summary and negative sample summary in the training sample by the first encoder to obtain word sequences, wherein the word sequences obtained here include word vectors, part-of-speech vectors and event keyword vectors of each word, wherein the word vector may include a vector represented individually or a vector represented in a distributed manner, the part-of-speech vector represents the part-of-speech of the word, and the part-of-speech vector may be a multi-dimensional vector that maps the part-of-speech tag of the current word to a multi-dimensional vector, and the same part-of-speech is initialized with the same vector. The part-of-speech vector is introduced because some verbs and nouns may contain richer event information; and the event keyword vector may represent whether the word is an event keyword. Event keywords may include keywords of the sample text, as well as words and some verbs that represent entities that can better reflect event information. In the case where the sample text includes event content, for example, the sample text is news, the event keywords therein may be marked by the event keyword vector. For example, if a word in the sample text is an event keyword, its event keyword vector may be represented by vector [1], otherwise it may be represented by vector [0].

[0087] It should be noted that event keywords include words determined through the following steps: determining whether the current word is a verb or represents an entity; in response to determining whether the current word is a verb or represents an entity, determining whether the current word is a keyword in a sample text, wherein the keywords in the sample text include words obtained by keyword extraction of the sample text; in response to determining that the current word is a keyword in the sample text, determining that the current word is an event keyword.

[0088] In actual implementation, after determining the word vector, part-of-speech vector, and event keyword vector for each word in the word sequence, the first sample text vector, the positive sample summary vector, and the negative sample summary vector are obtained. For example, the word vector, part-of-speech vector, and event keyword vector for each word can be concatenated or merged through other merging operations to obtain the corresponding first sample text vector, positive sample summary vector, and negative sample summary vector, respectively.

[0089] Step 103: decode the first sample text vector through a decoder to obtain a target summary.

[0090] In actual implementation, a decoding function included in the decoder obtains the probability distribution of the first sample text vector being a word in the dictionary, and selects words based on the probability to obtain a summary of the target text. Here, the decoding function may include a softmax (normalized exponential function) or the like.

[0091] In some embodiments, the text summarization model further includes a second encoder, here, see Figure 4 , Figure 4 This is an optional flow chart of the training method of the text summarization model provided in the embodiment of the present application, based on Figure 3 After step 103, you may also perform the following:

[0092] Step 1031: Encode the sample text through a second encoder to obtain a second sample text vector.

[0093] In actual implementation, the second encoder segments the sample text in the training sample to obtain a word sequence. The obtained word sequence includes the word vector, part-of-speech vector, and event keyword vector for each word in the word sequence. After determining the word vector, part-of-speech vector, and event keyword vector for each word in the word sequence, the second sample text vector is obtained. For example, the word vector, part-of-speech vector, and event keyword vector for each word can be concatenated or merged through other merging operations to obtain the corresponding second sample text vector.

[0094] Step 1032: Obtain the weights of the first encoder and the second encoder respectively.

[0095] In actual implementation, since the encoding capabilities of each encoder are different, different weights can be set based on encoders with different encoding capabilities. For example, an encoder with better encoding capabilities can be set with a higher weight, and an encoder with poorer encoding capabilities can be set with a lower weight.

[0096] Step 1033: Based on the weights of the first encoder and the second encoder, perform weighted summation on the first sample text vector and the second sample text vector to obtain a target sample text vector.

[0097] In actual implementation, after obtaining the corresponding weights of the first encoder and the second encoder, the first sample text vector encoded by the first encoder and the second sample text vector encoded by the second encoder can be weightedly summed based on the corresponding weights to obtain the target sample text vector.

[0098] It should be noted that in addition to performing weighted summation on the first sample text vector and the second sample text vector to obtain the target sample text vector, the first positive sample summary vector encoded by the first encoder and the second positive sample summary vector encoded by the second encoder can also be weighted summed to obtain the target positive sample summary vector, and the first negative sample summary vector encoded by the first encoder and the second negative sample summary vector encoded by the second encoder can be weighted summed to obtain the target negative sample summary vector.

[0099] In actual implementation, after obtaining the target sample text vector, the target sample text vector can be decoded through a decoder to obtain a target summary. Specifically, the decoder includes a decoding function to obtain the probability distribution of the target sample text vector being a word in the dictionary. The word is selected based on the probability to obtain a summary of the sample text. Here, the decoding function can include a softmax (normalized exponential function) or the like.

[0100] It should be noted that, in some embodiments, a text summary model weighted by an attention mechanism can also be used. The attention mechanism (Attention Mechanism) originates from the study of human vision, and its core goal is to select information that is more critical to the current task goal from a large amount of information. The text summary model weighted by the attention mechanism does not require the encoder to encode all input information into a vector of fixed length. On the contrary, at this time, the encoder needs to encode the input text into a sequence of vectors, and during decoding, each step will selectively select a subset from the vector for further processing. In this way, when generating each output, it is possible to fully utilize the information carried by the first sample text vector, the positive sample summary vector, and the negative sample summary vector.

[0101] Step 104 : determining a first loss of the text summarization model based on the target summary and the positive sample summary, and determining a second loss of the text summarization model based on the first sample text vector, the positive sample summary vector, and the negative sample summary vector.

[0102] In actual implementation, after determining the target summary and positive sample summary, the first loss of the text summary model will be determined based on the target summary and the positive sample summary. Figure 5 , Figure 5 This is a flow chart of determining the first loss by the text summarization model provided in the embodiment of the present application, based on Figure 3 , based on the target summary and the positive sample summary, the process of determining the first loss of the text summarization model includes:

[0103] Step 1041a: Determine the difference between the target summary and the positive sample summary.

[0104] In actual implementation, after obtaining the target summary, the difference between the target summary and the positive sample summary is compared. Specifically, since the target summary is the predicted result of summary extraction of the sample text in the training sample, each target summary consists of the corresponding sample text. Each target summary is compared with the positive sample summary corresponding to the sample text to determine the difference between the target summary and the corresponding positive sample summary. Here, determining the difference between the target summary and the corresponding positive sample summary is the difference between the words that constitute the target summary and the positive sample summary.

[0105] Step 1042a: Based on the differences, determine the probability of generating a positive sample summary based on the sample text.

[0106] In actual implementation, based on the difference between the target summary and the positive sample summary, the probability of generating a positive sample summary based on the sample text is determined. Specifically, based on the difference between the words that constitute the target summary and the positive sample summary, the generation probability of each word in the positive sample summary is determined, and then the generation probability of each word in the positive sample summary is multiplied to determine the probability of generating a positive sample summary based on the sample text. For example, for a sample text, if there are m words, the probability of generating the positive sample summary is expressed as formula (1), that is,

[0107]

[0108] In formula (1), y is the positive sample summary, x is the sample text, h(x) represents the probability of generating a certain word for the input sample text, and y (i) It represents the probability of decoding a certain word, and θ is the set of all relevant parameters of the text summary model. It should be noted that for a sample text, the probability P(y|x) of generating a positive sample summary based on the sample text is specifically P(y 1|x)P(y 2 |x)……P(y i |x).

[0109] Step 1043a: Determine the first loss of the text summarization model based on the probability.

[0110] In actual implementation, after determining the probability of generating a positive sample summary based on the sample text, the first loss of the text summary model is determined based on the probability, that is,

[0111]

[0112] Wherein, P(y|x) in formula (2) is the probability of generating a positive sample summary based on a given sample text for different θ, that is, formula (1).

[0113] In actual implementation, after determining the first sample text vector, the positive sample summary vector, and the negative sample summary vector, the second loss of the text summary model will be determined based on the first sample text vector, the positive sample summary vector, and the negative sample summary vector. Figure 6 , Figure 6 This is a flow chart of determining the second loss by the text summarization model provided in the embodiment of the present application, based on Figure 3 , based on the first sample text vector, the positive sample summary vector, and the negative sample summary vector, determining the second loss of the text summary model includes:

[0114] Step 1041b: Determine a first cosine similarity between the first sample text vector and the positive sample summary vector, and determine a second cosine similarity between the first sample text vector and the negative sample summary vector.

[0115] In actual implementation, the first cosine similarity sim(R(x), R(y)) between the first sample text vector and the positive sample summary vector is first determined, and then the second cosine similarity sim(R(x), R(y')) between the first sample text vector and the negative sample summary vector is determined. Here, it should be noted that the sim function represents the cosine similarity calculation between vectors, R(x) represents the text vector representation of x, that is, the first sample text vector, R(y) represents the text vector representation of y, that is, the positive sample summary vector, and R(y') represents the text vector representation of y', that is, the negative sample summary vector.

[0116] Step 1042b: Obtain a similarity coefficient between the positive sample summary and the negative sample summary; wherein the similarity coefficient is used to adjust the similarity between the positive sample summary and the negative sample summary.

[0117] In actual implementation, the similarity coefficient between the positive sample summary and the negative sample summary is also known as the temperature coefficient. This temperature coefficient is used to adjust the similarity between the positive sample summary and the negative sample summary. It should be noted that the temperature coefficient is used to adjust the similarity between the positive sample summary and the negative sample summary, that is, to adjust the degree of attention paid to the sample text corresponding to the negative sample summary (i.e., the negative sample). The smaller the temperature coefficient, the more attention is paid to separating the sample text corresponding to the positive sample summary (i.e., the positive sample) from the most similar negative sample. However, negative samples are often highly similar to positive samples. Many negative samples are actually potential positive samples. Excessively forcing the separation of negative samples will destroy the learned latent semantic structure. Therefore, the temperature coefficient cannot be too small.

[0118] Step 1043b: Determine a second loss of the text summarization model based on the first cosine similarity, the second cosine similarity, and the obtained similarity coefficient.

[0119] In actual implementation, after determining the first cosine similarity between the first sample text vector and the positive sample summary vector, the second cosine similarity between the first sample text vector and the negative sample summary vector, and the similarity coefficient between the positive sample summary and the negative sample summary, the second loss of the text summary model will be determined based on the first cosine similarity, the second cosine similarity and the obtained similarity coefficient, that is,

[0120]

[0121] Among them, r in formula (3) is the similarity coefficient between the positive sample summary and the negative sample summary.

[0122] Step 105: Update the model parameters of the text summarization model by combining the first loss and the second loss.

[0123] In actual implementation, after determining the first loss and the second loss, the model parameters of the text summarization model can be updated based on the first loss and the second loss. Here, there are two ways to update the model parameters of the text summarization model.

[0124] Next, two methods of updating the model parameters of the text summarization model are described.

[0125] In some embodiments, updating the model parameters of the text summary model in combination with the first loss and the second loss is specifically as follows: determining a corresponding first error signal based on the first loss, backpropagating the first error signal in the text summary model, and updating the model parameters of the decoder and the first encoder during the propagation process; determining a corresponding second error signal based on the second loss, backpropagating the second error signal in the text summary model, fixing the model parameters of the decoder during the propagation process, and updating the model parameters of the first encoder.

[0126] For example, see Figure 7 , Figure 7 This is an optional schematic diagram of updating the model parameters of the text summarization model by combining the first loss and the second loss provided in the embodiment of the present application, see Figure 7 After encoding x (i.e., sample text), y (positive sample summary), and y' (negative sample summary) based on the encoder, the corresponding vectors obtained by the encoding are decoded based on the decoder, and finally Y (i.e., the target summary corresponding to the sample text) is obtained. First, based on Y and y, the first loss L of the text summary model is determined by formula (2): MLE , and determine the corresponding first error signal based on the first loss, propagate the first error signal back in the text summary model, and update the model parameters of the decoder and the first encoder during the propagation process. At the same time, based on the first sample text vector, positive sample summary vector and negative sample summary vector obtained after encoding x (i.e. sample text), y (positive sample summary) and y' (negative sample summary), determine the second loss L of the text summary model CL , then based on the second loss L CL A corresponding second error signal is determined, and the second error signal is back-propagated in the text summarization model, during which the model parameters of the decoder are fixed and the model parameters of the first encoder are updated.

[0127] In some embodiments, updating the model parameters of the text summary model in combination with the first loss and the second loss is specifically as follows: obtaining weights corresponding to the first loss and the second loss, respectively; performing a weighted summation of the first loss and the second loss based on the weights corresponding to the first loss and the second loss to obtain a target loss; and updating the model parameters of the text summary model based on the target loss.

[0128] For example, see Figure 8 , Figure 8 This is an optional schematic diagram of updating the model parameters of the text summarization model by combining the first loss and the second loss provided in the embodiment of the present application, see Figure 8 After encoding x (i.e., sample text), y (positive sample summary), and y' (negative sample summary) based on the encoder, the corresponding vectors obtained by the encoding are decoded based on the decoder, and finally Y (i.e., the target summary corresponding to the sample text) is obtained. First, based on Y and y, the first loss L of the text summary model is determined by formula (2): MLE At the same time, based on the first sample text vector, positive sample summary vector and negative sample summary vector obtained after encoding x (i.e. sample text), y (positive sample summary) and y' (negative sample summary), the second loss L of the text summary model is determined CL , after getting the first loss L MLE and the second loss L CLAfter that, the weights corresponding to the first loss and the second loss are obtained respectively. Based on the weights corresponding to the first loss and the second loss, the first loss and the second loss are weighted summed to obtain the target loss, that is,

[0129] L=λ1*L MLE +λ2*L CL ...Formula (4);

[0130] In actual implementation, after obtaining the target loss, the model parameters of the text summarization model can be updated based on the target loss.

[0131] It should be noted that λ1 and λ2 in formula (4) are the weight coefficients corresponding to the first loss and the second loss obtained through training, and the value range of λ1 and λ2 is between 0 and 1, where λ1∈(0,1], λ2∈[0,1].

[0132] In some embodiments, before training the text summary model, the text summary model is pre-trained. Figure 9 , Figure 9 This is an optional flow chart of the pre-training of the text summary model provided in the embodiment of the present application, based on Figure 3 Before step 101, you can also execute:

[0133] Step 201: Obtain a pre-training sample set including a plurality of pre-training samples, wherein the pre-training samples include pre-training sample texts and corresponding pre-training sample summaries.

[0134] In actual implementation, after obtaining a pre-training sample set including multiple pre-training samples, the obtained pre-training sample text can first be pre-processed. Specifically, the pre-processing may include word segmentation, stop word removal, part-of-speech tagging, word vector conversion, entity word recognition, and keyword recognition. An entity refers to a real substance or concept that can exist independently and serve as the basis of all attributes and the origin of all things. It can usually refer to specific things, subjects, and supporters of phenomena, such as names of people, objects, places, organizations, and concepts.

[0135] Step 202: Encode the pre-training sample text through a first encoder to obtain a corresponding pre-training sample text vector.

[0136] In actual implementation, after obtaining a pre-training sample set including a plurality of pre-training samples, the pre-training sample texts contained in the pre-training sample set are encoded by a first encoder to obtain corresponding pre-training sample text vectors.

[0137] Step 203: decode the pre-trained sample text vector through a decoder to obtain a predicted summary.

[0138] In actual implementation, after obtaining the pre-training sample text vector, the pre-training sample text vector is decoded through a decoder to obtain a predicted summary.

[0139] Step 204 : Obtain the difference between the predicted summary and the pre-trained sample summary, and update the model parameters of the text summarization model based on the difference.

[0140] In actual implementation, after obtaining the predicted summary, the difference between the predicted summary and the pre-trained sample summary is obtained by comparing the predicted summary with the pre-trained sample summary, and the model parameters of the text summary model are updated based on the difference.

[0141] It should be noted that after updating the model parameters of the text summary model, a training sample including a sample text, a positive sample summary corresponding to the sample text, and a negative sample summary corresponding to the sample text can be obtained. Then, through the first encoder, the sample text, the positive sample summary, and the negative sample summary in the training sample are encoded respectively to obtain a first sample text vector, a positive sample summary vector, and a negative sample summary vector. Then, based on the first sample text vector, the positive sample summary vector, and the negative sample summary vector, the difference of the text summary model is determined, and based on the difference, the model parameters of the pre-trained text summary model are updated to obtain the final text summary model.

[0142] In some embodiments, after the training of the text summary model is completed, the corresponding text summary of the target text is extracted based on the trained text summary model. Figure 10 , Figure 10 This is an optional flow chart of the text summary extraction process provided in the embodiment of the present application, based on Figure 3 After step 105, you may also execute:

[0143] Step 301: Encode the target text through a first encoder to obtain a corresponding text vector.

[0144] In actual implementation, before the target text is encoded by the first encoder to obtain the corresponding text vector, the target text will first be obtained based on the client set on the terminal. It should be noted that the client can be a text summary client set on the terminal. The target text can be triggered by the user based on the human-computer interaction interface of the client, triggering the upload function item in the human-computer interaction interface so that the client presents the target text selection interface on the human-computer interaction interface. The user uploads the target text from the terminal locally based on the target text selection interface, so that the client obtains the uploaded target text.

[0145] In actual implementation, the target text can also be captured by a camera connected to the terminal. After capturing the target text, the camera transmits the target text to the terminal, which then automatically uploads it to the client. Alternatively, the target text can be written by a user on the terminal, and then the user takes a screenshot of the target text or saves a document containing the target text, and then uploads the image or document to the client, thereby uploading the target text to the client.

[0146] In actual implementation, after obtaining the target text, the target text is encoded through the first encoder to obtain a text vector corresponding to the target text.

[0147] Step 302: decode the text vector through a decoder to obtain a text summary corresponding to the target text.

[0148] In actual implementation, after obtaining the text vector corresponding to the target text, the text vector is decoded through a decoder to obtain the text summary corresponding to the target text.

[0149] In actual implementation, after obtaining the text summary corresponding to the target text, the server can send the text summary corresponding to the target text to the client. After receiving the corresponding text summary, the client can present the corresponding text summary in the human-computer interaction interface of the client, save the text summary locally on the terminal, and send the detection result to other devices connected to the terminal for communication, etc.

[0150] By applying the above-mentioned embodiment of the present application, in the process of training the text summary model, the training samples are divided into sample text, positive sample summary corresponding to the sample text, and negative sample summary corresponding to the sample text. By training the training samples containing the sample text, the positive sample summary, and the negative sample summary, a first loss and a second loss are obtained, and then the model parameters are updated based on the first loss and the second loss. In this way, by learning the differences between the positive samples (i.e., the sample text and the positive sample summary) and the negative samples (i.e., the sample text and the negative sample summary), the text summary model's ability to understand the differences between different summaries is enhanced, so that the text summary generation model can learn to obtain better text representations from the training samples, thereby making the text summary model have better performance and improving the accuracy of text summary extraction.

[0151] Next, we will continue to introduce the training method of the text summarization model provided in the embodiment of this application. Figure 11 This is an optional flow chart of the training method of the text summarization model provided in the embodiment of the present application, see Figure 11 The training method of the text summary model provided in the embodiment of the present application is implemented collaboratively by the client and the server.

[0152] In step 401 , the client obtains a sample set including a plurality of training samples in response to an upload operation for a training sample, wherein the training sample includes a sample text, a positive sample summary corresponding to the sample text, and a negative sample summary corresponding to the sample text.

[0153] In actual implementation, the client can be a text summary client set up on the terminal. The user can trigger the upload function item in the human-computer interaction interface based on the human-computer interaction interface of the client to make the client present a training sample selection interface on the human-computer interaction interface. The user can upload the training sample from the terminal locally based on the training sample selection interface, so that the client obtains the uploaded training sample.

[0154] Step 402: The client sends a training sample to the server.

[0155] In step 403 , the server inputs the received training sample into the text summarization model.

[0156] In step 404 , the sample text, the positive sample summary, and the negative sample summary in the training sample are respectively encoded by a first encoder to obtain a first sample text vector, a positive sample summary vector, and a negative sample summary vector.

[0157] Step 405: decode the first sample text vector through a decoder to obtain a target summary.

[0158] Step 406 : Determine a first loss of the text summarization model based on the target summary and the positive sample summary, and determine a second loss of the text summarization model based on the first sample text vector, the positive sample summary vector, and the negative sample summary vector.

[0159] Step 407: Update the model parameters of the text summarization model based on the first loss and the second loss.

[0160] In actual implementation, the server iterates the above training process until the loss function converges, completing the training of the text summarization model.

[0161] In step 408, the server generates a prompt message indicating that the text summary model training is completed.

[0162] Step 409: The server sends a prompt message to the client.

[0163] Step 410: The client obtains the target text in response to the upload operation on the target text.

[0164] In actual implementation, the target text can be captured by a camera connected to the terminal. After capturing the target text, the camera transmits the target text to the terminal, which then automatically uploads it to the client. Alternatively, the target text can be written by a user on the terminal, and then the user takes a screenshot of an image containing the target text or saves a document containing the target text, and then uploads the image or document to the client, thereby uploading the target text to the client.

[0165] In step 411 , the client sends the target text to the server in response to the text summary extraction instruction for the target text.

[0166] In actual implementation, the text summary extraction instruction can be automatically generated by the client under certain trigger conditions. For example, after the client obtains the target text, it automatically generates a text summary extraction instruction for the target text. It can also be sent to the client by other devices connected to the terminal. It can also be generated by the user based on the client's human-computer interaction interface, triggering the corresponding text summary extraction item.

[0167] In step 412 , the server encodes the target text through a first encoder to obtain a corresponding text vector.

[0168] Step 413: decode the text vector through a decoder to obtain a text summary corresponding to the target text.

[0169] Step 414: The server sends the text summary corresponding to the target text to the client.

[0170] Step 415: The client outputs a text summary corresponding to the target text.

[0171] In actual implementation, after obtaining the text summary corresponding to the target text, the server can send the text summary corresponding to the target text to the client. After receiving the corresponding text summary, the client can present the corresponding text summary in the human-computer interaction interface of the client, save the text summary locally on the terminal, and send the detection result to other devices connected to the terminal for communication, etc.

[0172] By applying the above-mentioned embodiment of the present application, in the process of training the text summary model, the training samples are divided into sample text, positive sample summary corresponding to the sample text, and negative sample summary corresponding to the sample text. By training the training samples containing the sample text, the positive sample summary, and the negative sample summary, a first loss and a second loss are obtained, and then the model parameters are updated based on the first loss and the second loss. In this way, by learning the differences between the positive samples (i.e., the sample text and the positive sample summary) and the negative samples (i.e., the sample text and the negative sample summary), the text summary model's ability to understand the differences between different summaries is enhanced, so that the text summary generation model can learn to obtain better text representations from the training samples, thereby making the text summary model have better performance and improving the accuracy of text summary extraction.

[0173] The following describes an exemplary application of the embodiments of the present application in a practical application scenario.

[0174] In the existing technology, text summarization technology is a high-level summary of information, which can help users quickly obtain the information they want in massive data. In the era of information explosion, we need to use text summarization technology to help users quickly obtain the information they want. Specifically, text summarization technology can be used in a variety of application scenarios such as search and recommendation. The existing text summary model usually consists of two parts: an encoder and a decoder. For the encoder, its task is to encode the natural text into a text vector, while the task of the decoder is to decode the text vector into the natural text that needs to be generated. In the text summary task, the generative method usually encodes the source document through the encoder, and then decodes it with the decoder to generate the summary result that needs to be generated. As for the training process of the existing text summary model, its training sample consists of the source document x and its corresponding target summary y. The generative text summary model will be used in a large number of<x,y> The training dataset composed of text pairs is obtained through MLE (maximum likelihood estimation) training objective training, and its specific loss function refers to formula (2).

[0175] In an embodiment of the present application, in order to enable the text summary model to learn the differences between different summaries, we introduce a contrastive learning method for training in the training process of generating the text summary model. Specifically, for the source document x, we use its corresponding target summary y as a positive example of x. On this basis, we randomly select the summary text y' (i.e., the target summary of other documents) of the non-source document x target summary in the data set as a negative example. In the model learning process, we hope that the model can distinguish y from y', so we introduce formula (3) as the loss function of the text summary model. In formula (3), the sim function represents the cosine similarity calculation between vectors; R(x) represents the text vector representation of x. Here, we use the encoder part in the text summary model to encode x, y, and y' to obtain the corresponding vector representation; r represents the temperature coefficient, which is used to control the weights of positive and negative examples. Finally, the loss function of formula (4) is used to train the text summary model.

[0176] The training method of the text summarization model provided in the embodiment of the present application, by adding the training objective of contrastive learning to the training of the text summarization model, enables the text summarization model to understand the differences between different summaries, thereby enhancing the text representation ability of the text summarization model. Specifically, in the training process of the text summarization model with contrastive learning, given the training sample of the text summary {x i ,y i}, first randomly select y from the training set i Other abstracts j (ie y i Not equal to y j ) is used to construct contrastive learning samples {x i ,y i ,y j Therefore, the training of the model mainly includes two parts of data: the data used for MLE training {x i ,y i} and the data {x i ,y i ,y j Here, for MLE training, the model will take xi as input, obtain a vector representation through the encoder network, and use the vector representation for decoding in the decoder network to finally obtain the target summary. The training goal of the model is to generate y i The probability of x is the largest, and the specific loss function calculation method is shown in formula (2); for contrastive learning training, the model will i ,y i ,y j All of them are used as input, and the vector representation of the three is obtained through the encoder network, which is recorded as R(x i ), R(y i ), R(yj ). On this basis, the loss value of contrastive learning is obtained by calculating formula (3).

[0177] As an example, the source document x i Group A released its fourth-quarter 2014 financial report and announced a tax-free divestiture plan for its 15% stake in Group B, intending to distribute this valuable investment, valued at approximately $40 billion, to shareholders. Group A's stock price rose approximately 7% to $51.45 at press time. i A randomly selected summary of Group A's announcement of divestiture of Group B's shares j The C Group expects its loan amount to exceed RMB 3 billion in 2015. Here, the training process of MLE is to input the source document x i , respectively, through the encoder and decoder to obtain the generated summary, according to formula (2) to maximize y i The generation probability of contrastive learning is x i ,y i ,y j , the model gets x through the encoder i ,y i ,y j After the vector representation of , the contrastive learning loss function is obtained according to formula (3). Finally, the two losses are added together as the final loss value for model training. In this way, the training samples are divided into positive samples (consisting of the source document and its corresponding target summary) and negative samples (consisting of the source document and its non-corresponding target summary). By learning the differences between positive and negative samples, the model's ability to understand the differences between different summaries is enhanced, thereby improving the model's performance.

[0178] The training method of the text summary model provided in the embodiment of the present application has better text summary extraction effect than the traditional method through the proposed text summary generation method based on contrastive learning. Specifically, the Chinese text summary public dataset LCSTS is used to verify the effect of the proposed method. Here, a traditional sequence-to-sequence text summary generation model and a text summary generation model with contrastive learning are designed for experiments, and then the results are evaluated using the evaluation indicators Rouge-1, Rouge-2, and Rouge-L commonly used in the summary field. Under the same experimental settings, the experimental results of the two methods are as follows: Figure 12 As shown, Figure 12 This is a comparison chart of the effects of the traditional text summarization model provided in the embodiment of this application and the text summarization model provided in this application, based on Figure 12 ,It can be determined that the text summary generation method based on contrastive learning proposed in this application has better results than the traditional method.

[0179] The following continues to describe the exemplary structure of the text summarization model training device 455 provided in the embodiment of the present application as a software module. In some embodiments, such as Figure 2 As shown, the software modules stored in the training device 455 of the text summarization model in the memory 440 may include:

[0180] An acquisition module 4551 is configured to acquire a sample set comprising a plurality of training samples, wherein the training samples include sample text, a positive sample summary corresponding to the sample text, and a negative sample summary corresponding to the sample text;

[0181] An encoding module 4552 is configured to encode the sample text, the positive sample summary, and the negative sample summary in the training sample using the first encoder to obtain a first sample text vector, a positive sample summary vector, and a negative sample summary vector;

[0182] A decoding module 4553 is configured to decode the first sample text vector using the decoder to obtain a target summary;

[0183] a determination module 4554 for determining a first loss of the text summarization model based on the target summary and the positive sample summary, and determining a second loss of the text summarization model based on the first sample text vector, the positive sample summary vector, and the negative sample summary vector;

[0184] An updating module 4555 is configured to update the model parameters of the text summarization model based on the first loss and the second loss.

[0185] In some embodiments, the update module 4555 is further used to determine a corresponding first error signal based on the first loss, back-propagate the first error signal in the text summary model, and update the model parameters of the decoder and the first encoder during the propagation process; determine a corresponding second error signal based on the second loss, back-propagate the second error signal in the text summary model, fix the model parameters of the decoder during the propagation process, and update the model parameters of the first encoder.

[0186] In some embodiments, the update module 4555 is further used to obtain the weights corresponding to the first loss and the second loss respectively; based on the weights corresponding to the first loss and the second loss, perform weighted summation of the first loss and the second loss to obtain a target loss; and update the model parameters of the text summary model based on the target loss.

[0187] In some embodiments, the text summary model also includes a second encoder, and the encoding module 4552 is further used to encode the sample text through the second encoder to obtain a second sample text vector; obtain the weights of the first encoder and the second encoder respectively; based on the weights of the first encoder and the second encoder, perform weighted summation on the first sample text vector and the second sample text vector to obtain a target sample text vector; the decoding module is further used to decode the target sample text vector through the decoder to obtain a target summary.

[0188] In some embodiments, the determination module 4554 is further used to determine a first cosine similarity between the first sample text vector and the positive sample summary vector, and to determine a second cosine similarity between the first sample text vector and the negative sample summary vector; obtain a similarity coefficient between the positive sample summary and the negative sample summary; wherein the similarity coefficient is used for the similarity between the positive sample summary and the negative sample summary; and determine a second loss of the text summary model based on the first cosine similarity, the second cosine similarity and the obtained similarity coefficient.

[0189] In some embodiments, the determination module 4554 is further used to determine the difference between the target summary and the positive sample summary; based on the difference, determine the probability of generating the positive sample summary based on the sample text; and based on the probability, determine the first loss of the text summarization model.

[0190] In some embodiments, the device further includes a pre-training module, which is used to obtain a pre-training sample set including multiple pre-training samples, wherein the pre-training samples include pre-training sample text and corresponding pre-training sample summaries; encode the pre-training sample text through the first encoder to obtain a corresponding pre-training sample text vector; decode the pre-training sample text vector through the decoder to obtain a predicted summary; obtain the difference between the predicted summary and the pre-training sample summary, and update the model parameters of the text summary model based on the difference.

[0191] In some embodiments, the acquisition module 4551 is also used to construct multiple positive samples, and the positive samples include: the sample text and the positive sample summary corresponding to the sample text; for each positive sample, select another positive sample, and use the positive sample summary of the other selected positive sample as the negative sample summary, and form the training sample together with the positive sample; obtain multiple training samples corresponding to the positive samples to obtain the sample set.

[0192] In some embodiments, the device further includes an application module, which is used to encode the target text through the first encoder to obtain a corresponding text vector; and decode the text vector through the decoder to obtain a text summary corresponding to the target text.

[0193] The present invention provides a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the text summarization model training method described in the present invention.

[0194] The embodiment of the present application provides a computer-readable storage medium storing executable instructions, wherein the executable instructions are stored. When the executable instructions are executed by a processor, the processor will execute the training method of the text summarization model provided in the embodiment of the present application, for example, Figure 3 The training method of the text summarization model is shown.

[0195] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or various devices including one or any combination of the above memories.

[0196] In some embodiments, executable instructions may be in the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0197] As an example, executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinating files (e.g., files storing one or more modules, subroutines, or code portions).

[0198] By way of example, executable instructions may be deployed to be executed on one computing device, or on multiple computing devices at one site, or on multiple computing devices distributed across multiple sites and interconnected by a communication network.

[0199] In summary, the embodiments of the present application have the following beneficial effects:

[0200] (1) By learning the differences between positive samples (i.e., sample text and positive sample summaries) and negative samples (i.e., sample text and negative sample summaries), the text summary model’s ability to understand the differences between different summaries is enhanced, so that the text summary generation model can learn better text representations from the training samples, thereby making the text summary model have better performance.

[0201] (2) By using the weighted text summary model of the attention mechanism, the information carried by the first sample text vector, the positive sample summary vector, and the negative sample summary vector can be fully utilized, so that the more critical information for the current task goal can be selected from a large amount of information, thereby improving the accuracy of text summary extraction.

[0202] (3) By cleaning web page data, a sample set including multiple training samples is automatically generated, thereby greatly improving the efficiency of sample set generation and reducing the labor cost generated by manual setting.

[0203] The above description is merely an embodiment of the present application and is not intended to limit the scope of protection of the present application. Any modifications, equivalent replacements, and improvements made within the spirit and scope of the present application are included in the scope of protection of the present application.

Claims

1. A training method for a text summarization model, characterized in that: The text summarization model includes a first encoder and a decoder, and the method includes: Acquire a sample set including a plurality of training samples, wherein the training samples include sample text, a positive sample summary corresponding to the sample text, and a negative sample summary corresponding to the sample text; Encoding the sample text, the positive sample summary, and the negative sample summary in the training sample by the first encoder respectively to obtain a first sample text vector, a positive sample summary vector, and a negative sample summary vector; Decoding the first sample text vector using the decoder to obtain a target summary; determining a difference between the target summary and the positive sample summary; Based on the difference, determining a probability of generating the positive sample summary based on the sample text; Based on the probability, determining a first loss of the text summarization model; Determining a second loss of the text summarization model based on the first sample text vector, the positive sample summary vector, and the negative sample summary vector; updating the model parameters of the text summarization model in combination with the first loss and the second loss; The first loss is used to update the model parameters of the decoder and the first encoder in the text summary model, and the second loss is used to update the model parameters of the first encoder in the text summary model while fixing the model parameters of the decoder in the text summary model.

2. The method according to claim 1, wherein The updating of the model parameters of the text summarization model by combining the first loss and the second loss includes: Determining a corresponding first error signal based on the first loss, backpropagating the first error signal in the text summarization model, and updating model parameters of the decoder and the first encoder during the propagation process; A corresponding second error signal is determined based on the second loss, and the second error signal is back-propagated in the text summarization model. During the propagation process, the model parameters of the decoder are fixed, and the model parameters of the first encoder are updated.

3. The method according to claim 1, wherein The text summarization model further includes a second encoder, and the method further includes: Encoding the sample text by the second encoder to obtain a second sample text vector; Obtaining weights of the first encoder and the second encoder respectively; Based on the weights of the first encoder and the second encoder, performing weighted summation on the first sample text vector and the second sample text vector to obtain a target sample text vector; Decoding the first sample text vector by the decoder to obtain a target summary includes: The target sample text vector is decoded by the decoder to obtain a target summary.

4. The method according to claim 1, wherein The determining, based on the first sample text vector, the positive sample summary vector, and the negative sample summary vector, a second loss of the text summarization model includes: Determining a first cosine similarity between the first sample text vector and the positive sample summary vector, and determining a second cosine similarity between the first sample text vector and the negative sample summary vector; Obtaining a similarity coefficient between the positive sample summary and the negative sample summary; wherein the similarity coefficient is used to adjust the similarity between the positive sample summary and the negative sample summary; A second loss of the text summarization model is determined based on the first cosine similarity, the second cosine similarity, and the obtained similarity coefficient.

5. The method according to claim 1, wherein Before obtaining the training sample, the method further includes: Acquire a pre-training sample set comprising a plurality of pre-training samples, wherein the pre-training samples include pre-training sample texts and corresponding pre-training sample summaries; Encoding the pre-training sample text by the first encoder to obtain a corresponding pre-training sample text vector; Decoding the pre-trained sample text vector using the decoder to obtain a predicted summary; Obtain a difference between the predicted summary and the pre-trained sample summary, and update model parameters of the text summarization model based on the difference.

6. The method according to claim 1, wherein The obtaining of a sample set including a plurality of training samples comprises: Constructing a plurality of positive samples, wherein the positive samples include: the sample text and a positive sample summary corresponding to the sample text; For each positive sample, select another positive sample, use the positive sample summary of the selected other positive sample as the negative sample summary, and form the training sample together with the positive sample; A plurality of training samples corresponding to the positive example samples are obtained to obtain the sample set.

7. The method according to claim 1, wherein The method further comprises: Encode the target text through the first encoder to obtain a corresponding text vector; The text vector is decoded by the decoder to obtain a text summary corresponding to the target text.

8. A training device for a text summarization model, characterized in that: The text summarization model includes a first encoder and a decoder, and the device includes: An acquisition module, configured to acquire a sample set comprising a plurality of training samples, wherein the training samples include sample text, a positive sample summary corresponding to the sample text, and a negative sample summary corresponding to the sample text; an encoding module, configured to encode the sample text in the training sample, the positive sample summary, and the negative sample summary respectively through the first encoder to obtain a first sample text vector, a positive sample summary vector, and a negative sample summary vector; a decoding module, configured to decode the first sample text vector through the decoder to obtain a target summary; a determination module configured to determine a difference between the target summary and the positive sample summary; based on the difference, determine a probability of generating the positive sample summary based on the sample text; and based on the probability, determine a first loss of the text summarization model; The determining module is further configured to determine a second loss of the text summarization model based on the first sample text vector, the positive sample summary vector, and the negative sample summary vector; An updating module, configured to update model parameters of the text summarization model based on the first loss and the second loss; The first loss is used to update the model parameters of the decoder and the first encoder in the text summary model, and the second loss is used to update the model parameters of the first encoder in the text summary model while fixing the model parameters of the decoder in the text summary model.

9. The device according to claim 8, characterized in that The update module is further configured to determine a corresponding first error signal based on the first loss, backpropagate the first error signal in the text summary model, and update the model parameters of the decoder and the first encoder during the propagation process; determine a corresponding second error signal based on the second loss, backpropagate the second error signal in the text summary model, fix the model parameters of the decoder during the propagation process, and update the model parameters of the first encoder.

10. The device according to claim 8, characterized in that The text summarization model further includes a second encoder, and the encoding module is further configured to encode the sample text using the second encoder to obtain a second sample text vector; obtain weights of the first encoder and the second encoder respectively; and perform a weighted summation of the first sample text vector and the second sample text vector based on the weights of the first encoder and the second encoder to obtain a target sample text vector; The decoding module is further configured to decode the target sample text vector through the decoder to obtain a target summary.

11. The device according to claim 8, characterized in that The device also includes a pre-training module, which is used to obtain a pre-training sample set including multiple pre-training samples, wherein the pre-training samples include pre-training sample text and corresponding pre-training sample summaries; encode the pre-training sample text through the first encoder to obtain a corresponding pre-training sample text vector; decode the pre-training sample text vector through the decoder to obtain a predicted summary; obtain the difference between the predicted summary and the pre-training sample summary, and update the model parameters of the text summarization model based on the difference.

12. An electronic device, characterized in that: include: a memory for storing executable instructions; The processor is configured to implement the text summarization model training method according to any one of claims 1 to 7 when executing the executable instructions stored in the memory.

13. A computer-readable storage medium, characterized in that Executable instructions are stored, which are used to cause a processor to execute and implement the text summarization model training method according to any one of claims 1 to 7.

14. A computer program product comprising computer-executable instructions, characterized in that When the computer-executable instructions are executed by a processor, the training method of the text summarization model according to any one of claims 1 to 7 is implemented.

Images