Generate topic-based summaries of text content

Abstract

The word generation model obtains text content and the requested topic of interest and generates a target summary of the text content adjusted to the topic of interest. To this end, the topic-aware encoding model encodes the text content using the topic tags corresponding to the topic of interest to generate topic-aware encoded text. The word generation model selects the next word for the topic-based summary from the topic-aware encoded text. The word generation model is trained using machine learning on training data to generate topic-based summaries, and the training data includes multiple documents, corresponding summaries for each document, and corresponding topics for each summary. Feedback of the next word selected is provided to the word generation model. The feedback enables the word generation model to select subsequent words for the topic-based summary based on the feedback of the next selected word.

Description

Background Art

[0001] Automatic summarization of textual content can be used to save time for end users by providing an overview of textual content (e.g., a document or article) that can be read quickly by the user. Conventional extractive summarization techniques extract key phrases from the input textual content and then select a subset of these phrases to place in the summary. However, the summaries generated by these traditional summarization techniques are generally not human-like. Furthermore, when the length of the desired summary is very small, such extractive methods become insufficient because this requires a method for concisely "rewriting" the input content to maximize the information conveyed in the summary rather than selecting the most "informative" sentences.

[0002] Recent advances in neural networks have led to the use of recurrent neural networks to generate summaries by rewriting documents. However, these algorithms only allow for the generation of a single summary. While a summary should contain the essential portion of the information in a document, what is considered "essential" varies from person to person. Furthermore, many documents include topics related to more than one topic of interest, such as politics and business. For some readers, business may be the primary area of interest, while others may be more interested in politics. In such scenarios, a single summary may not suit the topic preferences of all readers. Summary of the Invention

[0003] In order to overcome these problems, a topic-based summary of text content adjusted to a topic of interest is generated in a digital media environment. A word generation model obtains text content and a requested topic of interest and generates a target summary of text content adjusted to the topic of interest. To this end, a topic-aware coding model encodes the text content using topic tags corresponding to the topic of interest to generate topic-aware coded text. The word generation model selects the next word for the topic-based summary from the topic-aware coded text. The word generation model is trained to generate a topic-based summary using machine learning of training data, wherein the training data includes a plurality of documents, a corresponding summary for each document, and a corresponding topic for each summary. Feedback of the next word selected is provided to the word generation model. The feedback enables the word generation model to select a subsequent word for the topic-based summary based on the feedback of the next selected word.

[0004] This summary introduces some concepts in a simplified form that will be further described in the detailed description below. Therefore, this summary is not intended to identify essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] The detailed description is described with reference to the accompanying drawings.

[0006] Figure 1is an illustration of a digital media environment in an example implementation that is operable to employ the techniques for generating topic-based summaries of textual content described herein.

[0007] Figure 2 Depicts a system in an example implementation in which topic vectors are learned from a corpus of textual content.

[0008] Figure 3 A system is shown in which an intermediate data set is generated by applying topic vectors to a corpus comprising textual content of documents having corresponding summaries.

[0009] Figure 4 A system for artificially generating training data is shown.

[0010] Figure 5 Depicted is a system in an example implementation where Figure 1 The word generation model and topic-aware encoding model generate topic-based summaries of text content tuned to topics of interest.

[0011] Figure 6 An example of a word generation model when configured as a neural encoder is shown in accordance with one or more implementations.

[0012] Figure 7 Describes an example process for artificially generating training data to train a topic-aware encoding model and a word generation model using machine learning.

[0013] Figure 8 Describes an example process for generating topic-based summaries of text content using a topic-aware encoding model and a term generation model.

[0014] Figure 9 An example system is shown that includes an example computing device that is representative of one or more computing systems and / or devices that may implement the various techniques described herein. DETAILED DESCRIPTION

[0015] Overview

[0016] This paper describes a technology for generating a summary based on a theme of text content in a digital media environment. A summary module based on a theme utilizes a word generation model with a theme-aware coding model to generate a summary of text content that is adjusted to the requested topic of interest. The word generation model can correspond to a machine learning or rule-based summary model (e.g., a neural network) that utilizes extraction and / or abstract summary technology to generate a summary based on a theme of text content. The word generation model and the theme-aware coding model are trained using machine learning techniques on artificially generated training data to generate a summary based on a theme. It is noteworthy that the word generation model and the theme-aware coding model can be used to generate different summaries that are adjusted to different request topics of interest for a single document. Therefore, the technology described can generate a summary that is adjusted to different topics of interest to adapt to the topic preferences of different readers.

[0017] The subject-aware coding model and the word generation model are trained using machine learning of training data to generate topic-based summaries tuned to topics of interest, the training data comprising a plurality of data sets, each of which comprises a document, a corresponding summary of the document (e.g., an artificially generated summary), and the corresponding topic of the summary. However, the summary model requires a large amount of text content training corpus to train the model, and it is difficult to find a large text content data set that includes documents and summaries that are marked with topics. Therefore, in one or more implementations, the described technology artificially generates training data that can be used to train the subject-aware coding model and the word generation model.

[0018] To artificially generate training data, a supervised word frequency scheme is used to learn topic features (e.g., topic vectors) from a document corpus, where each document is labeled with a topic. The learned topic features are then applied to modify a large unlabeled document corpus and human-generated summaries. To this end, the learned features are applied to each summary to determine the topic of interest for each summary. All selected documents, their corresponding summaries, and the determined topics are then aggregated into an intermediate data set.

[0019] Finally, training data is artificially generated by randomly selecting a first document with a corresponding first summary and first topic, and a second document with a corresponding second summary and second topic from the intermediate data set. The first document is then merged with the second document to generate a first new document, which is then associated with the first summary and first topic in the training data. Similarly, the second document is merged with the first document to generate a second new document, which is then associated with the second summary and second topic in the training data. The first and second documents are then discarded, and these steps are repeated until all documents in the intermediate data set are exhausted. The artificially generated training data is then used to train a term generation model and a topic-aware encoding model to generate topic-based summaries using machine learning.

[0020] The summary module based on the theme uses the trained topic-aware coding model and the word generation model to generate the summary based on the theme. For this reason, the summary module based on the theme receives the request for generating the summary based on the theme of the text content that is adjusted to the topic of interest. The text of the text content of the input is provided to the topic-aware coding model, and the topic-aware coding model encodes the text content into the encoded text of the theme perception based on the requested topic of interest of the request, word by word. Therefore, based on the requested topic of interest for the summary, the input text of the text content is encoded differently. In order to encode the text content based on the topic of interest, the topic-aware coding model selects a topic label or a topic vector (such as a single-hot vector) based on the requested topic of interest, and then maps each word of the text content to embedding, and the selected topic vector representing the requested topic of interest is attached to each word. Then this overall concatenation vector is passed through the neural network of the topic-aware coding model, and this topic-aware coding model outputs encoding at each step, thereby causing a series of topic-aware encodings.

[0021] The fully encoded text of the text content is then provided to the word generation model. The decoder of the word generation model generates a topic-based summary by processing the encoding of the topic-based text content. Since the decoder of the word generation model uses these encodings to generate the summary, the generated summary is dependent on the requested topic of interest. In order to generate the topic-based summary, the word generation model generates an attention distribution of the words in the text content. For each word, the attention distribution indicates a selection probability value, which indicates the probability of the corresponding word being selected as the next generated word for the topic-based summary. Encoding the text content with the topic of interest encourages the selection of the most preferred words for the topic of interest learned through training.

[0022] The word generation model determines the selection probability value based on the words in the textual content of input and the words of the summary based on the theme generated by the word generation model. The described technology uses the theme-aware coding model to adjust the words generated by the word generation model to the interesting theme during the generation of the summary based on the theme. After using the word generation model and the theme-aware coding model to generate each word, the feedback about the generated words is provided back to the word generation model. The word generation model can use feedback to adjust the subsequent words of the summary. It should be noted that, since the textual content of input is encoded by the interesting theme during the generation summary, the summary is generated based on the words of the input text and the summary that has been generated that utilize the interesting theme to encode. Like this, the summary generated using the described technology generates a summary that is adjusted to the textual content of the requested interesting theme.

[0023] Thus, the described techniques utilize summarization algorithms to enhance the reader's reading experience and document consumption. In this regard, this topic-adapted summarization algorithm can assist in providing reader-specific summaries of longer documents. Since documents are often large in size, multiple topics may exist within a document, and this approach can help generate summaries with specific information based on the reader's topical interests.

[0024] In the following discussion, an example environment is first described in which the techniques described herein may be employed. Example implementation details and procedures are then described, which may be performed in the example environment as well as other environments. Therefore, performance of the example procedures is not limited to the example environment, and the example environment is not limited to performance of the example procedures.

[0025] Sample Environment

[0026] Figure 1 1 is an illustration of a digital media environment 100 in an example implementation that is operable to employ the techniques for generating topic-based summaries of text content described herein. The illustrated environment 100 includes a service provider system 102 and a client device 104 communicatively coupled via a network 106. The computing devices implementing the service provider system 102 and the client device 104 can be configured in various ways.

[0027] For example, a computing device can be configured as a desktop computer, a laptop computer, a mobile device (e.g., assuming a handheld configuration such as a tablet computer or mobile phone), etc. Thus, a computing device can range from a full-resource device with large amounts of memory and processor resources (e.g., a personal computer, a game console) to a low-resource device with limited memory and / or processing resources (e.g., a mobile device). Additionally, a computing device can represent multiple different devices, such as multiple servers used by an enterprise to perform operations "on the cloud," such as Figure 9 Described in .

[0028] The client device 104 is shown as including a communication module 108 that represents functionality for communicating with the service provider system 102 via the network 106, for example, using a browser, a network-enabled application, an operating system, etc. For example, the communication module 108 can form a request 110 for a summary of textual content 112 tuned to topics of interest 114. As described herein, the textual content 112 can include any type of textual content, such as an article, a word processing document, text on a web page, a PDF, etc. As described herein, the topics of interest 114 can include any type of topic or category, such as business, politics, sports, social networking, military, technology, education, health, etc.

[0029] The request 110 can be transmitted to the service provider system 102 via the network 106. In response to the request 110, the service provider system 102 generates and returns a topic-based summary 116 of the input text content 112 that is adjusted to the topic of interest 114. As described herein, the topic-based summary 116 corresponds to a shortened, concise version of the input text content 112 that is adjusted to the requested topic of interest 114. Although shown as being implemented remotely from the client device 104, the functionality of the illustrated service provider system 102 can also be implemented in whole or in part locally at the client device 104. For example, the topic-based summary module 118 can be implemented to generate the topic-based summary 116 locally at the client device 104 from which the request 110 was received.

[0030] To generate the topic-based summary 116, the service provider system 102 employs a topic-based summary module 118, which is configured to generate the topic-based summary 116, which is tuned to the requested topic of interest 114. The topic-based summary module 118 utilizes a word generation model 120 configured to generate a summary of the textual content, and a topic-aware encoding model 122 trained to encode the input textual content 112 with the requested topic of interest 114. The word generation model 120 can correspond to any type of word generation model or algorithm that can generate a summary of textual content. In some cases, the word generation model 120 is trained using machine learning (e.g., a neural network) to generate a summary of the textual content. For example, the word generation model 120 can be implemented as an extractive summarization algorithm, an abstractive summarization algorithm, a sequence to a neural encoder, and the like.

[0031] The service provider system is further shown to include a training module 124 that generates artificially generated training data 126 ("training data 126") containing multiple instances of the form (d, t, s), where d is the input document or article, t is the desired topic of the summary, and s is the ideal summary to be generated. The training module 124 then uses the artificially generated training data 126 to train the topic-aware encoding model 122 and the term generation model 120 using machine learning techniques.

[0032] In one or more implementations, the term generation model 120 generates terms for the topic-based summary 116 using a "term-by-term" summary process. To generate each "next term" for the topic-based summary 116, the term generation model 120 calculates a probability distribution based on a combination of the input text of the textual content 112 and the text of the target summary that has already been generated. After generating each term for the topic-based summary 116, the term generation model 120 receives feedback about the generated term. This feedback is used by the term generation model to generate subsequent terms for the topic-based summary 116. Unlike existing term generation techniques, the term generation model 120 generates terms for the target summary based on both the context of the input textual content 112 and the terms of the summary that has been generated so far. In order to tune the topic-based summary 116 to the topic of interest 114, the term-by-term generation process employed by the term generation model 120 is constrained by a topic-aware encoding model 122, which encodes the input textual content 112 with a topic vector.

[0033] A topic-based summary module 118 can be implemented to provide topic-based summaries 116 for a variety of different use cases. In some instances, the topic-based summary module 118 can be used by content authors of text content 112 (e.g., content authors of articles, papers, or books) to quickly and automatically generate summaries of their text content 112. The topic-based summary module 118 enables content authors to adjust summaries to different topics, which can enable authors to adjust summaries to multiple different target audiences. For example, content authors may want to provide different summaries on different platforms based on different audiences that frequently use these platforms. In other examples, the topic-based summary module 118 can be used by clients or end consumers of text content. In this context, the topic-based summary module 118 can assist in providing consumer-specific summaries of longer articles or papers that can be easily consumed through mobile notifications or on wearable devices. In this case, the topic-based summary module 118 can be implemented to adjust the summary to the requested consumer-interested topic.

[0034] In general, the functions, features, and concepts described with respect to the examples above and below can be employed in the context of the example processes described in this section. Furthermore, the functions, features, and concepts described with respect to the different figures and examples in this document are interchangeable with each other and are not limited to implementation in the context of a particular figure or process. Furthermore, the blocks associated with the different representative processes and corresponding figures herein can be applied together and / or combined in different ways. Thus, the individual functions, features, and concepts described with respect to the different example environments, devices, components, figures, and processes herein can be used in any suitable combination and are not limited to the specific combinations represented by the examples listed in this specification.

[0035] Figure 2 Depicted is system 200 in an example implementation in which topic vectors are learned from a corpus of textual content.

[0036] The training module 124 first uses a supervised word frequency-based approach to learn the characteristics of topics. In system 200, the training module 124 includes a grouping module 202 that receives a corpus 204 of textual content, which includes documents 206 labeled with topic tags 208, which identify the topic of each corresponding document 206. For example, the topic tags 208 can identify the topic of each document, such as "politics," "sports," "education," etc.

[0037] The grouping module 202 groups each document 206 of the corpus 204 of textual content into topic-based topic groups 210 based on topic tags 208, such that each document 206 having topic "t" is included in the set S t The representative is a theme-based theme group 210.

[0038] Frequency module 212 determines a term frequency 214 ("n") for each term in each topic-based topic group 210. Term frequency 214 corresponds to the number of occurrences of a given term in documents 206 for the topic-based topic group 210. Vector generation module 216 takes topic groups 210 and term frequency 214 and generates a topic vector 218 (e) for each topic t. t =(n1, n2..., n v )), where v=\V\ is a word {w1,w2,...,w v}, and n t In the theme group S t The word w appears in t In some cases, the vector e t Be standardized.

[0039] Figure 3 A system 300 is shown in which an intermediate data set is generated by applying topic vectors to a corpus of textual content comprising documents having corresponding summaries. In the system 300, the training module 124 is shown as comprising a topic identifier 302, which is configured to generate an intermediate data set 304. To this end, the topic identifier 302 receives a corpus 306 of textual content comprising documents 308 having corresponding human-generated summaries 310 as a collection of (document, summary) pairs. The topic identifier 302 modifies the corpus 306 of textual content to generate the intermediate data set 304 by determining a topic 312 for each summary 310.

[0040] To determine the topics 312 of the summaries 310, the topic identifier 302 applies the topic vector 218 to each corresponding summary 310. For example, the topic identifier 302 computes the dot product between the summary 310 (in its bag-of-words representation) and the previously extracted topic vector 218. In this case, <v s ,e tl > indicates the dot product between the summary 310 and the topic vector 218 of Topic 1. The topic identifier 302 then selects the topic with the highest calculated dot product as the determined topic 312 for the corresponding summary 310.

[0041] The confidence of the selected topic 312 can be determined based on a comparison of the highest calculated dot product with the second highest dot product. <v s ,e ti > has the highest dot product value, and <v s ,e tj > has the second highest dot product value in the set of dot products, then assigns the summary 310 to the topic 312 (t i ), with <v s ,e ti > / <v s ,e tj > a given confidence level. In some cases, if the confidence level is below a threshold, the document and corresponding summary are discarded from the intermediate data set 304. For example, if the confidence level is less than a given confidence threshold (e.g., 1.2), this indicates that the summary does not have a dominant topic of interest, and therefore the document is discarded from the intermediate data set 304. Doing so enables the training module 124 to generate the intermediate data set 304 to include only those summaries that have a dominant determined topic 312 in the text for training. The training module 124 then aggregates all selected documents and their corresponding summaries 310 and determined topics 312 into a triplet <document, topic, summary> for the intermediate data set 304.

[0042] Figure 4 A system 400 is shown in which training data is artificially generated. In the system 400, the training module 124 obtains the intermediate data set 304 and generates artificially generated training data 126. To this end, the merging module 402 of the training module 124 randomly selects a first document 404 having a first topic 408 and a corresponding first summary 406, and a second document 410 having a second summary 412 having a second topic 414 from the intermediate data set 304, wherein the first topic 408 and the second topic 414 are different.

[0043] Next, the merge module 402 generates a first new document 416 by merging the text from both the first document 404 and the second document 410. In one or more implementations, the merge module 402 generates the first new document 416 by sequentially picking up lines of text from the first document 404 and the second document 410. In some cases, each new line is added by randomly selecting one of the first document 404 or the second document 410 to extract the new line. This ensures that the lines of text from the first document 404 or the second document 410 occur in the same order in the first new document 416 as they did in each of the respective original documents. After the first new document 416 is generated, it is grouped with the first summary 406 and the first topic 408 and then added to the artificially generated training data 126.

[0044] Next, the merge module 402 repeats these steps to generate a second new document 418 to be grouped with the second summary 412 and the second topic 414. For example, the merge module 402 generates the second new document 418 by merging the text from the second document 410 and the first document 404. In one or more implementations, the merge module 402 generates the second new document 418 by sequentially picking up lines of text from the second document 410 and the first document 404. In some cases, each new line is added by randomly selecting one of the second document 410 or the first document 404 to extract the new line. This ensures that the lines of text from the second document 410 and the first document 404 occur in the same order in the second new document 418 as they did in each of the corresponding original documents. After the second new document 418 is generated, it is grouped with the second summary 412 and the second topic 414 and then added to the artificially generated training data 126.

[0045] In some cases, the training module 124 then discards the first document 404, the first summary 406, and the first topic 408, as well as the second document 410, the second summary 412, and the second topic 414 from the intermediate data set 304. These steps are then repeated until the intermediate data set 304 is exhausted. The artificially generated training data 126 is then used to train the term generation model 120 and the topic-aware encoding model 122 to generate topic-based summaries 116 using machine learning.

[0046] Figure 5 A system 500 is depicted in an example implementation, wherein Figure 1 The term generation model 120 and the topic-aware encoding model 122 generate a topic-based summary 116 of the text content 112 that is tuned to the topic of interest 114.

[0047] In system 500, topic-based summary module 118 receives a request 110 for generating a topic-based summary 116 of textual content 112 tuned to a topic of interest 114. In some cases, the request may be received from a content author of the textual content 112, such as an author of an article, a book, or the like. Alternatively, the request may be received from a consumer of the textual content 112 to generate a summary of the textual content. As described throughout, request 110 may include an indication of the textual content 112 and a topic of interest 114.

[0048] The topic-based summary module 118 provides the text of the input text content 112 to the topic-aware encoding model 122. The topic-aware encoding model 122 generates topic-aware encoded text 502 by encoding the text content 112 into a sequence of topic-aware encodings based on the interest topics 114 of the request 110. In other words, the input text of the text content 112 is encoded differently based on the expected interest topics 114 for the summary. In order to encode the text content 112 based on the interest topics 114, the topic-aware encoding model selects a topic vector 218 (e.g., a single-hot vector) based on the requested interest topics 114, then maps each word of the text content 112 to an embedding, and the selected topic vector 218 representing the requested interest topic 114 is attached to each word. This overall concatenated vector is then passed through the neural network of the topic-aware encoding model 122, which outputs an encoding at each step, resulting in a series of topic-based encodings. As described above, the topic vector 218 indicates the word preferences of the interest topics learned from the supervised word frequency-based scheme.

[0049] The topic-aware encoded text 502 of the textual content 112 is then provided to the term generation model 120. The attention decoder 504 of the term generation model 120 receives the topic-aware encoded text 502 of the textual content 112 as input and generates an attention distribution 506 for the terms 508 in the textual content 112. The attention distribution 506 indicates a selection probability value 510 for each term 508. The selection probability value 510 indicates the probability that the corresponding term 508 will be selected as the next generated term for the topic-based summary 116. The attention decoder 504 determines the selection probability value 510 based on the terms in the input textual content 112 and the terms of the topic-based summary 116 that have been generated by the term generation model 120.

[0050] The term generation model 120 is further shown to include a selection module 512 configured to select the next generated term 514 for the topic-based summary 116 based on the attention distribution 506, which is generated by the attention decoder 504 of the term generation model 120. As described above, encoding the text content to generate topic-aware encoded text increases the selection probability values of the preferred terms for the topic of interest 114 learned through machine learning training. Therefore, encoding the input text encourages the selection of the most preferred terms for the topic of interest 114 of the summary.

[0051] After the selection module 512 selects the next generated word 514, the generated word is added to the topic-based summary 116. In addition, the feedback module 516 provides the generated word feedback 518 to the attention decoder 504, which can use the generated word feedback 518 to select subsequent words of the topic-based summary 116 to maintain consistency in the word generation of the topic-based summary 116.

[0052] Figure 6 An example 600 of the word generation model 120 when configured as a neural encoder is shown in accordance with one or more implementations.

[0053] In example 600, the word generation model 120 is implemented as a neural encoder, which can be implemented to map a sequence of words to another sequence of words. For example, by training on a large corpus of English sentences and their French translations, the word generation model 120 can learn to output the French translation of an input sentence. This concept can be applied to configure the word generation model 120 to generate a summary of text content. In order to ensure that the next generated word of the summary does not depend solely on the previously generated words, the word generation model 120 can use a long short-term memory (LSTM) architecture. The LSTM architecture uses a gating mechanism that retains information in memory or discards previous information.

[0054] In example 600, the LSTM-based encoder receives word embedding vectors (“w i ”) as input and computes the sequence of encoder hidden states 602 (“h i ”). The final hidden state is passed to the decoder, which also calculates the decoder hidden state 604 (“s t ”) and generates word embeddings of the target word at each decoding time step (“y t ”).

[0055] As discussed throughout, the topic-aware encoding model 122 receives input text content 112 as a sequence of words w1 w2 . . . w n, and maps the input word to the embedding sequence e x e2...e n If there are k topics (t1, t2, ..., t k ), then each topic is represented by a one-hot vector of size k, and t i It is represented as (0,0,..1,0,0...) (e.g. the i-th position is set to 1 and the rest are set to 0). Therefore, if the input text content 112 is w1w2...w n , and the topic of interest 114 of the requested summary is t, then the words of the text content 112 are mapped to the sequence (e1,t),(e2,t),......,(e n ,t). Notably, the vector of topic t is appended to each word embedding. The encoded text is then fed to the LSTM-based encoder of the topic-aware encoding model, which computes a sequence of encoder hidden states 602 (“h i ”). The final hidden state of the LSTM-based encoder is also passed to the decoder, which also calculates the hidden state 604 (“s t ”) and generates word embeddings of the target word at each decoding time step (“y t ”).

[0056] When generating the words for the topic-based summary 116, the word generation model 120 pays more attention to certain parts of the input text content 112 than to other parts. For example, in the example system, the word generation model 120 is generating the topic-based summary 116 for the input text content 112: "Germany emerges victorious in a 2-0 win against Argentina." The word generation model 120 can learn to generate the word "beat" by paying more attention to the words "victorious" and "win." To do this, the word generation model 120 uses the following algorithm to calculate the attention distribution 506 over all words in the text content 112:

[0057]

[0058] a t =softmax(e t )

[0059] In the above attention distribution algorithm, machine learning techniques are used to train v and W on the corpus of text content. h 、W s and b attAs mentioned above, the attention distribution 506 can be considered as a probability distribution over the words in the input text content 112 learned from the training corpus. This context vector, which is a weighted sum of the encoder hidden states 602 Used to determine the next word of a summary to generate based on the words already generated within a defined "context" window.

[0060] In order to reproduce the factual fragments of the input text content 112, the word generation model 120 can be extended to include a probability term to decide whether to generate words as described above or pick them from the input text content 112 to copy the factual details. At each stage of the decoder, a probability pgen is calculated, which indicates whether the word is generated from the vocabulary or copied from the source text content 112 using the attention distribution 506. This probability can be calculated using the following algorithm:

[0061]

[0062] In the above algorithm, the training is based on the text training corpus and b gen . Then you can use p gen To determine the attention distribution 506 of the words in the vocabulary given by the algorithm:

[0063]

[0064] The second term allows the word generation model 120 to select words to copy from the input text content 112 using the attention distribution 506. P(w) yields the probability that the next word is being generated given the current word and the context.

[0065] Example Process

[0066] This section describes example processes for generating topic-based summaries of text content. Aspects of the processes can be implemented in hardware, firmware, or software, or a combination thereof. These processes are shown as a collection of blocks that specify operations performed by one or more devices and are not necessarily limited to the commands shown for performing the operations of the corresponding blocks.

[0067] Figure 7 Depicted is an example process 700 for artificially generating training data to train a topic-aware encoding model and a term generation model using machine learning.

[0068] A corpus of textual content including documents and corresponding summaries for each document is obtained by at least one computing device (block 702). For example, the topic identifier 302 of the training module 124 receives the corpus of textual content 306 including documents 308 with corresponding human-generated summaries 310 as a set of (document, summary) pairs.

[0069] The topic of each corresponding summary is determined by applying the topic vector to each corresponding summary (block 704). For example, the training module 124 first uses a supervised word frequency-based approach to learn the characteristics of the topics. To this end, the frequency module 212 determines the word frequency 214 of the words in each topic-based topic group 210. The word frequency 214 corresponds to the number of occurrences of a given word in the documents 206 of the topic-based topic group 210. The vector generation module 216 obtains the topic group 210 and the word frequency 214 and generates a topic vector 218 for each topic.

[0070] To determine the topics 312 of the summaries 310, the topic identifier 302 applies the topic vector 218 to each corresponding summary 310. For example, the topic identifier 302 computes the dot product between the summary 310 (in its bag-of-words representation) and the previously extracted topic vector 218. In this case, <v s ,e t1 > indicates the dot product between the summary 310 and the topic vector 218 of topic 1. The topic identifier 302 selects the topic with the highest calculated dot product as the determined topic 312 for the corresponding summary 310.

[0071] An intermediate data set is generated to include the documents of the corpus of textual content, the corresponding summary of each document, and the determined topics of each summary (block 706). For example, the training module 124 aggregates all selected documents and their corresponding summaries 310 and determined topics 312 into a triple of <document, topic, summary> in the intermediate data set 304.

[0072] Training data is artificially generated by merging text from a first document and a second document from an intermediate data set to generate a first new document, associating the first new document with the summary and topics of the first document in the training data, and merging text from a second document and the first document from the intermediate data set to generate a second new document, associating the second new document with the summary and topics of the second document in the training data (block 708). For example, the merge module 402 of the training module 124 generates the first new document 416 by merging text from the first document 404 and the second document 410. In one or more implementations, the merge module 402 generates the first new document 416 by sequentially picking text lines from the first document 404 and the second document 410. In some cases, each new text line is added by randomly selecting one of the first document 404 or the second document 410 to extract the new line. This ensures that the text lines from the first document 404 or the second document 410 occur in the same order in the first new document 416 as they did in each respective original document. After the first new document 416 is generated, it is grouped with the first summary 406 and the first topic 408 and then added to the artificially generated training data 126 .

[0073] Next, the merge module 402 repeats these steps to generate a second new document 418 to be grouped with the second summary 412 and the second topic 414. For example, the merge module 402 generates the second new document 418 by merging text from both the second document 410 and the first document 404. In one or more implementations, the merge module 402 generates the second new document 418 by sequentially picking up lines of text from the second document 410 and the first document 404. After the second new document 418 is generated, it is grouped with the second summary 412 and the second topic 414 and then added to the artificially generated training data 126.

[0074] The first document and the second document are discarded (block 710). For example, the training module 124 discards the first document 404, the first summary 406, and the first topic 408, and the second document 410, the second summary 412, and the second topic 414 from the intermediate data set 304. After discarding the first document and the second document, the merging and discarding of steps 708 and 710, respectively, are repeated until the intermediate data set 304 is exhausted.

[0075] Then, the artificially generated training data is used to train the topic-aware coding model and the word generation model to generate topic-based summaries using the artificially generated training data and machine learning (block 712). For example, the training module 124 uses the artificially generated training data 126 to train the word generation model 120 and the topic-aware coding model 122 to generate topic-based summaries 116 using machine learning.

[0076] Figure 8 Depicted is an example process 800 for generating topic-based summaries of textual content using a topic-aware encoding model and a term generation model.

[0077] A request is received by at least one computing device to generate a topic-based summary of textual content tailored to a topic of interest (block 802). As an example, topic-based summary module 118 receives request 110 to generate a topic-based summary 116 of textual content 112 tailored to a topic of interest 114. In some cases, the request may be received from a content author of the textual content 112, such as an author of an article or book. Alternatively, the request may be received from a consumer of the textual content 112 to generate a summary of the textual content. As described throughout, request 110 may include an indication of the textual content 112 and a topic of interest 114.

[0078] The text content is encoded into topic-aware encoded text by a topic-aware encoding model of at least one computing device (block 804). As an example, the topic-aware encoding model 122 generates topic-aware encoded text 502 by encoding the text content 112 word by word into a sequence of topic-aware encodings based on the topic of interest 114 of the request 110. In other words, the input text of the text content 112 is encoded differently based on the desired topic of interest 114 for the summary. To encode the text content 112 based on the topic of interest 114, the topic-aware encoding model selects a topic vector 218 (e.g., a one-hot vector) based on the requested topic of interest 114, then maps each word of the text content 112 to an embedding, and appends the selected topic vector 218 representing the requested topic of interest 114 to each word. This entire concatenated vector is then passed through the neural network of the topic-aware encoding model 122, which outputs an encoding at each step, resulting in a series of topic-based encodings.

[0079] A word generation model of at least one computing device generates, based on topic-aware encoded text, an attention distribution for words from the textual content and a selection probability value for each word in the attention distribution, the selection probability value indicating the corresponding probability of each word being selected as the next word in a topic-based summary of the textual content (block 806). For example, the attention decoder 504 of the word generation model 120 receives the topic-aware encoded text 502 of the textual content 112 as input and generates an attention distribution 506 for words 508 in the textual content 112. The attention distribution 506 indicates a selection probability value 510 for each word 508. The selection probability value 510 indicates the probability of the corresponding word 508 being selected as the next generated word in the topic-based summary 116. The attention decoder 504 determines the selection probability value 510 based on the words in the input textual content 112 and the words of the topic-based summary 116 generated by the word generation model 120.

[0080] The next word of the topic-based summary is selected based on the attention distribution (block 808 ). For example, the selection module 512 of the word generation model 120 selects the next generated word 514 for the topic-based summary 116 based on the attention distribution 506 generated by the attention decoder 504 of the word generation model 120 .

[0081] Feedback of the selected next word is provided to the word generation model, which causes the word generation model to modify the attention distribution based on the feedback of the next generated word for selection of subsequent words of the topic-based summary (block 810). For example, after the next generated word 514 is selected by the selection module 512, the generated word is added to the topic-based summary 116. In addition, the feedback module 516 provides the generated word feedback 518 to the attention decoder 504, which can use the generated word feedback 518 to select subsequent words of the topic-based summary 116, thereby maintaining consistency in word generation of the topic-based summary 116.

[0082] Having described example procedures according to one or more implementations, consider now example systems and devices that can be used to implement the various techniques described herein.

[0083] Example systems and devices

[0084] Figure 9 An example system, generally at 900, is shown including an example computing device 902, which represents one or more computing systems and / or devices that may implement the various techniques described herein. This is illustrated by the inclusion of a topic-based summary module 118. The computing device 902 may be, for example, a server of a service provider, a device associated with a client (e.g., a client device), a system-on-chip, and / or any other suitable computing device or computing system.

[0085] The example computing device 902 shown includes a processing system 904, one or more computer-readable media 906, and one or more I / O interfaces 908 communicatively coupled to each other. Although not shown, the computing device 902 may also include a system bus or other data and command transmission system that couples various components to each other. The system bus may include any one or a combination of the following: a different bus structure, such as a memory bus or memory controller, a peripheral bus, a universal serial bus; and / or a processor or local bus utilizing any of a variety of bus architectures. Various other examples are also contemplated, such as control and data lines.

[0086] Processing system 904 represents the functionality of performing one or more operations using hardware. Thus, processing system 904 is shown as including hardware elements 910 that can be configured as processors, functional blocks, and the like. This can include other logic devices implemented in hardware as application-specific integrated circuits or formed using one or more semiconductors. Hardware elements 910 are not limited by the materials from which they are formed or the processing mechanisms employed therein. For example, a processor can include semiconductors and / or transistors (e.g., electronic integrated circuits (ICs)). In such a context, processor-executable instructions can be electronically executable instructions.

[0087] Computer-readable media 906 is shown as including memory / storage 912. Memory / storage 912 represents memory / storage capacity associated with one or more computer-readable media. Memory / storage 912 may include volatile media (such as random access memory (RAM)) and / or non-volatile media (such as read-only memory (ROM), flash memory, optical disks, magnetic disks, etc.). Memory / storage 912 may include fixed media (e.g., RAM, ROM, fixed hard drives, etc.) and removable media (e.g., flash memory, removable hard drives, optical disks, etc.). Computer-readable media 906 may be configured in various other ways, as further described below.

[0088] The input / output interface 908 represents functionality that allows a user to input commands and information to the computing device 902, and also allows information to be presented to the user and / or other components or devices using various input / output devices. Examples of input devices include a keyboard, a cursor control device (e.g., a mouse), a microphone, a scanner, touch functionality (e.g., a capacitive sensor or other sensor configured to detect physical touch), a camera (e.g., which may employ visible or invisible wavelengths, such as infrared frequencies, for identifying motion as gestures that do not involve touch), and the like. Examples of output devices include a display device (e.g., a monitor or projector), speakers, a printer, a network card, a tactile response device, and the like. Thus, the computing device 902 can be configured in various ways, as further described below, to support reader interaction.

[0089] Various techniques may be described herein in the general context of software, hardware elements, or program modules. Typically, such modules include routines, programs, objects, elements, components, data structures, etc. that perform specific tasks or implement specific abstract data types. As used herein, the terms "module," "function," and "component" generally refer to software, firmware, hardware, or a combination thereof. A feature of the techniques described herein is that they are platform-independent, meaning that they can be implemented on a variety of commercial computing platforms with a variety of processors.

[0090] An implementation of the described modules and techniques may be stored on or transmitted via some form of computer-readable media. Computer-readable media may include various media that can be accessed by computing device 902. By way of example, and not limitation, computer-readable media may include "computer-readable storage media" and "computer-readable signal media."

[0091] "Computer-readable storage media" may refer to media and / or devices that enable persistent and / or non-transitory storage of information, as compared to mere signal transmission, carrier waves, or the signals themselves. Thus, computer-readable storage media refers to non-signal-bearing media. Computer-readable storage media include hardware, such as volatile and non-volatile, removable and non-removable media and / or storage devices implemented with methods or technologies suitable for the storage of information, such as computer-readable instructions, data structures, program modules, logic elements / circuits, or other data. Examples of computer-readable storage media may include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical storage, hard disks, cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or other storage devices, tangible media, or articles of manufacture suitable for storing the desired information and accessible by a computer.

[0092] "Computer-readable signal media" may refer to signal-bearing media that is configured to send instructions to the hardware of computing device 902, such as via a network. Signal media generally may embody computer-readable instructions, data structures, program modules, or other data in a modulated data signal, such as a carrier wave, data signal, or other transport mechanism. Signal media also include any information delivery media. The term "modulated data signal" refers to a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, communication media includes wired media, such as a wired network or direct-wired connection, and wireless media, such as acoustic, RF, infrared, and other wireless media.

[0093] As previously described, hardware elements 910 and computer-readable media 906 represent modules, programmable device logic, and / or fixed device logic implemented in hardware that may be employed in some embodiments to implement at least some aspects of the technology described herein, such as executing one or more instructions. The hardware may include components of an integrated circuit or system on a chip, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a complex programmable logic device (CPLD), and other implementations of silicon or other hardware. In this context, the hardware may operate as a processing device that performs program tasks defined by instructions and / or logic embodied in the hardware and for storing instructions for execution, such as the computer-readable storage medium described previously.

[0094] The aforementioned combination can also be used to implement the various technologies described herein. Therefore, software, hardware or executable modules can be implemented as one or more instructions and / or logic embodied on some form of computer-readable storage medium and / or by one or more hardware elements 910. The computing device 902 can be configured to implement specific instructions and / or functions corresponding to the software and / or hardware modules. Therefore, the implementation of the module as software that can be executed by the computing device 902 can be at least partially implemented in hardware, for example, through the use of computer-readable storage media and / or hardware elements 910 of the processing system 904. The instruction / or function can be executable / operable by one or more products (e.g., one or more computing devices 902 and / or processing systems 904) to implement the technology, modules and examples described herein.

[0095] The technology described herein can be supported by various configurations of computing device 902 and is not limited to the specific examples of the technology described herein. The functionality can also be implemented in whole or in part through the use of a distributed system, such as on a "cloud" 914 via a platform 916 as described below.

[0096] Cloud 914 includes and / or represents a platform 916 for resources 918. Platform 916 abstracts the underlying functionality of the hardware (e.g., servers) and software resources of cloud 914. Resources 918 may include applications and / or data that can be utilized while executing computer processing on servers remote from computing device 902. Resources 918 may also include services provided over the Internet and / or over a subscriber network such as a cellular network or a Wi-Fi network.

[0097] The platform 916 can abstract resources and functionality to connect the computing device 902 with other computing devices. The platform 916 can also be used to abstract the scaling of resources to provide corresponding scaling levels to meet the demand for resources 918 implemented via the platform 916. Therefore, in an interconnected device embodiment, the implementation of the functionality described herein can be distributed throughout the system 900. For example, the functionality can be implemented partially on the computing device 902 and via the platform 916 that abstracts the functionality of the cloud 914.

[0098] in conclusion

[0099] Although the invention has been described in language specific to structural features and / or methodological acts, it should be understood that the invention defined in the appended claims is not necessarily limited to the specific features or acts described. Rather, the specific features and acts are disclosed as example forms of implementing the claimed invention.

Claims

1. A method implemented by at least one computing device in a digital media environment for generating a topic-based summary of textual content tuned to topics of interest, the method comprising: receiving a request to generate the topic-based summary of textual content adjusted to the topic of interest, the request including the textual content and defining the topic of interest; encoding, by a topic-aware encoding model of the at least one computing device, the textual content with topic tags corresponding to the requested topic of interest to generate topic-aware encoded text; selecting, by a term generation model of the at least one computing device, a next term for the topic-based summary from the topic-aware encoded text, the term generation model trained using machine learning on training data for generating topic-based summaries, the training data comprising a plurality of documents, a corresponding summary for each document, and a corresponding topic for each summary; as well as providing feedback of the selected next word to the word generation model during the generating of the topic-based summary, the feedback causing the word generation model to select a subsequent word for the topic-based summary based on both the topic-aware encoded text and the feedback of the selected next word; and The topic-based summary of the text content tuned to the topic of interest is output, the topic-based summary including the terms selected by the term generation model.

2. The method of claim 1, wherein the encoding further comprises generating the topic-aware encoded text by mapping each word of the text content to the topic label corresponding to the requested topic of interest. 3 . The method of claim 2 , wherein the topic labels indicate word preferences of the topic of interest learned from a supervised word frequency based scheme.

4. The method according to claim 1, wherein encoding the text content to generate topic-aware encoded text increases the selection probability value of the preferred word of the topic of interest in the text content through the word generation model.

5. The method of claim 1, wherein the training data is generated by modifying a dataset of documents and corresponding summaries to include the topic of each summary.

6. The method of claim 1, wherein the topic-aware encoding model and the term generation model are operable to generate a topic-based summary tuned to a plurality of different topics of interest.

7. The method of claim 1 , receiving an additional request to generate a different topic-based summary of textual content adjusted to a different topic of interest, the request including the textual content and defining the different topic of interest; The different topic-based summaries of the textual content adjusted to the different topics of interest are output.

8. The method of claim 1, wherein the topic-aware encoding model and the term generation model comprise neural networks.

9. A method implemented by at least one computing device in a digital media environment, the method comprising: Obtaining an intermediate data set, the intermediate data set comprising documents, a corresponding summary for each document, and a topic for each summary; Generate training data by doing the following: selecting a first document and a second document from the intermediate data set; Merging text from a first document and a second document from the intermediate data set to generate a first new document and associating the first new document with corresponding summaries and topics of the first document in the training data, and merging text from the second document and the first document from the intermediate data set to generate a second new document and associating the second new document with corresponding summaries and topics of the second document in the training data; discarding the first document and the second document; as well as Repeating the merging and discarding until the intermediate data set is exhausted; and training a topic-aware encoding model and a word generation model to generate topic-based summaries using the generated training data and machine learning.

10. The method according to claim 9, further comprising generating the intermediate data set by: obtaining, by the at least one computing device, textual content comprising the documents and the corresponding summary for each document; and The subject matter of each corresponding summary is determined.

11. The method of claim 10 , wherein the topic of each summary is determined by applying a topic vector to each respective summary, the topic vector being generated by: obtaining, by the at least one computing device, documents tagged with subject tags defining a subject matter of each respective document; grouping the documents into topic-based groups based on the topic tags; determining frequencies of terms in the documents for the topic-based groups; The topic vector is generated for each topic-based group based on the frequency of terms in the documents of the topic-based group.

12. The method of claim 11 , wherein determining the topic of each respective summary by applying a topic vector to each respective summary further comprises: For each summary, computing a dot product between the summary and the topic vector; as well as The topic having the highest calculated dot product is selected as the determined topic for the corresponding summary.

13. The method according to claim 12, further comprising: calculating a confidence value for the determined topic by dividing the highest calculated dot product by a second highest calculated dot product; as well as If the confidence value is below a confidence threshold, the document and corresponding summary are discarded from the intermediate data set.

14. A system implemented in a digital media environment for generating a topic-based summary of textual content tuned to topics of interest, the system comprising: at least one processor; a memory having computer-readable instructions stored thereon, the computer-readable instructions being executable by the processor to implement a topic-based summary module to perform operations comprising: receiving a request to generate the topic-based summary of textual content adjusted to the topic of interest, the request including the textual content and defining the topic of interest; encoding the text content using a topic label corresponding to the requested topic of interest by a topic-aware encoding model to generate a topic-aware encoded text; selecting, by a term generation model of at least one computing device, a next term for the topic-based summary from the topic-aware encoded text, the term generation model trained using machine learning on training data for generating topic-based summaries, the training data comprising a plurality of documents, a corresponding summary for each document, and a corresponding topic for each summary; as well as providing feedback of the selected next word to the word generation model during the generating of the topic-based summary, the feedback causing the word generation model to select a subsequent word for the topic-based summary based on both the topic-aware encoded text and the feedback of the selected next word; and The topic-based summary of the text content tuned to the topic of interest is output, the topic-based summary including the terms selected by the term generation model.

15. The system of claim 14, wherein the encoding further comprises generating the topic-aware encoded text by mapping each word of the text content to the topic label corresponding to the requested topic of interest.

16. The system of claim 15, wherein the topic labels indicate word preferences of the topic of interest learned from a supervised word frequency based scheme.

17. The system of claim 14, wherein encoding the text content to generate topic-aware encoded text increases selection probability values of preferred terms of the topic of interest in the text content through the term generation model.

18. The system of claim 14, wherein the topic-aware encoding model and the term generation model are trained using machine learning on training data for generating topic-based summaries tuned to topics of interest.

19. The system of claim 18, wherein the training data is generated by modifying a dataset of documents and corresponding summaries to include the topic of each summary.

20. The system of claim 14, wherein the topic-aware encoding model and the term generation model are operable to generate a topic-based summary tuned to a plurality of different topics of interest.

Images