Unsupervised Text Summarization with Reinforcement Learning

Abstract

A computer-implemented method for performing Q-learning using a language model for unsupervised text summarization is proposed. The method includes: mapping each word of a sentence to a vector using word embeddings through a deep learning natural language processing model; assigning each word to an action and an operation state; determining a state for each of the words whose operation state is represented as "not operated" by calculating a local encoding and a global encoding; and connecting the local encoding and the global encoding in a self-attention manner, where the local encoding is calculated based on the vector, action, and operation state of the word, and the global encoding is calculated based on each local encoding of the word; and determining the Q-value of each word based on the state according to each of three actions via an editing agent.

Description

Background Art

[0001] The present invention generally relates to unsupervised text summarization, and more particularly, to unsupervised text summarization using a Q-language method that utilizes a language model.

[0002] A vast amount of information exists in text form, such as free, unstructured, or semi-structured text, including many database fields, reports, memorandums, emails, websites, and news articles. This information can be of interest to various individuals and entities, such as business managers, market analysts, researchers, private companies, public companies, and government agencies. As a result, methods for analyzing text resources have been developed. Text analysis or text data analysis can involve functions such as document classification, document clustering, information visualization, text or document summarization, and document cross-referencing.

[0003] Text summarization refers to the technique of shortening long text segments. The present invention aims to create a coherent and fluent summary that only has the key points outlined in the literature. Automatic text summarization is a common problem in machine learning and natural language processing (NLP). Summary of the Invention

[0004] According to one embodiment, there is provided a method for performing Q-learning using a language model for unsupervised text summarization. The method includes: mapping each word of a sentence to a vector using word embeddings via a deep learning natural language processing (NPL) model; assigning each word to an action and an operation state; determining a state for each of the words whose operation state represents "not operated" by calculating a local encoding and a global encoding and concatenating the local encoding and the global encoding, wherein the local encoding is calculated based on the word's vector, action, and operation state, and the global encoding is calculated in a self-attention manner based on each of the local encodings of the words; and determining a Q-value for each word based on the state via an editing agent for each of three actions.

[0005] According to another embodiment, there is provided a system for performing Q-learning using a language model for unsupervised text summarization. The system includes a memory and one or more processors communicatively coupled to the memory, the one or more processors being configured to map each word of a sentence to a vector using word embeddings via a deep learning natural language processing (NPL) model, assign each word to an action and an operation state, determine a state for each of the words whose operation state represents "not operated" by calculating a local encoding and a global encoding, and concatenate the local encoding and the global encoding, wherein the local encoding is calculated based on the word's vector, action, and operation state, and the global encoding is calculated in a self-attention manner based on each of the local encodings of the words, and determine a Q-value for each word based on the state via an editing agent for each of three actions.

[0006] According to another embodiment, a non - transitory computer - readable storage medium is provided that includes a computer - readable program for performing Q - learning using a language model for unsupervised text summarization. The non - transitory computer - readable storage medium performs the following steps: mapping each word of a sentence to a vector using word embeddings through a deep - learning natural language processing (NPL) model; assigning each word to an action and an operation state; for each word whose operation state represents "not operated", by calculating local encoding and global encoding and concatenating the local encoding and the global encoding, where the local encoding is calculated based on the vector, action, and operation state of the word, and the global encoding is calculated in a self - attention manner based on each of the local encodings of the words; and via an editing agent, determining the Q - value of each word based on the state according to each of three actions.

[0007] According to one embodiment, a method for performing Q - learning using a language model for unsupervised text summarization is provided. The method includes: mapping each word of a sentence to a vector using word embeddings through a deep - learning natural language processing (NPL) model; assigning each word to an action and an operation state; determining a state for each of the words whose operation state represents "not operated" by calculating local encoding and global encoding, where the local encoding and the global encoding are concatenated; and via an editing agent, determining the Q - value of each word based on the edit operations "replace", "keep", and "remove".

[0008] According to another embodiment, a system for performing Q - learning using a language model for unsupervised text summarization is provided. The system includes a memory and one or more processors in communication with the memory, the one or more processors being configured to map each word of a sentence to a vector using word embeddings through a deep - learning natural language processing (NPL) model, assign each word to an action and an operation state, determine a state for each of the words whose operation state represents "not operated" by calculating local encoding and global encoding, where the local encoding and the global encoding are concatenated; and via an editing agent, determine the Q - value of each word based on the edit operations "replace", "keep", and "remove".

[0009] It should be noted that the exemplary embodiments are described with reference to different subjects. Specifically, some embodiments are described with reference to method - type claims, while other embodiments are described with reference to apparatus - type claims. However, those skilled in the art will understand from the above and the following descriptions that, unless otherwise indicated, any combination between features related to different subjects, in particular any combination between the features of method - type claims and the features of apparatus - type claims, in addition to any combination of features belonging to one type of subject, is also considered to be described within this document.

[0010] These and other features and advantages will become apparent from the following detailed description of illustrative embodiments of the invention when read in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The present invention will be described in detail in the following description of preferred embodiments with reference to the following drawings, in which:

[0012] Figure 1 is a block diagram / flowchart showing a Q-learning method using a language model for unsupervised text summarization according to an embodiment of the present invention;

[0013] Figure 2 is a table showing states and actions of a Q-learning method using a language model for unsupervised text summarization according to an embodiment of the present invention;

[0014] Figure 3 is an exemplary system for iterative action prediction according to an embodiment of the present invention;

[0015] Figure 4 is an exemplary mechanism for deterministic transformation by a language model converter according to an embodiment of the present invention;

[0016] Figure 5 is a block diagram / flowchart of an exemplary method for implementing a Q-learning method using a language model for unsupervised text summarization according to an embodiment of the present invention;

[0017] Figure 6 is an example of calculating a reward after compression and reconstruction using an action sequence in each step according to an embodiment of the present invention;

[0018] Figure 7 is an exemplary processing system including an editing agent and a language model converter according to an embodiment of the present invention;

[0019] Figure 8 is a block diagram / flowchart of an exemplary cloud computing environment according to an embodiment of the present invention; and

[0020] Figure 9 is a schematic diagram of an exemplary abstract model layer according to an embodiment of the present invention.

[0021] In all the drawings, the same or similar reference numerals denote the same or similar elements. DETAILED DESCRIPTION OF THE INVENTION

[0022] Embodiments according to the present invention provide a method and apparatus for unsupervised text summarization using Q-learning with a language model. Text summarization is a technique for generating a concise and precise summary of a large amount of text while focusing on the parts that convey useful information without losing the overall meaning. The goal of automatic text summarization is to convert a long document into a shortened version, which can be difficult and expensive to do manually. Machine learning algorithms can be trained to understand the document and identify sections that convey important facts and information before producing a summarized text.

[0023] As large amounts of data cycle through the digital space, there is a need to develop machine learning algorithms that can automatically shorten longer texts and deliver an accurate summary that can convey the intended message. Additionally, applying text summarization reduces reading time, speeds up the information search process, and increases the amount of information that can fit in a given area. There are two main types of summarized text in natural language processing (NLP), namely, extraction-based summarization and abstraction-based summarization. Extractive text summarization techniques involve extracting key phrases from the source document and combining them to form a summary. The extraction is performed according to a defined metric without making any changes to the text. Abstraction techniques require interpreting and shortening parts of the source document. When applied to text summarization in deep learning problems, abstraction can overcome the syntactic inconsistencies of the extraction method. Abstract text summarization algorithms create new phrases and sentences that relay the most useful information from the original text. Conventional methods use two encoder-decoders, one set of encoder-decoders for compression learning and another set for reconstruction learning.

[0024] In contrast, exemplary embodiments of the present invention approximate summarization as an editing process by using a Q-learning method with a language model. The Q-learning method employs two modules, namely, an editing agent and a language model (LM) transformer. The editing agent determines an action for each word in a given sentence, while the LM transformer deterministically compresses and reconstructs with the given action. The editing agent is trained to predict the appropriate action that leads to favorable compression and reconstruction by the LM transformer. In action prediction, the editing agent leverages global language knowledge through a language model trained with a large amount of data. Thus, Q-learning is applied to unsupervised text summarization. Q-learning is an off-policy reinforcement learning algorithm that seeks to find the best action to take given the current state. Q-learning is considered off-policy because the Q-learning function learns from actions outside the current policy. More specifically, Q-learning seeks to learn a policy that maximizes the total reward.

[0025] It should be understood that the present invention will be described in accordance with a given illustrative architecture; however, other architectures, structures, substrate materials, and process features and steps / blocks may vary within the scope of the present invention. It should be noted that, for clarity, some features may not be shown in all of the figures. This should not be construed as a limitation on the scope of any particular embodiment or description or claim.

[0026] Figure 1 is a block diagram / flowchart showing a Q-learning method using a language model for unsupervised text summarization according to an embodiment of the present invention.

[0027] The block / flowchart shows a traditional method and a Q-learning method using a language model. In the traditional method, a pair of encoder-decoders is employed. For example, the sentence "Machine learning is not perfect"5 is fed into the compression encoder-decoder 10, and the compression encoder-decoder 10 outputs the sentence 12, i.e., "AI is not imperfect". The output sentence 12 is provided to the reconstruction encoder-decoder 20, and the reconstruction encoder-decoder 20 reconstructs the initial sentence 22, i.e., "Machine learning is not perfect". Thus, the first encoder-decoder 10 is used for compression, while the second encoder-decoder 20 is used for reconstruction.

[0028] In contrast, the Q-learning method using a language model provides the sentence 102, i.e., "Machine learning is not perfect" and feeds it into the editing agent 104. The editing agent 104 determines an action 106 for each word. There are three actions 106. The three actions 106 are "remove", "replace", and "keep". Each word of the sentence 102 and its corresponding editing action 106 are fed into the language model (LM) transformer 108. The LM transformer 108 deterministically compresses and reconstructs for each given action. The output of the LM transformer 108 is the sentence "AI is not imperfect" (output 110) which is a compression transformation, while the sentence "Machine learning is not perfect" (output 112) is a reconstruction transformation.

[0029] Therefore, the Q - learning method using a language model generalizes summarization as an editing process, where each word is given one of three actions. The actions are "remove", "replace", and "keep". These actions can also be referred to as edit operations. Two modules are employed. The first module is the editing agent 104, and the second module is the LM converter 108. The editing agent 104 determines which action 106 to assign to each word of the sentence 102, while the LM converter 108 deterministically compresses and reconstructs for each given action. Compression is the transformation from the original sentence to a compressed sentence (e.g., summary). Reconstruction is the transformation from the compressed sentence back to the original sentence. The editing agent 104 is trained to predict the appropriate actions that lead to favorable compression and reconstruction by the LM converter 108. In action prediction, the editing agent 104 leverages global language knowledge through a language model 108 trained with a large amount of data.

[0030] Broadly speaking, reinforcement learning differs from supervised learning in that instead of presenting correct input - output pairs, the machine (software agent) learns to take actions in some environment to maximize some form of reward or minimize cost. Taking an action moves the environment / system from one state to another, and in the specific case of Q - learning, the "Q - value" of a state - action combination is computed, which describes the action - value function that can be used to determine the expected utility of an action.

[0031] Q - learning is a model - free reinforcement learning algorithm for learning a policy that tells an agent what action to take in what situation. Q - learning does not require a model of the environment (hence the connotation "model - free"), and Q - learning can handle problems of stochastic transitions and rewards without adjustment. For any finite Markov decision process (FMDP), Q - learning finds the optimal policy in the sense of maximizing the expected value of the total reward over any and all successive steps starting from the current state. Q - learning can identify the optimal action - selection policy and a partially stochastic policy for any given FMDP and given infinite exploration time. The "Q" nomenclature returns the function that provides the reward for reinforcement and can be said to represent the "quality" of the action taken in a given state.

[0032] Reinforcement learning involves an agent, a set of states, and a set of actions for each state. By performing an action, the agent transitions from one state to another. Performing an action in a particular state provides a reward (numerical score) to the agent. The goal of the agent is to maximize its total reward. The agent achieves this by effectively influencing the current action with potential future rewards by adding the maximum reward obtainable from future states to the reward for reaching its current state. This potential return is a weighted sum of the expected values of the rewards for all future steps starting from the current state. This will be referred to Figure 2 To further illustrate with an example.

[0033] Figure 2 It is a table showing the states and actions of a Q - learning method using a language model for unsupervised text summarization according to an embodiment of the present invention.

[0034] When performing Q - learning, a Q - table or matrix of the form [state, action] is created and the values are initialized to zero. Then the Q - values are updated and stored after an episode. This Q - table becomes a reference table for the agent to select the best action based on the Q - values. The next step is for the agent to simply interact with the environment and update the state - action pairs in the Q - table Q[state, action]. The agent uses the Q - table as a reference and looks at all possible actions for a given state.

[0035] Specifically, the editing agent 104 makes decisions for each word in the sentence. In the current case, the sentence 130 is "Machine learning is not perfect". This sentence 130 consists of 5 words. The "state" 122 is the words in each sentence. The "action" 124 is one of the three editing operations 106, that is, the "action" 124 is "remove", "keep", or "replace". In one example, the agent 104 assigns the action "remove" to the word "Machine". The word "learning" can be assigned the action "remove". The word "is" can be assigned the action "replace". The word "not" can be assigned the action "replace". The word "perfect" can be assigned the action "keep". The agent 104 makes decisions for each word in the sentence one by one. Thus, predictions occur one after another. The agent 104 decides the action for each word in the sentence. The summary is generated word by word. The question is how the agent 104 determines what action to give to each word in the sentence. The following Figure 3 Outlines the method.

[0036] Figure 3 It is an exemplary system for iterative action prediction according to an embodiment of the present invention.

[0037] The iterative action prediction system 200 enables the editing agent 104 to assign actions to each word of a sentence. A state 205 is provided to the deep learning NPL model 210. For example, a first state 205(1), a second state 205(2), and a third state 205(3) are fed into the deep learning NPL model 210. Each state represents a word of the sentence. The deep learning NPL model 210 can be, for example, the Bidirectional Encoder Representations from Transformers (BERT) 210 (a deep learning NPL model from Google). BERT 210 outputs a vector 222, and the action 224 and operation state 226 of each word 205 are determined by previous operations. In this first step, all actions are initialized with "hold", and all operations are assigned the "not operated" state.

[0038] Local encoding 230 is performed, followed by global encoding 240. At 250, the local encoding and the global encoding are concatenated. Both the local encoding 230 and the global encoding 240 are weighted sums of vectors. However, the local encoding 230 is different from the global encoding 240 based on what vectors are employed. In the local encoding 230, the feature vectors (BERT embeddings, actions, operation states) of a specific word are combined. In the global encoding 240, the local encoding vectors of all the words in the sentence are combined. Thus, there are two vectors (local vector and global vector) for each word. As described, at 250, the local encoding and the global encoding are concatenated.

[0039] At 260, the highest Q value is calculated. A token 270 indicates a first prediction, while a token 272 indicates a second prediction, and a token 274 indicates a third prediction. In the first prediction 270, the third state 205(3) is processed.

[0040] Thereafter, in the second prediction 272, the state 3 (205(3))) has been operated, so 228 indicates the state "operated" of state 3. States 1 and 2 remain visible (since they have not been operated). In the third prediction 274, states 1 and 3 have been operated, so 229 indicates the state "operated" of state 1. As a result, each state is processed during each stage or prediction phase. 228 and 229 thus indicate the updated predicted actions and the updated operation states. The prediction ends when there are no more unoperated words. In other words, the prediction ends when all words have the "operated" state. Thus, according to the iterative action prediction system 200, an action sequence of the sentence can be generated for each time step, which enables compression and reconstruction for each time step. Additionally, this allows determining which action makes the compression and reconstruction worse.

[0041] In the world of NLP, representing words or sentences in vector form or as word embeddings opens the door to various potential applications. This ability to encode words into vectors is a powerful tool for NLP tasks such as calculating semantic similarity between words, which can be used to build semantic search engines. For example, Google's application of word embeddings to better understand search queries is called BERT. BERT is one of the most powerful language models to have become popular in the machine learning community.

[0042] The BERT (Bidirectional Encoder Representations from Transformers) model is pre-trained using a large number of sentences. Briefly, the training is done by masking several words (about 15% of the words) in the sentence and assigning the model the task of predicting the masked words. And when the model is trained to make predictions, the model learns to generate powerful internal representations of the words as word embeddings.

[0043] Embeddings are just low-dimensional representations of points in a higher-dimensional vector space. In the same way, word embeddings are dense vector representations of words in a lower-dimensional space. The first word embedding model using neural networks was made public in 2013. Since then, word embeddings are encountered in almost every NLP model in actual use today. Of course, the reason for this widespread adoption is their efficiency. By converting words into embeddings, the semantic importance of words can be modeled in numerical form and mathematical operations can be performed on them.

[0044] Figure 4 is an exemplary mechanism for deterministic transformation through a language model transformer according to an embodiment of the present invention.

[0045] Figure 4 Shows how to generate compressed and reconstructed sentences by using an LM transformer. Specifically, sentence 130 is "Machine learning is not perfect". This sentence 130 consists of 5 words. The first word 131 is "Machine", the second word 133 is "learning", the third word 135 is "is", the fourth word 137 is "not", and the fifth word 139 is "perfect". The editing agent 104 selects actions 106 for each of the words 131, 133, 135, 137, 139 by using the iterative action prediction system 200. The first word 131 has been assigned action 106(1) or "remove", the second word 133 has been assigned action 106(2) or "replace", the third word 135 has been assigned action 106(3) or "keep", the fourth word 137 has been assigned action 106(1) or "remove", and the fifth word 139 has been assigned action 106(2) or "replace".

[0046] In the next step, the empty token 302 (or ε) is assigned to the actions "remove" and "replace". Thus, in this example, 4 empty tokens 302 have been assigned. The third word 135 remains untouched. The next step is the compression phase. In the compression phase, the words "Machine" and "learning" are compressed into "AI". As a result, the word "AI" or L(z2) 304 is inserted into the second block. Additionally, in the compression phase, the words "not" and "perfect" are compressed into "imperfect". As a result, the word "imperfect" or L(z5) 306 is inserted into the fifth block. Two empty tokens 302 are also retained. In the compression phase, the "remove" word is not predicted. Instead, in the compression phase, the "replace" word is predicted. In the reconstruction phase, the sentence 130 is reconstructed. Thus, the sentence "Machinelearning is not perfect" is restored. In the "remove" and "replace" slots, words are predicted to reconstruct the original sentence 130. Thus, by using the ability of the LM to guess parts of the sentence from the fragments, the compression and reconstruction are deterministically operated. As a result, there is no training for generation, which alleviates the problem of having difficulty co-training multiple generators (as shown in the conventional method; Figure 1 the left side).

[0047] Figure 5 is a block diagram / flowchart of an exemplary method for implementing a Q-learning method using a language model for unsupervised text summarization according to an embodiment of the present invention.

[0048] In block 320, each word of the sentence is mapped to a vector by using word embeddings.

[0049] In block 322, each word is assigned to an action and an operation state.

[0050] In block 324, for each word whose operation state represents "not operated", a state is determined by calculating a local encoding and a global encoding and concatenating the local encoding and the global encoding, where the local encoding is calculated based on the vector, action, and operation state of the word, and the global encoding is calculated in a self-attention manner based on each of the local encodings of the words.

[0051] In self-attention or K = V = Q, if the input is, for example, a sentence, each word in the sentence needs to undergo an attention calculation. The goal is to learn the dependencies between the words in the sentence and use this information to capture the internal structure of the sentence. Since self-attention is applied both to each word and to all words together, the longest possible path is one, no matter how far apart they are, enabling the system to capture long-distance dependency relationships.

[0052] At block 326, based on the state, the Q-value of each word is determined according to each of the three actions ("replace", "keep", and "remove").

[0053] At block 328, the action and operation state of the word with the highest Q-value are updated to the action and operation state with the highest Q-value, "operated".

[0054] At block 330, the above determination step and update step are repeated until all operation states of the above words represent "operated".

[0055] At block 332, the sentence is converted into a masked sentence, where the words with "keep" are retained, while the words with "replace" and "remove" are changed to empty tokens.

[0056] At block 334, the sentence is compressed by using a masked language model for predicting the masked part in a given sentence to convert each empty token with "replace" in the masked sentence into a predicted word.

[0057] Figure 6 It is an example of calculating the reward after compression and reconstruction using the action sequence in each step according to an embodiment of the present invention.

[0058] In Figure 6 , after compression and reconstruction using the action sequence in each step, then in each step, the reward (e.g., value) of the action is calculated according to the change of the generated sentence, which specifically indicates what action is good or bad under what circumstances. The traditional method calculates the reward only from the final output. This makes the reward scarce and the training of the agent more difficult. Instead, in the present invention, there is no experience after the agent obtains a negative return. For example, if the original sentence 350 is "May the force be with you", different rewards can be assigned for different states.

[0059] In the first state 352, the action "remove" (354) can be given to the word "May". In the compression stage 360, the word "May" is removed. In the reconstruction stage 370, the original sentence 350 is correctly reconstructed. Therefore, the reward 380 is a positive reward.

[0060] In the second state 352, the action "remove" (354) can be given to the word "May" and the word "the". In the compression stage 360, the words "May" and "the" are removed. In the reconstruction stage 370, the original sentence 350 is correctly reconstructed. Therefore, the reward 380 is a positive reward.

[0061] In the third state 352, the actions "remove" (354) can be given to the words "May", "the", and "force". "At compression level 360, the words "May", "the", and "force" are removed. In the reconstruction phase 370, the original sentence 350 is not reconstructed correctly. The 3 missing words are reconstructed with "I will always" instead of "May the force". Thus, the reward 380 is a negative reward. This is sequential reward design.

[0062] Figure 7 is an exemplary processing system according to an embodiment of the present invention.

[0063] Now referring to Figure 7 , the general-purpose computer system 400 is programmed to implement functions corresponding to those shown in Figure 1 , Figure 3 and Figure 4 . The system includes a deep Q-learner 410, which includes a processor 412, a memory 414, an editing agent 104, and an LM converter 108. The memory 414 can store, for example, neural network code, action selection code, target Q generation code, and weight update code. The deep learner 410 receives a state 404 from the system or neural network environment 402 and sends an action 406 back to the system or neural network environment 402.

[0064] Figure 8 is a block diagram / flowchart of an exemplary cloud computing environment according to an embodiment of the present invention.

[0065] It should be understood that although the present invention includes a detailed description of cloud computing, the implementation of the teachings described herein is not limited to a cloud computing environment. Instead, embodiments of the present invention can be implemented in conjunction with any other type of computing environment now known or later developed.

[0066] Cloud computing is a service delivery model for enabling convenient on-demand network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services), which can be rapidly provisioned and released with minimal management effort or interaction with the provider of the service. The cloud model can include at least five characteristics, at least three service models, and at least four deployment models.

[0067] The characteristics are as follows:

[0068] On-demand self-service: Cloud consumers can unilaterally and automatically provision computing capabilities such as server time and network storage as needed, without human interaction with the provider of the service.

[0069] Wide Area Network Access: The capabilities are available over a network and accessed through standard mechanisms that facilitate use by heterogeneous thin or thick client platforms (such as mobile phones, laptops, and PDAs).

[0070] Resource Pooling: The provider's computing resources are pooled to serve multiple consumers using a multi-tenant model, where different physical and virtual resources are dynamically assigned and reassigned according to demand. There is a location-independent aspect, as consumers generally do not control or know the exact location of the provided resources, but can specify a location at a higher level of abstraction (such as country, state, or data center).

[0071] Rapid Elasticity: In some cases, the ability to rapidly scale out and scale in can be provided quickly and elastically. To the consumer, the available capacity for provisioning generally appears unlimited and can be purchased in any quantity at any time.

[0072] Measured Service: The cloud system automatically controls and optimizes resource use by leveraging metering capabilities at a certain level of abstraction appropriate to the type of service (such as storage, processing, bandwidth, and active user accounts). Resource use can be monitored, controlled, and reported, providing transparency for both the provider and consumer of the utilized service.

[0073] The service models are as follows:

[0074] Software as a Service (SaaS): The capabilities provided to the consumer are to use the provider's applications running on the cloud infrastructure. The applications can be accessed from various client devices through a thin client interface such as a web browser (e.g., web-based email). The consumer does not manage or control the underlying cloud infrastructure, including the network, servers, operating systems, storage, or even individual application capabilities, with the possible exception of limited user-specific application configuration settings.

[0075] Platform as a Service (PaaS): The capabilities provided to the consumer are to deploy the applications created or acquired by the consumer onto the cloud infrastructure, where the applications are created using programming languages and tools supported by the provider. The consumer does not manage or control the underlying cloud infrastructure, including the network, servers, operating systems, or storage, but has control over the deployed applications and possibly the configuration of the application hosting environment.

[0076] Infrastructure as a Service (IaaS): The capabilities provided to the consumer are to provide processing, storage, networking, and other basic computing resources where the consumer can deploy and run arbitrary software, which can include operating systems and applications. The consumer does not manage or control the underlying cloud infrastructure, but has control over the operating systems, storage, deployed applications, and possibly limited control over selected networking components (such as a host firewall).

[0077] The deployment models are as follows:

[0078] Private cloud: The cloud infrastructure is for the exclusive use of an organization's operations. It can be managed by the organization or a third party and can exist either on - premise or off - premise.

[0079] Community cloud: The cloud infrastructure is shared by several organizations and supports a specific community with shared concerns (e.g., tasks, security requirements, policies, and compliance considerations). It can be managed by the organization or a third party and can exist either on - premise or off - premise.

[0080] Public cloud: The cloud infrastructure is available for the general public or large industry groups and is owned by an organization that sells cloud services.

[0081] Hybrid cloud: The cloud infrastructure is a combination of two or more clouds (private, community, or public) that remain unique entities but are bound together by standardized or proprietary technologies (e.g., cloud bursting for load balancing between clouds) that enable data and application portability.

[0082] The cloud computing environment is service - oriented, with a focus on statelessness, low coupling, modularity, and semantic interoperability. At the core of cloud computing is an infrastructure of a network that includes interconnected nodes.

[0083] Now referring to Figure 8 , an illustrative cloud computing environment 750 for enabling use cases of the present invention is depicted. As shown, the cloud computing environment 750 includes one or more cloud computing nodes 710 with which local computing devices used by cloud consumers can communicate, such as personal digital assistants (PDAs) or cellular phones 754A, desktop computers 754B, laptop computers 754C, and / or in - vehicle computer systems 754N. The nodes 710 can communicate with each other. They can be physically or virtually grouped (not shown) in one or more networks, such as a private cloud, community cloud, public cloud, or hybrid cloud or a combination thereof as described above. This allows the cloud computing environment 750 to provide infrastructure, platform, and / or software as a service, and the cloud consumer does not need to maintain resources on a local computing device. It should be understood that Figure 8 the types of computing devices 754A - N shown in

[0084] Figure 9 is a schematic diagram of an exemplary abstract model layer according to an embodiment of the present invention. It should be understood in advance that Figure 9The components, layers, and functions shown are for illustrative purposes only, and embodiments of the present invention are not limited thereto. As depicted, the following layers and corresponding functions are provided:

[0085] The hardware and software layer 860 includes hardware and software components. Examples of hardware components include: host 861; server 862 based on RISC (Reduced Instruction Set Computer) architecture; server 863; blade server 864; storage device 865; and network and network components 866. In some embodiments, the software components include network application server software 867 and database software 868.

[0086] The virtualization layer 870 provides an abstraction layer from which the following examples of virtual entities can be provided: virtual server 871; virtual memory 872; virtual network 873, including virtual private networks; virtual applications and operating systems 874; and virtual clients 875.

[0087] In one example, the management layer 880 can provide the functions described below. Resource provisioning 881 provides dynamic procurement of computing resources and other resources used to perform tasks within a cloud computing environment. Metering and pricing 882 provides cost tracking when resources are utilized within a cloud computing environment, as well as accounting or invoicing for the consumption of these resources. In one example, these resources can include application software licenses. Security provides authentication for cloud consumers and tasks, as well as protection for data and other resources. User portal 883 provides access to the cloud computing environment for consumers and system administrators. Service level management 884 provides cloud computing resource allocation and management such that the required service levels are met. Service level agreement (SLA) planning and fulfillment 885 provides pre-arrangement and procurement of cloud computing resources, where future requirements are anticipated according to the SLA.

[0088] The workload layer 890 provides examples of functions that can utilize a cloud computing environment. Examples of workloads and functions that can be provided from this layer include: mapping and navigation 891; software development and lifecycle management 892; virtual classroom education delivery 893; data analysis processing 894; transaction processing 895; and Q-learning methods using language models 896.

[0089] As used herein, the terms "data," "content," "information," and similar terms may be used interchangeably to refer to data that can be captured, sent, received, displayed, and / or stored according to various example embodiments. Thus, the use of any such terms should not be taken to limit the spirit and scope of the present disclosure. Further, in instances where a computing device is described herein as receiving data from another computing device, the data may be received directly from the other computing device or may be received indirectly via one or more intermediate computing devices such as one or more servers, repeaters, routers, network access points, base stations, and the like.

[0090] To provide for interaction with a user, embodiments of the subject matter described in this specification may be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user may provide input to the computer. Other kinds of devices may also be used to provide for interaction with the user; for example, feedback provided to the user may be any form of sensory feedback such as visual feedback, auditory feedback, or tactile feedback; and input from the user may be received in any form including acoustic, speech, or tactile input.

[0091] The present invention may be a system, a method, and / or a computer program product. The computer program product may include a computer-readable storage medium (or media) having computer-readable program instructions thereon for causing a processor to perform aspects of the present invention.

[0092] A computer-readable storage medium may be a tangible device that is capable of retaining and storing instructions for use by an instruction execution device. A computer-readable storage medium may be, by way of example and not limitation, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of the computer-readable storage medium includes the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-

[0093] ROM), a digital versatile disc (DVD), a memory stick, a floppy disk, a mechanical encoding device such as a punched card or raised structures in a groove having instructions recorded thereon, and any suitable combination of the foregoing. As used herein, a computer-readable storage medium should not be construed as being a transitory signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through an optical fiber cable), or an electrical signal transmitted through a wire.

[0094] The computer-readable program instructions described herein can be downloaded to a respective computing / processing device from a computer-readable storage medium or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network may include copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in a computer-readable storage medium within the respective computing / processing device.

[0095] The computer-readable program instructions for carrying out operations of the present invention may be assembly instructions, instruction set architecture (ISA) instructions, machine-related instructions, microcode, firmware instructions, state-setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages such as Smalltalk, C++, etc., and conventional procedural programming languages such as the "C" programming language or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer, or entirely on the remote computer or server. In the latter case, the remote computer may be connected to the user's computer through any type of network connection, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, to carry out aspects of the present invention, an electronic circuit, including, for example, a programmable logic circuit, a field-programmable gate array (FPGA), or a programmable logic array (PLA), may execute the computer-readable program instructions by utilizing state information of the computer-readable program instructions to personalize the electronic circuit.

[0096] Aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0097] These computer-readable program instructions can be provided to at least one processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions executed via the processor of the computer or other programmable data processing apparatus create means for implementing the functions / actions specified in one or more boxes or modules of the flowchart and / or block diagram. These computer-readable program instructions may also be stored in a computer-readable storage medium that can direct a computer, a programmable data processing apparatus, and / or other devices to function in a particular manner, such that the computer-readable storage medium in which the instructions are stored comprises an article of manufacture including instructions that implement aspects of the functions / actions specified in one or more boxes or modules of the flowchart and / or block diagram.

[0098] The computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other devices to cause a series of operational blocks / steps to be performed on the computer, other programmable apparatus, or other devices to produce a computer-implemented process, such that the instructions executed on the computer, other programmable apparatus, or other devices implement the functions / actions specified in one or more blocks or modules of the flowchart and / or block diagram.

[0099] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each box in the flowchart or block diagram may represent a module, segment, or portion of instructions, which comprises one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions noted in the boxes may not occur in the order noted in the figures. For example, two boxes shown in succession may in fact be executed substantially concurrently, or the boxes may sometimes be executed in the reverse order, depending on the functionality involved. It will also be noted that each box of the block diagrams and / or flowchart illustrations, and combinations of boxes in the block diagrams and / or flowchart illustrations, can be implemented by special-purpose hardware-based systems that perform the specified functions or actions, or combinations of special-purpose hardware and computer instructions.

[0100] References in the specification to "one embodiment" or "an embodiment" and other variations thereof mean that a particular feature, structure, characteristic, etc. described in connection with the embodiment is included in at least one embodiment of the present principle. Thus, the appearances of the phrases "in one embodiment" or "in an embodiment" and other variations thereof in various places throughout the specification are not necessarily all referring to the same embodiment.

[0101] It should be understood that in cases such as "A / B", "A and / or B", and "at least one of A and B", the use of any of the following, i.e., " / ", "and / or", and "at least one of", is intended to cover the selection of only the first-listed option (A), or only the second-listed option (B), or the selection of both options (A and B). As a further example, in the cases of "A, B, and / or C" and "at least one of A, B, and C", such wording is intended to include the selection of only the first-listed option (A), or only the second-listed option (B), or only the third-listed option (C), or the selection of only the first and second-listed options (A and B), or only the first and third-listed options (A and C), or only the second and third-listed options (B and C), or the selection of all three options (A and B and C). This can be extended to any number of items listed, which will be apparent to those of ordinary skill in the art and related fields.

[0102] Preferred embodiments of systems and methods for unsupervised text summarization using the Q-language approach that utilizes a language model have been described (which are intended to be illustrative and not restrictive), noting that those skilled in the art can make modifications and variations based on the above teachings. Accordingly, it should be understood that changes can be made in the specific embodiments described, which are within the scope of the invention as defined by the appended claims. Thus, aspects of the invention have been described with the details and characteristics required by patent law, and what is claimed and desired to be protected by letters patent is set forth in the appended claims.

Claims

1. A computer-implemented method for performing Q-learning using a language model for unsupervised text summarization on a processor, the method comprising: Mapping each word of a sentence to a vector using word embeddings via a deep learning natural language processing (NLP) model; Assigning each of the words to an action and an operation state; Determining a state for each of the words whose operation state represents "not operated" by calculating a local encoding and a global encoding and concatenating the local encoding and the global encoding, wherein the local encoding is calculated based on the vector, action, and operation state of the word, and the global encoding is calculated in a self-attention manner based on each of the local encodings of the word; And Determining a Q-value for each of the words based on the state according to each of three actions via an editing agent, wherein the three actions are "replace", "keep", and "remove".

2. The method according to claim 1, further comprising updating the action and the operation state of the word having the highest Q-value to the action having the highest Q-value and updating the operation state to "operated".

3. The method according to claim 2, further comprising repeating the step of determining the state and the step of updating the operation state until all the operation states of the words represent "operated".

4. The method according to claim 3, further comprising converting the sentence into a masked sentence via a language model converter, wherein the words having "keep" are retained, and the words having "replace" and "remove" are changed to empty tokens.

5. The method according to claim 4, further comprising compressing the sentence by converting each empty token having "replace" in the masked sentence into a predicted word using a masked language model for predicting masked portions in a given sentence.

6. The method according to claim 5, further comprising reconstructing the sentence from the compressed sentence by converting each of the empty tokens in the masked sentence into a predicted word using the masked language model.

7. A non-transitory computer-readable storage medium comprising a computer-readable program executed on a processor in a data processing system, the computer-readable program for performing Q-learning using a language model for unsupervised text summarization, wherein the computer-readable program causes a computer to perform the following steps when executed on the processor: Mapping each word of a sentence to a vector using word embeddings via a deep learning natural language processing (NLP) model; Assigning each of the words to an action and an operation state; Determining a state for each of the words whose operation state represents "not operated" by calculating a local encoding and a global encoding and concatenating the local encoding and the global encoding, wherein the local encoding is calculated based on the vector, action, and operation state of the word, and the global encoding is calculated in a self-attention manner based on each of the local encodings of the word; and Via an editing agent, based on the state, determine the Q-value of each of the words according to each of three actions, where the three actions are "replace", "keep", and "remove".

8. The non-transitory computer-readable storage medium according to claim 7, wherein the action and the operation state of the word having the highest Q-value are updated to the action having the highest Q-value and the operation state is updated to "operated".

9. The non-transitory computer-readable storage medium according to claim 8, wherein the steps of determining the state and updating the operation state are repeated until all operation states of the words indicate "operated".

10. The non-transitory computer-readable storage medium according to claim 9, wherein the sentence is converted into a masked sentence via a language model converter, where words having "keep" are retained, and words having "replace" and "remove" are changed to empty tokens.

11. The non-transitory computer-readable storage medium according to claim 10, wherein the sentence is compressed by converting each of the empty tokens having "replace" in the masked sentence into a predicted word by using a masked language model for predicting masked parts in a given sentence.

12. The non-transitory computer-readable storage medium according to claim 11, wherein the sentence is reconstructed from the compressed sentence by converting each of the empty tokens in the masked sentence into a predicted word by adopting the masked language model.

13. A system for performing Q-learning using a language model for unsupervised text summarization, the system comprising: a memory; and one or more processors in communication with the memory, the one or more processors being configured to: map each word of a sentence into a vector by using a deep learning natural language processing (NLP) model with word embeddings; assign each of the words to an action and an operation state; determine a state for each of the words whose operation state represents "unoperated" by calculating local encoding and global encoding and concatenating the local encoding and the global encoding, where the local encoding is calculated based on the vector, action, and operation state of the word, and the global encoding is calculated in a self-attention manner based on each of the local encodings of the word; and via an editing agent, based on the state, determine the Q-value of each of the words according to each of three actions, where the three actions are "replace", "keep", and "remove".

14. The system according to claim 13, wherein the action and the operation state of the word having the highest Q-value are updated to the action having the highest Q-value, and the operation state is updated to "operated".

15. The system according to claim 14, wherein the steps of determining the state and updating the operation state are repeated until all operation states of the words indicate "operated".

16. The system according to claim 15, wherein the sentence is converted into a masked sentence by a language model converter, where words with "keep" are retained, while words with "replace" and "remove" are changed to empty tokens.

17. The system according to claim 16, wherein the sentence is compressed by converting each empty token with "replace" in the masked sentence into a predicted word by using a masked language model for predicting masked parts in a given sentence.

18. The system according to claim 17, wherein the sentence is reconstructed from the compressed sentence by converting each empty token in the masked sentence into a predicted word by employing the masked language model.

19. A computer-implemented method for performing Q-learning using a language model for unsupervised text summarization, executed on a processor, the method comprising: mapping each word of a sentence into a vector by using word embeddings through a deep learning natural language processing model; assigning each of the words to an action and an operation state; determining a state for each where the operation state represents "not operated" by calculating local encoding and global encoding, where the local encoding and the global encoding are concatenated; and determining the Q-value of each word via an editing agent based on the edit operations "replace", "keep", and "remove".

20. The method according to claim 19, wherein the local encoding is calculated based on the vector, action, and operation state of the word, and the global encoding is calculated in a self-attention manner based on each of the local encodings of the word.

21. The method according to claim 20, wherein the sentence is converted into a masked sentence by a language model converter, where words with "keep" are retained, while words with "replace" and "remove" are changed to empty tokens; and wherein the sentence is compressed by converting each empty token with "replace" in the masked sentence into a predicted word by using a masked language model for predicting masked parts in a given sentence.

22. A system for performing Q-learning using a language model for unsupervised text summarization, the system comprising: a memory; and one or more processors in communication with the memory, the one or more processors being configured to: map each word of a sentence into a vector by using word embeddings through a deep learning natural language processing model; assign each of the words to an action and an operation state; determine a state for each where the operation state represents "not operated" by calculating local encoding and global encoding, where the local encoding and the global encoding are concatenated; and determine the Q-value of each word via an editing agent based on the edit operations "replace", "keep", and "remove".

Images