Contextual completion for messages in a conversation

Abstract

The present disclosure provides a method and apparatus for performing contextual completion on a message in a conversation. A message may be obtained. Contextual omissions in the message may be detected. A determination may be made as to whether the message remains in the current domain of the conversation. In response to determining that the message remains in the current domain, supplementary text may be selected in the current domain to restore the contextual omissions. A completion message may be generated based on the message and the supplementary text.

Description

Technical Field

[0001] The present invention relates to information processing, and more particularly, to a method and apparatus for performing context completion on messages in a conversation. Background Art

[0002] Artificial intelligence (AI) chatbots are becoming increasingly popular and are being used in a growing number of scenarios. Chatbots are designed to simulate human speech and can engage in conversations with users via text, voice, and images. Typically, chatbots can recognize the language content within user-entered messages or apply natural language processing to them, and then provide responses to the user. Summary of the Invention

[0003] This summary is provided to introduce a set of concepts that will be further described in the following detailed description. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.

[0004] Embodiments of the present disclosure provide a method and apparatus for contextual completion of a message in a conversation. A message may be obtained. Contextual omissions in the message may be detected. A determination may be made as to whether the message remains in the current domain of the conversation. In response to determining that the message remains in the current domain, supplementary text may be selected in the current domain to restore the contextual omissions. A completion message may be generated based on the message and the supplementary text.

[0005] It should be noted that one or more of the above aspects include features described in detail below and particularly pointed out in the claims. The following description and drawings set forth in detail certain illustrative features of the one or more aspects. These features are merely indicative of the various ways in which the principles of the various aspects may be implemented, and the present disclosure is intended to include all such aspects and their equivalents. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] The disclosed aspects will be described below with reference to the accompanying drawings, which are provided to illustrate rather than limit the disclosed aspects.

[0007] Figure 1 An example session is shown with context elision.

[0008] Figure 2 An exemplary process for providing a response by contextually completing a message in a conversation is shown, according to an embodiment.

[0009] Figure 3 An exemplary process for making a domain hold determination according to an embodiment is shown.

[0010] Figure 4 An exemplary process for obtaining a domain-preserving classifier according to an embodiment is shown.

[0011] Figure 5 An exemplary process for performing convolutional neural network (CNN)-based ranking of candidate texts according to an embodiment is shown.

[0012] Figure 6 An exemplary process for performing regression-based ranking of candidate texts according to an embodiment is shown.

[0013] Figure 7 An exemplary session is shown in which contextual completion according to an embodiment is applied.

[0014] Figure 8 An exemplary session is shown in which contextual completion according to an embodiment is applied.

[0015] Figure 9 A flow chart of an exemplary method for contextual completion of messages in a conversation according to an embodiment is shown.

[0016] Figure 10 An exemplary apparatus for performing contextual completion on a message in a conversation according to an embodiment is shown.

[0017] Figure 11 An exemplary apparatus for performing contextual completion on a message in a conversation according to an embodiment is shown. DETAILED DESCRIPTION

[0018] The present disclosure will now be discussed with reference to various exemplary embodiments. It should be understood that the discussion of these embodiments is only for enabling those skilled in the art to better understand and thereby implement the embodiments of the present disclosure, and does not teach any limitation on the scope of the present disclosure.

[0019] Generally, a chatbot can conduct automated conversations within a conversation with a user. In this document, a "conversation" may refer to a continuous conversation between two chat participants and may include messages and responses within the conversation. A "message" may refer to any information input by a user, such as a query from the user, a user's response to a chatbot's question, or a user's opinion. The terms "message" and "query" may also be used interchangeably. A "response" may refer to any information provided by a chatbot, such as an answer to a user's question, a comment from the chatbot, or the like.

[0020] In a conversation between a user and a chatbot, the user often inputs semantically incomplete messages, such as messages with context ellipsis. Such messages may omit some of the contextual content that has been involved in the previous chat. For example, if the user has previously discussed with the chatbot about arrangements for a trip to Beijing, the user may then input the message "What fun places are there?". This message omits the previously discussed context "Beijing", and what it actually wants to express is "What fun places are there in Beijing?". When a message with context ellipsis is obtained, the chatbot needs to identify what contextual content has been omitted and perform context completion on the message accordingly, so that a response can be provided for the message with the omitted contextual content completed.

[0021] Several technologies have been proposed specifically for context completion of sentences with context omissions. These technologies aim to select supplementary text from a predetermined set of candidate texts in order to convert sentences known to have context omissions into completed sentences with context completion. The candidate text set is predetermined and fixed, and these existing technologies do not provide a mechanism for updating the candidate text set based on specific application scenarios. Furthermore, these existing technologies are generally applicable only to open domains or specific domains and cannot efficiently and quickly achieve migration between different domains. For example, the models or network architectures underlying these existing technologies require the use of labeled training data obtained from a corpus of the open domain or a specific domain. Therefore, when these technologies are to be applied to a new domain, new labeled training data obtained from the corpus of the new domain is required to rebuild or train these models or network architectures. The acquisition of new labeled training data and the rebuilding or training process are both time-consuming and costly.

[0022] Embodiments of the present disclosure propose an end-to-end architecture for performing contextual completion on messages in a conversation. The conversation may occur, for example, in an automated chat between a user and a chatbot. In this end-to-end architecture, a series of processes for performing contextual completion on messages may be automatically executed as the conversation progresses. For example, these processes may include, but are not limited to, at least one of the following: determining whether the current message has contextual omissions, determining whether the current message remains in the current domain, continuously updating candidate texts to be selected for restoring the contextual omissions, selecting supplementary texts for performing contextual completion on the current message from the candidate texts, generating a completion message, and the like. In addition, in the event that the current message does not remain in the current domain, a domain corresponding to the current message may also be determined, and the current message may be contextually completed in the corresponding domain. Embodiments of the present disclosure may provide simple, efficient, and highly accurate contextual completion for messages with contextual omissions in the context of automated chats where the conversation content is constantly evolving.

[0023] In this article, context omission may include topic omission and / or predicate omission. A topic may refer to an entity or subject involved in a conversation, which may include one or more nouns or noun phrases. A predicate may refer to a statement with a verb as the subject in a conversation, or a statement about an attribute of an entity in a conversation, which may include one or more verbs or verb phrases or other phrases used to describe the attribute. Taking "What are the fun places in Beijing?" as an example, the topic may be "Beijing" and the predicate may be "What are the fun places." Taking "What is the capital of China?" as an example, the topic may be "China" and the predicate may be "Where is the capital."

[0024] In one aspect, embodiments of the present disclosure can maintain domain-specific information streams during a conversation, such as a domain topic stream and a domain predicate stream. Different information streams corresponding to different domains can be maintained. Thus, when contextual completion is required for a message in a certain domain, supplementary text can be selected from the candidate texts in the information stream maintained for that domain to complete the message contextually. Furthermore, by maintaining information streams, chat context from multiple rounds of conversation can be recorded, thereby effectively enabling contextual completion of messages in multiple rounds of conversation.

[0025] In one aspect, embodiments of the present disclosure can apply the same model or network architecture for ranking candidate texts to different domains. Since the ranking model basically does not require annotated data of the domain, the ranking model can be easily migrated between different domains. More accurate and lightweight features such as pointwise mutual information (PMI) are used in the ranking of candidate texts. Compared with the labeled training data used in the existing technology, a large amount of PMI data in different domains can be more easily obtained. When implementing or applying the ranking model, ranking can be performed in different domains by directly using the PMI data in different domains.

[0026] In one aspect, the contextual completion performed on messages by embodiments of the present disclosure is explicit contextual completion. For example, informative words or expressions in a conversation are explicitly maintained in the information stream, and the completion message generated based on the supplementary text selected from the information stream is also readable. Existing response retrieval or generation techniques can be directly used to provide responses to the completion message without having to change the existing response retrieval or generation strategy.

[0027] Figure 1 An example session is shown with context elision.

[0028] Conversation 110 illustrates topic omission in a previous message. The user first enters the message "What fun places are there in Beijing?" The chatbot responds with "The Great Wall and the Forbidden City are in Beijing. Let's go!" The user then enters the message "What delicious food is there?" This message omits the topic, for example, omitting the topic "Beijing" from the previous message "What fun places are there in Beijing?" Therefore, the message "What delicious food is there?" actually expresses "What delicious food is there in Beijing?"

[0029] In conversation 120, the topic of the previous response is omitted. The user first enters the message "What delicious food is there in Beijing?" The chatbot responds with "Roast duck." The user then enters the message "Is it delicious?", which omits the topic "roast duck" from the previous response. Therefore, the message "Is it delicious?" actually means "Is the roast duck delicious?"

[0030] Conversation 130 illustrates predicate omission for the previous message. The user first enters the message "What is the capital of China?" The chatbot may respond with "Beijing." The user then enters the message "What about Thailand?", which exhibits predicate omission. For example, the predicate "where is the capital" from the previous message "What is the capital of China?" is omitted. Therefore, the message "What about Thailand?" actually expresses "What is the capital of Thailand?"

[0031] Conversation 140 illustrates topic omission across multiple conversations. In the first conversation, the user enters the message "What fun places are there in Beijing?", to which the chatbot responds, "The Great Wall and the Forbidden City are in Beijing." In the second conversation, the user enters the message "The Great Wall is world-famous," to which the chatbot responds, "The Great Wall is magnificent." In the third conversation, the user enters the message "What delicious food is there?", which exhibits topic omission. For example, the topic "Beijing" is omitted from the first conversation. Therefore, the message "What delicious food is there?" actually expresses "What delicious food is there in Beijing?"

[0032] Conversation 150 illustrates both topic and predicate omission for the previous message. The user first enters the message "What fun places are there in Beijing?" The chatbot responds with "The Great Wall and the Forbidden City are in Beijing." The user then enters the message "What else?", which exhibits both topic and predicate omission. For example, the message "Beijing" and the predicate "What fun places are there" from the previous message "What fun places are there in Beijing?" are omitted. Therefore, the message "What else" actually expresses "What other fun places are there in Beijing?"

[0033] It should be understood that Figure 1 Only some conversation examples with context omission are shown. In actual application scenarios, there may be many other context omission situations. The embodiments of the present disclosure can be implemented in various context omission situations.

[0034] Figure 2 An exemplary process 200 for providing a response by contextually completing a message in a conversation is shown, according to an embodiment.

[0035] At 202, a message from a user may be obtained in a conversation between the user and the chatbot. For example, the message may be input by the user in a chat conversation with the chatbot.

[0036] According to an embodiment of the present invention, the content in the message can be saved in an information stream maintained in real time, such as a domain topic stream and a domain predicate stream, so as to record context information in the conversation.

[0037] In one embodiment, at 204, topics and / or predicates may be extracted from the obtained message. If context elision exists in the message, the message obtained at 202 may include only topics but not predicates, or only predicates but not topics, or neither topics nor predicates. Accordingly, at 204, only topics, only predicates, or neither topics nor predicates may be extracted from the message. If context elision does not exist in the message, both topics and predicates may be extracted from the message at 204.

[0038] In one embodiment, a pre-established knowledge graph may be used to extract topics and / or predicates from a message. The knowledge graph may be domain-specific, for example, different knowledge graphs may be established for different domains. The knowledge graph of a certain domain may include topics and predicates pre-mined from a large amount of network data in that domain. For example, the knowledge graph of the tourism domain may include topics such as "Beijing", "The Great Wall", "The Forbidden City in Beijing", "Air Tickets", "Hotels", etc., and may include predicates such as "Food", "Fun Places", "Capital", "How to Go", "What's the Climate Like", etc. By comparing the content in the message with the knowledge graph, topics and / or predicates that have been recorded in the knowledge graph may be identified from the message.

[0039] If topics are extracted from the message at 204, the extracted topics may be stored in the domain topic stream of the current domain. The domain topic stream may include a list of topics extracted from the messages and responses of the conversation, such as topic 1, topic 2, topic 3, and the like. Optionally, process 200 may also include determining conversation-level information associated with the topics extracted at 204. Accordingly, the domain topic stream may also include conversation-level information determined for each topic. The conversation-level information for a topic may include statistical information determined for the topic in the conversation, such as the frequency or number of times the topic has appeared in the conversation, whether the topic appeared in the previous message, whether the topic appeared in the previous response, the number of turns that have passed since the topic last appeared, whether the topic is from a message, whether the topic is from a response, and the like. The topic and conversation-level information included in the domain topic stream may be continuously updated as the conversation progresses.

[0040] If predicates are extracted from the message at 204, the extracted predicates may be stored in the domain predicate stream for the current domain. The domain predicate stream may include a list of predicates extracted from the messages and responses of the conversation, e.g., predicate 1, predicate 2, predicate 3, etc. Optionally, process 200 may also include determining conversation-level information associated with the predicates extracted at 204. Accordingly, the domain predicate stream may also include conversation-level information determined for each predicate. The conversation-level information for a predicate may include statistical information determined for the predicate in the conversation, e.g., the frequency or number of times the predicate has appeared in the conversation, whether the predicate appeared in the last message, whether the predicate appeared in the last response, the number of turns since the last occurrence of the predicate, whether the predicate originated from a message, whether the predicate originated from a response, etc. The predicates and conversation-level information included in the domain predicate stream may be continuously updated as the conversation progresses.

[0041] It should be understood that although process 200 only illustrates updating the domain topic stream and domain predicate stream using topics and / or predicates extracted from messages and conversation-level information, embodiments of the present disclosure may also update the domain topic stream and domain predicate stream in real time using topics and / or predicates extracted from responses provided by the chatbot and conversation-level information.

[0042] At 206, it may be determined whether context omitting exists in the obtained message. In one embodiment, the determination of whether context omitting exists in the message may be based on the results of the extraction at 204. For example, if both a topic and a predicate are extracted from the message, it may be determined that context omitting does not exist in the message. If no topic is extracted from the message, it may be determined that topic omitting exists in the message. If no predicate is extracted from the message, it may be determined that predicate omitting exists in the message. If neither a topic nor a predicate is extracted from the message, it may be determined that both topic omitting and predicate omitting exist in the message.

[0043] If it is determined at step 206 that there is no context omission in the message, a response may be provided based on existing response retrieval or generation techniques at step 208. For example, in the context of automated chat, a variety of response providing techniques based on retrieval or generation have been proposed to respond to messages from users.

[0044] If it is determined at 206 that the message contains contextual omissions, process 200 proceeds to 210. At 210, it may be further determined whether the message contains a predetermined dialog act. A dialog act may refer to a label assigned to a dialog statement based on the category or intent of the statement's meaning. Predefined dialog acts may include certain pre-specified dialog acts, which may correspond to, for example, abbreviated language expressions that do not include a clear topic or predicate. For example, a message containing predetermined dialog acts may contain topic omissions and predicate omissions. An example of a pre-specified dialog act may be inquiring about the details of a previous response. For example, in response to the chatbot's previous response, "Beijing is worth a visit," the user may enter the message "Why?", which is intended to inquire about the specific reasons why "Beijing is worth a visit." Another example of a pre-specified dialog act may be repeating the previous message. For example, a user may enter the first message, "What are some fun places in Beijing?" After the chatbot responds, "The Great Wall and the Forbidden City," the user may continue to enter a second message, "Anything else?", which essentially repeats the question in the first message to elicit a further answer. Another example of a pre-designated conversation behavior may be an intention to end the conversation. For example, if the user enters the message "I understand," this message may indicate that the user does not want to continue the conversation or has a low desire to continue the conversation. The above examples of pre-designated conversation behaviors are merely illustrative. Depending on specific application requirements, it may also be determined at step 210 whether the message has any other type of conversation behavior.

[0045] In one embodiment, the determination at 210 may utilize a predetermined dialogue act classifier, such as a classifier based on K-means or similar techniques. The predetermined dialogue act classifier may be trained to determine whether a message contains any predetermined dialogue acts. For example, training data including dialogue sentences labeled with predetermined dialogue acts may be prepared in advance, and the predetermined dialogue act classifier may be trained using such training data. When a message is provided to the trained predetermined dialogue act classifier, it may determine whether the message contains any predetermined dialogue acts and output the predetermined dialogue acts contained in the message.

[0046] If it is determined at 210 that the message has a predetermined dialogue behavior, a response may be provided for the message based on a corresponding policy at 212. Different response policies may be pre-specified for different predetermined dialogue behaviors. For example, if the dialogue behavior of the message is determined to be inquiring for details of a previous response, a response including a detailed description of the previous response may be retrieved or generated. For example, if the dialogue behavior of the message is determined to be a repetition of a previous message, a response may be retrieved or generated again based on the previous message. If the dialogue behavior of the message is determined to be an attempt to end the conversation, the conversation may be paused or a response may be provided to shift the conversation to another topic.

[0047] If it is determined at 210 that the message is not a message with a predetermined conversation behavior, process 200 proceeds to 214 .

[0048] At 214, it may be determined whether the message remains in the current domain of the conversation. The current domain may refer to the domain in which the conversation immediately preceding the message was located.

[0049] If a predicate is omitted in the message and a topic can be detected, it can be determined whether the message remains in the current domain based on the topic included in the message. In one embodiment, if the topic included in the message is located in the domain topic stream of the current domain or in the knowledge graph of the current domain, it can be determined that the message remains in the current domain. If the topic included in the message is neither in the domain topic stream of the current domain nor in the knowledge graph of the current domain, it can be determined that the message does not remain in the current domain.

[0050] In the case where topic omission exists in the message and a predicate can be detected, a domain-preserving classifier can be used to determine whether the message remains in the current domain. In one embodiment, the domain-preserving classifier can be based on a recurrent neural network (RNN) model. Optionally, a classifier based on a Bidirectional Encoder Representation from Transformers (BERT) model can be first established to determine whether a sentence remains in a specific domain, and then a domain-preserving classifier based on the RNN model can be obtained through knowledge distillation. This will be combined with Figure 3 and Figure 4 The working process and acquisition process of the domain-preserving classifier are discussed in detail.

[0051] If it is determined at 214 that the message remains in the current domain, process 200 proceeds to 216. At 216, supplementary text for restoring the topic omission or predicate omission may be selected in the current domain. In this document, supplementary text may refer to text corresponding to the contextual content omitted in the message, such as a topic or predicate. Accordingly, the supplementary text may be used to combine with the message to form a semantically complete completed message. In the case where the message remains in the current domain, the supplementary text may come from a topic in the domain topic stream or a predicate in the domain predicate stream of the current domain. For example, if there is a topic omission in the message, the supplementary text may be selected from the domain topic stream. If there is a predicate omission in the message, the supplementary text may be selected from the domain predicate stream. Topics in the domain topic stream and predicates in the domain predicate stream may be considered as candidate texts, and supplementary text may be selected from these candidate texts.

[0052] In one embodiment, supplementary text may be selected from multiple candidate texts in a domain topic stream and / or a domain predicate stream through convolutional neural network (CNN)-based ranking. CNN-based ranking may rank multiple candidate texts so that the highest-ranked candidate text is selected as the supplementary text. CNN-based ranking may utilize features such as PMI and text similarity between the candidate text and the message.

[0053] In one embodiment, a supplementary text may be selected from multiple candidate texts in a domain topic stream and / or a domain predicate stream through regression-based ranking. The regression-based ranking may rank the multiple candidate texts so that the highest-ranked candidate text is selected as the supplementary text. The features used in the regression-based ranking may include, for example, at least one item of session-level information from the domain topic stream and the domain predicate stream.

[0054] In one embodiment, the supplementary text can be selected from multiple candidate texts in the domain topic stream and / or domain predicate stream by using both CNN-based ranking and regression-based ranking. In this case, the ranking results of the two rankings can be weighted combined to determine the highest-ranked candidate text and use it as the supplementary text.

[0055] Will be combined later Figure 5 and Figure 6 Specifically discuss CNN-based sorting and regression-based sorting.

[0056] After determining the supplementary text, embodiments of the present disclosure may further generate a completed message based on the message and the supplementary text. In this context, a completed message may refer to a semantically complete message formed by completing the context of a message with omitted context. The completed message may be readable text.

[0057] In one embodiment, at 218, at least one candidate completion message can be generated by placing supplementary text at different locations within the message. For example, the supplementary text can be placed before, after, or within the message. For example, taking the message "Is it delicious?" and the supplementary text "roast duck," the supplementary text can be placed before the message to generate the candidate completion message "Is the roast duck delicious?", while the supplementary text can be placed after the message to generate the candidate completion message "Is the roast duck delicious?". For example, taking the message "Let's go visit that place" and the supplementary text "The Forbidden City," the supplementary text can be placed within the message to perform coreference resolution, generating the candidate completion message "Let's go visit the Forbidden City." Then, at 220, a completion message can be selected from the at least one candidate completion message using, for example, a language model. A language model can be used to measure the probability that a text is a natural language sentence and can be based on neural networks or statistics. For example, "Is the roast duck delicious?" is closer to natural language than "Is the roast duck delicious?"; therefore, the language model may assign a higher score to the former and a lower score to the latter. The final completion message can be selected from the at least one candidate completion message based on the language model score.

[0058] After the completion message is determined, a response may be provided for the completion message at 222. For example, the response to the completion message may be provided based on existing response retrieval or generation techniques.

[0059] If it is determined at 214 that the message does not remain in the current domain, for example, the message has been transferred from the current domain to another domain, process 200 proceeds to 224. At 224, the domain corresponding to the message may be determined. In one embodiment, the domain corresponding to the message may be determined by comparing the message with topics or predicates in knowledge graphs of different domains.

[0060] As previously described, embodiments of the present disclosure can maintain different information flows for different domains involved in a conversation. Therefore, after determining the domain corresponding to the message, supplementary text can be selected in the corresponding domain at 226. For example, the supplementary text can be selected from candidate texts in the domain topic flow or domain predicate flow of the corresponding domain.

[0061] At 228, at least one candidate completion message may be generated in a manner similar to the process at 218. Then, at 230, a completion message may be selected from the at least one candidate completion message in a manner similar to the process at 220. After the completion message is determined, a response may be provided for the completion message at 222. At this point, the corresponding domain becomes the current domain for executing process 200 for subsequent messages.

[0062] It should be understood that the various steps and execution order of process 200 discussed above are exemplary, and process 200 may be modified in any manner according to specific application scenarios and design requirements.

[0063] In one embodiment, the process of determining whether the message remains in the current domain at 214 can be replaced by determining the domain corresponding to the message, similar to the process at 224. After the corresponding domain is determined, the completed message can be generated through processes similar to those at 226, 228, 230, etc.

[0064] In one embodiment, if it is determined at 214 that the message does not remain in the current domain, the topics and / or predicates and conversation-level information extracted at 204 may be transferred from the domain topic stream and / or domain predicate stream of the current domain to the domain topic stream and / or domain predicate stream of the domain corresponding to the message.

[0065] In one embodiment, when topics and / or predicates and conversation-level information are extracted from the message at 204, the extracted content may not be immediately added to the domain topic stream and / or domain predicate stream of the current domain. Instead, the content extracted at 204 may be added to the domain topic stream and / or domain predicate stream of the current domain after it is determined at 214 that the message remains in the current domain.

[0066] In one embodiment, the order of determining whether the message is a predetermined dialogue behavior message at 210 may be adjusted in process 200. For example, the process at 210 may be performed before 206.

[0067] In one embodiment, instead of determining whether the message is a predetermined dialogue act message at 210, it may be determined whether the message contains both topic omission and predicate omission at 210. If so, a response is provided for the message according to a predetermined response rule. The response rule defined for messages containing both topic omission and predicate omission may be empirically predetermined.

[0068] In one embodiment, instead of generating a readable completion message through the processing at 218 and 220 , the message and the supplemental text may not be semantically combined, but instead provided directly to the response retrieval or generation strategy as a bigram.

[0069] In one embodiment, instead of generating at least one candidate completion message at 218 and then selecting a completion message at 220, a completion message may be generated directly based on a language model after the supplementary text is selected at 216. For example, a language model may be trained to generate a completion message having a natural language style based on the message and the supplementary text.

[0070] In one embodiment, process 200 can be executed cyclically during an ongoing conversation. For example, each message input by a user can trigger the execution of process 200. Furthermore, the domain topic stream and domain predicate stream of each domain are also updated in real time. For example, each time a message or response is detected, the domain topic stream and domain predicate stream of the corresponding domain can be updated using the topic and / or predicate extracted from the message or response and session-level information.

[0071] Figure 3 An exemplary process 300 for making a domain hold determination according to an embodiment is shown. Process 300 may be performed, for example, in Figure 2 At 214 it is determined whether the message remains in the current domain of the session.

[0072] In process 300, a domain-preserving classifier 310 is used to make a domain-preserving determination. The domain-preserving classifier 310 can be trained for a specific domain. The domain-preserving classifier can include an encoder 312 and a fully connected layer 314. The encoder 312 can convert the text into a vector representation in a dense vector space. The fully connected layer 314 can convert the vector representation of the text into a domain-preserving score that indicates the degree of relevance of the text to the domain to which the domain-preserving classifier 310 belongs.

[0073] According to process 300, message 302 may be provided to domain-preserving classifier 310. After being processed by encoder 312 and fully connected layer 314, domain-preserving classifier 310 outputs a domain-preserving score 304 for message 302. A domain-preserving determination result 306 may be ultimately obtained based on domain-preserving score 304. For example, a score threshold may be predetermined. If domain-preserving score 304 exceeds the threshold, domain-preserving determination result 306 may indicate that message 302 remains in the domain to which domain-preserving classifier 310 belongs. Otherwise, if domain-preserving score 304 does not exceed the threshold, domain-preserving determination result 306 may indicate that message 302 does not remain in the domain to which domain-preserving classifier 310 belongs.

[0074] Figure 4 An exemplary process 400 for obtaining a domain preserving classifier according to an embodiment is shown. It should be understood that the present disclosure is not limited to obtaining a domain preserving classifier through the process 400, but the domain preserving classifier can be obtained through any other means.

[0075] According to process 400, a BERT model 410 may be pre-trained or obtained first. BERT model 410 is a general model for generating vector representations of text in an open domain. When BERT model 410 receives a text input, such as a message, it may output a vector representation corresponding to the text.

[0076] The BERT model 410 can be fine-tuned to obtain a domain-preserving classifier 420 based on the BERT model, wherein the BERT model 410 serves as the encoder of the domain-preserving classifier 420. The fine-tuning can use training data for domain-preserving determination to adjust the parameters or weights of the BERT model to be suitable for performing the domain-preserving determination task. The positive examples in the training data can include sentences from a specific domain, and domain keywords, such as topics belonging to the domain, are removed from these sentences. For example, if a domain-preserving classifier for the tourism domain is to be obtained, initial sentences can be captured from social network data, website data, chat logs, etc. related to tourism, and positive examples can be obtained by removing domain keywords. The negative examples in the training data can be sentences from any other domain, and words can be randomly removed from these sentences or no word removal can be performed. The sentences in the negative examples can be selected as conversation sentences with a similar length to the sentences in the positive examples.

[0077] The domain-preserving classifier 420 based on the BERT model can be used to determine whether the message remains in the current domain. However, considering the complexity and high computational complexity of the BERT model itself, in order to facilitate actual deployment, the embodiments of the present disclosure can also obtain a faster lightweight domain-preserving classifier 430 from the domain-preserving classifier 420 based on the BERT model through knowledge distillation. The lightweight domain-preserving classifier 430 can adopt an RNN model as an encoder to replace the BERT model. The RNN model may include, for example, a 3-layer RNN. Knowledge distillation is a general technology that can convert a complex model into a simple model. By performing knowledge distillation on the domain-preserving classifier 420 based on the BERT model, a smaller and faster domain-preserving classifier 430 based on the RNN model can be obtained. The performance of the domain-preserving classifier 430 based on the RNN model can approach the performance of the domain-preserving classifier 420 based on the BERT model. When performing knowledge distillation, the RNN-based domain-preserving classifier 430 may be trained using the text input received by the BERT-based domain-preserving classifier 420 and the generated vector representation and domain-preserving score.

[0078] Figure 5 An exemplary process 500 for performing convolutional neural network (CNN)-based ranking of candidate texts according to an embodiment is shown. The process 500 may be performed, for example, in Figure 2 At 216, a supplementary text is selected for the message from multiple candidate texts in the domain topic stream and / or domain predicate stream. For example, through process 500, ranking scores can be generated for multiple candidate texts respectively, and then the candidate text with the highest ranking can be selected as the supplementary text.

[0079] The sorting model on which process 500 is based can be regarded as an extension and improvement of the existing CNN-based text matching method. An example of a CNN-based text matching method can be a pyramid matching (MatchPyramid) model, which is a CNN-based model for predicting text similarity. The sorting model on which process 500 is based can not only consider the text similarity between the candidate text and the message, but also consider the PMI between the candidate text and the message, the PMI between the synonymous representation of the candidate text and the message, the PMI between the candidate text and the synonymous representation of the message, etc. PMI indicates the possibility of co-occurrence of the candidate text and the message, thereby indicating the correlation between the candidate text and the message. Due to the introduction of effective information such as PMI, the sorting model can be made more efficient, more controllable and more accurate.

[0080] For the message 502, a candidate text i 504 may be extracted from the domain topic stream and / or the domain predicate stream. Then, a ranking score of the candidate text i 504 may be calculated.

[0081] In branch 510, the text similarity matrix 512 between the candidate text i and the message can be calculated first. The text similarity matrix 512 can be calculated based on the cosine distance, dot product, etc. between the vector representation of the candidate text i and the vector representation of the message. Assuming that the candidate text i includes n words and the message includes m words, the similarity value between each word in the n words of the candidate text i and each word in the m words of the message can be calculated respectively, thereby forming an n╳m-dimensional text similarity matrix 512. Branch 510 can include one or more 2-dimensional (2D) convolutional layers and a 2D pooling layer optionally connected to each convolutional layer. The 2D convolutional layer can be used to perform feature extraction on the similarity matrix 512. The CNN result output by branch 510 can be provided to the fully connected layer 550.

[0082] In branch 520, the PMI matrix 522 between the candidate text i and the message can be first calculated. Any existing PMI calculation method can be used to calculate the PMI matrix 522. Assuming that the candidate text i includes n words and the message includes m words, the PMI values ​​between each of the n words of the candidate text i and each of the m words of the message can be calculated separately, thereby forming an n╳m-dimensional PMI matrix 522. Similar to branch 510, branch 520 can also include a series of 2D convolutional layers and a 2D pooling layer optionally connected to each convolutional layer. The CNN result output by branch 520 can be provided to the fully connected layer 550.

[0083] In branch 530, a PMI matrix 532 between the synonymous representations of candidate text i and the message can be first calculated. The synonymous representation of candidate text i can refer to the text after replacing at least one word or all words in candidate text i with corresponding semantically similar words. Here, semantically similar words can refer to words with similar linguistic meanings or words with high semantic similarity in the field of natural language processing. For example, taking the candidate text i "Nanjing" and the message "What are the fun places?" as an example, since the synonymous representation of candidate text i "Nanjing" can be "Beijing", the PMI between "Beijing" and "What are the fun places?" can be calculated. By calculating the PMI matrix between the synonymous representations of candidate text i and the message, the accuracy of the assessment of the relevance between candidate text i and the message can be enhanced. For example, if candidate text i has a low co-occurrence frequency with the message but actually has a high relevance, the synonymous representation of candidate text i may have a higher PMI metric with the message. This PMI metric can be used to compensate for the lower PMI metric between candidate text i and the message, thereby more accurately reflecting the relevance between candidate text i and the message. Continuing with the previous example, the candidate text i "Nanjing" and the message "What are the fun places?" may have a high correlation, for example, a high probability of being combined into "What are the fun places in Nanjing?" However, the PMI metric between "Nanjing" and "What are the fun places?" may be low, for example, 0.5, thus failing to accurately reflect the high correlation between the two. In this case, since "Beijing," a synonym of "Nanjing," has a high PMI metric with the message "What are the fun places?", for example, 0.9, by considering this PMI metric, the correlation between the candidate text i "Nanjing" and the message "What are the fun places?" can be more accurately reflected.

[0084] Assuming that candidate text i includes n words and the message includes m words, an n╳m-dimensional PMI matrix 532 can be formed. The PMI value at each element [x, y] in the PMI matrix 532 can be calculated in various ways, where x∈[1, n] and y∈[1, m]. For example, in one approach, the PMI value at element [x, y] can be the PMI value between the most semantically similar word of the xth word of candidate text i and the yth word of the message. In one approach, the PMI value at element [x, y] can be the average of multiple PMI values ​​between multiple semantically similar words of the xth word of candidate text i and the yth word of the message. In one approach, the PMI value at element [x, y] can be the maximum value among multiple PMI values ​​between multiple semantically similar words of the xth word of candidate text i and the yth word of the message. Similar to branch 510, branch 530 can also include a series of 2D convolutional layers and a 2D pooling layer optionally connected to each convolutional layer. The CNN result output by branch 530 may be provided to a fully connected layer 550 .

[0085] In branch 540, the PMI matrix 542 between the candidate text i and the synonymous representation of the message can be first calculated. The synonymous representation of the message may refer to the text after at least one word or all words in the message are replaced with corresponding semantically similar words. By calculating the PMI matrix between the candidate text i and the synonymous representation of the message, the accuracy of the evaluation of the correlation between the candidate text i and the message can be enhanced. For example, in the case where the candidate text i has a low co-occurrence frequency with the message but actually has a high correlation, the candidate text i may have a higher PMI measure with the synonymous representation of the message, which can be used to compensate for the lower PMI measure between the candidate text i and the message, thereby more realistically reflecting the correlation between the candidate text i and the message.

[0086] Assuming that candidate text i includes n words and the message includes m words, an n╳m-dimensional PMI matrix 542 can be formed. The PMI value at each element [x, y] in the PMI matrix 542 can be calculated in various ways, where x∈[1, n] and y∈[1, m]. For example, in one approach, the PMI value at element [x, y] can be the PMI value between the xth word of candidate text i and the word with the closest semantics to the yth word of the message. In one approach, the PMI value at element [x, y] can be the average of multiple PMI values ​​between the xth word of candidate text i and multiple semantically similar words of the yth word of the message. In one approach, the PMI value at element [x, y] can be the maximum value among multiple PMI values ​​between the xth word of candidate text i and multiple semantically similar words of the yth word of the message. Similar to branch 510, branch 540 can also include a series of 2D convolutional layers and a 2D pooling layer optionally connected to each convolutional layer. The CNN result output by branch 540 may be provided to a fully connected layer 550 .

[0087] The fully connected layer 550 may transform the CNN results of the branches 510 , 520 , 530 , and 540 , and may generate a ranking score 506 for the candidate text i using, for example, a softmax function.

[0088] It should be understood that although Figure 5 The figure shows that the ranking model includes branches 510, 520, 530 and 540. The ranking model may also include only some of these branches, so that a ranking score can be generated based on one or more of the text similarity between the candidate text and the message, the PMI between the candidate text and the message, the PMI between the synonymous representation of the candidate text and the message, and the PMI between the synonymous representation of the candidate text and the message.

[0089] After the ranking scores are obtained for all candidate texts, the candidate text with the highest ranking score may be selected as the supplementary text according to the ranking scores of the candidate texts.

[0090] Figure 6 An exemplary process 600 for performing regression-based ranking of candidate texts according to an embodiment is shown. The process 600 may be performed, for example, in Figure 2 At 216, a supplementary text is selected for the message from multiple candidate texts in the domain topic stream and / or domain predicate stream. For example, through process 600, ranking scores can be generated for multiple candidate texts respectively, and then the candidate text with the highest ranking can be selected as the supplementary text.

[0091] Process 600 may employ a regression-based ranking model 610. Ranking model 610 may be based on various regression algorithms, such as gradient boosted decision trees (GBDT). Ranking model 610 may employ at least one conversation-level information in the domain topic stream and domain predicate stream as a feature, such as the frequency of candidate text appearance in the conversation, whether the candidate text appears in the previous message, whether the candidate text appears in the previous response, the number of turns since the last appearance of the candidate text in the conversation, whether the candidate text comes from a message, whether the candidate text comes from a response, etc. In addition, ranking model 610 may also employ the PMI between the candidate text and the message, the text similarity between the candidate text and the message, etc. as features. Ranking model 610 may be trained using training data labeled with at least one of the above features.

[0092] For message 602, candidate text i 604 can be extracted from the domain topic stream and / or domain predicate stream. A ranking model 610 can then calculate a ranking score 606 for candidate text i 604 using features extracted from message 602, candidate text i 604, the domain topic stream, the domain predicate stream, and the like. After ranking scores are obtained for all candidate texts, the candidate text with the highest ranking score can be selected as the supplementary text based on the ranking scores of these candidate texts.

[0093] Although the above are combined Figure 5 We discuss the selection of supplementary texts through CNN-based ranking and the combination of Figure 6 While selecting supplementary texts through regression-based ranking, embodiments of the present disclosure may also combine these two ranking methods to select supplementary texts. For example, for candidate text i, after obtaining ranking score 506 through process 500 and ranking score 606 through process 600, ranking score 506 and ranking score 606 may be combined, such as by weighted summation, to determine a final ranking score for candidate text i. After determining the final ranking scores for all candidate texts, the candidate text with the highest final ranking score may be selected as the supplementary text.

[0094] Furthermore, it should be understood that Figure 5 The sorting model on which process 500 is based and Figure 6 The training of the ranking model 610 in the embodiment of the present invention does not require domain-specific annotated data, and when implementing or applying these ranking models, PMI data in different domains can be directly used to perform ranking in different domains. Therefore, these ranking models can be easily transferred between different domains.

[0095] Figure 7An exemplary conversation 700 in which contextual completion according to an embodiment is applied is shown. Conversation 700 relates to a specific domain, such as tourism. In conversation 700, a chatbot applies a method for performing contextual completion on messages in the conversation according to an embodiment of the present disclosure.

[0096] At the beginning of session 700, a message 702 "What fun places are there in Beijing?" is received from the user. The topic "Beijing" extracted from message 702 can be added to the domain topic stream of the current domain, for example, an entry is created for the topic "Beijing", and the predicate "What fun places are there" extracted from message 702 can be added to the domain predicate stream of the current domain, for example, an entry is created for the predicate "What fun places are there". In addition, session-level information can be determined and added to the corresponding entries in the domain topic stream and the domain predicate stream. For example, for the topic "Beijing", the session-level information "from message" is determined, indicating that the topic appears in the message for the first time, and the "frequency 1" is determined, indicating that the topic has appeared once in the session. For example, for the predicate "What fun places are there", the session-level information "from message" is determined, indicating that the predicate appears in the message for the first time, and the "frequency 1" is determined, indicating that the predicate has appeared once in the session.

[0097] For message 702, it can be determined that there is no context omission. Therefore, a response 704 "The Great Wall and the Forbidden City are in Beijing. Let's go!" can be provided through existing response retrieval or generation methods. Response 704 will trigger updates to the domain topic stream and domain predicate stream (not shown).

[0098] When message 706 "What delicious food?" is received, the domain topic flow and domain predicate flow can be updated accordingly. It can be determined that there is a topic omission in message 706. The supplementary text "Beijing" can be selected from the domain topic flow, and a completion message 706-1 "What delicious food is there in Beijing?" can be generated. In response to completion message 706-1, a response 708 "Roast Duck" can be provided, and the domain topic flow and domain predicate flow (not shown) can be updated accordingly.

[0099] When message 710 "Is it delicious?" is received, the domain topic stream and domain predicate stream can be updated accordingly. It can be determined that topic omission exists in message 710. The supplementary text "roast duck" can be selected from the domain topic stream, and a completed message 710-1 "Is roast duck delicious?" can be generated. In response to completed message 710-1, a response 712 "Roast duck is delicious!" can be provided, and the domain topic stream and domain predicate stream (not shown) can be updated accordingly.

[0100] When the message 714 "Where is the Forbidden City located?" is received, the domain topic flow and domain predicate flow (not shown) may be updated accordingly. It may be determined that there is no contextual omission in the message 714. In response to the message 714, a response 716 "Beijing City Center" may be provided, and the domain topic flow and domain predicate flow (not shown) may be updated accordingly.

[0101] When message 718 "Very good" is received, it can be determined that topic and predicate omissions are present in message 718, and it can be further determined that message 718 is a predetermined dialog act message. Response 720 "It's very convenient to go to the Forbidden City" can be provided based on the response policy corresponding to the dialog act of message 718, and the domain topic flow and domain predicate flow (not shown) can be updated accordingly.

[0102] When message 722 "Where is the Great Wall?" is received, the domain topic stream and domain predicate stream can be updated accordingly. It can be determined that predicate omission exists in message 722. The supplementary text "Where is the location?" can be selected from the domain predicate stream, and a completed message 722-1 "Where is the location of the Great Wall?" can be generated. In response to completed message 722-1, a response 724 "In the northwest suburbs" can be provided, and the domain topic stream and domain predicate stream (not shown) can be updated accordingly.

[0103] When message 726 "I understand" is received, it can be determined that topic and predicate omissions are present in message 726, and that message 726 is further determined to be a predetermined dialog act message. A response 728 "Beijing is in peak tourist season now, go buy your tickets!" can be provided based on the response policy corresponding to the dialog act of message 726, and the domain topic flow and domain predicate flow (not shown) can be updated accordingly.

[0104] It should be understood that Figure 7 The session 700 and the accompanying processing procedures are exemplary and may be modified in any manner depending on the specific application scenario. Figure 7 Only session level information about where the topic / predicate comes from and how many times the topic / predicate has appeared in the session is shown, but any other session level information can also be determined and added to the domain topic stream and domain predicate stream.

[0105] Figure 8 An exemplary conversation 800 in which contextual completion according to an embodiment is applied is shown. Conversation 800 involves transitions between different domains, such as a travel domain and a sports domain. In conversation 800, a chatbot applies a method for performing contextual completion on messages in a conversation according to an embodiment of the present disclosure.

[0106] When receiving the message 802 "What are some fun places in Beijing?", the chatbot can determine that the message relates to the tourism domain, and can update the tourism domain topic stream and / or tourism domain predicate stream maintained for the tourism domain. The chatbot can provide a response 804 "The Great Wall and the Forbidden City are in Beijing. Let's go!" and update the tourism domain topic stream and / or tourism domain predicate stream accordingly.

[0107] When receiving message 806 "What delicious food?", it can be determined that the message remains in the current domain, such as the tourism domain, and the tourism domain topic stream and / or tourism domain predicate stream can be updated. The chatbot can use the tourism domain topic stream and / or tourism domain predicate stream to generate a completion message for message 806 "What delicious food is there in Beijing?" and provide a response 808 "Roast duck", and accordingly update the tourism domain topic stream and / or tourism domain predicate stream.

[0108] When receiving the message 810 "The new NBA season is about to begin", it can be determined that the message has been converted to the sports domain, and the sports domain topic stream and / or sports domain predicate stream maintained for the sports domain can be updated. The chatbot can provide a response 812 "Excited!" and update the sports domain topic stream and / or sports domain predicate stream accordingly.

[0109] When receiving the message 814 "When?", it can be determined that the message remains in the current domain, for example, the sports domain, and the sports domain topic stream and / or the sports domain predicate stream can be updated. The chatbot can use the sports domain topic stream and / or the sports domain predicate stream to generate a completion message for the message 814 "When does the new NBA season start?" and provide a response 816 "This weekend", and update the sports domain topic stream and / or the sports domain predicate stream accordingly.

[0110] When receiving the message 818 "Where is the Forbidden City?", it can be determined that the message has been converted back to the tourism domain, and the tourism domain topic flow and / or tourism domain predicate flow maintained for the tourism domain can be updated. The chatbot can provide a response 820 "Beijing City Center" and update the tourism domain topic flow and / or tourism domain predicate flow accordingly.

[0111] When receiving message 822 "Where is the Great Wall?", it can be determined that the message remains in the current domain, such as the tourism domain, and the tourism domain topic flow and / or tourism domain predicate flow can be updated. The chatbot can use the tourism domain topic flow and / or tourism domain predicate flow to generate a completion message for message 822 "Where is the Great Wall?" and provide a response 824 "In the northwest suburbs", and update the tourism domain topic flow and / or tourism domain predicate flow accordingly.

[0112] Figure 9A flow chart of an exemplary method 900 for contextual completion of a message in a conversation is shown, according to an embodiment.

[0113] At 910, a message can be obtained.

[0114] At 920, the presence of context omission in the message can be detected.

[0115] At 930, it can be determined whether the message remains in the current domain of the session.

[0116] At 940 , in response to determining that the message remains in the current domain, supplemental text for restoring the contextual omission can be selected in the current domain.

[0117] At 950, a completion message can be generated based on the message and the supplemental text.

[0118] In one embodiment, detecting the presence of context omission in the message may include: extracting a topic and / or a predicate from the message; and determining the presence of a topic omission or a predicate omission in the message based on a result of the extraction.

[0119] In one embodiment, the method 900 may further include: adding the extracted topics and / or predicates to the domain topic flow and / or domain predicate flow of the current domain respectively.

[0120] In one embodiment, the method 900 may further include: determining session level information associated with the extracted topics and / or predicates; and adding the session level information to the domain topic flow and / or domain predicate flow of the current domain, respectively.

[0121] In one embodiment, the extraction may be performed based at least on a pre-established knowledge graph.

[0122] In one embodiment, determining whether the message remains in the current domain of the conversation may include: in response to determining that topic omission exists in the message, determining whether the message remains in the current domain using a domain-preserving classifier. The domain-preserving classifier may be based on an RNN model. The domain-preserving classifier may be obtained through knowledge distillation from a classifier based on a BERT model.

[0123] In one embodiment, determining whether the message remains in the current domain of the conversation may include: in response to determining that there is a predicate omission in the message, determining whether the message remains in the current domain based on a topic included in the message.

[0124] In one embodiment, selecting the supplementary text may include: selecting the supplementary text from the domain topic stream in response to determining that a topic omission exists in the message; or selecting the supplementary text from the domain predicate stream in response to determining that a predicate omission exists in the message.

[0125] In one embodiment, selecting the supplementary text may include: selecting the supplementary text from a plurality of candidate texts in the domain topic stream and / or domain predicate stream by at least one of CNN-based sorting and regression-based sorting. The CNN-based sorting may be performed based on at least one of the following information of the candidate text: text similarity between the candidate text and the message, PMI between the candidate text and the message, PMI between the candidate text and a synonym of the message, and PMI between the synonym of the candidate text and the message. The regression-based sorting may be performed based on at least one of the following features of the candidate text: frequency of the candidate text appearing in the conversation, whether the candidate text appears in the previous message, whether the candidate text appears in the previous response, the number of turns in the conversation since the last appearance of the candidate text, PMI between the candidate text and the message, and text similarity between the candidate text and the message.

[0126] In one embodiment, generating the completion message may include: generating at least one candidate completion message by placing the supplementary text at different positions in the message; and selecting the completion message from the at least one candidate completion message using a language model.

[0127] In one embodiment, method 900 may also include: in response to determining that the message is not maintained in the current domain, determining a domain corresponding to the message; selecting supplementary text for restoring the context omission in the domain corresponding to the message; and generating a completion message based on the message and the supplementary text.

[0128] It should be understood that method 900 may also include any steps / processes for performing context completion on messages in a conversation according to the above-mentioned embodiments of the present disclosure.

[0129] Figure 10 An exemplary apparatus 1000 for contextual completion of messages in a conversation according to an embodiment is shown.

[0130] The apparatus 1000 may include: a message obtaining module 1010 for obtaining a message; a context omission detection module 1020 for detecting the presence of context omissions in the message; a domain retention determination module 1030 for determining whether the message is retained in the current domain of the conversation; a supplementary text selection module 1040 for selecting supplementary text for restoring the context omission in the current domain in response to determining that the message is retained in the current domain; and a completion message generation module 1050 for generating a completion message based on the message and the supplementary text.

[0131] In one embodiment, the context omission detection module 1020 may be configured to: extract topics and / or predicates from the message; and determine whether topic omission or predicate omission exists in the message based on the extraction result.

[0132] In one embodiment, the apparatus 1000 may further include: a stream updating module, configured to add the extracted topics and / or predicates to the domain topic stream and / or domain predicate stream of the current domain, respectively.

[0133] In one embodiment, the domain retention determination module 1030 can be used to: in response to determining that there is a topic omission in the message, determine whether the message is retained in the current domain through a domain retention classifier; or in response to determining that there is a predicate omission in the message, determine whether the message is retained in the current domain based on the topic included in the message.

[0134] In one embodiment, the supplementary text selection module 1040 may be configured to: select the supplementary text from the domain topic stream in response to determining that a topic omission exists in the message; or select the supplementary text from the domain predicate stream in response to determining that a predicate omission exists in the message.

[0135] In one embodiment, the supplementary text selection module 1040 may be configured to select the supplementary text from a plurality of candidate texts in the domain topic stream and / or the domain predicate stream by at least one of CNN-based ranking and regression-based ranking.

[0136] In addition, the apparatus 1000 may further include any other modules configured to perform context completion on messages in a conversation according to the above-mentioned embodiments of the present disclosure.

[0137] Figure 11 An exemplary apparatus 1100 for contextual completion of messages in a conversation according to an embodiment is shown.

[0138] Apparatus 1100 may include at least one processor 1110 and a memory 1120 storing computer-executable instructions. When executing the computer-executable instructions, processor 1110 may: obtain a message; detect the presence of contextual omissions in the message; determine whether the message remains in the current domain of the conversation; in response to determining that the message remains in the current domain, select supplementary text in the current domain for restoring the contextual omissions; and generate a completion message based on the message and the supplementary text. Processor 1110 may also perform any other processing for performing contextual completion on messages in a conversation according to the aforementioned embodiments of the present disclosure.

[0139] The embodiments of the present disclosure may be implemented in a non-transitory computer-readable medium. The non-transitory computer-readable medium may include instructions that, when executed, cause one or more processors to perform any operations of the method for contextual completion of messages in a conversation according to the above-described embodiments of the present disclosure.

[0140] It should be understood that all operations in the method described above are merely exemplary, and the present disclosure is not limited to any operation in the method or the order of these operations, but should cover all other equivalent transformations under the same or similar concept.

[0141] It should also be understood that all modules in the above-described device can be implemented in various ways. These modules can be implemented as hardware, software, or a combination thereof. In addition, any module in these modules can be further divided into submodules or combined together in function.

[0142] Processor has been described in conjunction with various devices and methods.These processors can be implemented using electronic hardware, computer software or its arbitrary combination.Whether these processors are implemented as hardware or software will depend on specific application and the overall design constraint imposed on the system.As an example, the processor provided in this disclosure, any part of the processor or any combination of processors can be implemented as microprocessor, microcontroller, digital signal processor (DSP), field programmable gate array (FPGA), programmable logic device (PLD), state machine, gate logic, discrete hardware circuit and other suitable processing components configured for performing the various functions described in this disclosure.The function of the processor provided in this disclosure, any part of the processor or any combination of processors can be implemented as software performed by microprocessor, microcontroller, DSP or other suitable platform.

[0143] Software should be broadly considered to mean instructions, instruction sets, codes, code segments, program codes, programs, subroutines, software modules, applications, software applications, software packages, routines, subroutines, objects, running threads, processes, functions, etc. Software can reside in a computer-readable medium. A computer-readable medium can include, for example, a memory, which can be, for example, a magnetic storage device (e.g., a hard disk, a floppy disk, a magnetic stripe), an optical disk, a smart card, a flash memory device, a random access memory (RAM), a read-only memory (ROM), a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), a register, or a removable disk. Although the memory is shown as being separate from the processor in various aspects provided in the present disclosure, the memory can also be located inside the processor (e.g., a cache or register).

[0144] The above description is provided to enable any person skilled in the art to implement the various aspects described herein. Various modifications to these aspects will be apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects. Therefore, the claims are not intended to be limited to the aspects shown herein. All structural and functional equivalents of the elements of the various aspects described in this disclosure that are known or to be known to those skilled in the art are expressly incorporated herein by reference and are intended to be covered by the claims.

Claims

1. A method for performing contextual completion on a message in a conversation, comprising: Obtaining a message, wherein the content of the message is saved to an information stream maintained in real time; Detecting the presence of context omission in the message, wherein the detecting comprises: extracting topics and / or predicates from the message; and determining, based on the extracted result, whether topic omission or predicate omission exists in the message; In response to determining that a topic omission or a predicate omission exists in the message, determining whether the message remains in the current domain of the conversation; In response to determining that the message remains in the current domain, selecting supplemental text for restoring the contextual omission from candidate texts in the information stream maintained for the current domain using a neural network-based ranking, wherein the selecting comprises: calculating a matrix between the supplementary text and the message; extracting features from the matrix using a convolutional layer coupled with a pooling layer; outputting a result to a connection layer based on the matrix and the features; and generating a ranking using the results, wherein the selecting is based on the ranking; and Generating a completion message based on the message and the supplemental text, wherein the generating comprises: generating at least one candidate completion message by placing the supplementary text at different positions in the message; and The completion message is selected from the at least one candidate completion message using a language model.

2. The method of claim 1, further comprising: The extracted topics and / or predicates are added to the domain topic stream and / or domain predicate stream of the current domain respectively.

3. The method of claim 1 , further comprising: determining conversation-level information associated with the extracted topics and / or predicates; as well as The session level information is added to the domain topic stream and / or domain predicate stream of the current domain respectively.

4. The method according to claim 1, wherein The extraction is performed based at least on a pre-established knowledge graph.

5. The method according to claim 1, wherein Determining whether the message remains in the current domain of the session includes: In response to determining that topic omission exists in the message, determining, by a domain-preserving classifier, whether the message remains in the current domain.

6. The method according to claim 5, wherein: The domain preserving classifier is based on a recurrent neural network (RNN) model, and The domain-preserving classifier is obtained through knowledge distillation from a classifier based on the Bidirectional Encoder Representations from Transformers (BERT) model.

7. The method of claim 1, wherein: Determining whether the message remains in the current domain of the session includes: In response to determining that a predicate omission exists in the message, it is determined whether the message remains in the current domain based on a topic included in the message.

8. The method of claim 2, wherein: Selected supplementary texts include: In response to determining that a topic omission exists in the message, selecting the supplementary text from the domain topic stream; or In response to determining that a predicate omission exists in the message, the supplemental text is selected from the domain predicate stream.

9. The method of claim 2, wherein: Selected supplementary texts include: The supplementary text is selected from a plurality of candidate texts in the domain topic stream and / or the domain predicate stream by at least one of convolutional neural network (CNN)-based ranking and regression-based ranking.

10. The method of claim 9, wherein: The ranking based on the convolutional neural network (CNN) is performed according to at least one of the following information of the candidate texts: The text similarity between the candidate text and the message, Pointwise Mutual Information (PMI) between the candidate text and the message, the PMI between the candidate text and a close representation of the message, and The PMI between the candidate text's synonymous representation and the message; and The regression-based ranking is performed based on at least one of the following characteristics of the candidate texts: the frequency with which the candidate text has appeared in the conversation, Whether the candidate text appears in the previous message, Whether the candidate text appears in the previous response, the number of turns that have passed since the last occurrence of the candidate text in the session, The PMI between the candidate text and the message, and The text similarity between the candidate text and the message.

11. The method of claim 1 , further comprising: In response to determining that the message is not held in the current domain, determining a domain corresponding to the message; selecting, in a field corresponding to the message, supplementary text for restoring the contextual omission; as well as A completion message is generated based on the message and the supplemental text.

12. A device for performing context completion on a message in a conversation, comprising: A message acquisition module, configured to acquire a message, wherein the content of the message is saved in an information stream maintained in real time; A context omission detection module is configured to detect the presence of context omission in the message, wherein the detection includes: extracting topics and / or predicates from the message; and determining, based on the extracted result, whether topic omission or predicate omission exists in the message; a domain preservation determination module for determining whether the message is preserved in the current domain of the conversation in response to determining that there is a topic omission or a predicate omission in the message; a supplementary text selection module configured to, in response to determining that the message remains in the current domain, select supplementary text for restoring the contextual omission from candidate texts in the information stream maintained for the current domain using a neural network-based ranking, wherein the selection comprises: calculating a matrix between the supplementary text and the message; extracting features from the matrix using a convolutional layer coupled with a pooling layer; outputting a result to a connection layer based on the matrix and the features; and generating a ranking using the results, wherein the selecting is based on the ranking; and A completion message generation module is configured to generate a completion message based on the message and the supplementary text, wherein the generation includes: generating at least one candidate completion message by placing the supplementary text at different positions in the message; and The completion message is selected from the at least one candidate completion message using a language model.

13. The apparatus of claim 12, further comprising: A stream updating module is configured to add the extracted topics and / or predicates to the domain topic stream and / or domain predicate stream of the current domain respectively.

14. The apparatus of claim 12, wherein: The domain holding determination module is used for: In response to determining that topic omission exists in the message, determining, by a domain-preserving classifier, whether the message remains in the current domain; or In response to determining that a predicate omission exists in the message, it is determined whether the message remains in the current domain based on a topic included in the message.

15. The apparatus of claim 13, wherein: The supplementary text selection module is used to: In response to determining that a topic omission exists in the message, selecting the supplementary text from the domain topic stream; or In response to determining that a predicate omission exists in the message, the supplemental text is selected from the domain predicate stream.

16. The apparatus of claim 13, wherein: The supplementary text selection module is used to: The supplementary text is selected from a plurality of candidate texts in the domain topic stream and / or the domain predicate stream by at least one of convolutional neural network (CNN)-based ranking and regression-based ranking.

17. A device for performing context completion on a message in a conversation, comprising: at least one processor; as well as a memory storing computer-executable instructions that, when executed, cause the at least one processor to: Obtaining a message, wherein the content of the message is saved to an information stream maintained in real time, Detecting the presence of context omission in the message, wherein the detecting comprises: extracting topics and / or predicates from the message; and Determining whether topic omission or predicate omission exists in the message based on the extracted result, In response to determining that a topic omission or a predicate omission exists in the message, determining whether the message remains in the current domain of the conversation, In response to determining that the message remains in the current domain, selecting supplemental text for restoring the contextual omission from candidate texts in the information stream maintained for the current domain using a neural network-based ranking, wherein the selecting comprises: calculating a matrix between the supplementary text and the message; extracting features from the matrix using a convolutional layer coupled with a pooling layer; outputting a result to a connection layer based on the matrix and the features; and generating a ranking using the results, wherein the selection is based on the ranking, and Generating a completion message based on the message and the supplemental text, wherein the generating comprises: generating at least one candidate completion message by placing the supplementary text at different positions in the message; and The completion message is selected from the at least one candidate completion message using a language model.

Images