Methods, devices, and computer programs for generating dialog models

Abstract

A method for generating a neural network-based open-domain dialog model includes receiving an input utterance from a device having a conversation with the dialog model, obtaining a plurality of candidate answers to the input utterance from the dialog model, determining a plurality of discriminator scores for the candidate answers based on a reference-free discriminator, determining a plurality of quality scores associated with the candidate answers, and training the dialog model based on the quality scores.

Description

[Technical Field]

[0001] 1. Field This disclosure relates to generating a dialogue model, and more specifically, to training a neural network-based dialogue model based on a reference-free discriminator. [Background technology]

[0002] 2. Description of related fields Dialogue response generation aims to generate engaging and coherent responses given a dialogue history. Research interest in this area is growing, primarily due to increasing commercial demand. Increasingly powerful neural models have been proposed, such as using token-level cross-entropy loss to maximize the likelihood of a human-annotated response for each dialogue context. However, significant failures, such as generating meaningless or illogical responses, are frequently observed in dialogue response generation.

[0003] To address this problem, early attempts leverage external knowledge for richer functionality. However, these assume a single gold response (typically chosen by a human) for a given context, ignoring any potential error propagation, as various valid responses may exist with different meanings depending on the dialogue context. As a result, problems can arise due to the acquisition of incorrect knowledge. Recently, large-scale training has been proposed as a solution, based on the hypothesis that the above problem can be greatly mitigated when the model has seen enough instances (perhaps billions). However, this solution is challenged by the increasingly rapid evolution of languages, which continue to introduce new topics, words, and slogans. Furthermore, large-scale training is time-consuming and resource-intensive, and therefore may not be a sustainable direction. [Overview of the Initiative]

[0004] According to several possible implementations, a method (preferably generating a neural network-based open-domain dialogue model) includes the steps of: receiving an input utterance from a device having a conversation with the dialogue model (where the device from which the input utterance is received is a first device, and the dialogue model is run on a second device in conversational communication with the first device); obtaining a plurality of candidate answers for the input utterance from the dialogue model (i.e., from the second device on which the dialogue model is running), wherein the plurality of candidate answers comprises a first candidate answer and a second candidate answer; and determining a plurality of discriminator scores for the first candidate answer, wherein the plurality of discriminator scores comprises information corresponding to the first candidate answer and contextual information corresponding to the conversation history. The process may include: a step of providing a plurality of discriminators that evaluate the quality of a first candidate answer based on information and contextual information corresponding to the conversation history; a step of determining a plurality of discriminator scores for a second candidate answer, provided by the plurality of discriminators that evaluate the quality of a second candidate answer based on information corresponding to the second candidate answer and contextual information corresponding to the conversation history; a step of determining a first quality score associated with the first candidate answer, the first quality score being based on a weighted sum of the plurality of discriminator scores corresponding to the first candidate answer; a step of determining a second quality score associated with the second candidate answer, the second quality score being based on a weighted sum of the plurality of discriminator scores corresponding to the second candidate answer; and a step of training a dialogue model based on at least one of the first or second quality scores.

[0005] According to several possible implementations, the device may comprise at least one memory configured to store program code and a neural network-based open-domain dialogue model, and at least one processor configured to read the program code and act as directed by the program code. The program code comprises: a receive code configured to cause at least one processor to receive input utterances from a device having a conversation with a dialogue model; an acquire code configured to cause at least one processor to acquire a plurality of candidate answers for an input utterance from the dialogue model, wherein the plurality of candidate answers comprises a first candidate answer and a second candidate answer; a first decision code configured to cause at least one processor to determine a plurality of discriminator scores for the first candidate answer, wherein the plurality of discriminator scores are provided by a plurality of discriminators that evaluate the quality of the first candidate answer based on information corresponding to the first candidate answer and contextual information corresponding to the history of the conversation; and a second decision code configured to cause at least one processor to determine a plurality of discriminator scores for the second candidate answer, wherein the plurality of discriminators Taskcore may include: a second decision code provided by a plurality of discriminators that evaluate the quality of a second candidate answer based on information corresponding to a second candidate answer and contextual information corresponding to the conversation history; a third decision code configured to cause at least one processor to determine a first quality score associated with a first candidate answer, wherein the first quality score is based on a weighted sum of a plurality of discriminator scores corresponding to the first candidate answer; a fourth decision code configured to cause at least one processor to determine a second quality score associated with a second candidate answer, wherein the second quality score is based on a weighted sum of a plurality of discriminator scores corresponding to the second candidate answer; and training code configured to cause at least one processor to train a dialogue model based on at least one of the first or second quality scores.

[0006] According to several possible implementations, a non-temporary computer-readable medium stores instructions, and when an instruction is executed by one or more processors of a mobile device, the one or more processors: receive an input utterance from the device having a conversation with a dialog model; obtain multiple candidate answers for the input utterance from the dialog model, the multiple candidate answers comprising a first candidate answer and a second candidate answer; determine multiple discriminator scores for the first candidate answer, the multiple discriminator scores provided by multiple discriminators that evaluate the quality of the first candidate answer based on information corresponding to the first candidate answer and contextual information corresponding to the conversation history; and multiple disks for the second candidate answer. The invention may include one or more instructions to determine a limiter score, provided by multiple discriminators that evaluate the quality of the second-choice answer based on information corresponding to the second-choice answer and contextual information corresponding to the conversation history; to determine a first quality score associated with the first-choice answer, which is based on a weighted sum of multiple discriminator scores corresponding to the first-choice answer; to determine a second quality score associated with the second-choice answer, which is based on a weighted sum of multiple discriminator scores corresponding to the second-choice answer; and to train a dialogue model based on at least one of the first or second quality scores. [Brief explanation of the drawing]

[0007] The above and other aspects, features and embodiments of this disclosure will become more apparent from the following description, which will be considered in conjunction with the accompanying drawings.

[0008] [Figure 1] This is a schematic diagram illustrating an exemplary implementation described herein.

[0009] [Figure 2] This is a diagram of an exemplary environment in which the systems and / or methods described herein may be implemented.

[0010] [Figure 3] It is a diagram of exemplary components of one or more devices in FIG. 2.

[0011] [Figure 4] It is a flowchart of an exemplary process of a method for generating a neural network-based open domain dialogue model.

Mode for Carrying Out the Invention

[0012] The following detailed description of the exemplary embodiments refers to the accompanying drawings. The same reference numerals in different drawings may identify the same or similar elements.

[0013] The foregoing disclosure provides illustration and description, but is not intended to be exhaustive or to limit the implementation to the exact form disclosed. Modifications and variations are possible in light of the above disclosure or may be obtained from practice of the implementation.

[0014] It will be apparent that the systems and / or methods described herein may be implemented in different forms of hardware, firmware, or combinations of hardware and software. The actual special control hardware or software code used to implement these systems and / or methods is not limiting. Thus, although the operation and behavior of the systems and / or methods have been described herein without reference to specific software code, it is understood that software and hardware can be designed to implement the systems and / or methods based on the description herein.

[0015] Particular combinations of functions are recited in the claims and / or disclosed in the specification, but these combinations are not intended to limit the disclosure of possible implementations. In fact, many of these functions may be combined in ways not specifically recited in the claims and / or not disclosed in the specification. Each of the dependent claims listed below may depend directly on only one claim, but the disclosure of possible implementations includes each dependent claim in combination with all other claims in the set of claims.

[0016] Elements, acts, or instructions used herein should not be construed as important or essential unless expressly described as such. Also, as used herein, the articles “a” and “an” are intended to include one or more items and may be used interchangeably with “one or more.” Where only one item is intended, the term “one” or similar words are used. Also, as used herein, terms such as “has,” “have,” “having,” “include,” “including,” etc. are intended to be open-ended terms. Further, the phrase “based on” is intended to mean “at least partially based on” unless otherwise expressly stated.

[0017] FIG. 1 is a diagram outlining an embodiment of a method for training a neural network-based dialog model. Method 100 may be executed on a platform such as a server, a desktop computer, a handheld computing device, or a combination thereof.

[0018] As shown in FIG. 1, in operation 102, an utterance from a conversation may be obtained by the platform. The utterance may be a text string such as “Do you want to play a game?” The utterance may be received from an external device communicating with the platform.

[0019] In operation 104, based on the acquired utterance, the dialogue model may generate a series of candidate responses to the acquired utterance. As shown in Figure 1, candidate responses from the 1st to the Nth may be generated. According to one embodiment, the dialogue model may use a beam search algorithm.

[0020] In operation 106, the 1st to Nth no-reference discriminator scores may be determined by the platform. Each no-reference discriminator score may correspond to a separate no-reference discriminator. The no-reference discriminators may calculate their respective no-reference discriminator scores based on the input of candidate answers and the conversation history. According to one embodiment, the no-reference discriminator may consider only the candidate answers and the conversation history and not any external knowledge.

[0021] According to one embodiment, each discriminator may be based on complementary factors. Complementary factors may include the specificity, consistency, fluency, and relatedness of the candidate answer. Referring to Figure 1, for example, the first no-reference discriminator score may indicate the specificity of the candidate answer, the second no-reference discriminator score may indicate the consistency of the answer with previous answers, and additional no-reference discriminator scores may consider other complementary factors or other aspects of already considered complementary factors.

[0022] Figure 1 shows the reference-free discriminator score obtained only for the first candidate answer. However, reference-free discriminators can be obtained for each of the first to Nth candidate answers.

[0023] In operation 108, a weighted sum of the first candidate answers can be generated based on each of the no-reference discriminator scores corresponding to the first candidate answers. For example, each no-reference discriminator score corresponding to the first candidate answers can be assigned a certain weight based on empirical data, and then each weighted no-reference discriminator score can be summed to obtain a weighted sum of the candidate answers. According to one embodiment, a weighted sum can be generated for each candidate answer generated in operation 104.

[0024] In operation 110, the dialogue model may be trained based on a weighted sum corresponding to the first candidate answer. For example, the training algorithm may be based on a loss function that incorporates the weighted sum of the first candidate answers. According to one embodiment, the dialogue model may be trained based on a weighted sum corresponding to a plurality of candidate answers generated in operation 104.

[0025] In this way, some implementations herein provide dialogue models that generate more meaningful, logical, and relevant answers based on less resource-intensive training.

[0026] Figure 2 is a diagram of an exemplary environment 200 in which the systems and / or methods described herein may be implemented. As shown in Figure 2, the environment 200 may include user devices 210, a platform 220, and a network 230. The devices in environment 200 may be interconnected via wired connections, wireless connections, or a combination of wired and wireless connections.

[0027] The user device 210 includes one or more devices capable of receiving, generating, storing, processing, and / or providing information associated with the platform 220. For example, the user device 210 may include computing devices (e.g., desktop computers, laptop computers, tablet computers, handheld computers, smart speakers, servers, etc.), mobile phones (e.g., smartphones, wireless phones, etc.), wearable devices (e.g., a pair of smart glasses or a smartwatch), or similar devices. In some implementations, the user device 210 may receive information from and / or transmit information to the platform 220.

[0028] Platform 220 includes one or more devices capable of training a dialog model using a referenceless discriminator, as described elsewhere in this specification. In some implementations, Platform 220 may include a cloud server or a group of cloud servers. In some implementations, Platform 220 may be designed to be modular so that specific software components can be swapped in or out depending on specific needs. Thus, Platform 220 can be easily and / or quickly reconfigured for different applications.

[0029] In some implementations, as shown, platform 220 may be hosted in a cloud computing environment 222. In particular, the implementations described herein describe platform 220 as being hosted within a cloud computing environment 222, but in some implementations, platform 220 may be non-cloud-based (i.e., implemented outside a cloud computing environment) or partially cloud-based.

[0030] The cloud computing environment 222 includes an environment that hosts platform 220. The cloud computing environment 222 may provide services such as computing, software, data access, and storage that do not require the end user's (e.g., user device 210) knowledge of the physical location and configuration of the system and / or device that hosts platform 220. As shown, the cloud computing environment 222 may include a group of computing resources 224 (collectively referred to as “computing resources 224 (plural)” and individually as “computing resource 224 (singular)”).

[0031] Computing resource 224 includes one or more personal computers, workstation computers, server devices, or other types of computing and / or communication devices. In some implementations, computing resource 224 may host platform 220. Cloud resources may include computing instances running within computing resource 224, storage devices provided by computing resource 224, data transfer devices provided by computing resource 224, etc. In some implementations, computing resource 224 may communicate with other computing resources 224 via wired connections, wireless connections, or a combination of wired and wireless connections.

[0032] As further shown in Figure 2, the computing resource 224 includes a group of cloud resources such as one or more applications ("APP") 224-1, one or more virtual machines ("VM") 224-2, virtualized storage ("VS") 224-3, one or more hypervisors ("HYP") 224-4, etc.

[0033] Application 224-1 includes one or more software applications that may be provided to or accessed by the user device 210 and / or sensor devices. Application 224-1 eliminates the need to install and run software applications on the user device 210. For example, Application 224-1 may include any other software that can be provided via the software associated with the platform 220 and / or the cloud computing environment 222. In some implementations, one application 224-1 may send information to / receive information from one or more other applications 224-1 via a virtual machine 224-2.

[0034] The virtual machine 224-2 includes a software implementation of a machine (e.g., a computer) that runs programs like a physical machine. Depending on the extent to which the virtual machine 224-2 is used and corresponds to any real machine, the virtual machine 224-2 may be either a system virtual machine or a process virtual machine. A system virtual machine may provide a complete system platform that supports the execution of a complete operating system ("OS"). A process virtual machine may run a single program and may support a single process. In some implementations, the virtual machine 224-2 may run on behalf of a user (e.g., a user device 210) and may manage the infrastructure of a cloud computing environment 222, such as data management, synchronization, or long-duration data transfer.

[0035] Virtualized storage 224-3 includes one or more storage systems and / or one or more devices that use virtualization technology within the storage system or device of the computing resource 224. In some implementations, within the context of a storage system, the types of virtualization may include block virtualization and file virtualization. Block virtualization may refer to the abstraction (or separation) of logical storage from physical storage so that the storage system can be accessed regardless of the physical storage or heterogeneous structure. This separation may allow the administrator of the storage system to have flexibility in how they manage storage for end users. File virtualization can eliminate the dependency between data accessed at the file level and where the file is physically stored. This may enable optimized storage usage, server consolidation, and / or uninterrupted file movement performance.

[0036] Hypervisor 224-4 may provide hardware virtualization technology that enables multiple operating systems (e.g., "guest operating systems") to run simultaneously on a host computer such as computing resource 224. Hypervisor 224-4 may present a virtual operating system platform to the guest operating system and may manage the execution of the guest operating system. Multiple instances of various operating systems may share virtualized hardware resources.

[0037] Network 230 includes one or more wired and / or wireless networks. For example, Network 230 may include cellular networks (e.g., fifth-generation (5G) networks, Long-Term Evolution (LTE) networks, third-generation (3G) networks, code division multiple access (CDMA) networks, etc.), public land mobile networks (PLMN), local area networks (LANs), wide area networks (WANs), metropolitan area networks (MANs), telephone networks (e.g., public switched telephone networks (PSTNs)), private networks, ad hoc networks, intranets, the Internet, fiber optic-based networks, etc., and / or combinations of these or other types of networks.

[0038] The number and arrangement of devices and networks shown in Figure 2 are provided as an example. In practice, there may be additional devices and / or networks, fewer devices and / or networks, different devices and / or networks, or devices and / or networks arranged differently than those shown in Figure 2. Furthermore, two or more devices shown in Figure 2 may be implemented within a single device, or a single device shown in Figure 2 may be implemented as multiple distributed devices. Additionally or alternatively, a set of devices in environment 200 (e.g., one or more devices) may perform one or more functions that are described as being performed by another set of devices in environment 200.

[0039] Figure 3 is a diagram of exemplary components of device 300. Device 300 may correspond to user device 210 and / or platform 220. As shown in Figure 3, device 300 may include a bus 310, a processor 320, memory 330, storage components 340, input components 350, output components 360, and a communication interface 370.

[0040] Bus 310 includes components that enable communication between components of device 300. Processor 320 is implemented in hardware, firmware, or a combination of hardware and software. Processor 320 is a central processing unit (CPU), graphics processing unit (GPU), high-speed processing unit (APU), microprocessor, microcontroller, digital signal processor (DSP), field-programmable gate array (FPGA), application-specific integrated circuit (ASIC), or another type of processing component. In some implementations, processor 320 includes one or more processors that can be programmed to perform functions. Memory 330 includes random access memory (RAM), read-only memory (ROM), and / or another type of dynamic or static storage device (e.g., flash memory, magnetic memory, and / or optical memory) that stores information and / or instructions for use by processor 320.

[0041] The storage component 340 stores information and / or software related to the operation and use of device 300. For example, the storage component 340 may include a hard disk (e.g., magnetic disk, optical disk, magneto-optical disk and / or solid-state disk), a compact disk (CD), a digital multipurpose disk (DVD), a floppy disk, a cartridge, magnetic tape and / or another type of non-temporary computer-readable media, along with a corresponding drive.

[0042] The input component 350 includes components that enable the device 300 to receive information, such as through user input (e.g., a touchscreen display, keyboard, keypad, mouse, buttons, switches, and / or microphone). Additionally or alternatively, the input component 350 may include sensors for sensing information (e.g., a Global Positioning System (GPS) component, accelerometer, gyroscope, and / or actuator). The output component 360 includes components that provide output information from the device 300 (e.g., a display, speaker, and / or one or more light-emitting diodes (LEDs)).

[0043] The communication interface 370 includes transceiver-like components (e.g., transceivers and / or separate receivers and transmitters) that enable device 300 to communicate with other devices via wired connections, wireless connections, or a combination of wired and wireless connections. The communication interface 370 may enable device 300 to receive information from and / or provide information to other devices. For example, the communication interface 370 may include Ethernet® interfaces, optical interfaces, coaxial interfaces, infrared interfaces, radio frequency (RF) interfaces, Universal Serial Bus (USB) interfaces, Wi-Fi interfaces, cellular network interfaces, etc.

[0044] Device 300 may perform one or more processes as described herein. Device 300 may perform these processes in response to the processor 320 executing software instructions stored in a non-temporary computer-readable medium, such as memory 330 and / or storage component 340. A computer-readable medium is defined herein as a non-temporary memory device. A memory device includes a memory space within a single physical storage device or a memory space that extends across multiple physical storage devices.

[0045] Software instructions may be read into memory 330 and / or storage component 340 from another computer-readable medium or from another device via the communication interface 370. When executed, the software instructions stored in memory 330 and / or storage component 340 can cause the processor 320 to execute one or more processes described herein. In addition or alternatively, hardwired circuits may be used instead of or in combination with software instructions to execute one or more processes described herein. Therefore, the implementations described herein are not limited to any particular combination of hardware circuits and software.

[0046] The number and arrangement of components shown in Figure 3 are provided as an example. In practice, device 300 may include additional components, fewer components, different components, or components arranged differently than those shown in Figure 3. Additionally or alternatively, a set of components of device 300 (e.g., one or more components) may perform one or more functions that are described as being performed by another set of components of device 300.

[0047] Figure 4 is a flowchart of an exemplary process 400 for generating a dialog model based on a no-reference discriminator. In some implementations, one or more process blocks in Figure 4 may be executed by platform 220. In some implementations, one or more process blocks in Figure 4 may be executed by another device or group of devices separate from or including platform 220, such as user device 210.

[0048] As shown in Figure 4, process 400 may include receiving an input utterance (block 410). The input utterance may be in the form of a text string.

[0049] For example, platform 220 may receive an input utterance from a user or training set, such as "Do you want to play the game?". The input utterance may be part of a dialogue or conversation.

[0050] As further shown in Figure 4, process 400 may include determining a discriminator score for each of the candidate answers (block 420).

[0051] Platform 220 may determine discriminator scores based on discriminators that quantify complementary factors of good responses. Good responses may be significant, logical, and relevant. Discriminators may be reference-free, meaning they do not consider external knowledge, but rather candidate answers and contextual information from the current conversation or dialogue history.

[0052] This approach minimizes computational waste and allows for the selection of complementary factors for good answers, minimizing duplication. For example, factors may include the specificity of the candidate answer, consistency between the candidate answer and previously outputted answers, fluency of the candidate answer, and relevance of the candidate answer. The discriminator may correspond to one of these complementary factors for a good answer.

[0053] According to one embodiment, discriminator spec u Using this method, the specificity score for candidate responses u can be calculated as the average of the normalized inverse document frequency (NIDF) values.

[0054] The NIDF value for word w can be based on the inverse document frequency (IDF) of word w. The IDF of word w is IDF(w) = log(|U| / |U w |U| is defined as the number of candidate answers in the set of generated candidate answers, where |U| is the number of candidate answers in the set of candidate answers. w | represents the number of candidate answers that include "w".

[0055] NIDF can be determined based on the following Equation 1:

[0056]

Equation

[0057] In Equation 1, min_idf represents the minimum IDF value of the candidate answer, and max_idf represents the maximum IDF value of the candidate answer.

[0058] Discriminator spec u can be determined using the following Equation 2:

Equation

[0059] The argument c (context information) is included for consistency with other discriminators but is not considered when determining spec u That is, spec u (u, c) considers the candidate answer without considering the context information (e.g., conversation or dialogue history).

[0060] According to one embodiment, discriminator spec c can be used to calculate the specificity score for candidate answer u by considering the context information c. Discriminator spec c can be determined using the following Equation 3:

[0061]

Equation

[0062] As shown in Equation 3, spec c considers only the words w that appear in both the candidate answer u and the context information c.

[0063] Discriminator specu and spec c Both can facilitate the generation of detailed and engaging responses in the dialogue model. For example, in response to the utterance "How about going for come bowling?", the discriminator spec u and spec c This can prevent the dialogue model from using generic and meaningless suggested answers such as "I don't know."

[0064] According to one embodiment, the discriminator cons can evaluate whether a candidate answer contradicts a previous answer output by the dialogue model during the conversation. Thus, the discriminator cons is based on consistency complementarity factors.

[0065] The discriminator cons may be based on pre-training of deep bidirectional transforms for language understanding. For example, the discriminator cons may be based on an optimized BERT pre-training approach such as the RoBERTa model. The RoBERTa model may be fine-tuned on a dataset such as the Multi-Genre Natural Language Inference (MNLI) dataset. The MNLI dataset considers the following three possible relationships between each pair of sentences: contradictory, entailment, and neutral.

[0066] For example, if someone in a conversation says they "love dogs," it's unlikely they would also say they "are afraid of dogs and usually avoid them." Therefore, the discriminator cons generates a consistency discriminator score by considering the candidate answer u and contextual information c, by calculating the probability that the candidate answer u is consistent with previous answers in the dialogue model during the conversation or current dialogue session.

[0067] According to one embodiment, discriminator flu determines whether a candidate answer is fluent and natural. Therefore, discriminator flu is based on fluency complement factors.

[0068] The discriminator flu may be based on a tunable neural conversation generation model. For example, the discriminator flu may be based on a tunable gigaword-scale neural network model for generating conversational responses, such as the dialogue generative pre-training transformer DialoGPT.

[0069] The discriminator flu, given context information c, obtains the perplexity of each generated turn u and generates a fluency discriminator score.

[0070] According to one embodiment, discriminator rel s This measures how closely the generated response relates to the dialogue model. Therefore, the discriminator rel s It is based on relevance and complementary factors.

[0071] Discriminator rel s This sometimes focuses on semantic similarity at the sentence level. For example, discriminators sThis can be based on the cosine similarity between the representation vector of candidate answer u and the representation vector of context c, which is generated by a language representation model that pre-trains deep bidirectional representations from unlabeled text by jointly conditioning both left and right contexts at all layers. For example, the cosine similarity between the representation vector of candidate answer u and the representation vector of context c can be generated by a model such as BERT (Bidirectional Encoder Representations from Transforms), which is designed to pre-train deep bidirectional representations from unlabeled text by jointly conditioning both left and right contexts at all layers.

[0072] As further shown in Figure 4, process 400 may further include determining a quality score for each candidate answer based on the discriminator score (block 440). The quality score for candidate answer u may be based on a weighted sum of the discriminator scores of candidate answer u. For example, platform 220 may determine the quality score r(u,c) of candidate answer u in context c based on the following reward function equation [4]:

[0073]

number

[0074] In Equation 4, d j (u,c) represents the discriminator score, |d| represents the amount of the discriminator, and Φ jΦ represents the weight assigned to a discriminator. The weight Φ assigned to a discriminator score can be determined based on empirical evidence. For example, each discriminator score may be assigned a specific weight Φ determined through experimentation. The weight Φ for each discriminator can be determined only for that discriminator. Therefore, different weights Φ may correspond to each discriminator. According to one embodiment, the weight Φ can be optimized for different situations.

[0075] As further shown in Figure 4, process 400 may further include training a dialogue model based on quality scores (block 450).

[0076] For example, platform 220 may use a self-critical training algorithm to train the dialogue model. This algorithm could be a policy-gradient reinforcement learning (RL) algorithm. The loss function for training is shown in Equation 5.

[0077]

number

[0078] In Equation 5,

number

number

[0079] In equation 5, |u s | is a sampled utterance u s This represents the length of the conversation. As mentioned earlier, 'w' represents a word within the candidate answer, and 'c' represents contextual information such as conversation or dialogue history.

[0080] According to one embodiment, a self-critical training algorithm can be a special case of reinforcement by a baseline algorithm.

[0081] According to one embodiment, the above RL-based target l rl The cross-entropy-based target l ce Combined with this, the final training objective shown in Equation 6 can be generated:

[0082]

number

[0083] In Equation 6, λ can be determined empirically. According to one embodiment, λ can be empirically set to 0.7.

[0084] Cross-entropy based target l ce This allows for determining the token-level cross-entropy loss based on human-annotated responses.

[0085] Figure 4 shows an exemplary block of process 400, but in some implementations, process 400 may include additional blocks, fewer blocks, different blocks, or blocks arranged differently than those shown in Figure 4. Additionally or alternatively, two or more blocks of process 400 may be executed in parallel.

[0086] The foregoing disclosures provide examples and explanations, but are not intended to be exhaustive or to limit implementations to the exact form disclosed. Modifications and variations may be possible in light of the foregoing disclosures or may be derived from implementation practice.

[0087] As used herein, the term "component" is intended to be interpreted broadly as hardware, firmware, or a combination of hardware and software.

[0088] It will be apparent that the systems and / or methods described herein may be implemented in different forms of hardware, firmware, or combinations of hardware and software. The specific control hardware or software code used to implement these systems and / or methods is not limiting to the implementation. Therefore, although the operation and behavior of the systems and / or methods are described herein without reference to specific software code, it is understood that software and hardware may be designed to implement the systems and / or methods based on the descriptions herein.

[0089] Certain combinations of functions are described in the claims and / or disclosed in the specification, but these combinations are not intended to limit the disclosure of possible implementations. In fact, many of these functions may be combined in ways not specifically described in the claims and / or disclosed in the specification. Each dependent claim listed below may depend directly on only one claim, but the disclosure of possible implementations includes each dependent claim in combination with all other claims in the set of claims.

[0090] Any elements, actions, or instructions used herein should not be construed as important or essential unless expressly stated otherwise. Furthermore, when used herein, the articles “a” and “an” are intended to include one or more items and may be used interchangeably with “one or more.” Additionally, when used herein, the term “set” is intended to include one or more items (e.g., related items, unrelated items, combinations of related and unrelated items, etc.) and may be used interchangeably with “one or more.” When only one item is intended, the term “one” or similar is used. Furthermore, when used herein, terms such as “has,” “have,” and “having” are intended to be open-ended terms. Additionally, the phrase “based on” is intended to mean “at least partially based on” unless otherwise explicitly stated.

Claims

1. A method that is executed by a processor, The steps include receiving input utterances from the user's device during a conversation with the dialogue model, A step of obtaining a plurality of candidate responses for the input utterance from the dialog model, wherein the plurality of candidate responses include a first candidate response and a second candidate response. A step of determining multiple first discriminator scores for the first candidate answer using multiple discriminators, wherein each of the multiple discriminators evaluates the quality of the candidate answer based on one of different complementary factors and determines a discriminator score; The steps include determining a plurality of second discriminator scores for the second candidate answer using the plurality of discriminators, A step of determining a first quality score associated with the first candidate answer, wherein the first quality score is based on a weighted sum of the plurality of first discriminator scores corresponding to the first candidate answer, A step of determining a second quality score associated with the second candidate answer, wherein the second quality score is based on a weighted sum of the plurality of second discriminator scores corresponding to the second candidate answer, A step of training the dialogue model based on at least one of the first quality score or the second quality score, Includes, A method comprising: a first discriminator that evaluates the fluency of a candidate answer by determining the perplexity of the candidate answer based on information corresponding to the candidate answer and contextual information corresponding to the history of the conversation; a second discriminator that evaluates the singularity of the candidate answer by determining the normalized inverse function document frequency of the candidate answer; a third discriminator that evaluates the consistency of the candidate answer by calculating the probability that the candidate answer contradicts a previous candidate answer output by the dialogue model during the conversation; and a fourth discriminator that evaluates the relevance of the candidate answer to the conversation by calculating the cosine similarity between the representation vector of the candidate answer and the representation vector of the contextual information.

2. The dialog model is trained based on the quality scores of candidate answers, having the highest quality score among the candidate answers and the quality score of randomly selected candidate answers. The method according to claim 1.

3. The step of training the aforementioned dialogue model is: A step of determining a reinforcement learning-based goal based on at least one of the first quality score or the second quality score, The steps include determining a cross-entropy-based target to evaluate the candidate answers based on token-level cross-entropy loss, A step of determining the final training goal based on the combination of the reinforcement learning-based goal and the cross-entropy-based goal, The method according to claim 1, including the method described in claim 1.

4. At least one memory that stores program code and a neural network-based open-domain dialogue model, A device comprising at least one processor that reads the program code and executes the program code, wherein the program code is The at least one processor is provided with a receiving code for receiving input utterances from a user's device that engages in conversation with the dialog model, Acquisition code for causing at least one processor to obtain a plurality of candidate answers for the input utterance from the dialog model, wherein the plurality of candidate answers comprises a first candidate answer and a second candidate answer. A first decision code for causing at least one processor to determine a plurality of first discriminator scores for the first candidate answer by a plurality of discriminators, wherein each of the plurality of discriminators evaluates the quality of the candidate answer based on one of different complementary factors and determines a discriminator score; The at least one processor is provided with a second decision code for causing the plurality of discriminators to determine a plurality of second discriminator scores for the second candidate answer, A third decision code for causing at least one processor to determine a first quality score associated with the first candidate answer, wherein the first quality score is based on a weighted sum of the plurality of first discriminator scores corresponding to the first candidate answer, A fourth decision code for causing at least one processor to determine a second quality score associated with the second candidate answer, wherein the second quality score is based on a weighted sum of the plurality of second discriminator scores corresponding to the second candidate answer, The at least one processor is provided with training code for training the dialog model based on at least one of the first quality score or the second quality score, A device comprising: a first discriminator that evaluates the fluency of a candidate answer by determining the perplexity of the candidate answer based on information corresponding to the candidate answer and contextual information corresponding to the history of the conversation; a second discriminator that evaluates the specificity of the candidate answer by determining the normalized inverse function document frequency of the candidate answer; a third discriminator that evaluates the consistency of the candidate answer by calculating the probability that the candidate answer is inconsistent with previous candidate answers output by the dialog model during the conversation; and a fourth discriminator that evaluates the relevance of the candidate answer to the conversation by calculating the cosine similarity between the representation vector of the candidate answer and the representation vector of the contextual information.

5. The plurality of discriminators evaluate the candidate answer based only on the information corresponding to the first candidate answer and the context information corresponding to the conversation history. The device according to claim 4.

6. The training code is further configured to cause the at least one processor to train the dialog model based on the quality scores of the candidate answers, which include the highest quality score among the candidate answers and the quality score of a randomly selected candidate answer. The device according to claim 4.

7. The training code is provided to at least one processor, Based on at least one of the first quality score or the second quality score, a reinforcement learning-based goal is determined. Determine a cross-entropy-based goal that evaluates the candidate answers based on token-level cross-entropy loss. Based on the combination of the reinforcement learning-based goal and the cross-entropy-based goal, the final training goal is determined. The device according to claim 4, further configured as follows.

8. A computer program that, when executed by one or more processors, causes the one or more processors to perform the method described in any one of claims 1 to 3.

Images