Electronic device for determining conversational comprehension and method for operating the same

Abstract

An electronic device for determining conversational understanding and a method of operation thereof are disclosed. The method of operation of the disclosed electronic device includes the operation of transmitting conversational data between a plurality of speakers and a predetermined query related to said conversational data to a conversational agent, the operation of receiving a response to said query from said conversational agent, the operation of comparing said response with a correct answer corresponding to said query, and the operation of determining the conversational understanding of said conversational agent based on said comparison result, wherein the conversational agent, in response to receiving said query, generates a response to said query based on said conversational data.

Description

Technology Field

[0001] The following disclosure relates to an electronic device for determining conversational understanding and a method of operation thereof. Background Technology

[0003] A conversational agent refers to a system or software implemented to interact with users via a messenger based on predefined response rules. To facilitate smooth conversation, conversational agents can utilize pattern recognition technology—which enables machines to identify speech and text based on artificial intelligence and big data analysis—natural language processing technology—which allows computers to recognize human language for question-answering and translation, semantic web technology—which enables computers to understand information and reason logically, text mining technology—which extracts useful information from text-based data, and context-aware computing technology—which grasps the situation and context of the conversation partner. Conversational agents can be deployed in systems across various fields, for example, in the form of chatbots or service agents. Methods for evaluating the performance of conversational agents are being researched to assess and optimize their conversational comprehension and response speed.

[0004] The background technology described above is possessed or acquired by the inventor in the process of deriving the content of the disclosure of the present application, and cannot necessarily be considered as prior art disclosed to the general public prior to the filing of this application. means of solving the problem

[0006] A method of operation of an electronic device according to one embodiment includes the operation of transmitting conversation data between a plurality of speakers and a predetermined query related to the conversation data to a conversation agent, the operation of receiving a response to the query from the conversation agent, the operation of comparing the response with a correct answer corresponding to the query, and the operation of determining the conversational understanding of the conversation agent based on the comparison result, wherein the conversation agent, in response to receiving the query, generates a response to the query based on the conversation data.

[0007] The operation of comparing the above response with the above correct answer can determine the similarity between the above response and the above correct answer, and if the similarity is greater than or equal to a predetermined value, the comparison result can be determined as a first value, and if the similarity is less than a predetermined value, the comparison result can be determined as a second value.

[0008] The operation of determining the above conversational understanding can determine the conversational understanding based on the comparison results determined by repeatedly performing the operation of transmitting the above query, the operation of receiving the above response, and the operation of comparing the above response with the above correct answer.

[0009] The operation of determining the conversational understanding can determine the conversational understanding based on the ratio of the first value and the second value among the comparison results.

[0010] The operation for determining the conversation understanding above may determine the comparison result as the second value if the response is not received within a predetermined time.

[0011] The operation of transmitting the above conversation data transmits the above conversation data sequentially to the conversation agent according to the flow of the conversation, and the conversation agent can generate a response to the query based on the conversation data received up to the point in time when the query is received.

[0012] The conversation data stored in the conversation agent, which is utilized when the conversation agent performs a conversation with the plurality of speakers, is updated based on the conversation performed by the conversation agent with the plurality of speakers, and the conversation agent can perform a subsequent conversation by utilizing the updated conversation data.

[0013] If the update to the above conversation data is not performed within a predetermined time, the update to the above conversation data may be skipped.

[0014] The above conversation data includes a predetermined number of tokens or more, and the conversation agent can generate the response by determining the conversation context based on the tokens.

[0015] The operation of transmitting the above query may transmit any one of a predetermined set of queries at a random time during the conversation.

[0016] The operation of comparing the above response with the above correct answer may compare the above response with a correct answer determined differently for each of the plurality of speakers.

[0017] An electronic device according to one embodiment includes a processor and a memory for storing instructions, and when the instructions are executed by the processor, the electronic device transmits conversation data between a plurality of speakers and a predetermined query related to the conversation data to a conversation agent, receives a response to the query from the conversation agent, compares the response with a correct answer corresponding to the query, and determines the conversational understanding of the conversation agent based on the comparison result, and the conversation agent can generate a response to the query based on the conversation data in response to receiving the query. Brief explanation of the drawing

[0019] FIG. 1 is a drawing for explaining an electronic device according to one embodiment. FIG. 2 is a diagram illustrating the process of determining the conversational understanding of a conversation agent according to one embodiment. FIGS. 3 and 4 are drawings for explaining operations between an electronic device and a conversation agent according to one embodiment. FIG. 5 is a diagram illustrating a conversation agent according to one embodiment. FIG. 6 is a diagram illustrating an example of an operation for determining conversational understanding according to one embodiment. FIG. 7 is a flowchart illustrating the operation method of an electronic device according to one embodiment. Specific details for implementing the invention

[0020] Specific structural or functional descriptions of the embodiments are disclosed for illustrative purposes only and may be modified and implemented in various forms. Accordingly, actual implementations are not limited to the specific embodiments disclosed, and the scope of this specification includes modifications, equivalents, or substitutions included in the technical concept described by the embodiments.

[0021] In this document, each of the following phrases may include any one of the items listed together in the corresponding phrase, or any combination of A, B, and C, or any combination of all of them. Terms such as "A or B," "at least one of A and B," "at least one of A, B, and C," "at least one of A, B, or C," and "a combination of one or more of A, B, and C" may be used to describe various components, but these terms should be interpreted solely for the purpose of distinguishing one component from another. For example, the first component may be named the second component, and similarly, the second component may also be named the first component.

[0022] When it is stated that a component is "connected" to another component, it should be understood that it may be directly connected to or joined to that other component, or that there may be other components in between.

[0023] The singular expression includes the plural expression unless the context clearly indicates otherwise. In this specification, terms such as "comprising" or "having" are intended to specify the existence of the described features, numbers, steps, actions, components, parts, or combinations thereof, and should be understood as not precluding the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.

[0024] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as generally understood by those skilled in the art. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant technology, and should not be interpreted in an ideal or overly formal sense unless explicitly defined in this specification.

[0025] Hereinafter, embodiments will be described in detail with reference to the attached drawings. In the description with reference to the attached drawings, identical components are given the same reference numeral regardless of the drawing number, and redundant descriptions thereof will be omitted.

[0027] FIG. 1 is a drawing for explaining an electronic device according to one embodiment.

[0028] Referring to FIG. 1, an electronic device (110) may include a processor (111) and a memory (112). The processor (111) may include at least one processor. The memory (112) may store instructions (e.g., programs) executable by the processor (111). For example, the instructions may include instructions for executing the operation of the processor (111) and / or the operation of each component of the processor (111). The processor (111) may include various processors, such as a CPU (Central Processing Unit) or a GPU (Graphic Processing Unit), as a device for executing instructions or programs or controlling the electronic device (110).

[0029] The electronic device (110) is a device for evaluating the conversational understanding of a conversational agent (120) and may include, but is not limited to, various computing devices such as mobile phones, smartphones, tablets, e-book devices, laptops, personal computers, desktops, workstations, or servers; various wearable devices such as smart watches, smart glasses, HMDs (Head-Mounted Displays), or smart clothing; various home appliances such as smart speakers, smart TVs, or smart refrigerators; smart cars, smart kiosks, IoT (Internet of Things) devices, WADs (Walking Assist Devices), drones, or robots. For convenience of explanation in this specification, the electronic device (110) may be referred to as a conversational simulation device or a conversational agent evaluation system.

[0030] The electronic device (110) can transmit and receive data by communicating with the conversation agent (120) via wired or wireless connection. The electronic device (110) can transmit conversation data and a predetermined query related to the conversation data to the conversation agent (120). The electronic device (110) can receive a response to the transmitted query from the conversation agent (120). In FIG. 1, only one conversation agent (120) is shown for illustrative purposes, but the embodiment is not limited thereto, and the number of conversation agents communicating with the electronic device (110) may be multiple, and the electronic device (110) can determine the conversation understanding of the multiple conversation agents.

[0031] According to an embodiment, the electronic device (110) may include a conversation agent (120). In this case, the conversation agent (120) can generate a response to a query using conversation data stored in memory (112). Additionally, the electronic device (110) can evaluate the conversational understanding of a specific conversation agent (120) included within the electronic device (110).

[0032] The conversation agent (120) may represent a device that generates natural language responses or a device including a processor that generates natural language responses. The conversation agent (120) may include an artificial intelligence (AI) model. The conversation agent (120) may generate a response to a query based on conversation data in response to receiving a query. The conversation agent (120) may transmit the generated response to an electronic device (110). The conversation agent (120) may store conversation data and update the conversation data based on conversations performed with multiple speakers according to the flow of conversation. The conversation agent (120) may understand the conversation context between multiple speakers included in the conversation data and generate a response to a random query received at a random time.

[0033] According to one embodiment, the electronic device (110) can determine the conversational understanding of the conversational agent (120) based on the received response. Conversational understanding may indicate the ability of the conversational agent (120) to understand a long-term conversation included in the conversation data and to respond appropriately based on the conversation. In one embodiment, the electronic device (110) can compare the received response with the correct answer corresponding to the query and determine the conversational understanding of the conversational agent (120) based on the comparison result. The electronic device (110) can determine the conversational understanding of the conversational agent (120) based on the comparison results determined by repeatedly performing the operation of transmitting a query, the operation of receiving a response, and the operation of comparing the response with the correct answer for a predetermined time. By determining the conversational understanding of the conversational agent (120), the electronic device (110) can evaluate whether the conversational agent (120) can generate an appropriate response according to the conversation within an appropriate time. Through this, the electronic device (110) can help increase the performance, reliability, and efficiency of the conversation agent (120) and help select a conversation agent suitable for the service to be provided.

[0034] The operation of the electronic device (110) evaluating the conversational understanding of the conversation agent (120) is described in detail below through FIGS. 2 to 4.

[0036] FIG. 2 is a diagram illustrating the process of determining the conversational understanding of a conversation agent according to one embodiment.

[0037] Referring to FIG. 2, in the following embodiments, each operation may be performed sequentially, but is not necessarily performed sequentially. For example, the order of each operation may be changed, and at least two operations may be performed in parallel. The operations illustrated in FIG. 2 may be performed by at least one component of an electronic device (e.g., a processor, etc.) or a conversational agent.

[0038] In operation (210), a conversation between multiple speakers may take place. Each of the multiple speakers transmits conversation data to an electronic device via a user terminal (e.g., computer, mobile phone), and the electronic device transmits the received conversation data to other speakers, thereby allowing the conversation to proceed. The electronic device may provide one or more sessions in which the conversation takes place, and in each session, a conversation may proceed according to different conversation data. For example, each session may include conversation data in which the conversation was performed at different times. Additionally, each session may correspond to a predetermined set of queries and a set of correct answers corresponding to the set of queries. The electronic device may transmit conversation data to a conversation agent. For example, the electronic device may transmit conversation data corresponding to a simulated session to a conversation agent. The electronic device may transmit conversation data to a conversation agent sequentially according to the flow of the conversation.

[0039] In operation (211), the conversation agent can update conversation data within a predetermined time limit (e.g., 1 second, 3 seconds). In one embodiment, the conversation agent can store conversation data in memory and update conversation data in response to receiving conversation data from an electronic device. The conversation data may be updated based on conversations performed by the conversation agent with multiple speakers.

[0040] In operation (212), if the conversation agent succeeds in updating the conversation data within a predetermined time limit, it can use the updated conversation data to generate a response to a query and perform a subsequent conversation regarding the received conversation data. The electronic device can generate a response to a query based on the conversation data received up to the point of receiving the query. The conversation agent can transmit to the electronic device whether the update of the conversation data was successful. By successfully updating the conversation data, the conversation agent can generate a more accurate and appropriate response to the query.

[0041] In operation (213), if the conversation agent fails to update the conversation data within a predetermined time limit, it can use the stored conversation data to generate a response to the query and perform a subsequent conversation regarding the received conversation data. If the update to the conversation data is not performed within the predetermined time limit, the update to the conversation data may be skipped. Afterwards, the next utterance may proceed.

[0042] In operation (220), the electronic device may transmit a predetermined query related to conversation data to the conversation agent. The electronic device may transmit any one of the predetermined sets of queries related to conversation data to the conversation agent at a random time during the conversation. For example, the electronic device may transmit any one of the randomly determined queries from the predetermined sets of queries corresponding to the simulated session to the conversation agent. Additionally, the predetermined query may be any one of multiple-choice queries, open-ended queries, and descriptive queries. For each conversation session, the electronic device may transmit a randomly determined query to the conversation agent at a random time, which is determined by a randomly determined queryer.

[0043] In operation (230), the electronic device may receive a response to a query from a conversation agent within a predetermined time limit (e.g., 1 second, 3 seconds, 5 seconds). If the electronic device does not receive a response within the predetermined time limit, it may skip receiving the corresponding response from the conversation agent.

[0044] In operation (240), the electronic device can determine whether the received response corresponds to the correct answer corresponding to the question. The electronic device can determine the comparison result by comparing the response with the correct answer corresponding to the question. For example, a multiple-choice question may correspond to the correct answer which is one of the multiple-choice options, and a subjective question and a descriptive question may correspond to any one word, phrase, or sentence. Additionally, the electronic device can compare the response with the correct answer determined differently for each of multiple speakers. The correct answer corresponding to the question may vary depending on the speaker who asked the question.

[0045] In one embodiment, the electronic device determines the similarity between a response and a correct answer, and if the similarity is greater than or equal to a predetermined value, determines the comparison result as a first value (e.g., "1", "success"), and if the similarity is less than a predetermined value, determines the comparison result as a second value (e.g., "0", "failure"). In the case of a subjective or descriptive question, the electronic device may determine the similarity using a large language model. According to an embodiment, the electronic device may determine the similarity between a response and a correct answer as the comparison result.

[0046] In operation (241), if the electronic device determines that the received response corresponds to the correct answer, it may determine that the conversation agent has succeeded in the response. The electronic device may determine the comparison result for the response as the first value.

[0047] In operation (242), if the electronic device determines that the received response does not correspond to the correct answer, it may determine that the conversation agent has failed the response. Additionally, if the response is not received within a predetermined time limit, the conversation agent may determine that the response has failed. The electronic device may determine the comparison result for the response as a second value.

[0048] After operation (241) or operation (242), the electronic device may continue to transmit conversation data and queries to the agent. For example, the electronic device may determine comparison results by repeatedly performing the operation of transmitting queries (220), the operation of receiving responses (230), and the operation of comparing the responses with the correct answer (240). The electronic device may determine the conversational understanding of the conversational agent based on the ratio of the number of first values ​​and second values ​​among the comparison results. Alternatively, the electronic device may determine the conversational understanding of the conversational agent based on the similarities determined by the comparison results. For example, the electronic device may determine the conversational understanding of the conversational agent using the average or distribution of the similarities.

[0050] FIGS. 3 and 4 are drawings for explaining operations between an electronic device and a conversation agent according to one embodiment.

[0051] Referring to FIG. 3, the electronic device (320) can determine the conversational understanding (330) of the conversational agent (310) through random queries, queries requiring understanding of long-term conversations, and responses within a time limit. The electronic device (320) can determine the conversational understanding (330) by evaluating the conversational agent (310)'s real-time interaction, understanding of multi-party conversations, and long-term context dependency.

[0052] A conversation agent (310) may take on a specific role in a conversation. An electronic device (320) can simulate the role of the conversation agent (310) by transmitting conversation data regarding various situations that may be encountered during the conversation. During the simulation, the electronic device (320) transmits a random query to the conversation agent (310), and the conversation agent (310) may need to respond accurately to the query. Through this, the electronic device (320) can evaluate the conversation agent (310)'s ability to respond appropriately to the conversation in real time.

[0053] A time limit for receiving a response may be set to evaluate whether the conversation agent (310) can respond appropriately in real time. The time limit may be determined by the user of the electronic device who wishes to evaluate the conversation agent (310), or by the developer or user of the conversation agent (310). The electronic device (320) can evaluate the real-time response capability of the conversation agent (310) by simulating a situation requiring an immediate response during a conversation through the time limit. Through this, the electronic device (320) can quantitatively evaluate the accuracy and response speed of the conversation agent (310) to determine the conversation comprehension (330). The conversation comprehension (330) of the conversation agent (310) may be determined differently depending on whether a time limit is set or not. For example, if a time limit is set, the conversation comprehension may be determined to be high because the smaller the size of the conversation agent (310)'s model, the faster the inference speed. If a time limit is not set, the conversation comprehension may be determined to be high because the larger the size of the conversation agent (310)'s model, the higher the inference performance. The size of the model of the conversation agent (310) can be selected by considering the balance between inference time and inference ability for performing real-time conversation.

[0054] The electronic device (320) can evaluate the conversational understanding (330) of the conversation agent (310) using multi-party long-term conversational data. For example, the conversational data may contain a predetermined number (e.g., hundreds of thousands) or more of tokens. A token may represent a basic unit used by the conversation agent (310) to process and understand the conversational data. For example, a token may be determined as four alphabets or one syllable, but the embodiments are not limited thereto. The electronic device (320) may transmit a query requiring the understanding of a long-term conversation to the conversation agent (310). The conversation agent (310) may determine the conversational context based on the tokens and generate a response. Through this, the electronic device (320) can evaluate whether the conversation agent (310) can store conversational data for multiple conversational sessions and generate an appropriate response based on the conversational data. The electronic device (320) can simultaneously perform the long-term conversational understanding and real-time response evaluation of the conversation agent (310).

[0055] Additionally, the electronic device (320) can randomly determine one of a predetermined set of queries and transmit the determined query to the conversation agent (310), thereby evaluating whether the conversation agent (310) can consistently maintain high performance for the randomly given query.

[0056] Referring to FIG. 4, an exemplary flow is illustrated in which a conversation agent (410) generates a response to a query as the conversation proceeds in operation (420).

[0057] In operation (420), the electronic device can perform operations (421) to (424) on the conversation agent (410) while the conversation is in progress.

[0058] In operation (421), the electronic device can send a query to the conversation agent (410) at a randomly determined time while the conversation is in progress.

[0059] In operation (422), the electronic device can transmit a query to a conversation agent (410) for a questioner randomly selected from among a plurality of speakers. The conversation agent (410) can transmit different responses to the electronic device depending on the questioner. The electronic device can compare the correct answer and the response corresponding to the query and the questioner.

[0060] In operation (423), the electronic device can send a query requiring understanding of a long conversation to a conversation agent.

[0061] In operation (424), the electronic device can determine whether it has received a response from the conversation agent within a time limit.

[0062] In operation (411), the conversation agent (410) can update conversation data stored in memory. Based on the updated conversation data, the conversation agent (410) can generate a response and proceed with a subsequent conversation.

[0063] In operation (430), the electronic device can determine the conversational understanding of the conversation agent (410) based on the responses received as the conversation progresses.

[0065] FIG. 5 is a diagram illustrating a conversation agent according to one embodiment.

[0066] Referring to FIG. 5, the conversation agent (500) includes a model (510) and can store conversation data (520) in memory. The conversation data (520) can be stored in memory inside the conversation agent (500) or in external memory.

[0067] A conversational agent (500) can generate a response to a query using a pre-trained model (510). The method by which the conversational agent (500) generates a response to a query based on conversational data (520) may vary depending on the embodiment. The model (510) may be a model trained on other natural language data as well as conversational data (520). For example, the conversational agent (500) can generate a response to a query based on a large language model (LLM). A large language model is implemented as an artificial neural network including multiple parameters and multiple layers, and can be trained by various learning methods (e.g., supervised learning, unsupervised learning). The parameters and weights of the large language model may be determined according to the learning, or may be quantized or pruned for optimization. A large language model can process, understand, and generate natural language using pre-trained data, data on a connected network, and stored conversational data (520). For example, a conversational agent (500) can generate a response to a query based on conversational data (520) based on a pre-trained large language model. Additionally, according to an embodiment, the conversational agent (500) can generate a response to a query using a natural language processing (NLP) model, a natural language understanding (NLU) model, or a natural language generation (NLG) model.

[0068] The conversation agent (500) can store and update conversation data (520) for the model (510) to reference. The storage and updating of conversation data (520) may affect the conversational understanding of the conversation agent (500). In one embodiment, the conversation agent (500) can store and update conversation data (520) in memory. For example, the conversation agent (500) can update conversation data (520) based on retrieval-augmented generation (RAG) or update conversation data (520) by storing it in a conversation context, but the embodiments are not limited thereto.

[0069] In the case of a RAG-based memory storage method, the conversation agent (500) can efficiently manage and search conversation data (520) by utilizing external memory. In the case of a RAG-based memory storage method, for example, the conversation agent (500) can store conversation data (520) in a manner such as session-unit storage, utterance-unit storage, or conversation session summary storage. The session-unit storage method stores an entire conversation session, making it suitable for searching for a received query when a query regarding a specific session is received. The utterance-unit storage method stores each utterance individually, enabling more detailed searching. The summary storage method compresses and stores key information of the conversation, thereby enabling a reduction in the size of conversation data (520) and an improvement in search speed.

[0070] For example, memory search by the conversation agent (500) can be performed by utilizing a search algorithm (e.g., BM25) and / or a vector database. When using BM25 as the search algorithm, the conversation agent (500) can search for highly relevant information based on keyword matching between text and queries. BM25 can be useful for memory search when the query includes specific keywords. When using a vector database, the conversation agent (500) can perform memory search based on semantic similarity by storing conversation data (520) in an embedding form. A vector database can be useful for effectively searching for relevant information even when the query is not specific.

[0071] In the case of a method of storing the conversation context, the conversation agent (500) can internally update the conversation context and maintain the conversation context. The conversation agent (500) can store the conversation context without searching external memory and generate a response based on the stored conversation context. In this case, the conversation data (520) may include the conversation context. The method of storing the conversation context is advantageous in situations where there is a time limit and can enable the continuity of the conversation and the maintenance of a natural context.

[0072] In the simulation, depending on the situation, the conversation agent (500) can update the conversation data (520) by selecting an appropriate memory update method, and the electronic device can determine the conversation comprehension level by evaluating the conversation agent (500)'s real-time response ability, understanding of long conversations, and reasoning ability. Through this, the electronic device can comprehensively measure how effectively the conversational agent (500) can operate in various environments.

[0074] FIG. 6 is a diagram illustrating an example of an operation for determining conversational understanding according to one embodiment.

[0075] Referring to FIG. 6, queries (611, 621, 631) received by a conversation agent and responses (612, 622, 632) generated by the conversation agent in simulations (610, 620, 630) performed by an electronic device are illustrated as examples. The simulations (610, 620, 630) illustrated in FIG. 6 are exemplary for illustrative purposes and are not limited thereto; various queries may be determined depending on the situation, and various responses may be generated in response to the queries.

[0076] In the simulation (610), the electronic device may send a multiple-choice question (611) to the conversation agent that has no corresponding correct answer. The conversation agent may generate a response (612) in response to receiving the question (611). The electronic device may compare the conversation agent's response (612) with the correct answer. If there is no correct answer corresponding to the question (611), the electronic device may determine whether the conversation agent's response (612) indicates that there is no correct answer (e.g., "I don't know", "Unanswerable").

[0077] In the simulation (620), the electronic device may send a query (621) to a conversation agent that requires reference to time information (e.g., timestamp). In response to receiving the query (621), the conversation agent may generate a response (622) by referencing the time information. If the response (622) is not received from the conversation agent within a time limit, the electronic device may determine that the conversation agent failed to respond to the query (621). In this case, the electronic device may determine that the response failed without needing to compare the response (622) with the correct answer.

[0078] In the simulation (630), the electronic device may send a subjective-type query (631) to the conversation agent that requires understanding of the conversation of another session (e.g., a previous session). A set of queries corresponding to the session being simulated may include queries that require understanding of the conversation of the previous session. For example, the set of queries may include queries that require a response that must refer to information in the conversation data of the previous session. For convenience of explanation in this specification, a query that requires understanding of the conversation of another session may be referred to as a multi-hop query. In response to receiving the query (631), the conversation agent may generate a response (632) by referring to other sessions. The electronic device may compare the conversation agent's response (632) with the correct answer.

[0079] The electronic device can determine the conversational agent's understanding of the conversation based on the conversational agent's responses (612, 622, 632).

[0081] FIG. 7 is a flowchart illustrating the operation method of an electronic device according to one embodiment.

[0082] In the following embodiments, each operation may be performed sequentially, but is not necessarily performed sequentially. For example, the order of each operation may be changed, and at least two operations may be performed in parallel. Operations (710) to (740) may be performed by at least one component of an electronic device (e.g., a processor, etc.).

[0083] In operation (710), the electronic device can transmit conversation data between multiple speakers and a predetermined query related to the conversation data to a conversation agent. The electronic device can transmit the conversation data to the conversation agent sequentially according to the flow of the conversation. The electronic device can transmit any one of the predetermined sets of queries at a random time while the conversation is in progress.

[0084] A conversational agent can generate a response to a query based on conversational data in response to a received query. The conversational agent can generate a response to a query based on conversational data received up to the point in time when the query is received. The conversational data stored in the conversational agent, which is utilized when the conversational agent engages in conversations with multiple speakers, can be updated based on the conversations conducted by the conversational agent with multiple speakers. The conversational agent can perform subsequent conversations using the updated conversational data. If the update to the conversational data is not performed within a predetermined time, the update to the conversational data may be skipped. The conversational data includes a predetermined number of tokens, and the conversational agent can generate a response by determining the conversational context based on the tokens.

[0085] In operation (720), the electronic device can receive a response to a query from a conversation agent.

[0086] In operation (730), the electronic device can compare the response with the correct answer corresponding to the question. The electronic device can determine the similarity between the question and the correct answer, and if the similarity is greater than or equal to a predetermined value, determine the comparison result as a first value, and if the similarity is less than a predetermined value, determine the comparison result as a second value. The electronic device can compare the response with the correct answer determined differently for each of the multiple speakers.

[0087] In operation (740), the electronic device can determine the conversational understanding of the conversational agent based on comparison results. The electronic device can determine the conversational understanding based on comparison results determined by repeatedly performing the operation of transmitting a query, the operation of receiving a response, and the operation of comparing the response with the correct answer. The electronic device can determine the conversational understanding based on the ratio of a first value and a second value among the comparison results. If a response is not received within a predetermined time, the electronic device can determine the comparison result as the second value.

[0088] Since the details described in FIG. 1 to 6 apply to each operation illustrated in FIG. 7, a more detailed description is omitted.

[0090] The embodiments described above may be implemented as hardware components, software components, and / or combinations of hardware and software components. For example, the devices, methods, and components described in the embodiments may be implemented using a general-purpose computer or a special-purpose computer, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. The processing unit may execute an operating system (OS) and software applications executed on said operating system. Additionally, the processing unit may access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing unit may be described as being used as a single unit, but those skilled in the art will understand that the processing unit may include multiple processing elements and / or multiple types of processing elements. For example, the processing unit may include multiple processors or one processor and one controller. In addition, other processing configurations, such as parallel processors, are also possible.

[0091] Software may include computer programs, code, instructions, or a combination of one or more of these, and may configure a processing unit to operate as desired or instruct the processing unit independently or collectively. Software and / or data may be stored on any type of machine, component, physical device, virtual equipment, computer storage medium, or device so as to be interpreted by the processing unit or to provide instructions or data to the processing unit. Software may be distributed over networked computer systems and stored or executed in a distributed manner. Software and data may be stored on computer-readable recording media.

[0092] The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may store program instructions, data files, data structures, etc., either individually or in combination, and the program instructions recorded on the medium may be those specifically designed and configured for the embodiment or those known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, and flash memory. Examples of program instructions include machine code, such as that generated by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc.

[0093] The hardware device described above may be configured to operate as one or more software modules to perform the operation of the embodiment, and vice versa.

[0094] Although the embodiments have been described above with reference to the limited drawings, those skilled in the art can apply various technical modifications and variations based thereon. For example, suitable results may be achieved even if the described techniques are performed in a different order than described, and / or if the components of the described system, structure, device, circuit, etc. are combined or assembled in a form different from described, or replaced or substituted by other components or equivalents.

[0095] Therefore, other implementations, other embodiments, and equivalents to the claims also fall within the scope of the claims set forth below.

Claims

Claim 1 A method of operation of an electronic device comprising: transmitting conversation data between multiple speakers and a predetermined query related to said conversation data to a conversation agent; receiving a response to said query from said conversation agent; comparing said response with a correct answer corresponding to said query; and determining the conversational understanding of said conversation agent based on said comparison result, wherein the conversation agent, in response to receiving said query, generates a response to said query based on said conversation data. Claim 2 A method of operation of an electronic device according to claim 1, wherein the operation of comparing the response with the correct answer determines the similarity between the response and the correct answer, and if the similarity is greater than or equal to a predetermined value, determines the comparison result as a first value, and if the similarity is less than a predetermined value, determines the comparison result as a second value. Claim 3 A method of operation of an electronic device according to paragraph 2, wherein the operation of determining the conversational understanding is determined based on comparison results obtained by repeatedly performing the operation of transmitting the query, the operation of receiving the response, and the operation of comparing the response with the correct answer. Claim 4 In paragraph 3, the operation of determining the conversational understanding is a method of operation of an electronic device that determines the conversational understanding based on the ratio of the first value and the second value among the comparison results. Claim 5 A method of operation of an electronic device according to paragraph 2, wherein the operation of determining the conversation understanding determines the comparison result as the second value when the response is not received within a predetermined time. Claim 6 A method of operation of an electronic device according to claim 1, wherein the operation of transmitting the conversation data transmits the conversation data sequentially to a conversation agent according to the flow of the conversation, and the conversation agent generates a response to the query based on the conversation data received up to the point in time when the query is received. Claim 7 A method of operation of an electronic device according to claim 1, wherein conversation data stored in the conversation agent, which is utilized when the conversation agent performs a conversation with the plurality of speakers, is updated based on the conversation performed by the conversation agent with the plurality of speakers, and the conversation agent performs a subsequent conversation using the updated conversation data. Claim 8 A method of operation of an electronic device according to claim 7, wherein if the update to the conversation data is not performed within a predetermined time, the update to the conversation data is skipped. Claim 9 A method of operation of an electronic device according to claim 1, wherein the conversation data includes a predetermined number of tokens or more, and the conversation agent determines a conversation context based on the tokens and generates the response. Claim 10 A method of operation of an electronic device according to claim 1, wherein the operation of transmitting the above query is to transmit any one of a predetermined set of queries at a random time during the course of a conversation. Claim 11 A method of operation of an electronic device according to claim 1, wherein the operation of comparing the response with the correct answer is to compare the response with a correct answer determined differently for each of the plurality of speakers. Claim 12 An electronic device comprising: a processor; and a memory for storing instructions, wherein when the instructions are executed by the processor, the electronic device transmits conversation data between multiple speakers and a predetermined query related to the conversation data to a conversation agent, receives a response to the query from the conversation agent, compares the response with a correct answer corresponding to the query, and determines the conversational understanding of the conversation agent based on the comparison result, and the conversation agent generates a response to the query based on the conversation data in response to receiving the query. Claim 13 An electronic device according to claim 12, wherein when the instructions are executed by the processor, the electronic device determines the similarity between the response and the correct answer, and if the similarity is greater than or equal to a predetermined value, determines the comparison result as a first value, and if the similarity is less than a predetermined value, determines the comparison result as a second value. Claim 14 An electronic device according to claim 13, wherein when the instructions are executed by the processor, the electronic device causes the electronic device to repeatedly perform the operation of transmitting the query, the operation of receiving the response, and the operation of comparing the response with the correct answer, and determine the conversational understanding based on the determined comparison results. Claim 15 An electronic device according to claim 14, wherein when the instructions are executed by the processor, the electronic device causes the conversation understanding to be determined based on the ratio of the first value and the second value among the comparison results. Claim 16 An electronic device according to claim 13, wherein when the instructions are executed by the processor, the electronic device causes the comparison result to be determined as the second value if the response is not received within a predetermined time. Claim 17 In paragraph 12, the electronic device, when the instructions are executed by the processor, causes the electronic device to sequentially transmit the conversation data to a conversation agent according to the flow of the conversation, and the conversation agent generates a response to the query based on the conversation data received up to the point of receiving the query. Claim 18 An electronic device according to claim 12, wherein conversation data stored in the conversation agent, which is utilized when the conversation agent performs a conversation with the plurality of speakers, is updated based on the conversation performed by the conversation agent with the plurality of speakers, and the conversation agent performs a subsequent conversation using the updated conversation data. Claim 19 An electronic device according to claim 18, wherein if the update to the above conversation data is not performed within a predetermined time, the update to the above conversation data is skipped. Claim 20 A computer-readable recording medium storing a computer program that executes the method of any one of paragraphs 12 through 19.

Images