Electronic device, method, and non-transitory computer-readable storage medium for training artificial intelligence model for responding to query

Abstract

According to an embodiment, a method performed by an electronic device for training a first model configured to generate a response corresponding to a query may comprise the operations of: searching for a designated number of second queries appearing to be similar to a first query among a plurality of queries by using a second model; calculating a similarity between the first query and each of the second queries by using a third model; adjusting the similarity on the basis of whether each of the second queries is included in a group including the first query among a plurality of groups; and performing training on the first model by using pair data including the adjusted similarity.

Description

Electronic device, method, and non-transitory computer-readable storage medium for training an artificial intelligence model to answer questions

[0001] The present disclosure relates to an electronic device, a method, and a non-transitory computer-readable storage medium for training an artificial intelligence model to answer questions.

[0002] Artificial intelligence (AI) is a technology for simulating human (or biological) neural activity, such as perception and / or inference. It can be implemented using hardware, software, or any combination thereof designed to perform computations for simulating neural activity. Based on AI, AI models for answering questions, such as chatbot services, can be provided.

[0003] The above information may be provided as background art to aid in understanding the present disclosure. No claim or determination is made as to whether any of the above-described matters constitute prior art related to the present disclosure.

[0004] A method for training a model is disclosed. In one embodiment, the method, performed by an electronic device for training a first model, configured to generate a response corresponding to a query, may include an operation of searching for a second question among a plurality of questions that appears similar to the first question using a second model. The method may include an operation of calculating a similarity between the second question and the first question using a third model. The method may include an operation of adjusting the similarity based on whether the second question is included in a group among a plurality of groups in which the first question is included. The method may include an operation of performing training on the first model using pair data including the adjusted similarity.

[0005] A non-transitory computer-readable storage medium is disclosed. In one embodiment, a non-transitory computer-readable storage medium comprising instructions, wherein the instructions, when executed by an electronic device, may be configured to cause the electronic device to retrieve, from a database comprising a plurality of questions, a specified number of second questions that appear similar to a first question among the plurality of questions, using a second model. The database may include information for classifying each of the plurality of questions into one of a plurality of groups. The instructions, when executed by the electronic device, may be configured to cause the electronic device to determine whether each of the second questions is included in a group among the plurality of groups in which the first question is included. The instructions, when executed by the electronic device, may be configured to cause the electronic device to generate pair data based on the determination. The above pair data may include a similarity between the third question and the first question based on a function determined from among the functions specified according to whether the third question corresponding to the decision is included in the group.

[0006] The above instructions, when executed by the electronic device, may be configured to cause the electronic device to perform training on the first model using the pair data.

[0007] An electronic device is disclosed. According to one embodiment, the electronic device may include a memory storing instructions and at least one processor. The instructions, when executed by the at least one processor, may be configured to retrieve, from a database including a plurality of questions, a specified number of second questions that appear similar to a first question among the plurality of questions, using a second model, to train a first model configured to generate a response corresponding to the question. The database may include information for classifying each of the plurality of questions into one of a plurality of groups. The instructions, when executed by the at least one processor, may be configured to determine whether each of the second questions is included in a group among the plurality of groups in which the first question is included, in order to train the first model. The instructions, when executed by the at least one processor, may be configured to generate pair data based on the determination, in order to train the first model. The pair data may include a similarity between the third question and the first question based on a function determined from among designated functions depending on whether the third question corresponding to the decision is included in the group. The instructions, when executed by the at least one processor, may be configured to perform training on the first model using the pair data to train the first model.

[0008] A method of an electronic device is disclosed. According to one embodiment, the method of the electronic device may include an operation of receiving a first question from an external electronic device. The method may include an operation of executing a first model using the received first question to obtain a response to the first question. The method may include an operation of transmitting the obtained response to the external electronic device. The first model may be trained using pair data. The pair data may include a first question, a second question, and a similarity between the first question and the second question based on a function determined from among functions designated according to whether the first question and the second question are included in the same group among groups for classifying similar questions.

[0009] Figures 1a, 1b, and 1c illustrate exemplary operations of an electronic device in relation to responding to a question received from an external electronic device.

[0010] Figure 2 is an exemplary diagram illustrating operations for training an artificial intelligence model to answer questions.

[0011] Figure 3 is an exemplary diagram illustrating operations for training an artificial intelligence model to answer questions.

[0012] Figure 4 is a graph illustrating exemplary functions for training an artificial intelligence model to answer questions.

[0013] Figure 5 is a flowchart illustrating operations for training an artificial intelligence model to answer questions.

[0014] Figure 6 is a block diagram of an exemplary electronic device.

[0015] Hereinafter, various embodiments of this document are described with reference to the attached drawings.

[0016] The various embodiments of this document and the terminology used therein are not intended to limit the technology described in this document to a specific embodiment, but should be understood to include various modifications, equivalents, and / or substitutes of the embodiment. In connection with the description of the drawings, similar reference numerals may be used for similar components. The singular expression may include plural expressions unless the context clearly indicates otherwise. In this document, expressions such as "A or B", "at least one of A and / or B", "A, B, or C", or "at least one of A, B, and / or C" may include all possible combinations of the items listed together. Expressions such as "first", "second", "first", or "second" may modify the corresponding components regardless of order or importance, and are only used to distinguish one component from another, but do not limit the corresponding components. When it is said that a component (e.g., a first component) is “(functionally or communicatively) connected” or “connected” to another component (e.g., a second component), said component may be directly connected to said other component, or may be connected via another component (e.g., a third component).

[0017] The term "module" as used in this document includes a unit composed of hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be an integral component, or a minimal unit or portion thereof that performs one or more functions. For example, a module may be composed of an application-specific integrated circuit (ASIC).

[0018] Figures 1a, 1b, and 1c illustrate exemplary operations of an electronic device in relation to responding to a question received from an external electronic device.

[0019] Referring to FIGS. 1A, 1B, and 1C, the electronic device (101) may include a server of a service provider. The server may include a computer designed to provide server functionality, such as a workstation. Embodiments are not limited thereto, and the server may include a combination of a personal computer and a software application installed on the personal computer. The server may include a group (or cluster) of the example workstations and / or personal computers. The server may be configured by the service provider to execute a software application for providing the service over a network, such as the Internet.

[0020] According to one embodiment, the service provided by the electronic device (101) may include a conversational interaction service (e.g., a chatbot service). The electronic device (101) may store question information received from subscribers to the conversational interaction service. For example, the electronic device (101) may be a server configured to manage the conversational interaction service.

[0021] According to one embodiment, a processor of an electronic device (101) (e.g., processor (610) of FIG. 6) may execute or utilize artificial intelligence provided as at least one neural network (e.g., first model (210) of FIG. 2, neural network (625) of FIG. 6) to execute a function related to natural language. The function may include a function of generating or transmitting at least one natural language sentence expressing information related to one or more natural language sentences (e.g., one or more natural language sentences received from a user of an external electronic device (102). The function may include a function of executing or calling another function supported by the electronic device (101) and expressed by one or more natural language sentences. For example, the external electronic device (102) may execute an application for providing a conversational interaction service provided through the electronic device (101) based on a user input to the external electronic device (102). The external electronic device (102) may be referred to as, for example, a user terminal. The external electronic device (102) may include a smartphone, a smartpad, and / or a tablet PC. The embodiment is not limited thereto, and the external electronic device may include a smart accessory such as a smartwatch and / or a head-mounted device (HMD). An exemplary hardware configuration of the electronic device (101) and / or the external electronic device (102) is exemplarily described with reference to FIG. 6.

[0022] Referring to FIGS. 1A to 1C , exemplary states of an electronic device (101) and an external electronic device (102) connected to each other to provide a service for interactive interaction are illustrated. The processor (640) of the external electronic device (102) of FIG. 6 may execute an application for an interactive interaction service provided by the electronic device (101) based on a user input. While the application is executing, the external electronic device (102) may display the screen (110) of FIGS. 1A to 1C . The application may include a software application executable by the external electronic device (102). The embodiment is not limited thereto, and the external electronic device (102) may display the screen (110) by displaying a web page provided from the electronic device (101) through a network. The screen (110) may occupy at least a portion of a display (e.g., a display module (660) of FIG. 6) controlled by an external electronic device (102) (or a processor (640) of the external electronic device (102) of FIG. 6).

[0023] Within the exemplary screen (110) of FIG. 1A, the external electronic device (102) may display a text box (121) for receiving text, and / or a button (122) for receiving an input for transmitting the received text to the electronic device (101) using the text box (121). Although one embodiment of displaying the text box (121) and / or the button (122) at the bottom of the screen (110) is illustrated, the embodiment is not limited thereto. With the text box (121) in focus, the external electronic device (102) may, in response to a text input received via a user's voice input (e.g., a voice input based on an audio signal received via a microphone (672) of FIG. 6) and / or a keyboard (e.g., a keyboard device electrically connected to the external electronic device (102) and / or a software keyboard occupying at least a portion of the display), display one or more characters represented by the voice input and / or the text input within the text box (121). Referring to FIG. 1A, an exemplary state of an external electronic device (102) is illustrated that displays at least one natural language sentence (e.g., “Would you like to borrow a book that is not in the library?”) within a text box (121) in response to a user’s voice input and / or text input.

[0024] In the exemplary state of FIG. 1A, in response to an input indicating a selection of a button (122), the external electronic device (102) may transmit a first signal indicating one or more characters included in a text box (121) to the electronic device (101). The first signal may include a request to execute a function of the electronic device (101) associated with the one or more characters (e.g., a function of identifying a prompt sentence included in the one or more characters using natural language processing). Referring to FIG. 1B, a screen (110) displayed by the external electronic device (102) after receiving an input indicating a selection of the button (122) is illustrated. Referring to the exemplary state of FIG. 1B, the external electronic device (102) may remove one or more characters that were displayed in the text box (121) prior to the state of FIG. 1B, or may reset the text box (121). An external electronic device (102) may display a visual object (130) representing one or more characters corresponding to the input within a screen (110). The visual object (130) may be referred to as a bubble. An exemplary screen (110) is illustrated in which the visual object (130) is positioned along the right edge of the screen (110), but the location of the visual object (130) is not limited thereto.

[0025] In one embodiment, an electronic device (101) that receives a first signal related to a prompt sentence (e.g., an exemplary natural language sentence such as “How do I borrow a book that is not in the library?” in FIG. 1A) may execute a first model for natural language processing (e.g., the first model (210) in FIG. 2) using the prompt sentence. The electronic device (101) may transmit one or more natural language sentences obtained from the first model (210) and related to the prompt sentence to an external electronic device (102). For example, the electronic device (101) may transmit a second signal related to the one or more natural language sentences to the external electronic device (102).

[0026] Referring to FIG. 1C, the external electronic device (102) that receives the second signal may display a visual object (140) including one or more natural language sentences (e.g., “Please use the library search service or use a nearby bookstore”) indicated by the second signal within the screen (110). The visual object (140) may be referred to as a bubble. In order to distinguish the source of the visual objects (130, 140) (e.g., the user of the external electronic device (102) and / or the electronic device (101)), the visual object (140) may be displayed on the left edge of the screen (110), which is different from the right edge of the screen, which is the reference for the location of the visual object (130). The location at which the visual object (140) is displayed is not limited thereto.

[0027] According to one embodiment, the conversational interaction service provided by the electronic device (101) may be referred to as a QnA service for providing an answer to a question. The electronic device (101) may provide a user with an answer to the question by classifying and / or identifying one or more natural language sentences corresponding to the question, received from, for example, an external electronic device (102), using a first model (210).

[0028] Hereinafter, with reference to FIGS. 2 and 3, the method by which the first model (210) is trained and the operation of an electronic device (101) and / or a processor (e.g., the processor (610) of FIG. 6) that stores or outputs information related to one or more natural language sentences obtained based on the execution of the first model (210) are described.

[0029] Figure 2 is an exemplary diagram illustrating operations for training an artificial intelligence model to answer questions.

[0030] Referring to FIG. 2, an electronic device (101) may include a processor (e.g., a processor 610 of FIG. 6), a memory (e.g., a memory 620 of FIG. 6), and a communication circuit (e.g., a communication circuit 630 of FIG. 6). The electronic device (101) may be an electronic device and / or a server for providing an interactive interaction service. The operations described below may be individually or collectively performed by the electronic device (101), the processor (610) within the electronic device (101), an external electronic device (102), a processor within the external electronic device (102) (e.g., a processor 640 of FIG. 6), one or more electronic devices including models (210, 220, 230), and / or one or more processors including a processing circuit included in the one or more electronic devices.

[0031] According to one embodiment, a first model (210) may be provided as a neural network for classifying one or more questions similar to a question input through a communication circuit (630) of the electronic device (101) to perform a function of providing a response corresponding to a question, or for generating a response corresponding to the input question. The first model (210) may be configured to generate a response corresponding to the question based on the question input to the electronic device (101). The electronic device (101) may provide a response provided by the first model (210) in response to a question received from an external electronic device (102) to the electronic device (101), as illustrated in FIGS. 1A to 1C (e.g., text referenced by a visual object (140).

[0032] According to one embodiment, the first model (210) may be provided as a part (or bundle) of the electronic device (101) and / or an external electronic device (102), as a neural network (e.g., the neural network (625) of FIG. 6) for generating a response corresponding to a question input into the electronic device (101). The processor (610) of the electronic device (101) may be configured to drive the first model (210) based on execution of a function for inputting the input question into the first model (210). The processor (610) may be configured to execute a function for driving the first model (210) based on receiving a question from the external electronic device (102). The electronic device (101) may be configured to transmit a response corresponding to the question, output from the first model (210), to the external electronic device (102). For example, the first model (210) may be provided as part of the electronic device (101) so as to be driven based on a plurality of parameters contained in the memory (630) of the electronic device (101). However, the embodiment is not limited thereto.

[0033] Although not shown, the first model (210) may be provided by an electronic device other than the electronic device (101) and / or the external electronic device (102). The electronic device (101) may be configured to transmit a response corresponding to a question to the external electronic device (102) based on receiving the response from the other electronic device.

[0034] In one embodiment, a model trained based on a set of answers to preset questions may generate or provide inappropriate answers to questions when the input question does not fall within the preset set. To overcome this, a first model (210) different from the above model may be trained to provide appropriate answers to various types of questions.

[0035] According to one embodiment, a second model (220) may be provided as a neural network for training the first model (210). The second model (220) may be trained, for example, using a public dataset. For example, the second model (220) may be trained using a semantic textual similarity dataset (STS dataset) as a natural language processing public dataset. The second model (220) may be trained by calculating the similarity between sentences using vectors obtained through embedding of sentences provided from the public dataset. The second model (220) may be referred to as a Bi-Encoder (or Sentence Encoder) in that it uses vectors obtained by embedding each of the sentences, but is not limited thereto.

[0036] According to one embodiment, a third model (230) may be provided as a neural network for training the first model (210). For example, the third model (230), like the second model (220), may be trained from a natural language processing public data set. For example, unlike the second model (220), the third model (230) may be trained by calculating the similarity between any two sentences provided from the public data set using a preset function. The function may be a cross-entropy loss function that utilizes the difference between probability distributions, and the third model (230) may be referred to as a cross-encoder (or scoring model) in that it calculates the similarity between any two input sentences using the function, but the embodiment is not limited thereto.

[0037] In one embodiment, the second model (220) and the third model (230) may be fine-tuned from a natural language processing public data set during the training process. The trained second model (220) and the trained third model (230) may be configured to output the similarity between two input sentences (or questions), respectively. For example, the second model (220) may be used to retrieve a specified number of questions with a high similarity to the input question from a database (250) in which a plurality of questions (310) are each grouped, based on the input question. For example, the third model (230) may be used to re-rank the specified number of questions with a high similarity to the input question, based on the specified number of questions retrieved through the second model (220). For example, the third model (230) may be configured to reorder a specified number of questions for the input questions searched through the second model (220) using a function. However, the embodiment is not limited thereto.

[0038] Although the first model (210), the second model (220), and the third model (230) are illustrated and described as being distinct from each other, the embodiments are not limited thereto. For example, the second model (220) and the third model (230) may be included in the first model (210) or provided as part of the first model (210). The models (210, 220, 230) may be configured to provide different functions for training the first model (210), for example. The models (210, 220, 230) may be provided in various forms as artificial intelligences each trained to perform operations for training the first model (210) and may not be clearly distinguished from each other.

[0039] Hereinafter, with reference to FIG. 3, an operation for training a first model (210) configured to generate a response corresponding to a question using a second model (220) and a third model (230) is described.

[0040] Figure 3 is an exemplary diagram illustrating operations for training an artificial intelligence model to answer questions.

[0041] Referring to FIG. 3, a first model (210) configured to generate a response corresponding to a question can be trained by pair data (330) generated by operations illustrated and described in FIG. 3. The operations described below can be individually or collectively performed by one or more electronic devices, each including an electronic device (e.g., electronic device (101) of FIG. 1A), a processor within the electronic device (101) (e.g., processor (610) of FIG. 6), an external electronic device (e.g., external electronic device (102) of FIG. 1A), a processor within the external electronic device (102) (e.g., processor (640) of FIG. 6), a first model (210), a second model (e.g., second model (220) of FIG. 2), and a third model (e.g., third model (230) of FIG. 2), and / or one or more processors including processing circuitry included in the one or more electronic devices.

[0042] Referring to FIG. 3, a processor (610) of an electronic device (101) may receive a first question (e.g., question (331a)) from an external electronic device (102). Based on receiving the first question, the processor (610) may be configured to search, using a second model (220), from a database (250) including a plurality of questions (310), for a specified number (n) of second questions (e.g., questions (302)) that appear similar to the first question among the plurality of questions (310).

[0043] For example, the database (250) may include information for classifying each of a plurality of questions (310) into one of a plurality of groups (320). The database (250) may include, for example, a group (321) including a set of questions (311) including similar questions (311a, 311b, 311c), a group (322) including a set of questions (312) including similar questions (312a, 312b, 312c), a group (323) including a set of questions (313) including similar questions (313a, 313b, 313c), and a plurality of groups (320) including groups (321, 322, 323) may be stored in the database (250). The processor (610) of the electronic device (101) can, based on a question (311a) received through an external electronic device (102), retrieve a specified number (n) of questions (302) that appear similar to the question (311a) from the second model (220) using the second model (220).

[0044] According to one embodiment, the processor (610) of the electronic device (101) may reorder each of a specified number of second questions retrieved using the second model (220) using the third model (230). For example, the processor (610) may be configured to reorder questions (302) that appear similar to a question (311a) determined using the second model (220) according to the magnitude of the similarity indicated using the third model (230).

[0045] According to one embodiment, the processor (610) of the electronic device (101) may be configured to receive similar questions for expanding each of a plurality of groups (320) including a plurality of questions (310) in the database (250) based on a user input. The similar questions may be obtained from a user. For example, the processor (610) of the electronic device (101) may receive user feedback (e.g., feedback such as like / unlike related to the similarity between natural language sentences corresponding to questions) or obtain information related to the user feedback through a service provided by the first model (210). The feedback may be statistically used to determine the similarity. The processor (610) may be configured to expand each of the plurality of groups (320), for example, by using history information and / or survey information obtained through a service provided via the electronic device (101). For example, the processor (610) may increase the number of similar questions included in each of the sets of questions (321, 322, 323) based on user input. However, the above-mentioned embodiments are exemplary and are not limited thereto.

[0046] According to one embodiment, the processor (610) of the electronic device (101) may be configured to determine whether each of a specified number of second questions searched using the second model (220) is included in a group including the first question among a plurality of groups (320). For example, the processor (610) may determine whether each of a specified number (n) of questions (302) related to a question (311a) searched using the second model (220) is included in a group (321) to which the question (311a) belongs among a plurality of groups (310) of the database (250). Through the determination, the specified number (n) of pair data between the question (311a) and each of the specified number (n) of questions (302) may be generated.

[0047] According to one embodiment, the processor (610) may generate pair data (330) based on a determination as to whether each of the retrieved second questions is included in a group including the first question among a plurality of groups (320). The pair data (330) may include a similarity between the third question and the first question based on a function determined from among designated functions depending on whether the third question corresponding to the determination is included in the group.

[0048] For example, the processor (610) of the electronic device (101) may determine whether a question (312b) among the questions (302) searched through the second model (220) based on a question (311a) received from an external electronic device (102) is included in a group (321) including the question (311a). The processor (610) may generate pair data (330a) based on identifying that the question (312b) is not included in the group (321). For example, the processor (610) may determine whether a question (311c) among the questions (302) is included in a group (321) including the question (311a) received from the external electronic device (102). The processor (610) may generate pair data (330b) based on identifying that the question (311c) is included in the group (321).

[0049] For example, the processor (610) of the electronic device (610) may be configured to determine a function for adjusting the similarity evaluated by the third model (230) based on whether each of the questions (302) searched using the second model (220) and reordered using the third model (230) is included in a group (321) that includes the question (311a). For example, the processor (610) may be configured to adjust the similarity between the questions (311a, 312b) evaluated by the third model (230) through one of the designated functions (e.g., the second similarity adjustment function (420) of FIG. 4) based on identifying that the question (312b) is not included in the group (321) that includes the question (311a). The processor (610) may adjust the similarity between the questions (311a, 311c) evaluated by the third model (230) through another similarity adjustment function (e.g., the first similarity adjustment function (410) of FIG. 4) that is distinct from the one similarity adjustment function among the above-mentioned functions, based on identifying that the question (311c) is included in the group (321) that includes the question (311a). However, the above-mentioned embodiments are exemplary and are not limited thereto. A description of a function for adjusting the similarity between questions found to be similar to an input question is provided through FIG. 4.

[0050] For example, pair data (330) may include a similarity between an input question and a question to be compared, and data (336) representing the similarity. For example, pair data (330a) may include an adjusted similarity (336a) based on the fact that question (312b) is not included in a group (321) that includes question (311a). For example, pair data (330b) may include an adjusted similarity (336b) based on the fact that question (311c) is included in a group (321) that includes question (311a).

[0051] According to one embodiment, the processor (610) of the electronic device (101) may be configured to identify a ratio of questions included in a group including the first question among a specified number (n) of second questions retrieved using the second model (220). The processor (610) may be configured to retrieve additional questions from among the plurality of questions (310) that appear similar to the first question using the second model (220) based on the identified ratio being less than a predetermined ratio. For example, the processor (610) may be configured to retrieve additional questions from the database (250) that appear similar to the question (311a) using the second model (220) based on identifying that a ratio of questions included in a group (321) including the question (311a) among the questions (302) retrieved using the second model (220) as being similar to the question (311a) is less than a predetermined ratio.

[0052] According to one embodiment, the pair data (330) may include various data between the first question and the second questions. The pair data (330) may include, for example, a first question (or data (331) representing the first question), a third question (or data (332) representing the third question) determined among second questions searched as similar to the first question through the second model (220), data (333) representing whether the third question is included in a group including the first question, a first similarity (or data (334) representing the first similarity) between the third question and the first question evaluated by the second model (220), and a second similarity (or data (335) representing the second similarity) between the third question and the first question evaluated by the third model (230). For example, the pair data (330a) may include a question (311a), a question to be compared (312b), a parameter (333a) indicating that the question (312b) is not included in a group (321) including the question (311a), a similarity (334a) between the questions (311a, 312b) evaluated by the second model (220), and a similarity (335a) between the questions (311a, 312b) evaluated by the third model (230). The pair data (330a) may include a similarity (336a) calculated by one of the designated functions determined by the parameter (333a) from the similarity (335a) evaluated by the third model (230).

[0053] For example, the pair data (330b) may include a question (311a), a question to be compared (311c), a parameter (333b) indicating that the question (311c) is included in a group (321) including the question (311a), a similarity (334b) between the questions (311a, 311c) evaluated by the second model (220), and a similarity (335b) between the questions (311a, 311c) evaluated by the third model (230). The pair data (330b) may include a similarity (336b) calculated by one of the designated functions determined by the parameter (333b) from the similarity (335b) evaluated by the third model (230). However, the embodiment is not limited thereto, and pair data such as those in Table 1 below may be generated.

[0054]

[0055] However, the above-mentioned embodiments are exemplary and not limiting.

[0056] According to one embodiment, a first model (210) configured to generate a response corresponding to a question may be trained through pair data (330). The first model (210) trained through the pair data (330), which includes a similarity adjusted by specified functions, may be configured to provide an interactive interaction service through the electronic device (101). According to one embodiment, the processor (610) may be configured to generate training data for training the first model (210) based on history information obtained from a service provided using the trained first model (210). The first model (210) may be retrained through the training data, but is not limited thereto.

[0057] Among the retrieved second questions that appear similar to the first question, functions for adjusting the similarity between the first question and the third question, depending on whether the third question is included in the group including the first question, are described through FIG. 4.

[0058] Figure 4 is a graph illustrating exemplary functions for training an artificial intelligence model to answer questions.

[0059] Referring to FIG. 4, the horizontal axis of the graph (400) represents the similarity between the first question (e.g., question (311a) of FIG. 3) and the third question (e.g., question (312b) and question (311c) of FIG. 3) determined among the second questions (e.g., questions (302) of FIG. 3) evaluated by the third model (e.g., the third model (230) of FIG. 2). The vertical axis of the graph (400) represents the similarity determined through the similarity adjustment functions (410, 420).

[0060] Referring to FIG. 4, the similarity between the first question and the third question included in the pair data (e.g., the pair data (330) of FIG. 3) to be used for training the first model (e.g., the first model (210) of FIG. 2) can be adjusted and / or determined by preset similarity adjustment functions (410, 420).

[0061] For example, a processor (e.g., a processor (610) of FIG. 6) of an electronic device (e.g., an electronic device (101) of FIG. 1A) may be configured to increase a similarity between the first question and the third question obtained from a third model (230) through a first similarity adjustment function (410) based on identifying that a third question is included in a group including the first question among second questions that appear similar to the first question. The first similarity adjustment function (410) may be determined by the following mathematical expression 1.

[0062]

[0063] In the above mathematical expression 1, the x value is a value representing the similarity (e.g., data (335) of FIG. 3) evaluated through the third model (230). The y value is a value representing the similarity (e.g., data (336) of FIG. 3) adjusted by the first similarity adjustment function (410). However, the above-mentioned mathematical expression 1 is exemplary and is not limited thereto. The first similarity adjustment function (410) may include various functions for adjusting the similarity between the first question and the third question based on identifying that the third question is included in a group including the first question.

[0064] For example, the processor (610) of the electronic device (101) may be configured to reduce the similarity between the first question and the third question obtained from the third model (230) through a second similarity adjustment function (420) based on identifying that, among the second questions that appear similar to the first question, the third question is not included in the group including the first question. The second similarity adjustment function (420) may be determined by the following mathematical expression 2.

[0065]

[0066] In the above mathematical expression 1, the x value is a value representing the similarity (e.g., data (335) of FIG. 3) evaluated through the third model (230). The y value is a value representing the similarity (e.g., data (336) of FIG. 3) adjusted by the second similarity adjustment function (420). However, the above-mentioned mathematical expression 2 is exemplary and is not limited thereto. The second similarity adjustment function (420) may include various functions for adjusting the similarity between the first question and the third question based on identifying that the third question is not included in the group including the first question.

[0067] According to one embodiment, the first similarity adjustment function (410) and the second similarity adjustment function (420) may be symmetrical with respect to the function (430), but are not limited thereto. The pair data (330) for training the first model (210) may be adjusted through the similarity adjustment functions (410, 420), thereby providing an improved service through the first model (210).

[0068] Figure 5 is a flowchart illustrating operations for training an artificial intelligence model to answer questions.

[0069] The operations of FIG. 5 may be individually or collectively performed by one or more electronic devices, each including a first model (e.g., the electronic device (101) of FIG. 1A), a processor within the electronic device (101) (e.g., the processor (610) of FIG. 6), an external electronic device (e.g., the external electronic device (102) of FIG. 1A), a processor within the external electronic device (102) (e.g., the processor (640) of FIG. 6), a first model (210), a second model (e.g., the second model (220) of FIG. 2), and a third model (e.g., the third model (230) of FIG. 2), and / or one or more processors including processing circuitry included in the one or more electronic devices, for training a first model (e.g., the first model (210) of FIG. 2).

[0070] Referring to FIG. 5, in operation (501), the processor (610) may retrieve a specified number (e.g., n in FIG. 3) of second questions (e.g., questions (302) in FIG. 3) that appear similar to a first question (e.g., question (311a) in FIG. 3) using a second model (e.g., second model (220) in FIG. 2) from a database (e.g., database (250) in FIG. 2). For example, the processor (610) may retrieve the second questions that are similar to the first question input to the electronic device (101) through the similarities evaluated by the second model (220).

[0071] In operation (503), the processor (610) may generate pair data (e.g., pair data (330) of FIG. 3) including the similarity between the first question and the searched question. For example, the processor (610) may generate pair data including the similarity between the first question and a third question, which is one of the second questions searched as being similar to the first question. The operation (503) may include operations (505, 507, and 509).

[0072] In operation (505), the processor (610) may be configured to identify whether the retrieved question is included in a group including the first question. For example, the processor (610) may compare the first question with a third question, which is one of the second questions that appears similar to the first question through the second model (220), to determine whether the third question is included in a group including the first question (e.g., group (321) of FIG. 3). Based on the determination, a function (e.g., similarity adjustment functions (410, 420) of FIG. 4) for adjusting the similarity evaluated by the second model (220) and / or the third model (e.g., the third model (230) of FIG. 2) may be determined.

[0073] In operation (507), the processor (610) may determine a similarity through a first function (e.g., the first similarity adjustment function (410) of FIG. 4) based on identifying that the searched question is included in a group including the first question. For example, the processor (610) may increase a similarity evaluated by the third model (230) through the first similarity adjustment function (410) based on identifying that a third question among the second questions (e.g., question (311c) of FIG. 3) is included in a group including the first question.

[0074] In operation (509), the processor (610) may determine the similarity through a second function (e.g., the second similarity adjustment function (420) of FIG. 4) based on identifying that the searched question is not included in the group including the first question. For example, the processor (610) may reduce the similarity evaluated by the third model (230) through the first similarity adjustment function (410) based on identifying that a third question among the second questions (e.g., question (312b) of FIG. 3) is not included in the group including the first question.

[0075] In operation (511), the first model (210) can be trained using pair data (330). For example, the processor (610) can be configured to train the first model (210) using the pair data obtained from operation (503).

[0076] Figure 6 is a block diagram of an exemplary electronic device.

[0077] Referring to FIG. 6, the electronic device (101) may include a processor (610), a memory (620), and a communication circuit (630). The external electronic device (102) may include a processor (640), a communication circuit (650), and a display module (660).

[0078] According to one embodiment, the memory (620) of the electronic device (101) may include a neural network (625) configured to generate a response corresponding to a question. The neural network (625) may be referred to as a first model (e.g., the first model (210) of FIG. 1A) driven by a plurality of parameters included in the memory (620). For example, the processor (610) of the electronic device (101) may be configured to execute a function for generating a response to the question in the neural network (625) based on receiving a request for generating a response corresponding to the question from an external electronic device (102). The processor (610) may obtain a response output from the first model and, based on the output response, transmit the response to the external electronic device (102).

[0079] According to one embodiment, the processor (640) of the external electronic device (102) may be configured to receive a user input for entering a question. The processor (640) may be configured to display visual objects (e.g., button (122) of FIG. 1A) for receiving the user input through the display module (660) of the external electronic device (102), or to display a question and a response to the question (e.g., visual objects (130, 140) of FIG. 1C). The processor (640) may transmit a request for a response to the question to the electronic device (101) through the communication circuit (650) of the external electronic device (102).

[0080] According to one embodiment, the external electronic device (102) may include an audio module (670) for inputting and outputting sound. The audio module (670) may include a speaker (671) and a microphone (672). For example, the external electronic device (102) may receive and / or identify a natural language sentence corresponding to a question through the microphone (672) in order to transmit a question to the electronic device (101). For example, the processor (610) of the external electronic device (102) may be configured to output a natural language sentence corresponding to the response through the speaker (671) based on receiving a response to the question from the electronic device (101).

[0081] A method performed by an electronic device (e.g., the electronic device (101) of FIG. 1A) for training a first model (e.g., the first model (210) of FIG. 2) configured to generate a response corresponding to a query according to the above-described embodiment may include an operation of searching for a second question (e.g., questions (312b) and (311c) of FIG. 3) among a plurality of questions (e.g., the plurality of questions (310) of FIG. 3)) that appears similar to a first question (e.g., question (311a) of FIG. 3) using a second model (e.g., the second model (220) of FIG. 2). The method may include an operation of calculating a similarity between the second question and the first question using a third model (e.g., the third model (230) of FIG. 2). The method may include an operation of adjusting the similarity based on whether the second question is included in a group (e.g., group (321) of FIG. 3) that includes the first question among a plurality of groups. The method may include an operation of performing training on the first model using pair data (e.g., pair data (330) of FIG. 3) that includes the adjusted similarity.

[0082] For example, the operation of retrieving the second question may include an operation of retrieving a specified number of questions (e.g., questions (302) of FIG. 3) that include the second question. The operation of calculating the similarity may include an operation of rearranging the specified number of questions based on similarities between the searched specified number of questions and the first question to adjust the similarity.

[0083] For example, the operation of adjusting the similarity may include an operation of adjusting each of the similarities based on a function determined from among the specified functions depending on whether each of the specified number of questions is included in the group including the first question.

[0084] For example, the operation of adjusting the similarity may include an operation of adjusting the similarity to be higher than the similarity calculated by the third model through a first function (e.g., the first similarity adjustment function (410) of FIG. 4) when the second question is included in a group including the first question among the plurality of groups. The operation of adjusting the similarity may include an operation of adjusting the similarity to be lower than the similarity calculated by the third model through a second function (e.g., the second similarity adjustment function (420) of FIG. 4) when the second question is not included in a group including the first question among the plurality of groups.

[0085] For example, the operation of retrieving the second question may include the operation of retrieving a specified number of questions that include the second question. The method may further include the operation of identifying a ratio of questions included in the group that includes the first question among the specified number of questions, and, based on the ratio being less than a predetermined ratio, using the second model to retrieve additional questions among the plurality of questions that appear similar to the first question.

[0086] For example, the method may further include obtaining similar questions for expanding each of the plurality of groups including the plurality of questions based on user input.

[0087] For example, the method may further include an operation of generating learning data for training the first model based on history information obtained from a service provided using the first model.

[0088] In one embodiment, a non-transitory computer-readable storage medium comprising instructions, wherein the instructions, when executed by an electronic device, may be configured to cause the electronic device to retrieve, from a database comprising a plurality of questions, a specified number of second questions from among the plurality of questions, using a second model, that appear similar to a first question. The database may include information for classifying each of the plurality of questions into one of a plurality of groups. The instructions, when executed by the electronic device, may be configured to cause the electronic device to determine whether each of the second questions is included in a group among the plurality of groups in which the first question is included. The instructions, when executed by the electronic device, may be configured to cause the electronic device to generate pair data based on the determination. The pair data may include a similarity between the third question and the first question based on a function determined from among designated functions depending on whether a third question corresponding to the determination is included in the group.

[0089] The above instructions, when executed by the electronic device, may be configured to cause the electronic device to perform training on the first model using the pair data.

[0090] For example, the instructions, when executed by the electronic device, may be configured to cause the electronic device to generate data for determining, based on the determination, one of the functions to be used to calculate the similarity using a third model.

[0091] For example, the pair data may further include a parameter indicating whether the third question is included in the group, a first similarity between the third question and the first question evaluated by the second model, a second similarity between the third question and the first question evaluated by the third model, and a third similarity determined as the similarity based on one of the functions selected from the functions specified by the parameter.

[0092] For example, the third similarity may be determined as a higher similarity than the second similarity by obtaining the second similarity through a first function determined according to the third question included in the group. The third similarity may be determined as a lower similarity than the second similarity by obtaining the second similarity through a second function determined according to the third question not included in the group.

[0093] For example, the instructions, when executed by the electronic device, may be configured to cause the electronic device to identify a proportion of the questions among the second questions that are included in the group that includes the first question. The instructions, when executed by the electronic device, may be configured to cause the electronic device to retrieve, using the second model, additional questions among the plurality of questions that appear similar to the first question, based on the identified proportion being less than a specified proportion.

[0094] For example, the instructions, when executed by the electronic device, may be configured to cause the electronic device to receive similar questions for expanding each of the plurality of groups including the plurality of questions based on user input.

[0095] For example, the instructions, when executed by the electronic device, may be configured to cause the electronic device to generate learning data for training the first model based on historical information obtained from a service provided using the database and the first model.

[0096] In one embodiment, an electronic device may include a memory storing instructions and at least one processor. The instructions, when executed by the at least one processor, may be configured to retrieve, from a database including a plurality of questions, a specified number of second questions that appear similar to a first question among the plurality of questions, using a second model, in order to train a first model configured to generate a response corresponding to the question. The database may include information for classifying each of the plurality of questions into one of a plurality of groups. The instructions, when executed by the at least one processor, may be configured to determine whether each of the second questions is included in a group among the plurality of groups in which the first question is included, in order to train the first model. The instructions, when executed by the at least one processor, may be configured to generate pair data based on the determination, in order to train the first model. The pair data may include a similarity between the third question and the first question based on a function determined from among designated functions depending on whether the third question corresponding to the decision is included in the group. The instructions, when executed by the at least one processor, may be configured to perform training on the first model using the pair data to train the first model.

[0097] For example, the instructions, when executed by the at least one processor, may be configured to generate data for determining the one function to be used to calculate the similarity, based on the determination, using a third model to train the first model.

[0098] For example, the pair data may further include a parameter indicating whether the third question is included in the group, a first similarity between the third question and the first question evaluated by the second model, a second similarity between the third question and the first question evaluated by the third model, and a third similarity determined as the similarity based on one of the functions selected from the functions specified by the parameter.

[0099] For example, the third similarity may be determined as a higher similarity than the second similarity by obtaining the second similarity through a first function determined according to the third question included in the group. The third similarity may be determined as a lower similarity than the second similarity by obtaining the second similarity through a second function determined according to the third question not included in the group.

[0100] For example, the instructions, when executed by the at least one processor, may be configured to identify a proportion of questions included in the group including the first question among the second questions in order to train the first model. The instructions, when executed by the at least one processor, may be configured to retrieve, using the second model, additional questions among the plurality of questions that appear similar to the first question based on the proportion being less than a predetermined proportion, in order to train the first model.

[0101] A method of an electronic device according to one embodiment may include receiving a first question from an external electronic device. The method may include executing a first model using the received first question to obtain a response to the first question. The method may include transmitting the obtained response to the external electronic device. The first model may be trained using pair data. The pair data may include a first question, a second question, and a similarity between the first question and the second question based on a function determined from among functions designated according to whether the first question and the second question are included in the same group among groups for classifying similar questions.

[0102] The devices described above may be implemented as hardware components, software components, and / or a combination of hardware components and software components. For example, the devices and components described in the embodiments may be implemented using one or more general-purpose computers or special-purpose computers, such as a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing instructions and responding to them. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing device is sometimes described as being used alone; however, one of ordinary skill in the art will recognize that the processing device may include multiple processing elements and / or multiple types of processing elements. For example, a processing unit may include multiple processors, or a processor and a controller. Other processing configurations, such as parallel processors, are also possible.

[0103] Software may include a computer program, code, instructions, or a combination of one or more of these, which may configure a processing device to perform a desired operation or may independently or collectively command the processing device. The software and / or data may be embodied in any type of machine, component, physical device, computer storage medium, or device for interpretation by the processing device or for providing instructions or data to the processing device. The software may also be distributed over networked computer systems and stored or executed in a distributed manner. The software and data may be stored on one or more computer-readable recording media.

[0104] The method according to the embodiment may be implemented in the form of program commands that can be executed through various computer means and recorded on a computer-readable medium. In this case, the medium may be one that continuously stores a computer-executable program or one that temporarily stores it for execution or download. In addition, the medium may be various recording means or storage means in the form of a single or multiple hardware combinations, and is not limited to a medium directly connected to a computer system, but may also be distributed over a network. Examples of the medium may include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical recording media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and those configured to store program commands, including ROM, RAM, and flash memory. In addition, examples of other media may include recording media or storage media managed by app stores that distribute applications, sites that supply or distribute various software, servers, etc.

[0105] Although the embodiments described above have been described by way of limited examples and drawings, those skilled in the art will appreciate that various modifications and variations can be made based on the above teachings. For example, appropriate results can still be achieved even if the described techniques are performed in a different order than described, and / or components of the described systems, structures, devices, circuits, etc. are combined or combined in a different manner than described, or are replaced or substituted with other components or equivalents.

[0106] Therefore, other implementations, other embodiments, and equivalents to the claims also fall within the scope of the claims described below.

Claims

1. A method performed by an electronic device for training a first model configured to generate a response corresponding to a query, Using the second model, an operation of searching for a second question among multiple questions that appears similar to the first question; An operation of calculating the similarity between the first question and the second question using the third model; An operation of adjusting the similarity based on whether the second question is included in a group including the first question among a plurality of groups; and An operation of performing training on the first model using pair data including the adjusted similarity, method.

2. In paragraph 1, The operation of searching for the second question includes an operation of searching for a specified number of questions that include the second question, The operation of calculating the above similarity is as follows: An operation of rearranging the specified number of questions based on the similarities between the specified number of questions searched for adjusting the similarity and the first question, method.

3. In paragraph 2, The action of adjusting the above similarity is, An operation of adjusting each of the above similarities based on a function determined from among the specified functions depending on whether each of the above specified number of questions is included in the group including the first question, method.

4. In paragraph 1, The action of adjusting the above similarity is, An operation in which the second question is adjusted to be higher than the similarity calculated by the third model through the first function when the second question is included in a group among the plurality of groups including the first question; and If the second question is not included in a group that includes the first question among the plurality of groups, an operation is included in which the similarity calculated by the third model is adjusted lower than the similarity calculated by the second function. method.

5. In paragraph 1, The operation of searching for the second question includes an operation of searching for a specified number of questions that include the second question, The above method, An operation of identifying a proportion of questions included in the group including the first question among the specified number of questions; and Further comprising an operation of searching for additional questions that appear similar to the first question among the plurality of questions using the second model, based on the above ratio being less than a predetermined ratio. method.

6. In paragraph 1, Further comprising an action of obtaining similar questions for expanding each of the plurality of groups including the plurality of questions based on user input. method.

7. In paragraph 6, Further comprising an action of generating learning data for training the first model based on history information obtained from a service provided using the first model. method.

8. A non-transitory computer-readable storage medium containing instructions, wherein the instructions, when executed by an electronic device, cause the electronic device to: Using the second model, search for second questions that appear similar to the first question among multiple questions; Using the third model, the similarity between the second question and the first question is calculated; Adjusting the similarity based on whether the second question is included in a group including the first question among multiple groups; and configured to perform training on the first model using pair data including the adjusted above similarity, Non-transitory computer-readable storage medium.

9. In the 8th paragraph, when the instructions are executed by the electronic device, the electronic device: Retrieve a specified number of questions that include the second question; and configured to rearrange the specified number of questions based on the similarities between the specified number of questions searched for adjusting the similarity and the first question, Non-transitory computer-readable storage medium.

10. In the 9th paragraph, the instructions, when executed by the electronic device, are configured to cause the electronic device to adjust each of the similarities based on one of the functions determined from among the designated functions depending on whether each of the designated number of questions is included in the group including the first question. Non-transitory computer-readable storage medium.

11. In the 8th paragraph, when the instructions are executed by the electronic device, the electronic device: If the second question is included in a group that includes the first question among the plurality of groups, the similarity calculated by the third model is adjusted to be higher than that calculated by the first function; and If the second question is not included in a group in which the first question is included among the plurality of groups, the second function is configured to adjust the similarity lower than the similarity calculated by the third model. Non-transitory computer-readable storage medium.

12. In the 8th paragraph, when the instructions are executed by an electronic device, the electronic device: Retrieve a specified number of questions containing the second question; Identifying the proportion of questions included in the group including the first question among the specified number of questions; and Based on the above ratio being less than a predetermined ratio, the second model is configured to search for additional questions among the plurality of questions that appear similar to the first question. Non-transitory computer-readable storage medium.

13. In the 8th paragraph, the instructions, when executed by the electronic device, are configured to cause the electronic device to obtain similar questions for expanding each of the plurality of groups including the plurality of questions based on user input. Non-transitory computer-readable storage medium.

14. In the 13th paragraph, the instructions, when executed by the electronic device, are configured to cause the electronic device to generate learning data for training the first model based on history information obtained from a service provided using the first model. Non-transitory computer-readable storage medium.

15. In electronic devices, Memory that stores instructions; and To train a first model comprising at least one processor, wherein the instructions, when executed by the at least one processor, are configured to generate a response corresponding to a question: Using the second model, search for second questions that appear similar to the first question among multiple questions; Using the third model, the similarity between the second question and the first question is calculated; Adjusting the similarity based on whether the second question is included in a group including the first question among multiple groups; and configured to perform training on the first model using pair data including the adjusted above similarity, Electronic devices.

16. In the 15th paragraph, the instructions, when executed by the at least one processor, train the first model: Retrieve a specified number of questions that include the second question; and configured to rearrange the specified number of questions based on the similarities between the specified number of questions searched for adjusting the similarity and the first question, Electronic devices.

17. In the 16th paragraph, the instructions, when executed by the at least one processor, are configured to adjust each of the similarities based on one of the designated functions, determined based on whether each of the designated number of questions is included in the group including the first question, in order to train the first model. Electronic devices.

18. In paragraph 15, The above instructions, when executed by the at least one processor, train the first model: If the second question is included in a group that includes the first question among the plurality of groups, the similarity calculated by the third model is adjusted to be higher than that calculated by the first function; and If the second question is not included in a group in which the first question is included among the plurality of groups, the second function is configured to adjust the similarity lower than the similarity calculated by the third model. Electronic devices.

19. In paragraph 15, the instructions, when executed by the at least one processor, train the first model: Retrieve a specified number of questions containing the second question; Identifying the proportion of questions included in the group including the first question among the specified number of questions; and Based on the above ratio being less than a predetermined ratio, the second model is configured to search for additional questions among the plurality of questions that appear similar to the first question. Electronic devices.

20. In the method of an electronic device, An action of receiving a first question from an external electronic device; An operation of executing a first model using the received first question to obtain a response to the first question; Including an action of transmitting the acquired response to the external electronic device, The above first model is trained by pair data, and the pair data is Question 1; Question 2; and A similarity between the first question and the second question, based on a function determined from among the functions specified according to whether the first question and the second question are included in the same group among the groups for classifying similar questions, method.

Images