Electronic device and Control Method for performing natural language understanding on user speech using multilingual machine translation

Abstract

An electronic device and a method for controlling the same are provided. The electronic device includes a memory storing at least one instruction and a processor that controls the electronic device by executing at least one instruction stored in the memory. When a user voice is input, the processor obtains a first sentence in a first language corresponding to the user voice through a speech recognition model corresponding to the language of the user voice, obtains a second sentence in a second language corresponding to the first sentence in the first language through a machine translation model trained to translate multiple types of languages ​​into a predefined second language, and obtains a control command for the electronic device or a response to the second sentence corresponding to the obtained second sentence through a natural language understanding model trained based on the second language. In particular, at least part of the method for the electronic device to obtain a control command or response to the user voice may utilize an artificial intelligence model trained according to at least one of machine learning, neural networks, or deep learning algorithms.

Description

Technology Field

[0001] The present disclosure relates to an electronic device and a method for controlling the same, and more specifically, to an electronic device and a method for controlling the same capable of performing natural language understanding of a user's voice using multilingual machine translation. Background Technology

[0002] Recently, artificial intelligence systems are being used in various fields. In particular, artificial intelligence systems are electronic devices that learn, make judgments, and become smarter on their own. As artificial intelligence systems improve in recognition rate and become able to understand user preferences more accurately with use, existing rule-based smart systems are gradually being replaced by deep learning-based artificial intelligence systems.

[0003] In particular, recently, AI assistant systems (or conversational systems) capable of providing responses to user voices or controlling electronic devices based on user voices are being developed using artificial intelligence systems.

[0004] Language models used for natural language understanding in such AI assistant systems may require a number of models equal to the number of supported languages ​​or training using a large amount of data corresponding to that number of languages. For example, if two languages ​​are supported, two language models or data for two languages ​​are required for natural language understanding. In this case, creating a language model for natural language understanding requires a large amount of training data corresponding to the number of supported languages, and significant resources are required because CPU and memory are needed to perform training using such a large amount of data. Furthermore, as new words or domains are added over time, or existing words may be interpreted with new meanings, the language models used for natural language understanding may require additional updates. In other words, as the number or size of language models for natural language understanding increases, the resources required for updates increase, leading to the problem that time and costs rise proportionally to the number of supported languages. Prior art literature

[65535] Published Patent Application No. 10-2018-0005850 (2018.01.17.) Published Patent Application No. 10-2018-0113312 (2018.10.16.) Johnson,Melvin et.al, Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translation, Transactions of the association for computational linguistics, MIT Press, Dec. 2017, Vol.5, pp.339-351 The problem to be solved

[0005] The present disclosure is devised to solve the aforementioned problems, and the purpose of the present disclosure is to provide an electronic device and a method for controlling the same that can translate a sentence corresponding to a user's voice into a sentence of a specific language using a multilingual translation model and provide a service for the user's voice through a natural language understanding model for the specific language. means of solving the problem

[0006] According to one embodiment of the present disclosure, an electronic device comprises: a memory storing at least one instruction; and a processor that controls the electronic device by executing at least one instruction stored in the memory. When a user voice is input, the processor may obtain a first sentence in a first language corresponding to the user voice through a speech recognition model corresponding to the language of the user voice, obtain a second sentence in a second language corresponding to the first sentence in the first language through a machine translation model trained to translate a plurality of languages ​​into a predefined second language, and obtain a control command of the electronic device corresponding to the obtained second sentence or obtain a response to the second sentence through a natural language understanding model trained based on the second language.

[0007] Meanwhile, a control method for an electronic device according to another embodiment of the present disclosure comprises: a step of obtaining a first sentence in a first language corresponding to the user voice through a speech recognition model corresponding to the language of the user voice when a user voice is input; a step of obtaining a second sentence in a second language corresponding to the first sentence in the first language through a machine translation model trained to translate a plurality of types of languages ​​into a predefined second language; and a step of obtaining a control command for the electronic device corresponding to the obtained second sentence or obtaining a response to the second sentence through a natural language understanding model trained based on the second language. Brief explanation of the drawing

[0008] FIG. 1 is a drawing illustrating a method for an electronic device to obtain a control command or a response corresponding to a user voice through multilingual machine translation, according to one embodiment of the present disclosure. FIG. 2 is a block diagram showing the configuration of a voice recognition module according to one embodiment of the present disclosure, FIG. 3a is a drawing for illustrating a method for training a machine translation model according to one embodiment of the present disclosure, FIG. 3b is a block diagram showing the configuration of a machine translation module according to one embodiment of the present disclosure, FIG. 4 is a block diagram showing the configuration of a natural language understanding module according to one embodiment of the present disclosure, FIG. 5 is a flowchart illustrating a control method for an electronic device that obtains a control command for a user voice through multilingual machine translation, according to one embodiment of the present disclosure. FIG. 6 is a drawing illustrating a method for an electronic device to obtain a control command or a response corresponding to a user voice through multilingual machine translation, according to another embodiment of the present disclosure. FIG. 7a is a drawing for explaining an embodiment of performing natural language understanding of a user's voice in conjunction with an external server according to an embodiment of the present disclosure. FIG. 7b is a sequence diagram illustrating an embodiment of performing machine translation through a server based on a reliability value for machine translation according to one embodiment of the present disclosure. FIG. 7c is a sequence diagram illustrating an embodiment of performing machine translation and natural language understanding through a server based on a reliability value for machine translation according to another embodiment of the present disclosure. FIG. 8a is a drawing for illustrating a method for performing natural language understanding on a plurality of candidate sentences according to one embodiment of the present disclosure, FIG. 8b is a flowchart illustrating a method for performing natural language understanding on a plurality of candidate sentences according to one embodiment of the present disclosure, FIG. 8c is a sequence diagram illustrating a method for performing natural language understanding of a plurality of candidate sentences in conjunction with an external server according to an embodiment of the present disclosure, FIG. 9 is a block diagram showing the configuration of an electronic device according to one embodiment of the present disclosure, FIG. 10 is a block diagram showing the configuration of a conversational system that provides a response to a user voice or obtains a control command according to one embodiment of the present disclosure. FIG. 11 is a sequence diagram illustrating an embodiment for obtaining control commands for a user's voice by interacting with an external server, according to another embodiment of the present disclosure. Specific details for implementing the invention

[0009] The present disclosure will be described in more detail below with reference to the drawings.

[0010] FIG. 1 is a diagram illustrating a method for an electronic device (100) to obtain a control command or a response corresponding to a user voice through multilingual machine translation according to one embodiment of the present disclosure. Meanwhile, the electronic device (100) according to one embodiment of the present disclosure may be implemented as one of a smartphone, tablet PC, mobile phone, video phone, e-book reader, desktop PC, laptop PC, netbook computer, workstation, server, PDA, PMP (portable multimedia player), MP3 player, medical device, camera, home appliance (e.g., TV, refrigerator, etc.) or wearable device.

[0011] In particular, the electronic device (100) may include a conversational system for providing a response to the user's voice or controlling the electronic device (100) by performing voice recognition and natural language understanding on the user's voice. At this time, the conversational system may include a voice recognition module (220), a voice recognition model (225), a machine translation module (230), a multilingual machine translation model (235), a natural language understanding module (240), and a natural language understanding model (245), as illustrated in FIG. 1.

[0012] When the conversational system is executed by a user voice containing a wake-up word (or trigger word), the speech recognition module (220), machine translation module (230), and natural language understanding module (240) stored in memory (110) (e.g., non-volatile memory) may be loaded into memory (e.g., volatile memory) included in the processor (120). At this time, the speech recognition module (220), machine translation module (230), and natural language understanding module (240) may be loaded simultaneously, but this is merely one embodiment and they may be loaded sequentially.

[0013] And, each of the speech recognition module (220), machine translation module (230), and natural language understanding module (240) can load the speech recognition model (225), multilingual machine translation model (235), and natural language understanding model (245) stored in memory (110) (e.g., non-volatile memory) into memory (e.g., volatile memory) included in the processor (120). At this time, the speech recognition model (225), multilingual machine translation model (235), and natural language understanding model (245) may be loaded sequentially by the speech recognition module (220), machine translation module (230), and natural language understanding module (240), but this is merely one embodiment, and they may be loaded into volatile memory simultaneously by a wake-up word.

[0014] The microphone (140) can receive user voice input. At this time, the user voice input to the electronic device (100) may be a first language (e.g., Korean). The user voice input through the microphone (140) may be an analog audio signal in the form of voice data. Meanwhile, the user voice input to the microphone (140) may be a first language (e.g., Korean). The microphone (140) can process the input user voice in a digital form and output it to a voice recognition module (220) within the processor (120). Meanwhile, the electronic device (100) may receive user voice input through the microphone (140) provided within the electronic device (100) or connected to the electronic device (100) via a wired or wireless connection, but this is merely one embodiment, and user voice may be received from an external device.

[0015] The voice recognition module (220) can perform voice recognition on the input user voice using a voice recognition model (225) to obtain a first sentence of a first language in the form of text data corresponding to the user voice in the form of voice data.

[0016] Specifically, the voice recognition module (220) can remove noise from the user's voice in digital form, extract a segment of the user's voice, and input information about the extracted segment of the user's voice into the voice recognition model (225). The voice recognition model (225) can obtain a first sentence of a first language in the form of text data corresponding to the user's voice. At this time, the voice recognition model (225) may include an acoustic model and a language model. In particular, according to one embodiment of the present disclosure, a plurality of voice recognition models (225-1 to 225-N) corresponding to a plurality of languages ​​may be stored in the memory (110) to recognize multiple languages, and each voice recognition model may be equipped with a language model and a plurality of acoustic models. In particular, the voice recognition module (220) can determine the language of the user's voice among a plurality of languages, and input the user's voice into the language model and acoustic model corresponding to the language of the user's voice to obtain a first sentence of the first language corresponding to the user's voice. The first sentence may include at least one text and may include all forms of meaningful words, phrases, and clauses. Meanwhile, the characteristics of the voice recognition module (220) determining the language of the user's voice will be explained in detail with reference to FIG. 2.

[0017] The speech recognition module (220) can output a first sentence of a first language obtained through a speech recognition model (225) to a machine translation module (230). Then, the machine translation module (230) can output the first sentence of the first language to a multilingual machine translation model (235) to translate the user's voice into a second language (e.g., English).

[0018] A multilingual machine translation model (235) can obtain a second sentence in a second language by performing multilingual machine translation (20) on a first sentence in a first language. At this time, the multilingual machine translation model (235) may be a multilingual neural network translation model that translates multiple languages ​​into a predefined single language (here, the second language). The multilingual neural network translation model may be a neural network translation model trained using a multi-task learning method. This will be explained in detail later with reference to FIG. 3a.

[0019] That is, the multilingual machine translation model (235) can obtain a second sentence in the second language by translating a sentence in any language into a predefined second language. Meanwhile, the multilingual machine translation model (235) can obtain a sentence containing only the second language, but this is merely one embodiment, and it can obtain a sentence containing a language other than the second language.

[0020] Additionally, the multilingual machine translation model (235) can obtain at least one candidate sentence of the second language and a confidence value for each of the at least one candidate sentence and output it to the machine translation module (230). At this time, the confidence value for the candidate sentence may be a value representing the translation accuracy of the candidate sentence. Then, the machine translation module (230) can output the candidate sentence with the highest confidence value among the multiple candidate sentences as the second sentence.

[0021] Additionally, the multilingual machine translation model (235) can be implemented as the multilingual machine translation model (340) shown in FIG. 3b, and can modify some of the text included in the second sentence of the second language through the beam search decoder (360) and constrain DB (370).

[0022] The machine translation module (230) can output a second sentence in a second language output from a multilingual machine translation model (235) to a natural language understanding module (240), and the natural language understanding module (240) can output a second sentence in a second language to a natural language understanding model (245).

[0023] At this time, the natural language understanding model (245) may be a natural language understanding model trained based on a second language, and may acquire a user intent corresponding to the user's voice and a slot (or entity, parameter) required to perform the user intent. At this time, the slot is the main data of the electronic device for performing an action corresponding to the user intent, and may vary depending on the action or application being executed. For example, if the user intent is an alarm, the slot may be the type of alarm application and alarm time information, and if the user intent is a search, the slot may include the type of search application and search terms.

[0024] At this time, the natural language understanding model (245) can be learned based on text of a second language for performing functions or services provided by the electronic device (100). For example, the natural language understanding model (245) can identify user intent by selecting one candidate sentence based on the functions or services provided by the electronic device (100) from among a plurality of candidate sentences of a second language output by the multilingual machine translation model (235). At this time, the multilingual machine translation model (235) can be re-learned based on the results determined by the natural language understanding model (245).

[0025] The natural language understanding model (245) can output information about user intent and slots for the acquired user voice to the natural language understanding module (240). At this time, the natural language understanding module (240) can acquire or modify information about user intent and slots using intent classification (410), slot filling (420), slot replacement (430), and proper noun DB (440) as shown in FIG. 4.

[0026] The processor (120) can obtain a control command corresponding to the user voice (i.e., the second sentence) or obtain a response to the user voice (i.e., the second sentence) based on the user intent and slot information obtained by the natural language understanding module (240) in the manner described above.

[0027] As described above, the electronic device (100) can save a more lightweight natural language understanding model by translating any language into a predefined second language through a multilingual machine translation model (235) capable of translating multiple languages ​​into a predefined single language. Furthermore, not only are the resources required for future updates to the natural language understanding model reduced, but the update time can also be shortened.

[0028] Meanwhile, in the above-described embodiment, a voice recognition module (220), a machine translation module (230), and a natural language understanding module (240) may be included in the electronic device, but this is merely one embodiment, and a configuration for performing at least one of voice recognition, multilingual machine translation, and natural language understanding may be performed on an external server. This will be explained in detail later with reference to the drawings.

[0029] Meanwhile, although the above-described embodiment describes obtaining a first sentence of the first language by performing speech recognition on the acquired user voice, this is merely one embodiment, and the first sentence of the first language can be obtained using other methods. For example, when an image containing text is input, the electronic device (100) can obtain a first sentence of the first language contained in the image by performing character recognition on the image (e.g., OCR (optical character reader) analysis) or by inputting the image into a trained artificial intelligence model, and can perform machine translation of the obtained first sentence of the first language into a second sentence of the second language, and can perform natural language understanding using the second sentence of the second language. As another example, when text of the first sentence of the first language is input according to user operation while a text input UI is displayed, the electronic device (100) can perform machine translation of the obtained first sentence of the first language into a second sentence of the second language, and can perform natural language understanding using the second sentence of the second language.

[0030] Meanwhile, in FIG. 1, the speech recognition model (225), multilingual machine translation model (235), and natural language understanding model (245) are described as separate components from the speech recognition module (220), machine translation module (230), and natural language understanding module (240), but this is merely one embodiment, and the speech recognition model (225), multilingual machine translation model (235), and natural language understanding model (245) may be included in the speech recognition module (220), machine translation module (230), and natural language understanding module (240).

[0032] FIG. 2 is a block diagram showing the configuration of a voice recognition module according to one embodiment of the present disclosure. In particular, according to one embodiment of the present disclosure, the voice recognition module may include a language determination module (221) for determining a voice recognition model (235) corresponding to the language of the user's voice among a plurality of languages. And, the memory (110) may include voice recognition models (225) corresponding to a plurality of languages. For example, as shown in FIG. 2, the memory (110) may store a first voice recognition model (225-1) including a first AM and a first LM corresponding to English, a second voice recognition model (225-2) including a second AM and a second LM corresponding to Korean, and a third voice recognition model (225-3) including a third AM and a third LM corresponding to Japanese, etc.

[0033] For example, when a user voice is input, the language determination module (221) can determine the language type of the user voice based on user settings. Specifically, if the language setting of the electronic device is Korean, the language determination module (221) can determine the language of the user voice as Korean. As another example, when a user voice is input, the language determination module (221) can determine the language type of the user voice by analyzing the phonetic symbols of the input user voice. As yet another example, the language determination module (221) can determine the language type of the user voice by inputting the user voice into a neural network model trained to determine language.

[0034] When the language type of the user's voice is determined, the voice recognition module (220) can perform voice recognition using a second voice recognition model (225-2) corresponding to Korean determined by the language determination module (221). At this time, the voice recognition module (220) can load only the voice recognition model corresponding to the language of the user's voice among a plurality of voice recognition models.

[0035] Meanwhile, although the above-described embodiment describes a plurality of voice recognition models being stored in memory (110), this is merely one embodiment, and a plurality of voice recognition models may be stored in an external server. Furthermore, when the language of the user's voice is determined by the language determination module (221), the processor (120) may transmit a signal requesting information about the user's voice and a voice recognition model corresponding to the language of the user's voice to an external server through the communication interface (130), and may receive a voice recognition model corresponding to the language of the user's voice from the external server.

[0036] FIG. 3a is a diagram illustrating a method for training a multilingual machine translation model according to one embodiment of the present disclosure. According to one embodiment of the present disclosure, the multilingual machine translation model (235) may include a multilingual neural network translation model (340).

[0037] A multilingual neural network translation model (340) may be a neural network model constructed by performing multitasking learning based on multiple parallel corpora in which each of the multiple languages ​​is matched with a predefined language. For example, as illustrated in FIG. 3a, the multilingual neural network translation model (340) may be trained using a multitask learning method based on a first parallel corpus (310) that stores Korean-English pairs, a second parallel corpus (320) that stores Japanese-English pairs, and a third parallel corpus (330) that stores Chinese-English pairs. In this case, multitask learning is a method of improving prediction performance by training each of multiple tasks simultaneously, where a task may refer to a parallel corpus. Multitask learning is intended to maximize performance from a generalization perspective while guaranteeing the performance of the model itself for data collected from various tasks, and can consider both the global performance of the model and the local performance related to the task. In particular, the multilingual neural network translation model (340) may include individual parameters corresponding to the language to be translated and common parameters corresponding to a predefined target language for translation, and may learn the individual parameters and common parameters by receiving the first to third parallel corpora (310 to 330) as input.

[0038] Meanwhile, although FIG. 3a describes a multilingual neural network translation model (340) performing multitasking based on multiple parallel corpora in which each of the multiple languages ​​is matched with a predefined language, this is merely one embodiment, and the multilingual neural network translation model (340) can perform learning using parallel corpora in which multiple languages ​​are matched with each other. For example, the multilingual neural network translation model can be trained using parallel corpora in the form of Korean-English, English-Japanese, and Japanese-Korean.

[0039] As illustrated in FIG. 3a, when an input sentence in Korean / Japanese / Chinese is received by a multilingual neural network translation model (340) trained through multi-task learning, the multilingual neural network translation model (340) can output an output sentence in English. That is, if the first sentence is in Korean and the second sentence is in English, the multilingual neural network translation model (340) can translate the first sentence in Korean to obtain the second sentence in English.

[0040] Additionally, the multilingual neural network translation model (340) can translate a first sentence of the first language to obtain multiple candidate sentences of the second language and confidence values ​​for each of the candidate sentences. At this time, the multilingual neural network translation model (340) can output multiple candidate sentences and confidence values ​​for each of the multiple candidate sentences to the natural language understanding module (240) through the machine translation module (220), but this is merely one embodiment, and the candidate sentence with the highest confidence value can be determined as the second sentence and output the second sentence and the confidence value of the second sentence to the natural language understanding module (240).

[0041] FIG. 3b is a block diagram showing the configuration of a machine translation module according to one embodiment of the present disclosure.

[0042] The machine translation module (230) can translate a specific text of a first language into a text mapped to a specific text by using a Constrain database (Constrain DB) (or a first database) (370) that maps and stores input text and modified text of a multilingual machine translation model.

[0043] Specifically, as illustrated in FIG. 3b, the machine translation module (230) may include an encoder-decoder model (350) and a beam search decoder (360). In this case, the encoder-decoder model (350) is a machine translation model for translating a first sentence of a first language input, and can obtain a second sentence of a second language corresponding to the first sentence of the first language. The beam search decoder (360) can determine whether text stored in the Constrain DB (370) exists among the second sentences of the second language and modify them. For example, the first sentence of the first language, "Turn on text application," can be translated by the encoder-decoder model (350) into the second sentence of the second language, "Turn on text application." At this time, if "text application" and "message application" are matched and stored as input text and modified text in the Constrain DB (370), the beam search decoder (360) can modify the second sentence of the second language to "Turn on message application".

[0044] Meanwhile, although the above-described embodiment describes the input text and modified text of the beam search decoder (360) as both being in English, this is merely one embodiment, and the input text and modified text of the beam search decoder (360) may be in different languages. According to another embodiment, the beam search decoder (360) may be included in an encoder-decoder model (350) to directly translate the input text of the first language into the modified text of the second language. Specifically, the input text and modified text stored in the Constrain DB (370) may be in different languages. For example, the input text and modified text may be stored as "text application" and "message application". In this case, the beam search decoder (360) may directly translate "text application" of the first language into "message application" of the second language based on the input text and modified text stored in the Constrain DB (370).

[0045] At this time, the Constrain DB (370) can be updated based on the output result of the natural language understanding module (240). Specifically, if some text of the second sentence of the second language is replaced through the natural language understanding module (240), the Constrain DB (370) can update the input text and the modified text based on the replacement result.

[0046] FIG. 4 is a block diagram showing the configuration of a natural language understanding module (240) according to one embodiment of the present disclosure. As shown in FIG. 4, the natural language understanding module (240) may include an Intent classification (410), a slot filling (420), and a slot replacement (430). At this time, at least one of the Intent classification (410) and the slot filling (420) may be implemented as a natural language understanding model (245).

[0047] Intent classification (410) can classify (or judge, understand, analyze, identify) user intent for a second sentence of a second language (or, the highest priority sentence among multiple candidate sentences of the second language). In this case, Intent classification (410) can classify user intent through grammatical analysis or semantic analysis. However, if Intent classification (410) fails to classify user intent for a second sentence of the second language (i.e., the highest priority sentence of the second language) (i.e., if the confidence value regarding whether the user intent classified for the second sentence of the second language is accurate is below a threshold value), Intent classification (410) can classify user intent for a lower priority sentence of the second language. When intent for a lower priority sentence is classified, the natural language understanding module (240) can update the Constrain DB (370) by mapping at least one text included in the first sentence and at least one text included in the lower priority sentence to the input text and modified text. For example, regarding the first sentence of the first language, "Turn on message application," if "Turn on message application" is obtained as the highest priority sentence of the second language and "Open message application" is obtained as the second priority sentence, the Intent classification (410) cannot identify the user intent for the highest priority sentence, but can determine "LAUNCH APP" as the user intent for the second priority sentence. Accordingly, the natural language understanding module (240) can update the Constrain DB (370) to store the input text and modified text, respectively, as text related to the application, by mapping them to "Turn on" and "Open" or "Turn on" and "Open".

[0048] Slot filling (420) can determine a slot for a function that the user wants to perform by using a matching rule based on the user's intent. For example, for a second sentence in a second language that is "Open message application", Slot filling (420) can determine "message app" as a slot.

[0049] Slot replacement (430) can replace a slot filled by Slot filling (420) with another slot. At this time, slot replacement (430) can determine whether there is a slot that needs to be replaced by using the proper noun DB (440) (or a second database). For example, if an application name is stored in the first language in the proper noun DB (440), slot replacement (430) can replace the determined slot "message app" with "message application". At this time, the proper noun DB (440) may store not only application names but also contact information, schedule information, etc.

[0050] Additionally, the natural language understanding module (240) can update the Constrain DB (370) based on the replacement results from the Slot replacement (430). For example, the natural language understanding module (240) can update the Constrain DB (370) to store the input text and modified text, respectively, by mapping them to a "text application" and a "message application".

[0051] FIG. 5 is a flowchart illustrating a control method for an electronic device that obtains control commands for a user voice through multilingual machine translation, according to one embodiment of the present disclosure.

[0052] The electronic device (100) can acquire a user voice (S510). At this time, the user voice may include a first sentence of a first language in the form of voice data.

[0053] The electronic device (100) can obtain a first sentence of a first language corresponding to the user's voice through a voice recognition model corresponding to the language of the user's voice (S520). That is, as described above in FIG. 2, the electronic device (100) can determine the language of the user's voice and obtain a first sentence of a first language in the form of text data through a voice recognition model corresponding to the determined user's voice.

[0054] The electronic device (100) can obtain a second sentence of a second language corresponding to a first sentence of a first language through a machine translation model (S530). At this time, the machine translation model may be a multilingual neural network translation model (340) as described in FIG. 3a, and the multilingual neural network translation model (340) may be trained to translate any language into a predefined second language.

[0055] The electronic device (100) can obtain a control command of the electronic device (100) corresponding to the second sentence obtained through the natural language understanding model (245) or obtain a response to the second sentence (S540). That is, the electronic device (100) can determine a user intent and a slot corresponding to the second sentence through the natural language understanding model (245), and obtain a control command or a response based on the determined user intent and a slot.

[0056] FIG. 6 is a diagram illustrating a method for an electronic device to obtain a control command or a response corresponding to a user voice through multilingual machine translation, according to another embodiment of the present disclosure. Meanwhile, descriptions of FIG. 6 that overlap with descriptions of FIG. 1 are omitted. Additionally, the first machine translation model (235) shown in FIG. 6 may be the same module as the machine translation model (235) shown in FIG. 1.

[0057] When a first sentence of a first language is input into the first machine translation model (235), the first machine translation model (235) translates the first sentence of the first language to obtain a second sentence of the second language and a confidence value, and can output the obtained second sentence of the second language and the confidence value to the machine translation module (230). At this time, the first machine translation model (235) can obtain the sentence with the highest confidence value among a plurality of candidate sentences and obtain a confidence value for the sentence with the highest confidence value. At this time, the confidence value may be a value regarding the accuracy of the translation for the corresponding sentence. Meanwhile, this is merely one embodiment, and the first machine translation model (235) can transmit a plurality of candidate sentences and a confidence value for each of the plurality of candidate sentences to the machine translation module (230).

[0058] The machine translation module (240) can request a translation of the first sentence from an external server (600) based on a confidence value. That is, if the confidence value of the sentence with the highest confidence value is below a threshold value, the machine translation module (240) can transmit the first sentence of the first language to the server (600) through the communication interface (130).

[0059] The server (600) can translate a first sentence in a first language into a third sentence in a second language through a second machine translation model (237) and transmit it to a machine translation module (230) of an electronic device (100) through a communication interface (130). At this time, the second machine translation model (237) is a translation model trained with more data (e.g., a corpus) than the first machine translation model (235), and may have higher accuracy than the first machine translation model (235).

[0060] The machine translation module (230) can output a third sentence in a second language received from the server (600) to the natural language understanding module (240). Meanwhile, the machine translation module (230) can output the third sentence in a second language to the natural language understanding module (240) and can update (or retrain) the first machine translation model (235) based on the third sentence in the second language.

[0061] The natural language understanding module (240) can output a third sentence of the second language to the natural language understanding model (345).

[0062] The natural language understanding model (245) can perform natural language understanding based on a second sentence of a third language to obtain information about user intent and slots for user voice, and output the obtained information about user intent and slots for user voice to the natural language understanding module (240).

[0063] The processor (120) can provide a response to a user's voice or control an electronic device (100) based on user intent and slot information obtained by the natural language understanding module (240) in the manner described above. In particular, as shown in FIG. 6, by requesting machine translation from the server (600) based on the reliability value output by the machine translation model (235), it is possible to provide a response service or device control service to the user through a more accurate translation, and the performance of the first machine translation model (235) can be improved.

[0065] FIG. 7a is a drawing for explaining an embodiment of performing natural language understanding of a user's voice in conjunction with an external server according to one embodiment of the present disclosure.

[0066] First, as illustrated in FIG. 7a, the electronic device (100) may include a speech recognition module (710), a first machine translation module (720), and a first natural language understanding module (730), and the server (700) may include a second machine translation module (740) and a second natural language understanding module (750). In this case, the first machine translation module (720) and the second machine translation module (740) may each include a first machine translation model and a second machine translation model capable of translating any language into a predefined second language. In this case, the second machine translation model is a machine translation model trained using a larger amount of corpus than the first machine translation model, and can perform more accurate machine translation than the first machine translation model. Additionally, the first natural language understanding module (730) and the second natural language understanding module (750) may each include a first natural language understanding model and a second natural language understanding model for performing natural language understanding on user speech to obtain user intent and slots. The first natural language understanding model can understand natural language in a domain corresponding to the functions and services provided by the electronic device (100), but the second natural language understanding model can understand natural language in a wider domain without limitation to the functions and services provided by the electronic device (100).

[0067] In particular, the electronic device (100) can perform natural language understanding of the user's voice by linking with the server (700) based on the reliability value of the second sentence of the second language obtained from the first machine translation module (720).

[0068] Hereinafter, with reference to FIGS. 7b and FIGS. 7c, an embodiment of performing machine translation or natural language understanding through a server based on a reliability value for machine translation will be described.

[0069] FIG. 7b is a sequence diagram illustrating an embodiment in which a server (700) performs machine translation when the reliability value is low, according to one embodiment of the present disclosure.

[0070] First, the electronic device (100) can acquire a user voice (S705). At this time, the input user voice may include at least one text in the form of voice data.

[0071] The electronic device (100) can obtain a first sentence of a first language corresponding to the user's voice through a voice recognition module (S710). That is, the electronic device (100) can obtain a first sentence of a first language in the form of text data through a voice recognition module.

[0072] The electronic device (100) can obtain a second sentence and a confidence value of a second language by translating a first sentence of a first language through a first machine translation model (S715). At this time, the electronic device (100) can obtain the second sentence with the highest confidence value among the multiple candidate sentences of the second language obtained by translating the first sentence of the first language through the first machine translation model.

[0073] The electronic device (100) can determine whether the acquired reliability value is greater than a threshold value (S720). At this time, the threshold value represents a reliable value for the highest priority sentence as user voice, and may be, for example, 0.95.

[0074] If the confidence value is greater than the threshold value (S720-Y), the electronic device (100) can obtain a control command corresponding to the second sentence or a response to the second sentence using the first natural language understanding model (S740). That is, the electronic device (100) can determine a user intent and a slot corresponding to the user voice by inputting the second sentence of the second language into the first natural language understanding model, and can obtain a control command or a response based on the user intent and the slot.

[0075] If the reliability value is less than or equal to the threshold value (S720-N), the electronic device (100) can transmit the first sentence of the first language to an external server (700) (S725). At this time, the electronic device (100) can transmit the first sentence of the first language in the form of text data, but this is merely one embodiment and it can also transmit it in the form of voice data.

[0076] The server (700) can obtain a third sentence in the second language by translating a first sentence in the first language through a second machine translation model (S730). At this time, the server (700) can obtain a third sentence in the second language by translating a first sentence in the first language through a second machine translation model that has been trained with more data (e.g., a corpus) than the first machine translation model. Additionally, when a first sentence in the first language in the form of voice data is received, the server (700) can obtain a first sentence in the first language in the form of text data through a voice recognition module and translate the obtained sentence through a second machine translation model.

[0077] The server (700) can transmit the third sentence of the second language to the electronic device (100) (S735).

[0078] The electronic device (100) can obtain a control command corresponding to the third sentence or obtain a response to the third sentence using the first natural language understanding model (S740). That is, the electronic device (100) can determine a user intent and slot corresponding to the user voice by inputting the third sentence of the second language into the first natural language understanding model, and can obtain a control command or response based on the user intent and slot. At this time, the electronic device (100) can retrain a multilingual neural network translation model based on the obtained first sentence of the first language and the third sentence of the second language.

[0079] FIG. 7c is a sequence diagram illustrating an embodiment in which a server (600) performs machine translation and natural language understanding when the reliability value is low, according to one embodiment of the present disclosure. Since steps S750 to S775 according to an embodiment of the present invention are identical to steps S705 to S730 disclosed in FIG. 7b, a redundant description is omitted.

[0080] The server (700) can obtain a control command or response corresponding to a third sentence of the second language obtained through a second machine translation model using a second natural language understanding model (S780). Specifically, the server (700) can determine a user intent and a slot corresponding to the third sentence through the second natural language understanding model, and obtain an action rule or response as a control command based on the determined user intent and the slot. That is, the server (700) can obtain a more accurate control command or response by performing natural language understanding through a second natural language understanding model capable of performing natural language understanding for more domains.

[0081] The server (700) can transmit a control command or a response to the electronic device (100) (S785), and the electronic device (100) can receive the control command or response obtained from the server (700) (S790). Then, the electronic device (100) can control the electronic device (100) or output a response based on the received control command.

[0083] FIG. 8a is a diagram illustrating a method for performing natural language understanding on a plurality of candidate sentences according to one embodiment of the present disclosure.

[0084] The electronic device (100) can obtain a first sentence of a first language (① in FIG. 8a). For example, the electronic device (100) can obtain a first sentence in Korean, "Turn on the text application."

[0085] The electronic device (100) can obtain multiple candidate sentences of a second language using a machine translation module (330) (② in FIG. 8a). At this time, the electronic device (100) can obtain not only multiple candidate sentences of the second language but also the reliability values ​​of each of the multiple candidate sentences. For example, the electronic device (100) can obtain "turn on message application", "open message application", and "play message application" as candidate sentences of the second language translated from "turn on message application", and can obtain "0.98", "0.94", and "0.88" as reliability values ​​for each.

[0086] The electronic device (100) can perform N-best rescoring after analyzing the intent of the highest priority sentence with the highest confidence value among multiple candidate sentences through the natural language understanding module (340) (③ in FIG. 8a). That is, the electronic device (100) can determine the user intent regarding the highest priority sentence. At this time, if the user intent regarding the highest priority sentence is determined, the electronic device (100) can determine the slot based on the determined user intent. If the user intent regarding the highest priority sentence is not determined, the electronic device (100) can determine the user intent regarding the next priority sentence with a lower confidence value than the highest priority sentence. At this time, the electronic device (100) can rescore the confidence value for the next priority sentence.

[0087] For example, the electronic device (100) can analyze the user intent for "turn on message application" through the natural language understanding module (240). At this time, the electronic device (100) cannot analyze the user intent for "turn on message application" and can analyze the user intent for the next-ranked sentence, "open message application". At this time, the electronic device (100) can determine the user intent as "LAUNCH APP". Accordingly, the electronic device (100) can rescore the confidence value for "open message application" and retrain the multilingual neural network translation model included in the machine translation module (230) based on the rescored confidence value. At this time, the electronic device (100) can retrain the multilingual neural network translation model based on the alignment information of the text. That is, when the electronic device (100) performs a translation of "turn on" as an action for an application within a sentence, it can retrain the multilingual neural network translation model (340) so that the confidence value of "open" is higher than that of "turn on". That is, when the electronic device (100) performs a translation of "turn on" when there is no text for an application within a sentence, the confidence value of "turn on" may be higher than that of "open".

[0088] The electronic device (100) can determine a slot for performing an action corresponding to the user's voice after the user's intention is determined. At this time, the electronic device (100) can replace the slot by determining whether there is a proper noun stored in the proper noun DB (440) in the sentence where the intention is determined (④ in FIG. 8a). For example, the electronic device (100) can obtain "APP NAME: message application" as a slot for the user's intention "LAUNCH APP". At this time, if "text application" is stored in Korean in the proper noun DB (440), the electronic device (100) can replace the existing slot with "APP NAME: text application" based on the text application included in the first language.

[0089] The electronic device (100) can obtain intent and slot information through the natural language understanding module (240) by going through the process described above (⑤ in FIG. 8a). For example, the electronic device (100) can obtain "LAUNCH APP" as a user intent and "APP NAME: text application" as a slot. Accordingly, the electronic device (100) can obtain a control command based on the obtained user intent and slot information. That is, the electronic device (100) can obtain a control command of "execute text application" and can execute a text application based on the obtained control command.

[0090] Additionally, the electronic device (100) can update the Constrain DB (370) based on the information stored in the proper noun DB (440) after the proper noun replacement (⑥ in FIG. 8a). For example, the electronic device (100) can update the Constrain DB (370) so that the machine translation module (230) outputs the translation result for "text application" as "message application" rather than translating it as "text application". That is, the input test and the modified text can be stored in the Constrain DB (370) by matching them as "message application" and "text application," respectively.

[0091] FIG. 8b is a flowchart illustrating a method for performing natural language understanding on a plurality of candidate sentences according to one embodiment of the present disclosure.

[0092] The electronic device (100) can obtain multiple candidate sentences through a machine translation module (S805). At this time, the electronic device (100) can obtain not only multiple candidate sentences but also confidence values ​​for each of the multiple candidate sentences.

[0093] The electronic device (100) can identify the highest priority candidate sentence (S810). At this time, the electronic device (100) can identify the sentence with the highest reliability value among the multiple candidate sentences as the highest priority candidate sentence.

[0094] The electronic device (100) can analyze the user's intent regarding the candidate sentence (S815). That is, the electronic device (100) can analyze the user's intent regarding the highest priority candidate sentence by performing grammatical analysis and semantic analysis.

[0095] At this time, the electronic device (100) can determine whether the user's intention has been analyzed (S820).

[0096] If the user intent is not analyzed (S820-N), the electronic device (100) can identify a next-ranked candidate sentence (S825). That is, the electronic device (100) can identify a next-ranked candidate sentence that has a lower confidence value than the highest-ranked candidate sentence.

[0097] And, the electronic device (100) can again analyze the user's intent regarding the candidate sentence (S815).

[0098] When the user intent is analyzed through the process described above (S820-Y), the electronic device (100) can determine whether a proper noun exists in the slot corresponding to the user intent (S830). That is, the electronic device (100) can determine whether a proper noun stored in the proper noun DB (440) of the electronic device (100) exists in the slot corresponding to the user intent.

[0099] If no proper noun exists (S835-N), the electronic device (100) can obtain user intent and slot information (S840). However, if a proper noun exists (S835-Y), the electronic device (100) can replace the proper noun included in the slot based on the proper noun DB (440) (S835), and obtain user intent and slot information based on the replaced slot.

[0100] And, the electronic device (100) can update the Constrain DB (370) based on the replacement result of the proper noun (S845).

[0102] Meanwhile, only a limited number of proper nouns can be stored within the electronic device (100). That is, only proper nouns related to the electronic device (100) (e.g., application name, contact information, schedule information, etc.) are stored, and general proper nouns (e.g., place name, store name, etc.) may not be stored. Accordingly, the electronic device can replace proper nouns by linking with an external server (200). FIG. 8c is a sequence diagram for explaining a method of performing natural language understanding of a plurality of candidate sentences by linking with an external server according to an embodiment of the present disclosure. Meanwhile, S850 to S870 shown in FIG. 8c overlap with S805 to S825 described in FIG. 8b, so a detailed description is omitted.

[0103] The electronic device (100) can transmit the candidate sentence for which intent analysis has been performed to the server (800) (S875).

[0104] The server (800) can determine whether a proper noun exists in the slot corresponding to the user's intention (S880). That is, the server (800) can determine whether a proper noun stored in the proper noun DB stored by the server (800) exists in the slot corresponding to the user's intention. At this time, the proper noun DB stored by the server (800) can store proper nouns in domains other than the domains related to the services and functions provided by the electronic device (100), as well as domains related to the services and functions provided by the electronic device (100).

[0105] If no proper noun exists (S880-N), the server (800) can obtain user intent and slot information for candidate sentences for which intent analysis has been performed (S890). However, if a proper noun exists (S880-Y), the server (800) can replace the proper noun included in the slot based on the proper noun DB stored in the server (800) (S885), and obtain user intent and slot information for candidate sentences based on the replaced slot (S890).

[0106] And, the server (800) can transmit user intent and slot information to the electronic device (100) (S895), and the electronic device (100) can update the proper noun DB (440) and the constrain DB (370) based on the replacement result of the proper noun (S897).

[0108] FIG. 9 is a block diagram illustrating the configuration of an electronic device according to one embodiment of the present disclosure. As shown in FIG. 9, the electronic device (100) may include a communication interface (130), a memory (110), a microphone (140), a display (150), a speaker (160), a sensor (170), and a processor (120). Meanwhile, it is understood that some components of the configuration of the electronic device (100) shown in FIG. 9 may be added or omitted depending on the type of electronic device (100).

[0109] The communication interface (130) is a configuration that performs communication with various types of external devices according to various types of communication methods. The communication interface (130) may include a Wi-Fi module, a Bluetooth module, an infrared communication module, and a wireless communication module. In particular, the processor (120) can perform communication with various external devices using the communication interface (130). At this time, the communication interface (130) can perform communication with an external server for voice recognition, machine translation, and natural language understanding. That is, the communication interface (130) can transmit user voice in the form of voice data to an external server for voice recognition, transmit a first sentence of a first language in the form of text data to an external server for machine translation, and transmit a second sentence of a second language in the form of text data to an external server for natural language understanding. In addition, the communication interface (130) can receive a first sentence of a first language as a result of voice recognition from an external server, a second sentence of a second language as a result of machine translation, and information about control commands, responses, user intentions, and slots as a result of natural language understanding.

[0110] The memory (110) may store commands or data related to at least one other component of the electronic device (100). In particular, the memory (110) may include non-volatile memory and volatile memory, and may be implemented, for example, as flash memory, a hard disk drive (HDD), or a solid-state drive (SSD). The memory (110) is accessed by a processor (120), and the processor (120) may perform read / write / modify / delete / update operations on the data. Additionally, the memory (110) may store an artificial intelligence agent for operating a conversational system. Specifically, the electronic device (100) may use an artificial intelligence agent to generate natural language or obtain control commands in response to user utterances. In this case, the artificial intelligence agent is a dedicated program for providing AI-based services (e.g., voice recognition service, assistant service, translation service, search service, etc.). In particular, the artificial intelligence agent can be executed by a conventional general-purpose processor (e.g., CPU) or a separate AI-dedicated processor (e.g., GPU, NPU, etc.). Additionally, the memory (110) may include a plurality of configurations (or modules) that constitute a conversational system as illustrated in FIG. 10.

[0111] A microphone (140) is configured to receive a user's voice and may be provided within the electronic device (100), but this is merely one embodiment, and may be connected to the electronic device (100) via a wired or wireless connection outside the electronic device (100). In particular, the microphone (140) may receive a user's voice to control the electronic device (100) or an external device around the electronic device (100).

[0112] The display (150) can display images or UI received from the outside. In particular, the display (150) can be implemented as various types of displays such as an LCD (Liquid Crystal Display), an OLED (Organic Light Emitting Diodes) display, or a PDP (Plasma Display Panel). The display (150) may also include a driving circuit, a backlight unit, etc., which can be implemented in forms such as an a-si TFT, an LTPS (low temperature poly silicon) TFT, or an OTFT (organic TFT). Meanwhile, the display (150) can be implemented as a touch screen combined with a touch sensor, a flexible display, a 3D display, etc. Additionally, according to one embodiment of the present invention, the display (150) may include not only a display panel that outputs images but also a bezel that houses the display panel. In particular, according to one embodiment of the present invention, the bezel may include a touch sensor (not shown) for detecting user interaction. In particular, the display (150) can provide a response result to the user's voice in the form of text or an image.

[0113] The speaker (160) may be a component that outputs various audio data received externally, as well as various notification sounds or voice messages. At this time, the electronic device (100) may include an audio output device such as the speaker (160), but may also include an output device such as an audio output terminal. In particular, the speaker (160) may provide response results and operation results for user voice in the form of voice.

[0114] The sensor (170) can acquire various information related to the electronic device (100). In particular, the sensor (170) may include a GPS capable of acquiring location information of the electronic device (100), and may include various sensors such as a bio-sensor for acquiring bio-information of a user using the electronic device (100) (e.g., heart rate sensor, PPG sensor, etc.), and a motion sensor for detecting movement of the electronic device (100).

[0115] In addition, the electronic device (100) may include an input interface for receiving user commands to control the electronic device (100). At this time, the input interface may be implemented as a device such as a button, touch pad, mouse, and keyboard, or as a touch screen capable of performing the aforementioned display function and operation input function. Here, the button may be various types of buttons, such as a mechanical button, touch pad, or wheel, formed in any area such as the front, side, or back of the main body exterior of the electronic device (100).

[0116] The processor (120) is electrically connected to the memory (110) and can control the overall operation of the electronic device (100). In particular, the processor (120) can obtain a first sentence in a first language corresponding to the user's voice through a speech recognition model corresponding to the language of the user's voice by executing at least one command stored in the memory (110) when the user's voice is input. Then, the processor (120) can obtain a second sentence in the second language corresponding to the first sentence in the first language through a machine translation model trained to translate multiple types of languages ​​into a second language. Then, the processor (120) obtains a control command of the electronic device (100) corresponding to the obtained second sentence or obtains a response corresponding to the second sentence through a natural language understanding model trained based on the second language.

[0117] In particular, the processor (120) can perform natural language understanding by linking with an external server based on a second sentence of a second language and a reliability value of the second sentence obtained by a machine translation model. Specifically, the processor (120) can obtain a second sentence and a reliability value of the second sentence by inputting a first sentence into a first machine translation model stored in memory (110). At this time, if the reliability value is greater than or equal to a threshold value, the processor (120) can obtain a control command corresponding to the second sentence or obtain a response corresponding to the second sentence by inputting the second sentence into a first natural language understanding model stored in memory (110).

[0118] In one embodiment, if the confidence value is below a threshold value, the processor (120) may transmit the first sentence to an external server storing a second machine translation model via a communication interface. At this time, the second machine translation model is a machine translation model trained with more data (e.g., a corpus) than the first machine translation module, and is capable of performing more accurate machine translation than the first machine translation model. When a third sentence of the second language obtained through the second machine translation model is received from the external server via a communication interface, the processor (120) may input the received third sentence into the first natural language understanding model to obtain a control command corresponding to the third sentence or obtain a response for the third sentence. At this time, the first machine translation model may be retrained based on the first sentence and the third sentence.

[0119] In another embodiment, if the reliability value is below a threshold, the processor (120) may transmit the first sentence to an external server including a second machine translation model and a second natural language understanding model through a communication interface. At this time, the second natural language understanding model may be a natural language understanding model having improved accuracy and processing speed compared to the first natural language understanding model. The processor (120) may receive a control command corresponding to a third sentence obtained through the second machine translation model and the second natural language understanding model, or a response corresponding to a third sentence, from the external server through a communication interface.

[0120] Additionally, the processor (120) can obtain multiple candidate sentences corresponding to the first sentence and confidence values ​​corresponding to the multiple candidate sentences through a machine translation model stored in memory (110). Then, the processor (120) can determine the intent for the highest priority sentence by inputting the highest priority sentence, which has the highest confidence value among the multiple candidate sentences, into a learned natural language understanding model. If the intent for the highest priority sentence cannot be determined, the processor (120) can determine the intent for the next priority sentence by inputting the next priority sentence, which has a lower confidence value than the highest priority sentence among the multiple candidate sentences, into the learned natural language understanding model. When the intent for the next priority sentence is determined, the processor (120) can output a result of natural language understanding (e.g., user intent and slot) based on the next priority sentence. Furthermore, the processor (120) can map at least one text included in the first sentence and at least one text included in the next priority sentence as input text and modified text, and store them in a first database (or constrain DB) included in the machine translation module. At this time, at least one text included in the first sentence and at least one text included in the next sentence may be of the same language.

[0121] Additionally, if the second sentence contains a proper noun stored in a second database associated with a natural language understanding model, the processor (120) may replace at least one text included in the second sentence with a proper noun of the first language stored in the second database, and obtain a control command of the electronic device (100) corresponding to the replaced second sentence or obtain a response to the second sentence. Then, the processor (120) may map at least one text included in the second sentence and the replaced proper noun to the input text and modified text and store them in the first database. At this time, at least one text included in the second sentence and the replaced proper noun may be of different languages.

[0122] In particular, the artificial intelligence-related functions according to the present disclosure are operated through a processor (120) and a memory (110). The processor (120) may be composed of one or more processors. In this case, the one or more processors may be general-purpose processors such as a CPU, AP, DSP (Digital Signal Processor), graphics-dedicated processors such as a GPU, VPU (Vision Processing Unit), or artificial intelligence-dedicated processors such as an NPU. The one or more processors control the processing of input data according to predefined operation rules or artificial intelligence models stored in the memory (110). Alternatively, if the one or more processors are artificial intelligence-dedicated processors, the artificial intelligence-dedicated processors may be designed with a hardware structure specialized for processing a specific artificial intelligence model.

[0123] The predefined rules of operation or artificial intelligence models are characterized by being created through learning. Here, being created through learning means that a predefined rules of operation or artificial intelligence models configured to perform desired characteristics (or objectives) are created by a basic artificial intelligence model being trained using multiple learning data by a learning algorithm. Such learning may be performed on the device itself where the artificial intelligence according to the present disclosure is executed, or it may be performed through a separate server and / or system. Examples of learning algorithms include supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but are not limited to the examples described above.

[0124] An artificial intelligence model may be composed of multiple neural network layers. Each of the multiple neural network layers has multiple weight values ​​and performs neural network operations through operations between the results of previous layers and the multiple weights. The multiple weights possessed by the multiple neural network layers can be optimized based on the learning results of the artificial intelligence model. For example, the multiple weights may be updated so that the loss value or cost value obtained from the artificial intelligence model during the learning process is reduced or minimized. Artificial neural networks may include deep neural networks (DNNs), such as Convolutional Neural Networks (CNNs), Deep Neural Networks (DNNs), Recurrent Neural Networks (RNNs), Restricted Boltzmann Machines (RBMs), Deep Belief Networks (DBNs), Bidirectional Recurrent Deep Neural Networks (BRDNNs), or Deep Q-Networks, but are not limited to the examples mentioned above.

[0125] FIG. 10 is a block diagram illustrating a conversational system of an artificial intelligence agent system according to one embodiment of the present disclosure. The conversational system (200) illustrated in FIG. 10 is configured to perform a conversation with a virtual artificial intelligence agent through natural language or to control an electronic device (100). According to one embodiment of the present disclosure, a module included in the conversational system (200) may be stored in the memory (110) of the electronic device (100), but this is merely one embodiment and may be implemented in a combined form of hardware and software. Additionally, at least one module included in the conversational system (200) may be included in at least one external server.

[0126] As illustrated in FIG. 10, the conversational system (200) may include a speech input module (210), a speech recognition (SR) module (220), a machine translation (MT) module (230), a natural language understanding (NLU) module (240), a dialogue manager (DM) module (250), an action planner (AP) module (260), a natural language generator (NLG) module (270), a text to speech (TTS) module (280), an output module (290), and a device control module (295).

[0127] The voice input module (210) can receive user voice in the form of voice data. At this time, the voice input module (210) may include a microphone and can receive an analog audio signal containing the user voice through the microphone and convert the analog signal into a digital signal. In one embodiment, the voice input module (210) can obtain voice data corresponding to the user voice by removing noise components from the digital signal. However, according to another embodiment, the voice input module (210) may not remove separate noise components, and noise included in the digital signal may be removed through a voice recognition model. In particular, the user voice received through the voice input module (210) may be in the form of a sentence or phrase containing at least one text, but this is merely one embodiment, and it may include multiple sentences or phrases.

[0128] Meanwhile, the microphone included in the voice input module (210) may be provided inside the electronic device (100), but this is merely one embodiment and may be provided externally and connected to the electronic device (100) via wired or wireless connection. In addition, the voice input module (210) may include a communication module that receives user voice from an external device.

[0129] The voice recognition module (220) can convert user voice in the form of voice data received from the voice input module (210) into text data. At this time, the text data is data composed of character codes, and the voice data may be data containing frequency information regarding the user voice. The voice recognition module (220) may include an acoustic model and a language model. At this time, the acoustic model may include information related to vocalization, and the language model may include information regarding unit phoneme information and combinations of unit phoneme information. The voice recognition module can convert the user voice into text data using information related to vocalization and information regarding unit phoneme information. Information regarding the acoustic model and the language model may be stored, for example, in an automatic speech recognition database (ASR DB). In particular, the voice recognition module (220) may include multiple language models and multiple acoustic models corresponding to multiple languages ​​supported by the electronic device (100). For example, the speech recognition module (220) may include a first language model and a second acoustic model trained with a Korean corpus, a second language model and a second acoustic model trained with an English corpus, and a third language model and a third acoustic model trained with a Japanese corpus, but is not limited thereto.

[0130] In particular, the voice recognition module (220) can obtain a first sentence of a first language corresponding to the user's voice using an acoustic model and a language model. At this time, the first sentence may include at least one text.

[0131] Additionally, the voice recognition module (220) can determine the language of the user's voice through the language determination module (221) described in FIG. 2 and perform voice recognition through a voice recognition model corresponding to the determined language of the user's voice.

[0132] The machine translation module (230) can obtain a second sentence of a predefined second language by translating a first sentence of a first language obtained through the speech recognition module (220). At this time, the machine translation module (230) may include a multilingual neural network translation model. Since the machine translation module (230) has been described in FIGS. 3a and 3b, a redundant description is omitted.

[0133] The natural language understanding module (240) can identify the domain and user intent of the user's voice by performing syntactic analysis or semantic analysis based on the second sentence of the second language. At this time, syntactic analysis can divide the user input into grammatical units (e.g., words, phrases, morphemes, etc.) and identify what grammatical elements the divided units have. Semantic analysis can be performed using semantic matching, rule matching, formula matching, etc. Since the natural language understanding module (230) has been described in FIG. 4, a redundant description is omitted.

[0134] The conversation manager module (250) can provide a response to the user's voice based on the user intent and slot obtained through the natural language understanding module (240). At this time, the conversation manager module (250) can provide a response to the user's voice based on a knowledge base. At this time, the knowledge base may be included within the electronic device (100), but this is merely one embodiment and may be included in an external server.

[0135] Additionally, the conversation manager module (250) can determine whether the user's intent identified by the natural language understanding module (240) is clear. For example, the conversation manager module (250) can determine whether the user's intent is clear based on whether there is sufficient information regarding the slot. Additionally, the conversation manager module (250) can determine whether the slot identified by the natural language understanding module (240) is sufficient for the user to perform the function they intend to perform. According to one embodiment, if the user's intent is unclear, the conversation manager module (250) can provide feedback requesting necessary information from the user.

[0136] The natural language generation module (270) can change specified information into a text form. The information changed into a text form may be in the form of natural language. The specified information may be, for example, information about additional input, information guiding the completion of an action corresponding to user input, information in response to a question, or information guiding the user's additional input (e.g., feedback information regarding user input). The information changed into a text form may be displayed on the display of the electronic device (100) or changed into a speech form by the text-to-speech conversion module (280).

[0137] Additionally, the natural language generation module (270) may include a configuration for machine translation. For example, if the natural language generation module (270) obtains text in a second language, the natural language generation module (270) may obtain text in a first language by translating the text in the second language into text in a first language.

[0138] The text-to-speech conversion module (280) can convert information in text form into information in voice form. The text-to-speech conversion module (280) receives information in the form of text data of the first language from the natural language generation module (270) and can convert the information in the form of text data into information in the form of voice data.

[0139] The output module (290) can output information in the form of voice data received from the text-to-speech conversion module (280). At this time, the output module (290) can output information in the form of voice data through a speaker or a voice output terminal. Alternatively, the output module (290) can output information in the form of text data obtained through the natural language generation module (270) through a display or a video output terminal.

[0140] The action planner module (260) can generate at least one action rule (or path rule) using user intent and slots. For example, the action planner module (260) can generate at least one action rule by determining an application to be executed and an action to be executed in the application based on user intent and slots.

[0141] At this time, the action planner module (260) can determine the application to be executed based on the language of the user's voice input by the voice input module (210) or the current location of the electronic device (100). For example, if the language of the user's voice is Korean, the current location of the electronic device (100) is Korea, and the user's intention is to search, the action planner module (260) can determine a search application provided in Korea as the application to be executed, and if the language of the user's voice is Japanese, the current location of the electronic device is Japan, and the user's intention is to search, the action planner module (260) can determine a search application provided in Japan as the application to be executed.

[0142] The device control module (295) can control the device according to at least one action rule generated by the action planner module (260). For example, the device control module (395) can execute an application corresponding to a user's intention based on at least one action rule and perform a control operation corresponding to an action included in a slot.

[0144] Meanwhile, although the above-described embodiment is described as including a voice recognition module, a machine translation module, and a natural language understanding module inside the electronic device (100), this is merely one embodiment, and at least one of the voice recognition module, the machine translation module, and the natural language understanding module may exist on an external server. For example, the voice recognition module and the machine translation module may be provided in the electronic device (100), and the natural language understanding module may exist on a server.

[0145] FIG. 11 is a sequence diagram illustrating an embodiment of obtaining control commands for a user's voice by linking with an external server according to another embodiment of the present disclosure. At this time, steps S1110 to S1130 of FIG. 11 are identical to steps S510 to S530 of FIG. 5, so a redundant description is omitted.

[0146] The electronic device (100) can transmit a second sentence of a second language to a server (1100) (S1140).

[0147] The server (1100) can obtain a control command of the electronic device (100) corresponding to the second sentence obtained through a natural language understanding model or a response to the second sentence (S1150). That is, the server (1100) can determine a user intent and a slot corresponding to the second sentence through a natural language understanding model that has natural language understanding capabilities for more domains than the natural language understanding model stored in the electronic device (100), and obtain a control command or a response based on the determined user intent and a slot.

[0148] The server (1100) can transmit a control command or response obtained by the electronic device (100) (S1160). According to another embodiment, the server (1100) can transmit information about the user intent and slot corresponding to the second sentence instead of transmitting a control command or response.

[0149] The electronic device (100) can perform an operation or output a response based on a control command received from the server (1100) (S1170).

[0151] Meanwhile, the terms “part” or “module” as used in this disclosure include a unit composed of hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit, for example. A “part” or “module” may be a component formed integrally, or a minimum unit or part thereof that performs one or more functions. For example, a module may be composed of an application-specific integrated circuit (ASIC).

[0152] Various embodiments of the present disclosure may be implemented as software comprising instructions stored on a machine-readable storage medium (e.g., a computer). The machine may include an electronic device (e.g., an electronic device (100)) according to the disclosed embodiments, which is a device capable of calling instructions stored from the storage medium and operating according to the called instructions. When the instructions are executed by a processor, the processor may perform a function corresponding to the instructions directly or using other components under the control of the processor. The instructions may include code generated or executed by a compiler or an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, "non-transitory" means only that the storage medium does not contain a signal and is tangible, and does not distinguish whether data is stored semi-permanently or temporarily on the storage medium.

[0153] According to a few examples, the method according to the various embodiments disclosed herein may be provided as included in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)) or online through an application store (e.g., Play Store™). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily created in a storage medium such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0154] Each component (e.g., module or program) according to various embodiments may consist of a singular or multiple entities, and some of the aforementioned sub-components may be omitted, or other sub-components may be further included in various embodiments. Generally or additionally, some components (e.g., module or program) may be integrated into a single entity to perform the same or similar functions as those performed by each of the respective components prior to integration. The operations performed by the module, program, or other components according to various embodiments may be executed sequentially, in parallel, iteratively, or heuristically, or at least some operations may be executed in a different order, omitted, or other operations added. Explanation of the symbols

[0155] 110: Memory 120: Processor

Claims

Claim 1 An electronic device comprising: a memory storing at least one instruction; and a processor that controls the electronic device by executing at least one instruction stored in the memory; wherein, when a user voice is input, the processor obtains a first sentence of a first language corresponding to the user voice through a speech recognition model corresponding to the language of the user voice, obtains a plurality of candidate sentences corresponding to the first sentence of the first language and confidence values ​​corresponding to the plurality of candidate sentences through a machine translation model trained to translate a plurality of types of languages ​​into a predefined second language, inputs the highest priority sentence among the plurality of candidate sentences with the highest confidence value to a natural language understanding model trained based on the second language to determine the intent for the highest priority sentence, and if the intent for the highest priority sentence cannot be determined, inputs a next-priority sentence among the plurality of candidate sentences with a confidence value smaller than that of the highest priority sentence to the trained natural language understanding model to determine the intent for the next-priority sentence. Claim 2 An electronic device according to claim 1, wherein the memory includes a first machine translation model and a first natural language understanding model, and the processor inputs the first sentence into the first machine translation model to obtain a second sentence and a reliability value of the second sentence, and if the reliability value is greater than or equal to a threshold value, inputs the second sentence into the first natural language understanding model to obtain a control command corresponding to the second sentence or obtain a response to the second sentence. Claim 3 An electronic device according to claim 2 further comprising a communication interface communicating with an external server, wherein the external server includes a second machine translation model learned to translate a plurality of types of languages ​​into a second language, and the processor controls the communication interface to transmit the first sentence to the external server when the reliability value is less than a threshold value, and when a third sentence in the second language obtained through the second machine translation model is received from the external server, the received third sentence is input into the first natural language understanding model to obtain the control command corresponding to the third sentence or to obtain a response to the third sentence. Claim 4 In paragraph 3, the electronic device in which the first machine translation model is retrained based on the first sentence and the third sentence. Claim 5 An electronic device according to claim 3, further comprising a communication interface communicating with an external server; wherein the external server includes a second natural language understanding model learned based on the second language, and the processor controls the communication interface to transmit the first sentence to the external server when the confidence value is less than a threshold value, and receives a control command corresponding to the second text obtained through the second machine translation model and the second natural language understanding model, or a response to the second text, from the external server through the communication interface. Claim 6 delete Claim 7 In claim 1, the electronic device further comprises a first database that maps and stores input text and modified text of the machine translation model, and the processor, when an intention for the next sentence is determined, maps at least one text included in the first sentence and at least one text included in the next sentence as input text and modified text and stores them in the first database. Claim 8 In claim 7, the electronic device includes a second database that stores information about a proper noun stored in the electronic device, and the processor, if a second sentence of the second language includes a proper noun stored in the second database, replaces at least one text included in the second sentence with a proper noun of the first language stored in the second database, and obtains a control command of the electronic device corresponding to the replaced second sentence. Claim 9 In paragraph 8, the processor is an electronic device that maps at least one text included in the second sentence and the substituted proper noun to the input text and modified text and stores them in the first database. Claim 10 delete Claim 11 A control method for an electronic device comprises: a step of, when a user voice is input, obtaining a first sentence of a first language corresponding to the user voice through a speech recognition model corresponding to the language of the user voice; a step of obtaining a plurality of candidate sentences corresponding to the first sentence of the first language and confidence values ​​corresponding to the plurality of candidate sentences through a machine translation model trained to translate a plurality of types of languages ​​into a predefined second language; a step of inputting the highest priority sentence with the highest confidence value among the plurality of candidate sentences into a natural language understanding model trained based on the second language to determine the intent for the highest priority sentence; and, if the intent for the highest priority sentence cannot be determined, a next-priority sentence among the plurality of candidate sentences with a confidence value smaller than that of the highest priority sentence into the trained natural language understanding model to determine the intent for the next-priority sentence. Claim 12 delete Claim 13 delete Claim 14 delete Claim 15 delete Claim 16 delete Claim 17 delete Claim 18 delete Claim 19 delete Claim 20 delete

Images