Electronic Device and Method for Analyzing Speech Recognition Results

Abstract

An electronic device and method for analyzing voice recognition results are disclosed. An electronic device according to various embodiments comprises a display module that provides information to the outside of the electronic device, a processor electrically connected to the display module, and a memory electrically connected to the processor. The processor generates feature information of the text based on text corresponding to a user utterance, determines an output domain for processing the user utterance based on the feature information of the text, identifies an expected domain predetermined by the user, extracts feature information related to the output domain and feature information related to the expected domain from the memory, and can display the feature information related to the output domain and feature information related to the expected domain using the display module.

Description

Technology Field

[0001] The following disclosure relates to an electronic device and method for analyzing voice recognition results. Background Technology

[0002] Recently, with the advancement of speech recognition technology, the utilization of AI-powered speech assistants in electronic devices such as smartphones is increasing. Speech assistants recognize user utterances and can provide the services intended by the user.

[0003] To provide the service intended by the user, user utterances must be recognized, and the appropriate domain for processing the intended service must be determined. For example, when a user requests a search for nearby restaurants, a natural language processing process must determine whether to provide search results from social apps or restaurant search results from map apps.

[0004] However, machine learning-based learning models can be utilized in natural language processing. If the learning model fails to appropriately determine the domain for processing user utterances, developers may face limitations in improving the model's performance because the computational process itself is difficult to analyze. Therefore, technology is required that enables developers to identify the data that influenced the learning model's determination of the appropriate domain for processing user utterances. The problem to be solved

[0005] According to various embodiments disclosed in this document, feature information related to a user utterance can be identified among feature information pre-stored in a domain determined by a learning model for a user utterance, thereby providing an electronic device and method that allow a developer to analyze the results of a learning model.

[0006] According to various embodiments disclosed in this document, an electronic device and method are provided that can analyze the cause of the user-expected domain not being determined by displaying feature information related to user utterances among the feature information of the user-expected domain on a display module. means of solving the problem

[0007] An electronic device according to various embodiments comprises a display module that provides information to the outside of the electronic device, a processor electrically connected to the display module, and a memory electrically connected to the processor. The processor generates feature information of the text based on text corresponding to a user utterance, determines an output domain for processing the user utterance based on the feature information of the text, identifies an expected domain predetermined by the user, extracts feature information related to the output domain and feature information related to the expected domain from the memory, and can display the feature information related to the output domain and feature information related to the expected domain using the display module.

[0008] In an electronic device according to various embodiments, the electronic device comprises: a display module that provides information to the outside of the electronic device; a processor electrically connected to the display module; and a memory electrically connected to the processor, wherein the processor generates feature information of the text based on text corresponding to a user utterance, determines an output domain for processing the user utterance based on the generated feature information, extracts feature information related to the output domain from the memory, and displays the feature information related to the output domain using the display module.

[0009] A method for analyzing speech recognition results according to various embodiments may include: generating feature information of text based on text corresponding to a user utterance; determining an output domain for processing the user utterance based on the feature information of the text; identifying an expected domain predetermined by a user; extracting feature information related to the output domain and feature information related to the expected domain; and displaying feature information related to the output domain and feature information related to the expected domain. Effects of the invention

[0010] According to various embodiments disclosed in this document, feature information related to a user utterance can be identified among feature information pre-stored in a domain determined by the learning model for the user utterance, so that a developer can analyze the results of the learning model.

[0011] According to various embodiments disclosed in this document, by displaying feature information related to user utterances among the feature information of the domain expected by the user on a display module, the cause of the domain expected by the user not being determined can be analyzed. Brief explanation of the drawing

[0012] FIG. 1 is a block diagram of an electronic device in a network environment according to various embodiments. FIG. 2 is a block diagram of a module that analyzes voice recognition results within an electronic device according to one embodiment. FIG. 3 shows text corresponding to a user utterance and classification information of the text according to one embodiment. FIG. 4 is a diagram illustrating an example of feature information of text corresponding to a user utterance according to one embodiment. FIG. 5 is a diagram illustrating an interface for inputting an expected domain according to one embodiment. FIG. 6 is a diagram illustrating an interface that displays feature information of an expected domain and an output domain according to one embodiment. FIG. 7 is a diagram illustrating an example of an interface that displays feature information of an expected domain and an output domain according to one embodiment. FIG. 8 is a graph illustrating the weights of classification information by domain according to one embodiment. FIG. 9 is a diagram illustrating an interface representing the weights and frequencies of feature information for a domain according to one embodiment. FIG. 10 is a flowchart of a method for analyzing voice recognition results according to one embodiment. FIG. 11 is a block diagram showing an integrated intelligence system according to one embodiment. FIG. 12 is a diagram showing the form in which relationship information between a concept and an action is stored in a database according to one embodiment. FIG. 13 is a drawing illustrating a user terminal that displays a screen for processing voice input received through an intelligent app, according to one embodiment. Specific details for implementing the invention

[0013] Hereinafter, embodiments will be described in detail with reference to the attached drawings. In the description with reference to the attached drawings, identical components are given the same reference numeral regardless of the drawing number, and redundant descriptions thereof will be omitted.

[0015] FIG. 1 is a block diagram of an electronic device (101) in a network environment (100) according to various embodiments. Referring to FIG. 1, in the network environment (100), the electronic device (101) may communicate with an electronic device (102) through a first network (198) (e.g., a short-range wireless communication network) or may communicate with at least one of an electronic device (104) or a server (108) through a second network (199) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (101) may communicate with the electronic device (104) through a server (108). According to one embodiment, the electronic device (101) may include a processor (120), memory (130), input module (150), sound output module (155), display module (160), audio module (170), sensor module (176), interface (177), connection terminal (178), haptic module (179), camera module (180), power management module (188), battery (189), communication module (190), subscriber identification module (196), or antenna module (197). In some embodiments, at least one of these components (e.g., connection terminal (178)) may be omitted from the electronic device (101), or one or more other components may be added. In some embodiments, some of these components (e.g., sensor module (176), camera module (180), or antenna module (197)) may be integrated into a single component (e.g., display module (160)).

[0016] The processor (120) can control at least one other component (e.g., hardware or software component) of the electronic device (101) connected to the processor (120) by executing software (e.g., program (140)), and can perform various data processing or operations. According to one embodiment, as at least part of the data processing or operations, the processor (120) can store commands or data received from other components (e.g., sensor module (176) or communication module (190)) in volatile memory (132), process the commands or data stored in volatile memory (132), and store the resulting data in non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., central processing unit or application processor) or an auxiliary processor (123) that can operate independently or together with it (e.g., graphics processing unit, neural processing unit (NPU), image signal processor, sensor hub processor, or communication processor). For example, if the electronic device (101) includes a main processor (121) and an auxiliary processor (123), the auxiliary processor (123) may be configured to use lower power than the main processor (121) or to be specialized for a designated function. The auxiliary processor (123) may be implemented separately from the main processor (121) or as part thereof.

[0017] The auxiliary processor (123) may control at least some of the functions or states associated with at least one component of the electronic device (101) (e.g., display module (160), sensor module (176), or communication module (190)) on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. According to one embodiment, the auxiliary processor (123) (e.g., image signal processor or communication processor) may be implemented as part of another functionally related component (e.g., camera module (180) or communication module (190)). According to one embodiment, the auxiliary processor (123) (e.g., neural network processing unit) may include a hardware structure specialized for processing an artificial intelligence model. The artificial intelligence model may be generated through machine learning. Such learning may be performed, for example, on the electronic device (101) itself where the artificial intelligence model is executed, or through a separate server (e.g., server (108)). The learning algorithm may include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model may include a plurality of artificial neural network layers.An artificial neural network may be a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to the hardware structure, the artificial intelligence model may include a software structure, either additionally or substantially.

[0018] The memory (130) can store various data used by at least one component of the electronic device (101) (e.g., processor (120) or sensor module (176)). The data may include, for example, input data or output data for software (e.g., program (140)) and related commands. The memory (130) may include volatile memory (132) or non-volatile memory (134).

[0019] The program (140) may be stored as software in memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0020] The input module (150) can receive commands or data to be used for a component of the electronic device (101) (e.g., processor (120)) from outside the electronic device (101) (e.g., user). The input module (150) may include, for example, an input module, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0021] The sound output module (155) can output a sound signal to the outside of the electronic device (101). The sound output module (155) may include, for example, a sound output module or a receiver. The sound output module may be used for general purposes, such as multimedia playback or recording playback. The receiver may be used to receive incoming calls. According to one embodiment, the receiver may be implemented separately from the sound output module or as part thereof.

[0022] The display module (160) can visually provide information to an external (e.g., user) of the electronic device (101). The display module (160) may include, for example, a display module, a holographic device, or a projector and a control circuit for controlling said device. According to one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of the force generated by said touch.

[0023] The audio module (170) can convert sound into an electrical signal or, conversely, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150) or output sound through the sound output module (155) or an external electronic device (e.g., electronic device (102)) (e.g., sound output module or headphones) that is directly or wirelessly connected to the electronic device (101).

[0024] The sensor module (176) can detect the operating state of the electronic device (101) (e.g., power or temperature) or the external environmental state (e.g., user state) and generate an electrical signal or data value corresponding to the detected state. According to one embodiment, the sensor module (176) may include, for example, a gesture sensor, a gyroscope sensor, a barometric pressure sensor, a magnetic sensor, an accelerometer sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biosensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0025] The interface (177) may support one or more specified protocols that can be used for the electronic device (101) to be connected directly or wirelessly to an external electronic device (e.g., electronic device (102)). According to one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0026] The connection terminal (178) may include a connector through which the electronic device (101) can be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0027] The haptic module (179) can convert an electrical signal into a mechanical stimulus (e.g., vibration or movement) or an electrical stimulus that the user can perceive through tactile or kinesthetic senses. According to one embodiment, the haptic module (179) may include, for example, a motor, a piezoelectric element, or an electric stimulation device.

[0028] The camera module (180) can capture still images and video. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0029] The power management module (188) can manage the power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented, for example, as at least part of a power management integrated circuit (PMIC).

[0030] The battery (189) can supply power to at least one component of the electronic device (101). According to one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0031] The communication module (190) can support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between an electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may include one or more communication processors that operate independently of the processor (120) (e.g., application processor) and support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., cellular communication module, short-range wireless communication module, or GNSS (global navigation satellite system) communication module) or a wired communication module (194) (e.g., LAN (local area network) communication module, or power line communication module). The corresponding communication module among these communication modules can communicate with an external electronic device (104) through a first network (198) (e.g., a short-range communication network such as Bluetooth, WiFi (wireless fidelity) direct, or IrDA (infrared data association)) or a second network (199) (e.g., a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules may be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can identify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) using subscriber information (e.g., International Mobile Subscriber Identifier (IMSI)) stored in the subscriber identification module (196).

[0032] The wireless communication module (192) can support 5G networks and next-generation communication technologies following 4G networks, for example, new radio access technology. NR access technology can support high-speed transmission of high-capacity data (enhanced mobile broadband (eMBB)), minimization of terminal power and connection of multiple terminals (massive machine type communications (mMTC)), or high reliability and low latency (ultra-reliable and low-latency communications (URLLC)). The wireless communication module (192) can support a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate, for example. The wireless communication module (192) can support various technologies for securing performance in the high-frequency band, such as beamforming, massive MIMO (multiple-input and multiple-output), full-dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large-scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), external electronic device (e.g., electronic device (104)), or network system (e.g., second network (199)). According to one embodiment, the wireless communication module (192) can support a Peak data rate (e.g., 20 Gbps or more) for realizing eMBB, loss coverage (e.g., 164 dB or less) for realizing mMTC, or U-plane latency (e.g., downlink (DL) and uplink (UL) each 0.5 ms or less, or round trip 1 ms or less) for realizing URLLC.

[0033] An antenna module (197) can transmit a signal or power to or from an external source (e.g., an external electronic device). According to one embodiment, the antenna module (197) may include an antenna comprising a radiator made of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). According to one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as a first network (198) or a second network (199), may be selected from the plurality of antennas, for example, by a communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device through the selected at least one antenna. According to some embodiments, in addition to the radiator, other components (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as part of the antenna module (197).

[0034] According to various embodiments, the antenna module (197) may form a mmWave antenna module. According to one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent to a first surface (e.g., bottom surface) of the printed circuit board and capable of supporting a specified high frequency band (e.g., mmWave band), and a plurality of antennas (e.g., array antennas) disposed on or adjacent to a second surface (e.g., top surface or side surface) of the printed circuit board and capable of transmitting or receiving a signal of the specified high frequency band.

[0035] At least some of the above components can be connected to each other via a communication method between peripheral devices (e.g., bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)) and exchange signals (e.g., commands or data) with each other.

[0036] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) through a server (108) connected to a second network (199). Each of the external electronic devices (102, or 104) may be the same or different type of device as the electronic device (101). According to one embodiment, all or part of the operations performed on the electronic device (101) may be performed on one or more of the external electronic devices (102, 104, or 108). For example, if the electronic device (101) needs to perform a function or service automatically or in response to a request from a user or another device, the electronic device (101) may request one or more external electronic devices to perform at least part of the function or service instead of performing the function or service itself or additionally. One or more external electronic devices that receive the above request may execute at least part of the requested function or service, or additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may provide the result as is or additionally processed as at least part of the response to the request. For this purpose, for example, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used. The electronic device (101) may provide ultra-low latency services using, for example, distributed computing or mobile edge computing. In another embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server using machine learning and / or neural networks. According to one embodiment, the external electronic device (104) or the server (108) may be included within a second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0038] FIG. 2 is a block diagram of a module that analyzes voice recognition results within an electronic device according to one embodiment.

[0039] The operations processed in the modules (202-206) illustrated in FIG. 2 can be performed by a processor (120). User speech (201) can be received through an input module (150) (e.g., an input module) of an electronic device (101). User speech (201) can be converted into text through an automatic speech recognition (ASR) module.

[0040] The method of converting user speech (201) into text through automatic speech recognition is not limited to specific examples, and various methods that can be easily adopted by a person skilled in the art of the present invention may be used.

[0041] According to one embodiment, the feature information determination module (202) can generate feature information from text corresponding to a user utterance (201). The feature information may include at least one of classification information, text information, length information, and previous utterance information of words constituting a sentence.

[0042] Text information can be information about a single word of text. Classification information can be information indicating the meaning of the text information. For example, classification information can indicate whether the text information is about a place, time, a restaurant, a particle, a person, an object, etc.

[0043] Length information may be information about the length of a sentence. Length information may be classified according to multiple reference ranges. For example, if the length of the text corresponding to the user utterance (201) is 10 and the reference range is from 5 to 15, the length information of the text corresponding to the user utterance (201) may be determined as 'medium'. Previous utterance information may mean information about the keywords or context of the previously recognized user utterance (201) if there is a previously recognized user utterance (201).

[0044] According to one embodiment, the feature information determination module (202) can divide text corresponding to a user utterance (201) into a plurality of token information. Token information may refer to text in word units that constitute a sentence.

[0045] The feature information determination module (202) can distinguish text corresponding to a user utterance (201) as token information based on token information stored in advance in memory (130). The feature determination module can distinguish text corresponding to a user utterance (201) as token information by comparing text corresponding to the user utterance (201) with token information stored in advance in memory (130).

[0046] According to one embodiment, the feature information determination module (202) can extract feature information from the text corresponding to the user utterance (201) by comparing the feature information stored in advance in the memory (130) with the text corresponding to the user utterance (201).

[0047] The feature information determination module (202) can generate feature information of text corresponding to a user utterance (201) by dividing text corresponding to a user utterance (201) into a plurality of token informations and extracting feature information associated with token information of text corresponding to a user utterance (201) from among the feature information stored in memory (130).

[0048] According to one embodiment, the memory (130) can store sentences used for training an artificial intelligence model by domain and can store feature information for each sentence. According to one embodiment, a plurality of feature information can be stored separately by domain. Identical feature information may be included between domains.

[0049] The output domain determination module (203) can determine the output domain from feature information of text corresponding to the user utterance (201). The output domain may refer to a domain related to the user utterance (201) or a domain that processes the user utterance (201). The output domain determination module (203) can determine the output domain through a natural language processing process of the user utterance (201).

[0050] For example, if the text corresponding to the user utterance (201) is “Find hotels nearby seoul”, the feature information determination module (202) can divide the text corresponding to the user utterance (201) into multiple token information (e.g., 'Find', 'hotels', 'nearby', 'seoul'). The feature information determination module (202) can determine feature information (e.g., [text: find], [text: hotels], [text: nearby], [location: seoul], [length: medium]) based on each token information.

[0051] The output domain determination module (203) can determine the output domain using feature information. For example, if the text corresponding to the user utterance (201) is “Find hotels nearby seoul”, the output domain determination module (203) can determine the output domain as a domain capable of processing hotel search through natural language processing.

[0052] The output domain determination module (203) can determine the output domain from the feature information of the text corresponding to the user utterance (201) using a learning model trained to determine the output domain from the text corresponding to the user utterance (201).

[0053] A learning model for determining an output domain from text corresponding to a user utterance (201) may be a deep learning-based neural network model, a rule-based model, or a neural network model (e.g., a feedforward neural network (FNN), a recurrent neural network (RNN), or a convolutional neural network). The learning model is not limited to specific examples, and various methods that can be easily adopted by a person skilled in the art of the present invention may be used.

[0054] The feature information determination module (205) can identify an expected domain (204) determined by the user. The expected domain (204) may refer to a domain intended by the user for a user utterance (201) and may be input through an input module (150) of the electronic device (101) (e.g., keyboard, input module) and a display module (160) including a touch sensor.

[0055] The feature information determination module (205) can extract pre-stored feature information for each of the expected domain (204) and the output domain. The feature determination module (205) can extract sentences and feature information of those sentences that are pre-stored for the same domain as the expected domain (204) from the memory (130). The feature determination module (205) can extract sentences and feature information of those sentences that are pre-stored for the same domain as the output domain from the memory (130).

[0056] The similarity determination module (206) can calculate a first similarity between feature information of the output domain and feature information of text corresponding to user utterance (201), and a second similarity between feature information of the expected domain (204) and feature information corresponding to user utterance (201). Similarity may be a numerical value indicating the degree of similarity between feature information.

[0057] The similarity determination module (206) can determine a first similarity for each of the feature information of the output domain. For example, the similarity determination module (206) can determine the first similarity by comparing the classification information included in the feature information of the text with the classification information included in the feature information related to the output domain.

[0058] The similarity determination module (206) can determine a first similarity for the output domain based on the number of classification information included in the feature information of the text corresponding to the user utterance (201) among the classification information included in the feature information of the output domain.

[0059] The similarity determination module (206) can determine a second similarity for each of the feature information of the expected domain (204). For example, the similarity determination module (206) can determine the second similarity by comparing the classification information included in the feature information of the text with the classification information included in the feature information related to the expected domain (204).

[0060] The similarity determination module (206) can determine a second similarity for the feature information of the expected domain (204) based on the number of classification information included in the feature information of the text corresponding to the user utterance (201) among the classification information included in the feature information of the expected domain (204).

[0061] According to one embodiment, the similarity determination module (206) can determine a first similarity or a second similarity using the following mathematical formula 1.

[0062]

[0063] In mathematical formula 1, A may mean a set of information included in the feature information of the output domain or expected domain (204) (e.g., classification information, length information, at least one of the above). B may mean a set of information included in the feature information of the text corresponding to the user utterance (201) (e.g., classification information, length information, at least one of the above). J(A, B) may mean the similarity between A and B.

[0064] The similarity determination module (206) can determine a first similarity or a second similarity based on information that is common among i) information included in the feature information of the text corresponding to the user utterance (201) and information included in the output domain or ii) information included in the feature information of the expected domain (204).

[0065] The similarity determination module (206) can determine a first similarity or a second similarity based on common information between the information included in the feature information of the output domain or expected domain (204) and the information included in the feature information of the text corresponding to the user utterance (201).

[0066] The similarity determination module (206) can determine a first similarity or a second similarity by considering the weights of classification information, text information, length information, and previous utterance information included in the feature information of the output domain and the feature information of the expected domain (204).

[0067] For example, the first similarity when the classification information with a high weight among the classification information included in the feature information of the output domain is the same as the classification information included in the feature information of the text corresponding to the user utterance (201) can be determined to be higher than the first similarity when the classification information with a low weight among the classification information included in the feature information of the output domain is the same as the classification information included in the feature information of the text corresponding to the user utterance (201).

[0068] Weights can be predetermined for each feature information stored in memory (130). Even for the same feature information, weights can be determined differently for each domain. Weights can be determined as higher values ​​as the correlation with the domain increases. The weights of feature information for each domain can be predetermined by the user.

[0069] According to one embodiment, among the classification information included in the feature information of a specific domain, the higher the frequency of inclusion in the feature information, the higher the weight of the classification information may be determined. According to one embodiment, among the text information included in the feature information of a specific domain, the higher the frequency of inclusion in the feature information, the higher the weight of the text information may be determined.

[0070] According to one embodiment, feature information of an output domain and feature information of an expected domain (204) may be included in a display module (160). A first similarity to the feature information of the output domain or a second similarity to the feature information of the expected domain (204) may be displayed on the display module (160).

[0071] FIG. 3 shows text corresponding to a user utterance and classification information of the text according to one embodiment.

[0072] FIG. 3 (A) is a diagram illustrating an example in which a user utterance (e.g., user utterance (201) of FIG. 2) is converted into text through automatic speech recognition and separated into multiple token information (301). FIG. 3 (B) is a diagram illustrating an example of classification information extracted based on the token information (301) included in FIG. 3 (A).

[0073] For example, as in (A) of FIG. 3, if the text corresponding to the user utterance is “find a flight arriving at Incheon Airport sometime tomorrow,” then ‘sometimes’, ‘sometimes’, ‘at’, ‘Incheon’, ‘airport’, ‘at’, ‘arrival’, ‘ha’, ‘ne’, ‘flight’, ‘flight’, ‘ship’, ‘find’, ‘a’, ‘give’ may each be token information (301).

[0074] For each token information (301), classification information stored in memory (130) can be extracted. For example, for 'airport', a place (e.g., geo.PlaceType) and an airport (e.g., flight.Airport) can be determined as classification information, as shown in (B) of FIG. 3.

[0075] FIG. 4 is a diagram illustrating an example of feature information of text corresponding to a user utterance according to one embodiment.

[0076] FIG. 4 shows token information (401) of text corresponding to a user utterance (e.g., user utterance (201) of FIG. 2) and feature information of text corresponding to a user utterance. 'Tomorrow', 'among', 'at', 'Incheon', 'airport', 'at', 'arrival', 'do', 'is', 'airline', 'flight', 'convenience', 'find', 'a', 'give' may each be token information (401).

[0077] Previous utterance information (402) may be information about keywords or context of a previously recognized user utterance. For example, if the previously recognized user utterance is an utterance related to flight, the previous utterance information (402) may be flight (e.g., flight context) as shown in FIG. 4. When a user utterance is input, the processor (120) can perform natural language processing and update the previous utterance information (402).

[0078] Classification information (403) may be information indicating the meaning of token information (401). Classification information (403) of text corresponding to a user utterance may be determined according to classification information (403) that is pre-stored in memory (130) for each token information (401). Text information (404) may mean information about text corresponding to a user utterance. Text information (404) may include token information (401).

[0079] Length information (405) may be information indicating the length of text corresponding to a user utterance. If the length of the text corresponding to the user utterance falls within a certain standard range, an index corresponding to that standard range may be determined as length information (405). For example, if the length of the text is 10 or more and 20 or less, the length information (405) may be determined as 'Long' (e.g., LONG-LEN).

[0080] FIG. 5 is a diagram illustrating an interface for inputting an expected domain according to one embodiment.

[0081] The user can input an intended expected domain (502) through the graphical user interface (GUI) of FIG. 5. The GUI interface may be displayed by a display module (160) of an electronic device (101). The GUI interface may include text (504) corresponding to a user utterance (501) recognized through automatic speech recognition (e.g., user utterance (201) of FIG. 2).

[0082] According to one embodiment, a user can select or input an expected domain (502) (e.g., the expected domain (204) of FIG. 2) on an interface. Referring to FIG. 5, the user can select, directly input, or input the expected domain (502) using an input interface (505).

[0083] The GUI interface may include an input interface (506) that can select the type (503) of the expected domain (502). The user can determine the type (503) of the user utterance (501) through the input interface (506). When the type (503) of the user utterance (501) is determined, the processor (120) can extract feature information containing classification information associated with the type (503) of the user utterance (501) among the classification information of the expected domain (502).

[0084] For example, as shown in FIG. 5, for a user utterance (501) (e.g., "Find me a flight arriving at Incheon Airport tomorrow"), if the expected domain (502) is determined to be 'Expedia', an airline-related reservation application, and the type (503) is determined to be 'travel', the processor (120) can extract feature information for 'Expedia' and extract feature information that includes classification information related to 'travel' among the classification information of the feature information.

[0085] When the execution button (507) of the GUI interface illustrated in FIG. 5 is input, the processor (120) can extract sentences and feature information that have been stored in advance for the expected domain (502). The processor (120) can determine the feature information with the highest similarity by comparing the extracted feature information with the feature information extracted from the user utterance (501).

[0086] FIG. 6 is a diagram illustrating an interface that displays feature information of an expected domain and an output domain according to one embodiment.

[0087] Through the GUI interface of FIG. 6, the developer can analyze the feature information that influenced the output of the learning model by identifying the feature information of the output domain (621) determined by the learning model performing natural language processing that has a high similarity to the feature information of the user utterance (e.g., user utterance (201) of FIG. 2) and the feature information of the prediction domain (611) (e.g., prediction domain (611) (204) of FIG. 2) that has a high similarity to the feature information of the user utterance.

[0088] The GUI interface of FIG. 6 may be displayed by a display module (160) of an electronic device (101) and may include an interface (600) for user speech, an interface (610) for an expected domain (611), and an interface (620) for an output domain (621). Data included in the GUI interface of FIG. 6 may be arranged in various forms and are not limited to the illustrated examples.

[0089] According to one embodiment, in the GUI interface of FIG. 6, the interface (610) for the expected domain (611) may not be displayed and may be displayed as the interface (620) for the output domain (621) through a separate input process.

[0090] The interface (600) for a user utterance may include text (601) corresponding to the user utterance and classification information (602) determined based on the text (601) corresponding to the user utterance. The interface (600) for a user utterance may include other feature information in addition to the classification information of the text (601) corresponding to the user utterance.

[0091] The interface (610) for the expected domain (611) may include the expected domain (611) determined by the user. The interface (610) for the expected domain (611) may include feature information (612, 616) for the expected domain (611). The interface (610) for the expected domain (611) may include a first similarity (615, 619) between each feature information (612, 616) and the feature information of the text (601) corresponding to the user utterance.

[0092] The feature information (612, 616) displayed on the interface (610) for the expected domain (611) may be the feature information (612, 616) that has the highest first similarity (615, 619) with the feature information of the text (601) corresponding to the user utterance among the feature information stored in advance for the expected domain (611). The processor (120) may display the feature information on the display module (160) in order of highest first similarity (615, 619) with the feature information of the text (601) corresponding to the user utterance among the feature information related to the output domain (621).

[0093] Feature information (612) may include a sentence (613) corresponding to the feature information (613) and classification information (614). Among the classification information of feature information related to the expected domain (611), the higher the correlation with the expected domain (611), the higher the weight of the classification information may be determined. The processor (120) may display classification information (614) that has a weight higher than the standard among the classification information included in the feature information related to the domain on the display module (160) so as to distinguish it from other classification information (614).

[0094] For example, the processor (120) can display the classification information (614) so ​​that the magnitude of the weight is distinguished by increasing the shading of the classification information (614) as the weight increases. The feature information (616) may include a sentence (617) corresponding to the feature information (616) and the classification information (618).

[0095] The feature information (622) displayed on the interface (620) for the output domain (621) may be the feature information (622) that has the highest second similarity (625) with the feature information of the text (601) corresponding to the user utterance among the feature information stored in advance for the output domain (621). The processor (120) may display the feature information on the display module (160) in order of highest second similarity with the feature information of the text (601) corresponding to the user utterance among the feature information related to the output domain (621).

[0096] Feature information (622) may include a sentence (623) corresponding to the feature information (623) and classification information (624). Among the classification information of feature information related to the output domain (621), the higher the correlation with the output domain (621), the higher the weight of the classification information may be determined. The processor (120) may display classification information (624) with a weight higher than the standard among the classification information included in the feature information related to the domain on the display module (160) so as to distinguish it from other classification information.

[0097] FIG. 7 is a diagram illustrating an example of an interface that displays feature information of an expected domain and an output domain according to one embodiment.

[0098] The GUI interface (700) illustrated in FIG. 7 can be displayed through a display module (160). The GUI interface (700) may include at least one of i) text (701) corresponding to a user utterance, feature information (702) extracted from the text (701) corresponding to the user utterance, an expected domain (704), feature information (705, 706) among the feature information related to the expected domain that has a similarity to the feature information (702) that is greater than or equal to a standard, an output domain (707), and feature information (708, 709) among the feature information related to the output domain that has a similarity to the feature information (702) that is greater than or equal to a standard.

[0099] For example, referring to FIG. 7, if the text (701) corresponding to the user utterance is "Find me a flight arriving at Incheon Airport tomorrow," classification information (702) for 'tomorrow', 'Incheon', 'airport', and 'flight' can be extracted as feature information (702). The user's expected domain (704) is 'Expedia', but the output domain (707) determined by the learning model can be 'Hana Tour'.

[0100] The processor (120) can analyze which feature information caused the learning model to output the output domain (707) instead of the expected domain (704) by displaying the feature information (705, 706, 708, 709) that has the highest similarity to the user utterance among the feature information stored in advance as training data of the learning model in relation to the output domain (707) and the expected domain (704) through the display module (160).

[0101] FIG. 8 is a graph illustrating the weights of classification information by domain according to one embodiment.

[0102] The processor (120) can display a graph showing how the weights of classification information previously stored in memory (130) are determined by domain through the display module (160). The processor (120) can determine the weights of feature information, such as classification information, by domain according to user guidelines or frequency.

[0103] The processor (120) can extract domain-specific weights of pre-stored classification information and display them as a graph on the display module (160). For example, the weights of classification information such as 'travel', 'date', and 'hotel' may be determined to be higher in the domain for booking flights or accommodations than in the domain related to finance. Through the graph shown in FIG. 8, the developer can check how the weights of classification information are determined according to the domain.

[0104] FIG. 9 is a diagram illustrating an interface representing the weights and frequencies of feature information for a domain according to one embodiment.

[0105] FIG. 9 is a graph illustrating classification information and the frequency of classification information included in feature information stored in advance for a specific domain, domain A (901). The feature information can be stored in advance in memory (130) as feature information associated with domain A (901).

[0106] Referring to Fig. 9 (A), the frequency of classification information included in sentences related to domain A (901) is shown as a graph. Referring to Fig. 9 (A), the frequency of classification information included in sentences related to domain A (901) is shown as a graph.

[0107] Figure 9 (B) shows classification information related to domain A (901) and weights of classification information for domain A (901). The processor (120) can extract classification information and weights of classification information that are stored in advance for a specific domain and display the frequency or weights of classification information through the display module (160).

[0108] The developer can analyze which classification information is most frequently included in the training data in relation to domain A (901) and which classification information is determined to have a high weight.

[0109] FIG. 10 is a flowchart of a method for analyzing voice recognition results according to one embodiment.

[0110] In operation (1001), the processor (120) can generate feature information based on text corresponding to user utterance. The processor (120) can obtain text corresponding to user utterance by converting user utterance received through automatic speech recognition into text. The processor (120) can determine feature information of text corresponding to user utterance based on feature information stored in advance in memory (130).

[0111] In operation (1002), the processor (120) can determine an output domain for processing user utterances based on feature information of text. The processor (120) can determine the output domain by inputting feature information of text corresponding to user utterances into a learning model trained to determine a domain for processing user utterances from text.

[0112] In operation (1003), the processor (120) can extract feature information related to the output domain from the memory (130). The memory (130) can store feature information used for training the learning model for each domain. The processor (120) can extract feature information that is pre-stored for the output domain from the memory (130). The processor (120) can identify the expected domain determined by the user and extract feature information that is pre-stored for the expected domain from the memory (130).

[0113] In operation (1004), the processor (120) can display feature information related to the output domain using the display module (160). The processor (120) can display at least one of i) text corresponding to a user utterance, ii) feature information of the text corresponding to the user utterance, iii) feature information of the output domain, iv) feature information of the expected domain, v) a first similarity between the feature information of the output domain and the feature information of the text corresponding to the user utterance, and vii) a second similarity between the feature information of the expected domain and the feature information of the text corresponding to the user utterance on the display module (160).

[0114] The processor (120) can display feature information of the output domain in order of first high similarity and feature information of the expected domain in order of second high similarity. The processor (120) can display classification information with a weight higher than a standard among the classification information included in the feature information related to the expected domain or the output domain on the display module (160).

[0115] According to one embodiment, the processor (120) may extract feature information regarding an output domain without receiving a separate expected domain from the user, determine feature information with a first similarity to the feature information of text corresponding to the user's utterance among the feature information related to the output domain, and display it on the display module (160). In determining the first similarity, the processor (120) may determine the first similarity by considering a predetermined weight for the information included in the feature information.

[0116] FIG. 11 is a block diagram showing an integrated intelligence system according to one embodiment.

[0117] Referring to FIG. 11, an integrated intelligent system of one embodiment may include a user terminal (101), an intelligent server (1100), and a service server (1191).

[0118] The user terminal (101) of one embodiment may be a terminal device (or electronic device) capable of connecting to the internet, and may be, for example, a mobile phone, a smartphone, a PDA (personal digital assistant), a laptop computer, a TV, a white goods appliance, a wearable device, an HMD, or a smart sound output module.

[0119] According to the illustrated embodiment, the user terminal (101) may include an interface (177), an input module (150), an audio output module (155), a display module (160), a memory (130), or a processor (120). The listed components may be operatively or electrically connected to each other.

[0120] The interface (177) of one embodiment may be configured to be connected to an external device to transmit and receive data. The input module (150) of one embodiment may receive sound (e.g., user speech) and convert it into an electrical signal. The sound output module (155) of one embodiment may output the electrical signal as sound (e.g., voice). The display module (160) of one embodiment may be configured to display an image or video. The display module (160) of one embodiment may also display a graphic user interface (GUI) of an app (or application program) being executed.

[0121] In one embodiment, the memory (130) can store a client module (144-2), an SDK (software development kit) (144-1), and a plurality of apps (155). The client module (144-2) and the SDK (144-1) can form a framework (or solution program) for performing general-purpose functions. Additionally, the client module (144-2) or the SDK (144-1) can form a framework for processing voice input.

[0122] In one embodiment, the memory (130) may be a program for performing a designated function, wherein the plurality of apps (146) are programs. According to one embodiment, the plurality of apps (146) may include a first app (146_1) and a second app (146_3). According to one embodiment, each of the plurality of apps (146) may include a plurality of operations for performing a designated function. For example, the apps may include an alarm app, a message app, and / or a schedule app. According to one embodiment, the plurality of apps (146) may be executed by a processor (120) to sequentially execute at least some of the plurality of operations.

[0123] A processor (120) of one embodiment can control the overall operation of a user terminal (101). For example, the processor (120) can be electrically connected to an interface (177), an input module (150), an audio output module (155), and a display module (160) to perform a specified operation.

[0124] The processor (120) of one embodiment may also perform a specified function by executing a program stored in the memory (130). For example, the processor (120) may execute at least one of the client module (144-2) or the SDK (144-1) to perform the following operations for processing voice input. The processor (120) may, for example, control the operation of a plurality of apps (155) through the SDK (144-1). The following operations described as the operation of the client module (144-2) or the SDK (144-1) may be operations performed by the execution of the processor (120).

[0125] A client module (144-2) of one embodiment can receive voice input. For example, the client module (144-2) can receive a voice signal corresponding to a user utterance detected through an input module (150). The client module (144-2) can transmit the received voice input to an intelligent server (1100). Along with the received voice input, the client module (144-2) can transmit status information of the user terminal (101) to the intelligent server (1100). The status information may be, for example, execution status information of an app.

[0126] A client module (144-2) of one embodiment can receive a result corresponding to a received voice input. For example, if the client module (144-2) can produce a result corresponding to the received voice input from an intelligent server (1100), the client module (144-2) can receive a result corresponding to the received voice input. The client module (144-2) can display the received result on a display module (160).

[0127] In one embodiment, the client module (144-2) can receive a plan corresponding to the received voice input. The client module (144-2) can display the results of executing multiple actions of the app according to the plan on the display module (160). The client module (144-2) can, for example, display the results of executing multiple actions sequentially on the display module. The user terminal (101) can, for another example, display only some of the results of executing multiple actions (e.g., the result of the last action) on the display module.

[0128] According to one embodiment, the client module (144-2) may receive a request from the intelligent server (1100) to obtain information necessary to produce a result corresponding to a voice input. According to one embodiment, the client module (144-2) may transmit the necessary information to the intelligent server (1100) in response to the request.

[0129] The client module (144-2) of one embodiment can transmit result information of executing multiple operations according to a plan to the intelligent server (1100). The intelligent server (1100) can use the result information to confirm that the received voice input has been processed correctly.

[0130] A client module (144-2) of one embodiment may include a voice recognition module. According to one embodiment, the client module (144-2) can recognize voice input that performs a limited function through the voice recognition module. For example, the client module (144-2) can perform an intelligent app for processing voice input to perform an organic action through a specified input (e.g., Wake Up!).

[0131] An intelligent server (1100) of one embodiment can receive information related to user voice input from a user terminal (101) via a communication network. According to one embodiment, the intelligent server (1100) can convert the data related to the received voice input into text data. According to one embodiment, the intelligent server (1100) can generate a plan for performing a task corresponding to the user voice input based on the text data.

[0132] According to one embodiment, a plan may be generated by an artificial intelligence (AI) system. The AI ​​system may be a rule-based system or a neural network-based system (e.g., a feedforward neural network (FNN), a recurrent neural network (RNN)). Alternatively, it may be a combination of the foregoing or a different AI system. According to one embodiment, the plan may be selected from a set of predefined plans or may be generated in real time in response to a user request. For example, the AI ​​system may select at least one plan from a plurality of predefined plans.

[0133] An intelligent server (1100) of one embodiment may transmit a result according to a generated plan to a user terminal (101) or transmit a generated plan to a user terminal (101). According to one embodiment, the user terminal (101) may display a result according to a plan on a display module. According to one embodiment, the user terminal (101) may display a result of executing an action according to a plan on a display module.

[0134] An intelligent server (1100) of one embodiment may include a front end (1110), a natural language platform (1120), a capsule database (1130), an execution engine (1160), an end user interface (1150), a management platform (1150), a big data platform (1170), or an analytic platform (1180).

[0135] A front end (1110) of one embodiment can receive voice input received from a user terminal (101). The front end (1110) can transmit a response corresponding to the voice input.

[0136] According to one embodiment, the natural language platform (1120) may include an automatic speech recognition module (ASR module) (1121), a natural language understanding module (NLU module) (1123), a planner module (1125), a natural language generator module (NLG module) (1127), or a text to speech module (TTS module) (1129).

[0137] An automatic speech recognition module (1121) of one embodiment can convert voice input received from a user terminal (101) into text data. A natural language understanding module (1123) of one embodiment can identify the user's intention using the text data of the voice input. For example, the natural language understanding module (1123) can identify the user's intention by performing a syntactic analysis or a semantic analysis. A natural language understanding module (1123) of one embodiment can identify the meaning of a word extracted from the voice input using linguistic features (e.g., grammatical elements) of a morpheme or phrase, and determine the user's intention by matching the identified meaning of the word to the intention.

[0138] A planner module (1125) of one embodiment can generate a plan using intentions and parameters determined by a natural language understanding module (1123). According to one embodiment, the planner module (1125) can determine multiple domains necessary to perform a task based on the determined intention. The planner module (1125) can determine multiple actions included in each of the multiple domains determined based on the intention. According to one embodiment, the planner module (1125) can determine parameters necessary to execute the determined multiple actions or result values ​​output by the execution of the multiple actions. The parameters and result values ​​may be defined as concepts of a specified format (or class). Accordingly, the plan may include multiple actions and multiple concepts determined by the user's intention. The planner module (1125) can determine the relationship between the multiple actions and the multiple concepts in a stepwise (or hierarchical) manner. For example, the planner module (1125) can determine the execution order of multiple actions determined based on the user's intentions based on multiple concepts. In other words, the planner module (1125) can determine the execution order of multiple actions based on parameters required for the execution of multiple actions and results output by the execution of multiple actions. Accordingly, the planner module (1125) can generate a plan that includes association information (e.g., ontology) between multiple actions and multiple concepts. The planner module (1125) can generate the plan using information stored in a capsule database (1130) in which a set of relationships between concepts and actions is stored.

[0139] A natural language generation module (1127) of one embodiment can change specified information into a text form. The information changed into a text form may be in the form of a natural language utterance. A text-to-speech conversion module (1129) of one embodiment can change information in a text form into information in a speech form.

[0140] According to one embodiment, some or all functions of the natural language platform (1120) may also be implemented on a user terminal (101).

[0141] The capsule database (1130) can store information regarding the relationships between multiple concepts and actions corresponding to multiple domains. A capsule according to one embodiment may include multiple action objects (or action information) and concept objects (or concept information) included in a plan. According to one embodiment, the capsule database (1130) may store multiple capsules in the form of a CAN (concept action network). According to one embodiment, multiple capsules may be stored in a function registry included in the capsule database (1130).

[0142] The above capsule database (1130) may include a strategy registry in which strategy information necessary for determining a plan corresponding to a voice input is stored. The strategy information may include reference information for determining one plan when there are multiple plans corresponding to the voice input. According to one embodiment, the capsule database (1130) may include a follow-up registry in which information of a follow-up action is stored for suggesting a follow-up action to a user in a specified situation. The follow-up action may include, for example, a follow-up utterance. According to one embodiment, the capsule database (1130) may include a layout registry in which layout information of information output through a user terminal (101) is stored. According to one embodiment, the capsule database (1130) may include a vocabulary registry in which vocabulary information included in the capsule information is stored. According to one embodiment, the capsule database (1130) may include a dialog registry in which information regarding a conversation (or interaction) with a user is stored. The capsule database (1130) may update stored objects through a developer tool. The developer tool may include, for example, a function editor for updating action objects or concept objects. The developer tool may include a vocabulary editor for updating vocabulary. The developer tool may include a strategy editor for creating and registering strategies for determining plans. The developer tool may include a dialog editor for creating conversations with a user.The above developer tool may include a follow-up editor capable of activating a follow-up goal and editing a follow-up utterance that provides a hint. The follow-up goal may be determined based on a currently set goal, user preferences, or environmental conditions. In one embodiment, the capsule database (1130) may also be implemented within the user terminal (101).

[0143] An execution engine (1160) of one embodiment can produce a result using the generated plan. An end-user interface (1150) can transmit the produced result to a user terminal (101). Accordingly, the user terminal (101) can receive the result and provide the received result to the user. A management platform (1150) of one embodiment can manage information used in the intelligent server (1100). A big data platform (1170) of one embodiment can collect user data. An analysis platform (1180) of one embodiment can manage the quality of service (QoS) of the intelligent server (1100). For example, the analysis platform (1180) can manage the components and processing speed (or efficiency) of the intelligent server (1100).

[0144] A service server (1191) of one embodiment may provide a service designated to a user terminal (101), such as ordering food or making a hotel reservation. According to one embodiment, the service server (1191) may be a server operated by a third party. A service server (1191) of one embodiment may provide information to an intelligent server (1100) for generating a plan corresponding to a received voice input. The provided information may be stored in a capsule database (1130). Additionally, the service server (1191) may provide result information according to the plan to the intelligent server (1100).

[0145] In the integrated intelligent system described above, the user terminal (101) can provide various intelligent services to the user in response to user input. The user input may include, for example, input via a physical button, touch input, or voice input.

[0146] In one embodiment, the user terminal (101) may provide a voice recognition service through an intelligent app (or voice recognition app) stored internally. In this case, for example, the user terminal (101) may recognize a user utterance or voice input received through the input module and provide a service to the user corresponding to the recognized voice input.

[0147] In one embodiment, the user terminal (101) may perform a specified action, either alone or together with the intelligent server and / or service server, based on the received voice input. For example, the user terminal (101) may execute an app corresponding to the received voice input and perform a specified action through the executed app.

[0148] In one embodiment, when a user terminal (101) provides services together with an intelligent server (1100) and / or a service server, the user terminal can detect a user utterance using the input module (150) and generate a signal (or voice data) corresponding to the detected user utterance. The user terminal can transmit the voice data to the intelligent server (1100) using an interface (177).

[0149] An intelligent server (1100) according to one embodiment may generate a plan for performing a task corresponding to a voice input, or a result of performing an operation according to the plan, as a response to a voice input received from a user terminal (101). The plan may include, for example, a plurality of operations for performing a task corresponding to a user's voice input, and a plurality of concepts related to the plurality of operations. The concepts may define parameters input to the execution of the plurality of operations or result values ​​output by the execution of the plurality of operations. The plan may include association information between the plurality of operations and the plurality of concepts.

[0150] A user terminal (101) of one embodiment can receive the response using an interface (177). The user terminal (101) can output a voice signal generated inside the user terminal (101) to the outside using the sound output module (155), or output an image generated inside the user terminal (101) to the outside using the display module (160).

[0151] FIG. 12 is a diagram showing the form in which relationship information between concepts and operations is stored in a database according to various embodiments.

[0152] The capsule database (e.g., capsule database (1130)) of the intelligent server (1100) can store capsules in the form of a CAN (concept action network). The capsule database can store actions for processing tasks corresponding to the user's voice input, and parameters required for said actions, in the form of a CAN (concept action network).

[0153] The above capsule database may store a plurality of capsules (capsule(A)(1201), capsule(B)(1204)) corresponding to each of a plurality of domains (e.g., application). According to one embodiment, one capsule (e.g., capsule(A)(1201)) may correspond to one domain (e.g., location (geo), application). Additionally, one capsule may correspond to at least one service provider (e.g., CP 1 (1202) or CP 2 (1203)) for performing functions for the domain associated with the capsule. According to one embodiment, one capsule may include at least one operation (1210) and at least one concept (1220) for performing a specified function.

[0154] The above natural language platform (1120) can generate a plan for performing a task corresponding to a received voice input using capsules stored in a capsule database. For example, the planner module (1125) of the natural language platform can generate a plan using capsules stored in a capsule database. For example, a plan (1207) can be generated using the actions (12011, 12013) and concepts (12012, 12014) of capsule A (1201) and the actions (12041) and concepts (12042) of capsule B (1204).

[0155] FIG. 13 is a diagram showing a screen in which a user terminal according to various embodiments processes voice input received through an intelligent app.

[0156] The user terminal (101) can run an intelligent app to process user input through an intelligent server (1100).

[0157] According to one embodiment, on a screen 1310, the user terminal (101) can execute an intelligent app for processing voice input when it recognizes a designated voice input (e.g., Wake up!) or receives input via a hardware key (e.g., a dedicated hardware key). The user terminal (101) can execute the intelligent app while, for example, a schedule app is running. According to one embodiment, the user terminal (101) can display an object (e.g., an icon) (1311) corresponding to the intelligent app on the display module (160). According to one embodiment, the user terminal (101) can receive voice input by user speech. For example, the user terminal (101) can receive voice input saying, “Tell me this week’s schedule!” According to one embodiment, the user terminal (101) can display a UI (user interface) (1313) (e.g., an input window) of the intelligent app on the display module, in which text data of the received voice input is displayed.

[0158] According to one embodiment, on a 1320 screen, the user terminal (101) can display a result corresponding to the received voice input on the display module. For example, the user terminal (101) can receive a plan corresponding to the received user input and display 'this week's schedule' on the display module according to the plan.

[0160] In an electronic device (101) according to various embodiments, the electronic device (101) includes a display module (160) that provides information to the outside of the electronic device (101), a processor (120) electrically connected to the display module (160), and a memory (130) electrically connected to the processor (120). The processor (120) generates feature information of the text based on text corresponding to a user utterance, determines an output domain that processes the user utterance based on the feature information of the text, identifies an expected domain that is predetermined by the user, extracts feature information related to the output domain and feature information related to the expected domain from the memory (130), and can display the feature information related to the output domain and feature information related to the expected domain using the display module (160).

[0161] The processor (120) can determine i) a first similarity between the feature information of the text and the feature information related to the output domain and ii) a second similarity between the feature information of the text and the feature information related to the expected domain, and can display the first similarity and the second similarity using the display module (160).

[0162] The above feature information includes one or more of text information, classification information, length information, and text information of words included in a sentence, and the processor (120) can determine the first similarity by comparing the classification information included in the feature information of the text with the classification information included in the feature information related to the output domain.

[0163] The processor (120) can determine the first similarity by determining the weights of the classification information included in the feature information related to the output domain and comparing the classification information included in the feature information related to the text with the classification information included in the feature information related to the output domain, taking the weights into consideration.

[0164] The above feature information includes one or more of text information, classification information, length information, and text information of words included in a sentence, and the processor (120) can determine the second similarity by comparing the classification information included in the feature information of the text with the classification information included in the feature information related to the expected domain.

[0165] The processor (120) can determine the second similarity by determining the weights of the classification information included in the feature information related to the predicted domain and comparing the classification information included in the feature information related to the text with the classification information included in the feature information related to the predicted domain, taking the weights into consideration.

[0166] The processor (120) can display feature information on the display module (160) in order of highest first similarity with the feature information of the text among the feature information related to the output domain.

[0167] The processor (120) can display feature information on the display module (160) in order of highest first similarity with the feature information of the text among the feature information related to the expected domain.

[0168] The processor (120) can display classification information with a weight higher than a standard among the classification information included in the feature information related to the output domain on the display module (160).

[0169] The processor (120) can display classification information with a weight higher than a standard among the classification information included in the feature information related to the expected domain on the display module (160).

[0170] The processor (120) can determine the output domain by inputting feature information of the text into a learning model trained to determine the domain associated with the user utterance.

[0171] The processor (120) can determine feature information of the text by dividing the text into a plurality of token information and comparing the divided token information with token information previously stored in the memory (130). Electronic device (101).

[0172] In an electronic device (101) according to various embodiments, the electronic device (101) comprises: a display module (160) that provides information to the outside of the electronic device (101); a processor (120) electrically connected to the display module (160); and a memory (130) electrically connected to the processor (120). The processor (120) generates feature information of the text based on the text corresponding to the user utterance, determines an output domain that processes the user utterance based on the generated feature information, extracts feature information related to the output domain from the memory (130), and can display the feature information related to the output domain using the display module (160).

[0173] The processor (120) can determine the similarity between the feature information of the text and the feature information related to the output domain, and display the similarity using the display module (160).

[0174] The above feature information includes one or more of text information, classification information, length information, and text information of words included in a sentence, and the processor (120) can determine the similarity by comparing the classification information included in the feature information of the text with the classification information included in the feature information related to the output domain.

[0175] The processor (120) can determine the similarity by determining the weights of the classification information included in the feature information related to the output domain and comparing the classification information included in the feature information related to the text with the classification information included in the feature information related to the output domain, taking the weights into consideration.

[0176] A method for analyzing speech recognition results according to various embodiments may include: generating feature information of text based on text corresponding to a user utterance; determining an output domain for processing the user utterance based on the feature information of the text; identifying an expected domain predetermined by a user; extracting feature information related to the output domain and feature information related to the expected domain; and displaying feature information related to the output domain and feature information related to the expected domain.

[0178] The electronic device according to the various embodiments disclosed in this document may be of various forms. The electronic device may include, for example, a portable communication device (e.g., a smartphone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a consumer electronics device. The electronic device according to the embodiments of this document is not limited to the devices described above.

[0179] The various embodiments of this document and the terms used therein are not intended to limit the technical features described in this document to specific embodiments, and should be understood to include various modifications, equivalents, or substitutions of said embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of said items unless the relevant context clearly indicates otherwise. In this document, phrases such as "A or B," "at least one of A and B," "at least one of A or B," "A, B or C," "at least one of A, B and C," and "at least one of A, B, or C" may each include any one of the items listed together in the corresponding phrase, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used simply to distinguish said components from other said components and do not limit said components in any other aspect (e.g., importance or order). Where any (e.g., 1st) component is referred to as “coupled” or “connected” to another (e.g., 2nd) component, with or without the terms “functionally” or “communicationly,” it means that said any component may be connected to said other component directly (e.g., via a wire), wirelessly, or through a third component.

[0180] The term “module” as used in the various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit, for example. A module may be a component formed integrally, or a minimum unit of said component or a part thereof that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0181] Various embodiments of the present document may be implemented as software (e.g., program (140)) comprising one or more instructions stored in a storage medium (e.g., internal memory (136) or external memory (138)) readable by a machine (e.g., electronic device (101)). For example, a processor (e.g., processor (120)) of the machine (e.g., electronic device (101)) may call at least one of the one or more instructions stored in the storage medium and execute it. This enables the machine to be operated to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code that can be executed by an interpreter. The storage medium readable by the machine may be provided in the form of a non-transitory storage medium. Here, 'non-temporary' simply means that the storage medium is a tangible device and does not contain a signal (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently and cases where it is stored temporarily.

[0182] According to one embodiment, the method according to the various embodiments disclosed herein may be provided as included in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or distributed online (e.g., download or upload) through an application store (e.g., Play Store™) or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily created on a device-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0183] According to various embodiments, each component (e.g., module or program) of the components described above may include a singular or multiple entities, and some of the multiple entities may be separated and placed in other components. According to various embodiments, one or more of the components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Generally or additionally, multiple components (e.g., module or program) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each of the multiple components in the same or similar manner as those performed by the corresponding component among the multiple components prior to integration. According to various embodiments, operations performed by the module, program, or other components may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

Claims

Claim 1 In an electronic device, a display module that provides information to the outside of the electronic device; a processor electrically connected to the display module; An electronic device comprising a memory electrically connected to the processor, wherein the processor generates feature information of the text based on text corresponding to a user utterance, determines an output domain for processing the user utterance based on the feature information of the text, identifies an expected domain predetermined by the user, extracts feature information related to the output domain and feature information related to the expected domain from the memory, and displays the feature information related to the output domain and feature information related to the expected domain using the display module, wherein the processor determines a first similarity based on information common between a set of information included in the feature information of the text corresponding to the user utterance and a set of feature information related to the output domain, determines a second similarity based on information common between a set of information included in the feature information of the text corresponding to the user utterance and a set of feature information related to the expected domain, and displays the first similarity and the second similarity using the display module. Claim 2 delete Claim 3 An electronic device according to claim 1, wherein the feature information includes one or more of text information, classification information, length information, and text information of words included in a sentence, and the processor determines the first similarity by comparing the classification information included in the feature information of the text with the classification information included in the feature information related to the output domain, and the processor determines the weight of the classification information included in the feature information related to the output domain, and determines the first similarity by comparing the classification information included in the feature information of the text with the classification information included in the feature information related to the output domain, taking into account the weight. Claim 4 delete Claim 5 An electronic device according to claim 1, wherein the feature information includes one or more of text information, classification information, length information, and text information of words included in a sentence, and the processor determines the second similarity by comparing the classification information included in the feature information of the text with the classification information included in the feature information related to the expected domain, and the processor determines the weight of the classification information included in the feature information related to the expected domain, and determines the second similarity by comparing the classification information included in the feature information of the text with the classification information included in the feature information related to the expected domain, taking into account the weight. Claim 6 delete Claim 7 An electronic device according to claim 1, wherein the processor displays feature information on the display module in order of highest first similarity with the feature information of the text among feature information related to the output domain. Claim 8 An electronic device according to claim 1, wherein the processor displays feature information on the display module in order of highest first similarity with the feature information of the text among the feature information related to the expected domain. Claim 9 delete Claim 10 delete Claim 11 An electronic device according to claim 1, wherein the processor determines the output domain by inputting feature information of the text into a learning model trained to determine the domain associated with the user utterance. Claim 12 An electronic device according to claim 1, wherein the processor divides the text into a plurality of token information and determines feature information of the text by comparing the divided token information with token information previously stored in the memory. Claim 13 An electronic device comprising: a display module that provides information to the outside of the electronic device; a processor electrically connected to the display module; and a memory electrically connected to the processor, wherein the processor generates feature information of the text based on text corresponding to a user utterance, determines an output domain that processes the user utterance based on the generated feature information, extracts feature information related to the output domain from the memory, and displays the feature information related to the output domain using the display module, wherein the processor determines a similarity based on common information between a set of information included in the feature information of the text corresponding to the user utterance and a set of feature information related to the output domain, and displays the similarity using the display module. Claim 14 delete Claim 15 delete Claim 16 delete Claim 17 A method for analyzing a voice recognition result of an electronic device comprises: a step of generating feature information of a text based on a text corresponding to a user utterance; a step of determining an output domain that processes the user utterance based on the feature information of the text; a step of identifying an expected domain that is predetermined by a user; a step of extracting feature information related to the output domain and feature information related to the expected domain; and a step of displaying the feature information related to the output domain and the feature information related to the expected domain. The method for analyzing a voice recognition result further comprises: a step of determining a first similarity based on information common between a set of information included in the feature information of the text corresponding to the user utterance and a set of feature information related to the output domain; a step of determining a second similarity based on information common between a set of information included in the feature information of the text corresponding to the user utterance and a set of feature information related to the expected domain; and a step of displaying the first similarity and the second similarity using a display module of the electronic device.

Images