Electronic device for providing utterance category determined on basis of utterance log, and control method therefor

Abstract

Disclosed are an electronic device for providing an utterance category determined on the basis of an utterance log, and a control method therefor. The electronic device according to an embodiment of the present document comprises a touch screen display, a communication module, and a processor. The processor may be configured to: transmit information about an utterance history of a user to an external device via the communication module, the utterance history of the user transmitted to the external device being categorized in at least one utterance category by at least one error detection module stored in the external device; in response to the transmission, acquire information about the at least one utterance category in which the utterance history was categorized and at least one representative utterance sentence belonging to the utterance category from the external device via the communication module; and provide the acquired utterance category and information about the representative utterance sentence via the touch screen display.

Description

Electronic device providing ignition categories determined based on ignition logs and control method thereof

[0001] This document relates to an electronic device that provides an utterance category determined based on an utterance log and a control method thereof.

[0002] The variety of services and additional features offered through electronic devices, such as smartphones, is steadily increasing. To enhance the utility of these devices and satisfy the diverse needs of users, telecommunications service providers and electronic device manufacturers are competitively developing electronic devices that offer diverse features and differentiate themselves from competitors. Accordingly, the various functions offered through wearable devices are also becoming increasingly sophisticated.

[0003] Typically, when a user of an electronic device utilizes an intelligent service (e.g., Samsung® Bixby™), the electronic device (e.g., the Bixby™ server) estimates the characteristics and quality of the target service based on reviews and opinions from other users regarding the target service the user is attempting to perform, and then provides the service to the user. For example, when providing a service for a specific application, the electronic device (e.g., a smartphone) only provides items such as "Say this" and / or "Review" on the intelligent service screen. However, with such limited information, it is difficult to accurately judge the intelligent service and determine whether the target service is useful. Accordingly, the electronic device (e.g., the Bixby™ server) may provide a service that does not provide the user with the results he or she expected, or a service that is of lower quality than expected.

[0004] According to one embodiment of the present document, an electronic device may be provided that can provide objective and accurate information to a user who selects and uses each target service by automatically (e.g., not based on review information of other users) providing information on successful utterances, failed utterances, and / or unsupported utterances of each target service to the user based on the user's utterance log information.

[0005] According to one embodiment of the present document, a control method of an electronic device can be provided that can provide objective and accurate information to a user who selects and uses each target service by automatically (e.g., not based on review information of other users) providing information on successful utterances, failed utterances, and / or unsupported utterances of each target service to the user based on the user's utterance log information.

[0006] An electronic device according to one embodiment of the present document includes a touchscreen display, a communication module, and a processor, wherein the processor is configured to transmit information on a user's speech history to an external device through the communication module, determine at least one speech category from the user's speech history transmitted to the external device by at least one error detection module stored in the external device, and, in response to the transmission, obtain, from the external device through the communication module, information on a representative sentence belonging to the determined at least one speech category, and provide information on the obtained representative sentence through the touchscreen display.

[0007] A method for controlling an electronic device according to one embodiment of the present document may include an operation of transmitting information on a user's speech history to an external device through a communication module of the electronic device, determining at least one speech category from the user's speech history transmitted to the external device by at least one error detection module stored in the external device, and, in response to the transmission, obtaining information on a representative sentence belonging to the determined at least one speech category from the external device through the communication module, and providing information on the obtained representative sentence through a touch screen display of the electronic device.

[0008] According to one embodiment of the present document, an electronic device may be provided that can provide objective and accurate information to a user who selects and uses each target service by automatically (e.g., not based on review information of other users) providing information on successful utterances, failed utterances, and / or unsupported utterances of each target service to the user based on the user's utterance log information.

[0009] FIG. 1 is a block diagram of an electronic device within a network environment according to various embodiments.

[0010] FIG. 2 is a block diagram illustrating an integrated intelligence system according to one embodiment of the present document.

[0011] FIG. 3 is a diagram showing a screen for processing voice input received through an intelligent app by a user terminal according to one embodiment of the present document.

[0012] FIG. 4 is a diagram showing a screen for processing voice input received through an intelligent app by a user terminal according to one embodiment of the present document.

[0013] FIG. 5 is an exemplary drawing for explaining a function or operation of an electronic device according to an embodiment of the present document to provide representative utterance sentences according to utterance categories based on utterance logs.

[0014] FIG. 6 is an exemplary drawing for explaining the configuration of an intelligent system according to one embodiment of the present document.

[0015] FIG. 7 is an exemplary diagram illustrating a function or operation of an error detection module generated and / or learned according to one embodiment of the present document.

[0016] Figure 8 is an exemplary drawing for explaining an error detection module according to one embodiment of the present document.

[0017] Figure 9 is an example drawing for explaining utterance log information according to one embodiment of the present document.

[0018] FIG. 10 is an exemplary drawing for explaining the function or operation of an end-to-end error detection module according to one embodiment of the present document.

[0019] FIG. 11 is an exemplary drawing for explaining the function or operation of a failure type detection module according to one embodiment of the present document.

[0020] FIG. 12 is an exemplary drawing for explaining the function or operation of a slot error detection module according to one embodiment of the present document.

[0021] FIG. 13 and FIG. 14 are exemplary drawings for explaining the function or operation of a client and server error detection module according to one embodiment of the present document.

[0022] FIG. 15 is an exemplary drawing for explaining the function or operation of an integrated module according to one embodiment of the present document.

[0023] FIG. 16 is an exemplary drawing for explaining the data format stored in a prediction database according to one embodiment of the present document.

[0024] FIG. 17 is an exemplary diagram illustrating a data format for an unsupported utterance as a utterance category transmitted to an electronic device according to one embodiment of the present document.

[0025] FIG. 18 is an exemplary drawing for explaining the function or operation of a failed ignition extraction module according to one embodiment of the present document.

[0026] FIG. 19 is an exemplary diagram illustrating a data format for a failed utterance as an utterance category transmitted to an electronic device according to one embodiment of the present document.

[0027] FIG. 20 is an exemplary drawing for explaining the function or operation of a successful utterance extraction module according to one embodiment of the present document.

[0028] FIG. 21 is an exemplary diagram illustrating a data format for a successful utterance as an utterance category transmitted to an electronic device according to one embodiment of the present document.

[0029] FIG. 22A is an example drawing for explaining a speech category and at least one representative speech sentence provided through an electronic device according to one embodiment of the present document.

[0030] FIG. 22b is an example diagram illustrating utterance categories and at least one representative utterance sentence provided through a website according to one embodiment of the present document.

[0031] FIG. 23 is an exemplary diagram for explaining a function or operation for providing a service related to an unsupported utterance when an unsupported utterance is acquired according to one embodiment of the present document.

[0032] FIG. 24 is a system flowchart for explaining a function or operation of an electronic device according to an embodiment of the present document to provide representative utterance sentences according to utterance categories based on utterance logs.

[0033] FIG. 1 is a block diagram of an electronic device (101) within a network environment (100) according to various embodiments.

[0034] Referring to FIG. 1, in a network environment (100), an electronic device (101) may communicate with an electronic device (102) via a first network (198) (e.g., a short-range wireless communication network), or may communicate with at least one of an electronic device (104) or a server (108) via a second network (199) (e.g., a long-range wireless communication network). In one embodiment, the electronic device (101) may communicate with the electronic device (104) via the server (108). According to one embodiment, the electronic device (101) may include a processor (120), a memory (130), an input module (150), an audio output module (155), a display module (160), an audio module (170), a sensor module (176), an interface (177), a connection terminal (178), a haptic module (179), a camera module (180), a power management module (188), a battery (189), a communication module (190), a subscriber identification module (196), or an antenna module (197). In some embodiments, the electronic device (101) may omit at least one of these components (e.g., the connection terminal (178)), or may have one or more other components added. In some embodiments, some of these components (e.g., the sensor module (176), the camera module (180), or the antenna module (197)) may be integrated into one component (e.g., the display module (160)).

[0035] The processor (120) may, for example, execute software (e.g., a program (140)) to control at least one other component (e.g., a hardware or software component) of the electronic device (101) connected to the processor (120) and perform various data processing or operations. According to one embodiment, as at least a part of the data processing or operations, the processor (120) may store commands or data received from other components (e.g., a sensor module (176) or a communication module (190)) in a volatile memory (132), process the commands or data stored in the volatile memory (132), and store result data in a non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., a central processing unit or an application processor) or an auxiliary processor (123) (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor) that can operate independently or together with the main processor (121). For example, when the electronic device (101) includes the main processor (121) and the auxiliary processor (123), the auxiliary processor (123) may be configured to use less power than the main processor (121) or to be specialized for a given function. The auxiliary processor (123) may be implemented separately from the main processor (121) or as a part thereof.

[0036] The auxiliary processor (123) may control at least a portion of functions or states associated with at least one component (e.g., a display module (160), a sensor module (176), or a communication module (190)) of the electronic device (101), for example, on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. In one embodiment, the auxiliary processor (123) (e.g., an image signal processor or a communication processor) may be implemented as a part of another functionally related component (e.g., a camera module (180) or a communication module (190)). In one embodiment, the auxiliary processor (123) (e.g., a neural network processing unit) may include a hardware structure specialized for processing artificial intelligence models. The artificial intelligence models may be generated through machine learning. This learning can be performed, for example, on the electronic device (101) itself where the artificial intelligence model is executed, or can be performed through a separate server (e.g., server (108)). The learning algorithm can include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model can include multiple artificial neural network layers.The artificial neural network may be one of a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to, or alternatively to, a hardware structure, an artificial intelligence model may include a software structure.

[0037] The memory (130) can store various data used by at least one component (e.g., processor (120) or sensor module (176)) of the electronic device (101). The data can include, for example, software (e.g., program (140)) and input data or output data for commands related thereto. The memory (130) can include volatile memory (132) or non-volatile memory (134).

[0038] The program (140) may be stored as software in the memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0039] The input module (150) can receive commands or data to be used in a component of the electronic device (101) (e.g., a processor (120)) from an external source (e.g., a user) of the electronic device (101). The input module (150) can include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0040] The audio output module (155) can output audio signals to the outside of the electronic device (101). The audio output module (155) can include, for example, a speaker or a receiver. The speaker can be used for general purposes, such as multimedia playback or recording playback. The receiver can be used to receive incoming calls. In one embodiment, the receiver can be implemented separately from the speaker or as part of the speaker.

[0041] The display module (160) can visually provide information to an external party (e.g., a user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling the device. According to one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of a force generated by the touch.

[0042] The audio module (170) can convert sound into an electrical signal, or vice versa, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150), output sound through the sound output module (155), or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphone) directly or wirelessly connected to the electronic device (101).

[0043] The sensor module (176) can detect the operating status (e.g., power or temperature) of the electronic device (101) or the external environmental status (e.g., user status) and generate an electrical signal or data value corresponding to the detected status. According to one embodiment, the sensor module (176) can include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0044] The interface (177) may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device (101) with an external electronic device (e.g., the electronic device (102)). In one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0045] The connection terminal (178) may include a connector through which the electronic device (101) may be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0046] The haptic module (179) can convert electrical signals into mechanical stimuli (e.g., vibration or movement) or electrical stimuli that a user can perceive through tactile or kinesthetic sensations. According to one embodiment, the haptic module (179) can include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

[0047] The camera module (180) can capture still images and videos. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0048] The power management module (188) can manage power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented as, for example, at least a part of a power management integrated circuit (PMIC).

[0049] A battery (189) may power at least one component of the electronic device (101). In one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0050] The communication module (190) may support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may operate independently from the processor (120) (e.g., application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module (194) (e.g., a local area network (LAN) communication module, or a power line communication module). Among these communication modules, the corresponding communication module can communicate with an external electronic device (104) via a first network (198) (e.g., a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network (199) (e.g., a long-range communication network such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules can be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can verify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) by using subscriber information (e.g., an international mobile subscriber identity (IMSI)) stored in the subscriber identification module (196).

[0051] The wireless communication module (192) can support 5G networks and next-generation communication technologies following the 4G network, such as NR access technology (new radio access technology). The NR access technology can support high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimization of terminal power and connection of multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency (URLLC (ultra-reliable and low-latency communications)). The wireless communication module (192) can support, for example, a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate. The wireless communication module (192) can support various technologies for securing performance in a high-frequency band, such as beamforming, massive multiple-input and multiple-output (MIMO), full dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), an external electronic device (e.g., the electronic device (104)), or a network system (e.g., the second network (199)). According to one embodiment, the wireless communication module (192) can support a peak data rate (e.g., 20 Gbps or more) for eMBB realization, a loss coverage (e.g., 164 dB or less) for mMTC realization, or a U-plane latency (e.g., 0.5 ms or less for downlink (DL) and uplink (UL), or 1 ms or less for round trip) for URLLC realization.

[0052] The antenna module (197) can transmit or receive signals or power to or from an external device (e.g., an external electronic device). In one embodiment, the antenna module (197) may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). In one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as the first network (198) or the second network (199), may be selected from the plurality of antennas, for example, by the communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device via the at least one selected antenna. In some embodiments, in addition to the radiator, another component (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as a part of the antenna module (197).

[0053] According to various embodiments, the antenna module (197) may form a mmWave antenna module. In one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent a first side (e.g., a bottom side) of the printed circuit board and capable of supporting a designated high-frequency band (e.g., a mmWave band), and a plurality of antennas (e.g., an array antenna) disposed on or adjacent a second side (e.g., a top side or a side side) of the printed circuit board and capable of transmitting or receiving signals in the designated high-frequency band.

[0054] At least some of the above components can be interconnected and exchange signals (e.g., commands or data) with each other via a communication method between peripheral devices (e.g., a bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)).

[0055] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) via a server (108) connected to a second network (199). Each of the external electronic devices (102 or 104) may be the same or a different type of device as the electronic device (101). According to one embodiment, all or part of the operations executed in the electronic device (101) may be executed in one or more of the external electronic devices (102, 104, or 108). For example, when the electronic device (101) is to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device (101) may, instead of or in addition to executing the function or service itself, request one or more external electronic devices to perform the function or at least a part of the service. One or more external electronic devices that receive the request may execute at least a portion of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may process the result as is or additionally and provide it as at least a portion of a response to the request. For this purpose, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used, for example. The electronic device (101) may provide an ultra-low latency service by using distributed computing or mobile edge computing, for example. In another embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server utilizing machine learning and / or a neural network. According to one embodiment, the external electronic device (104) or the server (108) may be included in the second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0056] FIG. 2 is a block diagram illustrating an integrated intelligence system according to one embodiment.

[0057] Referring to FIG. 2, an integrated intelligence system (10) of one embodiment may include a user terminal (290), an intelligent server (200), and a service server (300).

[0058] The user terminal (290) of one embodiment may be a terminal device (or electronic device) that can connect to the Internet, and may be, for example, a mobile phone, a smart phone, a personal digital assistant (PDA), a laptop computer, a TV, white goods, an electronic device, an HMD, or a smart speaker.

[0059] According to the illustrated embodiment, the user terminal (290) may include a communication interface (291), a microphone (295), a speaker (294), a display (293), a memory (299), or a processor (292). The above-listed components may be operatively or electrically connected to each other.

[0060] The communication interface (291) of one embodiment may be configured to connect to an external device and transmit and receive data. The microphone (295) of one embodiment may receive sound (e.g., user speech) and convert it into an electrical signal. The speaker (294) of one embodiment may output the electrical signal as sound (e.g., voice). The display (293) of one embodiment may be configured to display an image or video. The display (293) of one embodiment may also display a graphical user interface (GUI) of an app (or application program) being executed.

[0061] In one embodiment, the memory (299) may store a client module (298), a software development kit (SDK) (297), and multiple apps (296). The client module (298) and the SDK (297) may constitute a framework (or solution program) for performing general functions. In addition, the client module (298) or the SDK (297) may constitute a framework for processing voice input.

[0062] In one embodiment, the memory (299) may be a program for performing a designated function for the plurality of apps (296). According to one embodiment, the plurality of apps (296) may include a first app (296_1) and a second app (296_2). According to one embodiment, each of the plurality of apps (296) may include a plurality of operations for performing a designated function. For example, the apps may include an alarm app, a message app, and / or a schedule app. According to one embodiment, the plurality of apps (296) may be executed by the processor (292) to sequentially perform at least some of the plurality of operations.

[0063] The processor (292) of one embodiment can control the overall operation of the user terminal (290). For example, the processor (292) can be electrically connected to a communication interface (291), a microphone (295), a speaker (294), and a display (293) to perform a designated operation.

[0064] The processor (292) of one embodiment may also execute a program stored in the memory (299) to perform a designated function. For example, the processor (292) may execute at least one of the client module (298) or the SDK (297) to perform the following operations for processing voice input. The processor (292) may control the operations of multiple apps (296), for example, through the SDK (297). The following operations described as operations of the client module (298) or the SDK (297) may be operations executed by the processor (292).

[0065] The client module (298) of one embodiment can receive a voice input. For example, the client module (298) can receive a voice signal corresponding to a user utterance detected through a microphone (295). The client module (298) can transmit the received voice input to the intelligent server (200). The client module (298) can transmit status information of the user terminal (290) together with the received voice input to the intelligent server (200). The status information can be, for example, execution status information of an app.

[0066] The client module (298) of one embodiment may receive a result corresponding to the received voice input. For example, the client module (298) may receive a result corresponding to the received voice input if the intelligent server (200) can produce a result corresponding to the received voice input. The client module (298) may display the received result on the display (293).

[0067] In one embodiment, the client module (298) can receive a plan corresponding to the received voice input. The client module (298) can display the results of executing multiple operations of the app according to the plan on the display (293). For example, the client module (298) can sequentially display the results of executing multiple operations on the display (293). The user terminal (290) can, for example, display only some of the results of executing multiple operations (e.g., the result of the last operation) on the display (293).

[0068] According to one embodiment, the client module (298) may receive a request from the intelligent server (200) to obtain information necessary to produce a result corresponding to a voice input. According to one embodiment, the client module (298) may transmit the necessary information to the intelligent server (200) in response to the request.

[0069] The client module (298) of one embodiment can transmit result information of executing multiple operations according to a plan to the intelligent server (200). The intelligent server (200) can use the result information to confirm that the received voice input has been processed correctly.

[0070] The client module (298) of one embodiment may include a voice recognition module. According to one embodiment, the client module (298) may recognize voice inputs to perform limited functions through the voice recognition module. For example, the client module (298) may execute an intelligent app to process voice inputs to perform organic actions through designated inputs (e.g., "Wake up!").

[0071] An intelligent server (200) of one embodiment may receive information related to a user voice input from a user terminal (290) via a communication network. According to one embodiment, the intelligent server (200) may convert data related to the received voice input into text data. According to one embodiment, the intelligent server (200) may generate a plan for performing a task corresponding to the user voice input based on the text data.

[0072] In one embodiment, the plan may be generated by an artificial intelligence (AI) system. The AI ​​system may be a rule-based system, a neural network-based system (e.g., a feedforward neural network (FNN) or a recurrent neural network (RNN)), or a combination of the foregoing or another AI system. In one embodiment, the plan may be selected from a set of predefined plans or may be generated in real time in response to a user request. For example, the AI ​​system may select at least one plan from a plurality of predefined plans.

[0073] An intelligent server (200) of one embodiment may transmit results according to a generated plan to a user terminal (290), or transmit the generated plan to the user terminal (290). According to one embodiment, the user terminal (290) may display results according to the plan on a display (293). According to one embodiment, the user terminal (290) may display results of executing an operation according to the plan on a display (293).

[0074] An intelligent server (200) of one embodiment may include a front end (210), a natural language platform (220), a capsule database (230), an execution engine (240), an end user interface (250), a management platform (260), a big data platform (270), or an analytic platform (280).

[0075] The front end (210) of one embodiment can receive a voice input from a user terminal (290). The front end (210) can transmit a response corresponding to the voice input.

[0076] According to one embodiment, the natural language platform (220) may include an automatic speech recognition module (ASR module) (221), a natural language understanding module (NLU module) (223), a planner module (225), a natural language generator module (NLG module) (227), or a text to speech module (TTS module) (229).

[0077] An automatic speech recognition module (221) of one embodiment can convert a voice input received from a user terminal (290) into text data. A natural language understanding module (223) of one embodiment can use the text data of the voice input to determine the user's intention. For example, the natural language understanding module (223) can perform syntactic analysis or semantic analysis to determine the user's intention. The natural language understanding module (223) of one embodiment can use linguistic features (e.g., grammatical elements) of morphemes or phrases to determine the meaning of words extracted from the voice input, and can match the meaning of the determined words to the intention to determine the user's intention.

[0078] The planner module (225) of one embodiment can generate a plan using the intent and parameters determined by the natural language understanding module (223). According to one embodiment, the planner module (225) can determine a plurality of domains necessary to perform a task based on the determined intent. The planner module (225) can determine a plurality of operations included in each of the plurality of domains determined based on the intent. According to one embodiment, the planner module (225) can determine parameters necessary to execute the determined plurality of operations or result values ​​output by the execution of the plurality of operations. The parameters and the result values ​​can be defined as concepts of a specified format (or class). Accordingly, the plan can include a plurality of operations and a plurality of concepts determined by the user's intent. The planner module (225) can operationally (or hierarchically) determine the relationship between the plurality of operations and the plurality of concepts. For example, the planner module (225) can determine the execution order of a plurality of actions based on the user's intention based on a plurality of concepts. In other words, the planner module (225) can determine the execution order of a plurality of actions based on parameters required for the execution of the plurality of actions and results output by the execution of the plurality of actions. Accordingly, the planner module (225) can generate a plan including association information (e.g., ontology) between the plurality of actions and the plurality of concepts. The planner module (225) can generate the plan using information stored in a capsule database (230) in which a set of relationships between concepts and actions is stored.

[0079] The natural language generation module (227) of one embodiment can convert specified information into text format. The information converted into text format may be in the form of natural language speech. The text-to-speech conversion module (229) of one embodiment can convert information in text format into information in speech format.

[0080] According to one embodiment, some or all of the functions of the natural language platform (220) may also be implemented in the user terminal (290).

[0081] The capsule database (230) can store information on the relationships between multiple concepts and actions corresponding to multiple domains. According to one embodiment, a capsule can include multiple action objects (or action information) and concept objects (or concept information) included in a plan. According to one embodiment, the capsule database (230) can store multiple capsules in the form of a concept action network (CAN). According to one embodiment, the multiple capsules can be stored in a function registry included in the capsule database (230).

[0082] The capsule database (230) may include a strategy registry that stores strategy information necessary for determining a plan corresponding to a voice input. The strategy information may include reference information for determining one plan when there are multiple plans corresponding to a voice input. According to one embodiment, the capsule database (230) may include a follow-up registry that stores information on follow-up actions for suggesting follow-up actions to a user in a given situation. The follow-up actions may include, for example, follow-up utterances. According to one embodiment, the capsule database (230) may include a layout registry that stores layout information of information output through the user terminal (290). According to one embodiment, the capsule database (230) may include a vocabulary registry that stores vocabulary information included in capsule information. According to one embodiment, the capsule database (230) may include a dialog registry that stores information on dialogue (or interaction) with a user. The capsule database (230) may update stored objects through a developer tool. The developer tool may include, for example, a function editor for updating an action object or a concept object. The developer tool may include a vocabulary editor for updating a vocabulary. The developer tool may include a strategy editor for creating and registering a strategy that determines a plan. The developer tool may include a dialog editor for creating a dialogue with a user.The developer tool may include a follow-up editor that activates follow-up goals and allows editing of follow-up utterances that provide hints. The follow-up goals may be determined based on currently set goals, user preferences, or environmental conditions. In one embodiment, the capsule database (230) may also be implemented within the user terminal (290).

[0083] The execution engine (240) of one embodiment can produce a result using the generated plan. The end user interface (250) can transmit the produced result to the user terminal (290). Accordingly, the user terminal (290) can receive the result and provide the received result to the user. The management platform (260) of one embodiment can manage information used in the intelligent server (200). The big data platform (270) of one embodiment can collect user data. The analysis platform (280) of one embodiment can manage the quality of service (QoS) of the intelligent server (200). For example, the analysis platform (280) can manage the components and processing speed (or efficiency) of the intelligent server (200).

[0084] In one embodiment, the service server (300) may provide a designated service (e.g., food ordering or hotel reservation) to the user terminal (290). According to one embodiment, the service server (300) may be a server operated by a third party. In one embodiment, the service server (300) may provide information for generating a plan corresponding to the received voice input to the intelligent server (200). The provided information may be stored in the capsule database (230). In addition, the service server (300) may provide result information according to the plan to the intelligent server (200).

[0085] In the integrated intelligence system (10) described above, the user terminal (290) can provide various intelligent services to the user in response to user input. The user input may include, for example, input via a physical button, touch input, or voice input.

[0086] In one embodiment, the user terminal (290) may provide a voice recognition service through an intelligent app (or voice recognition app) stored internally. In this case, for example, the user terminal (290) may recognize a user utterance or voice input received through the microphone and provide the user with a service corresponding to the recognized voice input.

[0087] In one embodiment, the user terminal (290) may perform a designated action based on the received voice input, either alone or together with the intelligent server (200) and / or the service server (300). For example, the user terminal (290) may execute an app corresponding to the received voice input and perform a designated action through the executed app.

[0088] In one embodiment, when a user terminal (290) provides a service together with an intelligent server (200) and / or a service server (300), the user terminal (290) can detect a user's speech using the microphone (295) and generate a signal (or voice data) corresponding to the detected user speech. The user terminal (290) can transmit the voice data to the intelligent server (200) using a communication interface (291).

[0089] According to one embodiment, an intelligent server (200) may generate a plan for performing a task corresponding to a voice input received from a user terminal (290), or a result of performing an operation according to the plan, in response to the voice input. The plan may include, for example, a plurality of operations for performing a task corresponding to the user's voice input, and a plurality of concepts related to the plurality of operations. The concepts may define parameters input to the execution of the plurality of operations, or result values ​​output by the execution of the plurality of operations. The plan may include association information between the plurality of operations and the plurality of concepts.

[0090] In one embodiment, the user terminal (290) can receive the response using the communication interface (291). The user terminal (290) can output a voice signal generated within the user terminal (290) to the outside using the speaker (294), or can output an image generated within the user terminal (290) to the outside using the display (293).

[0091] FIG. 3 is a diagram showing a form in which relationship information between concepts and actions is stored in a database according to various embodiments.

[0092] The capsule database (e.g., capsule database (230)) of the intelligent server (200) can store capsules in the form of a CAN (concept action network). The capsule database can store operations for processing tasks corresponding to a user's voice input and parameters required for the operations in the form of a CAN (concept action network).

[0093] The capsule database may store a plurality of capsules (e.g., capsule(A)(401), capsule(B)(404)) corresponding to each of a plurality of domains (e.g., applications). According to one embodiment, one capsule (e.g., capsule(A)(401)) may correspond to one domain (e.g., location (geo), application). In addition, one capsule may correspond to at least one service provider (e.g., CP 1(402) or CP 2(403)) for performing a function for a domain related to the capsule. According to one embodiment, one capsule may include at least one operation (410) and at least one concept (420) for performing a specified function.

[0094] The above natural language platform (220) can generate a plan for performing a task corresponding to a received voice input using capsules stored in a capsule database. For example, the planner module (225) of the natural language platform can generate a plan using capsules stored in a capsule database. For example, a plan (407) can be generated using actions (4011, 4013) and concepts (4012, 4014) of capsule A (410) and actions (4041) and concepts (4042) of capsule B (404).

[0095] FIG. 4 is a diagram showing a screen for processing voice input received through an intelligent app by a user terminal according to various embodiments.

[0096] The user terminal (290) can execute an intelligent app to process user input through the intelligent server (200).

[0097] According to one embodiment, on screen 310, when the user terminal (290) recognizes a designated voice input (e.g., wake up!) or receives an input via a hardware key (e.g., a dedicated hardware key), the user terminal (290) may execute an intelligent app for processing the voice input. For example, the user terminal (290) may execute the intelligent app while the schedule app is running. According to one embodiment, the user terminal (290) may display an object (e.g., an icon) (311) corresponding to the intelligent app on the display (293). According to one embodiment, the user terminal (290) may receive a voice input by a user's speech. For example, the user terminal (290) may receive a voice input such as "Tell me my schedule for this week!" According to one embodiment, the user terminal (290) may display a user interface (UI) (313) (e.g., an input window) of the intelligent app, in which text data of the received voice input is displayed, on the display.

[0098] According to one embodiment, on screen 320, the user terminal (290) may display a result corresponding to the received voice input on the display. For example, the user terminal (290) may receive a plan corresponding to the received user input and display "This Week's Schedule" on the display according to the plan.

[0099] FIG. 5 is an exemplary drawing for explaining a function or operation of an electronic device (101) according to one embodiment of the present document to provide representative utterance sentences according to utterance categories based on utterance logs.

[0100] Referring to FIG. 5, an electronic device (101) according to an embodiment of the present document may transmit information on a user's utterance history to an external device (e.g., a server (108)) through a communication module in operation 510. For example, the electronic device (101) according to an embodiment of the present document may transmit utterance log information (900) to the external device. The utterance log information (900) according to an embodiment of the present document may be stored in the electronic device. FIG. 9 is an exemplary diagram for explaining utterance log information (900) according to an embodiment of the present document. Referring to FIG. 9, the utterance log information (900) according to an embodiment of the present document may include at least one of utterance ID information, device information, user utterance information, intelligent service response information, a label, client error information, and / or server error information. According to one embodiment of the present document, the utterance history information (900) may determine at least one utterance category by at least one error detection module (615) stored in an external device (e.g., server (108)). Fig. 6 is an exemplary diagram for explaining the configuration of an intelligent system according to one embodiment of the present document. Referring to Fig. 6, the error detection module according to one embodiment of the present document may include at least one module among an end-to-end error detection module (620), a slot error detection module (630), a failure type detection module (640), an unsupported utterance detection module (650), and / or a client and server error detection module (650). Additionally, an intelligent external device (600) according to one embodiment of the present document (e.g., server (108) of FIG. 1) may include at least one of a log collector (610) configured to obtain utterance history information from an electronic device (101), a prediction database (670) configured to store results from an error detection module (615), and / or a real-time usability detection system (690) that integrates results from the error detection module (615) and transmits the integrated results.

[0101] FIG. 7 is an exemplary diagram for explaining a function or operation generated and / or learned by an error detection module (615) according to one embodiment of the present document. FIG. 8 is an exemplary diagram for explaining an error detection module according to one embodiment of the present document. Referring to FIGS. 7 and 8, an electronic device (101) according to one embodiment of the present document may obtain an utterance such as "Order me a caramel macchiato" as a user utterance. An electronic device (101) according to one embodiment of the present document may output a message indicating that such a user utterance is an utterance not supported by an intelligent service. Such an output result may be stored in the electronic device (101). An electronic device (101) according to one embodiment of the present document may transmit such utterance history information to an inspector (e.g., an inspection server). The utterance history information according to one embodiment of the present document may be transmitted to an inspector from electronic devices other than the electronic device (101). According to one embodiment of the present document, an inspector can perform an inspection on utterance history information using an inspection tool (710). According to one embodiment of the present document, an inspector can store the inspection result in an inspection database (720). According to one embodiment of the present document, the inspection result can be transmitted to an error detection module (615). According to one embodiment of the present document, an error detection module (615) (e.g., an end-to-end error detection module (620), a slot error detection module (630), a failure type detection module (640), an unsupported utterance detection module (650), and / or a client and server error detection module (650)) can be trained based on the inspection result. According to one embodiment of the present document, the training result can be stored in a model storage (730) included in an intelligent external device.

[0102] FIG. 10 is an exemplary diagram for explaining the function or operation of an end-to-end error detection module (620) according to one embodiment of the present document. Referring to FIG. 10, utterance log information (900) according to one embodiment of the present document may be input to a pre-trained BERT (bidirectional encoder representation from transformer) model (1010). User utterance history information (900) as an output result of the pre-trained BERT model (1010) according to one embodiment of the present document may be input to an FNN (fast forward network) module (1020). User utterance history information (900) as an output result of the FNN module (1020) according to one embodiment of the present document may be output by being classified as utterance success or utterance failure. In this case, the output result according to one embodiment of the present document may be output as an utterance success probability and / or an utterance failure probability. Based on such functions or operations, the end-to-end error detection module (620) according to one embodiment of the present document can determine whether the process for a user utterance has succeeded or failed end-to-end. The end-to-end error detection module (620) according to one embodiment of the present document can output the determination result to the integration module (1510).

[0103] Fig. 11 is an exemplary diagram for explaining the function or operation of a failure type detection module (630) according to one embodiment of the present document. Referring to Fig. 11, utterance log information (900) according to one embodiment of the present document can be input to a pre-trained BERT (bidirectional encoder representation from transformer) model (1010). User utterance history information (900) as an output result of the pre-trained BERT model (1010) according to one embodiment of the present document can be input to a fast forward network (FNN) module (1020). User utterance history information (900) as an output result of the FNN module (1020) according to one embodiment of the present document can be classified and output by the type of utterance failure. In this case, the output result according to one embodiment of the present document may include, for example, the probability of a failure type such as “utterance with unclear intent”, “multiple intent question”, “incorrect service connection”, “connection error”, “ASR error” and / or “unsupported utterance”. The functions or operations performed by other modules according to one embodiment of the present document (e.g., the unsupported utterance detection module (650)) may also be performed as at least a part of the functions or operations performed by the failure type detection module (630). According to such functions or operations, the failure type detection module (640) according to one embodiment of the present document may determine the result regarding the failure type of the user utterance. The failure type detection module (640) according to one embodiment of the present document may output the determination result to the integration module (1510).

[0104] FIG. 12 is an exemplary diagram for explaining the function or operation of a slot error detection module (630) according to one embodiment of the present document. Referring to FIG. 12, utterance log information (900) according to one embodiment of the present document may be input to a pre-trained BERT (bidirectional encoder representation from transformer) model (1010). User utterance history information (900) as an output result of the pre-trained BERT model (1010) according to one embodiment of the present document may be input to an FNN (fast forward network) module (1020). User utterance history information (900) as an output result of the FNN module (1020) according to one embodiment of the present document may be output by being classified as utterance success or utterance failure. In this case, the output result according to one embodiment of the present document may be output as an utterance success probability and / or an utterance failure probability. According to such functions or operations, the end-to-end error detection module (620) according to one embodiment of the present document can determine whether a slot error (e.g., whether tagging for a specific parameter (e.g., "Mom") was performed correctly) exists in the process for a user utterance. The slot error detection module (630) according to one embodiment of the present document can output the determination result to the integration module (1510).

[0105] FIGS. 13 and 14 are exemplary drawings for explaining the function or operation of a client and server error detection module (660) according to one embodiment of the present document. Referring to FIGS. 13 and 14, utterance log information (900) according to one embodiment of the present document may be input to a post-processing module (1310). User utterance history information (900) as an output result of the post-processing module (1310) according to one embodiment of the present document may be classified and output as utterance success or utterance failure. According to such function or operation, the client and server error detection module (660) according to one embodiment of the present document may determine whether an error exists in the client and / or server for the process for the user utterance. The client and server error detection module (660) according to one embodiment of the present document may output the determination result to the integration module (1510). A client error according to one embodiment of the present document may include an error event, such as whether the user intentionally terminated the intelligent service. The client and server error detection module (660) according to one embodiment of the present document may predict success even if the intelligent service has been terminated if, by checking the client error log, it is confirmed that the termination of the intelligent service was intentional by the user. The client and server error detection modules (660) according to one embodiment of the present document may each be configured as separate modules.

[0106] FIG. 15 is an exemplary diagram for explaining the function or operation of an integrated module (1510) according to one embodiment of the present document. FIG. 16 is an exemplary diagram for explaining the format of data stored in a prediction database according to one embodiment of the present document. Referring to FIG. 15, an integrated module (1510) according to one embodiment of the present document can integrate the results of an error detection module (615) (e.g., an end-to-end error detection module (620), a slot error detection module (630), a failure type detection module (640), an unsupported utterance detection module (650), and / or a client and server error detection module (650)) and output the integrated result as prediction data (1520). According to one embodiment of the present document, when a failure result is output by the end-to-end error detection module (620) as illustrated in FIG. 15, and the failure type is predicted as an unsupported utterance by the failure type detection module (640), the prediction data (1520) may include a result of “utterance failure” as an utterance category. In addition, the prediction data (1510) according to one embodiment of the present document may include at least one of “end-to-end error” and / or “unsupported utterance” as a failure reason as a failure type. The integrated module (1510) according to one embodiment of the present document may be included and configured in a real-time usability detection system (690). Referring to FIG. 16, “Device” may be the name of the device on which the corresponding utterance operated. “Utterance” may refer to a sentence uttered by a user through an intelligent service. “Message Speech” may refer to a sentence in which an intelligent service responded to a user utterance. "Service Goal Name" may refer to the name of an application, domain, service, module, or function provided by the intelligent service. "Label" may refer to the value of a variable used by the intelligent service in a user utterance."Client Error" and "Server Error" may refer to error values ​​output from the client and server, respectively. "PASS / FAIL" may refer to a success / failure value for the corresponding utterance as judged by the integration module (1510). "Type" may refer to a category indicating the reason for failure in the event of a failure in the utterance.

[0107] FIG. 17 is an exemplary diagram for explaining the data format for "unsupported utterance" as an utterance category transmitted to an electronic device (101) according to one embodiment of the present document. FIG. 18 is an exemplary diagram for explaining the function or operation of a failed utterance extraction module according to one embodiment of the present document. FIG. 19 is an exemplary diagram for explaining the data format for "unsupported utterance" as an utterance category transmitted to an electronic device (101) according to one embodiment of the present document. Referring to FIG. 17, information on unsupported utterance according to one embodiment of the present document may be transmitted to the electronic device (101) in the data format as illustrated in FIG. 17. Referring to FIGS. 17 and 19, "Device" may be the name of the device on which the corresponding utterance was operated. "Utterance" may refer to a sentence uttered by a user through an intelligent service. "Service Goal name" may refer to the name of an application, domain, service, module, or function provided by an intelligent service. A "similar utterance" may mean an utterance similar to a user's utterance. Such a similar utterance may be determined by a failed utterance extraction module illustrated in FIG. 18. The failed utterance extraction module according to one embodiment of the present document may be configured to be included in an intelligent external device (600). Referring to FIG. 18, the failed utterance extraction module according to one embodiment of the present document may obtain information about a failed utterance (e.g., "Call the person who called yesterday", "What is the phone number of the person who called earlier?") from a prediction database (670). The failed utterance extraction module according to one embodiment of the present document may extract and / or determine an utterance similar to the failed utterance using a sentence BERT module (1810) and a cosine similarity module (1820). The similar utterance determined according to one embodiment of the present document (e.g., "Call that kid yesterday") may be output as failed data (1830).Failure data (1830) according to one embodiment of the present document may be transmitted to the electronic device (101) by the intelligent external device (600). The intelligent external device (600) (e.g., a failed utterance extraction module) according to one embodiment of the present document may determine at least one representative utterance sentence to be transmitted to the electronic device (101) among the “failed utterance” and the “similar utterance.”

[0108] FIG. 20 is an exemplary diagram for explaining the function or operation of a successful utterance extraction module according to one embodiment of the present document. FIG. 21 is an exemplary diagram for explaining the data format for "successful utterance" as an utterance category transmitted to an electronic device according to one embodiment of the present document. Referring to FIG. 21, information on a successful utterance according to one embodiment of the present document may be transmitted to the electronic device (101) in the data format illustrated in FIG. 21. Referring to FIG. 21, "Device" may be the name of the device on which the utterance was operated. "Utterance" may refer to a sentence uttered by a user through an intelligent service. "Service name" may refer to the name of an application, domain, service, module, or function provided by the intelligent service. "Similar utterance" may refer to an utterance similar to a user's utterance. Such similar utterances may be determined by the successful utterance extraction module illustrated in FIG. 20. A successful utterance extraction module according to one embodiment of the present document may be configured to be included in an intelligent external device (600). Referring to FIG. 20, the successful utterance extraction module according to one embodiment of the present document may obtain information about successful utterances (e.g., "Call Mom", "Call Mother") from a prediction database (670). The successful utterance extraction module according to one embodiment of the present document may extract and / or determine utterances similar to the successful utterances using a sentence BERT module (1810) and a cosine similarity module (1820). The similar utterances determined according to one embodiment of the present document (e.g., "Tell me I called earlier") may be output as successful data (1910). The successful data (1910) according to one embodiment of the present document may be transmitted to an electronic device (101) by the intelligent external device (600).An intelligent external device (600) (e.g., a successful utterance extraction module) according to one embodiment of the present document can determine at least one representative utterance sentence to be transmitted to the electronic device (101) among “successful utterances” and “similar utterances.”

[0109] Returning to FIG. 2, the electronic device (101) according to one embodiment of the present document, in response to the transmission according to operation 510, may obtain information about a representative sentence belonging to at least one determined utterance category from an external device (e.g., an intelligent external device (600)) through the communication module (190). An utterance similar to a failed utterance determined according to one embodiment of the present document (e.g., “Call that kid yesterday”) may be output as failure data (1830). The failure data (1830) according to one embodiment of the present document may be transmitted to the electronic device (101) by the intelligent external device (600). The intelligent external device (600) according to one embodiment of the present document (e.g., a failed utterance extraction module) may determine at least one representative utterance sentence to be transmitted to the electronic device (101) from among the “failed utterance” and the “similar utterance”. Additionally, an utterance similar to a successful utterance determined according to one embodiment of the present document (e.g., “Tell me I got a call earlier”) may be output as success data (1910). The success data (1910) according to one embodiment of the present document may be transmitted to the electronic device (101) by the intelligent external device (600). The intelligent external device (600) according to one embodiment of the present document (e.g., a successful utterance extraction module) may determine at least one representative utterance sentence to be transmitted to the electronic device (101) among the “successful utterance” and the “similar utterance.”

[0110] Referring to FIG. 2, an electronic device (101) according to an embodiment of the present document may provide information on acquired representative utterance sentences through a touchscreen display in operation 530. Representative utterance sentences according to an embodiment of the present document may be displayed by being classified according to utterance categories. FIG. 22A is an example diagram for explaining utterance categories and at least one representative utterance sentence provided through an electronic device (101) according to an embodiment of the present document. FIG. 22B is an example diagram for explaining utterance categories and at least one representative utterance sentence provided through a website according to an embodiment of the present document. Referring to FIG. 22A, information on “successful utterance” (e.g., “well-performing utterance”), “failed utterance” (e.g., “poorly-performing utterance”), “unsupported utterance” (e.g., “unsupported function”), and / or utterance success rate may be provided through the electronic device (101). Referring to FIG. 22a, through a designated application (e.g., a coffee service application and / or an intelligent service application), information about “successful utterances” (e.g., “well-sounding utterances”), “failed utterances” (e.g., “poor-sounding utterances”), “unsupported utterances” (e.g., “unsupported functions”), and / or utterance success rates related to the designated application may be provided through the electronic device (101). Also, referring to FIG. 22a, through a designated application (e.g., a music service application and / or an intelligent service application), information about “successful utterances” (e.g., “well-sounding utterances”), “failed utterances” (e.g., “poor-sounding utterances”), and / or utterance success rates related to the designated application may be provided through the electronic device (101). Referring to FIG. 22b, a designated website (e.g., www.samsung.com) may be provided.Information about “successful utterances” (e.g., “well-executed utterances”), “failed utterances” (e.g., “not-well-executed utterances”), “unsupported utterances” (e.g., “unsupported functions”) and / or utterance success rates related to a specified application may also be provided through the electronic device (101).

[0111] FIG. 23 is an exemplary diagram for explaining a function or operation of providing a service related to an unsupported utterance when an unsupported utterance is acquired according to an embodiment of the present document. Referring to FIG. 23, an electronic device (101) according to an embodiment of the present document may output a first message (2310) indicating that another application related to the designated application is executed when an unsupported utterance related to a designated application or an utterance similar to the unsupported utterance is acquired. Alternatively, an electronic device (101) according to an embodiment of the present document may output a second message (2320) indicating that a task according to the user utterance is not performed and the designated application is executed when an unsupported utterance related to a designated application or an utterance similar to the unsupported utterance is acquired. According to one embodiment of the present document, when an unsupported utterance or an utterance similar to an unsupported utterance related to a designated application is acquired, the electronic device (101) may execute another application that supports a task according to the unsupported utterance or execute the designated application without performing the task according to the user utterance. Information about the designated application and / or other applications according to one embodiment of the present document may be transmitted from the intelligent external device (600) to the electronic device (101).

[0112] FIG. 24 is a system flowchart for explaining a function or operation of an electronic device according to an embodiment of the present document to provide representative utterance sentences according to utterance categories based on utterance logs.

[0113] Referring to FIG. 24, an electronic device according to an embodiment of the present document may transmit information about a utterance log stored in the electronic device (101) to an intelligent external device (600) in operation 2410. The intelligent external device (600) (e.g., log collector (610)) according to an embodiment of the present document may transmit the acquired utterance log to an error detection module (615) (e.g., end-to-end error detection module (620), slot error detection module (630), failure type detection module (640), unsupported utterance detection module (650), and / or client and server error detection module (650)) in operation 2420. An intelligent referral device (600) according to one embodiment of the present document (e.g., an error detection module (615)) can predict a failure type in operation 2430. For example, utterance log information (900) according to one embodiment of the present document can be input to a pre-trained BERT (bidirectional encoder representation from transformer) model (1010). User utterance history information (900) as an output result of the pre-trained BERT model (1010) according to one embodiment of the present document can be input to a fast forward network (FNN) module (1020). User utterance history information (900) as an output result of the FNN module (1020) according to one embodiment of the present document can be classified and output by the type of utterance failure. In this case, the output result according to one embodiment of the present document can be, for example, "utterance with unclear intent", "multiple intent question", "incorrect service connection", "connection error", "ASR error" and / or It may include the probability of a failure type such as "unsupported utterance." According to one embodiment of the present document, the intelligent external device (600) (e.g., error detection module (615)) may transmit the predicted result according to operation 2430 to the real-time usability detection system (690) at operation 2340.A real-time usability detection system (690) according to an embodiment of the present document may determine an utterance category and a representative utterance sentence in operation 2450. An intelligent external device (600) according to an embodiment of the present document (e.g., a failed utterance extraction module) may determine at least one representative utterance sentence to be transmitted to the electronic device (101) among “failed utterances” and “similar utterances.” An intelligent external device (600) according to an embodiment of the present document (e.g., a successful utterance extraction module) may determine at least one representative utterance sentence to be transmitted to the electronic device (101) among “successful utterances” and “similar utterances.” An intelligent external device (600) according to an embodiment of the present document (e.g., a successful utterance extraction module) may determine at least one representative utterance sentence to be transmitted to the electronic device (101) among “unsupported utterances” and “similar utterances.” A real-time usability detection system (690) according to an embodiment of the present document may transmit utterance category information and representative utterance sentence information to an electronic device (101) in operation 2460. The electronic device (101) according to an embodiment of the present document may provide utterance categories and representative utterance sentences. According to an embodiment of the present document, information on “successful utterance” (e.g., “good utterance”), “failed utterance” (e.g., “bad utterance”), “unsupported utterance” (e.g., “unsupported function”), and / or utterance success rate may be provided through the electronic device (101). Through such functions or operations, when the electronic device (101) acquires an utterance according to a guide utterance (e.g., an utterance included in the “Try saying it like this”), a technical effect may be achieved that allows the user to know whether a normal response can be received “now.”

[0114] An electronic device (e.g., the electronic device (101) of FIG. 1) according to one embodiment of the present document may include a touchscreen display (e.g., the display module (160) of FIG. 1), a communication module (e.g., the communication module (190) of FIG. 1), and a processor (e.g., the processor (120) of FIG. 1), wherein the processor may be configured to transmit information on a user's speech history to an external device through the communication module, determine at least one speech category from the user's speech history transmitted to the external device by at least one error detection module stored in the external device, and, in response to the transmission, obtain information on a representative sentence belonging to the determined at least one speech category from the external device through the communication module, and provide information on the obtained representative sentence through the touchscreen display.

[0115] A method for controlling an electronic device according to one embodiment of the present document may include an operation of transmitting information on a user's speech history to an external device through a communication module of the electronic device, determining at least one speech category from the user's speech history transmitted to the external device by at least one error detection module stored in the external device, and, in response to the transmission, obtaining information on a representative sentence belonging to the determined at least one speech category from the external device through the communication module, and providing information on the obtained representative sentence through a touch screen display of the electronic device.

[0116] Electronic devices according to the various embodiments disclosed in this document may take various forms. Electronic devices may include, for example, portable communication devices (e.g., smartphones), computer devices, portable multimedia devices, portable medical devices, cameras, wearable devices, or home appliances. Electronic devices according to the embodiments of this document are not limited to the aforementioned devices.

[0117] The various embodiments of this document and the terminology used therein are not intended to limit the technical features described in this document to specific embodiments, but should be understood to include various modifications, equivalents, or substitutes of the embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of the items, unless the context clearly indicates otherwise. In this document, each of the phrases "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and "at least one of A, B, or C" can include any one of the items listed together in the corresponding phrase among those phrases, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used merely to distinguish one component from another, and do not limit the components in any other respect (e.g., importance or order). When a component (e.g., a first component) is referred to as "coupled" or "connected" to another component (e.g., a second component), with or without the terms "functionally" or "communicatively," it means that the component can be connected to the other component directly (e.g., wired), wirelessly, or through a third component.

[0118] The term "module" used in various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be an integral component, or a minimum unit or part of such a component that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0119] Various embodiments of the present document may be implemented as software (e.g., a program (2540)) including one or more instructions stored in a storage medium (e.g., an internal memory (2536) or an external memory (2538)) readable by a machine (e.g., an electronic device (2501)). For example, a processor of the machine (e.g., an electronic device (2501)) may call at least one instruction among the one or more instructions stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' simply means that the storage medium is a tangible device and does not contain signals (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently or temporarily on the storage medium.

[0120] According to one embodiment, the method according to various embodiments disclosed in the present document may be provided as included in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) via an application store (e.g., Play Store™) or directly between two user devices (e.g., smart phones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily generated in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0121] According to one embodiment of the present document, each component (e.g., a module or a program) of the above-described components may include one or more entities, and some of the entities may be separated and placed in other components. According to one embodiment of the present document, one or more components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (e.g., a module or a program) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to the integration. According to one embodiment of the present document, operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

Claims

1. In electronic devices, Touchscreen display, Communication module, at least one processor, and A memory comprising: a memory configured to store a plurality of instructions; wherein the plurality of instructions, when executed by the at least one processor, cause the electronic device to: Through the above communication module, information about the user's speech history is transmitted to an external device, and the user's speech history transmitted to the external device is classified into at least one speech category by at least one error detection module stored in the external device. In response to the above transmission, through the communication module, information about at least one classified utterance category and at least one representative utterance sentence belonging to the utterance category is obtained from the external device, and An electronic device characterized by including instructions for providing information about the acquired utterance category and the representative utterance sentence through the touch screen display.

2. In paragraph 1, An electronic device, characterized in that the above error detection module is generated by learning based on the user's speech history and the speech history of other users.

3. In paragraph 1 or 2, An electronic device, characterized in that the error detection module includes at least one of an end-to-end error detection module, a slot error detection module, a failure type detection module, or an unsupported ignition detection module.

4. In any one of paragraphs 1 to 3, An electronic device, wherein the above utterance category comprises at least one of utterance success, utterance failure, or unsupported utterance.

5. In any one of paragraphs 1 to 4, An electronic device, characterized in that the external device further comprises an integration module configured to integrate the results of the error detection module.

6. In any one of paragraphs 1 to 5, An electronic device, characterized in that the external device is set to transmit a message recommending an application capable of performing a task according to the user's utterance to the electronic device.

7. In any one of paragraphs 1 to 6, An electronic device, characterized in that the above representative sentence is determined based on the similarity with an utterance included in the utterance history.

8. In any one of paragraphs 1 to 7, An electronic device, characterized in that the plurality of instructions further include an instruction causing the electronic device to provide information on the utterance success rate transmitted from the external device together with the representative sentence through the touch screen display.

9. In any one of paragraphs 1 to 8, The above external device further includes a client error checking module, An electronic device characterized in that the client error verification module is set to classify the category of an utterance included in the utterance history as a successful utterance when an error related to an utterance included in the utterance history is confirmed by the user's intention.

10. In any one of paragraphs 1 to 9, The above external device further includes a server error checking module, An electronic device characterized in that the server error confirmation module is set to classify the category of an utterance included in the utterance history as an utterance failure when an error related to an utterance included in the utterance history is confirmed as a server error.

11. A computer-readable non-transitory recording medium configured to store a plurality of instructions, wherein the plurality of instructions, when executed by at least one processor of an electronic device, cause the electronic device to: Through the communication module of the electronic device, information on the user's speech history is transmitted to an external device, and the user's speech history transmitted to the external device is classified into at least one speech category by at least one error detection module stored in the external device. In response to the above transmission, through the communication module, information about at least one classified utterance category and at least one representative utterance sentence belonging to the utterance category is obtained from the external device, and A computer-readable non-transitory recording medium, characterized in that it includes instructions for providing information about the acquired utterance category and the representative utterance sentence through a touch screen display of the electronic device.

12. In paragraph 11, A computer-readable non-transitory recording medium, characterized in that the above error detection module is generated by learning based on the user's speech history and the speech history of other users.

13. In paragraph 11 or 12, A computer-readable non-transitory recording medium, characterized in that the error detection module includes at least one of an end-to-end error detection module, a slot error detection module, a failure type detection module, or an unsupported ignition detection module.

14. In any one of paragraphs 11 to 13, A computer-readable non-transitory recording medium, characterized in that the above utterance category includes at least one of utterance success, utterance failure, or unsupported utterance.

15. In any one of paragraphs 11 to 14, A computer-readable non-transitory recording medium, characterized in that the external device further comprises an integration module set to integrate the results of the error detection module.

Images