Electronic device and user utterance processing method

Abstract

An electronic device according to an embodiment may include a memory to store instructions. The electronic device may include at least one processor. The instructions, when executed by the at least one processor, may cause the at least one processor to: receive a first signal corresponding to a first utterance of a user; providing a first response corresponding to the first utterance; obtaining cross-domain data based on the first utterance or the first response; receiving a second signal corresponding to a second utterance following the first utterance; and providing a second response corresponding to the second utterance based on the cross-domain data if the second utterance includes a referral. The first utterance and the second utterance may respectively correspond to different domains.

Description

Technical Field

[0001] The disclosed embodiments relate to an electronic device and a method for processing user speech. Background Art

[0002] Electronic devices including voice assistant functions that provide services based on user speech have been widely distributed. The electronic device can recognize user speech through an artificial intelligence (AI) server, and can figure out the meaning and intention of the user speech. The AI ​​server can infer the user's intention by interpreting the user's speech and perform tasks based on the inferred intention. The AI ​​server can perform tasks based on the user's intention expressed through natural language interaction between the user and the AI ​​server.

[0003] The AI ​​server's operation of classifying the user's intent to perform a task may be performed by using a neural network-based model. A neural network-based classification model may require many training utterances. Summary of the invention

[0004] Technical Solution According to one aspect of the present disclosure, an electronic device may include: a memory storing instructions. The electronic device may include at least one processor. When the instructions are executed by the at least one processor, the at least one processor may: receive a first signal corresponding to a first utterance of a user; provide a first response corresponding to the first utterance; obtain cross-domain data based on the first utterance or the first response; receive a second signal corresponding to a second utterance of the user after the first utterance; and if the second utterance includes a pronoun, provide a second response corresponding to the second utterance based on the cross-domain data. The first utterance and the second utterance may correspond to different domains, respectively.

[0005] According to one aspect of the present disclosure, a method for operating an electronic device may include: receiving a first signal corresponding to a first utterance; guiding a user interface to provide a first response corresponding to the first utterance; obtaining cross-domain data based on the first utterance or the first response; receiving a second signal corresponding to a second utterance of a user after the first utterance; and when the second utterance includes a pronoun, guiding the user interface to provide a second response corresponding to the second utterance based on the cross-domain data. The first utterance and the second utterance may correspond to different domains, respectively. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] Figure 1 is a block diagram illustrating electronic devices in a network environment according to an embodiment.

[0007] Figure 2 is a block diagram illustrating an integrated intelligent system according to an embodiment.

[0008] Figure 3 is a diagram showing a form in which relationship information between concepts and actions is stored in a database according to an embodiment.

[0009] Figure 4 is a diagram illustrating a screen in which an electronic device processes a received voice input through a smart application according to an embodiment.

[0010] Figure 5 is a diagram illustrating an operation of an electronic device processing a user utterance according to an embodiment.

[0011] Figure 6 is a schematic block diagram of an electronic device according to an embodiment.

[0012] Figure 7 is a diagram illustrating an operation of an electronic device processing a user utterance according to an embodiment.

[0013] Figure 8 is a diagram illustrating an operation of an electronic device processing a user utterance according to an embodiment.

[0014] Fig. 9 is a flowchart illustrating an operating method of an electronic device according to an embodiment. DETAILED DESCRIPTION

[0015] Hereinafter, the embodiments are described in detail with reference to the accompanying drawings. When describing the embodiments with reference to the accompanying drawings, the same reference numerals denote the same elements, and repeated descriptions thereof will be omitted.

[0016] Figure 1 1 is a block diagram showing an electronic device 101 in a network environment 100 according to an embodiment. Figure 1, the electronic device 101 in the network environment 100 may communicate with the electronic device 102 via the first network 198 (e.g., a short-range wireless communication network), or communicate with the electronic device 104 or the server 108 via the second network 199 (e.g., a long-range wireless communication network). The electronic device 101 may communicate with the electronic device 104 via the server 108. The electronic device 101 may include a processor 120, a memory 130, an input module 150, a sound output module 155, a display module 160, an audio module 170, and a sensor module 176, an interface 177, a connection terminal 178, a haptic module 179, a camera module 180, a power management module 188, a battery 189, a communication module 190, a user identification module (SIM) 196, or an antenna module 197. In some embodiments, at least one of the components (e.g., the connection terminal 178) may be omitted from the electronic device 101, or one or more other components may be added to the electronic device 101. In some embodiments, some of the components (eg, sensor module 176, camera module 180, or antenna module 197) may be integrated into a single component (eg, display module 160).

[0017] The processor 120 may run, for example, software (e.g., program 140) to control at least one other component (e.g., hardware component or software component) of the electronic device 101 connected to the processor 120, and may perform various data processing or calculations. As at least part of the data processing or calculation, the processor 120 may store a command or data received from another component (e.g., sensor module 176 or communication module 190) in the volatile memory 132, process the command or data stored in the volatile memory 132, and store the resultant data in the non-volatile memory 134. The processor 120 may include a main processor 121 (e.g., a central processing unit (CPU) or an application processor (AP)) and an auxiliary processor 123 (e.g., a graphics processing unit (GPU), a neural processing unit (NPU), an image signal processor (ISP), a sensor hub processor, or a communication processor (CP)) that is independent of or combined with the main processor 121 in operation. For example, when the electronic device 101 includes a main processor 121 and an auxiliary processor 123, the auxiliary processor 123 may be adapted to consume less power than the main processor 121, or adapted to be specifically used for a specified function. The auxiliary processor 123 may be implemented as a separate part from the main processor 121, or as part of the main processor 121.

[0018] When the main processor 121 is in an inactive (e.g., sleep) state, the auxiliary processor 123 (rather than the main processor 121), for example, may control at least some of the functions or states related to at least one component among the components of the electronic device 101 (e.g., display module 160, sensor module 176, or communication module 190), or when the main processor 121 is in an active state (e.g., running an application), the auxiliary processor 123 may control at least some of the functions or states related to at least one component among the components of the electronic device 101 (e.g., display module 160, sensor module 176, or communication module 190) together with the main processor 121. According to one embodiment, the auxiliary processor 123 (e.g., image signal processor or communication processor) may be implemented as part of another component (e.g., camera module 180 or communication module 190) that is functionally related to the auxiliary processor 123. According to one embodiment, the auxiliary processor 123 (e.g., NPU) may include a hardware structure dedicated to artificial intelligence model processing. The artificial intelligence (AI) model may be generated by machine learning, and such learning may be performed by, for example, the electronic device 901 where the artificial intelligence is performed or via a separate server (e.g., server 908). The learning algorithm may include, but is not limited to, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning. The artificial intelligence (AI) model may include multiple artificial neural network layers. The artificial neural network may include, for example, a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q network, or a combination of two or more thereof, but is not limited thereto. Additionally or optionally, the AI ​​model may include a software structure in addition to a hardware structure.

[0019] The memory 130 may store various pieces of data used by at least one component of the electronic device 101 (e.g., the processor 120 or the sensor module 176). The data may include, for example, software (e.g., the program 140) and input data or output data for commands related thereto. The memory 130 may include a volatile memory 132 or a nonvolatile memory 134.

[0020] The program 140 may be stored as software in the memory 130 , and may include, for example, an operating system (OS) 142 , middleware 144 , or applications 146 .

[0021] The input device 150 may receive commands or data from outside the electronic device 101 (e.g., a user) to be used by other components of the electronic device 101 (e.g., the processor 120). The input device 150 may include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus).

[0022] The sound output module 155 can output sound signals to the outside of the electronic device 101. The sound output module 155 may include, for example, a speaker or a receiver. The speaker can be used for general purposes such as playing multimedia or playing records. The receiver can be used to receive incoming calls. The receiver can be implemented as a separate part from the speaker, or as part of the speaker.

[0023] The display module 160 may visually provide information to the outside of the electronic device 101 (e.g., a user). The display module 160 may include, for example, a display, a holographic device, or a projector, and a control circuit for controlling a corresponding one of the display, the holographic device, and the projector. The display module 160 may include a touch sensor adapted to detect a touch or a pressure sensor adapted to measure the strength of a force caused by a touch.

[0024] The audio module 170 may convert sound into an electrical signal, or vice versa. According to an embodiment, the audio module 170 may obtain sound via the input module 150, or output sound via the sound output module 155 or an external electronic device (e.g., the electronic device 102, such as a speaker or earphone) directly or wirelessly connected to the electronic device 101.

[0025] The sensor module 176 may detect an operating state (e.g., power or temperature) of the electronic device 101 or an environmental state (e.g., a state of a user) outside the electronic device 101, and generate an electrical signal or data value corresponding to the detected state. According to an embodiment, the sensor module 176 may include, for example, a gesture sensor, a gyro sensor, an atmospheric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an infrared (IR) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illumination sensor.

[0026] The interface 177 may support one or more designated protocols for connecting the electronic device 101 to an external electronic device (e.g., electronic device 102) directly (e.g., via a cable) or wirelessly. According to an embodiment, the interface 177 may include, for example, a high-definition multimedia interface (HDMI), a universal serial bus (USB) interface, a secure digital (SD) card interface, or an audio interface.

[0027] The connection end 178 may include a connector, wherein the electronic device 101 can be physically connected to an external electronic device (e.g., the electronic device 102) via the connector. According to an embodiment, the connection end 178 may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0028] The haptic module 179 may convert the electrical signal into mechanical stimulation (eg, vibration or motion) or electrical stimulation that can be recognized by the user via their sense of touch or kinesthetic sense. According to an embodiment, the haptic module 179 may include, for example, a motor, a piezoelectric element, or an electrical stimulator.

[0029] The camera module 180 may capture still images and moving images. According to an embodiment, the camera module 180 may include one or more lenses, an image sensor, an image signal processor, or a flash.

[0030] The power management module 188 may manage power supply to the electronic device 101. According to an embodiment, the power management module 188 may be implemented as, for example, at least a part of a power management integrated circuit (PMIC).

[0031] The battery 189 may power at least one component of the electronic device 101. According to an embodiment, the battery 189 may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0032] The communication module 190 may support establishing a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device 101 and an external electronic device (e.g., electronic device 102, electronic device 104, or server 108), and perform communication via the established communication channel. The communication module 190 may include one or more CPs that can operate independently with the processor 120 (e.g., AP) and support direct (e.g., wired) communication or wireless communication. According to an embodiment, the communication module 190 may include a wireless communication module 192 (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module 194 (e.g., a local area network (LAN) communication module or a power line communication (PLC) module). A corresponding one of these communication modules may communicate with the external electronic device 104 via a first network 198 (e.g., a short-range communication network such as Bluetooth, Wireless Fidelity (Wi-Fi) Direct, or Infrared Data Association (IrDA)) or a second network 199 (e.g., a long-range communication network such as a traditional cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or a Wide Area Network (WAN))). These various types of communication modules may be implemented as a single component (e.g., a single chip), or may be implemented as a plurality of components (e.g., a plurality of chips) separated from each other. The wireless communication module 192 may identify and authenticate the electronic device 101 in a communication network such as the first network 198 or the second network 199 using user information (e.g., an International Mobile Subscriber Identity (IMSI)) stored in the SIM 196.

[0033] The wireless communication module 192 may support 5G networks after 4G networks and next generation communication technologies (e.g., new radio (NR) access technologies). NR access technologies may support enhanced mobile broadband (eMBB), massive machine type communications (mMTC), or ultra-reliable low latency communications (URLLC). The wireless communication module 192 may support high frequency bands (e.g., millimeter wave bands) to achieve, for example, high data transmission rates. The wireless communication module 192 may support various technologies for ensuring performance on high frequency bands, such as, for example, beamforming, massive multiple input multiple output (MIMO), full dimensional MIMO (FD-MIMO), array antennas, analog beamforming, or massive antennas. The wireless communication module 192 may support various requirements specified in the electronic device 101, an external electronic device (e.g., electronic device 104), or a network system (e.g., a second network 199). According to one embodiment, the wireless communication module 192 may support a peak data rate for implementing eMBB (e.g., 20 Gbps or greater), loss coverage for implementing mMTC (e.g., 164 dB or less), or U-plane latency for implementing URLLC (e.g., 0.5 ms or less for each of the downlink (DL) and uplink (UL), or 1 ms or less round trip).

[0034] The antenna module 197 may transmit or receive a signal or power to or from the outside of the electronic device 101 (e.g., an external electronic device). According to one embodiment, the antenna module 197 may include an antenna including a radiating element formed of a conductive material or a conductive pattern formed in or on a substrate (e.g., a printed circuit board (PCB)). According to one embodiment, the antenna module 197 may include a plurality of antennas (e.g., an antenna array). In this case, at least one antenna suitable for a communication scheme used in a communication network (such as the first network 198 or the second network 199) may be selected from the plurality of antennas by, for example, the communication module 190. A signal or power may be transmitted or received between the communication module 190 and the external electronic device via the selected at least one antenna. According to one embodiment, another component (e.g., a radio frequency integrated circuit (RFIC)) other than the radiating element may be additionally formed as a part of the antenna module 197.

[0035] According to an embodiment, the antenna module 197 may form a millimeter wave antenna module. According to one embodiment, the millimeter wave antenna module may include a PCB, an RFIC, and a plurality of antennas (e.g., array antennas), wherein the RFIC is on a first surface (e.g., bottom surface) of the printed circuit board, or is adjacent to the first surface of the PCB and is capable of supporting a specified high frequency band (e.g., millimeter wave band), and the plurality of antennas are disposed on a second surface (e.g., top surface or side surface) of the PCB, or is adjacent to the second surface of the PCB and is capable of transmitting or receiving signals of the specified high frequency band.

[0036] At least some of the above components may be connected to each other via an inter-peripheral communication scheme (e.g., a bus, a general purpose input output (GPIO), a serial peripheral interface (SPI), or a mobile industry processor interface (MIPI)) and communicatively transfer signals (e.g., commands or data) therebetween.

[0037] According to an embodiment, a command or data may be sent or received between the electronic device 101 and the external electronic device 104 via the server 108 connected to the second network 199. Each of the external electronic device 102 or the external electronic device 104 may be a device of the same type as the electronic device 101, or a device of a different type from the electronic device 101. According to one embodiment, all or some operations to be performed by the electronic device 101 may be performed by one or more of the external electronic devices (e.g., the electronic device 102 and the electronic device 104 and the server 108). For example, if the electronic device 101 needs to automatically perform a function or service or should perform a function or service in response to a request from a user or another device, the electronic device 101 may request one or more external electronic devices to perform at least part of the function or service instead of running the function or service, or the electronic device 101 may request one or more external electronic devices to perform at least part of the function or service in addition to running the function or service. The one or more external electronic devices that receive the request may execute at least part of the function or service, or execute another function or another service related to the request, and may transmit the result of the execution to the electronic device 101. The electronic device 101 may provide the result as at least part of the response to the request with or without further processing the result. To this end, cloud computing technology, distributed computing technology, mobile edge computing (MEC) technology, or client-server computing technology, for example, may be used. The electronic device 101 may use, for example, distributed computing or MEC to provide ultra-low latency services. In an embodiment, the external electronic device (e.g., electronic device 104) may include an Internet of Things (IoT) device. The server 108 may be an intelligent server using machine learning and / or neural networks. According to one embodiment, the external electronic device (e.g., electronic device 104) or the server 108 may be included in the second network 199. The electronic device 101 may be applied to intelligent services (e.g., smart homes, smart cities, smart cars, or healthcare) based on 5G communication technology or IoT-related technologies.

[0038] Figure 2 is a block diagram illustrating an integrated intelligent system according to an embodiment.

[0039] refer to Figure 2 , the integrated intelligent system 20 may include an electronic device 201 (e.g., Figure 1 electronic device 101), intelligent server 200 (for example, Figure 1 Server 108 and service server 300 (eg, Figure 1 server 108).

[0040] The electronic device 101 of one embodiment may be a terminal device (or electronic device) that can be connected to the Internet, and may be, for example, a mobile phone, a smart phone, a personal digital assistant (PDA), a notebook computer, a TV, a white appliance, a wearable device, a head-mounted display (HMD), or a smart speaker.

[0041] According to the illustrated embodiment, the electronic device 201 may include a communication interface 202 (eg, Figure 1 interface 177), microphone 206 (eg, Figure 1 Input module 150), speaker 205 (e.g., Figure 1 The sound output module 155), the display module 204 (for example, Figure 1 display module 160), memory 207 (for example, Figure 1 memory 130) or processor 203 (e.g., Figure 1 The components listed above may be operatively connected to each other or electrically connected to each other.

[0042] The communication interface 202 of one embodiment can be connected to an external device and is configured to send data to and receive data from the external device. The microphone 206 can receive sound (e.g., user speech) and convert the sound into an electrical signal. The speaker 205 can output the electrical signal as sound (e.g., voice).

[0043] The display module 204 of one embodiment may be configured to display images or videos. The display module 204 may also display a graphical user interface (GUI) of an application (or application program) being executed. The display module 204 may receive touch input via a touch sensor. For example, the display module 204 may receive text input in an on-screen keyboard area displayed in the display module 204 via a touch sensor.

[0044] The memory 207 of one embodiment may store a client module 209, a software development kit (SDK) 208, and a plurality of applications 211. The client module 209 and the SDK 208 may be configured as a framework (or solution program) for performing common functions. In addition, the client module 209 or the SDK 208 may be configured as a framework for processing user input (e.g., voice input, text input, or touch input).

[0045] The plurality of applications 211 stored in the memory 207 of one embodiment may be programs for performing a specified function. The plurality of applications 211 may include a first application 211-1 and a second application 211-2. Each of the plurality of applications 211 may include a plurality of actions for performing a specified function. For example, the application may include an alarm clock application, a messaging application, and / or a calendar application. The plurality of applications 211 may be executed by the processor 203 to sequentially perform at least a portion of the plurality of actions.

[0046] The processor 203 of one embodiment may control the overall operation of the electronic device 201. For example, the processor 203 may be electrically connected to the communication interface 202, the microphone 206, the speaker 205, and the display module 204 to perform designated operations.

[0047] The processor 203 of one embodiment may also perform a designated function by executing a program stored in the memory 207. For example, the processor 203 may execute at least one of the client module 209 or the SDK 208 to perform the following operations for processing user input. The processor 203 may control the operations of the plurality of applications 211 through, for example, the SDK 208. The following operations as operations of the client module 209 or the SDK 208 may be performed by the processor 203.

[0048] The client module 209 of one embodiment can receive user input. For example, the client module 209 can receive a voice signal corresponding to a user's speech sensed by the microphone 206. As another example, the client module 209 can receive a touch input sensed by the display module 204. As yet another example, the client module 209 can receive a text input sensed by a keyboard or an on-screen keyboard. In addition, the client module 209 can receive various types of user input sensed by an input module included in the electronic device 201 or an input module connected to the electronic device 201. The client module 209 can send the received user input to the intelligent server 200. The client module 209 can send the state information of the electronic device 201 to the intelligent server 200 together with the received user input. The state information can be, for example, execution state information of an application.

[0049] The client module 209 of one embodiment may receive a result corresponding to the received user input. For example, when the intelligent server 200 is capable of calculating a result corresponding to the received user input, the client module 209 may receive a result corresponding to the received user input. The client module 209 may display the received result on the display module 204. In addition, the client module 209 may output the received result in audio form through the speaker 205.

[0050] The client module 209 of one embodiment may receive a plan corresponding to the received user input. The client module 209 may display the results of executing multiple actions of the application according to the plan on the display module 204. For example, the client module 209 may sequentially display the results of executing multiple actions on the display module 204, and output the results in audio form through the speaker 205. As another example, the electronic device 201 may display only a portion of the results of executing multiple actions (e.g., the result of the last action) on the display module 204, and output a portion of the results in audio form through the speaker 205.

[0051] According to an embodiment, the client module 209 may receive a request for obtaining information necessary to calculate a result corresponding to a user input from the smart server 200. The client module 209 may transmit the necessary information to the smart server 200 in response to the request.

[0052] The client module 209 of one embodiment may send information about the results of performing the multiple actions according to the plan to the smart server 200. The smart server 200 may use the information about the results to confirm that the received user input has been processed correctly.

[0053] The client module 209 of one embodiment may include a voice recognition module. The client module 209 may recognize voice input for executing a limited function through the voice recognition module. For example, the client module 209 may execute an intelligent application for processing voice input to perform an organic operation through a specified input (e.g., wake up!).

[0054] The intelligent server 200 of one embodiment may receive information related to the user voice input from the electronic device 201 through the communication network. The intelligent server 200 may change the data related to the received voice input into text data. The intelligent server 200 may generate a plan for executing a task corresponding to the user voice input based on the text data.

[0055] According to an embodiment, the plan may be generated by an AI system. The AI ​​system may be a rule-based system or a neural network-based system (e.g., a feedforward neural network (FNN) or a recurrent neural network (RNN)). Alternatively, the AI ​​system may be a combination thereof or other AI systems. The plan may be selected from a set of predefined plans, or may be generated in real time in response to a user request. For example, the AI ​​system may select at least one plan from the predefined plans.

[0056] The intelligent server 200 of one embodiment may send the result according to the generated plan to the electronic device 201, or send the generated plan to the electronic device 201. The electronic device 201 may display the result according to the plan on the display module 204. The electronic device 201 may display the result of executing the action according to the plan on the display module 204.

[0057] The intelligent server 200 of one embodiment may include a front end 210 , a natural language platform 220 , a capsule database (DB) 230 , an execution engine 240 , an end-user interface 250 , a management platform 260 , a big data platform 270 , or an analysis platform 280 .

[0058] The front end 210 of one embodiment may receive the received user input from the electronic device 201. The front end 210 may send a response corresponding to the user input.

[0059] According to an embodiment, the natural language platform 220 may include an automatic speech recognition (ASR) module 221, a natural language understanding (NLU) module 223, a planner module 225, a natural language generator (NLG) module 227, or a text to speech (TTS) module 229.

[0060] The ASR module 221 of one embodiment may convert data related to the voice input received from the electronic device 201 into text data. The NLU module 223 of one embodiment may determine the domain (and / or intent information) corresponding to the voice input by using the text data of the voice input (e.g., user utterance). The domain may be a category (or service) associated with the operation (or function) that the user desires to perform by using the device. The domain may be classified based on the service (e.g., application) associated with the text. For example, the Gracenote domain may correspond to a music search service (e.g., Gracenote service). The Melon domain may correspond to a music playback service (e.g., Melon service). The domain may be associated with the intent information corresponding to the text. The NLU module 223 may discern the user's intent by using the text data of the voice input. For example, the NLU module 223 may discern the user's intent by performing syntactic analysis or semantic analysis on the user input in the form of text data. The NLU module 223 may discern the meaning of the words extracted from the user input using the linguistic features (e.g., grammatical elements) of morphemes or phrases, and determine the user's intent by matching the discerned meaning of the words with the intent. The NLU module 223 may obtain intent information corresponding to the user utterance. The intent information may be information indicating the intent of the user determined by analyzing the text data. The intent information may include information indicating the action (or function) that the user intends to perform using the device. The slot information may be detailed information related to the intent information. The slot information may be obtained based on the domain corresponding to the utterance. The slot information may be variable information required to perform the operation. The variable information included in the slot may include a named entity. For example, when the text converted from the user's voice input is "What time is it in San Francisco now?", the domain may be a "date and time domain", the intent information may correspond to "date / time information provision", and the slot information may be "San Francisco". For example, when the text converted from the user's voice input is "What's the weather like here?", the domain may be a "weather domain", the intent information may correspond to "weather information provision", and there may be no slot information.

[0061] The planner module 225 of one embodiment may generate a plan using the parameters and intent determined by the NLU module 223. The planner module 225 may determine multiple domains required to perform a task based on the determined intent. The planner module 225 may determine multiple actions included in each of the multiple domains determined based on the intent. The planner module 225 may determine the parameters required to perform the multiple actions determined or the result values ​​output by performing the multiple actions. The parameters and the result values ​​may be defined as concepts of a specified form (or category). Therefore, the plan may include multiple actions and multiple concepts determined by the user's intent. The planner module 225 may determine the relationship between the multiple actions and the multiple concepts step by step (or hierarchically). For example, the planner module 225 may determine the execution order of the multiple actions determined based on the user's intent based on the multiple concepts. In other words, the planner module 225 may determine the execution order of the multiple actions based on the parameters required to perform the multiple actions and the results output by performing the multiple actions. Therefore, the planner module 225 may generate a plan including connection information (e.g., ontology) between the multiple actions and the multiple concepts. The planner module 225 may generate a plan using information stored in the capsule DB 230 , where the capsule DB 230 stores a set of relationships between concepts and actions.

[0062] The NLG module 227 of one embodiment may change the specified information into a text form. The information changed into a text form may be in the form of a natural language speech. The TTS module 229 may change the information in the text form into the information in the voice form.

[0063] According to an embodiment, some or all functions of the natural language platform 220 may also be implemented in the electronic device 201 .

[0064] The capsule DB 230 may store information about the relationship between a plurality of concepts and actions corresponding to a plurality of domains. A capsule according to an embodiment may include a plurality of action objects (or action information) and a concept object (or concept information) included in a plan. According to an embodiment, the capsule DB 230 may store a plurality of capsules in the form of a concept action network (CAN). According to an embodiment, a plurality of capsules may be stored in a function registry included in the capsule DB 230.

[0065] The capsule DB 230 may include a policy registry storing policy information required to determine a plan corresponding to the voice input. The policy information may include reference information for determining a plan when there are multiple plans corresponding to the user input. The capsule DB 230 may include a subsequent registry storing information about subsequent actions for suggesting subsequent actions to the user in a specified situation. The subsequent action may include, for example, a subsequent utterance. The capsule DB 230 may include a layout registry storing layout information, wherein the layout information is information output by the electronic device 201. The capsule DB 230 may include a vocabulary registry storing vocabulary information included in the capsule information. The capsule DB 230 may include a dialogue registry storing information about a dialogue (or interaction) with the user. The capsule DB 230 may update the stored objects through the developer tool. The developer tool may include, for example, a function editor for updating an action object or a concept object. The developer tool may include a vocabulary editor for updating. The developer tool may include a policy editor for generating and registering a policy for determining a plan. The developer tool may include a dialogue editor for generating a dialogue with the user. The developer tool may include a subsequent editor for activating a subsequent target and editing a subsequent utterance providing a prompt. The subsequent goal may be determined based on the currently set goal, the user's preference, or environmental conditions. In an embodiment, the capsule DB 230 may also be implemented in the electronic device 201 .

[0066] The execution engine 240 of one embodiment can use the generated plan to calculate the result. The terminal user interface 250 can send the calculated result to the electronic device 201. Therefore, the electronic device 201 can receive the result and provide the received result to the user. The management platform 260 can manage the information used by the intelligent server 200. The big data platform 270 can collect the user's data. The analysis platform 280 can manage the quality of service (QoS) of the intelligent server 200. For example, the analysis platform 280 can manage the components and processing rate (or efficiency) of the intelligent server 200.

[0067] The service server 300 of one embodiment may provide a specified service (e.g., food ordering or hotel reservation) to the electronic device 201. According to an embodiment, the service server 300 may be a server operated by a library administrator. The service server 300 of one embodiment may provide the smart server 200 with information to be used to generate a plan corresponding to the received user input. The provided information may be stored in the capsule DB 230. In addition, the service server 300 may provide the smart server 200 with result information according to the plan.

[0068] In the integrated intelligent system 20, the electronic device 101 can provide various intelligent services to the user in response to user input. The user input can include, for example, input through a physical button, touch input, or voice input.

[0069] In an embodiment, the electronic device 201 can provide a voice recognition service through an intelligent application (or voice recognition application) stored therein. In this case, for example, the electronic device 201 can recognize a user's speech or voice input received through a microphone and provide a service corresponding to the recognized voice input to the user.

[0070] In an embodiment, the electronic device 201 can perform a specified action based on the received voice input alone or together with the intelligent server and / or the service server. For example, the electronic device 201 can execute an application corresponding to the received voice input and perform a specified action through the executed application.

[0071] In an embodiment, when the electronic device 201 provides a service together with the intelligent server 200 and / or the service server 300, the electronic device 201 can detect user speech using the microphone 206 and generate a signal (or voice data) corresponding to the detected user speech. The electronic device 201 can use the communication interface 202 to send the voice data to the intelligent server 200.

[0072] In response to the voice input received from the electronic device 201, the intelligent server 200 of one embodiment can generate a plan for performing a task corresponding to the voice input or the result of performing an action according to the plan. The plan may include, for example, multiple actions for performing tasks corresponding to the user's voice input, and multiple concepts related to the multiple actions. A concept may be defined as a parameter input for the execution of multiple actions or a result value output by the execution of multiple actions. The plan may include connection information between multiple actions and multiple concepts.

[0073] The electronic device 201 of one embodiment may receive the response using the communication interface 202. The electronic device 201 may output a voice signal generated inside the electronic device 201 to the outside using the speaker 205, or output an image generated inside the electronic device 201 to the outside using the display module 204.

[0074] Figure 3 : is a diagram showing a form in which relationship information between concepts and actions according to an embodiment is stored in a database.

[0075] Intelligent servers (e.g. Figure 2 The capsule DB of the intelligent server 200 (for example, Figure 2The capsule DB 230 may store capsules in the form of CAN 400. The capsule DB may store an action for processing a task corresponding to a user's voice input and parameters required for the action in the form of CAN.

[0076] The capsule DB may store a plurality of capsules (capsule A 401 and capsule B 404) corresponding to a plurality of domains (e.g., applications), respectively. One capsule (e.g., capsule A 401) may correspond to one domain (e.g., location (geography) or application). In addition, one capsule may correspond to at least one service provider (e.g., CP1 402 or CP2 403) for performing a function of a domain related to the capsule. One capsule may include at least one action 410 and at least one concept 420 for performing a specified function.

[0077] Above, natural language platforms (e.g. Figure 2 The natural language platform 220 of the present invention can use the capsules stored in the capsule DB to generate a plan for performing a task corresponding to the received speech input. For example, a planner module of the natural language platform (e.g., Figure 2 The planner module 225 of the embodiment can generate a plan using capsules stored in the capsule DB. For example, the action 4011 and 4013 and the concepts 4012 and 4014 of capsule A 401 and the action 4041 and the concept 4042 of capsule B 404 can be used to generate a plan 407.

[0078] Figure 4 is a diagram illustrating a screen in which an electronic device processes a received voice input through a smart application according to an embodiment.

[0079] The electronic device 201 may execute the smart application to communicate with the smart server (eg, Figure 2 The intelligent server 200) processes the user input.

[0080] According to an embodiment, on the screen 310, when a designated voice input (e.g., wake up!) is recognized or an input through a hardware key (e.g., a dedicated hardware key) is received, the electronic device 201 may execute a smart application for processing the voice input. The electronic device 201 may, for example, execute the smart application in a state where a scheduling application is executed. The electronic device 201 may display the smart application on the display module 204 (e.g., Figure 1 The display module 160 and Figure 2The electronic device 201 may display an object (e.g., icon) 311 corresponding to the smart application on the display module 204 of the electronic device 201. The electronic device 201 may receive a voice input through a user's speech. For example, the electronic device 201 may receive a voice input of "Tell me about this week's schedule!" The electronic device 201 may display a user interface (UI) 313 (e.g., input window) of the smart application on the display module 204, wherein the text data of the received voice input is displayed in the user interface 313.

[0081] According to an embodiment, on screen 320, the electronic device 201 may display a result corresponding to the received voice input on the display module 204. For example, the electronic device 201 may receive a plan corresponding to the received user input, and display “this week’s schedule” on the display module 204 according to the plan.

[0082] Figure 5 is a diagram illustrating an operation of an electronic device processing a user utterance according to an embodiment.

[0083] refer to Figure 5 In an embodiment, the electronic device 501 may include a reference Figure 1 The electronic device 101 described and the reference Figure 2 At least some components of the electronic device 201 described herein. The intelligent server 601 may include reference Figure 2 At least some components of the smart server 200 described herein. For the electronic device 501 and the smart server 601, refer to Figures 1 to 4 The duplicate descriptions provided are omitted.

[0084] Electronic device 501 (eg, Figure 1 The electronic device 101 or Figure 2 The electronic device 201) can be connected to the intelligent server 601 (e.g., Figure 2 The electronic device 501 and the smart server 601 may communicate with each other through a wired communication method or a wireless communication method (e.g., wireless LAN (Wi-Fi), Bluetooth, Bluetooth low energy, ZigBee, Wi-Fi Direct (WFD), ultra-wideband (UWB), infrared data association (IrDA), and near field communication (NFC)).

[0085] According to an embodiment, the electronic device 501 can be implemented as at least one of a smart phone, a tablet personal computer (PC), a mobile phone, a speaker (e.g., an artificial intelligence (AI) speaker), a video phone, an e-book reader, a desktop PC, a laptop PC, a netbook computer, a workstation, a server, a personal digital assistant (PDA), a portable multimedia player (PMP), an MP3 player, a mobile medical device, a camera, or a wearable device.

[0086] According to an embodiment, the electronic device 501 may obtain a voice signal corresponding to the user's speech, and may send the voice signal to the intelligent server 601. The intelligent server 601 may obtain text data corresponding to the user's speech based on the voice signal. The text data may be obtained by converting the speech portion into a computer-readable text by performing ASR on the voice signal. The intelligent server 601 may use the text data to analyze the user's speech. The intelligent server 601 may use the analysis results (e.g., intent information, entities, and / or capsules) to perform the desired function, or may provide a response (e.g., question and answer) to be provided to the user to the device (e.g., the electronic device 501). The intelligent server 601 may be implemented as software. Some or all of the intelligent servers 601 may be implemented on the electronic device 501 and / or the intelligent server 601 (e.g., Figure 2 The on-device AI for processing speech without communicating with the intelligent server 601 can be installed on the electronic device 501. Figures 2 to 4 The described components of the natural language platform 220 may be implemented in an electronic device 501 .

[0087] According to an embodiment, the ASR module 221 (eg, Figure 2 The ASR module of the natural language platform 220 can convert the user utterance into text data. The NLU module 223 included in the natural language platform 220 can determine the domain (and / or intent information) corresponding to the user utterance by using the text data corresponding to the user utterance. The domain can be a category (or service) associated with the operation (or function) that the user expects to perform by using the device. The domain can be classified based on the service (e.g., application) related to the user utterance. For example, the Gracenote domain can correspond to a music search service (e.g., Gracenote service). For example, the Melon domain can correspond to a music playback service (e.g., Melon service). The multiple user utterances can be processed separately based on the respective domains corresponding to the multiple user utterances.

[0088] According to an embodiment, the electronic device 501 may support a cross-domain scenario. A cross-domain scenario may mean that a plurality of utterances (e.g., continuous utterances) may correspond to different domains, respectively. When processing utterances (e.g., continuous utterances) in a single domain, information (e.g., information corresponding to a pronoun) included in a previous utterance (e.g., an utterance before a subsequent utterance) may be used to process a subsequent utterance (e.g., a subsequent utterance in a continuous utterance) including a pronoun (or not enough to include the information required for utterance processing). When processing continuous utterances (e.g., a cross-domain scenario) separately in different domains, the information included in the previous utterance may not be easily used to process a subsequent utterance including a pronoun (or not enough to include the information required for utterance processing). According to information security policies, in a cross-domain scenario, information exchange may not be easily available between different domains. In a cross-domain scenario, the electronic device 501 may use information (e.g., cross-domain data) included in a previous utterance. The electronic device 501 may support a cross-domain scenario by using cross-domain data obtained from a user utterance.

[0089] Reference Figure 5 , the electronic device 501 may receive an utterance (e.g., "What time is it in San Francisco now?"). The electronic device 501 may receive a subsequent utterance (e.g., "What's the weather like here?") after the utterance (e.g., "What time is it in San Francisco now?"). The utterance (e.g., "What time is it in San Francisco now?") may correspond to a date&time domain associated with a service that provides dates and times. The subsequent utterance (e.g., "What's the weather like here?") may correspond to a weather domain associated with a service that provides weather. Since consecutive utterances correspond to different domains, Figure 5 The context shown in can be a cross-domain scenario. Subsequent utterances (e.g., "What's the weather like here?") can include a pronoun (e.g., here). In a cross-domain scenario, the electronic device 501 can use the information (e.g., cross-domain data) (e.g., San Francisco) included in the utterance (e.g., "What time is it in San Francisco now?") to process a subsequent utterance (e.g., "What's the weather like here?") that includes a pronoun (e.g., here) (or is insufficient to include the information required for utterance processing). The electronic device 501 can provide a response (e.g., "It's sunny in San Francisco") corresponding to the subsequent utterance (e.g., "What's the weather like here?") based on the cross-domain data (e.g., location-San Francisco). The electronic device 501 can support cross-domain scenarios based on the cross-domain data obtained from the user utterance.

[0090] According to an embodiment, some or all of the operations performed by the electronic device 501 may be performed by the electronic device 501 and / or the smart server 601. Hereinafter, a description is provided based on the assumption that the electronic device 501 performs the operations.

[0091] Figure 6is a schematic block diagram of an electronic device according to an embodiment.

[0092] refer to Figure 6 In an embodiment, the electronic device 501 may include a reference Figure 1 The electronic device 101 described and the reference Figure 2 At least some components of the electronic device 201 described above are used to communicate with the intelligent server (e.g., Figure 2 Smart Server 200 or Figure 5 The AI ​​on the device for processing speech in the case of communicating with the intelligent server 502 may be installed in the electronic device 501. Figures 2 to 4 The natural language platform 220 described herein may be implemented in the electronic device 501. Figures 1 to 4 The electronic device 501 may include a processor 520 (eg, Figure 1 The processor 120 and Figure 2 203) and a memory 530 (eg, Figure 1 The memory 130 and Figure 2 The memory 207 of the electronic device 501). The processor 520 (eg, an application processor) may execute instructions by accessing the memory 530. The processor 520 may perform an operation for providing a response to a user. The memory 530 may store various data used by at least one component of the electronic device 501 (eg, the processor 520).

[0093] According to an embodiment, the electronic device 501 can support cross-domain scenarios. Figure 5 As described, a cross-domain scenario may mean that multiple utterances (e.g., consecutive utterances) correspond to different domains, respectively. The electronic device 501 may also provide a suitable response corresponding to a subsequent utterance (e.g., an utterance after a user utterance) in a cross-domain scenario by using cross-domain data obtained based on a user utterance and / or a response provided to the user.

[0094] According to an embodiment, the processor 520 may receive a first utterance. The first utterance may correspond to a first domain 521. The processor 520 may provide a first response corresponding to the first utterance based on the first domain 521.

[0095] According to an embodiment, the processor 520 may obtain cross-domain data based on the first utterance and / or the first response. The processor 520 may obtain cross-domain data from the first utterance and / or the first response based on the first domain 521. The cross-domain data may be used to support a cross-domain scenario, where the cross-domain scenario means that utterances (e.g., continuous utterances) correspond to different domains respectively. The cross-domain data may include data obtained based on the first utterance and / or the first response ( Figure 6The value of (value) and the category of the data obtained (for example, Figure 6 The category of the obtained data may be predefined. The processor 520 may generate activation information for the cross-domain data based on the first domain 521. The processor 520 may send the cross-domain data and the activation information for the cross-domain data to the library domain 523.

[0096] According to an embodiment, the processor 520 may store cross-domain data and activation information of the cross-domain data based on the library domain 523. For example, the activation information may include session information including information about a period during which a speech recognition service (e.g., a speech recognition service activated based on a first utterance) continues to process a subsequent utterance (e.g., a second utterance) after the first utterance. For example, the activation information may include timeout information corresponding to a category of the cross-domain data.

[0097] According to an embodiment, the processor 520 may receive a second utterance after the first utterance. The second utterance may correspond to a second domain 522. The second domain 522 may be different from the first domain 521. For example, the first domain may be a Gracenote domain corresponding to a music search service (e.g., a Gracenote service). For example, the second domain may be a Melon domain corresponding to a music playback service (e.g., a Melon service). The second utterance may be an utterance including a pronoun. The processor 520 may classify the pronoun (e.g., the pronoun included in the second utterance) into a category (e.g., Figure 6 key) is sent to the repository domain 523.

[0098] According to an embodiment, the processor 520 may compare a category corresponding to a pronoun (e.g., a pronoun included in the second utterance) with a category of cross-domain data (e.g., cross-domain data obtained based on the first utterance) based on the library domain 523. Based on the library domain 523, when the pronoun (e.g., a pronoun included in the second utterance) matches a category of the cross-domain data (e.g., cross-domain data obtained based on the first utterance) and the cross-domain data is activated, the processor 520 may read the cross-domain data.

[0099] According to an embodiment, the processor 520 may provide a second response corresponding to a second utterance (e.g., a second utterance including a pronoun) based on cross-domain data (e.g., cross-domain data obtained based on a first utterance). As described above, the first domain 521 and the second domain 522 corresponding to the first utterance and the second utterance, respectively, may be different from each other, and the electronic device 501 may support a cross-domain scenario. An example of a cross-domain scenario is provided in Table 1.

[0100] [Table 1]

[0101]

[0102] Referring to Table 1, according to an embodiment, the electronic device 501 may receive an utterance (e.g., "What time is it in San Francisco now?"). The electronic device 501 may receive a subsequent utterance (e.g., "What time is it in San Francisco now?") after the utterance (e.g., "What time is it in San Francisco now?"). The utterance (e.g., "What time is it in San Francisco now?") may correspond to a date & time domain associated with a service providing date and time. The subsequent utterance (e.g., "What weather is it here?") may correspond to a weather domain associated with a service providing weather. Since the continuous utterances correspond to different domains, respectively, the aforementioned example may be a cross-domain scenario. The subsequent utterance (e.g., "What weather is it here?") may include a pronoun (e.g., here). In a cross-domain scenario, the electronic device 501 may use information (e.g., cross-domain data) (e.g., San Francisco) included in the utterance (e.g., "What time is it in San Francisco now?") to process a subsequent utterance (e.g., "What weather is it here?") that includes a pronoun (e.g., here) (or is insufficient to include information required for utterance processing). The electronic device 501 may provide a response (e.g., "What weather is it here?") corresponding to the subsequent utterance (e.g., "What weather is it here?") based on the cross-domain data (e.g., location-San Francisco). "San Francisco is sunny."). The electronic device 501 may support cross-domain scenarios based on cross-domain data obtained from user utterances.

[0103] According to an embodiment, when subsequent cross-domain data of the same category as the cross-domain data is obtained, the processor 520 may overwrite the cross-domain data with the subsequent cross-domain data. An example of the overwriting operation is provided in Table 2.

[0104] [Table 2]

[0105] Figure 7 is a diagram illustrating an operation of an electronic device processing a user utterance according to an embodiment.

[0106] Reference Figure 7 , the electronic device 501 can support cross-domain scenarios. Figure 5 As described, the cross-domain scenario may mean that multiple utterances (eg, consecutive utterances) correspond to different domains respectively. The electronic device 501 may provide adaptation corresponding to subsequent utterances (eg, utterances after the user utterance) in the cross-domain scenario by using cross-domain data obtained from the user utterance.

[0107] According to an embodiment, the processor 520 may receive a first utterance (eg, “What time is it in San Francisco?”). The first utterance may correspond to the date & time field 721. The processor 520 may provide a first response corresponding to the first utterance based on the date & time field 721.

[0108] According to an embodiment, the processor 520 may obtain cross-domain data (e.g., location-San Francisco) based on the first utterance (e.g., “What time is it in San Francisco now?”). The processor 520 may obtain cross-domain data (e.g., location-San Francisco) from the first utterance (e.g., “What time is it in San Francisco now?”) based on the date & time domain 721. The cross-domain data may be used to support cross-domain scenarios (e.g., the scenario shown in FIG. 7 ), wherein the cross-domain scenario means that utterances (e.g., continuous utterances) correspond to different domains (e.g., the date & time domain 721 and the weather domain 722), respectively. The cross-domain data (e.g., location-San Francisco) may include data obtained based on the first utterance (e.g., Figure 7 = San Francisco) and the category of the data obtained (e.g. Figure 7 =Location). The processor 520 may generate activation information (not shown) of the cross-domain data based on the date & time field 721. The activation data (not shown) may be session information, wherein the session information includes information about a period during which a speech recognition service (e.g., a speech recognition service activated based on a first utterance) continues to process a subsequent utterance (e.g., a second utterance) after the first utterance. The processor 520 may send the cross-domain data (e.g., location-San Francisco) and the activation data (not shown) of the cross-domain data to the library field 523.

[0109] According to an embodiment, the processor 520 may store the cross-domain data (eg, location-San Francisco) and activation information (not shown) of the cross-domain data in the memory 530 based on the library domain 523. Figure 7 Assuming that the device is operated, the cross-domain data and the activation information of the cross-domain data are stored in the electronic device 501, but the example is not limited thereto. The cross-domain data and the activation information of the cross-domain data can be stored in the intelligent server (for example, Figure 2 Smart Server 200 and Figure 6 in the intelligent server 601).

[0110] According to an embodiment, the processor 520 may receive a second utterance (e.g., “What time is it in San Francisco now?”) after a first utterance (e.g., “What’s the weather like here?”). The second utterance (e.g., “What’s the weather like here?”) may correspond to a weather domain 722. The weather domain 722 may be different from the date & time domain 721. For example, the date & time domain 721 may correspond to a date and time provision service. The weather domain 722 may correspond to a weather provision service. Figure 7The context shown in may correspond to a cross-domain scenario. In addition, the second utterance (e.g., “How is the weather here?”) may be an utterance including a pronoun (e.g., here). The processor 520 may assign a category (e.g., Figure 7 The key = location) is sent to the library domain 523.

[0111] According to an embodiment, the processor 520 may compare the category (e.g., place) corresponding to the reference word (e.g., here) with the category (e.g., place) of the cross-domain data (e.g., place-San Francisco) based on the library domain 523. Based on the library domain 523, when the reference word (e.g., here) matches the category (e.g., place) of the cross-domain data and the cross-domain data is activated, the processor 520 may read the cross-domain data (e.g., place-San Francisco). Figure 7 , but the speech recognition service activated based on the first utterance (e.g., "What time is it in San Francisco?") can continue until the second utterance (e.g., "What's the weather like here?") is received. The cross-domain data (e.g., location - San Francisco) obtained based on the first utterance (e.g., "What time is it in San Francisco?") can be activated.

[0112] According to an embodiment, the processor 520 may provide a second response corresponding to a second utterance (e.g., "How is the weather here?") including a reference word (e.g., here) based on the cross-domain data (e.g., location-San Francisco). As described above, the date & time domain 721 and the weather domain 722 corresponding to the first utterance and the second utterance, respectively, may be different from each other, and the electronic device 501 may support cross-domain scenarios.

[0113] Figure 8 is a diagram illustrating an operation of an electronic device processing a user utterance according to an embodiment.

[0114] Reference Figure 8 , the electronic device 501 can support cross-domain scenarios. Figure 5 As described, a cross-domain scenario may mean that multiple utterances (e.g., consecutive utterances) correspond to different domains respectively. The electronic device 501 may also provide a suitable response corresponding to a subsequent utterance (e.g., an utterance after a user utterance) in a cross-domain scenario by using cross-domain data obtained based on a response provided to the user.

[0115] According to an embodiment, the processor 520 may receive a first utterance (e.g., “What is this song?”). The first utterance (e.g., “What is this song?”) may correspond to a Gracenote domain 821. The Gracenote domain 821 may correspond to a music search service. The processor 520 may provide a first response (e.g., “This is a play by IU.”) corresponding to the first utterance (e.g., “What is this song?”) based on the Gracenote domain 821.

[0116] According to an embodiment, the processor 520 may obtain cross-domain data (e.g., music-artist: [IU], song: [drama]) based on the first response (e.g., “This is a drama by IU.”). The processor 520 may obtain cross-domain data (e.g., music-artist: [IU], song [drama]) from the first response (e.g., “This is a drama by IU.”) based on the Gracenote domain 821. The cross-domain data may be used to support cross-domain scenarios (e.g., Figure 8 ), where the cross-domain scenario means that the utterances (e.g., continuous utterances) correspond to different domains (e.g., Gracenote domain 821 and Melon domain 822). Cross-domain data (e.g., music-artist: [IU], song: [drama]) may include data obtained based on the first response (e.g., “This is a drama by IU.”) (e.g., Figure 8 = Artist: [IU], Song: [Drama]) and the category of the data obtained (e.g. Figure 8 =Music). The processor 520 may generate activation information (e.g., 300 ms) of the cross-domain data based on the Gracenote domain 821. The activation information (e.g., 300 ms) may be timeout information corresponding to the category (e.g., music) of the cross-domain data. The timeout information may be preset, but the example is not limited thereto. The processor 520 may send the cross-domain data (e.g., music-artist: [IU], song: [drama]) and the activation information (e.g., 300 ms) of the cross-domain data to the library domain 523.

[0117] According to an embodiment, the processor 520 may store cross-domain data (eg, music-artist: [IU], song: [drama]) and activation information (eg, 300 ms) of the cross-domain data based on the library domain 523. Figure 8 Assuming that the device is operated, the cross-domain data and the activation information of the cross-domain data are stored in the electronic device 501, but the example is not limited thereto. The cross-domain data and the activation information of the cross-domain data can be stored in the intelligent server (for example, Figure 2 Smart Server 200 and Figure 6 in the intelligent server 601).

[0118] According to an embodiment, the processor 520 may receive a second utterance (e.g., “Add this song to my playlist.”) after the first utterance (e.g., “This is IU's play.”). The second utterance (e.g., “Add this song to my playlist.”) may correspond to the Melon domain 822. The Melon domain 822 may be different from the Gracenote domain 821. For example, the Gracenote domain 821 may correspond to a music search service. For example, the Melon domain 822 may correspond to a music playback service. Figure 8 The context shown in may correspond to a cross-domain scenario. In addition, the second utterance (e.g., "Add this song to my playlist.") may be an utterance that is insufficient to include information required for utterance processing (e.g., information about the song to be stored in the playlist). The second utterance (e.g., "Add this song to my playlist.") may be an utterance that does not include slot information (e.g., information corresponding to the song to be stored) for performing an operation corresponding to the second utterance (e.g., storing the song in the playlist). The processor 520 may select a category (e.g., Figure 8 key = music) is sent to library domain 523.

[0119] According to an embodiment, the processor 520 may compare the category (e.g., music) corresponding to the slot information (e.g., information about the song) with the category (e.g., music) of the cross-domain data (e.g., music-artist: [IU], song: [Drama]) based on the library domain 523. Based on the library domain 523, when the slot information (e.g., information about the song) matches the category (e.g., music) of the cross-domain data and the cross-domain data is activated, the processor 520 may read the cross-domain data (e.g., music-artist: [IU], song: [Drama]). Although not in Figure 8 , but after the cross-domain data (e.g., music-artist: [IU], song: [drama]) obtained based on the first response (e.g., "This is IU's drama.") is stored, it may not exceed 300ms. The cross-domain data (e.g., music-artist: [IU], song: [drama]) can be activated.

[0120] According to an embodiment, the processor 520 may provide a second response corresponding to a second utterance (e.g., "Add this song to my playlist.") that is insufficient to include information required for utterance processing (e.g., information about the song) based on the cross-domain data (e.g., music-artist: [IU], song: [drama]). As described above, the Gracenote domain 821 and the Melon domain 822 corresponding to the first utterance and the second utterance, respectively, may be different from each other, and the electronic device 501 may support a cross-domain scenario.

[0121] Fig. 9 is a flowchart illustrating an operating method of an electronic device according to an embodiment.

[0122] Operations 910 to 940 may be performed sequentially, but may not necessarily be performed sequentially. For example, the order of operations 910 to 940 may be changed, and at least two of operations 910 to 940 may be performed in parallel.

[0123] According to an embodiment, it is understood that operations 910 to 940 may be performed by an electronic device (eg, Figure 5 A processor (eg, Figures 6 to 8 processor 520) to execute.

[0124] In operation 910, an electronic device (eg, Figure 5 The electronic device 501) can receive the first utterance and provide a first response corresponding to the first utterance.

[0125] In operation 920, the electronic device 501 may obtain cross-domain data based on the first utterance or the first response. The cross-domain data may be used to support cross-domain scenarios, where the cross-domain scenario means that multiple utterances correspond to different domains, respectively. The cross-domain data may include data obtained based on the first utterance or the first response and the category of the obtained data.

[0126] In operation 930 , the electronic device 501 may receive a second utterance subsequent to the first utterance.

[0127] In operation 940 , when the second utterance includes the pronoun (or the second utterance is insufficient to include information required for utterance processing), the electronic device 501 may provide a second response corresponding to the second utterance based on the cross-domain data.

[0128] According to an electronic device (eg, Figure 1 electronic device 101, Figure 2 The electronic device 201 and Figure 5 The electronic device 501 may include: a memory storing instructions (eg, Figure 1 Memory 130, Figure 2 The memory 201 and Figure 6 The electronic device may include at least one processor (e.g., Figure 1 processor 120, Figure 2 The processor 203 and Figure 6The instructions, when executed by the at least one processor, may cause the at least one processor to: receive a first signal corresponding to a first utterance of a user; provide a first response corresponding to the first utterance; obtain cross-domain data based on the first utterance or the first response; receive a second signal corresponding to a second utterance of the user following the first utterance; and if the second utterance includes a replacement word, provide a second response corresponding to the second utterance based on the cross-domain data. The first utterance and the second utterance may correspond to different domains, respectively.

[0129] According to an embodiment, the cross-domain data may be used to support a cross-domain scenario, in which a plurality of utterances correspond to different domains, respectively.

[0130] According to an embodiment, the cross-domain data may include data obtained based on the first utterance or the first response and a category of the obtained data.

[0131] According to an embodiment, when the instructions are executed by the at least one processor, the at least one processor may cause the at least one processor to: store the cross-domain data and activation information of the cross-domain data in the memory based on a library domain.

[0132] According to an embodiment, the activation information may include session information, wherein the session information includes information about a period of time during which the speech recognition service activated based on the first utterance continues to process an utterance after the first utterance.

[0133] According to an embodiment, the activation information may further include timeout information, wherein the timeout information corresponds to a category of the acquired data included in the cross-domain data.

[0134] According to an embodiment, when the instructions are executed by at least one processor, the at least one processor may: based on a library domain, when the reference term matches a category of the obtained data included in the cross-domain data, read the cross-domain data.

[0135] According to an embodiment, when the instructions are executed by at least one processor, the at least one processor may cause the at least one processor to: read the cross-domain data based on the library domain when the cross-domain data is activated.

[0136] According to an embodiment, when the instruction is executed by the at least one processor, it may cause the at least one processor to: when obtaining subsequent cross-domain data of the same category as the obtained data included in the cross-domain data, overwrite the cross-domain data with the subsequent cross-domain data.

[0137] According to an embodiment, when the instructions are executed by the at least one processor, the at least one processor may cause the at least one processor to: provide the second response based on the cross-domain data when the second utterance is insufficient to include information required for utterance processing.

[0138] According to an embodiment, a method of operating an electronic device (e.g., Figure 1 electronic device 101, Figure 2 The electronic device 201 and Figure 5 The method of the electronic device 501) may include: receiving a first signal corresponding to a first utterance. The method includes guiding a user interface to provide a first response corresponding to the first utterance. The method includes obtaining cross-domain data based on the first utterance or the first response. The method includes receiving a second signal corresponding to a second utterance of a user after the first utterance; and when the second utterance includes a pronoun, guiding the user interface to provide a second response corresponding to the second utterance based on the cross-domain data. The first utterance and the second utterance may correspond to different domains, respectively.

[0139] The cross-domain data may be used to support a cross-domain scenario, wherein in the cross-domain scenario, a plurality of utterances correspond to different domains, respectively.

[0140] According to an embodiment, the cross-domain data may include data obtained based on the first utterance or the first response and a category of the obtained data.

[0141] According to an embodiment, the method may further include: storing the cross-domain data and activation information of the cross-domain data in a memory based on a library domain.

[0142] According to an embodiment, the activation information may include session information, wherein the session information includes information about a period of time during which the speech recognition service activated based on the first utterance continues to process an utterance after the first utterance.

[0143] According to an embodiment, the activation information may further include timeout information, wherein the timeout information corresponds to a category of the obtained data included in the cross-domain data.

[0144] According to an embodiment, directing the user interface to provide the second response may include: based on a repository domain, when the referent matches a category of the obtained data included in the cross-domain data, reading the cross-domain data.

[0145] According to an embodiment, reading the cross-domain data may include: based on the library domain, reading the cross-domain data when the cross-domain data is activated.

[0146] According to an embodiment, the method may further include: when subsequent cross-domain data of the same category as the obtained data included in the category of the cross-domain data is obtained, rewriting the cross-domain data with the subsequent cross-domain data.

[0147] According to an embodiment, guiding the user interface to provide the second response may further include: based on the cross-domain data when the second utterance is insufficient to include information required for utterance processing.

[0148] According to an embodiment, the electronic device may be one of various types of electronic devices. The electronic device may include, for example, a portable communication device (e.g., a smart phone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a household appliance device. According to one embodiment, the electronic device is not limited to those electronic devices described above.

[0149] It should be understood that the various embodiments and the terms used therein are not intended to limit the technical features set forth herein to specific embodiments, but include various changes, equivalents or alternative forms for the corresponding embodiments. In association with the description of the accompanying drawings, reference numerals that are thought of can be used for similar or related components. It will be understood that the nouns in the singular form corresponding to the item may include one or more things unless the relevant context clearly indicates otherwise. As used herein, "A or B", "at least one of A and B", "at least one of A or B", "A, B or C", "at least one of A, B and C" and "A, B or C", each of which may include any one of the items listed together with the corresponding one of the multiple phrases or all possible combinations thereof. Terms such as "1st", "2nd" or "first" and "second" can be simply used to distinguish components from other components discussed, and can refer to components that are not limited in other aspects (e.g., importance or order). It will be understood that if an element (e.g., a first element) is referred to as being “coupled with another element (e.g., the second element)”, “coupled to another element (e.g., the second element)”, “connected with another element (e.g., the second element)”, or “connected to another element (e.g., the second element)” when the term “operably” or “communicatively” is used or when the term “operably” or “communicatively” is not used, it means that the element may be directly connected to the other element (e.g., via a cable), wirelessly connected to the other element, or connected to the other element via a third element.

[0150] As used in connection with various embodiments, the term "module" may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with other terms (e.g., "logic," "logic block," "portion," or "circuit"). A module may be a single integrated component adapted to perform one or more functions or a minimum unit or portion of the single integrated component. For example, according to one embodiment, a module may be implemented in the form of an ASIC.

[0151] The various embodiments described herein may be implemented as software (e.g., program 140) including one or more instructions stored in a storage medium (e.g., internal memory 136 or external memory 138) that can be read by a machine (e.g., electronic device 101). For example, under the control of a processor, a processor (e.g., processor 120) of the machine (e.g., electronic device 101) may call at least one of the one or more instructions stored in the storage medium and execute it. This enables the machine to operate to perform at least one function according to the at least one instruction called. The one or more instructions may include code generated by a compiler or code that can be run by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. The term "non-transitory" only means that the storage medium is a tangible device and does not include signals (e.g., electromagnetic waves), but the term does not distinguish between data being semi-permanently stored in the storage medium and data being temporarily stored in the storage medium.

[0152] The method may be included and provided in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., a compact disk read-only memory (CD-ROM)), or may be available via an application store (e.g., PlayStore). TM ) the computer program product may be published (e.g., downloaded or uploaded) online, or the computer program product may be distributed (e.g., downloaded or uploaded) directly between two user devices (e.g., smart phones). If published online, at least part of the computer program product may be temporarily generated, or at least part of the computer program product may be at least temporarily stored in a machine-readable storage medium (such as a memory of a manufacturer's server, a server of an application store, or a forwarding server).

[0153] Each component (e.g., module or program) in the above-mentioned components may include a single entity or multiple entities, and some of the multiple entities may be separately arranged in different components. One or more of the above-mentioned components may be omitted, or one or more other components may be added. Alternatively or additionally, multiple components (e.g., modules or programs) may be integrated into a single component. In this case, according to one embodiment, the integrated component may still perform the one or more functions of each of the multiple components in the same or similar manner as a corresponding one of the multiple components performing one or more functions before integration. The operations performed by a module, program or another component may be performed sequentially, in parallel, repeatedly or in a heuristic manner, or one or more of the operations may be run or omitted in a different order, or one or more other operations may be added.

Claims

1. An electronic device (101; 201; 501), including: a memory (130; 207; 530) storing instructions; and at least one processor (120; 203; 520), Wherein, when the instruction is executed by the at least one processor (120; 203; 520), the instruction may enable the at least one processor (120; 203; 520): receiving a first signal corresponding to a first utterance of a user, providing a first response corresponding to the first utterance, obtaining cross-domain data based on the first utterance or the first response, receiving a second signal corresponding to a second utterance of the user subsequent to the first utterance, If the second utterance includes a pronoun, providing a second response corresponding to the second utterance based on the cross-domain data; and The first utterance and the second utterance may correspond to different domains respectively.

2. The electronic device (101; 201; 501) according to claim 1, wherein: The cross-domain data is used to support a cross-domain scenario, wherein in the cross-domain scenario, multiple utterances correspond to different domains respectively.

3. The electronic device (101; 201; 501) according to any one of claims 1 and 2, wherein: The cross-domain data includes data obtained based on the first utterance or the first response and a category of the obtained data.

4. The electronic device (101; 201; 501) according to any one of claims 1 to 3, in, The instructions, when executed by the at least one processor (120; 203; 520), may cause the at least one processor (120; 203; 520): The cross-domain data and activation information of the cross-domain data are stored in the memory (130; 207; 530) based on the library domain (523).

5. The electronic device (101; 201; 501) according to any one of claims 1 to 4, wherein: The activation information includes session information, wherein the session information includes information about a period of time during which the speech recognition service activated based on the first utterance continues to process an utterance after the first utterance.

6. The electronic device (101; 201; 501) according to any one of claims 1 to 5, wherein: The activation information further includes timeout information, wherein the timeout information corresponds to a category of the obtained data included in the cross-domain data.

7. The electronic device (101; 201; 501) according to any one of claims 1 to 6, wherein: The instructions, when executed by the at least one processor (120; 203; 520), may cause the at least one processor (120; 203; 520): Based on the library domain (523), when the reference word matches the category of the obtained data included in the cross-domain data, the cross-domain data is read.

8. The electronic device (101; 201; 501) according to any one of claims 1 to 7, wherein: The instructions, when executed by the at least one processor (120; 203; 520), may cause the at least one processor (120; 203; 520): Based on the library domain (523), when the cross-domain data is activated, the cross-domain data is read.

9. The electronic device (101; 201; 501) according to any one of claims 1 to 8, wherein: The instructions, when executed by the at least one processor (120; 203; 520), may cause the at least one processor (120; 203; 520): When subsequent cross-domain data of the same category as that of the obtained data included in the cross-domain data is obtained, the cross-domain data is overwritten with the subsequent cross-domain data.

10. The electronic device (101; 201; 501) according to any one of claims 1 to 9, wherein: The instructions, when executed by the at least one processor (120; 203; 520), may cause the at least one processor (120; 203; 520): Providing the second response also includes being based on the cross-domain data when the second utterance is insufficient to include information required for utterance processing.

11. An operating electronic device (101; 201; 501), the method comprising: receiving a first signal corresponding to a first utterance; directing the user interface to provide a first response corresponding to the first utterance; obtaining cross-domain data based on the first utterance or the first response; receiving a second signal corresponding to a second utterance of a user subsequent to the first utterance; and When the second utterance includes a pronoun, directing the user interface to provide a second response corresponding to the second utterance based on the cross-domain data, and The first utterance and the second utterance correspond to different domains respectively.

12. The method according to claim 11, wherein: The cross-domain data is used to support a cross-domain scenario, wherein in the cross-domain scenario, multiple utterances correspond to different domains respectively.

13. The method according to any one of claims 11 and 12, wherein: The cross-domain data includes data obtained based on the first utterance or the first response and a category of the obtained data.

14. The method according to any one of claims 11 to 13, further comprising: Based on the library domain (523), the cross-domain data and the activation information of the cross-domain data are stored in the memory (130; 207; 530).

15. The method according to any one of claims 11 to 14, wherein: The activation information includes session information, wherein the session information includes information about a period of time during which the speech recognition service activated based on the first utterance continues to process an utterance after the first utterance.

Images