False rejection determination method and electronic device for performing same

Abstract

Disclosed are a false rejection determination method and an electronic device for performing the same. An electronic device (101, 201, 520, 540, 701, 702, 705) according to an embodiment may comprise: a memory (130, 207) containing instructions; and a processor (120, 203) operably connected with the memory (130, 207) and configured to execute the instructions. The processor (120, 203) may: generate, on the basis of a user input, a detected event corresponding to a related event, where the related event is related to a speech for which speaker recognition fails; in response to the detection event, outputting information about the related event; and obtaining feedback of the information about the related event.

Description

Technical Field

[0001] The present disclosure relates to a false rejection determination method and an electronic device for executing the method. Background Art

[0002] Users can use electronic devices to obtain various services. With the development of voice recognition technology, users can input voice (e.g., a speech) into electronic devices and receive a reply message based on the input voice through a voice assistant (e.g., a voice assistant service).

[0003] In addition, based on the recognition of the input voice (e.g., voice call command) by the voice assistant, the electronic device can perform certain functions or support another electronic device (e.g., Internet of Things (IoT) device) to perform functions. Device control based on voice call command can include understanding the user's intention from the voice call command and executing the device command desired by the user. Summary of the invention

[0004] Technical Solution

[0005] According to one aspect of the present disclosure, an electronic device (101; 201; 520; 540; 701; 702; 705) includes: a memory (130; 207) in which instructions are stored; and a processor (120; 203) operably connected to the memory (130; 207) and configured to execute the instructions to: receive user input; generate a detection event corresponding to a related event based on the user input, wherein the related event is related to an utterance for which speaker recognition fails; output information about the related event generated in response to the detection event; and obtain feedback on the information about the related event, wherein the feedback is used to learn a speaker model for the utterance for which speaker recognition fails.

[0006] According to one aspect of the present disclosure, a method for operating an electronic device (101; 201; 520; 540; 701; 702; 705) includes: receiving user input; generating a detection event corresponding to a relevant event based on the user input, wherein the relevant event is related to an utterance for which speaker recognition fails; outputting information about the relevant event generated in response to the detection event; and obtaining feedback on the information about the relevant event, wherein the feedback is used to learn a speaker model for the utterance for which speaker recognition fails.

[0007] According to one aspect of the present disclosure, an electronic device (101; 201; 520; 540; 701; 702; 705) includes: a memory (130; 207) storing instructions; and a processor (120; 203) operably connected to the memory (130; 207) and configured to execute the instructions to: classify, among received utterances, utterances in which speaker recognition fails; receive user input within a specified time from the time of occurrence of the utterance in which speaker recognition fails; generate a detection event corresponding to a related event based on the user input, wherein the related event is related to the utterance in which speaker recognition fails; output information about the related event generated in response to the detection event; and obtain feedback on the information about the related event, wherein the feedback is used to learn a speaker model for the utterance in which speaker recognition fails. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 A block diagram of an electronic device in a network environment according to an embodiment is shown.

[0009] Figure 2 A block diagram of an integrated intelligent system according to an embodiment is shown.

[0010] Figure 3 A schematic diagram showing a form in which relationship information between concepts and actions is stored in a database according to an embodiment.

[0011] Figure 4 A schematic diagram of a screen showing an electronic device processing a received voice input through a smart application according to an embodiment is shown.

[0012] Figure 5 A schematic diagram of an intelligent voice system according to an embodiment is shown.

[0013] Figure 6 A flow chart showing an operation of an electronic device obtaining feedback to determine an utterance that is mistakenly rejected according to an embodiment.

[0014] Fig. 7A A schematic diagram showing an operation of another device determining whether an utterance for which speaker recognition fails on a device is erroneously rejected according to an embodiment.

[0015] Figure 7B A schematic diagram showing an operation of a device according to an embodiment to determine whether an utterance for which speaker recognition fails on the device is mistakenly rejected.

[0016] Figure 8 A schematic diagram showing an operation of an electronic device classifying an utterance for which speaker recognition fails according to an embodiment.

[0017] Fig. 9 A schematic diagram showing an operation example of an intelligent server updating a speaker model of an utterance for which speaker recognition fails based on an utterance that was mistakenly rejected according to an embodiment.

[0018] Fig.10 A schematic diagram showing another operation example of an intelligent server updating a speaker model of an utterance for which speaker recognition fails based on an utterance that was mistakenly rejected according to an embodiment.

[0019] Fig.11 A flowchart of the operation of an intelligent server updating a speaker model of a falsely rejected utterance based on the falsely rejected utterance according to an embodiment is shown.

[0020] Fig. 12A An example of a user interface for obtaining feedback according to an embodiment is shown.

[0021] Fig. 12B An example of a user interface for obtaining feedback according to an embodiment is shown.

[0022] Fig. 12C An example of a user interface for obtaining feedback according to an embodiment is shown. DETAILED DESCRIPTION

[0023] Hereinafter, some example embodiments will be described in detail with reference to the accompanying drawings. When describing these embodiments with reference to the accompanying drawings, the same reference numerals represent the same elements, and repeated description of the related elements will be omitted.

[0024] Figure 1 A block diagram of an electronic device 101 in a network environment 100 according to an embodiment is shown.

[0025] Reference Figure 1, the electronic device 101 in the network environment 100 may communicate with the electronic device 102 via the first network 198 (e.g., a short-range wireless communication network), or communicate with at least one of the electronic device 104 or the server 108 via the second network 199 (e.g., a long-range wireless communication network). According to an embodiment, the electronic device 101 may communicate with the electronic device 104 via the server 108. According to an embodiment, the electronic device 101 may include a processor 120, a memory 130, an input module 150, a sound output module 155, a display module 160, an audio module 170, a sensor module 176, an interface 177, a connection terminal 178, a haptic module 179, a camera module 180, a power management module 188, a battery 189, a communication module 190, a user identification module (SIM) 196, or an antenna module 197. In some embodiments, at least one component (e.g., the connection terminal 178) may be omitted in the electronic device 101, or one or more other components may be added in the electronic device 101. In some embodiments, some components (eg, sensor module 176 , camera module 180 , or antenna module 197 ) may be integrated into a single component (eg, display module 160 ).

[0026] The processor 120 may execute, for example, software (e.g., program 140) to control at least one other component (e.g., hardware or software component) of the electronic device 101 connected to the processor 120, and may perform various data processing or calculations. According to an embodiment, as at least part of the data processing or calculation, the processor 120 may store a command or data received from another component (e.g., sensor module 176 or communication module 190) in the volatile memory 132, process the command or data stored in the volatile memory 132, and store the resultant data in the non-volatile memory 134. According to an embodiment, the processor 120 may include a main processor 121 (e.g., a central processing unit (CPU) or an application processor (AP)), or include an auxiliary processor 123 (e.g., a graphics processing unit (GPU), a neural processing unit (NPU), an image signal processor (ISP), a sensor hub processor, or a communication processor (CP)), which may operate independently of the main processor 121 or in conjunction with the main processor 121. For example, when the electronic device 101 includes a main processor 121 and an auxiliary processor 123, the auxiliary processor 123 may be adapted to consume less power than the main processor 121 or be dedicated to a specific function. The auxiliary processor 123 may be implemented separately from the main processor 121 or as part of the main processor 121.

[0027] When the main processor 121 is in an inactive (e.g., sleep) state, the auxiliary processor 123 may control at least some functions or states related to at least one component of the electronic device 101 (e.g., display module 160, sensor module 176, or communication module 190) instead of the main processor 121, or when the main processor 121 is in an active state (e.g., executing an application), the auxiliary processor 123 may control these functions or states together with the main processor 121. According to an embodiment, the auxiliary processor 123 (e.g., ISP or CP) may be implemented as a part of another component (e.g., camera module 180 or communication module 190) that is functionally related to the auxiliary processor 123. According to an embodiment, the auxiliary processor 123 (e.g., NPU) may include a designated hardware structure for artificial intelligence (AI) model processing. The artificial intelligence model may be generated by machine learning. For example, such learning may be performed by the electronic device 101 that performs artificial intelligence, or via a separate server (e.g., server 108). The learning algorithm may include, but is not limited to, for example, a supervised learning algorithm, an unsupervised learning algorithm, a semi-supervised learning algorithm, or a reinforcement learning algorithm. The AI ​​model may include multiple artificial neural network layers. The artificial neural network may include, for example, a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), and a bidirectional recurrent deep neural network (BRDNN), a deep Q network, or a combination of two or more thereof, but is not limited thereto. The AI ​​model may additionally or alternatively include a software structure in addition to a hardware structure.

[0028] The memory 130 may store various data used by at least one component of the electronic device 101 (e.g., the processor 120 or the sensor module 176). These data may include, for example, input data or output data of software (e.g., program 140) and commands related thereto. The memory 130 may include a volatile memory 132 or a non-volatile memory 134.

[0029] The program 140 may be stored as software in the memory 130 , and may include, for example, an operating system (OS) 142 , middleware 144 , or applications 146 .

[0030] The input module 150 may receive a command or data to be used by another component (e.g., the processor 120) of the electronic device 101 from outside (e.g., a user) of the electronic device 101. The input module 150 may include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0031] The sound output module 155 can output sound signals to the outside of the electronic device 101. The sound output module 155 may include, for example, a speaker or a receiver. The speaker can be used for general purposes, such as playing multimedia content or playing recordings. The receiver can be used to receive calls. Depending on the embodiment, the receiver can be implemented separately from the speaker, or as part of the speaker.

[0032] The display module 160 can provide information to the outside of the electronic device 101 (e.g., a user) in a visual manner. The display module 160 may include, for example, a display, a holographic device, or a projector, and a control circuit for controlling a corresponding one of the display, the holographic device, and the projector. According to an embodiment, the display module 160 may include a touch sensor adapted to sense a touch, or a pressure sensor adapted to measure the strength of a force generated by a touch.

[0033] The audio module 170 may convert sound into an electrical signal and vice versa. According to an embodiment, the audio module 170 may obtain sound via the input module 150 or output sound via the sound output module 155 or an external electronic device (e.g., an electronic device 102 such as a speaker or earphone) directly or wirelessly connected to the electronic device 101.

[0034] The sensor module 176 may detect an operating state (e.g., power or temperature) of the electronic device 101 or an environmental state (e.g., a user's state) outside the electronic device 101, and generate an electrical signal or data value corresponding to the detected state. According to an embodiment, the sensor module 176 may include, for example, a gesture sensor, a gyro sensor, an atmospheric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an infrared (IR) sensor, a biosensor, a temperature sensor, a humidity sensor, or an illumination sensor.

[0035] The interface 177 may support one or more designated protocols for the electronic device 101 to be directly (e.g., via a wired connection) or wirelessly coupled to an external electronic device (e.g., the electronic device 102). According to an embodiment, the interface 177 may include, for example, a high-definition multimedia interface (HDMI), a universal serial bus (USB) interface, a secure digital (SD) card interface, or an audio signal interface.

[0036] The connection terminal 178 may include a connector through which the electronic device 101 can be physically connected to an external electronic device (e.g., the electronic device 102). According to an embodiment, the connection terminal 178 may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio signal connector (e.g., a headphone connector).

[0037] The haptic module 179 may convert the electrical signal into mechanical stimulation (eg, vibration or motion) or electrical stimulation, which the user may sense through their sense of touch or kinesthetic sense. According to an embodiment, the haptic module 179 may include, for example, a motor, a piezoelectric element, or an electrical stimulator.

[0038] The camera module 180 may capture still images and moving images. According to an embodiment, the camera module 180 may include one or more lenses, image sensors, ISPs, or flashes.

[0039] The power management module 188 may manage power supplied to the electronic device 101. According to an embodiment, the power management module 188 may be implemented as, for example, at least a part of a power management integrated circuit (PMIC).

[0040] The battery 189 may supply power to at least one component of the electronic device 101. According to an embodiment, the battery 189 may include, for example, a non-rechargeable primary battery, a rechargeable energy storage battery, or a fuel cell.

[0041] The communication module 190 may support establishing a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device 101 and an external electronic device (e.g., electronic device 102, electronic device 104, or server 108), and communicate through the established communication channel. The communication module 190 may include one or more CPs that operate independently of the processor 120 (e.g., AP) and support direct (e.g., wired) communication or wireless communication. According to an embodiment, the communication module 190 may include a wireless communication module 192 (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module 194 (e.g., a local area network (LAN) communication module or a power line communication module). A corresponding one of these communication modules can communicate with the external electronic device 104 via a first network 198 (e.g., a short-range communication network such as Bluetooth, WiFi (Wireless Fidelity) Direct, or IrDA (Infrared Data Association)) or a second network 199 (e.g., a long-range communication network such as a traditional cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules may be implemented as a single component (e.g., a single chip), or may be implemented as multiple components (e.g., multiple chips) separated from each other. The wireless communication module 192 may identify and authenticate the electronic device 101 in a communication network such as the first network 198 or the second network 199 using user information (e.g., an International Mobile Subscriber Identity (IMSI)) stored in the SIM 196.

[0042] The wireless communication module 192 can support 5G networks after 4G networks and next-generation communication technologies, such as new air interface (NR) access technology. NR access technology can support enhanced mobile broadband (eMBB), massive machine type communication (mMTC), or ultra-reliable low latency communication (URLLC). The wireless communication module 192 can support high-frequency bands (e.g., millimeter wave bands) to achieve, for example, high data transmission rates. The wireless communication module 192 can support various technologies for ensuring performance on high-frequency bands, such as beamforming, massive multiple-input multiple-output (MIMO), full-dimensional MIMO (FD-MIMO), array antennas, analog beamforming, or massive antennas. The wireless communication module 192 can support various requirements specified in the electronic device 101, an external electronic device (e.g., electronic device 104), or a network system (e.g., a second network 199). According to an embodiment, the wireless communication module 192 can support a peak data rate for implementing eMBB (e.g., 20 Gbps or higher), support a coverage loss for implementing mMTC (e.g., 164 dB or lower), or support a user plane delay for implementing URLLC (e.g., 0.5 ms or lower for downlink (DL) and uplink (UL) respectively, or a round-trip time of 1 ms or lower).

[0043] The antenna module 197 may transmit a signal or power to the outside of the electronic device 101 (e.g., an external electronic device), or receive a signal or power from the outside of the electronic device 101. According to an embodiment, the antenna module 197 may include an antenna including a radiation element including a conductive material or a conductive pattern formed in or on a substrate (e.g., a PCB). According to an embodiment, the antenna module 197 may include a plurality of antennas (e.g., an array antenna). In this case, for example, the communication module 190 may select at least one antenna suitable for a communication scheme used in a communication network such as the first network 198 or the second network 199 from the plurality of antennas. A signal or power may be transmitted or received between the communication module 190 and the external electronic device through the selected at least one antenna. According to an embodiment, another component other than the radiation element (e.g., an RFIC (Radio Frequency Integrated Circuit)) may be additionally formed as a part of the antenna module 197.

[0044] According to an embodiment, the antenna module 197 may form a millimeter wave antenna module. According to an embodiment, the millimeter wave antenna module may include a PCB, an RFIC disposed on a first surface (e.g., bottom surface) of the PCB or adjacent to the first surface of the PCB and capable of supporting a specified high frequency band (e.g., millimeter wave band), and a plurality of antennas (e.g., array antennas) disposed on a second surface (e.g., top surface or side surface) of the PCB or adjacent to the second surface of the PCB and capable of sending or receiving signals in the specified high frequency band.

[0045] At least some of the above components may be coupled to each other, and signals (e.g., commands or data) may be transmitted therebetween via an inter-peripheral communication scheme (e.g., a bus, GPIO (General Purpose Input Output), SPI (Serial Peripheral Interface), or MIPI (Mobile Industry Processor Interface)).

[0046] According to an embodiment, a command or data may be sent or received between the electronic device 101 and the external electronic device 104 via a server 108 coupled to the second network 199. Each of the external electronic devices 102 and 104 may be a device of the same type or a different type as the electronic device 101. According to an embodiment, all or some operations to be performed by the electronic device 101 may be performed on one or more external electronic devices (e.g., external electronic devices 102 and 104 and server 108). For example, if the electronic device 101 needs to automatically perform a function or service, or respond to a request from a user or another device, the electronic device 101 may request one or more external electronic devices to perform at least a part of the function or service, instead of performing the function or service by itself, or perform such a request in addition to performing the function or service by itself. The one or more external electronic devices that receive the request may perform at least a part of the requested function or service, or perform additional functions or additional services related to the request, and may transmit the execution result to the electronic device 101. The electronic device 101 may provide the result (whether or not the result is further processed) as at least a part of the response to the request. To this end, for example, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used. The electronic device 101 may use, for example, distributed computing or MEC to provide ultra-low latency services. In an embodiment, the external electronic device 104 may include an Internet of Things (IoT) device. The server 108 may be an intelligent server using machine learning and / or a neural network. According to an embodiment, the external electronic device 104 or the server 108 may be included in the second network 199. The electronic device 101 may be applied to intelligent services (e.g., smart homes, smart cities, smart cars, or healthcare) based on 5G communication technology or IoT-related technologies.

[0047] Figure 2 A block diagram of an integrated intelligent system according to an embodiment is shown.

[0048] Reference Figure 2 According to an embodiment, the integrated intelligent system 20 may include an electronic device 201 (eg, Figure 1 electronic device 101 in the example), intelligent server 290 (e.g., Figure 1 108 in the server) and the service server 300 (for example, Figure 1 The electronic device 201 may be a user terminal.

[0049] The electronic device 201 in one embodiment may be a terminal device (or electronic device) that can be connected to the Internet, and may be, for example, a mobile phone, a smart phone, a personal digital assistant (PDA), a laptop computer, a TV, a white appliance, a wearable device, a head-mounted display (HMD), or a smart speaker.

[0050] According to the illustrated embodiment, the electronic device 201 may include a communication interface 202 (eg, Figure 1 interface 177 in), microphone 206 (e.g., Figure 1 Input module 150 in), speaker 205 (e.g., Figure 1 The sound output module 155), the display module 204 (for example, Figure 1 , the display module 160 in the memory 207 (for example, Figure 1 130) or processor 203 (e.g., Figure 1 The above components may be operatively or electrically connected to each other.

[0051] The communication interface 202 in one embodiment can be connected to an external device and configured to send and receive data with the external device. The microphone 206 in one embodiment can receive sound (e.g., user's speech) and convert the sound into an electrical signal. The speaker 205 in one embodiment can output the electrical signal as sound (e.g., voice).

[0052] The display module 204 in one embodiment can be configured to display images or videos. The display module 204 in one embodiment can also display a graphical user interface (GUI) of an application (or application program) being executed. The display module 204 in one embodiment can receive touch input through a touch sensor. For example, the display module 204 can receive text input through a touch sensor in an on-screen keyboard area displayed in the display module 204.

[0053] The memory 207 in one embodiment may store a client module 209, a software development kit (SDK) 208, and a plurality of applications 210. The client module 209 and the SDK 208 may be configured as a framework (or solution program) for performing common functions. In addition, the client module 209 or the SDK 208 may be configured as a framework for processing user input (e.g., voice input, text input, or touch input).

[0054] In one embodiment, the plurality of applications 210 stored in the memory 207 may be programs for performing a specified function. According to an embodiment, the plurality of applications 210 may include a first application 210_1, a second application 210_2, etc. According to an embodiment, each of the plurality of applications 210 may include a plurality of actions for performing a specified function. For example, the applications may include an alarm clock application, a message application, and / or a schedule application. According to an embodiment, the plurality of applications 210 may be executed by the processor 203 to sequentially perform at least a portion of the plurality of actions.

[0055] The processor 203 in one embodiment may control the overall operation of the electronic device 201. For example, the processor 203 may be electrically or operatively connected to the communication interface 202, the microphone 206, the speaker 205, and the display module 204 to perform designated operations.

[0056] The processor 203 in one embodiment may also perform a specified function by executing a program stored in the memory 207. For example, the processor 203 may execute at least one of the client module 209 or the SDK 208 to perform the following operation for processing user input. The processor 203 may control the operation of the plurality of applications 210, for example, through the SDK 208. The following operations as the client module 209 or the SDK 208 may be performed by the processor 203.

[0057] The client module 209 in one embodiment can receive user input. For example, the client module 209 can receive a voice signal corresponding to a user's speech sensed by the microphone 206. As another example, the client module 209 can receive a touch input sensed by the display module 204. As another example, the client module 209 can receive a text input sensed by a keyboard or an on-screen keyboard. In addition, the client module 209 can receive various types of user input sensed by an input module included in the electronic device 201 or an input module connected to the electronic device 201. The client module 209 can send the received user input to the intelligent server 290. The client module 209 can send the state information of the electronic device 201 to the intelligent server 290 together with the received user input. The state information can be, for example, the execution state information of the application.

[0058] The client module 209 in one embodiment can receive a result corresponding to the received user input. For example, when the intelligent server 290 is capable of calculating a result corresponding to the received user input, the client module 209 can receive a result corresponding to the received user input. The client module 209 can display the received result on the display module 204. In addition, the client module 209 can output the received result in the form of an audio signal through the speaker 205.

[0059] The client module 209 in one embodiment may receive a plan corresponding to the received user input. The client module 209 may display the results of executing multiple actions of the application according to the plan on the display module 204. For example, the client module 209 may sequentially display the results of executing multiple actions on the display module 204, and output these results in the form of audio signals through the speaker 205. As another example, the electronic device 201 may display only a part of the results of executing multiple actions (e.g., the result of the last action) on the display module 204, and output this part of the results in the form of audio signals through the speaker 205.

[0060] According to an embodiment, the client module 209 may receive a request for obtaining information required to calculate a result corresponding to the user input from the smart server 290. According to an embodiment, the client module 209 may send necessary information to the smart server 290 in response to the request.

[0061] The client module 209 in one embodiment can send information about the results of performing multiple actions according to the plan to the intelligent server 290. The intelligent server 290 can use the information about these results to confirm that the received user input has been processed correctly.

[0062] The client module 209 in one embodiment may include a voice recognition module. According to an embodiment, the client module 209 may recognize voice input for performing limited functions through the voice recognition module. For example, the client module 209 may execute an intelligent application for processing voice input by specifying input (e.g., "wake up!") to perform organic operations.

[0063] The intelligent server 290 in one embodiment may receive information related to the user voice input from the electronic device 201 through the communication network. According to an embodiment, the intelligent server 290 may convert the data related to the received voice input into text data. According to an embodiment, the intelligent server 290 may generate a plan for executing a task corresponding to the user voice input based on the text data.

[0064] According to an embodiment, the plan may be generated by an (AI) system. The AI ​​system may be a rule-based system or a neural network-based system (e.g., a feedforward neural network (FNN) or a recurrent neural network (RNN)). Alternatively, the AI ​​system may be a combination thereof or other artificial intelligence systems. According to an embodiment, the plan may be selected from a set of predefined plans, or may be generated in real time in response to a user request. For example, the AI ​​system may select at least one plan from the predefined plans.

[0065] The intelligent server 290 in one embodiment may send the result according to the generated plan to the electronic device 201, or send the generated plan to the electronic device 201. According to an embodiment, the electronic device 201 may display the result according to the plan on the display module 204. According to an embodiment, the electronic device 201 may display the result of executing the action according to the plan on the display module 204.

[0066] The intelligent server 290 in one embodiment may include a front end 215 , a natural language platform 220 , a capsule database (DB) 230 , an execution engine 240 , a terminal user interface 250 , a management platform 260 , a big data platform 270 , or an analysis platform 280 .

[0067] The front end 215 in one embodiment may receive the received user input from the electronic device 201. The front end 215 may send a response corresponding to the user input.

[0068] According to an embodiment, the natural language platform 220 may include an automatic speech recognition (ASR) module 221 , a natural language understanding (NLU) module 223 , a planner module 225 , a natural language generator (NLG) module 227 , or a text-to-speech (TTS) module 229 .

[0069] The ASR module 221 in one embodiment may convert the voice input received from the electronic device 201 into text data. The NLU module 223 may use the text data of the voice input to identify the user's intention. For example, the NLU module 223 may identify the user's intention by performing syntactic analysis or semantic analysis on the user input in the form of text data. The NLU module 223 may use the language features (e.g., grammatical elements) of morphemes or phrases to identify the meaning of words extracted from the user input, and determine the user's intention by matching the meaning of the identified words with the intention. The NLU module 223 may obtain intention information corresponding to the user's utterance. The intention information may be information indicating the user's intention determined by analyzing the text data. The intention information may include information indicating the action or function that the user intends to perform using the device.

[0070] The planner module 225 in one embodiment may generate a plan using parameters and the intent determined by the NLU module 223. According to an embodiment, the planner module 225 may determine multiple domains required to perform a task based on the determined intent. The planner module 225 may determine multiple actions included in each of the multiple domains determined based on the intent. According to an embodiment, the planner module 225 may determine the parameters required to perform the multiple actions determined, or determine the result values ​​output by performing the multiple actions. The parameters and the result values ​​may be defined as concepts of a specified form (or category). Therefore, the plan may include multiple actions and multiple concepts determined by the user's intent. The planner module 225 may determine the relationship between the multiple actions and the multiple concepts step by step (or hierarchically). For example, the planner module 225 may determine the execution order of the multiple actions determined based on the user's intent based on the multiple concepts. In other words, the planner module 225 may determine the execution order of the multiple actions based on the parameters required to perform the multiple actions, and determine the results output by performing the multiple actions. Therefore, the planner module 225 may generate a plan including connection information (e.g., ontology) between the multiple actions and the multiple concepts. The planner module 225 may generate a plan using information stored in the capsule DB 230 , which stores a set of relationships between concepts and actions.

[0071] The NLG module 227 in one embodiment may convert the specified information into a text form. The information converted into a text form may be in the form of a natural language speech. The TTS module 229 may convert the information in the text form into the information in the voice form.

[0072] According to an embodiment, part or all of the functions of the natural language platform 220 may also be implemented in the electronic device 201 .

[0073] The capsule DB 230 may store information about the relationship between a plurality of concepts and actions corresponding to a plurality of fields. A capsule according to an embodiment may include a plurality of action objects (or action information) and a concept object (or concept information) included in a plan. According to an embodiment, the capsule DB 230 may store a plurality of capsules in the form of a concept-action network (CAN). According to an embodiment, a plurality of capsules may be stored in a function registration table included in the capsule DB 230.

[0074] The capsule DB 230 may include a policy registry storing policy information required for determining a plan corresponding to the voice input. The policy information may include reference information for determining one plan when there are multiple plans corresponding to the user input. According to an embodiment, the capsule DB 230 may include a subsequent registry storing information about a subsequent action suggesting a subsequent action to the user in a specified situation. The subsequent action may include, for example, a subsequent utterance. According to an embodiment, the capsule DB 230 may include a layout registry storing layout information as information output by the electronic device 201. According to an embodiment, the capsule DB 230 may include a vocabulary registry storing vocabulary information included in the capsule information. According to an embodiment, the capsule DB 230 may include a dialogue registry storing information about a dialogue (or interaction) with the user. The capsule DB 230 may update the stored objects through a developer tool. The developer tool may include, for example, a function editor for updating an action object or a concept object. The developer tool may include a vocabulary editor for updating vocabulary. The developer tool may include a policy editor for generating and registering a policy for determining a plan. The developer tool may include a dialogue editor for generating a dialogue with the user. The developer tool may include a subsequent editor for activating a subsequent target and editing a subsequent utterance providing a prompt. The subsequent target may be determined based on the currently set target, the user's preferences, or environmental conditions. In an embodiment, the capsule DB 230 may also be implemented in the electronic device 201.

[0075] The execution engine 240 in one embodiment can use the generated plan to calculate the result. The terminal user interface 250 can send the calculated result to the electronic device 201. Therefore, the electronic device 201 can receive the result and provide the received result to the user. The management platform 260 can manage the information used by the intelligent server 290. The big data platform 270 can collect the user's data. The analysis platform 280 can manage the quality of service (QoS) of the intelligent server 290. For example, the analysis platform 280 can manage the components and processing rate (or efficiency) of the intelligent server 290.

[0076] The service server 300 in one embodiment may provide a specified service (e.g., ordering a meal or booking a hotel) to the electronic device 201. According to an embodiment, the service server 300 may be a server operated by a third party. The service server 300 may provide information for generating a plan corresponding to the received user input to the smart server 290. The provided information may be stored in the capsule DB 230. In addition, the service server 300 may provide the smart server 290 with result information according to the plan.

[0077] In the integrated intelligent system 20, the electronic device 201 can provide various intelligent services to the user in response to user input. The user input can include, for example, input through a physical button, touch input, or voice input.

[0078] In an embodiment, the electronic device 201 can provide a voice recognition service through an intelligent application (or voice recognition application) stored therein. In this case, for example, the electronic device 201 can recognize a user's speech or voice input received through a microphone, and provide a service corresponding to the recognized voice input to the user.

[0079] In an embodiment, the electronic device 201 can perform a specified action based on the received voice input alone or together with the intelligent server and / or the service server. For example, the electronic device 201 can execute an application corresponding to the received voice input and perform the specified action through the executed application.

[0080] In an embodiment, when the electronic device 201 provides a service together with the intelligent server 290 and / or the service server 300, the electronic device 201 can detect user speech using the microphone 206 and generate a signal (or voice data) corresponding to the detected user speech. The electronic device 201 can send the voice data to the intelligent server 290 using the communication interface 202.

[0081] The intelligent server 290 in one embodiment can generate a plan for performing a task corresponding to the voice input received from the electronic device 201, or generate the result of performing an action according to the plan as a response to the voice input. The plan can include, for example, multiple actions for performing tasks corresponding to the user's voice input, and multiple concepts related to the multiple actions. These concepts can define parameters input to perform multiple actions or result values ​​output by performing multiple actions. The plan can include connection information between multiple actions and multiple concepts.

[0082] The electronic device 201 may receive the response through the communication interface 202. The electronic device 201 may output a voice signal generated inside the electronic device 201 to the outside using the speaker 205, or may output an image generated inside the electronic device 201 to the outside using the display module 204.

[0083] Figure 3 A schematic diagram showing a form in which relationship information between concepts and actions is stored in a database according to an embodiment.

[0084] Intelligent servers (e.g. Figure 2 The capsule DB of the intelligent server 290 in FIG. Figure 2The capsule DB 230 in the CAN 400 may store capsules in the form of a CAN (Concept Action Network) 400. The capsule DB may store an action for processing a task corresponding to a user voice input and a parameter required for the action in the form of a CAN (Concept Action Network).

[0085] The capsule DB may store a plurality of capsules (capsule A 401 and capsule B 404) corresponding to a plurality of fields (e.g., applications), respectively. According to an embodiment, one capsule (e.g., capsule A 401) may correspond to one field (e.g., location (geography) or application). In addition, the one capsule may correspond to at least one service provider (e.g., CP 1 402 or CP 2 403) that performs a function on the field related to the capsule. According to an embodiment, one capsule may include at least one action 410 and at least one concept 420 for performing a specified function.

[0086] As mentioned above, natural language platforms (e.g. Figure 2 The natural language platform 220 in the example may use the capsules stored in the capsule DB to generate a plan for performing a task corresponding to the received speech input. For example, a planner module of the natural language platform (e.g., Figure 2 The planner module 225 in the capsule DB can generate the plan using capsules stored in the capsule DB. For example, the action 4011 and the action 4013 and the concept 4012 and the concept 4014 of the capsule A 401, and the action 4041 and the concept 4042 of the capsule B 404 can be used to generate the plan 470.

[0087] Figure 4 A schematic diagram of a screen showing an electronic device processing a received voice input through a smart application according to an embodiment is shown.

[0088] The electronic device 201 can execute the intelligent application to communicate with the intelligent server (e.g., Figure 2 The intelligent server 290 in processes the user input.

[0089] According to an example embodiment, when a specified voice input (e.g., "Wake up!") is recognized or an input through a hardware key (e.g., a dedicated hardware key) is received, the electronic device 201 may execute a smart application for processing the voice input on the screen 310. For example, the electronic device 201 may execute the smart application while executing a schedule application. According to an embodiment, the electronic device 201 may display a screen 310 (e.g., Figure 1 The display module 160 or Figure 2The object (e.g., icon 311) corresponding to the smart application is displayed on the display module 204 in the electronic device 201. According to an embodiment, the electronic device 201 may receive a voice input of a user's speech. For example, the electronic device 201 may receive a voice input of "Tell me about this week's schedule!" According to an embodiment, the electronic device 201 may display a user interface (UI) 313 (e.g., an input window) of the smart application on the display module 204, in which the text data of the received voice input is displayed.

[0090] According to an embodiment, the electronic device 201 may display a result corresponding to the received voice input on the screen 320 on the display module 204. For example, the electronic device 201 may receive a plan corresponding to the received user input and display "this week's schedule" on the display module 204 according to the plan.

[0091] Figure 5 A schematic diagram of an intelligent voice system according to an embodiment is shown.

[0092] Reference Figure 5 According to an embodiment, an intelligent speech system equipped with a speaker recognition function (e.g., Fig. 7A System 700-1 or Figure 7B The system 700-2 in the example may analyze a first event 510 associated with an utterance in which speaker recognition fails, so as to determine whether the utterance in which speaker recognition fails is a falsely rejected utterance based on a second event 530 corresponding to the analysis result. The electronic device 540 in the second event 530 may be substantially the same as or different from the electronic device 520 in the first event 510 associated with an utterance in which speaker recognition fails. The first event 510 may be an event occurring at t=t1, and the second event 530 may be an event occurring at t=t2, where t=t2 may be a time point within a specified time (e.g., the effective time of the first event) from t=t1. In a multi-device scenario in which the electronic device 540 and the electronic device 520 are different devices, the intelligent voice system may reduce performance degradation due to differences in the availability of each device (e.g., utterance distance and usage conditions).

[0093] According to an embodiment, in a first event 510, the electronic device 520 may not be able to identify the speaker of the user speech (e.g., "What is my schedule today?"). In a second event 530, the electronic device 540 may receive user input (e.g., "What is my next appointment?"). The user input may be in a variety of forms, such as voice, touch, gesture, and action. When the intention of the user input corresponds to the intention of the speech in which the speaker recognition fails, the electronic device 540 may perform an action according to the intention of the user input, and provide information for verifying whether the speech in which the speaker recognition fails is falsely rejected. For example, the information for verifying false rejection may include a question about whether the speaker of the speech in which the speaker recognition fails is the user of the electronic device 540. The information for verifying false rejection may be provided to the user using visual, auditory, or other functions. The electronic device 540 may receive feedback from the user on the information for verifying false rejection. Based on the feedback, the electronic device 540 may determine whether the speech in which the speaker recognition fails is falsely rejected, and update the speaker model of the falsely rejected speech based on the falsely rejected speech.

[0094] According to an embodiment, the intelligent voice system can update the speaker model of the falsely rejected utterance based on the falsely rejected utterance, thereby strengthening the corresponding speaker model. The falsely rejected utterance may be caused by the failure of the initial speaker model learned based on the collected voice clips during the sound source registration process to fully characterize the user's voice. For example, when the user uses the speaker recognition function under different conditions (e.g., different speech distances, different amounts and types of ambient noise) that are different from the sound samples recorded during the sound source registration, or when the speech style changes due to the speaker's internal factors (e.g., the user's health, emotional state), the initial speaker model may not be able to recognize the speaker, and the user's speech may be falsely rejected.

[0095] According to the embodiment, since the intelligent voice system updates the speaker model of the falsely rejected utterance based on the falsely rejected utterance, it is possible to overcome the limitation of relearning the existing model by receiving additional recording segments from the user or collecting audio recognized as the user's voice when the user uses the electronic device. For example, the intelligent voice system can avoid the inconvenience of the user having to intentionally make additional recordings. In addition, since the intelligent voice system uses the falsely rejected utterance as learning data rather than the sound source that has been identified, the falsely rejected utterance part can be actively improved.

[0096] Figure 6 A flow chart showing an operation of an electronic device obtaining feedback to determine an utterance that is mistakenly rejected according to an embodiment.

[0097] In the following embodiments, the operations may be performed sequentially, but the embodiments are not limited thereto, and the operations do not necessarily have to be performed sequentially. For example, the order of the operations may be changed, and at least two operations may be performed in parallel.

[0098] According to an embodiment, Figure 6 Operations 610 to 640 in the embodiment may be performed by an electronic device (eg, Figure 2 A processor (eg, Figure 2 Processor 203 in executes.

[0099] According to an embodiment, operations 610 to 640 may illustrate that a second electronic device (eg, Figure 5 The process of obtaining feedback from the second electronic device 540 in the device to determine the utterance that was mistakenly rejected.

[0100] In operation 610, the second electronic device 540 may receive user input. The user input may be in various forms. For example, the user input may be in the form of voice, gesture, or touch. For example, the second electronic device 540 may receive a touch input for opening a calendar application.

[0101] In operation 620, the second electronic device 540 may generate a detection event corresponding to an event related to a previous utterance in which speaker recognition failed based on a user input (e.g., a touch input for opening a calendar application). The utterance in which speaker recognition failed may be a utterance intended to be successfully recognized. The utterance in which speaker recognition failed may be a utterance in which the first electronic device (e.g., Figure 5 In one example, the first electronic device 520 may be substantially the same as the second electronic device 540. In another example, the first electronic device 520 may be a different device from the second electronic device 540.

[0102] In an embodiment, an event related to an utterance in which speaker recognition fails may be an event of performing an action corresponding to the intention of the utterance in which speaker recognition fails. A detection event may be an event for detecting an event related to user input. In other words, a detection event may be an event for detecting user input of an action corresponding to the intention of the utterance in which speaker recognition fails. The second electronic device 540 may generate a detection event based on the occurrence time of the utterance in which speaker recognition fails and the intention of the utterance in which speaker recognition fails.

[0103] In an embodiment, the second electronic device 540 may generate a detection event for detecting an event (e.g., a related event) of performing an action corresponding to the intention of the previous utterance in which speaker recognition failed by determining whether the user input (e.g., a touch input for opening a calendar application) performs an action corresponding to the intention of the previous utterance in which speaker recognition failed. For example, when "turn up the music volume" and "what is my first schedule today?" are included in the previous utterance in which speaker recognition failed, the user input (e.g., a touch input for opening a calendar application) performs an action (e.g., executing a calendar application) corresponding to the intention (e.g., schedule search) of "what is my first schedule today?", and thus the second electronic device 540 may generate a detection event.

[0104] In operation 630, the second electronic device 540 may output information about the relevant event generated in response to the detection event. For example, the second electronic device 540 may output a user interface including information about the relevant event. The user interface may include one or more of an audio signal of the utterance for which speaker recognition failed and a question to the speaker about the utterance for which speaker recognition failed.

[0105] In operation 640, the second electronic device 540 may obtain feedback on the information about the relevant event and send the feedback to the smart server (eg, Figure 2 The feedback can be used to learn a speaker model for the utterance in which speaker recognition fails. Based on the feedback, the intelligent server 290 can determine whether the utterance in which speaker recognition fails is mistakenly rejected. In one example, when the utterance in which speaker recognition fails is mistakenly rejected, the intelligent server 290 can update the speaker model for the utterance in which speaker recognition fails based on the mistakenly rejected utterance. In another example, when the utterance in which speaker recognition fails is mistakenly rejected, the second electronic device 540 can update the speaker model for the utterance in which speaker recognition fails in the form of a device end. That is, the second electronic device 540 can directly update the speaker model for the utterance in which speaker recognition fails without going through the intelligent server 290, thereby updating the model in the form of a device end.

[0106] Fig. 7A A schematic diagram showing an operation of another device according to an embodiment to determine whether an utterance for which speaker recognition fails on a device is mistakenly rejected, and Figure 7B A schematic diagram showing an operation of a device according to an embodiment to determine whether an utterance for which speaker recognition fails on the device is mistakenly rejected.

[0107] Reference Fig. 7A and Figure 7BAccording to an embodiment, the systems 700-1 and 700-2 may include an audio analysis module 710, an audio classification module 720, a matching database 730, a matching module 740, update modules 751 and 752, a UI analysis module 760, a generation module 770, and an output module 780. The audio analysis module 710 may receive an utterance and analyze the intention and speaker of the utterance. The audio classification module 720 may classify the utterance based on the analysis results of the intention and speaker of the utterance.

[0108] According to an embodiment, the audio classification module 720 may classify the speech into a plurality of groups based on whether the speech is recognized and whether the speaker is recognized. Whether the speech is recognized may refer to whether the intent of the speech is recognized, and whether the speaker is recognized may refer to whether the subject of the speech is recognized. For example, speech recognition may be performed using automatic speech recognition (ASR), natural language understanding (NLU), or other techniques for identifying the intent of speech by extracting contextual information from the speech. Speaker recognition may be performed using techniques for identifying the subject of speech or an object corresponding thereto (e.g., a speaker ID).

[0109] For example, the multiple groups may include successful speech recognition and successful speaker recognition (e.g., the first group), successful speech recognition but failed speaker recognition (e.g., the second group), failed speech recognition but successful speaker recognition (e.g., the third group), and failed speech recognition and failed speaker recognition (e.g., the fourth group).

[0110] According to an embodiment, the audio classification module 720 may store the utterances (e.g., the second group) for which speech recognition succeeds but speaker recognition fails in the matching database 730. The matching database 730 may include information related to the utterances for which speaker recognition fails. Table 1 shows an example of information related to the utterances for which speaker recognition fails stored in the matching database 730.

[0111] [Table 1]

[0112] Here, the information related to the utterances may include the intention of the utterances, the effective time of the event related to the utterances for which speaker recognition failed, and the ID and memory address of the device storing the utterance audio signal.

[0113] For example, entry 2 in Table 1 may indicate that the intent analysis of the utterance for which speaker recognition failed is “browse schedule”. The utterance may indicate that time t1 has passed since the time when the utterance occurred, and the effective time of the event related to the utterance may be the remaining T The time t1. T may be the maximum valid time of an event related to the speech for which speaker recognition fails. The audio signal of the corresponding speech may be stored in a memory of a device having a device ID of 3, and the address of the memory may be 0x01.

[0114] As another example, when audio signals of corresponding utterances are stored in a server (e.g., IoT server 703), information related to the utterances may include the intention of the utterance, the effective time of an event related to the utterance for which speaker recognition failed, and the ID of the device that sent the audio signal of the utterance and the memory address where the audio signal of the utterance is stored.

[0115] According to an embodiment, when a new entry is stored in the matching database 730, the matching module 740 may be activated. The matching module 740 may monitor events related to utterances in which speaker recognition fails within the effective time of the corresponding event. Events related to utterances in which speaker recognition fails may be events of performing actions corresponding to the intentions of utterances in which speaker recognition fails. The matching module 740 may monitor detection events corresponding to related events. That is, the matching module 740 may monitor whether an event for detecting user input for performing an action corresponding to the intentions of utterances in which speaker recognition fails occurs. When there is a detection event, the matching module 740 may select the subject of the action as a speaker candidate for the utterance in which speaker recognition fails.

[0116] For example, the matching module 740 can monitor the detection event based on the event listener. In this example, the matching module 740 can register a listener for one or more electronic devices. The listener can determine the event related to the speech in which the speaker recognition fails as the target event. When the matching module 740 receives the detection event corresponding to the relevant event from one or more electronic devices, the matching module 740 can activate the generation module 770 and delete the entry corresponding to the detection event from the matching database 730 at the same time.

[0117] As another example, when user input history (eg, application usage) of one or more electronic devices is stored and managed on a server (eg, smart server 704), the matching module 740 may register a listener in an activity log of the server (eg, smart server 704).

[0118] According to an embodiment, the UI analysis module 760 may receive user input. The user input may be in various forms. For example, the user input may be voice, touch, or gesture. The UI analysis module 760 may generate a detection event based on the user input. That is, when the action performed by the user input corresponds to the action corresponding to the intention of the utterance in which the speaker recognition failed, the UI analysis module 760 may generate a detection event. The UI analysis module 760 may send the detection event to the matching module 740.

[0119] According to an embodiment, according to the detection event received by the matching module 740, the generation module 770 may be called by the matching module 740. The generation module 770 may generate information about the relevant event in response to the detection event. The generation module 770 may generate a user interface including information about the relevant event in response to the detection event. The user interface may include one or more of an audio signal of the utterance for which the speaker recognition failed and a question to the speaker about the utterance for which the speaker recognition failed. An example of the user interface will be referred to in FIG. 12A to FIG. 12C Further description.

[0120] According to an embodiment, the output module 780 may output information about the relevant event. For example, the output module 780 may output a user interface on a display of an electronic device (e.g., the second electronic device 702 or the third electronic device 705). As another example, the output module 780 may store information about the relevant event in an activity log of a server (e.g., the intelligent server 704) so ​​that the user can view the activity of the server (e.g., the intelligent server 704) when necessary.

[0121] According to an embodiment, the output module 780 may obtain feedback on information about the relevant event. The feedback may include at least one of the following: i) whether the speech for which the speaker recognition failed is the user's own voice, or ii) when it is not the user's own voice, whose speech it is. The output module 780 may send the feedback to the matching module 740. Based on the feedback, the matching module 740 may determine whether the speech for which the speaker recognition failed is falsely rejected.

[0122] According to an embodiment, the update module 751 or 752 may receive information about whether the utterance in which the speaker recognition fails is falsely rejected from the matching module 740. The update module 751 or 752 may learn (e.g., update) the speaker model of the utterance in which the speaker recognition fails based on whether the utterance is falsely rejected. When the update module 751 or 752 determines that the utterance in which the speaker recognition fails is a falsely rejected utterance, the update module 751 or 752 may use the utterance in which the speaker recognition fails as learning data for the speaker model of the utterance in which the speaker recognition fails. Therefore, the speaker model can perform incremental learning and continuous learning.

[0123] According to an embodiment, in system 700-1, another device (e.g., second electronic device 702) may determine whether an utterance in which speaker recognition fails in one device (e.g., first electronic device 701) is mistakenly rejected. Here, first electronic device 701 and second electronic device 702 may share a common IoT platform. For ease of description, Fig. 7A Only two electronic devices (eg, a first electronic device 701 and a second electronic device 702) are included in the system 700-1, but this is merely an example, and according to an embodiment, the system 700-1 may include two or more electronic devices.

[0124] According to an embodiment, the audio analysis module 710 and the audio classification module 720 may be implemented on the first electronic device 701, the matching database 730 may be implemented on the IoT server 703, the matching module 740 and the update module 751 may be implemented on the intelligent server 704, and the update module 752, the UI analysis module 760, the generation module 770 and the output module 780 may be implemented on the second electronic device 702. The update module 751 and the update module 752 may be selectively implemented.

[0125] According to an embodiment, when the system 700 - 1 does not support device-side operation, the audio analysis module 710 may be implemented on the IoT server 703 .

[0126] According to an embodiment, since the matching database 730 is implemented on the IoT server 703, the matching database 730 can be read / written from a public device registered on the IoT platform or from a user's personal device registered as a family member. That is, the matching database 730 can be shared between corresponding devices, and the user's utterance input to the personal device can be additionally stored in the matching database 730 or deleted from the matching database 730.

[0127] In this case, among the user utterances input to the personal device, the utterances additionally stored in the matching database 730 may be utterances for which intent recognition succeeds but speaker recognition fails. Among the entries corresponding to the user utterances input to the personal device, the entries to be deleted from the matching database 730 may include entries corresponding to utterances for which a specified effective time has passed since the time when the utterance occurred and entries corresponding to utterances for which user input and intent matching have been completed on another device within the effective time.

[0128] According to an embodiment, when the update module 752 is implemented on the second electronic device 702, the second electronic device 702 can directly update the speaker model of the mistakenly rejected utterance based on the mistakenly rejected utterance, and share the updated speaker model with another electronic device (e.g., the first electronic device 701) that shares the IoT platform.

[0129] For ease of description, Fig. 7A In the embodiment, an example is described in which only some modules among the multiple modules 710, 720, 760, 770, 780 and 752 are implemented on the first electronic device 701 and the second electronic device 702, but this is only a description of the modules related to the operation of the system 700-1, and the first electronic device 701 and the second electronic device 702 can respectively include all modules among the multiple modules 710, 720, 760, 770, 780 and 752.

[0130] According to an embodiment, system 700-2 shows an operation of a device (e.g., a third electronic device 705) determining whether an utterance in which speaker recognition fails in the device (e.g., the third electronic device 705) is mistakenly rejected. The audio analysis module 710, the audio classification module 720, the update module 752, the UI analysis module 760, the generation module 770, and the output module 780 can be implemented on the third electronic device 705, the matching database 730 can be implemented on the IoT server 703, and the matching module 740 and the update module 751 can be implemented on the intelligent server 704. The update module 751 and the update module 752 can be implemented selectively.

[0131] Figure 8 A schematic diagram showing an operation of an electronic device classifying an utterance for which speaker recognition fails according to an embodiment.

[0132] Reference Figure 8 According to an embodiment, an electronic device (e.g., Fig. 7A The first electronic device 701 or Figure 7B The third electronic device 705 in the embodiment may classify the speech into a plurality of groups based on whether the speech is recognized and whether the speaker is recognized. Whether the speech is recognized may refer to whether the intention of the speech is recognized, and whether the speaker is recognized may refer to whether the subject of the speech is recognized. For example, speech recognition may be performed using ASR, NLU, or other technologies that extract context information from the speech to recognize the intention of the speech. Speaker recognition may be performed using a technology for recognizing the subject of the speech or an object corresponding thereto (e.g., speaker ID, etc.).

[0133] For example, the multiple groups may include successful speech recognition and successful speaker recognition (e.g., the first group), successful speech recognition but failed speaker recognition (e.g., the second group) 810, failed speech recognition but successful speaker recognition (e.g., the third group), and failed speech recognition and failed speaker recognition (e.g., the fourth group).

[0134] Fig. 9A schematic diagram showing an operation example of an intelligent server updating a speaker model of an utterance for which speaker recognition failed based on an utterance that was mistakenly rejected according to an embodiment, and Fig.10 A schematic diagram showing another operation example of an intelligent server updating a speaker model of an utterance for which speaker recognition fails based on an utterance that was mistakenly rejected according to an embodiment.

[0135] In the following embodiments, the operations may be performed sequentially, but not necessarily sequentially. For example, the order of the operations may be changed, and at least two operations may be performed in parallel.

[0136] Operations 901 to 907 may illustrate an intelligent server (eg, Fig. 7A An example of an operation in which the smart server 704 in FIG. 7 matches intents for electronic devices (eg, device 1, device 2, and device 3) that share a common IoT platform.

[0137] In operation 901 , speaker A may input an utterance (eg, a voice command) “play my playlist” through device 1 (eg, a speaker).

[0138] In operation 902, an audio analysis module (e.g., Fig. 7A The audio analysis module 710 in the example may receive the speech and analyze the intent and speaker of the speech. Fig. 7A The audio classification module 720 in the IoT server can classify the utterance into multiple groups based on whether the voice is recognized and whether the speaker is recognized. For example, the audio classification module 720 can classify the utterance as an utterance for which the voice recognition succeeds but the speaker recognition fails. The audio classification module 720 can send the classification result to the IoT server (e.g., Fig. 7A IoT server 703 in the example.

[0139] In operation 903, the IoT server 703 sends a response to the device 1 according to the classification result. For example, the response according to the classification result may include "speech recognition is successful, but due to speaker recognition failure, user registration is required". The IoT server 703 may write information about the utterance for which speaker recognition failed (e.g., audio, intent, etc.) into the matching database (e.g., Fig. 7A The sound matching failure history table 910 of the matching database 730).

[0140] In operation 904, the IoT server 703 may simultaneously register listeners for individual devices (e.g., device 2 and device 3) sharing the IoT platform. The target event of the listener may be an event related to the utterance in which speaker recognition failed. For example, the target event of the listener may be an event for performing an action corresponding to the intention of the utterance in which speaker recognition failed. The target event of the listener may be in an active state for a maximum of T hours from the time when the utterance in which speaker recognition failed occurred.

[0141] In operation 905, the UI analysis module (eg, Fig. 7A The UI analysis module 760 in the device 2 may generate a detection event based on the detection of a target event (e.g., an event of performing an action corresponding to playing a playlist) by a listener for the device 2. In response to the detection event, the IoT server 703 may delete the entry corresponding to the utterance for which the speaker recognition failed from the sound matching failure history table 910. The UI analysis module 760 may send the detection event to the smart server (e.g., Fig. 7A ), and the intelligent server 704 can trigger a generation module (e.g., Fig. 7A Generation module 770 in ), thereby providing information about events related to the utterance in which speaker recognition fails to be provided to the user of device 2 through the user interface.

[0142] In operation 906 , the user of device 2 may input feedback through the user interface. For example, when the user of device 2 matches with user A of device 1 , the feedback may be “‘Play my playlist’ was entered 5 minutes ago”. Device 2 may send the feedback to the smart server 704 .

[0143] In operation 907, based on the feedback, since the user of device 2 matches user A of device 1, the intelligent server 704 can determine that the utterance for which speaker recognition failed is mistakenly rejected, and update the speaker model of the utterance for which speaker recognition failed (e.g., the speaker model of account A) based on the mistakenly rejected utterance.

[0144] According to an embodiment, the speaker model may be operated for each account. For example, Fig. 9 One or more devices (for example, device 1, device 2, and device 3) sharing an IoT platform can share the speaker model of account A and the speaker model of account B.

[0145] According to an embodiment, each speaker model may operate for each account and device. For example, Fig.10One or more devices (for example, device 1, device 2, and device 3) sharing an IoT platform can share the device 1 speaker model of account A, the device 2 speaker model of account A, the device 1 speaker model of account B, and the device 3 speaker model of account B.

[0146] Fig.11 A flowchart of the operation of an intelligent server updating a speaker model of a falsely rejected utterance based on the falsely rejected utterance according to an embodiment is shown.

[0147] In the following embodiments, the operations may be performed sequentially, but not necessarily sequentially. For example, the order of the operations may be changed, and at least two operations may be performed in parallel.

[0148] Operations 1110 to 1190 may illustrate the operation of a smart server (eg, Fig. 7A and Figure 7B , wherein when the speaker recognition of the user utterance fails, the intelligent server (e.g., Fig. 7A and Figure 7B The intelligent server 704 in the example manages the user utterance as a failure history in the matching database, and when the speech recognition and speaker recognition of the user utterance are successful, the intelligent server (e.g., Fig. 7A and Figure 7B The intelligent server 704 in the embodiment determines whether the utterance for which the speaker recognition failed previously is falsely rejected, and updates the speaker model of the falsely rejected utterance.

[0149] In operation 1110, an electronic device (eg, Fig. 7A The first electronic device 701 or Figure 7B The third electronic device 705 in the example may receive the user's utterance (eg, "What is my schedule today?"). The user's utterance may correspond to a voice command.

[0150] In operation 1120, the electronic device 701 or 705 may determine whether the voice in the speech is recognized. Whether the voice is recognized may refer to whether the intention of the speech is recognized. When the voice recognition is successful, the electronic device 701 or 705 may perform operation 1140, and when the voice recognition fails, operation 1130 may be performed.

[0151] In operation 1130, when the voice recognition of the utterance fails, the electronic device 701 or 705 may ignore the utterance.

[0152] In operation 1140, the electronic device 701 or 705 may determine whether the speaker of the speech is recognized. Whether the speaker is recognized may refer to whether the subject of the speech is recognized. When the speaker recognition is successful, the electronic device 701 or 705 may perform operation 1160, and when the speaker recognition fails, operation 1150 may be performed.

[0153] In operation 1150, the electronic device 701 or 705 may output information corresponding to the user registration request to the user (eg, “Please register a user”). The electronic device 701 or 705 may store the utterance in a matching database (eg, Fig. 7A and Figure 7B In the matching database 730 in FIG. 7 , the matching database 730 may include, in addition to the utterance for which the speaker recognition failed, other utterances for which the speaker recognition failed. The intelligent server (e.g., Fig. 7A and Figure 7B The intelligent server 704 in the matching database 730 may generate events related to one or more utterances for which speaker recognition failed to be stored in the matching database 730. The related event may be an event for performing an action corresponding to the intent of the utterance for which speaker recognition failed. For example, when speaker recognition fails for the utterance "What is my schedule today?", the related event may be an event for performing an action (e.g., executing a calendar application) corresponding to the intent (e.g., schedule search) of "What is my schedule today?"

[0154] In operation 1160, the intelligent server 704 may determine whether to generate a detection event based on the speech in which the voice recognition is successful and the speaker recognition is successful. The intelligent server 704 may determine whether to generate an event (e.g., a detection event) for detecting an event related to a speech in which the previous speaker recognition failed based on the speech in which the voice recognition is successful and the speaker recognition is successful. That is, the intelligent server 704 may verify whether to generate a detection event by determining whether the intention of the speech in which the voice recognition is successful and the speaker recognition is successful corresponds to the intention of one or more speech in which the speaker recognition failed stored in the matching database 730. When a detection event is generated, the electronic device 701 or 705 may perform operation 1170, and when a detection event is not generated, operation 1200 may be performed. In other words, the intelligent server 704 may determine whether to generate a detection event corresponding to the intention by determining whether the intention (e.g., schedule search) of the speech "What is my schedule today?" corresponds to the intention of one or more speech in which the speaker recognition failed stored in the matching database 730. For example, when the speech recognition and speaker recognition of the utterance “What’s my schedule today?” are successful, and the intent (e.g., schedule search) of the utterance “What’s my schedule today?” corresponds to the intent of the utterance in which the speaker recognition failed previously, the intelligent server 704 may perform operation 1170. When the speech recognition and speaker recognition of the utterance “What’s my schedule today?” are successful, and the intent (e.g., schedule search) of the utterance “What’s my schedule today?” does not correspond to the intent of the utterance in which the speaker recognition failed previously, the intelligent server 704 may perform operation 1200.

[0155] In operation 1170, when a detection event is generated, the electronic device 701, 705 may generate a user interface in response to the detection event using a generation module activated by the intelligent server 704. The electronic device 701, 705 may obtain feedback on the speech corresponding to the detection event in the speech in which one or more speaker recognition fails through the user interface. The electronic device 701, 705 may send the feedback to the intelligent server 704.

[0156] In operation 1180, the intelligent server 704 can determine whether the speech corresponding to the detection event is a falsely rejected speech based on the feedback. When the speech corresponding to the detection event is a falsely rejected speech, the electronic device 701 or 705 can perform operation 1190, and when the speech corresponding to the detection event is not a falsely rejected speech, operation 1200 can be performed.

[0157] In operation 1190, the intelligent server 740 may update the speaker model corresponding to the falsely rejected utterance based on the falsely rejected utterance. In an embodiment, the electronic device 701 or 705 may directly update the speaker model corresponding to the falsely rejected utterance based on the falsely rejected utterance.

[0158] In operation 1200 , the electronic device 701 or 705 may perform an action corresponding to the intention of the utterance for which voice recognition and speaker recognition are successful.

[0159] Fig. 12A An example of a user interface for obtaining feedback according to an embodiment is shown.

[0160] Reference Fig. 12A According to an embodiment, when a detection event is generated based on a user input, an electronic device (eg, Fig. 7A The second electronic device 702 or Figure 7B The third electronic device 705 in the example may output information about the speech in which the speaker recognition fails corresponding to the detection event after the user inputs. For example, when the user requests to save the call record to the contact on the home device through a voice command, but the command is not executed due to the failure of the speaker recognition, the user may issue a voice command to another device (e.g., a smart phone) within the effective time of the event related to the speech in which the speaker recognition fails to be performed. The function of the intention (e.g., saving the call record) corresponding to the speech in which the speaker recognition fails can be executed by issuing a voice command to another device (e.g., a smart phone). In this example, a user interface may be provided to the user to determine whether the audio of the speech in which the speaker recognition fails (e.g., "Save the call record now") is the user's own voice. The user interface may include a playback of the speech in which the speaker recognition fails (e.g., "Save the call record now") and a positive / negative selection box (e.g., a display object) for the user's voice. The user may input feedback on the speech in which the speaker recognition fails (e.g., "Save the call record now") through the user interface.

[0161] Fig. 12B An example of a user interface for obtaining feedback according to an embodiment is shown.

[0162] Reference Fig. 12B According to the embodiment, in addition to referring to Fig. 12A In addition to the described user interface, electronic device 702 or 705 may generate a user interface that further includes a selection box (e.g., “No, this is the voice of my brother (Hongdukkae)”) for determining whether a speaker exists in one or more devices of the shared IoT platform when the user is not the speaker of the utterance for which speaker recognition fails.

[0163] Fig. 12CAn example of a user interface for obtaining feedback according to an embodiment is shown.

[0164] Reference Fig. 12C According to an embodiment, the electronic device 702 or 705 may store information about related events in an activity log of a server (e.g., the intelligent server 704) so ​​that the user can view the activities of the server (e.g., the intelligent server 704) when necessary. In addition, the electronic device 702 or 705 may also provide a user interface for obtaining feedback on utterances for which speaker recognition failed when viewing the activity log.

[0165] According to one aspect of the present disclosure, an electronic device (101; 201; 520; 540; 701; 702; 705) includes a memory (130; 207) containing instructions and a processor (120; 203) electrically connected to the memory (130; 207) and configured to execute the instructions. When the instructions are executed by the processor (120; 203), the processor (120; 203) can be configured to perform multiple operations. The multiple operations may include receiving user input. The multiple operations may include generating a detection event corresponding to an event related to an utterance in which speaker recognition fails based on the user input. The multiple operations may include outputting information about the related event generated in response to the detection event. The multiple operations may include obtaining feedback on the information about the related event. The feedback may be used to learn a speaker model for the utterance in which speaker recognition fails.

[0166] An utterance for which speaker recognition fails may be an utterance for which intent recognition succeeds.

[0167] The relevant event may be an event of performing an action corresponding to the intention of the utterance for which speaker recognition failed.

[0168] The detection event may be an event for detecting a user input to perform an action corresponding to an intention of an utterance in which speaker recognition failed.

[0169] Generating may include generating a detection event based on a time of occurrence of the utterance for which speaker recognition failed and an intent of the utterance for which speaker recognition failed.

[0170] The outputting may include outputting information of the utterance in which speaker recognition fails when it is determined based on the detection event that the intention of the user input matches the intention of the utterance in which speaker recognition fails.

[0171] Outputting may include outputting a user interface including information about the relevant event.

[0172] The user interface may include one or more of an audio signal of the utterance for which speaker recognition failed and a question to the speaker regarding the utterance for which speaker recognition failed.

[0173] Feedback may be used to determine whether an utterance for which speaker recognition failed was falsely rejected.

[0174] The plurality of operations may also include updating a speaker model based on the utterance for which speaker recognition failed.

[0175] The plurality of operations may also include sharing the updated speaker model with another electronic device.

[0176] The speaker model can be updated based on utterances for which speaker recognition fails.

[0177] According to an embodiment, a method of operating an electronic device (101; 201; 520; 540; 701; 702; 705) may include receiving user input. The method may include generating a detection event corresponding to an event related to an utterance for which speaker recognition fails based on the user input. The method may include outputting information about the related event generated in response to the detection event. The method may include obtaining feedback on the information about the related event. The feedback may be used to learn a speaker model for the utterance for which speaker recognition fails.

[0178] According to one aspect of the present disclosure, an electronic device (101; 201; 520; 540; 701; 702; 705) includes a memory (130; 207) containing instructions and a processor (120; 203) electrically connected to the memory (130; 207) and configured to execute the instructions. When the instructions are executed by the processor (120; 203), the processor (120; 203) may be configured to perform multiple operations. The multiple operations may include classifying utterances in which speaker recognition fails among received utterances. The multiple operations may include receiving user input within a specified time from the time when the utterance in which speaker recognition fails occurs. The multiple operations may include generating a detection event corresponding to an event related to the utterance in which speaker recognition fails based on the user input. The multiple operations may include outputting information about the related event generated in response to the detection event. The multiple operations may include obtaining feedback on the information about the related event. The feedback may be used to learn a speaker model for the utterance in which speaker recognition fails.

[0179] An utterance for which speaker recognition fails may be an utterance for which intent recognition succeeds.

[0180] The relevant event may be an event of performing an action corresponding to the intention of the utterance for which speaker recognition failed.

[0181] The detection event may be an event for detecting a user input to perform an action corresponding to an intention of an utterance in which speaker recognition failed.

[0182] The outputting may include outputting information of the utterance in which speaker recognition fails when it is determined based on the detection event that the intention of the user input matches the intention of the utterance in which speaker recognition fails.

[0183] Outputting may include outputting a user interface including information about the relevant event.

[0184] The user interface may include one or more of an audio signal of the utterance for which speaker recognition failed and a question to the speaker regarding the utterance for which speaker recognition failed.

[0185] The electronic device in the embodiments disclosed herein may be one of various types of electronic devices. The electronic device may include, for example, a portable communication device (e.g., a smart phone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a household appliance. According to the embodiments of the present disclosure, the electronic device is not limited to those described above.

[0186] It should be understood that the various embodiments of the present disclosure and the terms used therein are not intended to limit the technical features set forth herein to specific embodiments, but include various changes, equivalents or substitutions to the corresponding embodiments. When described in conjunction with the accompanying drawings, similar reference numerals may be used for similar or related components. It should be understood that, unless the relevant context clearly indicates otherwise, the singular form of a noun corresponding to a certain item may include one or more of the item. As used herein, "A or B", "at least one of A and B", "at least one of A or B", "A, B or C", "at least one of A, B and C", and "at least one of A, B or C", each of which may be included in any one of the items listed together in the corresponding phrase, or all possible combinations thereof. Terms such as "first", "second" and the like may be used only to distinguish the component from other components discussed, without limiting these components in other aspects (e.g., importance or order). It should be understood that if one element (e.g., a first element) is referred to as being “coupled with,” “coupled to,” “connected to,” or “connected with” another element (e.g., a second element), whether or not the term “operably” or “communicatively” is used, it means that the element can be coupled to the other element directly (e.g., by wires), wirelessly, or via a third element.

[0187] As used in conjunction with various embodiments of the present disclosure, the term "module" may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with other terms (e.g., "logic," "logic block," "component," or "circuit"). A module may be a single integral component adapted to perform one or more functions, or a minimum unit or portion thereof. For example, according to an example, a module may be implemented in the form of an application specific integrated circuit (ASIC).

[0188] The embodiments of the present disclosure described herein may be implemented as software (e.g., program 140) that includes one or more instructions stored in a storage medium (e.g., internal memory 136 or external memory 138) readable by a machine (e.g., electronic device 101). For example, a processor (e.g., processor 120) of a machine (e.g., electronic device 101) may call at least one of the one or more instructions stored in the storage medium and execute it. This enables the machine to be operated to perform at least one function according to at least one instruction called. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-volatile storage medium. Here, the term "non-volatile" simply means that the storage medium is a tangible device and does not include signals (e.g., electromagnetic waves), but the term does not distinguish whether data is semi-permanently stored or temporarily stored in the storage medium.

[0189] According to an embodiment, the method according to various embodiments of the present disclosure may be included and provided in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., a compact disc read-only memory (CD-ROM)), or may be downloaded through an application store (e.g., PlayStore). TM ) online distribution (e.g. downloading or uploading) or directly between two user devices (e.g. smartphones). If distributed online, at least part of the computer program product may be temporarily generated or at least temporarily stored in a machine-readable storage medium, such as a memory of a manufacturer's server, a server of an application store or a memory of a relay server.

[0190] According to an embodiment, each of the above-mentioned components (e.g., a module or a program) may include a single entity or multiple entities, and some of the multiple entities may be respectively arranged in different components. According to an embodiment, one or more of the above-mentioned components or operations may be omitted, or one or more other components or operations may be added. Alternatively or additionally, multiple components (e.g., modules or programs) may be integrated into a single component. In this case, the integrated component may still perform one or more functions of each of the multiple components in the same or similar manner as performed by a corresponding one of the multiple components before integration. According to an embodiment, the operations performed by a module, program or other component may be performed sequentially, in parallel, repeatedly or in a heuristic manner, or one or more operations may be performed in a different order or omitted, or one or more other operations may be added.

Claims

1. An electronic device (101; 201; 520； 540；701； 702; 705), including: a memory (130; 207), wherein instructions are stored in the memory; and A processor (120; 203) operatively connected to the memory (130; 207) and configured to execute the instructions to: Receive user input; Based on the user input, generating a detection event corresponding to a related event, wherein the related event is related to the utterance for which speaker recognition failed; outputting information about the relevant event generated in response to the detected event; and obtaining feedback on the information about the relevant event, The feedback is used to learn the speaker model of the utterance for which speaker recognition fails.

2. The electronic device (101; 201; 520; 540; 701; 702; 705) according to claim 1, wherein: The utterance for which speaker recognition fails is the utterance for which intention recognition succeeds.

3. The electronic device (101; 201; 520; 540; 701; 702; 705) according to any one of claims 1 and 2, wherein: The relevant event includes performing an action corresponding to the intention of the utterance for which speaker recognition fails.

4. The electronic device (101; 201; 520; 540; 701; 702; 705) according to any one of claims 1 to 3, wherein: The detection event is used to detect user input for performing an action corresponding to the intention of the utterance for which speaker recognition failed.

5. The electronic device (101; 201; 520; 540; 701; 702; 705) according to any one of claims 1 to 4, wherein: The processor (120; 203) is further configured to execute the instructions to generate the detection event based on the occurrence time of the utterance for which speaker recognition fails and the intention of the utterance for which speaker recognition fails.

6. The electronic device (101; 201; 520; 540; 701; 702; 705) according to any one of claims 1 to 5, wherein: The processor (120; 203) is also configured to execute the instructions to output the information about the related event by outputting information about the utterance in which speaker recognition failed when it is determined based on the detection event that the intention of the user input matches the intention of the utterance in which speaker recognition failed.

7. The electronic device (101; 201; 520; 540; 701; 702; 705) according to any one of claims 1 to 6, wherein: The processor (120; 203) is further configured to execute the instructions to output the information about the relevant events by outputting a user interface including the information about the relevant events.

8. The electronic device (101; 201; 520; 540; 701; 702; 705) according to any one of claims 1 to 7, wherein: The user interface includes at least one of the following: an audio signal of the utterance for which speaker recognition fails; and A question about the speaker of the utterance for which speaker identification failed.

9. The electronic device (101; 201; 520; 540; 701; 702; 705) according to any one of claims 1 to 8, wherein: The feedback is used to determine whether the utterance for which speaker recognition failed is falsely rejected.

10. The electronic device (101; 201; 520; 540; 701; 702; 705) according to any one of claims 1 to 9, wherein: The processor (120; 203) is further configured to execute the instructions to update the speaker model based on the utterance for which speaker recognition fails.

11. The electronic device (101; 201; 520; 540; 701; 702; 705) according to any one of claims 1 to 10, wherein: The processor (120; 203) is further configured to execute the instructions to provide the updated speaker model to another electronic device (101; 201; 520; 540; 701; 702; 705).

12. The electronic device (101; 201; 520; 540; 701; 702; 705) according to any one of claims 1 to 11, wherein: The speaker model is updated based on the utterance for which speaker recognition fails.

13. An operating electronic device (101; 201; 520； 540；701； 702； 705), the method comprising: Receive user input; Based on the user input, generating a detection event corresponding to a related event, wherein the related event is related to the utterance for which speaker recognition failed; outputting information about the relevant event generated in response to the detected event; and obtaining feedback on the information about the relevant event, The feedback is used to learn the speaker model of the utterance for which speaker recognition fails.

14. An electronic device (101; 201; 520； 540；701； 702; 705), including: a memory (130; 207), wherein instructions are stored in the memory; and A processor (120; 203) operatively connected to the memory (130; 207) and configured to execute the instructions to: Among the received utterances, classify the utterances for which speaker recognition fails; receiving user input within a specified time from the time of occurrence of the utterance for which speaker recognition failed; Based on the user input, generating a detection event corresponding to a related event, wherein the related event is related to the utterance for which speaker recognition failed; outputting information about the relevant event generated in response to the detected event; and obtaining feedback on the information about the relevant event, The feedback is used to learn the speaker model of the utterance for which speaker recognition fails.

15. The electronic device (101; 201; 520; 540; 701; 702; 705) according to claim 14, wherein: The utterance for which speaker recognition fails is the utterance for which intention recognition succeeds.

Images