Electronic device, intelligent server and speaker adaptive speech recognition method

Abstract

An electronic device according to an embodiment may include: a memory storing at least one instruction; and at least one processor operatively connected to the memory and configured to execute at least one instruction. The at least one processor may perform speaker verification on the speech input to determine whether the speech input matches the speech of the registered speaker. If the speech input does not match the speech of the registered speaker, the at least one processor may perform a first speech recognition on the speech input based on the first ASR model. If the speech input matches the speech of the registered speaker, the at least one processor may perform second speech recognition on the speech input based on the SSN and a second ASR model. Additional embodiments may be possible.

Description

Technical Field

[0001] The present disclosure relates to an electronic device, an intelligent server and a speaker-adaptive speech recognition method. Background Art

[0002] Electronic devices equipped with voice assistant functions to provide services based on user speech are being distributed. The electronic device can recognize the user's speech and figure out the meaning and intention of the speech through an artificial intelligence (AI) server. The AI ​​server can infer the user's intention by interpreting the user's speech, and can perform tasks according to the inferred intention. The AI ​​server can perform tasks according to the user's intention expressed through natural language interaction between the user and the AI ​​server. Summary of the invention

[0003] Technical Solution According to one aspect of the present disclosure, an electronic device includes: a memory storing at least one instruction; and at least one processor operatively connected to the memory. The at least one processor is configured to execute at least one instruction to: perform speaker verification on a voice input to determine whether the voice input matches the voice of a registered speaker; based on determining that the voice input does not match the voice of the registered speaker, perform first voice recognition on the voice input based on a first automatic speech recognition (ASR) model; and based on determining that the voice input matches the voice of the registered speaker, perform second voice recognition on the voice input based on a sequence summary neural network (SSN) and a second ASR model.

[0004] According to one aspect of the present disclosure, an intelligent server includes: a memory storing at least one instruction; and at least one processor operatively connected to the memory. The at least one processor is configured to run at least one instruction to: perform speaker verification on a voice input to determine whether the voice input matches the voice of a registered speaker; based on determining that the voice input does not match the voice of the registered speaker, perform a first voice recognition on the voice input based on a first automatic speech recognition (ASR) model; and based on determining that the voice input matches the voice of the registered speaker, perform a second voice recognition on the voice input based on a sequence summary neural network (SSN) and a second ASR model.

[0005] According to one aspect of the present disclosure, an operating method of an intelligent server includes: performing speaker verification on a voice input to determine whether the voice input matches the voice of a registered speaker; based on determining that the voice input does not match the voice of the registered speaker, performing a first voice recognition on the voice input based on a first automatic speech recognition (ASR) model; and based on determining that the voice input matches the voice of the registered speaker, performing a second voice recognition on the voice input based on a sequence summary neural network (SSN) and a second ASR model. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] Figure 1 is a block diagram illustrating an electronic device in a network environment according to an embodiment; Figure 2 is a block diagram illustrating an integrated intelligent system according to an embodiment; Figure 3 is a diagram showing relationship information between concepts and actions stored in a database according to an embodiment; Figure 4 is a diagram showing a screen of an electronic device processing a received voice input through a smart app according to an embodiment; Figure 5 is a diagram showing speech recognition by an intelligent server according to an embodiment; Figure 6a is a schematic block diagram of an electronic device according to an embodiment; Figure 6b is a schematic block diagram of a sequence summarization neural network (SSN) according to an embodiment; Figure 6c is a schematic block diagram of an adaptation layer according to an embodiment; Figure 7 is a diagram illustrating a method of training an SSN and an adaptation layer according to an embodiment; Figure 8 is a diagram showing a result of voice recognition performed by an electronic device according to an embodiment; Fig. 9 is a flowchart illustrating an operation method of an intelligent server according to an embodiment. DETAILED DESCRIPTION

[0007] Hereinafter, various embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. When describing various embodiments with reference to the accompanying drawings, like reference numerals refer to like elements, and repeated descriptions thereof may be omitted.

[0008] Figure 1 is a block diagram showing an electronic device 101 in a network environment 100 according to an embodiment. Figure 1, the electronic device 101 in the network environment 100 may communicate with the external electronic device 102 via the first network 198 (e.g., a short-range wireless communication network), or communicate with at least one of the external electronic device 104 or the server 108 via the second network 199 (e.g., a long-range wireless communication network). According to an embodiment, the electronic device 101 may communicate with the external electronic device 104 via the server 108. According to an embodiment, the electronic device 101 may include at least one processor 120, a memory 130, an input module 150, a sound output module 155, a display module 160, an audio module 170, and a sensor module 176, an interface 177, a connection terminal 178, a haptic module 179, a camera module 180, a power management module 188, a battery 189, a communication module 190, a user identification module (SIM) 196, or an antenna module 197. In an embodiment, at least one of the above components (e.g., the connection terminal 178) may be omitted from the electronic device 101, or one or more other components may be added to the electronic device 101. In an embodiment, at least one of the above-described components (eg, the sensor module 176 , the camera module 180 , or the antenna module 197 ) may be integrated into a single component (eg, the display module 160 ).

[0009] At least one processor 120 may be a single processor or a plurality of processors each running one or more operations. The processor 120 may run, for example, software (e.g., program 140) to control at least one other component (e.g., hardware component or software component) of the electronic device 101 connected to the processor 120, and may perform various data processing or calculations. According to an embodiment, as at least part of the data processing or calculation, the processor 120 may store a command or data received from another component (e.g., sensor module 176 or communication module 190) in a volatile memory 132, process the command or data stored in the volatile memory 132, and store the result data in a non-volatile memory 134. According to an embodiment, the processor 120 may include a main processor 121 (e.g., a central processing unit (CPU) or an application processor (AP)), an auxiliary processor 123 (e.g., a graphics processing unit (GPU), a neural processing unit (NPU), an image signal processor (ISP), a sensor hub processor, and / or a communication processor (CP)) that is independent of or combined with the main processor 121 in operation. For example, when the electronic device 101 includes a main processor 121 and an auxiliary processor 123, the auxiliary processor 123 may be adapted to consume less power than the main processor 121, or adapted to be dedicated to a specific function. The auxiliary processor 123 may be implemented separately from the main processor 121, or as part of the main processor 121.

[0010] When the main processor 121 is in an inactive (e.g., sleep) state, the auxiliary processor 123 (rather than the main processor 121) may control at least some of the functions or states related to at least one component among the components of the electronic device 101 (e.g., display module 160, sensor module 176, or communication module 190), or when the main processor 121 is in an active state (e.g., running an application), the auxiliary processor 123 may control at least some of the functions or states related to at least one component among the components of the electronic device 101 (e.g., display module 160, sensor module 176, or communication module 190) together with the main processor 121. According to an embodiment, the auxiliary processor 123 (e.g., ISP or CP) may be implemented as part of another component (e.g., camera module 180 or communication module 190) that is functionally related to the auxiliary processor 123. According to an embodiment, the auxiliary processor 123 (e.g., NPU) may include a hardware structure dedicated to artificial intelligence model processing. The artificial intelligence model may be generated through machine learning. Such learning may be performed, for example, by the electronic device 101 where the artificial intelligence model is executed or via a separate server (e.g., server 108). The learning algorithm may include, but is not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, for example. The artificial intelligence model may include multiple artificial neural network layers. The artificial neural network may include, for example, a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), and a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), or a deep Q network or a combination of two or more thereof, but is not limited thereto. Additionally or optionally, the artificial intelligence model may include a software structure in addition to a hardware structure.

[0011] The memory 130 may store various data used by at least one component of the electronic device 101 (e.g., the processor 120 or the sensor module 176). The various data may include, for example, software (e.g., the program 140) and input data or output data for commands related thereto. The memory 130 may include a volatile memory 132 or a non-volatile memory 134.

[0012] The program 140 may be stored as software in the memory 130 , and may include, for example, an operating system (OS) 142 , middleware 144 , or applications 146 .

[0013] The input module 150 may receive a command or data to be used by another component (e.g., the processor 120) of the electronic device 101 from outside (e.g., a user) of the electronic device 101. The input module 150 may include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus).

[0014] The sound output module 155 can output sound signals to the outside of the electronic device 101. The sound output module 155 may include, for example, a speaker or a receiver. The speaker may be used for general purposes such as playing multimedia or playing records. The receiver may be used to receive incoming calls. Depending on the embodiment, the receiver may be implemented as a separate part from the speaker, or as part of the speaker.

[0015] The display module 160 can visually provide information to the outside of the electronic device 101 (e.g., a user). The display module 160 may include, for example, a display, a holographic device, or a projector, and a control circuit for controlling a corresponding one of the display, the holographic device, and the projector. According to an embodiment, the display module 160 may include a touch sensor adapted to detect a touch or a pressure sensor adapted to measure the strength of a force caused by a touch.

[0016] The audio module 170 may convert sound into an electrical signal, or vice versa. According to an embodiment, the audio module 170 may obtain sound via the input module 150, or output sound via the sound output module 155 or an external electronic device (e.g., an external electronic device 102 such as a speaker or earphone) directly or wirelessly connected to the electronic device 101.

[0017] The sensor module 176 may detect an operating state (e.g., power or temperature) of the electronic device 101 or an environmental state (e.g., a state of a user) outside the electronic device 101, and generate an electrical signal or data value corresponding to the detected state. According to an embodiment, the sensor module 176 may include, for example, a gesture sensor, a gyro sensor, an atmospheric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an infrared (IR) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illumination sensor.

[0018] The interface 177 may support one or more specific protocols to be used to couple the electronic device 101 with an external electronic device (e.g., external electronic device 102) (e.g., wired) or wirelessly. According to an embodiment, the interface 177 may include, for example, a high-definition multimedia interface (HDMI), a universal serial bus (USB) interface, a secure digital (SD) card interface, or an audio interface.

[0019] The connection terminal 178 may include a connector, wherein the electronic device 101 can be physically connected to an external electronic device (e.g., the external electronic device 102) via the connector. According to an embodiment, the connection terminal 178 may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0020] The haptic module 179 may convert the electric signal into mechanical stimulation (eg, vibration or movement) or electric stimulation that can be recognized by the user via his or her sense of touch or kinesthetic sense. According to an embodiment, the haptic module 179 may include, for example, a motor, a piezoelectric element, or an electric stimulator.

[0021] The camera module 180 may capture still images and moving images. According to an embodiment, the camera module 180 may include one or more lenses, an image sensor, an image signal processor, or a flash.

[0022] The power management module 188 may manage power supply to the electronic device 101. According to an embodiment, the power management module 188 may be implemented as, for example, at least a part of a power management integrated circuit (PMIC).

[0023] The battery 189 may power at least one component of the electronic device 101. According to an embodiment, the battery 189 may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0024] The communication module 190 may support establishing a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device 101 and an external electronic device (e.g., external electronic device 102, external electronic device 104, or server 108), and perform communication via the established communication channel. The communication module 190 may include one or more communication processors that are capable of operating independently from the processor 120 (e.g., AP) and supporting direct (e.g., wired) communication or wireless communication. According to an embodiment, the communication module 190 may include a wireless communication module 192 (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module 194 (e.g., a local area network (LAN) communication module or a power line communication (PLC) module). A corresponding one of these communication modules may communicate via a first network 198 (e.g., a short-range communication network such as Bluetooth TM The wireless communication module 192 may communicate with the external electronic device 104 through a wireless communication network (e.g., a wireless Fidelity (Wi-Fi) Direct connection, or an Infrared Data Association (IrDA)) or a second network 199 (e.g., a long-distance communication network such as a traditional cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or a wide area network (WAN))). These various types of communication modules may be implemented as a single component (e.g., a single chip), or may be implemented as multiple components (e.g., multiple chips) separated from each other. The wireless communication module 192 may identify and authenticate the electronic device 101 in a communication network (such as the first network 198 or the second network 199) using user information (e.g., an International Mobile Subscriber Identity (IMSI)) stored in the SIM 196.

[0025] The wireless communication module 192 can support 5G networks after 4G networks and next-generation communication technologies (e.g., new radio (NR) access technologies). NR access technologies can support enhanced mobile broadband (eMBB), massive machine type communications (mMTC), or ultra-reliable low-latency communications (URLLC). The wireless communication module 192 can support high-frequency bands (e.g., millimeter wave bands) to achieve, for example, high data transmission rates. The wireless communication module 192 can support various technologies for ensuring performance on high-frequency bands, such as, for example, beamforming, massive multiple-input multiple-output (massive MIMO), full-dimensional MIMO (FD-MIMO), array antennas, analog beamforming, or massive antennas. The wireless communication module 192 can support various requirements specified in the electronic device 101, an external electronic device (e.g., an external electronic device 104), or a network system (e.g., a second network 199). According to an embodiment, the wireless communication module 192 may support a peak data rate for implementing eMBB (e.g., 20 Gbps or greater), loss coverage for implementing mMTC (e.g., 164 dB or less), or U-plane latency for implementing URLLC (e.g., 0.5 ms or less for each of the downlink (DL) and uplink (UL), or 1 ms or less for a round trip).

[0026] The antenna module 197 may transmit or receive a signal or power to or from the outside of the electronic device 101 (e.g., an external electronic device). According to an embodiment, the antenna module 197 may include an antenna including a radiating element formed of a conductive material or a conductive pattern formed in or on a substrate (e.g., a printed circuit board (PCB)). According to an embodiment, the antenna module 197 may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication scheme used in a communication network (such as the first network 198 or the second network 199) may be selected from the plurality of antennas by, for example, the communication module 190. A signal or power may be transmitted or received between the communication module 190 and the external electronic device via at least one selected antenna. According to an embodiment, another component (e.g., a radio frequency integrated circuit (RFIC)) other than the radiating element may be additionally formed as a part of the antenna module 197.

[0027] According to an embodiment, the antenna module 197 may form a millimeter wave antenna module. According to an embodiment, the millimeter wave antenna module may include a printed circuit board, an RFIC, and a plurality of antennas (e.g., array antennas), wherein the RFIC is disposed on a first surface (e.g., bottom surface) of the printed circuit board, or is adjacent to the first surface and can support a specified high frequency band (e.g., millimeter wave band), and the plurality of antennas are disposed on a second surface (e.g., top surface or side surface) of the printed circuit board, or is adjacent to the second surface and can transmit or receive signals of the specified high frequency band.

[0028] At least some of the above components may be coupled to each other via an inter-peripheral communication scheme (e.g., a bus, a general purpose input output (GPIO), a serial peripheral interface (SPI), or a mobile industry processor interface (MIPI)) and communicatively transmit signals (e.g., commands or data) therebetween.

[0029] According to an example embodiment, a command or data may be sent or received between the electronic device 101 and the external electronic device 104 via the server 108 coupled to the second network 199. Each of the external electronic devices (e.g., the external electronic device 102 or the external electronic device 104) may be a device of the same type as the electronic device 101, or a device of a different type from the electronic device 101. According to an embodiment, all or some operations to be executed by the electronic device 101 may be executed in one or more of the external electronic devices (e.g., the external electronic device 102 and the external electronic device 104) and the server 108. For example, if the electronic device 101 needs to automatically execute a function or service or should execute a function or service in response to a request from a user or another device, the electronic device 101 may request the one or more external electronic devices to execute at least part of the function or service without executing the function or service, or the electronic device 101 may request the one or more external electronic devices to execute at least part of the function or service in addition to executing the function or service. The one or more external electronic devices that receive the request may execute at least part of the requested function or service, or execute another function or another service related to the request, and may transmit the result of the execution to the electronic device 101. The electronic device 101 may provide the result as at least a partial reply to the request with or without further processing the result. To this end, cloud computing technology, distributed computing technology, mobile edge computing (MEC) technology, or client-server computing technology, for example, may be used. The electronic device 101 may use, for example, distributed computing or mobile edge computing to provide ultra-low latency services. In another embodiment, the external electronic device 104 may include an Internet of Things (IoT) device. The server 108 may be an intelligent server using machine learning and / or neural networks. According to an embodiment, the external electronic device 104 or the server 108 may be included in the second network 199. The electronic device 101 may be applied to intelligent services (e.g., smart homes, smart cities, smart cars, or healthcare) based on 5G communication technology or IoT-related technologies.

[0030] Reference Figure 2 According to an embodiment, the integrated intelligent system 20 may include an electronic device 201 (eg, the electronic device 101), an intelligent server 200 (eg, the server 108), and a service server 300 (eg, Figure 1 server 108).

[0031] The electronic device 201 may be a terminal device (or electronic device) connectable to the Internet, and may be, for example, a mobile phone, a smart phone, a personal digital assistant (PDA), a notebook computer, a TV, a white appliance, a wearable device, a head mounted display (HMD), or a smart speaker.

[0032] According to the illustrated embodiment, the electronic device 201 may include a communication interface 202 (e.g., interface 177), a microphone 206 (e.g., input module 150), a speaker 205 (e.g., sound output module 155), a display module 204 (e.g., display module 160), a memory 207 (e.g., memory 130), or a processor 203 (e.g., processor 120). The components listed above may be operatively connected or electrically connected to each other.

[0033] The communication interface 202 can be connected to an external device and is configured to send and receive data to and from the external device. The microphone 206 can receive sound (e.g., user speech) and convert the sound into an electrical signal. The speaker 205 can output the electrical signal as sound (e.g., voice).

[0034] The display module 204 may be configured to display images or videos. The display module 204 may also display a graphical user interface (GUI) of a running app (or application). The display module 204 may receive touch input via a touch sensor. For example, the display module 204 may receive text input via a touch sensor in an on-screen keyboard area displayed in the display module 204.

[0035] The memory 207 may store a client module 209, a software development kit (SDK) 208, and a plurality of apps 211. The client module 209 and the SDK 208 may configure a framework (or solution program) for performing general functions. In addition, the client module 209 or the SDK 208 may configure a framework for processing user input (e.g., voice input, text input, or touch input).

[0036] The plurality of apps 211 stored in the memory 207 may be programs for executing designated functions. The plurality of apps 211 may include, for example, a first app 211A, a second app 211B, up to and including an Nth app 211C. Each of the plurality of apps 211 may include a plurality of actions for executing designated functions. For example, the apps may include an alarm clock app, a messaging app, and / or a calendaring app. The plurality of apps 211 may be executed by the processor 203 to sequentially execute at least a portion of the plurality of actions.

[0037] The processor 203 may control the overall operation of the electronic device 201. For example, the processor 203 may be electrically connected to the communication interface 202, the microphone 206, the speaker 205, and the display module 204 to perform a designated operation.

[0038] The processor 203 may also execute a designated function by running a program stored in the memory 207. For example, the processor 203 may run at least one of the client module 209 or the SDK 208 to perform the following operations in order to process user input. The processor 203 may control the operations of the plurality of apps 211 through, for example, the SDK 208. The following operations as operations of the client module 209 or the SDK 208 may be performed by the processor 203.

[0039] The client module 209 may receive user input. For example, the client module 209 may receive a voice signal corresponding to a user's speech sensed by the microphone 206. As another example, the client module 209 may receive a touch input sensed by the display module 204. As yet another example, the client module 209 may receive a text input sensed by a keyboard or an on-screen keyboard. In addition, the client module 209 may receive various types of user input sensed by an input module included in the electronic device 201 or an input module connected to the electronic device 201. The client module 209 may send the received user input to the intelligent server 200. The client module 209 may send the status information of the electronic device 201 together with the received user input to the intelligent server 200. The status information may be, for example, the running status information of the app.

[0040] The client module 209 may receive a result corresponding to the received user input. For example, when the intelligent server 200 is able to calculate a result corresponding to the received user input, the client module 209 may receive a result corresponding to the received user input. The client module 209 may display the received result on the display module 204. In addition, the client module 209 may output the received result in audio form through the speaker 205.

[0041] The client module 209 may receive a plan corresponding to the received user input. The client module 209 may display the results of executing the multiple actions of the app according to the plan on the display module 204. For example, the client module 209 may sequentially display the results of executing the multiple actions on the display module 204 and output the results in audio form through the speaker 205. As another example, the electronic device 201 may display only part of the results of executing the multiple actions (e.g., the result of the last action) on the display module 204 and output part of the results in audio form through the speaker 205.

[0042] According to an embodiment, the client module 209 may receive a request for obtaining information necessary to calculate a result corresponding to the user input from the smart server 200. According to an embodiment, the client module 209 may send necessary information to the smart server 200 in response to the request.

[0043] The client module 209 may send information about the results of running the multiple actions according to the plan to the smart server 200. The smart server 200 may use the information about the results to confirm that the received user input has been processed correctly.

[0044] The client module 209 may include a voice recognition module. According to an embodiment, the client module 209 may recognize a voice input for executing a limited function through the voice recognition module. For example, the client module 209 may run a smart app for processing the voice input to execute an organic operation through a specified input (e.g., wake up!).

[0045] The intelligent server 200 may receive information related to the user voice input from the electronic device 201 through the communication network. According to an embodiment, the intelligent server 200 may change the data related to the received voice input into text data. According to an embodiment, the intelligent server 200 may generate a plan for executing a task corresponding to the user voice input based on the text data.

[0046] According to an embodiment, the plan may be generated by an artificial intelligence (AI) system. The artificial intelligence system may be a rule-based system or a neural network-based system (e.g., a feedforward neural network (FNN) or a recurrent neural network (RNN)). Alternatively, the artificial intelligence system may be a combination thereof or other artificial intelligence systems. According to an embodiment, the plan may be selected from a set of predefined plans or may be generated in real time in response to a user request. For example, the AI ​​system may select at least one plan from among the predefined plans.

[0047] The intelligent server 200 may send the result according to the generated plan to the electronic device 201 or send the generated plan to the electronic device 201. According to an embodiment, the electronic device 201 may display the result according to the plan on the display module 204. According to an embodiment, the electronic device 201 may display the result of executing the action according to the plan on the display module 204.

[0048] The intelligent server 200 may include a front end 210 , a natural language platform 220 , a capsule database (DB) 230 , an execution engine 240 , an end-user interface 250 , a management platform 260 , a big data platform 270 , or an analysis platform 280 .

[0049] The front end 210 may receive the received user input from the electronic device 201. The front end 210 may send a response corresponding to the user input.

[0050] According to example embodiments, the natural language platform 220 may include an automatic speech recognition (ASR) module 221 , a natural language understanding (NLU) module 223 , a planner module 225 , a natural language generator (NLG) module 227 , or a text to speech (TTS) module 229 .

[0051] The ASR module 221 may convert data related to the voice input received from the electronic device 201 into text data. The NLU module 223 may use the text data of the voice input to discern the user's intention. For example, the NLU module 223 may discern the user's intention by performing syntactic analysis or semantic analysis on the user input in the form of text data. The NLU module 223 may discern the meaning of words extracted from the user input using linguistic features (e.g., grammatical elements) of morphemes or phrases, and determine the user's intention by matching the meaning of the discerned words with the intention. The NLU module 223 may obtain intention information corresponding to the user's utterance. The intention information may be information indicating the user's intention determined by analyzing the text data. The intention information may include information indicating an action or function that the user intends to perform using the device.

[0052] The planner module 225 may generate a plan using parameters and the intent determined by the NLU module 223. According to an embodiment, the planner module 225 may determine multiple domains required to perform a task based on the determined intent. The planner module 225 may determine multiple actions included in each of the multiple domains determined based on the intent. According to an embodiment, the planner module 225 may determine parameters required to run the multiple actions determined or result values ​​output by running the multiple actions. Parameters and result values ​​may be defined as concepts of a specified form (or class). Therefore, a plan may include multiple actions and multiple concepts determined by the user's intent. The planner module 225 may determine the relationship between multiple actions and multiple concepts step by step (or hierarchically). For example, the planner module 225 may determine the running order of multiple actions determined based on the user's intent based on multiple concepts. In other words, the planner module 225 may determine the running order of multiple actions based on the parameters required to run the multiple actions and the results output by running the multiple actions. Therefore, the planner module 225 may generate a plan including connection information (e.g., ontology) about the connection between multiple actions and multiple concepts. The planner module 225 may generate a plan using information stored in the capsule DB 230 storing a set of relationships between concepts and actions.

[0053] The NLG module 227 may change the designated information into a text form. The information changed into a text form may be in the form of a natural language speech. The TTS module 229 may change the information in the text form into the information in the voice form.

[0054] According to an embodiment, some or all of the functions of the natural language platform 220 may also be implemented in the electronic device 201 .

[0055] The capsule DB 230 may store information about the relationship between a plurality of concepts and actions corresponding to a plurality of domains. A capsule according to an embodiment may include a plurality of action objects (or action information) and a concept object (or concept information) included in a plan. According to an embodiment, the capsule DB 230 may store a plurality of capsules in the form of a concept action network (CAN). According to an embodiment, a plurality of capsules may be stored in a function registry included in the capsule DB 230.

[0056] The capsule DB 230 may include a policy registry storing policy information necessary for determining a plan corresponding to the voice input. The policy information may include reference information for determining one plan when there are multiple plans corresponding to the user input. According to an embodiment, the capsule DB 230 may include a subsequent registry storing information about subsequent actions for suggesting subsequent actions to the user in a specified situation. The subsequent actions may include, for example, subsequent utterances. According to an embodiment, the capsule DB 230 may include a layout registry storing layout information of information output by the electronic device 201. According to an embodiment, the capsule DB 230 may include a vocabulary registry storing vocabulary information included in the capsule information. According to an embodiment, the capsule DB 230 may include a dialogue registry storing information about a dialogue (or interaction) with the user. The capsule DB 230 may update the stored objects through a developer tool. The developer tool may include, for example, a function editor for updating an action object or a concept object. The developer tool may include a vocabulary editor for updating vocabulary. The developer tool may include a policy editor for generating and registering a policy for determining a plan. The developer tool may include a dialogue editor for generating a dialogue with the user. The developer tool may include a subsequent editor for activating a subsequent target and editing a subsequent utterance providing a prompt. The subsequent target may be determined based on a currently set target, a user's preference, or an environmental condition. In an embodiment, the capsule DB 230 may also be implemented in the electronic device 201 .

[0057] The execution engine 240 can calculate the result using the generated plan. The end-user interface 250 can send the calculated result to the electronic device 201. Therefore, the electronic device 201 can receive the result and provide the received result to the user. The management platform 260 can manage the information used by the intelligent server 200. The big data platform 270 can collect the user's data. The analysis platform 280 can manage the quality of service (QoS) of the intelligent server 200. For example, the analysis platform 280 can manage the components and processing rate (or efficiency) of the intelligent server 200.

[0058] The service server 300 may provide a designated service (e.g., ordering food or booking a hotel) to the electronic device 201. According to an embodiment, the service server 300 may be a server operated by a third party. The service server 300 may provide the smart server 200 with information to be used for generating a plan corresponding to the received user input. The provided information may be stored in the capsule DB 230. In addition, the service server 300 may provide the smart server 200 with result information according to the plan.

[0059] In the above integrated intelligent system 20, the electronic device 201 can provide various intelligent services to the user in response to user input. The user input may include, for example, input through a physical button, touch input, or voice input.

[0060] In an embodiment, the electronic device 201 can provide a voice recognition service through a smart app (or voice recognition app) stored therein. In this case, for example, the electronic device 201 can recognize a user's speech or voice input received through a microphone, and provide a service corresponding to the recognized voice input to the user.

[0061] In an embodiment, the electronic device 201 can perform a specified action based on the received voice input alone or together with the intelligent server and / or the service server. For example, the electronic device 201 can run an app corresponding to the received voice input and perform a specified action through the running app.

[0062] In an embodiment, when the electronic device 201 provides a service together with the intelligent server 200 and / or the service server 300, the electronic device 201 can use the microphone 206 to detect the user's speech and generate a signal (or voice data) corresponding to the detected user's speech. The electronic device 201 can use the communication interface 202 to send the voice data to the intelligent server 200.

[0063] In response to the voice input received from the electronic device 201, the intelligent server 200 can generate a plan for performing a task corresponding to the voice input or a result of performing an action according to the plan. The plan may include, for example, multiple actions for performing tasks corresponding to the user's voice input and multiple concepts related to the multiple actions. The concept may define parameters input to the operation of multiple actions or result values ​​output by running multiple actions. The plan may include connection information between multiple actions and multiple concepts.

[0064] The electronic device 201 may receive the response using the communication interface 202. The electronic device 201 may output a voice signal generated internally by the electronic device 201 to the outside using the speaker 205 or output an image generated internally by the electronic device 201 to the outside using the display module 204.

[0065] Figure 3 is a diagram illustrating storing relationship information between concepts and actions in a database according to various exemplary embodiments.

[0066] The capsule DB (eg, capsule DB 230) of the smart server (eg, smart server 200) may store capsules in the form of CAN 400. The capsule DB may store an action for processing a task corresponding to a user's voice input and parameters required for the action in the form of CAN.

[0067] The capsule DB may store a plurality of capsules (capsule A 401 and capsule B 404) corresponding to a plurality of domains (e.g., applications), respectively. According to an embodiment, one capsule (e.g., capsule A 401) may correspond to one domain (e.g., location (geographic location) or application). In addition, one capsule may correspond to at least one service provider (e.g., CP-1 402 or CP-2 403) for performing a function for a domain related to the capsule. According to an embodiment, one capsule may include at least one action 410 and at least one concept 420 for performing a specified function.

[0068] The natural language platform (e.g., natural language platform 220) can use the capsules stored in the capsule DB to generate a plan for performing a task corresponding to the received voice input. For example, the planner module (e.g., planner module 225) of the natural language platform can use the capsules stored in the capsule DB to generate a plan. For example, the action 4011 and 4013 and the concepts 4012 and 4014 of capsule A 401 and the action 4041 and the concept 4042 of capsule B 404 can be used to generate a plan 407.

[0069] Figure 4is a diagram illustrating a screen of an electronic device processing a received voice input through a smart app according to various exemplary embodiments.

[0070] The electronic device 201 may run a smart app to process user input through a smart server (eg, smart server 200 ).

[0071] According to an embodiment, on the screen 310, when a specified voice input (e.g., wake up!) is recognized or an input through a hardware key (e.g., a dedicated hardware key) is received, the electronic device 201 may run a smart app for processing the voice input. The electronic device 201 may, for example, run the smart app in a state where a schedule app is running. According to an embodiment, the electronic device 201 may display an object (e.g., an icon) 311 corresponding to the smart app on a display module 204 (e.g., a display module 160 or a display module 204). According to an embodiment, the electronic device 201 may receive a voice input through a user's speech. For example, the electronic device 201 may receive a voice input of "Tell me about this week's schedule!" According to an embodiment, the electronic device 201 may display a user interface (UI) 313 (e.g., an input window) of the smart app, wherein text data of the received voice input is displayed on the display module 204.

[0072] According to an embodiment, on screen 320, the electronic device 201 may display a result corresponding to the received voice input on the display module 204. For example, the electronic device 201 may receive a plan corresponding to the received user input and display “this week’s schedule” on the display module 204 according to the plan.

[0073] Figure 5 is a diagram illustrating speech recognition performed by an intelligent server according to an embodiment.

[0074] Reference Figure 5 In an embodiment, the electronic device 501 may include a reference Figure 1 The electronic device 101 described and the reference Figure 2 The smart server 502 may include at least a portion of the components of the electronic device 201 described herein. Figure 2 At least a portion of the components of the smart server 200 described herein. Figures 1 to 4 The duplicate description provided.

[0075] The electronic device 501 (e.g., the electronic device 101 or the electronic device 201) may be connected to the intelligent server 502 (e.g., the intelligent server 200) via a LAN, a WAN, a value-added network (VAN), a mobile wireless communication network, a satellite communication network, or any combination thereof. The electronic device 501 and the intelligent server 502 may communicate with each other through a wired communication method or a wireless communication method (e.g., wireless LAN (Wi-Fi), Bluetooth, Bluetooth low energy, ZigBee, Wi-Fi Direct (WFD), ultra-wideband (UWB), infrared data association (IrDA), and near field communication (NFC)).

[0076] In an embodiment, the electronic device 501 may be implemented as at least one of a smart phone, a tablet personal computer (PC), a mobile phone, a speaker (e.g., an artificial intelligence (AI) speaker), a video phone, an e-book reader, a desktop PC, a laptop PC, a netbook computer, a workstation, a server, a personal digital assistant (PDA), a portable multimedia player (PMP), an MP3 player, a mobile medical device, a camera, or a wearable device.

[0077] In an embodiment, the electronic device 501 may obtain a voice signal corresponding to the user's speech and send the voice signal to the intelligent server 502. The intelligent server 502 may obtain text data corresponding to the user's speech based on the voice signal. The text data may be obtained by performing ASR on the voice signal to convert the voice portion into a computer-readable text. The intelligent server 502 may use the text data to analyze the user's speech. The intelligent server 502 may use the analysis results (e.g., intent information, entities, and / or capsules) to provide a response (e.g., a question or answer) to be provided to the user to the device (e.g., the electronic device 501). The intelligent server 502 may be implemented as software. Part or all of the intelligent server 502 may be implemented in the electronic device 501 and / or the intelligent server 502 (e.g., the intelligent server 200). An on-device AI for processing speech without communicating with the intelligent server 502 may be installed in the electronic device 501. Reference may be implemented in the electronic device 501 Figures 2 to 4 The natural language platform 220 described above, etc.

[0078] According to an embodiment, the intelligent server 502 may perform speaker-adaptive speech recognition. The intelligent server 502 may register a speaker in a speech recognition service. The intelligent server 502 may store personalized data (e.g., voice) of the registered speaker. When a voice signal of a registered speaker is input when performing speaker verification, the intelligent server 502 may determine that the subject of the input voice signal is the registered speaker. The intelligent server 502 may perform speaker-adaptive speech recognition based on a sequence summary neural network (SSN) and an adaptation layer. The SSN and the adaptation layer may be trained based on the personalized data (e.g., voice) of the registered speaker. The intelligent server 502 may store the SSN and the adaptation layer trained for each registered speaker. When the subject of the voice input is a registered speaker, the intelligent server 502 may perform speaker-adaptive speech recognition using the corresponding SSN and the adaptation layer.

[0079] Reference Figure 5 In an embodiment, the electronic device 501 may receive an utterance (e.g., “Call Hong Gil-dong”). The utterance (e.g., “Call Hong Gil-dong”) may include words that require speaker adaptation (e.g., Hong Gil-dong). The words included in the utterance (e.g., Hong Gil-dong) may be easily mistaken for words that are more frequently used by the public (e.g., Hong Gil-dong). The electronic device 501 may send a voice signal corresponding to the utterance to the intelligent server 502. The intelligent server 502 may perform speaker verification to determine whether the subject of the utterance is a registered speaker. When the subject of the utterance is a registered speaker, the intelligent server 502 may accurately recognize the utterance (e.g., “Call Hong Gil-dong”) based on the SSN and adaptation layer corresponding to the registered speaker (e.g., the SSN and adaptation layer trained based on the personalized data of the registered speaker). The intelligent server 502 may perform speaker adaptive speech recognition, thereby guiding the electronic device 501 to provide a correct response to the user (e.g., a response of saying “Calling Hong Gil-dong”).

[0080] According to an embodiment, even when input data different from the training data of the ASR system (e.g., the natural language platform 220 or the ASR module 221) is input, the intelligent server 502 can accurately perform speech recognition. The intelligent server 502 can perform speech recognition without modifying the ASR system (e.g., the ASR module 221) by selectively using the SSN and the adaptation layer for speaker adaptation. When the subject of the voice input is a registered speaker, the intelligent server 502 can obtain the SSN adaptation feature based on the SSN corresponding to the registered speaker. SSN reference Figure 6bDescribed in detail. When the subject of the voice input is a registered speaker, the intelligent server 502 may selectively add an adaptation layer corresponding to the registered speaker without modifying the ASR model of the ASR module 221 (e.g., a transformer-based ASR model, a conformer-based ASR model, or an RNN-transducer-based ASR model). The common parameters of the ASR model of the ASR module 221 (e.g., a transformer-based ASR model, a conformer-based ASR model, or an RNN-transducer-based ASR model) may not be modified, and only the adaptation parameters corresponding to the registered speaker may be selectively added. Adaptation parameter reference Figure 6c The intelligent server 502 may perform speech recognition according to the SSN adaptation feature based on the ASR model to which the adaptation parameters corresponding to the registered speakers are added.

[0081] In an embodiment, a part or all of the operations performed by the electronic device 501 may be performed by the smart server 502. Hereinafter, a description is provided based on the assumption that the electronic device 501 performs the operations.

[0082] Figure 6a is a schematic block diagram of an electronic device according to an embodiment, Figure 6b is a schematic block diagram of a SSN according to an embodiment, and Figure 6c is a schematic block diagram of an adaptation layer according to an embodiment.

[0083] Reference Figure 6a In an embodiment, the electronic device 501 may include a reference Figure 1 The electronic device 101 described and the reference Figure 2 At least some components of the electronic device 201 described above. As described above, an on-device AI for processing speech without communicating with an intelligent server (e.g., intelligent server 200 or intelligent server 502) can be installed in the electronic device 501. Figures 2 to 4 The natural language platform 220 described herein. Figures 1 to 4 The electronic device 501 may include a processor 520 (e.g., the processor 120 or the processor 203) and a memory 530 (e.g., the memory 130 or the memory 207) electrically connected to the processor 520. The processor 520 (e.g., the application processor) may execute instructions by accessing the memory 530. The processor 520 may perform an operation for providing a response to a user. The memory 530 may store various data used by at least one component of the electronic device 501 (e.g., the processor 520).

[0084] In an embodiment, operations 521 to 527 may be performed by the processor 520 of the electronic device 501. In the following examples, the operations may be performed sequentially, but not necessarily sequentially. For example, the operations may be performed in different orders, and at least two of the operations may be performed in parallel.

[0085] In operation 521, the processor 520 may perform partial speech recognition. Partial speech recognition may be performed for real-time speech recognition. For example, for a voice input of about 1200 milliseconds (ms), the processor 520 may sequentially output speech recognition results for accumulated voice input (e.g., voice input accumulated for about 400 ms).

[0086] In operation 522, the processor 520 may perform speaker verification. The speaker verification may be performed on the entire voice input (eg, “call Hong Gil-dong”). The processor 520 may determine whether the subject of the voice input (eg, “call Hong Gil-dong”) is a registered speaker by performing speaker verification.

[0087] In operation 523, when the subject of the voice input (eg, "call Hong Gil-dong") is an unregistered speaker, the processor 520 may obtain a voice feature for the first ASR. The speech input can be divided into time windows of approximately 25 ms and transformed into a log-Mel filter bank. The time windows can overlap each other by approximately 10 ms. Speech Features It can be a set of logarithmic Mel filter banks into which the entire speech input is transformed. In addition, the speech feature It can be a normalized feature (for example, a feature normalized by spectral mean and variance normalization (CMVN)).

[0088] In operation 524, the processor 520 may generate a signal based on the first ASR model according to the voice characteristics. Perform speech recognition. The first ASR model may be a transformer-based ASR model, a conformer-based ASR model, and / or an RNN-transducer-based ASR model. The transcription network of the RNN-transducer-based ASR model may include a transformer, a conformer, or a long short-term memory (LSTM) encoder. The first ASR model may be an end-to-end model. The processor 520 may extract the speech features from the speech features based on the first ASR model. Text data corresponding to a voice input (eg, a voice signal corresponding to an utterance) is generated.

[0089] In operation 525, when the subject of the voice input (eg, "call Hong Gil-dong") is a registered speaker, the processor 520 may obtain the voice features for the second ASR. The voice features obtained in operation 523 and the voice features obtained in operation 525 Can be substantially the same type of data.

[0090] In operation 526, the processor 520 may obtain the SSN adaptation feature . SSN Adaptation Features It can be obtained from a SSN (e.g., SSN 601) trained based on data (e.g., speech) of registered speakers. Figure 6b , shows a schematic block diagram of SSN 601. SSN 601 may be a simple neural network designed to replace i-vector. SSN 601 may receive speech features And accumulate the values ​​passed through the feedforward layer (e.g., fully connected (FC) layer). SSN 601 can calculate the average of the accumulated values ​​to obtain the auxiliary features . SSN 601 can be auxiliary characteristics Projection to speech features The SSN 601 can be added by adding the projection feature and voice characteristics To output SSN adaptation features . Get SSN adaptation characteristics The operation can be expressed by formula 1.

[0091] [Formula 1]

[0092] Formula 1 shown above is an example for better understanding, is not limited thereto, and may be modified, applied, or expanded in various ways.

[0093] In formula 1, Can represent FC layer. Can be replaced by an adaptation layer to reduce the number of parameters. Can indicate the context of voice input. Information about the speaker may be included. Processor 520 may perform speaker-adaptive speech recognition by selectively using SSNs (eg, SSN 601) corresponding to registered speakers.

[0094] In operation 527, the processor 520 may adapt the feature according to the SSN based on the second ASR model. Perform speech recognition. The second ASR model can be configured by selectively adding an adaptation layer 602 configured for personalization of the registered speaker to the first ASR model (eg, a transformer-based ASR model, a conformer-based ASR model, or an RNN-transducer-based ASR model). Figure 6c , the adaptation layer 602 may have a bottleneck structure for limiting the number of parameters. The adaptation layer 602 may project d-dimensional features into m-dimensions as lower dimensions. The adaptation layer 602 may apply nonlinearity to the m-dimensional features, and then project the m-dimensional features to which nonlinearity is applied back to the d-dimensions. The total number of parameters added for each adaptation layer (e.g., the adaptation layer 602) may be 2md + d + m. The adaptation layer may be added to an encoder (e.g., an encoder of a transformer, an encoder of a conformer, or an LSTM encoder). For example, an adaptation layer may be added between a feedforward layer and a nonlinear layer (e.g., ReLU) of an encoder of a transformer. When the subject of the voice input is a registered speaker, the processor 520 may perform speaker-adaptive speech recognition by selectively adding an adaptation layer corresponding to the registered speaker.

[0095] Figure 7 is a diagram illustrating a method of training an SSN and an adaptation layer according to an embodiment.

[0096] According to an embodiment, operations 701 to 713 may be interpreted as being performed by a processor 520 (e.g., processor 120 or processor 203) of an intelligent server 502 (e.g., intelligent server 200 or intelligent server 502) or an electronic device 501 (e.g., electronic device 101 or electronic device 201). In the following examples, the operations may be performed sequentially, but not necessarily sequentially. For example, the operations may be performed in different orders, and at least two of the operations may be performed in parallel.

[0097] In operation 701 and operation 703, the intelligent server 502 may collect a voice signal. In operation 701, the voice signal may be a voice signal having a signal-to-noise ratio (SNR) greater than about 10 decibels (dB). In operation 703, the voice signal may be a voice signal of a speaker registered through speaker verification. The processor 520 may collect a voice signal when a user who agrees to voice collection makes a call. When the user initializes the electronic device 501, the processor 520 may collect a voice signal of the user reading a previously prepared script.

[0098] In operations 705 to 707 , the intelligent server 502 may collect voice signals until a sufficient amount (eg, approximately 10 minutes) of voice signals are collected.

[0099] In operation 709 , the intelligent server 502 may generate transcription data based on the collected voice signals.

[0100] In operation 711, the intelligent server 502 may train the SSN and the adaptation layer based on the transcription data. The trained SSN may have a size of approximately 0.75 MiB (e.g., single precision, FC1{80 x 512}, FC2{512 x 256}, Projection{256 x 80}). When the SSN is changed to an adapter structure, the size of the SSN may be reduced to approximately 144 KiB (e.g., an adapter having a structure of single precision, projection1{80 x 128}, FC1{128 x 64}, FC2{64 x 128}, projection2{128 x 80}). The total size of the trained adaptation layer excluding the SSN may be approximately 1728 KiB (e.g., approximately 144 KiB per adaptation layer, where "2" adaptation layers are added for each self-attention layer and a total of "6" self-attention layers are included in the encoder of the transformer). The intelligent server 502 can obtain the trained SSN and the trained adaptation layer (or trained adaptation parameters).

[0101] In operation 713 , the intelligent server 502 may store the trained SSN and the trained adaptation layer corresponding to each registered speaker in a user database (DB).

[0102] Figure 8 is a diagram illustrating a result of voice recognition performed by an electronic device according to an embodiment.

[0103] Reference Figure 8 In case 801, the electronic device 800 may receive an utterance (e.g., "Calling Hong Gil-dong") including a word (e.g., Hong Gil-dong) that requires speaker adaptation. The electronic device 800 may be a device that is not speaker adapted. The electronic device 800 may misunderstand the word (e.g., Hong Gil-dong) included in the utterance as a word (e.g., Hong Gil-dong) that is more frequently used by the public and provide an inappropriate response to the user (e.g., calling Hong Gil-dong).

[0104] In situation 802, the electronic device 501 (e.g., electronic device 101 or electronic device 201) may receive an utterance (e.g., "Call Hong Gil-dong"). The utterance (e.g., "Call Hong Gil-dong") may include words that require speaker adaptation (e.g., Hong Gil-dong). The words included in the utterance (e.g., Hong Gil-dong) may be easily mistaken for words that are more frequently used by the public (e.g., Hong Gil-dong). The electronic device 501 may perform speaker verification to determine whether the subject of the utterance is a registered speaker. When the subject of the utterance is a registered speaker, the electronic device 501 may accurately recognize the utterance (e.g., "Call Hong Gil-dong") based on the SSN and adaptation layer corresponding to the registered speaker (e.g., the SSN and adaptation layer trained based on the personalized data of the registered speaker). The electronic device 501 may provide the user with a correct response (e.g., calling Hong Gil-dong) by performing speaker adaptive speech recognition.

[0105] Fig. 9 is a flowchart illustrating an operation method of an intelligent server according to an embodiment.

[0106] Operations 910 to 930 may be performed sequentially, but are not necessarily performed sequentially. For example, operations 910 to 930 may be performed in a different order, and at least two operations thereof may be performed in parallel.

[0107] In operation 910 , an intelligent server (eg, intelligent server 200 or intelligent server 502 ) may perform speaker verification on the voice input.

[0108] In operation 920, when the voice input does not match the voice of the registered speaker, the intelligent server 502 may perform a first speech recognition on the voice input based on the first ASR model.

[0109] In operation 930, when the voice input matches the voice of the registered speaker, the intelligent server 502 may perform a second speech recognition on the voice input based on the SSN and the second ASR model.

[0110] According to an electronic device (eg, Figure 5 and the electronic device 101, the electronic device 201 or the electronic device 501 of FIG. 6) may include a memory having instructions (eg, the memory 130, Figure 2The processor may include a memory 207 or memory 530 (e.g., a processor 120, a processor 203, or a processor 520) electrically connected to the memory and configured to execute the instructions. When the instructions are executed by the processor, the processor may perform speaker verification on the voice input. When the voice input does not match the voice of the registered speaker, the processor may perform a first speech recognition on the voice input based on the first ASR model. When the voice input matches the voice of the registered speaker, the processor may perform a second speech recognition on the voice input based on the SSN and the second ASR model.

[0111] According to an embodiment, the second ASR model may be configured by selectively adding an adaptation layer configured for personalization of a registered speaker to the first ASR model.

[0112] According to an embodiment, the processor may obtain a voice feature based on the voice input. The processor may perform a first speech recognition according to the voice feature based on the first ASR model.

[0113] According to an embodiment, the processor may obtain a voice feature based on the voice input. The processor may obtain an SSN adaptation feature from the voice feature based on the SSN. The processor may perform a second speech recognition based on the SSN adaptation feature based on the second ASR model.

[0114] According to an embodiment, the first ASR model may include any one of a transformer-based ASR model, a conformer-based ASR model, or an RNN-transducer-based ASR model.

[0115] According to an embodiment, the processor may perform partial speech recognition on the sequentially input voice input based on the first ASR model. When the voice input is fully input, the processor may perform speaker verification.

[0116] According to an embodiment, the SSN and the adaptation layer may be trained based on transcription data obtained by decoding a user's voice through speaker verification.

[0117] According to an embodiment, an intelligent server (e.g., intelligent server 200 or intelligent server 502) may include a memory having instructions and a processor electrically connected to the memory and configured to execute the instructions. When the instructions are executed by the processor, the processor may perform speaker verification on the voice input. When the voice input does not match the voice of the registered speaker, the processor may perform a first speech recognition on the voice input based on the first ASR model. When the voice input matches the voice of the registered speaker, the processor may perform a second speech recognition on the voice input based on the SSN and the second ASR model.

[0118] According to an embodiment, the second ASR model may be configured by selectively adding an adaptation layer configured for personalization of a registered speaker to the first ASR model.

[0119] According to an embodiment, the processor may obtain a voice feature based on the voice input. The processor may perform a first speech recognition according to the voice feature based on the first ASR model.

[0120] According to an embodiment, the processor may obtain a voice feature based on the voice input. The processor may obtain an SSN adaptation feature from the voice feature based on the SSN. The processor may perform a second speech recognition based on the SSN adaptation feature based on the second ASR model.

[0121] According to an embodiment, the first ASR model may include any one of a transformer-based ASR model, a conformer-based ASR model, or an RNN-transducer-based ASR model.

[0122] According to an embodiment, the processor may perform partial speech recognition on the sequentially input voice input based on the first ASR model. When the voice input is fully input, the processor may perform speaker verification.

[0123] According to an embodiment, the SSN and the adaptation layer may be trained based on transcription data obtained by decoding a user's voice through speaker verification.

[0124] The operation method of the intelligent server (e.g., intelligent server 200 or intelligent server 502) according to the embodiment may include performing speaker verification on the voice input. The operation method of the intelligent server may include performing a first speech recognition on the voice input based on a first ASR model when the voice input does not match the voice of a registered speaker. The operation method of the intelligent server may include performing a second speech recognition on the voice input based on an SSN and a second ASR model when the voice input matches the voice of a registered speaker.

[0125] According to an embodiment, the second ASR model may be configured by selectively adding an adaptation layer configured for personalization of a registered speaker to the first ASR model.

[0126] According to an embodiment, performing the first speech recognition may include obtaining a speech feature based on the speech input. Performing the first speech recognition may include performing the first speech recognition according to the speech feature based on the first ASR model.

[0127] According to an embodiment, performing the second speech recognition may include obtaining a speech feature based on the speech input. Performing the second speech recognition may include obtaining an SSN adaptation feature from the speech feature based on the SSN. Performing the second speech recognition may include performing the second speech recognition according to the SSN adaptation feature based on the second ASR model.

[0128] According to an embodiment, the first ASR model may include any one of a transformer-based ASR model, a conformer-based ASR model, or an RNN-transducer-based ASR model.

[0129] According to an embodiment, the SSN and the adaptation layer may be trained based on transcription data obtained by decoding a user's voice through speaker verification.

[0130] The electronic device according to the embodiment may be one of various types of electronic devices. The electronic device may include, for example, a portable communication device (e.g., a smart phone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a household appliance device. According to an embodiment of the present disclosure, the electronic device is not limited to those electronic devices described above.

[0131] It should be understood that the embodiments of the present disclosure and the terms used therein are not intended to limit the technical features set forth herein to specific embodiments, and include various changes, equivalents or replacements of the corresponding embodiments. Regarding the description of the accompanying drawings, the same reference numerals may be used for similar or related components. It should be understood that, unless the relevant context clearly indicates otherwise, the singular form of the noun corresponding to the project may include one or more things. As used herein, each of "A or B", "at least one of A and B", "at least one of A or B", "A, B or C", "at least one of A, B and C" and "at least one of A, B or C" may include any one of the items listed together in the corresponding phrases in these phrases or all possible combinations thereof. Terms such as "1st", "2nd" or "first" or "second" may be simply used to distinguish components from the other components described without limiting components in other aspects (e.g., importance or order). It should be understood that if an element (e.g., a first element) is referred to as being “coupled with”, “coupled to”, “connected to”, or “connected to” another element (e.g., the second element), with or without the term “operably” or “communicatively”, it means that the element may be coupled to the other element directly (e.g., by wire), wirelessly, or via a third element.

[0132] As used in relation to the embodiments of the present disclosure, the term "module" may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with other terms such as "logic", "logic block", "portion", or "circuit". A module may be a single integral component adapted to perform one or more functions or its smallest unit or portion. For example, according to an embodiment, a module may be implemented in the form of an application specific integrated circuit (ASIC).

[0133] The embodiments as described herein may be implemented as software (e.g., program 1740) including one or more instructions stored in a storage medium (e.g., internal memory 1736 or external memory 1738) readable by a machine (e.g., electronic device 1701). For example, a processor (e.g., processor 1720) of a machine (e.g., electronic device 1701) may call at least one of the one or more instructions stored in the storage medium and may run it. This enables the machine to be operated to perform at least one function according to at least one of the called instructions. The one or more instructions may include code generated by a compiler or code executable by an interpreter. A machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, the term "non-transitory" simply means that the storage medium is a tangible device and does not include a signal (e.g., an electromagnetic wave), but this term does not distinguish between a case where data is semi-permanently stored in a storage medium and a case where data is temporarily stored in a storage medium.

[0134] According to an embodiment, the method according to an embodiment of the present disclosure may be included and provided in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., a compact disc read-only memory (CD-ROM)), or via an application store (e.g., PlayStore). TM ) online distribution (e.g., download or upload), or directly between two user devices (e.g., smartphones). If distributed online, at least a portion of the computer program product may be temporarily generated or at least temporarily stored in a machine-readable storage medium such as a memory of a manufacturer's server, a memory of an application store's server, or a memory of a relay server.

[0135] According to an embodiment, each of the above-mentioned components (e.g., a module or a program) may include a single entity or multiple entities, and some of the multiple entities may be separately arranged on different components. According to an embodiment, one or more of the above-mentioned components may be omitted, or one or more other components may be added. Alternatively or additionally, multiple components (e.g., a module or a program) may be integrated into a single component. In such a case, the integrated components may still perform one or more functions of each of the multiple components in the same or similar manner as the multiple components were performed by their corresponding components before integration. According to an embodiment, the operations performed by a module, a program or another component may be performed sequentially, in parallel, repeatedly or heuristically, or one or more operations may be performed or omitted in a different order, or one or more other operations may be added.

[0136] Although the exemplary embodiments of the present disclosure have been shown and described, the present disclosure is not limited to the aforementioned specific embodiments, and it is clear that various modifications can be made by a person skilled in the art of the present disclosure without departing from the gist of the present disclosure as claimed in the appended claims and their equivalents. In addition, such modifications are not intended to be interpreted independently of the technical ideas or prospects of the present disclosure.

Claims

1. An electronic device (101; 201; 501), the electronic device (101; 201; 501) comprises: a memory (130; 207; 530), the memory (130; 207; 530) storing at least one instruction; and At least one processor (120; 203; 520) operatively connected to the memory (130; 207; 530) and configured to execute the at least one instruction to: performing speaker verification on the voice input to determine whether the voice input matches the voice of a registered speaker, Based on determining that the voice input does not match the voice of the registered speaker, performing a first speech recognition on the voice input based on a first automatic speech recognition (ASR) model, and Based on determining that the voice input matches the voice of the registered speaker, a second speech recognition is performed on the voice input based on a sequence summary neural network SSN (601) and a second ASR model.

2. The electronic device (101; 201; 501) according to claim 1, wherein: The second ASR model is configured by selectively adding an adaptation layer (602) configured for personalization of the registered speaker to the first ASR model.

3. The electronic device (101; 201; 501) according to any one of claims 1 and 2, wherein: The at least one processor (120; 203; 520) is further configured to execute the at least one instruction to: obtaining a voice feature based on the voice input, and The first speech recognition is performed according to the voice features based on the first ASR model.

4. The electronic device (101; 201; 501) according to any one of claims 1 to 3, wherein: The at least one processor (120; 203; 520) is further configured to execute the at least one instruction to: obtaining a voice feature based on the voice input, obtaining SSN adaptation features from the voice features based on the SSN (601), and The second speech recognition is performed according to the SSN adaptation feature based on the second ASR model.

5. The electronic device (101; 201; 501) according to any one of claims 1 to 4, wherein: The first ASR model includes one of a transformer-based ASR model, a conformer-based ASR model, or a recurrent neural network (RNN-transducer)-based ASR model.

6. The electronic device (101; 201; 501) according to any one of claims 1 to 5, wherein: The at least one processor (120; 203; 520) is further configured to execute the at least one instruction to: performing partial speech recognition on the sequentially input voice input based on the first ASR model, and Based on the voice input being fully input, the speaker verification is performed.

7. The electronic device (101; 201; 501) according to any one of claims 1 to 6, wherein: The SSN (601) and the adaptation layer (602) are trained based on transcription data obtained by decoding the user's voice through speaker verification.

8. An intelligent server (200; 502), the intelligent server (200; 502) includes: a memory storing at least one instruction; as well as at least one processor operatively connected to the memory and configured to execute the at least one instruction to: performing speaker verification on the voice input to determine whether the voice input matches the voice of a registered speaker, Based on determining that the voice input does not match the voice of the registered speaker, performing a first speech recognition on the voice input based on a first automatic speech recognition (ASR) model, and Based on determining that the voice input matches the voice of the registered speaker, a second speech recognition is performed on the voice input based on a sequence summary neural network SSN (601) and a second ASR model.

9. The intelligent server (200; 502) according to claim 8, wherein: The second ASR model is configured by selectively adding an adaptation layer (602) configured for personalization of the registered speaker to the first ASR model.

10. The intelligent server (200; 502) according to any one of claims 8 and 9, wherein: The at least one processor is further configured to execute the at least one instruction to: obtaining a voice feature based on the voice input, and The first speech recognition is performed according to the voice features based on the first ASR model.

11. The intelligent server (200; 502) according to any one of claims 8 to 10, wherein: The at least one processor is further configured to execute the at least one instruction to: obtaining a voice feature based on the voice input, obtaining SSN adaptation features from the voice features based on the SSN (601), and The second speech recognition is performed according to the SSN adaptation feature based on the second ASR model.

12. The intelligent server (200; 502) according to any one of claims 8 to 11, wherein: The first ASR model includes one of a transformer-based ASR model, a conformer-based ASR model, or a recurrent neural network (RNN-transducer)-based ASR model.

13. The intelligent server (200; 502) according to any one of claims 8 to 12, wherein: The at least one processor is further configured to execute the at least one instruction to: performing partial speech recognition on the sequentially input voice input based on the first ASR model; and Based on the voice input being fully input, the speaker verification is performed.

14. The intelligent server (200; 502) according to any one of claims 8 to 13, wherein: The SSN (601) and the adaptation layer (602) are trained based on transcription data obtained by decoding the user's voice through speaker verification.

15. An operating method of an intelligent server (200; 502), the operating method comprising: performing speaker verification on the voice input to determine whether the voice input matches a registered speaker's voice; Based on determining that the voice input does not match the voice of the registered speaker, performing a first speech recognition on the voice input based on a first automatic speech recognition (ASR) model; as well as Based on determining that the voice input matches the voice of the registered speaker, a second speech recognition is performed on the voice input based on a sequence summary neural network SSN (601) and a second ASR model.

Images