Electronic apparatus for processing user utterance and controlling method thereof

Abstract

An electronic device in one embodiment according to the present disclosure comprises a microphone and at least one processor operatively connected to the microphone, and the at least one processor may include a buffer memory that stores a first feature vector for a first voice signal obtained from the microphone as an inverse value, and a calculation circuit configured to perform a norm operation on the first feature vector and the second feature vector based on a second feature vector based on a second voice signal streamed from the microphone and the inverse value of the first feature vector stored in the buffer memory, or to calculate the similarity between the first feature vector and the second feature vector. In addition to this, various other embodiments identified through the specification are possible.

Description

Technology Field

[0001] Various embodiments relate to a technology that recognizes user voice commands and performs actions corresponding to user voice commands. Background Technology

[0002] An artificial intelligence system (or integrated intelligence system) is a computer system that implements human-level intelligence; it is a system in which machines learn and make judgments autonomously, and its recognition rate improves with use. Artificial intelligence technology consists of machine learning (deep learning) technology, which utilizes algorithms to autonomously classify and learn the characteristics of input data, and component technologies that utilize machine learning algorithms to mimic functions such as cognition and judgment of the human brain.

[0003] The above-mentioned elemental technologies may include, for example, at least one of linguistic understanding technology that recognizes human language / characters, visual understanding technology that perceives objects like human vision, reasoning / prediction technology that judges information to logically infer and predict, knowledge representation technology that processes human experience information into knowledge data, and motion control technology that controls autonomous driving of vehicles and the movement of robots. Among the aforementioned elemental technologies, linguistic understanding is a technology that recognizes and applies / processes human language / characters, and includes natural language processing, machine translation, dialogue systems, question answering, speech recognition / synthesis, etc.

[0004] Meanwhile, an electronic device equipped with an artificial intelligence system can analyze voice data input by a user's speech, determine a function to be performed by the electronic device or an external electronic device connected to the electronic device based on the analyzed result, and support the performance of said function by the electronic device or the external electronic device.

[0005] Recently, there has been a growing trend of electronic devices providing voice recognition capabilities. Electronic devices that recognize a user's speech can not only provide services to the user by executing built-in functions (e.g., applications installed on the device) but can also provide related services through various third-party service providers. In this case, the electronic device analyzes voice data input by the user's speech and can provide voice recognition services based on the analysis results, specifically whether the voice data contains keywords predetermined by the device's manufacturer. The problem to be solved

[0006] Meanwhile, various electronic devices that provide voice recognition functions can offer various services to users by executing functions built into the device, but the electronic device can launch a voice assistant app to provide voice recognition services upon receiving a predetermined voice keyword. The predetermined voice keyword may be referred to as a trigger word. The electronic device can execute a command corresponding to a user utterance following the predetermined voice keyword. For example, the predetermined keyword may be a keyword set by the manufacturer during the manufacturing of the electronic device. The user of the electronic device can launch the voice assistant app only by making an utterance that includes the predetermined keyword.

[0007] When a predetermined voice command must be spoken to execute the functions of a voice recognition service, a predetermined keyword must be spoken. Therefore, there is a disadvantage in that various users cannot launch the voice assistant app based on custom keywords personalized for each user.

[0008] In addition, to run a voice assistant app using personalized custom keywords, a large amount of custom keywords must be collected and deep learning must be performed on an electronic device or server. If deep learning based on a large amount of custom keywords is performed on an electronic device, it can increase power consumption. Furthermore, if the deep learning is performed through a server, there is a disadvantage of being vulnerable to security risks as a large amount of user voice is transmitted to the server.

[0009] The technical problems to be solved in this disclosure are not limited to those mentioned above, and other unmentioned technical problems will be clearly understood by those skilled in the art from the description below. means of solving the problem

[0010] An electronic device in one embodiment includes a microphone and at least one processor operatively connected to the microphone, and the at least one processor may include a buffer memory that stores a first feature vector for a first voice signal obtained from the microphone as an inverse value, and a calculation circuit configured to perform a norm operation on the first feature vector and the second feature vector based on a second feature vector based on a second voice signal streamed from the microphone and the inverse value of the first feature vector stored in the buffer memory, or to calculate the similarity between the first feature vector and the second feature vector.

[0011] A method of operation of an electronic device comprising a calculation circuit configured to perform a norm operation and a similarity operation according to one embodiment may include: an operation of inversely storing a first feature vector for a first voice signal obtained from a user of the electronic device through a microphone included in the electronic device in a memory included in the electronic device; an operation of performing a norm operation based on a second feature vector based on a second voice signal obtained by streaming from the microphone and the inverse value of the first vector; and an operation of calculating the similarity between the first feature vector and the second feature vector based on the operation result.

[0012] A keyword detection accelerator for performing neural network operations according to one embodiment includes an audio signal processor, a buffer memory, an operation circuit capable of performing norm operations, and at least one processor operatively connected to the audio signal processor, the buffer memory, and the operation circuit. The at least one processor stores a first feature vector for a first voice signal obtained through the audio signal processor as an inverse value in the buffer memory, obtains a second voice signal streamed through the audio signal processor, and performs a norm operation on the first feature vector and the second feature vector based on the second feature vector for the second voice signal and the inverse value of the first feature vector through the operation circuit, or calculates the similarity between the first feature vector and the second feature vector. Effects of the invention

[0013] An electronic device and a method of operation thereof according to various embodiments of the present disclosure can set a custom keyword for using a voice recognition service by user input.

[0014] An electronic device and a method of operation thereof according to various embodiments of the present disclosure can provide a voice recognition service based on the reception of voice data containing a custom keyword from a user's utterance.

[0015] An electronic device and a method of operation according to various embodiments of the present disclosure can perform an always-on custom keyword detection operation using a keyword detection accelerator included in the electronic device.

[0016] The electronic device and the method of operation according to various embodiments of the present disclosure can provide a voice recognition service based on custom keywords through a keyword detection accelerator that implements one-shot learning (or a more advanced form of network) without deep learning training executed on a server.

[0017] In addition, various effects that can be identified directly or indirectly through this document may be provided. Brief explanation of the drawing

[0018] FIG. 1 is a block diagram showing an integrated intelligence system according to one embodiment. FIG. 2 is a diagram showing the form in which relationship information between a concept and an action is stored in a database according to one embodiment. FIG. 3 is a block diagram of an electronic device according to one embodiment. FIG. 4 is a block diagram of a keyword detection accelerator according to one embodiment. FIG. 5 is a diagram showing the configuration of a keyword detection accelerator according to one embodiment. FIG. 6 is a diagram showing the logic operation structure of an operation circuit in one embodiment. Figure 7a is a diagram showing the operation process for the second feature vector. FIG. 7b is a diagram showing the operation process for the first feature vector and the second feature vector. FIG. 7c is a diagram illustrating the process of first norm operation for the first feature vector and the second feature vector. FIG. 7d is a diagram illustrating the process of a second norm operation for a first feature vector and a second feature vector. FIG. 8 is a diagram showing the similarity calculation process of an operation circuit according to one embodiment. FIG. 9 is a flowchart illustrating the custom keyword storage operation of an electronic device according to one embodiment. FIG. 10 is a flowchart illustrating the custom keyword setting operation of an electronic device according to one embodiment. FIG. 11 is a flowchart illustrating the keyword similarity operation of an electronic device according to one embodiment. FIG. 12 is a flowchart illustrating the operation of running a voice assistant app of an electronic device according to one embodiment. FIG. 13 is a drawing illustrating an electronic device that displays a screen for processing voice input received through an intelligent app, according to one embodiment. FIG. 14 is a drawing illustrating an electronic device that displays a screen for setting custom keywords according to one embodiment. FIG. 15 is a block diagram of an electronic device in a network environment according to one embodiment. In relation to the description of the drawings, the same or similar reference numerals may be used for identical or similar components. Specific details for implementing the invention

[0019] FIG. 1 is a block diagram showing an integrated intelligence system according to one embodiment.

[0020] Referring to FIG. 1, an integrated intelligent system of one embodiment may include a user terminal (100), an intelligent server (200), and a service server (290).

[0021] The user terminal (100) of one embodiment may be a terminal device (or electronic device) capable of connecting to the internet, and may be, for example, a mobile phone, a smartphone, a PDA (personal digital assistant), a laptop computer, a TV (television), a white goods appliance, a wearable device, an HMD (head mounted device), or a smart speaker.

[0022] According to the illustrated embodiment, the user terminal (100) may include a communication interface (110), a microphone (120), a speaker (130), a display (140), a memory (150), or a processor (160). The listed components may be operatively or electrically connected to each other.

[0023] A communication interface (110) of one embodiment may be configured to be connected to an external device to transmit and receive data. A microphone (120) of one embodiment may receive sound (e.g., user speech) and convert it into an electrical signal. A speaker (130) of one embodiment may output the electrical signal as sound (e.g., voice). A display (140) of one embodiment may be configured to display an image or video. A display (140) of one embodiment may also display a graphic user interface (GUI) of an app (or application program) being executed.

[0024] In one embodiment, the memory (150) can store a client module (151), an SDK (software development kit) (153), and a plurality of apps (155). The client module (151) and the SDK (153) can form a framework (or solution program) for performing general-purpose functions. Additionally, the client module (151) or the SDK (153) can form a framework for processing voice input.

[0025] The plurality of apps (155) may be programs for performing a designated function. According to one embodiment, the plurality of apps (155) may include a first app (155_1) and / or a second app (155_3). According to one embodiment, each of the plurality of apps (155) may include a plurality of operations for performing a designated function. For example, the apps may include an alarm app, a message app, and / or a schedule app. According to one embodiment, the plurality of apps (155) may be executed by a processor (160) to sequentially execute at least some of the plurality of operations.

[0026] A processor (160) of one embodiment can control the overall operation of a user terminal (100). For example, the processor (160) can perform specified operations by being electrically connected to a communication interface (110), a microphone (120), a speaker (130), and a display (140). For example, the processor (160) may include at least one processor.

[0027] A processor (160) of one embodiment may also perform a specified function by executing a program stored in the memory (150). For example, the processor (160) may execute at least one of a client module (151) or an SDK (153) to perform the following operations for processing voice input. The processor (160) may, for example, control the operation of a plurality of apps (155) through the SDK (153). The following operations described as the operation of the client module (151) or the SDK (153) may be operations performed by the execution of the processor (160).

[0028] A client module (151) of one embodiment can receive voice input. For example, the client module (151) can receive a voice signal corresponding to a user speech detected through a microphone (120). The client module (151) can transmit the received voice input (e.g., voice signal) to an intelligent server (200). Along with the received voice input, the client module (151) can transmit status information of the user terminal (100) to the intelligent server (200). The status information may be, for example, execution status information of an app.

[0029] A client module (151) of one embodiment can receive a result corresponding to a received voice input. For example, if the client module (151) can produce a result corresponding to the received voice input from an intelligent server (200), the client module (151) can receive a result corresponding to the received voice input. The client module (151) can display the received result on a display (140).

[0030] In one embodiment, the client module (151) can receive a plan corresponding to the received voice input. The client module (151) can display the results of executing multiple actions of the app according to the plan on the display (140). The client module (151) can, for example, display the results of executing multiple actions sequentially on the display. The user terminal (100) can, for another example, display only some of the results of executing multiple actions (e.g., the result of the last action) on the display.

[0031] According to one embodiment, the client module (151) may receive a request from the intelligent server (200) to obtain information necessary to produce a result corresponding to a voice input. According to one embodiment, the client module (151) may transmit the necessary information to the intelligent server (200) in response to the request.

[0032] A client module (151) of one embodiment can transmit result information of executing a plurality of operations according to a plan to an intelligent server (200). The intelligent server (200) can use the result information to confirm that the received voice input has been processed correctly.

[0033] A client module (151) of one embodiment may include a voice recognition module. According to one embodiment, the client module (151) can recognize voice input that performs a limited function through the voice recognition module. For example, the client module (151) can execute an intelligent app for processing voice input by performing an organic action in response to a specified voice input (e.g., Wake up!).

[0034] An intelligent server (200) of one embodiment can receive information related to user voice input from a user terminal (100) via a communication network. According to one embodiment, the intelligent server (200) can convert the data related to the received voice input into text data. According to one embodiment, the intelligent server (200) can generate at least one plan for performing a task corresponding to the user voice input based on the text data.

[0035] According to one embodiment, a plan may be generated by an artificial intelligence (AI) system. The AI ​​system may be a rule-based system or a neural network-based system (e.g., a feedforward neural network (FNN)), and / or a recurrent neural network (RNN)). Alternatively, it may be a combination of the foregoing or a different AI system. According to one embodiment, the plan may be selected from a set of predefined plans or may be generated in real time in response to a user request. For example, the AI ​​system may select at least one plan from a plurality of predefined plans.

[0036] An intelligent server (200) of one embodiment may transmit a result according to a generated plan to a user terminal (100) or transmit a generated plan to a user terminal (100). According to one embodiment, the user terminal (100) may display a result according to a plan on a display. According to one embodiment, the user terminal (100) may display a result of executing an action according to a plan on a display.

[0037] An intelligent server (200) of one embodiment may include a front end (210), a natural language platform (220), a capsule database (230), an execution engine (240), an end user interface (250), a management platform (260), a big data platform (270), or an analytic platform (280).

[0038] A front end (210) of one embodiment can receive voice input received from a user terminal (100). The front end (210) can transmit a response corresponding to the voice input to the user terminal (100).

[0039] According to one embodiment, the natural language platform (220) may include an automatic speech recognition module (ASR module) (221), a natural language understanding module (NLU module) (223), a planner module (225), a natural language generator module (NLG module) (227), and / or a text to speech module (TTS module) (229).

[0040] An automatic speech recognition module (221) of one embodiment can convert voice input received from a user terminal (100) into text data. A natural language understanding module (223) of one embodiment can identify the user's intention using the text data of the voice input. For example, the natural language understanding module (223) can identify the user's intention by performing a syntactic analysis or a semantic analysis. A natural language understanding module (223) of one embodiment can identify the meaning of a word extracted from the voice input using linguistic features (e.g., grammatical elements) of a morpheme or phrase, and determine the user's intention by matching the identified meaning of the word to the intention.

[0041] A planner module (225) of one embodiment can generate a plan using intentions and parameters determined by a natural language understanding module (223). According to one embodiment, the planner module (225) can determine multiple domains necessary to perform a task based on the determined intention. The planner module (225) can determine multiple actions included in each of the multiple domains determined based on the intention. According to one embodiment, the planner module (225) can determine parameters necessary to execute the determined multiple actions or result values ​​output by the execution of the multiple actions. The parameters and result values ​​may be defined as concepts of a specified format (or class). Accordingly, the plan may include multiple actions and / or multiple concepts determined by the user's intention. The planner module (225) can determine the relationship between the multiple actions and the multiple concepts in a stepwise (or hierarchical) manner. For example, the planner module (225) can determine the execution order of multiple actions determined based on the user's intentions based on multiple concepts. In other words, the planner module (225) can determine the execution order of multiple actions based on parameters required for the execution of multiple actions and results output by the execution of multiple actions. Accordingly, the planner module (225) can generate a plan that includes association information (e.g., ontology) between multiple actions and multiple concepts. The planner module (225) can generate the plan using information stored in a capsule database (230) in which a set of relationships between concepts and actions is stored.

[0042] A natural language generation module (227) of one embodiment can change specified information into a text form. The information changed into a text form may be in the form of natural language utterance. A text-to-speech conversion module (229) of one embodiment can change information in a text form into information in a speech form.

[0043] According to one embodiment, some or all functions of the natural language platform (220) may also be implemented on a user terminal (100).

[0044] The capsule database (230) may store information regarding the relationships between multiple concepts and actions corresponding to multiple domains. A capsule according to one embodiment may include multiple action objects (or action information) and concept objects (or concept information) included in a plan. According to one embodiment, the capsule database (230) may store multiple capsules in the form of a CAN (concept action network). According to one embodiment, multiple capsules may be stored in a function registry included in the capsule database (230).

[0045] The above capsule database (230) may include a strategy registry in which strategy information necessary for determining a plan corresponding to a voice input is stored. The strategy information may include reference information for determining one plan when there are multiple plans corresponding to the voice input. According to one embodiment, the capsule database (230) may include a follow-up registry in which information of a follow-up action is stored for suggesting a follow-up action to a user in a specified situation. The follow-up action may include, for example, a follow-up utterance. According to one embodiment, the capsule database (230) may include a layout registry in which layout information of information output through the user terminal (100) is stored. According to one embodiment, the capsule database (230) may include a vocabulary registry in which vocabulary information included in the capsule information is stored. According to one embodiment, the capsule database (230) may include a dialogue registry in which information regarding a conversation (or interaction) with a user is stored. The capsule database (230) may update stored objects through a developer tool. The developer tool may include, for example, a function editor for updating action objects or concept objects. The developer tool may include a vocabulary editor for updating vocabulary. The developer tool may include a strategy editor for creating and registering strategies for determining plans. The developer tool may include a dialogue editor for creating conversations with a user.The above developer tool may include a follow-up editor capable of activating a follow-up goal and editing a follow-up utterance that provides a hint. The follow-up goal may be determined based on a currently set goal, user preferences, or environmental conditions. In one embodiment, the capsule database (230) may also be implemented within the user terminal (100).

[0046] An execution engine (240) of one embodiment can produce a result using the generated plan. An end-user interface (250) can transmit the produced result to a user terminal (100). Accordingly, the user terminal (100) can receive the result and provide the received result to the user. A management platform (260) of one embodiment can manage information used in the intelligent server (200). A big data platform (270) of one embodiment can collect user data. An analysis platform (280) of one embodiment can manage the quality of service (QoS) of the intelligent server (200). For example, the analysis platform (280) can manage the components and processing speed (or efficiency) of the intelligent server (200).

[0047] A service server (290) of one embodiment may provide a service designated to a user terminal (100) (e.g., food ordering or hotel reservation). According to one embodiment, the service server (290) may be a server operated by a third party. A service server (290) of one embodiment may provide information to an intelligent server (200) for generating a plan corresponding to a received voice input. The provided information may be stored in a capsule database (230). Additionally, the service server (290) may provide result information according to the plan to the intelligent server (200).

[0048] In the integrated intelligent system described above, the user terminal (100) can provide various intelligent services to the user in response to user input. The user input may include, for example, input via a physical button, touch input, or voice input.

[0049] In one embodiment, the user terminal (100) may provide a voice recognition service through an intelligent app (or voice recognition app) stored internally. In this case, for example, the user terminal (100) may recognize a user utterance or voice input received through the microphone and provide a service to the user corresponding to the recognized voice input.

[0050] In one embodiment, the user terminal (100) may perform a specified action, either alone or together with the intelligent server and / or service server, based on the received voice input. For example, the user terminal (100) may execute an app corresponding to the received voice input and perform a specified action through the executed app.

[0051] In one embodiment, when a user terminal (100) provides services together with an intelligent server (200) and / or a service server, the user terminal can detect user speech using the microphone (120) and generate a signal (or voice data) corresponding to the detected user speech. The user terminal can transmit the voice data to the intelligent server (200) using a communication interface (110).

[0052] An intelligent server (200) according to one embodiment may generate, in response to a voice input received from a user terminal (100), a plan for performing a task corresponding to the voice input, or a result of performing an operation according to the plan. The plan may include, for example, a plurality of operations for performing a task corresponding to the user's voice input and / or a plurality of concepts related to the plurality of operations. The concepts may define parameters input to the execution of the plurality of operations or result values ​​output by the execution of the plurality of operations. The plan may include association information between the plurality of operations and / or the plurality of concepts.

[0053] A user terminal (100) of one embodiment can receive the response using a communication interface (110). The user terminal (100) can output a voice signal generated inside the user terminal (100) to the outside using the speaker (130), or output an image generated inside the user terminal (100) to the outside using the display (140).

[0054] FIG. 2 is a diagram showing the form in which relationship information between a concept and an operation is stored in a database according to various embodiments.

[0055] The capsule database (e.g., capsule database (230)) of the intelligent server (200) can store capsules in the form of a CAN (concept action network). The capsule database can store actions for processing tasks corresponding to the user's voice input, and parameters required for said actions, in the form of a CAN (concept action network).

[0056] The above capsule database may store a plurality of capsules (capsule(A)(401), capsule(B)(404)) corresponding to each of a plurality of domains (e.g., application). According to one embodiment, one capsule (e.g., capsule(A)(401)) may correspond to one domain (e.g., location (geo), application). Additionally, one capsule may correspond to at least one service provider (e.g., CP service A (291) or CP service B (292)) for performing functions for the domain associated with the capsule. According to one embodiment, one capsule may include at least one operation (410) and at least one concept (420) for performing a designated function.

[0057] The above natural language platform (220) can generate a plan for performing a task corresponding to a received voice input using capsules stored in a capsule database. For example, the planner module (225) of the natural language platform can generate a plan using capsules stored in a capsule database. For example, a plan (407) can be generated using the actions (4011, 4013) and concepts (4012, 4014) of capsule A (410) and the actions (4041) and concepts (4042) of capsule B (404).

[0058] FIG. 3 is a block diagram of an electronic device according to one embodiment.

[0059] Referring to FIG. 3, an electronic device (300) (e.g., user terminal (100) of FIG. 1) may include at least one processor (310), memory (320), microphone (330), communication circuit (340), and display (350). The listed components may be operatively or electrically connected to each other. The components of the electronic device (300) shown in FIG. 3 may be partially modified, deleted, or added as an example. According to one embodiment, the electronic device (300) may include all or part of the components of the user terminal (100) of FIG. 1. According to various embodiments, the electronic device (300) may communicate with the intelligent server (200) of FIG. 1 and perform operations according to various embodiments, either alone or jointly.

[0060] According to one embodiment, the processor (310) may be electrically or operatively connected to a memory (320), a microphone (330), a communication circuit (340), and a display (350). According to one embodiment, the processor (310) may execute operations or data processing regarding the control and / or communication of at least one other component of the electronic device (300) using instructions stored in the memory (320). According to one embodiment, the processor (310) may include at least one of a central processing unit (CPU), a graphics processing unit (GPU), a micro controller unit (MCU), a sensor hub, a supplementary processor, a communication processor, an application processor, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), and a Neural Processing Unit (NPU), and may have multiple cores.

[0061] According to one embodiment, the NPU included in the processor (310) can perform neural network-based tasks required according to the execution of the application. According to one embodiment, the NPU may be included in the application processor. The neural network may include at least one of various types of neural network models including CNN (Convolutional Neural Network), R-CNN (Region with Convolutional Neural Network), RPN (Region Proposal Network), RNN (Recurrent Neural Network), S-DNN (Stacking-based Deep Neural Network), S-SDNN (State-Space Dynamic Neural Network), Deconvolution Network, DBN (Deep Belief Network), RBM (Restricted Boltzmann Machine), Fully Convolutional Network, LSTM (Long Short-Term Memory) Network, and Classification Network.

[0062] According to one embodiment, the processor (310) can perform neural network operations based on received input data. For example, neural network operations may include convolution, pooling, activation function operations, inner product operations, vector operations, and sigmoid operations.

[0063] A neural network may include at least one layer. At least one layer may be a linear layer (e.g., a convolution layer) or a non-linear layer (e.g., a polling layer). An electronic device (300) according to one embodiment may perform neural network operations through a convolution layer. A convolution layer according to one embodiment may generate a feature vector based on input values. For example, the convolution layer may generate a feature vector corresponding to an input user voice signal.

[0064] According to one embodiment, the processor (310) may include a hardware structure specialized for executing a voice recognition service. For example, the processor (310) may include a buffer memory (311) and an arithmetic circuit (312). According to various embodiments, the processor (310) may include various configurations for determining whether to execute a voice recognition service.

[0065] According to one embodiment, the buffer memory (311) may store programs and / or data. For example, the buffer memory (311) may store computation parameters for a neural network, input data (e.g., feature vectors), and output data (e.g., similarity). According to one embodiment, the computation circuit (312) may perform a computation operation (e.g., norm operation) based on input data (e.g., feature vectors) received from the memory (311), and store the computation result (e.g., similarity) in the memory (311). A description of the buffer memory (311) and the computation circuit (312) according to one embodiment will be described in detail later with reference to FIG. 4.

[0066] According to one embodiment, the processor (310) can receive voice input by a user's speech through a microphone (330) and process the received voice input. Additionally, the processor (310) can convert the received voice input into text data and provide a voice recognition service (e.g., a voice assistant) based on the text data. For example, the processor (310) can execute a voice assistant app in response to the received voice input containing a keyword for executing a voice assistant app. According to one embodiment, the processor (310) can execute a voice assistant app in response to a custom keyword being entered by a user of the electronic device (300). The operation of the processor (310) executing a voice assistant app is described in detail later with reference to FIGS. 9 through 12.

[0067] To perform the above-described function, the processor (310) may include a natural language platform (220) described with reference to FIG. 1, or may include a hardware structure specialized for performing the above-described function.

[0068] According to one embodiment, the memory (320) may store instructions that allow the processor (310) to process data or control components of the electronic device (300) to perform operations of the electronic device (300) when executed. According to one embodiment, the memory (320) may store at least one application capable of performing various functions. According to one embodiment, the memory (320) may store instructions and data related to processing user voice input received through the microphone (330). For example, it may store data regarding the input user voice signal and / or data related to the execution of a voice recognition service.

[0069] According to one embodiment, the memory (320) may store computation parameters used for keyword similarity calculations to execute a voice assistant app. For example, the memory (320) may store data for custom keywords set by the processor (310) and / or data for keyword detection sensitivity input by the user of the electronic device (300).

[0070] According to one embodiment, the microphone (330) can receive user speech and convert it into an electrical signal. For example, the microphone (330) can receive user speech for executing a voice recognition service from a user of the electronic device (300) and convert it into an electrical signal. According to one embodiment, the processor (310) can determine the keyword of the voice signal received from the user using the electrical signal received through the microphone (330). For example, the processor (310) can determine whether the keyword of the voice signal received from the user is a custom keyword for executing a voice recognition service. According to one embodiment, if the keyword of the received voice signal is a keyword for executing a voice recognition service, the processor (310) can execute a voice assistant app. According to one embodiment, the operation of the processor (310) determining whether the keyword of the received voice signal is a keyword for executing a voice recognition service is described in detail later with reference to FIGS. 4 to 8.

[0071] According to one embodiment, the electronic device (300) can communicate with the intelligent server (200) described with reference to FIG. 1 through a communication circuit (340). The electronic device (300) can transmit voice data based on user speech to the intelligent server (200) through the communication circuit (340). Additionally, the electronic device (300) can receive a response from the intelligent server (200) using the communication circuit (340). For example, the electronic device (300) can transmit voice data based on user speech to the intelligent server (200) through the communication circuit (340) and receive text data generated by an automatic speech recognition module (221).

[0072] According to one embodiment, the display (350) may display various content (e.g., text, images, videos, icons, and / or symbols, etc.). According to one embodiment, the display (350) may include a liquid crystal display (LCD), a light-emitting diode (LED) display, or an organic light-emitting diode (OLED) display. According to one embodiment, the processor (310) may control the display (350) to display various content related to the execution of a voice recognition service. For example, the processor (310) may control the display (350) to display a screen requesting keyword input for the execution of a voice recognition service. According to various embodiments, the operation of the display (350) displaying content related to the execution of a voice recognition service is described later with reference to FIGS. 13 and FIGS. 14.

[0073] According to one embodiment, the processor (310) receives a first voice signal corresponding to a keyword for executing a voice recognition service from a user through a microphone (330), and can store a first feature vector for the first voice signal as an inverse value in a buffer memory (311). The first feature vector according to one embodiment may represent a vector generated by inputting the first voice signal into a convolution layer.

[0074] According to one embodiment, the processor (310) may acquire a second voice signal obtained via streaming through the microphone (330) and generate a second feature vector for the second voice signal. According to one embodiment, the second feature vector may represent a vector generated by inputting the second voice signal into a convolution layer.

[0075] According to one embodiment, the processor (310) can perform various operations based on the inverse value of the first feature vector and the second feature vector stored in the buffer memory (311). For example, the processor (310) can perform a norm operation on the inverse value of the first feature vector and the second feature vector using an operation circuit (312), or calculate the similarity between the inverse value of the first feature vector and the second feature vector. According to one embodiment, the processor (310) can execute a voice assistant app based on the result of the similarity calculation.

[0076] FIG. 4 is a block diagram of a keyword detection accelerator according to one embodiment.

[0077] Referring to FIG. 4, the keyword detection accelerator (3100) may include a buffer memory (3110), an operation circuit (3120), and an accelerator controller (3130). According to one embodiment, the keyword detection accelerator (3100) and the accelerator controller (3130) may be included in the processor (310) described with reference to FIG. 3. According to one embodiment, the keyword detection accelerator (3100) may be included in an application processor. According to various embodiments, the keyword detection accelerator (3100) and the accelerator controller (3130) may perform the operation of the processor (310) described with reference to FIG. 3. According to one embodiment, the accelerator controller (3130) may control the operation of the keyword detection accelerator (3100). According to one embodiment, the accelerator controller (3130) may be operated at low power.

[0078] According to one embodiment, the keyword detection accelerator (3100) can determine whether a custom keyword for executing a voice assistant app is included in a user voice signal obtained through a microphone (330). Additionally, the keyword detection accelerator (3100) can transmit an interrupt signal for executing a voice assistant app to a processor (310) in response to the inclusion of a custom keyword. For example, an accelerator controller (3130) included in the keyword detection accelerator (3100) can transmit an interrupt signal for executing a voice assistant app to a central processing unit (CPU). For example, the accelerator controller (3130) can transmit an interrupt signal to a central processing unit (CPU), an internal bus, and a memory controller (DRAM controller) included in the processor (310). According to one embodiment, the processor (310) can execute a voice assistant app based on the interrupt signal.

[0079] According to one embodiment, the buffer memory (3110) may store a custom keyword (3111), a streaming keyword (3112), and a similarity threshold (3113). According to one embodiment, the keyword detection accelerator (3100) may receive external data (E_D) from various configurations included in the electronic device (300). According to one embodiment, the external data may include a user's voice signal obtained through a microphone (330), data regarding keyword detection sensitivity input by the user, and / or data stored in memory (320). According to various embodiments, the external data may include various data.

[0080] According to one embodiment, the accelerator controller (3130) may store the external data (E_D) in the buffer memory (3110) in response to the receipt of external data (E_D). According to one embodiment, the external data (E_D) may include a feature vector for a voice signal input by a user. According to one embodiment, the feature vector may be obtained using a convolution layer described with reference to FIG. 3. For example, the processor (310) may obtain the feature vector for the voice signal using computational hardware (e.g., CPU, GPU and / or DSP). According to one embodiment, the accelerator controller (3130) may store a first feature vector for the input first voice signal in the custom keyword (3111) according to the custom keyword setting operation of the electronic device (300). According to one embodiment, the accelerator controller (3130) may obtain a first feature vector from memory (320) and store the obtained first feature vector inversely in buffer memory (3110). According to various embodiments, the custom keyword (3111), streaming keyword (3112), and similarity threshold (3113) stored in buffer memory (3110) may be stored in a single buffer memory or each may be stored in multiple buffer memories.

[0081] According to one embodiment, the accelerator controller (3130) may store a second feature vector for a second voice signal streamed through the microphone (330) in a streaming keyword (3112). According to one embodiment, the accelerator controller (3130) may receive data regarding keyword detection sensitivity from a user and store the inverse value therefrom in a buffer memory (3110) as a similarity threshold (3113). For example, data regarding keyword detection sensitivity may be obtained from a user via touch input through a display (350), and the obtained data may be stored in the buffer memory (3110) as a similarity threshold (3113). According to one embodiment, the larger the magnitude of the absolute value of the similarity threshold, the higher the similarity between the keyword for running the voice assistant app included in the first voice signal and the keyword included in the second voice signal, so that the voice assistant app can be run.

[0082] According to one embodiment, the operation circuit (3120) can perform various operations executed in the keyword detection accelerator. According to various embodiments, the operation circuit may include at least one multiplexer, at least one multiplier, at least one adder, at least one accumulator, and / or at least one two's complement circuit.

[0083] According to one embodiment, the keyword detection accelerator (3100) can determine whether a keyword for executing a voice assistant app is included in a second voice signal obtained via streaming. For example, the keyword detection accelerator (3100) can calculate the similarity between a first feature vector corresponding to a preset custom keyword and a second feature vector corresponding to a second voice signal obtained via streaming, and compare the calculated similarity with a similarity threshold (3113) to generate an interrupt signal for executing a voice assistant app. According to one embodiment, the accelerator controller (3130) can calculate the similarity between the inverse value of the first feature vector and the second feature vector through a calculation circuit (3120). According to one embodiment, the similarity between the inverse value of the first feature vector and the second feature vector can be determined through a norm operation. According to one embodiment, the accelerator controller (3130) can determine the similarity by performing a sigmoid operation on the result of a norm operation on the first feature vector and the second feature vector.

[0084] According to one embodiment, the accelerator controller (3130) can compare the similarity determined using the computation circuit (3120) with a threshold value, and if the similarity is greater than the threshold value, transmit an interrupt signal to the central processing unit to execute a voice assistant app. According to various embodiments, the process of the computation circuit (3120) performing a norm operation based on a first feature vector and a second feature vector and the process of comparing the similarity with the threshold value will be described in detail later with reference to FIGS. 6 to 8.

[0085] FIG. 5 is a diagram showing the configuration of a keyword detection accelerator according to one embodiment.

[0086] Referring to FIG. 5, the keyword detection accelerator (3100) may include SRAM (510), MFCC (530), Activation (540), Control (550), and MAC (520). According to one embodiment, SRAM (510) may be included in buffer memory (3110). According to one embodiment, Control (550) may be included in accelerator controller (3130), and MAC (520) may be included in computation circuit (3120). According to one embodiment, MFCC (530) may divide a user's voice signal input through a microphone (330) into certain intervals and analyze the spectrum to extract features. According to one embodiment, Activation (540) may perform activation function operations through an activation layer. For example, Activation (540) may perform operations using a sigmoid function and a Rectified Linear Unit (ReLU) function.

[0087] According to one embodiment, Control (550) can read the value of the inverse of the first feature vector corresponding to the custom keyword stored in memory (320) and the similarity threshold value to determine whether to run the voice assistant app, and store them in SRAM (510). According to one embodiment, Control (550) can perform a norm operation on the second feature vector and the value of the inverse of the first feature vector for the second voice signal obtained via streaming through MFCC (530) through MAC (520). Additionally, Control (550) can control the result of the norm operation through MAC (520) to be subjected to a sigmoid operation through Activation (540). According to one embodiment, the values ​​for the first feature vector and the second feature vector output using the sigmoid function can be represented as similarity.

[0088] According to one embodiment, Control (550) can compare the similarity obtained through Activation (540) with the similarity threshold stored in SRAM (510) through MAC (520). For example, Control (550) can perform a subtraction operation on the similarity and the similarity threshold through MAC (520). According to one embodiment, if the result of the comparison of the similarity and the similarity threshold obtained through MAC (520) is positive, Control (550) can transmit an interrupt signal to the central processing unit to execute a voice assistant app. According to another embodiment, if the result of the comparison of the similarity and the similarity threshold obtained through MAC (520) is negative, Control (550) can determine that there was no voice input to execute a voice assistant app.

[0089] FIG. 6 is a diagram showing the logic operation structure of an operation circuit according to one embodiment.

[0090] Referring to FIG. 6, the operation circuit (312) may include first to sixth multiplexers (601, 602, 604, 605, 608, 609), a multiplier (603), an adder (607), an accumulator (610), and / or a two's complement circuit (606). Depending on various embodiments, components included in the operation circuit (312) may be added or omitted. According to one embodiment, the operation circuit (312) may perform operation operations on data represented as binary numbers.

[0091] According to one embodiment, the first multiplexer (601) and the second multiplexer (602) can each select and output one of the data transmitted from other components included in the electronic device (300) and the data transmitted from the accumulator (610). According to one embodiment, the accumulator (610) (e.g., a register) can temporarily store the result of an operation performed in the operation circuit (312) and / or parameter values ​​used in the operation. Data output from the first multiplexer (601) according to one embodiment can be input to the multiplier (603).

[0092] According to one embodiment, the multiplier (603) can perform a multiplication operation on the input data and output the multiplication result. According to one embodiment, the multiplication result can be input to the third multiplexer (604). The third multiplexer (604) can select and output one of the multiplication result input from the multiplier (603) and data transmitted from other components included in the electronic device (300). The data selected through the third multiplexer (604) can be input to the adder (607). According to one embodiment, the fourth multiplexer (605) can select and output one of the data transmitted from the accumulator (610) and '0'. The data selected through the fourth multiplexer (605) can be input to the adder (607). An adder (607) according to one embodiment can perform an addition operation on data input from a third multiplexer (604) and a fourth multiplexer (605) and output an addition result.

[0093] According to one embodiment, the two's complement circuit (606) can perform a two's complement operation on the data output from the accumulator (610). The operation circuit (312) can output the absolute value of the input data through the two's complement operation. For example, the two's complement circuit (606) can check the sign bit of the input data and determine whether the input data is positive or negative. For example, the two's complement circuit (606) can determine whether the input data is positive or negative based on the most significant bit of the input data. According to one embodiment, if the sign bit of the input data is '0', it is determined to be positive, and if it is '1', it is determined to be negative. According to one embodiment, if the sign bit of the input data is '0', the two's complement circuit (606) outputs the input data as is, and if the sign bit is '1', it can perform a two's complement operation.

[0094] According to one embodiment, the sixth multiplexer (609) can receive absolute value data output from the two's complement circuit (606) and arbitrary data, select one, and output it. The output data can be input to the fifth multiplexer (608). According to one embodiment, the fifth multiplexer (608) can select and output one of the data input from the adder (607) and the sixth multiplexer (609). The output data can be stored in the accumulator (610).

[0095] According to one embodiment, an operation circuit comprises a plurality of multiplexers (e.g., first to sixth multiplexers (601, 602, 604, 605, 608, 609)), at least one register for storing data output from an operation step, at least one multiplier (e.g., multiplier (603)), at least one adder (e.g., adder (607)), at least one accumulator (e.g., accumulator (610)), and a two's complement circuit (e.g., two's complement circuit (606)). One of the plurality of multiplexers (e.g., third multiplexer (604)) outputs a value of the inverse of a first feature vector (-h1) or a similarity threshold (-Th) based on a first voice signal related to a custom keyword setting, and the two's complement circuit performs an addition operation of the value of the inverse of the first feature vector (-h1) and a second feature vector (h2) based on a second voice signal. The result can be output by performing an absolute value operation on the resulting subtraction data (h2-h1), and the similarity between the first voice signal and the second voice signal can be calculated by performing a norm operation through at least one multiplier, at least one adder, and at least one accumulator. According to various embodiments, the operation circuit (312) may include a register for storing data output at each operation step. According to one embodiment, the first to sixth multiplexers (601, 602, 603, 604, 605, 608, 609) can select and output one of the input data based on a control signal transmitted from the accelerator controller (3130) described with reference to FIG. 4. According to various embodiments, the configuration of the operation circuit (312) may be changed within an equal range capable of outputting the same result value.

[0096] Figure 7a is a diagram showing the operation process for the second feature vector.

[0097] Referring to FIG. 7a, in order to perform a norm operation on the first feature vector and the second feature vector, the operation circuit (312) can store the second feature vector in the accumulator (610). Since the description of the configuration of the operation circuit (312) is the same as that described with reference to FIG. 6, redundant descriptions are omitted.

[0098] According to one embodiment, the operation circuit (312) may receive the second feature vector (h2) and '1' described with reference to FIG. 3 as input data. According to one embodiment, the second feature vector (h2) may correspond to a second voice signal received via streaming through the microphone (330). According to one embodiment, the first multiplexer (601) may output the received second feature vector (h2) to the multiplier (603), and the second multiplexer (602) may output the input '1' to the multiplier (603). According to one embodiment, the multiplier (603) may perform a multiplication operation on the second feature vector (h2) and '1'. Thus, the multiplier (603) may output the second feature vector (h2) through the operation.

[0099] According to one embodiment, the second feature vector (h2) output from the multiplier (603) can be input to the third multiplexer (604). According to one embodiment, the third multiplexer (604) can output the input second feature vector (h2) to the adder (607). According to one embodiment, the fourth multiplexer (605) can receive '0' as input and output it to the adder (607). Thus, the adder (607) can receive the second feature vector (h2) and '0' as input. According to one embodiment, the adder (607) can perform an addition operation on the second feature vector and '0' and output the addition result to the fifth multiplexer (608). Thus, the fifth multiplexer (608) can receive the second feature vector (h2) as input.

[0100] According to one embodiment, the second feature vector (h2) input to the fifth multiplexer (608) can be output to the accumulator (610). The accumulator (610) according to one embodiment can store the second feature vector (h2).

[0101] FIG. 7b is a diagram showing the operation process for the first feature vector and the second feature vector.

[0102] Referring to FIG. 7b, in order to perform a norm operation on the first feature vector and the second feature vector, the operation circuit (312) may perform a subtraction operation on the first feature vector and the second feature vector and store the subtraction result in the accumulator (610). Since the description of the configuration of the operation circuit (312) is the same as that described with reference to FIG. 6, redundant descriptions are omitted.

[0103] According to one embodiment, the operation circuit (312) can receive the value (-h1) of the inverse of the first feature vector described with reference to FIG. 3 from the buffer memory (3110) through the accelerator controller (3130). According to one embodiment, the accelerator controller (3130) can load the value of the inverse of the first feature vector stored in the memory (320) into the buffer memory (311) and input the value of the inverse of the first feature vector to the operation circuit (312) in order to perform keyword similarity calculations based on the first feature vector and the second feature vector.

[0104] According to one embodiment, the value of the inverse of the first feature vector (-h1) may correspond to the first voice signal received through the microphone (330) in the operation of setting a custom keyword. According to one embodiment, the third multiplexer (604) may output the received value of the inverse of the first feature vector (-h1) to the adder (607). According to one embodiment, the accumulator (610) may output the second feature vector (h2) stored in the accumulator (610) to the fourth multiplexer (605) with reference to FIG. 7a.

[0105] According to one embodiment, the fourth multiplexer (605) can select and output the second feature vector (h2) from among the input second feature vector (h2) and '0'. The output second feature vector (h2) can be input to the adder (607).

[0106] According to one embodiment, the adder (607) can perform an addition operation on the inverse value of the first feature vector (-h1) and the second feature vector (h2) input through the third multiplexer (604). For example, the adder (607) can perform an addition operation on the inverse value of the first feature vector (-h1) and the second feature vector (h2), and output the subtraction data (h2-h1), which is the result of the addition operation, to the fifth multiplexer (608).

[0107] According to one embodiment, subtraction data (h2-h1) input to the fifth multiplexer (608) can be output to an accumulator (610). The accumulator (610) according to one embodiment can store subtraction data (h2-h1).

[0108] FIG. 7c is a diagram illustrating the first norm operation process for the first feature vector and the second feature vector.

[0109] Referring to FIG. 7c, in order to perform a norm operation on the first feature vector and the second feature vector, the operation circuit (312) can perform an absolute value operation on the subtraction data (h2-h1). Since the description of the configuration of the operation circuit (312) is the same as that described with reference to FIG. 6, redundant descriptions are omitted.

[0110] According to one embodiment, with reference to FIG. 7c, subtraction data (h2-h1) stored in the accumulator (610) can be input to a two's complement circuit (606). According to one embodiment, the two's complement circuit (606) can perform an absolute value operation on the input subtraction data (h2-h1) and output the subtraction absolute value data (|h2-h1|) to a sixth multiplexer (609). According to various embodiments, the two's complement circuit (606) may include various configurations necessary for the absolute value operation.

[0111] According to one embodiment, the sixth multiplexer (609) may output subtraction absolute value data (|h2-h1|) to the fifth multiplexer (608). Additionally, the fifth multiplexer (608) may select subtraction absolute value data (|h2-h1) and output it to the accumulator (610), and the output subtraction absolute value data (|h2-h1) may be stored in the accumulator (610). According to one embodiment, the subtraction absolute value data (|h2-h1|) stored in the accumulator (610) of the operation circuit (312) may be represented as a first norm value for similarity calculation of the first feature vector and the second feature vector. According to one embodiment, the processor (310) described with reference to FIG. 3 may perform a sigmoid operation on the first norm value. According to one embodiment, the first norm value obtained by performing a sigmoid operation can be represented as first similarity data. According to one embodiment, a similarity operation for determining whether a second voice signal contains a custom keyword based on the first similarity data is described in detail later with reference to FIG. 8.

[0112] FIG. 7d is a diagram illustrating the process of a second norm operation for a first feature vector and a second feature vector.

[0113] Referring to FIG. 7d, in order to perform a norm operation on the first feature vector and the second feature vector, the operation circuit (312) can perform a squaring operation on the subtraction data (h2-h1). Since the description of the configuration of the operation circuit (312) is the same as that described with reference to FIG. 6, redundant descriptions are omitted.

[0114] According to one embodiment, with reference to FIG. 7c, subtraction data (h2-h1) stored in the accumulator (610) can be input to the first multiplexer (601) and the second multiplexer (602), respectively. According to one embodiment, the first multiplexer (601) and the second multiplexer (602) can output the input subtraction data (h2-h1) to a multiplier (603). According to one embodiment, the multiplier (603) can perform a multiplication operation on the subtraction data (h2-h1) input from the first multiplexer (601) and the second multiplexer (602). According to one embodiment, the multiplier (603) can perform a multiplication operation on the subtraction data (h2-h1) and output subtraction squared data ((h2-h1)^2) as a result.

[0115] According to one embodiment, subtraction squared data ((h2-h1)^2) output from a multiplier (603) can be input to a third multiplexer (604). According to one embodiment, the third multiplexer (604) can output the input subtraction squared data ((h2-h1)^2) to an adder (607). According to one embodiment, the fourth multiplexer (605) can receive '0' as input and output '0' to the adder (607). According to one embodiment, the adder (607) can perform an addition operation on the subtraction squared data ((h2-h1)^2) input from the third multiplexer (604) and the '0' input from the fourth multiplexer (605), and output the addition result. Therefore, the adder (607) can output subtraction squared data ((h2-h1)^2) as the addition result.

[0116] According to one embodiment, subtraction square data ((h2-h1)^2) output from the adder (607) is input to the fifth multiplexer (608), and subtraction square data ((h2-h1)^2) input to the fifth multiplexer (608) can be input to the accumulator (610) and stored.

[0117] According to one embodiment, the subtraction squared data ((h2-h1)^2) stored in the accumulator (610) of the operation circuit (312) can be represented as a second norm value for similarity calculation of the first feature vector and the second feature vector. According to one embodiment, the processor (310) described with reference to FIG. 3 can perform a sigmoid operation on the second norm value. According to one embodiment, the second norm value after performing the sigmoid operation can be represented as second similarity data. According to one embodiment, a similarity calculation operation for determining whether the second voice signal contains a custom keyword based on the second similarity data is described in detail later with reference to FIG. 8.

[0118] FIG. 8 is a diagram showing the similarity calculation process of an operation circuit according to one embodiment.

[0119] Referring to FIG. 8, the processor (310) can compare the first similarity data and the second similarity data described with reference to FIG. 7a to 7d with data regarding keyword detection sensitivity using an operation circuit (312). According to one embodiment, the accelerator controller (3130) can determine whether to execute the voice assistant app based on the comparison result and generate an interrupt signal for executing the voice assistant app and transmit it to the processor (310). For example, the accelerator controller (3130) included in the keyword detection accelerator (3100) can generate an interrupt signal and transmit it to the processor (310) (e.g., central processing unit). According to one embodiment, the processor (310) can execute the voice assistant app based on the interrupt signal.

[0120] According to one embodiment, the operation circuit (312) can compare the magnitude of the first similarity data with the magnitude of the similarity threshold (-Th) (e.g., the similarity threshold (3113) of FIG. 4). Additionally, the operation circuit (312) can compare the magnitude of the second similarity data with the magnitude of the similarity threshold (-Th). According to one embodiment, the operation of comparing the magnitude of the first similarity data with the magnitude of the similarity threshold (-Th) and the operation of comparing the magnitude of the second similarity data with the magnitude of the similarity threshold (-Th) can be performed through the same operation process. Therefore, for convenience of explanation, the first similarity data and the second similarity data may be collectively referred to as similarity data (L).

[0121] According to one embodiment, the accelerator controller (3130) can input a similarity threshold (-Th) stored in the buffer memory (3110) to the third multiplexer (604). The third multiplexer (604) can output the input similarity threshold (-Th) to an adder (607).

[0122] According to one embodiment, with reference to FIGS. 7c and FIGS. 7d, similarity data (L) stored in the accumulator (610) can be input to the fourth multiplexer (604). According to one embodiment, the fourth multiplexer (605) receives similarity data (L) and '0', and can select and output similarity data (L). The output similarity data (L) can be input to the adder (607).

[0123] According to one embodiment, the adder (607) can perform an addition operation on the similarity threshold (-Th) output through the third multiplexer (604) and input to the adder (607), and on the similarity data (L) input to the adder (607) through the fourth multiplexer (605). According to one embodiment, the adder (607) can determine the sign bit of the addition result for the similarity threshold (-Th) and the similarity data (L). For example, the adder (607) can determine the most significant bit of the similarity comparison value (L-Th), which is the addition result for the similarity threshold (-Th) and the similarity data (L), as the sign bit.

[0124] According to one embodiment, the sign bit of the determined similarity comparison value (L-Th) can be output by the adder (607) as a similarity result (S(L-Th)). According to one embodiment, the output similarity result (S(L-Th)) can be input to the fifth multiplexer (608). According to one embodiment, the fifth multiplexer (608) can output the input similarity result (S(L-Th)) to the accumulator (610). According to one embodiment, the accumulator (610) can store the similarity result (S(L-Th)) output from the fifth multiplexer (608).

[0125] According to one embodiment, the accelerator controller (3130) may generate an interrupt signal to execute a voice assistant app when the similarity result (S(L-Th)) obtained from the computational circuit (312) indicates a positive value. For example, if a second voice signal obtained via streaming from a user contains a custom keyword for executing a voice assistant app, the obtained similarity result (S(L-Th)) may indicate a positive value. Accordingly, the accelerator controller (3130) may generate an interrupt signal for executing a voice assistant app and transmit it to the processor (310). For example, the accelerator controller (3130) included in the keyword detection accelerator (3100) may generate an interrupt signal and transmit it to the processor (310) (e.g., central processing unit). According to one embodiment, the processor (310) may execute a voice assistant app based on the interrupt signal.

[0126] According to another embodiment, if the similarity result (S(L-Th)) obtained from the operation circuit (312) indicates a negative value, it may be determined not to run the voice assistant app. For example, if the second voice signal obtained via streaming from the user does not contain a custom keyword for running the voice assistant app, the obtained similarity result (S(L-Th)) may indicate a negative value. Accordingly, the accelerator controller (3130) may not run the voice assistant app and may obtain the second voice signal again via streaming through the microphone (330).

[0127] FIG. 9 is a flowchart (900) for explaining the custom keyword storage operation of an electronic device according to one embodiment.

[0128] Referring to FIG. 9, the processor (310) can obtain a first voice signal from a user of the electronic device (300) using a microphone (330) in operation 901. According to one embodiment, the first voice signal is a voice command for executing a voice assistant app and may include a custom keyword set by user input.

[0129] According to one embodiment, the electronic device (300) may receive input regarding keyword detection sensitivity from a user in operation 903 and store the received similarity threshold in memory (320). According to one embodiment, when the processor (310) stores the similarity threshold in memory (320), it may store the inverse value of the similarity threshold in memory (320).

[0130] According to one embodiment, the processor (310) may store a first feature vector for the acquired first voice signal in memory (320) as an inverse value in operation 905. According to one embodiment, the first voice signal may include a voice signal included in at least one utterance of a user. According to one embodiment, the processor (310) may acquire a first feature vector for each of the first voice signals included in at least one utterance through neural network operations. For example, a first feature vector for each of the first voice signals may be acquired using a convolution layer included in a neural network.

[0131] According to various embodiments, the processor (310) may store the first feature vector for each of the first voice signals inversely in memory (320), or process (e.g., average) the representative value of the first feature vector for each of the first voice signals and store the calculated value inversely in memory (320). An operation in which the processor (310) according to one embodiment stores the value of the first feature vector inversely in memory (320) is described in detail later with reference to FIG. 10.

[0132] FIG. 10 is a flowchart (1000) for explaining the custom keyword setting operation of an electronic device according to one embodiment.

[0133] Referring to FIG. 10, the processor (310) can obtain a voice signal from a user of the electronic device (300) through a microphone (330) in operation 1001. According to one embodiment, the processor (310) can recognize a custom keyword included in the voice signal in operation 1003. According to one embodiment, the custom keyword may represent a keyword that the user sets as a command to run a voice assistant app.

[0134] According to one embodiment, the processor (310) can determine in operation 1005 whether the time taken to acquire a voice signal from a user is less than M seconds. That is, the processor (310) can determine whether the length of the user's utterance is acquired within a certain time. If the length of the voice signal is within M seconds, the processor (310) according to one embodiment can perform a similarity determination between the Kth custom keyword included in the voice signal and the K+1th custom keyword included in the voice signal in operation 1009. The operation of determining the similarity between the Kth custom keyword and the K+1th custom keyword according to one embodiment may be similar to the similarity determination operation between the first feature vector corresponding to the first voice signal and the second feature vector corresponding to the second voice signal described with reference to FIGS. 7a to 8. That is, the processor (310) can determine whether the Kth keyword and the K+1th keyword are similar by performing an operation to determine the similarity between the feature vector corresponding to the Kth custom keyword included in the voice signal and the feature vector corresponding to the K+1th custom keyword. According to various embodiments, the keyword setting sensitivity for setting custom keywords may differ from the keyword detection sensitivity. That is, the processor (310) can receive data regarding the keyword setting sensitivity from the user by performing an operation similar to receiving data regarding the keyword detection sensitivity from the user. According to one embodiment, the processor (310) can determine whether the Kth keyword and the K+1th keyword are similar based on a threshold value based on the keyword detection sensitivity or the keyword setting sensitivity.

[0135] According to one embodiment, the processor (310) can increase the value of K by 1 in response to the similarity between the Kth keyword and the K+1th keyword in operation 1011.

[0136] According to one embodiment, if the processor (310) determines that the time taken to acquire the voice signal is M seconds or more, or that the Kth keyword and the K+1th keyword are not similar, it may output a message requesting custom keyword input in operation 1007. For example, the processor (310) may control the display (350) to output a message requesting custom keyword input. According to one embodiment, the processor (310) may perform operations again starting from operation 1001 as a result of requesting custom keyword input.

[0137] According to one embodiment, the processor (310) may compare the value of K with N in operation 1013. According to one embodiment, N may represent the number of repetitions for a preset custom keyword setting. For example, N may represent the number of custom keyword inputs required to provide an always-on voice assistant service using custom keywords. According to one embodiment, if the value of K and the value of N are the same, the processor (310) may terminate the custom keyword setting operation. If the value of K and the value of N are not the same, the processor (310) may obtain a voice signal from the user again in operation 1001.

[0138] According to one embodiment, the processor (310) may set the value of K to 1 and perform operations 1001 to 1013 to perform a custom keyword setting operation. According to one embodiment, the processor (310) may perform operations 1001 to 1013 to perform neural network operations on the set custom keyword to extract a first feature vector, and may store the value of the inverse of the first feature vector in memory (320).

[0139] FIG. 11 is a flowchart (1100) for explaining the keyword similarity operation of an electronic device according to one embodiment.

[0140] Referring to FIG. 11, in operation 1101, the processor (310) can store the first feature vector for each of the first voice signals obtained from the user through the microphone (330) inversely in the buffer memory (311). As another example, the processor (310) can load the value of the inverse of the first feature vector stored in memory (320) into the buffer memory (311).

[0141] In one embodiment, the processor (310) can perform a norm operation in operation 1103 based on a second feature vector based on a second voice signal obtained from a microphone (330) and the value of the inverse of a first feature vector. For example, the processor (310) can perform a norm operation on the value of the inverse of the first feature vector and the second feature vector using an operation circuit (312). The norm operation process according to one embodiment can be performed through an operation similar to the operation described above with reference to FIGS. 7a to 7d. According to one embodiment, the processor (310) can obtain first similarity data and second similarity data described with reference to FIGS. 7a to 7d through a norm operation using the operation circuit (312).

[0142] According to one embodiment, the processor (310) may perform a similarity operation between a first feature vector and a second feature vector based on the result of the norm operation in operation 1105. According to one embodiment, the processor (310) may compare the first similarity data and the second similarity data obtained through the operation circuit (312) with the data for keyword detection sensitivity, respectively, and determine whether to execute the voice assistant app based on the comparison result. According to one embodiment, the similarity operation between the first feature vector and the second feature vector may be performed through the operation of the operation circuit (312) described above with reference to FIG. 8.

[0143] FIG. 12 is a flowchart (1200) for explaining the operation of running a voice assistant app of an electronic device according to one embodiment.

[0144] Referring to FIG. 12, the processor (310) may store the value of the first feature vector inverse and the similarity threshold in the buffer memory (311) to always perform the operation custom keyword detection operation in operation 1201. According to one embodiment, the processor (310) may load the value of the first feature vector inverse and the similarity threshold stored in memory (320) into the buffer memory (311).

[0145] According to one embodiment, the processor (310) may acquire a second voice signal from a user through a microphone (330) in operation 1203. According to one embodiment, the second voice signal acquired through the microphone (330) may represent a voice signal acquired via streaming. According to one embodiment, the processor (310) may perform a similarity operation between a second feature vector and a first feature vector for the second voice signal in operation 1205. For example, the processor (310) may perform a similarity data operation between the value of the inverse of the first feature vector stored in the buffer memory (311) and the second feature vector acquired via streaming through an operation circuit (312). The similarity data operation of the operation circuit (312) according to one embodiment may be performed through an operation process described with reference to FIGS. 7a to 7d.

[0146] According to one embodiment, the processor (310) can compare similarity data and a similarity threshold for keyword detection sensitivity in operation 1207. For example, the processor (310) can compare similarity data and a similarity threshold through an operation circuit (312). The similarity data according to one embodiment may represent data obtained by performing a sigmoid operation on similarity data obtained in operation 1205.

[0147] According to one embodiment, the processor (310) can execute a voice assistant app in operation 1209 when similarity data is greater than a similarity threshold. For example, the processor (310) can execute a voice assistant app by generating an interrupt signal to execute the voice assistant app when similarity data is greater than a similarity threshold and transmitting the generated interrupt signal to a central processing unit. According to one embodiment, the processor (310) can obtain a second voice signal via streaming through the microphone (330) in operation 1203 when similarity data is less than a similarity threshold.

[0148] According to one embodiment, the processor (310) can perform a similarity calculation between a custom keyword for executing a preset voice assistant app and a second voice signal obtained via streaming through operations 1201 to 1209, and perform an operation to execute a voice assistant app based on the custom keyword. For example, if the custom keyword set based on the first voice signal is "Eunhasuya", the processor (310) can calculate a first feature vector corresponding to "Eunhasuya" and store the value of the inverse of the first feature vector in memory (320). Subsequently, when the second voice signal of "Eunhasuya" is obtained via streaming from the user, the processor (310) can calculate a second feature vector corresponding to "Eunhasuya" and perform a similarity calculation between the value of the inverse of the first feature vector and the second feature vector. According to one embodiment, the processor (310) can execute a voice assistant app based on the result of the similarity calculation.

[0149] Each operation of the flowchart described with reference to FIGS. 9 through 12 may be partially omitted and is not necessarily executed in order. Additionally, some of the operations described with reference to FIGS. 9 through 12 may be included and performed in a flowchart described with reference to other drawings.

[0150] FIG. 13 is a drawing illustrating an electronic device that displays a screen for processing voice input received through an intelligent app, according to one embodiment.

[0151] FIG. 13 is a diagram showing a screen in which an electronic device according to various embodiments processes voice input received through an intelligent app.

[0152] An electronic device (300) (e.g., user terminal (100) of FIG. 1) can run an intelligent app to process user input through an intelligent server (200).

[0153] According to one embodiment, on the first screen (1310), the electronic device (300) may execute an intelligent app for processing voice input when it recognizes a designated voice input (e.g., Wake up!) or receives input via a hardware key (e.g., a dedicated hardware key). The electronic device (300) may execute the intelligent app while, for example, a schedule app is running. According to one embodiment, the electronic device (300) may include a display (not shown). The electronic device (300) may display an object (e.g., an icon) (1311) corresponding to the intelligent app on the display (e.g., the display (140) of FIG. 1). According to one embodiment, the electronic device (300) may receive voice input by user speech. For example, the electronic device (300) may receive voice input such as "Tell me this week's schedule!" According to one embodiment, the electronic device (300) can display a UI (user interface) (1312) (e.g., an input window) of an intelligent app on a display that displays text data of a received voice input.

[0154] According to one embodiment, on the second screen (1320), the electronic device (300) can display a result corresponding to the received voice input on the display. For example, the electronic device (300) can receive a plan corresponding to the received user input and display 'this week's schedule' on the display according to the plan.

[0155] FIG. 14 is a drawing illustrating an electronic device that displays a screen for setting custom keywords according to one embodiment.

[0156] FIG. 14 is a drawing showing a screen in which an electronic device according to various embodiments sets a custom keyword.

[0157] An electronic device (300) (e.g., user terminal (100) of FIG. 1) can acquire a first voice signal from a user via a microphone (330) to set a voice assistant app, and set it as a custom keyword. According to one embodiment, the electronic device (300) can execute a voice assistant app based on a keyword similar to the custom keyword being input via the microphone (330).

[0158] According to one embodiment, if the time taken to acquire a first voice signal through the microphone (330) exceeds a set time (e.g., 2 seconds), the processor (310) may output a first guidance message (1411) on the first screen (1410). That is, if the time taken to acquire the first voice signal exceeds a set time, the processor (310) may output a guidance message through the display (350) saying, "Please enter a short custom keyword (can be set to a voice keyword within 2 seconds)."

[0159] According to one embodiment, the processor (310) may receive at least one user utterance through the microphone (330) and obtain a first voice signal corresponding to at least one user utterance in order to perform a custom keyword setting operation. For example, the processor (310) may receive the user utterance repeatedly a set number of times (e.g., N times). The processor (310) may determine whether the custom keyword included in the obtained voice signal is a similar custom keyword. For example, the processor (310) may determine the similarity between the Kth custom keyword included in the Kth voice signal and the K+1th custom keyword included in the K+1th voice signal. The similarity determination operation according to one embodiment may include the operation described with reference to FIG. 10.

[0160] According to one embodiment, the processor (310) can output a second guidance message (1421) on the second screen (1420) if the acquired custom keywords are not similar to each other. That is, if the custom keywords included in at least one user utterance input while acquiring the first voice signal are not similar, the processor (310) can output a guidance message of "Please enter the same custom keyword" through the display (350).

[0161] FIG. 15 is a block diagram of an electronic device (1501) in a network environment (1500) according to various embodiments.

[0162] Referring to FIG. 15, in a network environment (1500), an electronic device (1501) may communicate with an electronic device (1502) through a first network (1598) (e.g., a short-range wireless communication network) or with at least one of an electronic device (1504) or a server (1508) through a second network (1599) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (1501) may communicate with the electronic device (1504) through a server (1508). According to one embodiment, the electronic device (1501) may include a processor (1520), memory (1530), input module (1550), sound output module (1555), display module (1560), audio module (1570), sensor module (1576), interface (1577), connection terminal (1578), haptic module (1579), camera module (1580), power management module (1588), battery (1589), communication module (1590), subscriber identification module (1596), or antenna module (1597). In some embodiments, at least one of these components (e.g., connection terminal (1578)) may be omitted from the electronic device (1501), or one or more other components may be added. In some embodiments, some of these components (e.g., sensor module (1576), camera module (1580), or antenna module (1597)) may be integrated into a single component (e.g., display module (1560)).

[0163] The processor (1520) can, for example, execute software (e.g., program (1540)) to control at least one other component (e.g., hardware or software component) of the electronic device (1501) connected to the processor (1520) and perform various data processing or operations. According to one embodiment, as at least part of the data processing or operations, the processor (1520) can store commands or data received from other components (e.g., sensor module (1576) or communication module (1590)) in volatile memory (1532), process the commands or data stored in volatile memory (1532), and store the resulting data in non-volatile memory (1534). According to one embodiment, the processor (1520) may include a main processor (1521) (e.g., a central processing unit or an application processor) or an auxiliary processor (1523) that can operate independently or together with it (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor), or an accelerator controller (accelerator processor). For example, if the electronic device (1501) includes a main processor (1521) and an auxiliary processor (1523), the auxiliary processor (1523) may be configured to use less power than the main processor (1521) or to be specialized for a specified function. The auxiliary processor (1523) may be implemented separately from the main processor (1521) or as part thereof.

[0164] The auxiliary processor (1523) may control at least some of the functions or states associated with at least one component of the electronic device (1501) (e.g., display module (1560), sensor module (1576), or communication module (1590)) on behalf of the main processor (1521) while the main processor (1521) is in an inactive (e.g., sleep) state, or together with the main processor (1521) while the main processor (1521) is in an active (e.g., application execution) state. According to one embodiment, the auxiliary processor (1523) (e.g., image signal processor or communication processor) may be implemented as part of another functionally related component (e.g., camera module (1580) or communication module (1590)). According to one embodiment, the auxiliary processor (1523) (e.g., neural network processing unit) may include a hardware structure specialized for processing an artificial intelligence model. The artificial intelligence model may be generated through machine learning. Such learning may be performed, for example, on the electronic device (1501) itself where the artificial intelligence model is executed, or through a separate server (e.g., server (1508)). The learning algorithm may include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model may include a plurality of artificial neural network layers.An artificial neural network may be a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to the hardware structure, the artificial intelligence model may include a software structure, either additionally or substantially.

[0165] The memory (1530) can store various data used by at least one component of the electronic device (1501) (e.g., processor (1520) or sensor module (1576)). The data may include, for example, input data or output data for software (e.g., program (1540)) and related commands. The memory (1530) may include volatile memory (1532) or non-volatile memory (1534).

[0166] The program (1540) may be stored as software in memory (1530) and may include, for example, an operating system (1542), middleware (1544), or an application (1546).

[0167] The input module (1550) can receive commands or data to be used for a component of the electronic device (1501) (e.g., processor (1520)) from outside the electronic device (1501) (e.g., user). The input module (1550) may include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0168] The sound output module (1555) can output a sound signal to the outside of the electronic device (1501). The sound output module (1555) may include, for example, a speaker or a receiver. The speaker may be used for general purposes, such as multimedia playback or recording playback. The receiver may be used to receive incoming calls. According to one embodiment, the receiver may be implemented separately from the speaker or as part thereof.

[0169] The display module (1560) can visually provide information to an external (e.g., user) of the electronic device (1501). The display module (1560) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling said device. According to one embodiment, the display module (1560) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of the force generated by said touch.

[0170] The audio module (1570) can convert sound into an electrical signal or, conversely, convert an electrical signal into sound. According to one embodiment, the audio module (1570) can acquire sound through an input module (1550) or output sound through an audio output module (1555) or an external electronic device (e.g., electronic device (1502)) (e.g., speaker or headphones) connected directly or wirelessly to the electronic device (1501).

[0171] The sensor module (1576) can detect the operating state of the electronic device (1501) (e.g., power or temperature) or the external environmental state (e.g., user state) and generate an electrical signal or data value corresponding to the detected state. According to one embodiment, the sensor module (1576) may include, for example, a gesture sensor, a gyroscope sensor, a barometric pressure sensor, a magnetic sensor, an accelerometer sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biosensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0172] The interface (1577) may support one or more specified protocols that can be used for the electronic device (1501) to be connected directly or wirelessly to an external electronic device (e.g., electronic device (1502)). According to one embodiment, the interface (1577) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0173] The connection terminal (1578) may include a connector through which the electronic device (1501) can be physically connected to an external electronic device (e.g., electronic device (1502)). According to one embodiment, the connection terminal (1578) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0174] The haptic module (1579) can convert an electrical signal into a mechanical stimulus (e.g., vibration or movement) or an electrical stimulus that the user can perceive through tactile or kinesthetic senses. According to one embodiment, the haptic module (1579) may include, for example, a motor, a piezoelectric element, or an electric stimulation device.

[0175] The camera module (1580) can capture still images and video. According to one embodiment, the camera module (1580) may include one or more lenses, image sensors, image signal processors, or flashes.

[0176] The power management module (1588) can manage the power supplied to the electronic device (1501). According to one embodiment, the power management module (1588) can be implemented, for example, as at least part of a power management integrated circuit (PMIC).

[0177] The battery (1589) can supply power to at least one component of the electronic device (1501). According to one embodiment, the battery (1589) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0178] The communication module (1590) can support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between an electronic device (1501) and an external electronic device (e.g., electronic device (1502), electronic device (1504), or server (1508)), and the performance of communication through the established communication channel. The communication module (1590) may include one or more communication processors that operate independently of the processor (1520) (e.g., application processor) and support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (1590) may include a wireless communication module (1592) (e.g., cellular communication module, short-range wireless communication module, or GNSS (global navigation satellite system) communication module) or a wired communication module (1594) (e.g., LAN (local area network) communication module, or power line communication module). The corresponding communication module among these communication modules can communicate with an external electronic device (1504) via a first network (1598) (e.g., a short-range communication network such as Bluetooth, WiFi (wireless fidelity) direct, or IrDA (infrared data association)) or a second network (1599) (e.g., a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN). These various types of communication modules may be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (1592) can identify or authenticate the electronic device (1501) within a communication network such as the first network (1598) or the second network (1599) using subscriber information (e.g., International Mobile Subscriber Identifier (IMSI)) stored in the subscriber identification module (1596).

[0179] The wireless communication module (1592) can support 5G networks and next-generation communication technologies following 4G networks, for example, new radio access technology. NR access technology can support high-speed transmission of high-capacity data (enhanced mobile broadband (eMBB)), minimization of terminal power and connection of multiple terminals (massive machine type communications (mMTC)), or high reliability and low latency (ultra-reliable and low-latency communications (URLLC)). The wireless communication module (1592) can support a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate, for example. The wireless communication module (1592) can support various technologies for securing performance in the high-frequency band, such as beamforming, massive MIMO (multiple-input and multiple-output), full-dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large-scale antenna. The wireless communication module (1592) can support various requirements specified in the electronic device (1501), external electronic device (e.g., electronic device (1504)), or network system (e.g., second network (1599)). According to one embodiment, the wireless communication module (1592) can support a Peak data rate (e.g., 20 Gbps or more) for realizing eMBB, loss coverage (e.g., 164 dB or less) for realizing mMTC, or U-plane latency (e.g., downlink (DL) and uplink (UL) each 0.5 ms or less, or round trip 1 ms or less) for realizing URLLC.

[0180] An antenna module (1597) can transmit a signal or power to or from an external source (e.g., an external electronic device). According to one embodiment, the antenna module (1597) may include an antenna comprising a radiator made of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). According to one embodiment, the antenna module (1597) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as a first network (1598) or a second network (1599), may be selected from the plurality of antennas, for example, by a communication module (1590). A signal or power may be transmitted or received between the communication module (1590) and an external electronic device through the selected at least one antenna. According to some embodiments, in addition to the radiator, other components (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as part of the antenna module (1597).

[0181] According to various embodiments, the antenna module (1597) may form a mmWave antenna module. According to one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent to a first surface (e.g., bottom surface) of the printed circuit board and capable of supporting a specified high frequency band (e.g., mmWave band), and a plurality of antennas (e.g., array antennas) disposed on or adjacent to a second surface (e.g., top surface or side surface) of the printed circuit board and capable of transmitting or receiving a signal of the specified high frequency band.

[0182] At least some of the above components can be connected to each other via a communication method between peripheral devices (e.g., bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)) and exchange signals (e.g., commands or data) with each other.

[0183] According to one embodiment, commands or data may be transmitted or received between the electronic device (1501) and an external electronic device (1504) through a server (1508) connected to a second network (1599). Each of the external electronic devices (1502, or 1504) may be the same or a different type of device as the electronic device (1501). According to one embodiment, all or part of the operations performed on the electronic device (1501) may be performed on one or more of the external electronic devices (1502, 1504, or 1508). For example, if the electronic device (1501) needs to perform a function or service automatically or in response to a request from a user or another device, the electronic device (1501) may request one or more external electronic devices to perform at least part of the function or service instead of performing the function or service itself or additionally. One or more external electronic devices that receive the above request may execute at least part of the requested function or service, or additional function or service related to the request, and transmit the result of the execution to the electronic device (1501). The electronic device (1501) may provide the result as is or additionally processed as at least part of the response to the request. For this purpose, for example, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used. The electronic device (1501) may provide ultra-low latency services using, for example, distributed computing or mobile edge computing. In another embodiment, the external electronic device (1504) may include an Internet of Things (IoT) device. The server (1508) may be an intelligent server using machine learning and / or neural networks.According to one embodiment, an external electronic device (1504) or server (1508) may be included within the second network (1599). The electronic device (1501) may be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0184] The electronic device according to the various embodiments disclosed in this document may be of various forms. The electronic device may include, for example, a portable communication device (e.g., a smartphone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a consumer electronics device. The electronic device according to the embodiments of this document is not limited to the devices described above.

[0185] The various embodiments of this document and the terms used therein are not intended to limit the technical features described in this document to specific embodiments, and should be understood to include various modifications, equivalents, or substitutions of said embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of said items unless the relevant context clearly indicates otherwise. In this document, phrases such as "A or B," "at least one of A and B," "at least one of A or B," "A, B or C," "at least one of A, B and C," and "at least one of A, B, or C" may each include any one of the items listed together in the corresponding phrase, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used simply to distinguish said components from other said components and do not limit said components in any other aspect (e.g., importance or order). Where any (e.g., 1st) component is referred to as “coupled” or “connected” to another (e.g., 2nd) component, with or without the terms “functionally” or “communicationly,” it means that said any component may be connected to said other component directly (e.g., via a wire), wirelessly, or through a third component.

[0186] The term “module” as used in the various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit, for example. A module may be a component formed integrally, or a minimum unit of said component or a part thereof that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0187] Various embodiments of the present document may be implemented as software (e.g., program (1540)) comprising one or more instructions stored in a storage medium (e.g., internal memory (1536) or external memory (1538)) readable by a machine (e.g., electronic device (1501)). For example, a processor (e.g., processor (1520)) of the machine (e.g., electronic device (1501)) may call at least one of the one or more instructions stored from the storage medium and execute it. This enables the machine to be operated to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code that can be executed by an interpreter. The storage medium readable by the machine may be provided in the form of a non-transitory storage medium. Here, 'non-temporary' simply means that the storage medium is a tangible device and does not contain a signal (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently and cases where it is stored temporarily.

[0188] According to one embodiment, the method according to the various embodiments disclosed herein may be provided by being included in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)) or an application store (e.g., Play Store). TM It can be distributed online (e.g., downloaded or uploaded) through ) or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily created on a device-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0189] According to various embodiments, each component (e.g., module or program) of the components described above may include a singular or multiple entities, and some of the multiple entities may be separated and placed in other components. According to various embodiments, one or more of the components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Generally or additionally, multiple components (e.g., module or program) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each of the multiple components in the same or similar manner as those performed by the corresponding component among the multiple components prior to integration. According to various embodiments, operations performed by the module, program, or other components may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

[0190] As described above, an electronic device according to one embodiment (e.g., the electronic device (300) of FIG. 3) comprises a microphone (e.g., the microphone (330) of FIG. 3) and at least one processor (e.g., the processor (310) of FIG. 3) operatively connected to the microphone, and the at least one processor may include a buffer memory (e.g., the buffer memory (311) of FIG. 3) that stores a first feature vector for a first voice signal obtained from the microphone as an inverse value, and an operation circuit (e.g., the operation circuit (312) of FIG. 3) configured to perform a norm operation on the first feature vector and the second feature vector based on a second feature vector based on a second voice signal streamed from the microphone and the inverse value of the first feature vector stored in the buffer memory, or to calculate the similarity between the first feature vector and the second feature vector.

[0191] According to one embodiment, the at least one processor can input the first voice signal into a convolution layer to obtain the first feature vector, and input the second voice signal into the convolution layer to obtain the second feature vector.

[0192] According to one embodiment, the operation circuit may include at least one multiplexer that selectively selects and outputs one of external data input from the buffer memory and data input from a configuration included in the operation circuit, at least one accumulator that accumulates the output of the at least one multiplexer, an adder that performs an addition operation based on the output value of the at least one multiplexer, and a multiplier that performs a multiplication operation based on the output value of the at least one multiplexer.

[0193] In one embodiment, the operation circuit may include a two's complement circuit that performs a two's complement operation based on the output value of the accumulator.

[0194] According to one embodiment, the at least one multiplexer may include a first multiplexer that receives a second feature vector from the buffer memory, a second multiplexer that receives a value of 1 from outside the operation circuit, and a third multiplexer that receives the inverse value of the first feature vector from the buffer memory.

[0195] According to one embodiment, the at least one processor may receive input regarding similarity from a user of the electronic device and store an inverse value regarding similarity as a similarity threshold in the buffer memory.

[0196] According to one embodiment, the at least one processor can perform a comparison operation on the result data of the norm operation and a similarity threshold through the operation circuit, and determine whether to execute the voice assistant app based on the comparison result output from the operation circuit.

[0197] According to one embodiment, the at least one processor may obtain subtraction absolute value data by calculating the difference between the inverse value of the first feature vector and the second feature vector through the calculation circuit, obtain first similarity data by performing a sigmoid operation on the subtraction absolute value data, and perform a comparison operation on the first similarity data and the similarity threshold value.

[0198] According to one embodiment, the at least one processor may obtain subtraction square data by calculating the square of the difference between the inverse value of the first feature vector and the second feature vector through the calculation circuit, obtain second similarity data by performing a sigmoid operation on the subtraction square data, and perform a comparison operation on the second similarity data and the similarity threshold.

[0199] According to one embodiment, the at least one processor can generate an interrupt signal for executing a voice assistant app and transmit the generated interrupt signal to a central processing unit (CPU).

[0200] According to one embodiment, the at least one processor can acquire at least one user utterance through the microphone, acquire the first feature vector for the first voice signal included in the at least one user utterance, and calculate the similarity of the first feature vector corresponding to the at least one user utterance through the calculation circuit.

[0201] According to one embodiment, the at least one processor can obtain the first feature vector by calculating the feature vector for the first voice signal corresponding to the at least one user utterance as a representative value.

[0202] In one embodiment, the device further includes a communication circuit configured to transmit and receive data to and from an external server, and the at least one processor can transmit and receive the first voice signal and the second voice signal to and from an intelligent server through the communication circuit.

[0203] According to one embodiment, the electronic device (300) further includes a display, and the at least one processor can control the display to display guidance for inputting a custom keyword to run a voice assistant app.

[0204] As described above, a method of operation of an electronic device including a calculation circuit configured to perform a norm operation and a similarity operation according to one embodiment may include the operation of inversely storing a first feature vector for a first voice signal obtained from a user of the electronic device through a microphone included in the electronic device in a memory included in the electronic device, the operation of performing a norm operation based on a second feature vector based on a second voice signal obtained via streaming from the microphone and the value of the inverse of the first vector, and the operation of calculating the similarity between the first feature vector and the second feature vector based on the result of the operation.

[0205] The method of operation of an electronic device according to one embodiment may further include the operation of executing a voice assistant app based on the calculation result of the similarity.

[0206] According to one embodiment, the operation of performing the norm operation may include the operation of obtaining subtraction absolute value data or subtraction square data by calculating the difference between the inverse value of the first feature vector and the second feature vector through the operation circuit, and the operation of obtaining similarity data by performing a sigmoid operation on the subtraction absolute value data.

[0207] According to one embodiment, the method further includes the operation of receiving input regarding similarity from a user of the electronic device and the operation of storing the inverse value of the similarity as a similarity threshold value in memory, and the operation of calculating the similarity may include the operation of performing a comparison operation on the similarity data and the similarity threshold value through the calculation circuit and the operation of determining whether to execute a voice assistant app based on the result of the calculation.

[0208] According to one embodiment, the operation circuit may include at least one multiplexer that selectively selects and outputs one of external data input from a buffer memory included in an electronic device and data input from a configuration included in the operation circuit; at least one accumulator that accumulates the output of the at least one multiplexer; an adder that performs an addition operation based on the output value of the at least one multiplexer; a multiplier that performs a multiplication operation based on the output value of the at least one multiplexer; and a two's complement circuit that performs a two's complement operation based on the output value of the accumulator.

[0209] As described above, a keyword detection accelerator for performing neural network operations according to one embodiment (e.g., keyword detection accelerator (3100) of FIG. 4) comprises an audio signal processor (e.g., MFCC (530) of FIG. 5), a buffer memory (e.g., buffer memory (3110) of FIG. 4), an operation circuit capable of performing norm operations (e.g., operation circuit (3120) of FIG. 4), and at least one processor (e.g., accelerator controller (3130) of FIG. 4) operatively connected to the audio signal processor, the buffer memory, and the operation circuit, wherein the at least one processor stores a first feature vector for a first voice signal obtained through the audio signal processor as an inverse value in the buffer memory, obtains a second voice signal streamed through the audio signal processor, and performs norm operations on the first feature vector and the second feature vector based on the second feature vector for the second voice signal and the inverse value of the first feature vector through the operation circuit, or the first feature The similarity between the vector and the second feature vector can be calculated.

Claims

Claim 1 An electronic device comprising: a microphone; at least one processor operatively connected to the microphone; wherein the at least one processor comprises a buffer memory that stores a first feature vector for a first voice signal obtained from the microphone as an inverse value, and a second feature vector based on a second voice signal streamed from the microphone and the inverse value of the first feature vector stored in the buffer memory, wherein the norm operation performs a norm operation on the first feature vector and the second feature vector based on a second feature vector based on a second voice signal streamed from the microphone and the inverse value of the first feature vector stored in the buffer memory, wherein the norm operation includes at least one of an operation to calculate the absolute value of the sum of the inverse value of the first feature vector and the second feature vector, or an operation to calculate the square value of the sum of the inverse value of the first feature vector and the second feature vector, and wherein the electronic device comprises an operation circuit configured to calculate the similarity between the first feature vector and the second feature vector using the result of the norm operation. Claim 2 An electronic device according to claim 1, wherein the at least one processor inputs the first voice signal to a convolution layer to obtain the first feature vector and inputs the second voice signal to the convolution layer to obtain the second feature vector. Claim 3 An electronic device according to claim 1, wherein the operation circuit comprises: at least one multiplexer that selectively selects and outputs one of external data input from the buffer memory and data input from a configuration included in the operation circuit; at least one accumulator that accumulates the output of the at least one multiplexer; an adder that performs an addition operation based on the output value of the at least one multiplexer; and a multiplier that performs a multiplication operation based on the output value of the at least one multiplexer. Claim 4 An electronic device according to claim 3, wherein the operation circuit comprises a two's complement circuit that performs a two's complement operation based on the output value of the accumulator. Claim 5 An electronic device according to claim 3, wherein the at least one multiplexer comprises: a first multiplexer receiving the second feature vector from the buffer memory; a second multiplexer receiving a value of 1 from outside the operation circuit; and a third multiplexer receiving the inverse value of the first feature vector from the buffer memory. Claim 6 delete Claim 7 An electronic device according to claim 1, wherein the at least one processor performs a comparison operation on the result data of the norm operation and a similarity threshold through the operation circuit, and determines whether to execute a voice assistant app based on the comparison result output from the operation circuit. Claim 8 The electronic device according to claim 1, wherein the absolute value includes the absolute value of the difference between the first feature vector and the second feature vector, and the at least one processor performs a sigmoid operation on the absolute value to obtain first similarity data and performs a comparison operation on the first similarity data and a similarity threshold. Claim 9 An electronic device according to claim 1, wherein the squared value includes the squared value of the difference between the first feature vector and the second feature vector, and the at least one processor: performs a sigmoid operation on the squared value to obtain second similarity data, and performs a comparison operation on the second similarity data and a similarity threshold. Claim 10 delete Claim 11 delete Claim 12 delete Claim 13 delete Claim 14 delete Claim 15 A method of operation of an electronic device comprising an operation circuit configured to perform a norm operation and a similarity operation, the method comprising: an operation of inversely storing a first feature vector for a first voice signal obtained from a user of the electronic device through a microphone included in the electronic device in a memory included in the electronic device; an operation of performing the norm operation based on a second feature vector based on a second voice signal obtained by streaming from the microphone and the inverse value of the first feature vector, wherein the norm operation includes at least one of an operation of calculating the absolute value of the sum of the inverse value of the first feature vector and the second feature vector, or an operation of calculating the square value of the sum of the inverse value of the first feature vector and the second feature vector; and an operation of performing the similarity operation of the first feature vector and the second feature vector based on the result of the norm operation. Claim 16 delete Claim 17 delete Claim 18 delete Claim 19 delete Claim 20 A keyword detection accelerator for performing neural network operations comprises: an audio signal processor; a buffer memory; an operation circuit capable of performing a norm operation; and at least one processor operatively connected to the audio signal processor, the buffer memory, and the operation circuit, wherein the at least one processor stores a first feature vector for a first voice signal obtained through the audio signal processor as an inverse value in the buffer memory, obtains a second voice signal streamed through the audio signal processor, and performs the norm operation on the first feature vector and the second feature vector based on a second feature vector for the second voice signal and the inverse value of the first feature vector through the operation circuit, wherein the norm operation includes at least one of an operation to calculate the absolute value of the sum of the inverse value of the first feature vector and the second feature vector, or an operation to calculate the square value of the sum of the inverse value of the first feature vector and the second feature vector, and calculates the similarity of the first feature vector and the second feature vector using the result of the norm operation.

Images