Electronic apparatus and control method thereof

Abstract

Disclosed are an electronic apparatus and a control method thereof. In particular, the electronic apparatus may comprise: a microphone; a memory (120) for storing at least one instruction; and a processor that executes the at least one instruction. The processor may: when a speech signal is received through the microphone, acquire a first score indicating a probability that the speech signal corresponds to a pre-registered keyword, by inputting each of frames included in the speech signal into a character identifier; acquire a speech feature vector indicating a feature of the speech signal, by inputting each of the frames into a speech encoder; identify whether or not the keyword is included in the speech signal, by inputting the first score, the speech feature vector, and a text feature vector indicating a feature of the keyword into a keyword detector; and when it is identified that the keyword is included in the speech signal, activate a speech recognition function of the electronic apparatus.

Description

Electronic device and method of controlling the same

[0001] The present disclosure relates to an electronic device and a method for controlling the electronic device, and more particularly, to an electronic device capable of detecting a keyword for activating a voice recognition function and a method for controlling the same.

[0002] With the recent advancement of technologies related to artificial intelligence (AI), voice recognition technologies using voice recognition models are rapidly developing.

[0003] However, because achieving high accuracy in speech recognition models requires massive amounts of data and significant computational effort, keeping the model always active can result in excessive power consumption. Therefore, recent technologies have been developed that keep the voice recognition model on standby, activating it only when the user utters a pre-registered keyword.

[0004] Among the conventional techniques for keyword spotting (KWS), there is a technique for registering keywords by having the user utter the keyword he or she wishes to register several times. However, this technique has been criticized for its limitations in that it may cause inconvenience to the user because it requires the user to utter the keyword several times, and inaccurate keywords may be entered.

[0005] Meanwhile, existing technologies utilize non-streaming keyword detection techniques that identify whether a keyword is spoken within a preset speech interval (e.g., 1.3 seconds). While these techniques offer superior performance because they consider the entire keyword and its surrounding context, they also require a significant amount of computation, as the model must process the entire speech signal simultaneously.

[0006] In addition, non-streaming keyword detection technology has limitations such as the difficulty in recognizing keywords uttered in a section exceeding the size of the utterance section, the possibility of including a lot of noise when detecting keywords uttered in a section shorter than the size of the utterance section, and the fact that the non-streaming method requires knowledge of the start and end points of the utterance.

[0007] Accordingly, there is a need for a technology that can effectively compare the registered keyword with the user's voice received in real time even when the user registers the keyword by text input, and can efficiently detect the keyword by processing one frame at a time (e.g., 25 ms) in real time according to the streaming method.

[0008] The present disclosure is intended to overcome the limitations of the prior art as described above, and an object of the present disclosure is to provide an electronic device and a control method thereof capable of detecting a keyword for activating a voice recognition function in an effective and efficient manner.

[0009] According to one or more embodiments of the present disclosure for achieving the above-described object, an electronic device includes a microphone, a memory storing at least one instruction, and a processor executing the at least one instruction, wherein when a voice signal is received through the microphone, the processor inputs each of the frames included in the voice signal into a character identifier to obtain a first score indicating a probability that the voice signal corresponds to a pre-registered keyword, inputs each of the frames into a voice encoder to obtain a voice feature vector indicating a characteristic of the voice signal, inputs the first score, the voice feature vector, and a text feature vector indicating a characteristic of the keyword into a keyword detector to identify whether the voice signal includes the keyword, and when it is identified that the voice signal includes the keyword, the processor activates a voice recognition function of the electronic device.

[0010] Meanwhile, when the processor receives a user input for entering text corresponding to the keyword, the processor can input the text into a text encoder to obtain the text feature vector, and store the text feature vector in the memory.

[0011] Meanwhile, the processor inputs each of the frames into the character identifier and the voice encoder each time each of the frames is received, and the character identifier obtains probability information about the probability that each of the frames corresponds to each of the plurality of characters included in the keyword, and based on the probability information, identifies characters among the plurality of characters for each of the frames that have the highest probability of corresponding to each of the frames, and accumulates probabilities corresponding to the identified characters, thereby obtaining the first score.

[0012] Meanwhile, the character identifier updates information about the probability that the frames input up to now among the frames include a frame corresponding to each of the plurality of characters based on the probability information, thereby identifying characters having the highest probability of corresponding to each of the frames among the plurality of characters for each of the frames.

[0013] Meanwhile, the speech feature vector includes a plurality of first speech feature vectors representing the characteristics of each of the frames and a plurality of second speech feature vectors representing the characteristics of each of the identified characters, and the speech encoder can obtain the plurality of first speech feature vectors by obtaining the first speech feature vector representing the characteristics of each of the frames whenever each of the frames is input.

[0014] Meanwhile, the keyword detector obtains a second score indicating a similarity between the voice feature vector and the text feature vector, obtains a weighted sum of the first score and the second score, and if the weighted sum is greater than a preset threshold value, it can identify that the keyword is included in the voice signal.

[0015] Meanwhile, the keyword detector can obtain a plurality of second voice feature vectors by combining first voice feature vectors having the same identified characters among the plurality of first voice feature vectors based on information about the identified characters and information about the time at which each of the frames was received.

[0016] Meanwhile, the text feature vector includes a plurality of character feature vectors representing the features of each of the plurality of characters, and the keyword detector can obtain the second score by comparing the plurality of second speech feature vectors with the characters corresponding to the plurality of character feature vectors.

[0017] Meanwhile, each of the character identifier, the voice encoder, and the text encoder includes a plurality of neural networks, and the processor can obtain a loss value based on the first score and the second score, and train at least one of the character identifier, the voice encoder, and the text encoder based on the loss value.

[0018] Meanwhile, the size of each of the above frames can be determined within the range of 20 ms to 40 ms.

[0019] According to one or more embodiments of the present disclosure for achieving the above-described object, a method for controlling an electronic device includes the steps of: when a voice signal is received through a microphone, inputting each of the frames included in the voice signal into a character identifier to obtain a first score indicating a probability that the voice signal corresponds to a pre-registered keyword; inputting each of the frames into a voice encoder to obtain a voice feature vector indicating a characteristic of the voice signal; inputting the first score, the voice feature vector, and a text feature vector indicating a characteristic of the keyword into a keyword detector to identify whether the voice signal includes the keyword; and when it is identified that the voice signal includes the keyword, activating a voice recognition function of the electronic device.

[0020] Meanwhile, the control method of the electronic device may further include, when a user input for inputting text corresponding to the keyword is received, a step of inputting the text into a text encoder to obtain the text feature vector, and a step of storing the text feature vector in the memory.

[0021] Meanwhile, the step of obtaining the first score includes a step of inputting each of the frames into the character identifier each time each of the frames is received, and the step of obtaining the voice feature vector includes a step of inputting each of the frames into the voice encoder each time each of the frames is received, and the character identifier obtains probability information about a probability that each of the frames corresponds to each of the plurality of characters included in the keyword, and based on the probability information, identifies characters having the highest probability of corresponding to each of the frames among the plurality of characters for each of the frames, and accumulates probabilities corresponding to the identified characters, thereby obtaining the first score.

[0022] Meanwhile, the character identifier updates information about the probability that the frames input up to now among the frames include a frame corresponding to each of the plurality of characters based on the probability information, thereby identifying characters having the highest probability of corresponding to each of the frames among the plurality of characters for each of the frames.

[0023] Meanwhile, the speech feature vector includes a plurality of first speech feature vectors representing the characteristics of each of the frames and a plurality of second speech feature vectors representing the characteristics of each of the identified characters, and the speech encoder can obtain the plurality of first speech feature vectors by obtaining the first speech feature vector representing the characteristics of each of the frames whenever each of the frames is input.

[0024] Meanwhile, the keyword detector obtains a second score indicating a similarity between the voice feature vector and the text feature vector, obtains a weighted sum of the first score and the second score, and if the weighted sum is greater than a preset threshold value, it can identify that the keyword is included in the voice signal.

[0025] Meanwhile, the keyword detector can obtain a plurality of second voice feature vectors by combining first voice feature vectors having the same identified characters among the plurality of first voice feature vectors based on information about the identified characters and information about the time at which each of the frames was received.

[0026] Meanwhile, the text feature vector includes a plurality of character feature vectors representing the features of each of the plurality of characters, and the keyword detector can obtain the second score by comparing the plurality of second speech feature vectors with the characters corresponding to the plurality of character feature vectors.

[0027] Meanwhile, each of the character identifier, the voice encoder, and the text encoder includes a plurality of neural networks, and the control method of the electronic device may include a step of obtaining a loss value based on the first score and the second score, and a step of training at least one of the character identifier, the voice encoder, and the text encoder based on the loss value.

[0028] According to one or more embodiments of the present disclosure for achieving the above-described object, there is provided a non-transitory computer-readable recording medium including a program for executing a method for controlling an electronic device, the method comprising: when a voice signal is received through a microphone, inputting each of the frames included in the voice signal into a character identifier to obtain a first score indicating a probability that the voice signal corresponds to a pre-registered keyword; inputting each of the frames into a voice encoder to obtain a voice feature vector indicating a characteristic of the voice signal; inputting the first score, the voice feature vector, and a text feature vector indicating a characteristic of the keyword into a keyword detector to identify whether the voice signal includes the keyword; and when it is identified that the voice signal includes the keyword, activating a voice recognition function of the electronic device.

[0029] Other aspects, features and advantages of one or more embodiments according to the present disclosure will become more apparent from the following detailed description taken in conjunction with the accompanying drawings.

[0030] FIG. 1 is a schematic diagram illustrating a configuration of an electronic device according to one or more embodiments of the present disclosure;

[0031] FIG. 2 is a diagram illustrating a plurality of modules according to one or more embodiments of the present disclosure;

[0032] FIG. 3 is a diagram illustrating a plurality of modules according to one or more embodiments of the present disclosure;

[0033] FIG. 4 is a drawing for explaining in detail the process of obtaining the first score and the second score according to one or more embodiments of the present disclosure;

[0034] FIG. 5 is a drawing detailing the configuration of an electronic device according to one or more embodiments of the present disclosure, and

[0035] FIG. 6 is a flowchart illustrating a method for controlling an electronic device according to one or more embodiments of the present disclosure.

[0036] The present embodiments may be modified and have various embodiments. Specific embodiments are illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the scope to specific embodiments, but should be understood to encompass various modifications, equivalents, and / or alternatives of the embodiments of the present disclosure. In connection with the description of the drawings, similar reference numerals may be used for similar components.

[0037] In describing the present disclosure, if it is determined that a specific description of a related known function or configuration may unnecessarily obscure the gist of the present disclosure, a detailed description thereof will be omitted.

[0038] Additionally, the following embodiments may be modified in various other forms, and the scope of the technical concepts of the present disclosure is not limited to the following embodiments. Rather, these embodiments are provided to further faithfully and completely convey the technical concepts of the present disclosure to those skilled in the art.

[0039] The terminology used in this disclosure is for the purpose of describing specific embodiments only and is not intended to limit the scope of the rights. Singular expressions include plural expressions unless the context clearly dictates otherwise.

[0040] In this disclosure, expressions such as “has,” “can have,” “includes,” or “may include” indicate the presence of a corresponding feature (e.g., a component such as a number, function, operation, or part), and do not exclude the presence of additional features.

[0041] In this disclosure, expressions such as “A or B,” “at least one of A and / or B,” or “one or more of A or / and B” can include all possible combinations of the listed items. For example, “A or B,” “at least one of A and B,” or “at least one of A or B” can all refer to (1) including at least one A, (2) including at least one B, or (3) including both at least one A and at least one B.

[0042] The expressions “first,” “second,” “first,” or “second,” etc., used in this disclosure can describe various components, regardless of order and / or importance, and are only used to distinguish one component from another, but do not limit the components.

[0043] When it is said that a component (e.g., a first component) is “(operatively or communicatively) coupled with / to” or “connected to” another component (e.g., a second component), it should be understood that said component may be directly coupled to said other component, or may be coupled via another component (e.g., a third component).

[0044] On the other hand, when it is said that a component (e.g., a first component) is "directly connected" or "directly connected" to another component (e.g., a second component), it can be understood that no other component (e.g., a third component) exists between said component and said other component.

[0045] The expression "configured to" as used in the present disclosure may be used interchangeably with, for example, "suitable for," "having the capacity to," "designed to," "adapted to," "made to," or "capable of." The term "configured to" may not necessarily mean only "specifically designed to" in terms of hardware.

[0046] Instead, in some contexts, the phrase "a device configured to" may mean that the device, in conjunction with other devices or components, is "capable of" performing A, B, and C. For example, the phrase "a processor configured (or set) to perform A, B, and C" may refer to a dedicated processor (e.g., an embedded processor) for performing those operations, or a general-purpose processor (e.g., a CPU or application processor) that can perform those operations by executing one or more software programs stored in a memory device.

[0047] In the embodiments, a 'module' or 'part' performs at least one function or operation, and may be implemented as hardware or software, or as a combination of hardware and software. Furthermore, a plurality of 'modules' or 'parts' may be integrated into at least one module and implemented as at least one processor, except for a 'module' or 'part' that needs to be implemented as a specific hardware.

[0048] Meanwhile, the various elements and areas in the drawings are schematically drawn. Therefore, the technical concept of the present invention is not limited by the relative sizes or spacing depicted in the attached drawings.

[0049] Meanwhile, an electronic device according to various embodiments of the present disclosure may include, for example, at least one of a smart phone, a tablet PC, a desktop PC, a laptop PC, or a wearable device. The wearable device may include at least one of an accessory type (e.g., a watch, a ring, a bracelet, an anklet, a necklace, glasses, contact lenses, or a head-mounted device (HMD)), a fabric or clothing-integrated type (e.g., an electronic garment), a body-attached type (e.g., a skin pad or tattoo), or a bio-implantable circuit.

[0050] In some embodiments, the electronic device may be, for example, a television, a digital video disk (DVD) player, an audio, a refrigerator, an air conditioner, a vacuum cleaner, an oven, a microwave oven, a washing machine, an air purifier, a set-top box, a home automation control panel, a security control panel, a media box (e.g., Samsung HomeSync). TM , Apple TV TM , or Google TV TM ), game consoles (e.g. Xbox TM , PlayStation TM ), may include at least one of an electronic dictionary, an electronic key, a camcorder, or an electronic picture frame.

[0051] In another embodiment, the electronic device may include at least one of various medical devices (e.g., various portable medical measuring devices (e.g., blood glucose meters, heart rate monitors, blood pressure monitors, or body temperature monitors), magnetic resonance angiography (MRA), magnetic resonance imaging (MRI), computed tomography (CT), cameras, or ultrasound machines), navigation devices, global navigation satellite systems (GNSS), event data recorders (EDR), flight data recorders (FDR), automotive infotainment devices, marine electronic equipment (e.g., marine navigation devices, gyrocompasses, etc.), avionics, security devices, vehicle head units, industrial or domestic robots, drones, ATMs of financial institutions, point of sales (POS) of stores, or Internet of Things devices (e.g., light bulbs, various sensors, sprinkler devices, fire alarms, thermostats, streetlights, toasters, exercise equipment, hot water tanks, heaters, boilers, etc.).

[0052] Hereinafter, with reference to the attached drawings, embodiments according to the present disclosure will be described in detail so that a person having ordinary knowledge in the technical field to which the present disclosure pertains can easily implement the present disclosure.

[0053] FIG. 1 is a schematic diagram illustrating the configuration of an electronic device (100) according to one or more embodiments of the present disclosure. FIG. 2 is a diagram illustrating a plurality of modules according to one or more embodiments of the present disclosure. The following description will be made with reference to FIG. 1 and FIG. 2 together.

[0054] As illustrated in FIG. 1, the electronic device (100) may include a microphone (110), a memory (120), and a processor (130).

[0055] The microphone (110) can acquire a signal for a sound or voice generated from outside the electronic device (100). Specifically, the microphone (110) can acquire a vibration according to a sound or voice generated from outside the electronic device (100) and convert the acquired vibration into an electrical signal.

[0056] In particular, the microphone (110) according to the present disclosure can acquire a voice signal for a user's voice generated by the user's speech. In addition, the acquired signal can be converted into a digital signal and stored in a memory (120). The microphone (110) can include an A / D converter (Analog to Digital Converter) and can also operate in conjunction with an A / D converter located outside the microphone (110).

[0057] In one or more embodiments, the processor (130) may receive a voice signal corresponding to a user's voice through the microphone (110) in units of preset frames. A 'frame' refers to a portion of a voice signal divided by a preset size, i.e., a length of time. Here, the preset size, i.e., the size of each frame, may be determined within a range of 20 ms to 40 ms. The processor (130) may continuously receive frames through the microphone (110) at intervals of time according to the size of the frame, and the continuously received frames may constitute a voice signal.

[0058] At least one instruction regarding the electronic device (100) may be stored in the memory (120). In addition, an O / S (Operating System) for driving the electronic device (100) may be stored in the memory (120). In addition, various software programs or applications for operating the electronic device (100) according to various embodiments of the present disclosure may be stored in the memory (120). In addition, the memory (120) may include a semiconductor memory (120) such as a flash memory (120) or a magnetic storage medium such as a hard disk.

[0059] Specifically, the memory (120) may store various software modules for operating the electronic device (100) according to various embodiments of the present disclosure, and the processor (130) may control the operation of the electronic device (100) by executing the various software modules stored in the memory (120). That is, the memory (120) is accessed by the processor (130), and data reading / recording / modifying / deleting / updating, etc. may be performed by the processor (130).

[0060] Meanwhile, in the present disclosure, the term memory (120) may be used to mean a memory (120), a ROM, a RAM in a processor (130), or a memory (120) card (e.g., a micro SD card, a memory (120) stick) mounted on an electronic device (100).

[0061] In one or more embodiments, the memory (120) may store various information / data such as pre-registered keywords, feature vectors representing the characteristics of keywords, voice signals and frames of voice signals, voice feature vectors, text feature vectors, first scores, second scores, etc.

[0062] The memory (120) may store data for a plurality of modules according to the present disclosure. In addition, if at least one of the plurality of modules is implemented as a neural network model, the memory (120) may store data for the neural network model.

[0063] In addition, various information necessary within the scope of achieving the purpose of the present disclosure may be stored in the memory (120), and the information stored in the memory (120) may be updated as received from an external device or input by a user.

[0064] The processor (130) controls the overall operation of the electronic device (100). Specifically, the processor (130) is connected to the configuration of the electronic device (100) including the microphone (110) and the memory (120), and can control the overall operation of the electronic device (100) by executing at least one instruction stored in the memory (120) as described above. There may be one or two or more processors (130).

[0065] The processor (130) may be implemented in various ways. For example, the processor (130) may be implemented as at least one of an application specific integrated circuit (ASIC), an embedded processor (130), a processor (130) with a microphone (110), hardware control logic, a hardware finite state machine (FSM), and a digital signal processor (130) (Digital Signal Processor, DSP). Meanwhile, the term "processor (130)" in the present disclosure may be used to mean a central processing unit (CPU), a graphic processing unit (GPU), and a microprocessor unit (MPU).

[0066] In one or more embodiments, the processor (130) may identify whether a voice signal corresponds to a pre-registered keyword and determine whether to activate the voice recognition function based on the identification result. The processor (130) may determine whether to activate the voice recognition function using multiple modules.

[0067] The plurality of modules may be software modules or hardware modules. If at least one module among the plurality of modules is implemented as a software module, the memory (120) may store data for the at least one module, and the processor (130) may load the data stored in the memory (120) to perform operations of the plurality of modules. If at least one module among the plurality of modules is implemented as a hardware module, the at least one module may be integrated into the processor (130) in an on-chip manner. The plurality of modules may be implemented on-device by the processor (130) of the electronic device (100), and at least one of the plurality of modules may be implemented by an external device such as a server.

[0068] At least one of the plurality of modules may include a neural network, and two or more of the plurality of modules may be integrated into a single neural network model. If two or more of the plurality of modules are integrated into a single neural network model, the two or more modules may be jointly trained. Here, the neural network model may be a Wake-on-Voice model (WoV model) trained to determine whether to activate the speech recognition function, and there are no particular restrictions on the type or structure of the neural network constituting the neural network model.

[0069] As illustrated in FIG. 2, the plurality of modules may include a character identifier (1110), a voice encoder (1120), a text encoder (1130), and a keyword detector (1140). The operations performed by each of the plurality of modules are described in the description of various embodiments implemented by the processor (130) below.

[0070] The processor (130) can receive a voice signal through the microphone (110). Specifically, when a user speaks, the processor (130) can receive a voice signal corresponding to the spoken voice in frame units through the microphone (110). Meanwhile, the processor (130) can also receive a voice signal from an external device through the communication unit (140). However, for convenience of explanation, various embodiments will be described below on the assumption that the processor (130) receives a voice signal through the microphone (110).

[0071] The processor (130) can input each frame included in the voice signal into the character identifier (1110) to obtain a first score indicating the probability that the voice signal corresponds to a pre-registered keyword. As illustrated in FIG. 2, the processor (130) can input each frame included in the voice signal into the character identifier (1110) each time the frames are received.

[0072] A 'keyword' refers to a word registered as a word for activating the voice recognition function of an electronic device (100). The term 'keyword' may be replaced with terms such as 'wake on word', 'wake-up word', 'trigger word', etc. A keyword may be determined by the developer's choice, or may be registered based on user input.

[0073] In particular, a keyword according to the present disclosure can be registered based on a user input of text corresponding to the keyword. While the present disclosure will be described assuming that the keyword is registered by a user's text input, the present disclosure is not limited thereto. The keyword can be registered based on a user's voice uttering a voice corresponding to the keyword. In this case, the processor (130) may obtain a text corresponding to the user's voice using a voice recognition model included in the electronic device (100) or included in an external device such as a server, and may register the keyword based on the obtained text.

[0074] The 'character identifier (1110)' refers to a module capable of identifying a character corresponding to each frame included in a voice signal. The character identifier (1110) may include a neural network and may be trained to identify a character corresponding to each frame included in the voice signal. As illustrated in FIG. 2, when each frame included in the voice signal is input, a first score may be output.

[0075] In this disclosure, the term "character" refers to each of the multiple characters included in a keyword. For example, if the keyword is "CAT," the multiple characters included in the keyword may be "C," "A," and "T." The term "character" may be replaced with terms such as "token."

[0076] The 'first score' may represent the probability that a voice signal corresponds to a pre-registered keyword. In other words, the first score may represent the probability that a user who uttered the voice corresponding to the voice signal uttered all of the multiple characters included in the keyword.

[0077] For example, if the probability of corresponding to the letter 'C' in the first to fifth frames is higher than the probability of corresponding to the letter 'A' and the letter 'T', the probability of corresponding to the letter 'A' in the sixth to tenth frames is higher than the probability of corresponding to the letter 'C' and the letter 'T', and the probability of corresponding to the letter 'T' in the eleventh to fifteenth frames is higher than the probability of corresponding to the letter 'C' and the letter 'A', the processor (130) can obtain the first score by accumulating all the probabilities that the first to fifth frames each correspond to the letter 'C', the probabilities that the sixth to tenth frames each correspond to the letter 'A', and the probabilities that the eleventh to fifteenth frames each correspond to the letter 'T'. Here, accumulating the probabilities may mean adding all the probabilities or multiplying all the probabilities, and various other operations may also be added.

[0078] Specifically, the character identifier (1110) can obtain probability information regarding the probability that each of the input frames corresponds to each of the multiple characters included in the keyword, and can obtain a first score based on the obtained probability information. A more specific process related to obtaining the first score by the character identifier (1110) will be described in more detail with reference to FIG. 4.

[0079] The processor (130) can input each frame to the voice encoder (1120) to obtain a voice feature vector representing the characteristics of the voice signal. As illustrated in FIG. 2, the processor (130) can input each frame included in the voice signal to the voice encoder (1120) each time the frames are received.

[0080] The term 'voice feature vector' may collectively refer to a vector representing the characteristics of a voice signal. Hereinafter, the term 'vector' may be replaced with terms such as 'embedding' or 'tensor'. In the present disclosure, the voice feature vector may include a plurality of first voice feature vectors representing the characteristics of each frame and a plurality of second voice feature vectors representing the characteristics of each character identified through a character identifier (1110). The first voice feature vector may be referred to as a 'frame-based voice feature vector', and the second voice feature vector may be referred to as a 'character-based voice feature vector'.

[0081] A "voice encoder (1120)" refers to a module capable of acquiring the characteristics of a voice signal. The voice encoder (1120) may include a neural network and may be trained to acquire a voice feature vector representing the characteristics of an input voice signal. As illustrated in FIG. 2, when each frame included in a voice signal is input, a voice feature vector representing the characteristics of the voice signal can be acquired.

[0082] Specifically, the voice encoder (1120) can obtain a plurality of first voice feature vectors by obtaining a first voice feature vector representing the characteristics of each frame each time each frame is input.

[0083]

[0084] *The second voice feature vector can be obtained through a keyword detector (1140), and the process of obtaining the second voice feature vector will be described later.

[0085] The processor (130) inputs a first score, a voice feature vector, and a text feature vector representing the characteristics of a keyword into a keyword detector (1140), thereby identifying whether a keyword is included in a voice signal. The first score and the voice feature vector have been described above. Below, the process of obtaining a text feature vector will first be described, and then the process of identifying whether a keyword is included in a voice signal will be described.

[0086] The term "text feature vector" can be a general term for a vector representing the characteristics of a previously registered keyword. "Text encoder (1130)" refers to a module capable of acquiring the characteristics of an input text. The text encoder (1130) may include a neural network and may be trained to acquire a text feature vector representing the characteristics of the input text. As illustrated in FIG. 2, when a keyword is input, the text encoder (1130) can acquire a text feature vector representing the characteristics of the keyword.

[0087] Specifically, the text encoder (1130) can obtain a plurality of character feature vectors (i.e., text feature vectors per character) representing the characteristics of each of the plurality of characters included in the keyword, and can also obtain a single text feature vector representing the characteristics of the entire keyword. For example, if the pre-registered keyword is 'CAT', the text encoder (1130) can obtain a character feature vector representing the characteristics of the character 'C', a character feature vector representing the characteristics of the character 'A', and a character feature vector representing the characteristics of the character 'T'.

[0088] As illustrated in FIG. 2, the process of acquiring a text feature vector by the text encoder (1130) may be performed during the keyword registration process. Specifically, when a user input for entering text corresponding to a keyword is received, the processor (130) may input the text corresponding to the keyword into the text encoder (1130), thereby acquiring a text feature vector. Furthermore, the processor (130) may store the acquired text feature vector in the memory (120).

[0089] A 'keyword detector (1140)' refers to a module capable of detecting keywords included in a voice signal. The keyword detector (1140) may include a neural network and may be trained to detect keywords included in a voice signal. When a first score obtained through a character identifier (1110), a voice feature vector obtained through a voice encoder (1120), and a text feature vector obtained through a text encoder (1130) are input, the keyword detector (1140) may use the first score voice feature vector and the text feature vector together to identify whether a keyword is included in the voice signal and output the identification result.

[0090] Specifically, the keyword detector (1140) can obtain a second score indicating the similarity between the speech feature vector and the text feature vector. The keyword detector (1140) can identify whether a keyword is included in the speech signal based on the first score and the second score. For example, the keyword detector (1140) can obtain a weighted sum of the first score and the second score, and if the obtained weighted sum is greater than or equal to a preset threshold value, the keyword detector (1140) can identify that the speech signal includes a keyword. On the other hand, if the obtained weighted sum is less than the preset threshold value, the keyword detector (1140) can identify that the speech signal does not include a keyword.

[0091] Meanwhile, if the obtained weighted sum is less than a first threshold value preset as a criterion for including a keyword in a voice signal, but is greater than or equal to a second threshold value preset as a value lower than the first threshold value, the processor (130) may identify that it is unclear whether the keyword is included in the voice signal and provide a message requesting re-speech to the user.

[0092] Here, the similarity between the speech feature vector and the text feature vector can be calculated by various methods such as cosine similarity, Euclidean distance, or Jaccard similarity.

[0093] The keyword detector (1140) may calculate similarity by comparing one voice feature vector representing the characteristics of a voice signal with one text feature vector representing the characteristics of a keyword, but may also calculate similarity by comparing multiple vectors with each other.

[0094] Specifically, the keyword detector (1140) can obtain a plurality of second speech feature vectors by combining first speech feature vectors having the same identified characters among the plurality of first speech feature vectors based on information about the characters identified through the character identifier (1110) and information about the time at which each frame was received. As described above, the plurality of speech feature vectors can represent the characteristics of each of the characters identified through the character identifier (1110).

[0095] For example, if the first to fifth frames correspond to the letter 'C', the sixth to tenth frames correspond to the letter 'A', and the eleventh to fifteenth frames correspond to the letter 'T', then the keyword detector (1140) can obtain a second voice feature vector corresponding to the letter 'C' by combining the first voice feature vectors corresponding to each of the first to fifth frames. In this case, the keyword detector (1140) can obtain a second voice feature vector corresponding to the letter 'A' by combining the first voice feature vectors corresponding to each of the sixth to tenth frames, and can obtain a second voice feature vector corresponding to the letter 'T' by combining the first voice feature vectors corresponding to each of the eleventh to fifteenth frames.

[0096] As described above, when a plurality of second speech feature vectors are obtained, the keyword detector (1140) can obtain a second score by comparing the plurality of second speech feature vectors (i.e., speech feature vectors in character units) with the plurality of character feature vectors (i.e., text feature vectors in character units) corresponding to each other. Here, the plurality of character feature vectors refers to vectors representing the features of each of the plurality of characters included in the keyword, as described above.

[0097] For example, the keyword detector (1140) may compare a second speech feature vector corresponding to the letter 'C' and a character feature vector representing the features of the letter 'C' to calculate a first similarity value, a second speech feature vector corresponding to the letter 'A' and a character feature vector representing the features of the letter 'A' to calculate a second similarity value, and a third similarity value may be calculated by comparing a second speech feature vector corresponding to the letter 'T' and a character feature vector representing the features of the letter 'T'. In addition, the keyword detector (1140) may perform an operation such as a sum or a weighted sum based on the first similarity, the second similarity, and the third similarity to obtain a second score.

[0098] If it is determined that the voice signal includes a keyword, the processor (130) may activate the voice recognition function of the electronic device (100). On the other hand, if it is determined that the voice signal does not include a keyword, the processor (130) may maintain the voice recognition function in a state where it is not activated.

[0099] Activating a voice recognition function means that the voice recognition model starts operating, detecting the user's voice, and is ready to acquire text corresponding to the user's voice. Because the voice recognition model requires a large amount of computation and data, it can consume significant power during operation. Therefore, the voice recognition function may remain in a standby (inactive) state until a preset event occurs, during which it can be activated. The preset event may be, as described above, the user uttering a voice corresponding to a keyword, or it may be a user input (e.g., touch input, button input, etc.) that activates the voice recognition function.

[0100] Meanwhile, the voice recognition model to be activated may be implemented by the electronic device (100) or by an external device. Accordingly, if the voice recognition model is implemented by an external device, the processor (130) may activate the voice recognition function of the external device by transmitting a request to activate the voice recognition function to the external device via the communication unit (140).

[0101] Meanwhile, as described above, at least one of the character identifier (1110), the voice encoder (1120) and the text encoder (1130) may include a neural network and thus may be trained based on training data.

[0102] For example, the processor (130) may train the character identifier (1110) to identify characters included in the input speech signal using a loss value corresponding to the first score. In addition, the processor (130) may train at least one of the speech encoder (1120) and the text encoder (1130) to make the speech feature vector output by the speech encoder (1120) and the text feature vector output by the text encoder (1130) similar using a loss value corresponding to the second score.

[0103] In addition, the processor (130) may obtain a loss value based on the first score and the second score, and perform joint training on at least one of the character identifier (1110), the voice encoder (1120), and the text encoder (1130) based on the obtained loss value. When supervised learning is used as a learning method, a pre-registered keyword may be used as a label.

[0104] According to the embodiments described above with reference to FIGS. 1 and 2, the electronic device (100) can detect keywords for activating a voice recognition function in an effective and efficient manner. Specifically, even when a user registers a keyword through text input, the electronic device (100) can effectively compare the registered keyword with the user's voice received in real time, and can efficiently detect keywords by processing one frame at a time in real time according to a streaming method.

[0105] FIG. 3 is a diagram illustrating a plurality of modules according to one or more embodiments of the present disclosure.

[0106] In the description of FIG. 2, various embodiments have been described based on the structure of a plurality of modules as illustrated in FIG. 2. Specifically, in the description of FIG. 2, when frames included in a voice signal are input to a character identifier (1110), the character identifier (1110) obtains a first score indicating a probability that the voice signal corresponds to a pre-registered keyword, and when frames included in a voice signal are input to a voice encoder (1120), the voice encoder (1120) obtains a voice feature vector indicating the characteristics of the voice signal. However, the structure illustrated in FIG. 2 is only an example, and the structure and operation of the plurality of modules may vary depending on the embodiment.

[0107] As illustrated in FIG. 3, the voice encoder (1120) may include a first transcription block (1121) and a second transcription block (1122). Here, the first transcription block (1121) may output probability information corresponding to an input frame. Here, the 'probability information' refers to information about the probability that each of the input frames corresponds to each of a plurality of characters included in a keyword. When the probability information for each of the frames is obtained through the first transcription block (1121), the processor (130) may input the probability information for each of the frames into the character identifier (1110) to obtain a first score.

[0108] In other words, in the description of FIG. 2, it was described that the character identifier (1110) receives each frame as input, obtains probability information for each frame, and obtains a first score based on the obtained probability information, but as illustrated in FIG. 3, the process of receiving each frame as input and obtaining probability information for each frame may also be performed through the first transcription block (1121) included in the voice encoder (1120).

[0109] Meanwhile, the second transcription block (1122) can output a voice feature vector (i.e., a first voice feature vector) corresponding to the input frame. That is, the operation performed by the second transcription block (1122) of FIG. 3 may be identical to the operation described as performed by the audio encoder of FIG. 2.

[0110] The operations of the text encoder (1130) and keyword detector (1140) are identical to those described with reference to FIG. 2, and thus, a detailed description of the same content will be omitted. If the structure can implement various embodiments according to the present disclosure, the structure of the multiple modules may vary beyond the structure of FIG. 2 or FIG. 3.

[0111] FIG. 4 is a diagram for explaining in detail the process of obtaining the first score and the second score according to one or more embodiments of the present disclosure.

[0112] In one or more embodiments, the character identifier (1110) may obtain probability information regarding the probability that each of the input frames corresponds to each of the multiple characters included in the keyword. The probability information in FIG. 4 represents frame-by-frame probability information output by the character identifier (1110) or the voice encoder (1120).

[0113] The character identifier (1110) can identify characters having the highest probability of corresponding to each of the frames among a plurality of characters for each input frame based on the acquired probability information. In addition, the character identifier (1110) can obtain a first score by accumulating probabilities corresponding to the identified characters. More specifically, the character identifier (1110) can identify characters having the highest probability of corresponding to each of the frames among a plurality of characters for each frame by updating information about the probability that the frames input up to now include a frame corresponding to each of the plurality of characters based on the probability information.

[0114] For example, the processor (130) can obtain an optimal CTC score ending at the current time point T by utilizing a CTC (Connectionist Temporal Classification) algorithm using a character identifier (1110). In this case, the CTC score may be a first score according to the present disclosure. The following describes a process of obtaining the first score using the CTC algorithm.

[0115] 410 of Figure 4 illustrates probability information, and 1, 2, 3, 4, 5, ..., T-1, T on the horizontal axis represent the order (time) in which each frame was received. The vertical axis represents predefined tokens, among which <pad>represents a token for matching the length of the sequence. 'C', 'A', and 'T' represent multiple characters included in the keyword when the previously registered keyword is 'CAT'. The filled circles corresponding to the characters 'C', 'A', and 'T' respectively represent the probability that each frame corresponds to 'C', 'A', and 'T' respectively, and the hollow circles represent blank tokens indicating transitions between characters. A blank token may indicate that the character corresponding to the previous token was continuously uttered. In addition, the graph of 410 in Fig. 4 represents the result of connecting the tokens with the highest probability that each frame corresponds to.

[0116] Referring to 410 of FIG. 4, the character identifier (1110) can identify that the sixth to seventh frames have the highest probability of corresponding to the letter 'C', the eighth to eleventh frames have the highest probability of corresponding to the letter 'A', and the twelfth to thirteenth frames have the highest probability of corresponding to the letter 'T'.

[0117] The character identifier (1110) can obtain a CTC score, i.e., a first score, by accumulating the probabilities that the sixth to seventh frames correspond to the letter 'C', the probabilities that the eighth to eleventh frames correspond to the letter 'A', and the probabilities that the twelfth to thirteenth frames correspond to the letter 'T'. Accordingly, the character identifier (1110) can obtain a first score indicating the probability that the voice signal corresponds to a pre-registered keyword at the current time point T.

[0118] 420 of FIG. 4 represents the structure of data that is updated each time probability information, such as that shown in 410 of FIG. 4, is acquired for each frame. That is, 410 of FIG. 4 and 420 of FIG. 4 represent the same example unfolded along the time axis and depicted as a data structure, respectively.

[0119] Specifically, the circles at 420 in FIG. 4 represent states of data corresponding to each token indicated on the vertical axis at 410 in FIG. 4. In addition, the arrows at 420 in FIG. 4 indicate that each state can cycle to itself or move to the next state. The character identifier (1110) can update all states whenever each frame is input. Specifically, the character identifier (1110) can update data corresponding to states based on the largest CTC score among the CTC scores corresponding to the arrows entering each state. Accordingly, all states can represent the probability that the frames input up to the current point in time correspond to the token of the corresponding state.

[0120] When the seventh frame of 410 in FIG. 4 is input, the C state can store a score indicating the probability that the frames input up to now correspond to the letter 'C', the time when the frame corresponding to the letter 'C' started to be input (i.e., the time corresponding to the sixth frame of 410 in FIG. 4), and a combination of the first speech feature vectors corresponding to the letter 'C' (i.e., the second speech feature vector corresponding to C in FIG. 4).

[0121] Meanwhile, until the seventh frame of 410 in FIG. 4 is input, the A state and the T state are also updated, but since the frames corresponding to the letters 'A' and 'T' have not yet been input, the scores of the A state and the T state may be very low.

[0122] When the eighth frame of 410 in FIG. 4 is input, the information stored in the C state is copied to the A state, and the A state can also store a score indicating the probability that the eighth frame corresponds to the letter 'A', the time when the frame corresponding to the letter 'A' started to be input (i.e., the time corresponding to the eighth frame in 420 in FIG. 4), and the first voice feature vector corresponding to the eighth frame.

[0123] When the ninth frame of 410 in FIG. 4 is input, the A state can store a combination of a score indicating a probability that the eighth frame corresponds to the letter 'A' and a score indicating a probability that the ninth frame corresponds to the letter 'A'. Here, the combination of scores indicating probabilities can be performed by calculating the sum of log probabilities, or can be performed according to various other operations. Thereafter, the A state can store a combination of a first speech feature vector corresponding to the eighth frame and a first speech feature vector corresponding to the ninth frame (i.e., a second speech feature vector corresponding to A in FIG. 4).

[0124] If the method described above is repeated until a frame corresponding to the current time T of 410 of FIG. 4 is input, the T state can store scores corresponding to the entire keyword including the letters 'C', 'A', and 'T' up to the current time T (i.e., the first score), the time when each of the letters 'C', 'A', and 'T' started to be input, and the second speech feature vector corresponding to each of the letters 'C', 'A', and 'T'.

[0125] As described above, the character identifier (1110) can obtain a score (i.e., a first score) corresponding to the entire keyword and identify the CTC alignment while identifying the characters with the highest probability of corresponding to each frame in a streaming manner for each input frame. In addition, the character identifier (1110) can obtain a plurality of second voice feature vectors corresponding to each of the plurality of characters included in the keyword according to the CTC alignment.

[0126] The process of obtaining the first score has been described above, and the process of obtaining the second score will be described below.

[0127] In one or more embodiments, the keyword detector (1140) may obtain a second score by combining a plurality of first speech feature vectors to obtain a single speech feature vector, combining a plurality of text feature vectors to obtain a single text feature vector corresponding to a keyword, and then comparing the single speech feature vector with the single text feature vector.

[0128] In another embodiment, the keyword detector (1140) can obtain a second score by comparing a plurality of second speech feature vectors (i.e., speech feature vectors in character units) with a plurality of character feature vectors (i.e., text feature vectors in character units) corresponding to each other.

[0129] For example, the keyword detector (1140) may compare a second speech feature vector corresponding to the letter 'C' and a character feature vector representing the features of the letter 'C' to calculate a first similarity value, a second speech feature vector corresponding to the letter 'A' and a character feature vector representing the features of the letter 'A' to calculate a second similarity value, and a third similarity value may be calculated by comparing a second speech feature vector corresponding to the letter 'T' and a character feature vector representing the features of the letter 'T'. In addition, the keyword detector (1140) may perform an operation such as a sum or a weighted sum based on the first similarity, the second similarity, and the third similarity to obtain a second score.

[0130] By comparing the character-based speech feature vectors with multiple character feature vectors and the corresponding characters to obtain a second score, the speech feature vectors of frames corresponding to noise, such as the first to third frames of 410 in Fig. 4, can be excluded, and since correspondence can be confirmed for each character, the accuracy of keyword detection can be further improved.

[0131] Meanwhile, the keyword detector (1140) can also obtain a second score by combining a score based on the result of comparing one voice feature vector with one text feature vector and a score based on the result of comparing a character-level voice feature vector with multiple character feature vectors corresponding to each other.

[0132] According to the embodiments described above, the electronic device (100) obtains an optimal CTC score (first score) and CTC alignment ending at the current time point T in a streaming manner, and compares a voice feature vector and a character feature vector on a character basis using not only the CTC score but also the CTC alignment to obtain a similarity score (second score), and performs keyword detection using the first score and the second score together. Accordingly, the electronic device (100) can detect a keyword for activating a voice recognition function in an effective and efficient manner.

[0133] FIG. 5 is a drawing detailing the configuration of an electronic device (100) according to one or more embodiments of the present disclosure.

[0134] As illustrated in FIG. 5, the electronic device (100) may further include a microphone (110), a memory (120), and a processor (130), as well as a communication unit (140), an input unit (150), and an output unit (160). However, the configurations illustrated in FIGS. 1 and 5 are merely exemplary, and it is to be understood that new configurations may be added or some configurations may be omitted in addition to the configurations illustrated in FIGS. 1 and 5 when implementing the present disclosure.

[0135] The communication unit (140) includes a circuit and can perform communication with an external device. Specifically, the processor (130) can receive various data or information from an external device connected via the communication unit (140) and can also transmit various data or information to the external device.

[0136] The communication unit (140) may include at least one of a WiFi module, a Bluetooth module, a wireless communication module, an NFC module, and an Ultra-Wide Band (UWB) module. Specifically, the WiFi module and the Bluetooth module may each perform communication in the WiFi or Bluetooth manner. When using a WiFi module or a Bluetooth module, various connection information, such as an SSID, may be first transmitted and received, and then communication may be established using this, after which various pieces of information may be transmitted and received.

[0137] In addition, the wireless communication module can perform communication according to various communication standards such as IEEE, Zigbee, 3G (3rd Generation), 3GPP (3rd Generation Partnership Project), LTE (Long Term Evolution), 5G (5th Generation), etc. And, the NFC module can perform communication in the NFC (Near Field Communication) method using the 13.56MHz band among various RF-ID frequency bands such as 135kHz, 13.56MHz, 433MHz, 860~960MHz, 2.45GHz, etc. In addition, the UWB module can accurately measure ToA (Time of Arrival), which is the time it takes for a pulse to reach a target, and AoA (Ange of Arrival), which is the pulse arrival angle at the transmitting device, through communication between UWB antennas, and accordingly, precise distance and location recognition is possible within an error range of several tens of centimeters indoors.

[0138] In one or more embodiments, when at least some of the plurality of modules are implemented by an external device, or when a voice recognition model is implemented by an external device, the processor (130) may control the communication unit (140) to transmit at least some of various information / data, such as a pre-registered keyword, a feature vector representing a feature of the keyword, a voice signal and frames of the voice signal, a voice feature vector, a text feature vector, a first score, a second score, etc., to the external device. In addition, the processor (130) may receive at least some of various information / data, such as a pre-registered keyword, a feature vector representing a feature of the keyword, a voice signal and frames of the voice signal, a voice feature vector, a text feature vector, a first score, a second score, etc., from the external device through the communication unit (140).

[0139] The input unit (150) includes a circuit, and the processor (130) can receive a user command to control the operation of the electronic device (100) through the input unit (150). Specifically, the input unit (150) can be configured with a microphone (110), a camera, a remote control signal receiving unit, and the like. In addition, the input unit (150) can be implemented in a form included in a display as a touch screen. In particular, the microphone (110) can receive a voice signal and convert the received voice signal into an electrical signal.

[0140] In one or more embodiments, the processor (130) may receive a user input for entering text corresponding to a keyword through the input unit (150). In addition, the processor (130) may also receive a user input (e.g., touch input, button input, etc.) for activating a voice recognition function through the input unit (150).

[0141] The output unit (160) includes a circuit, and the processor (130) can output various functions that the electronic device (100) can perform through the output unit (160). In addition, the output unit (160) can include at least one of a display, a speaker, and an indicator.

[0142] The display can output image data under the control of the processor (130). Specifically, the display can output an image previously stored in the memory (120) under the control of the processor (130). In particular, the display according to one or more embodiments of the present disclosure can also display a user interface stored in the memory (120). The display can be implemented as an LCD (Liquid Crystal Display Panel), an OLED (Organic Light Emitting Diodes), etc., and in some cases, the display can also be implemented as a flexible display, a transparent display, etc. However, the display according to the present disclosure is not limited to a specific type.

[0143] The speaker can output audio data under the control of the processor (130). The indicator can be turned on under the control of the processor (130). Specifically, the indicator can be turned on in various colors under the control of the processor (130). For example, the indicator can be implemented as a light emitting diode (LED), a liquid crystal display panel (LCD), a vacuum fluorescent display (VFD), etc., but is not limited thereto.

[0144] In one or more embodiments, if it is identified that a pre-registered keyword is included in the voice signal, the processor (130) may control the output unit (160) to output a message indicating that the voice signal has been identified as including the keyword.

[0145] Meanwhile, if the weighted sum obtained based on the first score and the second score is less than a first threshold value preset as a criterion for including a keyword in a voice signal, but is greater than or equal to a second threshold value preset as a value lower than the first threshold value, the processor (130) may identify that it is unclear whether the keyword is included in the voice signal, and control the output unit (160) to output a message requesting the user to re-enter the voice signal.

[0146] FIG. 6 is a flowchart illustrating a method for controlling an electronic device (100) according to one or more embodiments of the present disclosure.

[0147]

[0148] *As illustrated in FIG. 6, the electronic device (100) can receive a voice signal through the microphone (110) (S610). When a user speaks, the electronic device (100) can receive a voice signal corresponding to the spoken voice on a frame-by-frame basis through the microphone (110).

[0149] The electronic device (100) can input each frame included in the voice signal into a character identifier (1110) to obtain a first score indicating the probability that the voice signal corresponds to a pre-registered keyword (S620). Specifically, the electronic device (100) can obtain probability information regarding the probability that each of the input frames corresponds to each of a plurality of characters included in the keyword, and can obtain the first score based on the obtained probability information.

[0150] The electronic device (100) can obtain a voice feature vector representing the characteristics of a voice signal by inputting each frame to the voice encoder (1120) (S630). Specifically, the electronic device (100) can obtain a plurality of first voice feature vectors by obtaining a first voice feature vector representing the characteristics of each frame each time each frame is input.

[0151] The electronic device (100) inputs a first score, a voice feature vector, and a text feature vector representing the characteristics of a keyword into a keyword detector (1140), thereby identifying whether a keyword is included in a voice signal (S640).

[0152] The electronic device (100) can obtain a plurality of character feature vectors (i.e., text feature vectors per character) representing the characteristics of each of the plurality of characters included in the keyword, and can also obtain one text feature vector representing the characteristics of the entire keyword.

[0153] The electronic device (100) can identify whether a keyword is included in a voice signal by using the first score voice feature vector and the text feature vector together, and output the identification result.

[0154] The electronic device (100) can obtain a second score indicating the similarity between the voice feature vector and the text feature vector. The electronic device (100) can identify whether a keyword is included in the voice signal based on the first score and the second score.

[0155] For example, the electronic device (100) can obtain a weighted sum of the first score and the second score, and if the obtained weighted sum is greater than or equal to a preset threshold value, the electronic device (100) can identify that the voice signal includes a keyword. On the other hand, if the obtained weighted sum is less than the preset threshold value, the electronic device (100) can identify that the voice signal does not include a keyword.

[0156] If it is determined that the voice signal contains a keyword, the electronic device (100) can activate the voice recognition function of the electronic device (100) (S650). On the other hand, if it is determined that the voice signal does not contain a keyword, the professional electronic device (100) can maintain the voice recognition function in a state where it is not activated.

[0157] Meanwhile, the control method of the electronic device (100) according to the above-described embodiment may be implemented as a program and provided to the electronic device (100). In particular, the program including the control method of the electronic device (100) may be stored and provided in a non-transitory computer readable medium.

[0158] Specifically, in a non-transitory computer-readable recording medium including a program for executing a control method of an electronic device (100), the control method of the electronic device (100) may include, when a voice signal is received through a microphone (110), a step of inputting each of the frames included in the voice signal into a character identifier (1110) to obtain a first score indicating a probability that the voice signal corresponds to a pre-registered keyword, a step of inputting each of the frames into a voice encoder (1120) to obtain a voice feature vector indicating a characteristic of the voice signal, a step of inputting the first score, the voice feature vector, and a text feature vector indicating a characteristic of the keyword into a keyword detector (1140) to identify whether the keyword is included in the voice signal, and a step of activating a voice recognition function of the electronic device (100) when it is identified that the keyword is included in the voice signal.

[0159] In the above, a method for controlling an electronic device (100) and a computer-readable recording medium including a program for executing the method for controlling an electronic device (100) have been briefly described, but this is only to omit redundant descriptions, and it goes without saying that various embodiments of the electronic device (100) can also be applied to a method for controlling an electronic device (100) and a computer-readable recording medium including a program for executing the method for controlling an electronic device (100).

[0160] The artificial intelligence-related function according to the present disclosure is operated through the processor (130) and memory (120) of the electronic device (100).

[0161] The processor (130) may be composed of one or more processors (130). At this time, the one or more processors (130) may include at least one of a CPU (Central Processing Unit), a GPU (Graphic Processing Unit), and an NPU (Neural Processing Unit), but is not limited to the examples of the processors (130) described above.

[0162] The CPU is a general-purpose processor (130) capable of performing not only general calculations but also artificial intelligence calculations. Its multi-layer cache structure allows for the efficient execution of complex programs. The CPU is advantageous in a serial processing method, enabling organic linking of previous and subsequent calculation results through sequential calculations. The general-purpose processor (130) is not limited to the aforementioned examples, except in cases where it is specifically designated as a CPU.

[0163] A GPU is a processor (130) for large-scale operations such as floating-point operations used in graphic processing, and can perform large-scale operations in parallel by integrating a large number of cores. In particular, a GPU may be advantageous compared to a CPU in parallel processing methods such as convolution operations. In addition, a GPU may be used as a co-processor (130) to supplement the functions of a CPU. The processor (130) for large-scale operations is not limited to the examples described above, except in cases where it is specified as a GPU as described above.

[0164] An NPU is a processor (130) specialized in artificial intelligence operations using an artificial neural network, and each layer constituting the artificial neural network can be implemented with hardware (e.g., silicon). At this time, since the NPU is designed specifically according to the required specifications of the company, it has a lower degree of freedom compared to a CPU or GPU, but it can efficiently process the artificial intelligence operations requested by the company. Meanwhile, as a processor (130) specialized in artificial intelligence operations, the NPU can be implemented in various forms such as a TPU (Tensor Processing Unit), an IPU (Intelligence Processing Unit), a VPU (Vision processing unit), etc. The artificial intelligence processor (130) is not limited to the above-described examples, except in cases where it is specified as the above-described NPU.

[0165] Additionally, one or more processors (130) may be implemented as a SoC (System on Chip). In this case, the SoC may further include, in addition to one or more processors (130), a memory (120), and a network interface such as a bus for data communication between the processor (130) and the memory (120).

[0166] When a plurality of processors (130) are included in a SoC (System on Chip) included in an electronic device (100), the electronic device (100) may perform operations related to artificial intelligence (e.g., operations related to learning or inference of an artificial intelligence model) by using some of the plurality of processors (130). For example, the electronic device (100) may perform operations related to artificial intelligence by using at least one of a GPU, an NPU, a VPU, a TPU, and a hardware accelerator specialized in artificial intelligence operations such as convolution operations and matrix multiplication operations among the plurality of processors (130). However, this is merely an example, and it is of course possible to process operations related to artificial intelligence by using a CPU or a general-purpose processor (130).

[0167] In addition, the electronic device (100) can perform operations related to functions related to artificial intelligence by utilizing multiple cores (e.g., dual cores, quad cores, etc.) included in one processor (130). In particular, the electronic device (100) can perform artificial intelligence operations such as convolution operations, matrix multiplication operations, etc. in parallel by utilizing multiple cores included in the processor (130).

[0168] One or more processors (130) are controlled to process input data according to predefined operation rules or artificial intelligence models stored in the memory (120). The predefined operation rules or artificial intelligence models are characterized by being created through learning.

[0169] Here, "created through learning" means that a predefined set of behavioral rules or an AI model with desired characteristics is created by applying a learning algorithm to a large number of learning data. This learning may be performed on the device itself, where the AI ​​according to the present disclosure is implemented, or through a separate server / system.

[0170] An artificial intelligence model may be composed of multiple neural network layers. At least one layer has at least one weight value and performs its operation through the operation result of the previous layer and at least one defined operation. Examples of neural networks include a convolutional neural network (CNN), a deep neural network (DNN), a recurrent neural network (RNN), a restricted boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), deep Q-networks, and a transformer. The neural networks in the present disclosure are not limited to the above-described examples unless otherwise specified.

[0171] A learning algorithm is a method for training a target device (e.g., a robot) using a large amount of learning data, enabling the target device to make decisions or predictions on its own. Examples of learning algorithms include supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning. Unless otherwise specified, the learning algorithms in this disclosure are not limited to the aforementioned examples.

[0172] A device-readable storage medium may be provided in the form of a non-transitory storage medium. Here, the term "non-transitory storage medium" simply means a tangible device that does not contain signals (e.g., electromagnetic waves). This term does not distinguish between cases where data is permanently stored in the storage medium and cases where data is temporarily stored. For example, a "non-transitory storage medium" may include a buffer in which data is temporarily stored.

[0173] According to one or more embodiments, the method according to the various embodiments disclosed in the present document may be provided as a computer program product. The computer program product may be traded between sellers and buyers as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read only memory (CD-ROM)) or may be available through an application store (e.g., Play Store). TM ) or directly between two user devices (e.g., smartphones), online distribution (e.g., downloading or uploading). In the case of online distribution, at least a portion of the computer program product (e.g., downloadable app) may be at least temporarily stored or temporarily created in a device-readable storage medium, such as a manufacturer's server, an application store's server, or a memory (120) of an intermediary server.

[0174] Each of the components (e.g., modules or programs) according to the various embodiments of the present disclosure as described above may be composed of a single or multiple entities, and some of the sub-components described above may be omitted, or other sub-components may be further included in the various embodiments. Alternatively or additionally, some components (e.g., modules or programs) may be integrated into a single entity, which may perform the same or similar functions as those performed by each of the respective components prior to integration.

[0175] According to various embodiments, operations performed by a module, program or other component may be executed sequentially, in parallel, iteratively or heuristically, or at least some operations may be executed in a different order, omitted, or other operations may be added.

[0176] Meanwhile, the terms "part" or "module" used in the present disclosure include units composed of hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A "part" or "module" may be an integrally composed component, a minimum unit performing one or more functions, or a portion thereof. For example, a module may be composed of an application-specific integrated circuit (ASIC).

[0177] Various embodiments of the present disclosure may be implemented as software including instructions stored in a machine-readable storage medium that can be read by a machine (e.g., a computer). The device may include an electronic device (e.g., an electronic device (100)) according to the disclosed embodiments, which is a device capable of calling instructions stored in the storage medium and operating according to the called instructions.

[0178] When the above instruction is executed by the processor, the processor may perform the function corresponding to the instruction directly or by using other components under the control of the processor. The instruction may include code generated or executed by a compiler or interpreter.

[0179] Although the preferred embodiments of the present disclosure have been illustrated and described above, the present disclosure is not limited to the specific embodiments described above, and various modifications may be made by a person having ordinary skill in the art to which the present disclosure pertains without departing from the gist of the present disclosure as claimed in the claims, and such modifications should not be understood individually from the technical idea or prospect of the present disclosure.< / pad>

Claims

1. In electronic devices, mike; A memory (120) storing at least one instruction; and a processor for executing at least one instruction; The above processor, When a voice signal is received through the microphone, each frame included in the voice signal is input into a character identifier to obtain a first score indicating the probability that the voice signal corresponds to a previously registered keyword, By inputting each of the above frames into a voice encoder, a voice feature vector representing the characteristics of the voice signal is obtained, By inputting the first score, the voice feature vector, and the text feature vector representing the feature of the keyword into a keyword detector, it is identified whether the keyword is included in the voice signal, An electronic device that activates a voice recognition function of the electronic device when the voice signal is identified as containing the keyword.

2. In paragraph 1, The above processor, When a user input corresponding to the above keyword is received, the text is input into a text encoder to obtain the text feature vector, An electronic device that stores the above text feature vector in the above memory (120).

3. In paragraph 2, The above processor, Each of the above frames is input to the character identifier and the voice encoder as each of the above frames is received, The above character identifier is, Obtain probability information about the probability that each of the above frames corresponds to each of the multiple characters included in the above keyword, Based on the above probability information, for each of the frames, the characters having the highest probability of corresponding to each of the frames are identified among the plurality of characters, An electronic device that obtains the first score by accumulating probabilities corresponding to the identified characters.

4. In paragraph 3, The above character identifier is, An electronic device that updates information on the probability that the frames input up to now among the frames include a frame corresponding to each of the plurality of characters based on the probability information, thereby identifying characters among the plurality of characters with the highest probability of corresponding to each of the frames.

5. In paragraph 4, The above speech feature vector includes a plurality of first speech feature vectors representing the features of each of the frames and a plurality of second speech feature vectors representing the features of each of the identified characters, The above voice encoder, An electronic device that obtains the plurality of first voice feature vectors by obtaining the first voice feature vector representing the characteristics of each of the frames each time each of the frames is input.

6. In paragraph 5, The above keyword detector is, Obtain a second score indicating the similarity between the above speech feature vector and the above text feature vector, Obtain a weighted sum of the first score and the second score, An electronic device that identifies the voice signal as containing the keyword if the weighted sum is greater than or equal to a preset threshold value.

7. In paragraph 6, The above keyword detector is, An electronic device that obtains a plurality of second voice feature vectors by combining first voice feature vectors having the same identified characters among the plurality of first voice feature vectors based on information about the identified characters and information about the time at which each of the frames was received.

8. In paragraph 7, The above text feature vector includes a plurality of character feature vectors representing the features of each of the plurality of characters, The above keyword detector is, An electronic device that obtains the second score by comparing the plurality of second voice feature vectors with the plurality of character feature vectors corresponding to each other.

9. In paragraph 7, Each of the above character identifier, the above speech encoder and the above text encoder comprises a plurality of neural networks, The above processor, Obtain a loss value based on the first score and the second score, An electronic device that trains at least one of the character identifier, the speech encoder and the text encoder based on the loss value.

10. In paragraph 1, An electronic device wherein the size of each of the above frames is determined within a range of 20 ms to 40 ms.

11. In a method for controlling an electronic device, When a voice signal is received through a microphone, a step of inputting each frame included in the voice signal into a character identifier to obtain a first score indicating a probability that the voice signal corresponds to a previously registered keyword; A step of inputting each of the above frames into a voice encoder to obtain a voice feature vector representing the characteristics of the voice signal; A step of inputting the first score, the voice feature vector, and the text feature vector representing the feature of the keyword into a keyword detector to identify whether the voice signal includes the keyword; and A method for controlling an electronic device, comprising: a step of activating a voice recognition function of the electronic device when the keyword is identified as being included in the voice signal; 12. In paragraph 11, The method of controlling the above electronic device is as follows: When a user input corresponding to the keyword is received, a step of inputting the text into a text encoder to obtain the text feature vector; and A control method of an electronic device further comprising: a step of storing the text feature vector in the memory (120); 13. In paragraph 12, The step of obtaining the above first score is: A step of inputting each of the above frames into the character identifier each time each of the above frames is received; The step of obtaining the above voice feature vector is: A step of inputting each of the frames into the voice encoder each time each of the frames is received; The above character identifier is, Obtain probability information about the probability that each of the above frames corresponds to each of the multiple characters included in the above keyword, Based on the above probability information, for each of the frames, the characters having the highest probability of corresponding to each of the frames are identified among the plurality of characters, A control method of an electronic device for obtaining the first score by accumulating probabilities corresponding to the identified characters.

14. In paragraph 13, The above character identifier is, A control method of an electronic device for identifying characters having the highest probability of corresponding to each of the plurality of characters among the plurality of characters in each of the frames by updating information on the probability that the frames input up to now among the frames include a frame corresponding to each of the plurality of characters based on the probability information.

15. In paragraph 14, The above speech feature vector includes a plurality of first speech feature vectors representing the features of each of the frames and a plurality of second speech feature vectors representing the features of each of the identified characters, The above voice encoder, A control method of an electronic device for obtaining the plurality of first voice feature vectors by obtaining the first voice feature vector representing the characteristics of each of the frames each time each of the frames is input.

Images