Electronic device for enhancing recognition performance for target keyword, and control method thereof

Abstract

An electronic device is disclosed. The present electronic device comprises: a neural network model including an acoustic encoder and a text encoder trained on relationships between a plurality of first sample sounds and a plurality of sample texts; a memory for storing instructions; and at least one processor including processing circuitry, wherein the instructions, when executed individually or collectively by the at least one processor, may: input target text corresponding to a target keyword into the text encoder to obtain a text embedding; and retrain the neural network model on the basis of a plurality of second sample sounds including the target keyword and the text embedding to obtain a final neural network model in which the acoustic encoder is updated.

Description

Electronic device and control method for enhancing recognition performance for target keywords

[0001] The present disclosure relates to an electronic device and a control method thereof, and more particularly, to an electronic device and a control method thereof for enhancing recognition performance for a target keyword.

[0002] Advances in electronic technology have led to the development of a diverse range of electronic devices. In particular, electronic devices equipped with keyword spotting (KWS) technology have recently become widespread.

[0003] KWS technology is a technology that detects desired keywords from continuous voice input into the system, and can be mainly used for wake-up word detection (WWD) of voice assistants.

[0004] KWS technology can be divided into fixed KWS, which only supports fixed keywords, and custom KWS, which supports a variety of custom keywords. Custom KWS can be categorized into voice and text registration methods, depending on the registration method.

[0005] The voice registration method registers a desired keyword by uttering it n times, and an acoustic embedding (AE) can be extracted from the registered utterance. In the inference stage, the AE extracted from the input speech is compared with the registered AE, and actions can be taken based on similarity.

[0006] The text registration method trains an acoustic encoder and a text encoder using training data containing both speech and text. It then extracts text embeddings (TEs) by inputting desired keywords as text. In the inference stage, the AE extracted from the input speech is compared with the registered TE, and actions can be taken based on similarity.

[0007] According to one embodiment of the present disclosure for achieving the above object, an electronic device includes a neural network model including an acoustic encoder and a text encoder that have learned a relationship between a plurality of first sample sounds and a plurality of sample texts, and one or more processors including a memory and processing circuitry that store instructions, wherein the instructions, when individually or collectively executed by the one or more processors, input a target text corresponding to a target keyword into the text encoder to obtain a text embedding, and retrain the neural network model based on a plurality of second sample sounds including the target keyword and the text embedding, so that the acoustic encoder obtains an updated final neural network model.

[0008] Additionally, the instructions, when individually or collectively executed by the one or more processors, can retrain the neural network model based on the plurality of second sample sounds and the text embeddings to obtain the final neural network model in which at least one layer included in the audio encoder is updated.

[0009] And, the instructions, when individually or collectively executed by the one or more processors, can identify the at least one layer based on the number of parameters of each of the plurality of layers included in the acoustic encoder.

[0010] Additionally, the at least one layer may include at least one of a batch normalization (BN) layer that determines a feature distribution per channel or a squeeze-and-excitation (SE) module that determines feature importance per channel.

[0011] And, when the instructions are individually or collectively executed by the one or more processors, the preset activation function included in the acoustic encoder can be converted into a plurality of learnable activation functions, and the plurality of output values ​​output from the plurality of learnable activation functions can be weighted based on the text embedding to retrain the neural network model.

[0012] In addition, the instructions, when individually or collectively executed by the one or more processors, input the text embedding and conditioning parameters into a softmax function, and retrain the neural network model by weighting a plurality of output values ​​based on the softmax function, and the conditioning parameters may have values ​​updated during the retraining process.

[0013] And, the preset activation function includes a ReLU function, and the plurality of learnable activation functions may include at least one of an ELU function, a hard sigmoid function, the ReLU function, a softplus function, a swish function, or a tanh function.

[0014] In addition, the instructions, when individually or collectively executed by the one or more processors, can convert a preset activation function included in the acoustic encoder into a plurality of learnable activation functions, and retrain the neural network model by weighting a plurality of output values ​​output from the plurality of learnable activation functions based on the text embedding.

[0015] And, when the instructions are individually or collectively executed by the one or more processors, the text embedding and conditioning parameters are input to a softmax function, and a plurality of output values ​​are weighted based on the softmax function to retrain the neural network model, and the conditioning parameters can have values ​​updated during the retraining process.

[0016] In addition, the preset activation function may include a ReLU function, and the plurality of learnable activation functions may include at least one of an ELU function, a hard sigmoid function, the ReLU function, a softplus function, a swish function, or a tanh function.

[0017] And, the number of the plurality of second sample sounds may be less than the number of the plurality of first sample sounds.

[0018] Meanwhile, according to one embodiment of the present disclosure, a method for controlling an electronic device may include a step of obtaining a neural network model including an acoustic encoder and a text encoder that have learned a relationship between a plurality of first sample sounds and a plurality of sample texts, a step of inputting a target text corresponding to a target keyword into the text encoder to obtain a text embedding, and a step of retraining the neural network model based on a plurality of second sample sounds including the target keyword and the text embedding to obtain a final neural network model in which the acoustic encoder is updated.

[0019] Additionally, the step of obtaining the final neural network model may retrain the neural network model based on the plurality of second sample sounds and the text embedding to obtain the final neural network model in which at least one layer included in the audio encoder is updated.

[0020] And, the step of obtaining the final neural network model can identify at least one layer based on the number of parameters of each of the plurality of layers included in the acoustic encoder.

[0021] Additionally, the at least one layer may include at least one of a batch normalization (BN) layer that determines a feature distribution per channel or a squeeze-and-excitation (SE) module that determines feature importance per channel.

[0022] And, the step of obtaining the final neural network model may include converting a preset activation function included in the audio encoder into a plurality of learnable activation functions, and retraining the neural network model by weighting a plurality of output values ​​output from the plurality of learnable activation functions based on the text embedding.

[0023] In addition, the step of obtaining the final neural network model inputs the text embedding and conditioning parameters into a softmax function, and retrains the neural network model by weighting a plurality of output values ​​based on the softmax function, and the conditioning parameters may have values ​​updated during the retraining process.

[0024] And, the preset activation function includes a ReLU function, and the plurality of learnable activation functions may include at least one of an ELU function, a hard sigmoid function, the ReLU function, a softplus function, a swish function, or a tanh function.

[0025] In addition, the step of obtaining the final neural network model may convert a preset activation function included in the acoustic encoder into a plurality of learnable activation functions, and retrain the neural network model by weighting a plurality of output values ​​output from the plurality of learnable activation functions based on the text embedding.

[0026] And, the step of obtaining the final neural network model inputs the text embedding and conditioning parameters into a softmax function, and retrains the neural network model by weighting a plurality of output values ​​based on the softmax function, and the conditioning parameters may have values ​​updated during the retraining process.

[0027] In addition, the preset activation function may include a ReLU function, and the plurality of learnable activation functions may include at least one of an ELU function, a hard sigmoid function, the ReLU function, a softplus function, a swish function, or a tanh function.

[0028] And, the number of the plurality of second sample sounds may be less than the number of the plurality of first sample sounds.

[0029] FIG. 1A and FIG. 1B are drawings for explaining a keyword spotting (KWS) model to help understand the present disclosure.

[0030] FIG. 2 is a block diagram showing the configuration of an electronic device according to an embodiment of the present disclosure.

[0031] FIG. 3 is a block diagram showing a detailed configuration of an electronic device according to an embodiment of the present disclosure.

[0032] FIG. 4 is a diagram illustrating the structure and learning method of a final neural network model according to one embodiment of the present disclosure.

[0033] FIGS. 5 to 7 are drawings for explaining a method for updating an acoustic encoder according to an embodiment of the present disclosure.

[0034] FIG. 8 is a flowchart for explaining a control method of an electronic device according to an embodiment of the present disclosure.

[0035] The purpose of the present disclosure is to provide an electronic device and a control method thereof that enhance recognition performance for a target keyword.

[0036] Hereinafter, the present disclosure will be described in detail with reference to the attached drawings.

[0037] The terms used in the embodiments of this disclosure have been selected from widely used, current terms, taking into account the functions of this disclosure. However, these terms may vary depending on the intentions of those skilled in the art, precedents, the emergence of new technologies, etc. Furthermore, in certain cases, terms may be arbitrarily selected by the applicant, and in such cases, their meanings will be described in detail in the description of the relevant disclosure. Therefore, the terms used in this disclosure should not be defined simply as names of terms, but rather based on the meanings of the terms and the overall content of this disclosure.

[0038] In this specification, expressions such as “has,” “can have,” “includes,” or “may include” indicate the presence of a feature (e.g., a number, function, operation, or component such as a part), and do not exclude the presence of additional features.

[0039] The expression "at least one of A and / or B" should be understood to mean either "A" or "B" or "A and B".

[0040] As used herein, the expressions “first,” “second,” “first,” or “second,” etc., may describe various components, regardless of order and / or importance, and are only used to distinguish one component from another, but do not limit the components.

[0041] Singular expressions include plural expressions unless the context clearly dictates otherwise. In this application, terms such as "comprise" or "consist of" are intended to indicate the presence of a feature, number, step, operation, component, part, or combination thereof described in the specification, but should be understood not to preclude the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0042] In this specification, the term user may refer to a person using an electronic device or a device using an electronic device (e.g., an artificial intelligence electronic device).

[0043] Various embodiments of the present disclosure will be described in more detail with reference to the attached drawings below.

[0044] FIG. 1A and FIG. 1B are drawings for explaining a keyword spotting (KWS) model to help understand the present disclosure.

[0045] The KWS model can be a model that detects desired keywords from continuous speech input into the system.

[0046] The KWS model using voice registration requires the user to register the target keyword by speaking it multiple times, but the KWS model using text registration allows the user to register the target keyword through text input, which can improve user convenience.

[0047] The KWS model of the text registration method, as illustrated in Fig. 1a, converts registered text into text embedding (TE) through a text encoder, converts sound into sound (acoustic embedding (AE)) through an acoustic encoder, and can determine whether the sound contains the same keyword as the registered text through a similarity comparison of the text embedding and the acoustic embedding.

[0048] However, the KWS model using the text registration method has no restrictions on target keywords, so it can respond to all keywords, and has a problem of lower performance compared to the fixed KWS (fixed keyword detection) model trained using only target keywords. In addition, in order to train the fixed KWS model, data of more than tens of thousands of utterances corresponding to the target keywords are required, which incurs a significant cost.

[0049] The KWS model of the voice registration method is a model that is fine-tuned using a small amount of data on target keywords, as illustrated in Fig. 1b. It can be a method of adding a classification layer that classifies target keywords to a pre-trained model and fine-tuning the classification layer.

[0050] A technique for improving specific performance by fine-tuning a pre-trained model using only a small amount of data is called few-shot transfer learning.

[0051] However, because it uses a voice registration method, its performance may be lower than that of the text registration method, and there may be user inconvenience due to the voice registration process.

[0052] FIG. 2 is a block diagram showing the configuration of an electronic device (100) according to one embodiment of the present disclosure.

[0053] The electronic device (100) is a device for learning a neural network model, and may be implemented as a device such as a desktop PC, laptop, smartphone, tablet PC, server, TV, projector, smart glasses, smart watch, speaker, sound bar, etc. However, the present invention is not limited thereto, and the electronic device (100) may be any device capable of learning a neural network model.

[0054] According to FIG. 2, the electronic device (100) includes a memory (110) and a processor (120).

[0055] Memory (110) may refer to hardware that stores information such as data in an electrical or magnetic form so that a processor (120) or the like can access it. To this end, memory (110) may be implemented as at least one piece of hardware from among non-volatile memory, volatile memory, flash memory, hard disk drive (HDD), solid state drive (SSD), RAM, ROM, etc.

[0056] The memory (110) may store at least one instruction required for the operation of the electronic device (100) or the processor (120). Here, the instruction is a code unit that instructs the operation of the electronic device (100) or the processor (120), and may be written in machine language, which is a language that a computer can understand. Alternatively, the memory (110) may store a plurality of instructions for performing a specific task of the electronic device (100) or the processor (120) as an instruction set.

[0057] The memory (110) may store data, which is information in bit or byte units that can represent characters, numbers, images, etc. For example, a neural network model may be stored in the memory (110).

[0058] Here, the neural network model may include an acoustic encoder and a text encoder that have learned the relationship between a plurality of first sample sounds and a plurality of sample texts. For example, the neural network model is a general-purpose model, and is not limited to specific keywords, and may include an acoustic encoder and a text encoder that have learned the relationship between a plurality of first sample sounds corresponding to a plurality of keywords, respectively, and a plurality of sample texts corresponding to the plurality of first sample sounds, respectively. For example, the neural network model may be trained so that the difference between the acoustic embedding obtained when the sound of uttering "cat" is input to the acoustic encoder decreases and the text embedding obtained when the text "cat" is input to the text encoder, but the neural network model may be trained so that the difference between the acoustic embedding obtained when the sound of uttering "dog" is input to the acoustic encoder increases and the text embedding obtained when the text "cat" is input to the text encoder.

[0059] The memory (110) is accessed by the processor (120), and reading / writing / modifying / deleting / updating instructions, instruction sets, or data can be performed by the processor (120).

[0060] The processor (120) controls the overall operation of the electronic device (100). Specifically, the processor (120) is connected to each component of the electronic device (100) and can control the overall operation of the electronic device (100). For example, the processor (120) is connected to components such as the memory (110) and can control the operation of the electronic device (100).

[0061] The one or more processors (120) may include one or more of a CPU, a GPU (Graphics Processing Unit), an APU (Accelerated Processing Unit), a MIC (Many Integrated Core), an NPU (Neural Processing Unit), a hardware accelerator, or a machine learning accelerator. The one or more processors (120) may control one or any combination of other components of the electronic device (100) and perform operations related to communication or data processing. The one or more processors (120) may execute one or more programs or instructions stored in the memory (110). For example, the one or more processors (120) may perform a method according to an embodiment of the present disclosure by executing one or more instructions stored in the memory (110).

[0062] When a method according to an embodiment of the present disclosure includes multiple operations, the multiple operations may be performed by one processor or by multiple processors. For example, when a first operation, a second operation, and a third operation are performed by a method according to an embodiment, the first operation, the second operation, and the third operation may all be performed by the first processor, or the first operation and the second operation may be performed by the first processor (e.g., a general-purpose processor) and the third operation may be performed by the second processor (e.g., an artificial intelligence-specific processor).

[0063] One or more processors (120) may be implemented as a single core processor including one core, or may be implemented as one or more multicore processors including multiple cores (e.g., homogeneous multicores or heterogeneous multicores). When one or more processors (120) are implemented as a multicore processor, each of the multiple cores included in the multicore processor may include an internal processor memory, such as a cache memory or an on-chip memory, and a common cache shared by the multiple cores may be included in the multicore processor. In addition, each of the multiple cores (or some of the multiple cores) included in the multicore processor may independently read and execute a program instruction for implementing a method according to an embodiment of the present disclosure, or all (or some) of the multiple cores may be linked to read and execute a program instruction for implementing a method according to an embodiment of the present disclosure.

[0064] When a method according to an embodiment of the present disclosure includes a plurality of operations, the plurality of operations may be performed by one core among the plurality of cores included in a multi-core processor, or may be performed by the plurality of cores. For example, when a first operation, a second operation, and a third operation are performed by a method according to an embodiment, the first operation, the second operation, and the third operation may all be performed by a first core included in the multi-core processor, or the first operation and the second operation may be performed by a first core included in the multi-core processor, and the third operation may be performed by a second core included in the multi-core processor.

[0065] In embodiments of the present disclosure, one or more processors (120) may refer to a system on a chip (SoC) in which one or more processors and other electronic components are integrated, a single-core processor, a multi-core processor, or a core included in a single-core processor or a multi-core processor, wherein the core may be implemented as a CPU, a GPU, an APU, a MIC, an NPU, a hardware accelerator, or a machine learning accelerator, but the embodiments of the present disclosure are not limited thereto. However, for convenience of explanation, the operation of the electronic device (100) is described below using the expression processor (120).

[0066] The processor (120) inputs target text corresponding to a target keyword into a text encoder to obtain a text embedding, and retrains a neural network model based on a plurality of second sample sounds including the target keyword and the text embedding, so that the audio encoder can obtain a final neural network model with an updated sound.

[0067] For example, the processor (120) may input target text corresponding to the target keyword "refrigerator" into a text encoder to obtain a text embedding corresponding to "refrigerator", and retrain a neural network model based on a plurality of second sample sounds including "refrigerator" and a text embedding corresponding to "refrigerator" to obtain a final neural network model in which the acoustic encoder is updated. For example, the processor (120) may input a second sample sound corresponding to "refrigerator" among a plurality of second sample sounds into an acoustic encoder of the neural network model to obtain an acoustic embedding, and retrain the neural network model so that the difference between the obtained acoustic embedding and the text embedding corresponding to "refrigerator" is reduced. Alternatively, the processor (120) may input a second sample sound that does not correspond to “refrigerator” among a plurality of second sample sounds into an audio encoder of a neural network model to obtain an audio embedding, and retrain the neural network model so that the difference between the obtained audio embedding and the text embedding corresponding to “refrigerator” increases.

[0068] Here, the number of the plurality of second sample sounds may be smaller than the number of the plurality of first sample sounds. For example, if the number of the plurality of first sample sounds is more than one million, the number of the plurality of second sample sounds may be less than or equal to thirty. In addition, the number of second sample sounds corresponding to the target keyword among the plurality of second sample sounds may be less than or equal to half of the number of the plurality of second sample sounds. In other words, the neural network model is a general-purpose model that learns the relationship between the plurality of first sample sounds and the plurality of sample texts, and the final neural network model may be a specialized model with enhanced target keyword recognition performance in the neural network model. In particular, since the text registration method is used instead of the voice registration method of Fig. 1b, the recognition performance of the target keyword may be enhanced. In addition, since a separate classification layer is not added as in Fig. 1b, and the text encoder of the neural network model is fixed and only the audio encoder is updated, learning is possible with a smaller amount of sample data than in Fig. 1b.

[0069] The processor (120) can retrain the neural network model based on a plurality of second sample sounds and text embeddings to obtain a final neural network model in which at least one layer included in the audio encoder is updated.

[0070] For example, the processor (120) may identify at least one layer based on the number of parameters of each of the plurality of layers included in the acoustic encoder. For example, the at least one layer may include at least one of a batch normalization (BN) layer that determines a feature distribution per channel or a squeeze-and-excitation (SE) module that determines feature importance per channel.

[0071] That is, the processor (120) may update only some layers included in the acoustic encoder without updating all parameters included in the acoustic encoder, thereby enabling learning with a small amount of sample data.

[0072] The processor (120) can convert a preset activation function included in the audio encoder into a plurality of learnable activation functions, and retrain the neural network model by weighting a plurality of output values ​​output from the plurality of learnable activation functions based on text embedding.

[0073] For example, the processor (120) may input text embedding and conditioning parameters into a softmax function, and retrain the neural network model by weighting multiple output values ​​based on the softmax function. Here, the conditioning parameters may have updated values ​​during the retraining process, the preset activation function may include a ReLU function, and the multiple learnable activation functions may include at least one of an ELU function, a hard sigmoid function, a ReLU function, a softplus function, a swish function, or a tanh function.

[0074] This conditioning can further enhance the recognition performance of target keywords.

[0075] As described above, the processor (120) may retrain only the acoustic encoder or only some layers of the acoustic encoder. Alternatively, the processor (120) may perform retraining through retraining and conditioning some layers of the acoustic encoder.

[0076] However, it is not limited to this, and the processor (120) may only perform relearning through conditioning.

[0077] For example, the processor (120) can convert a preset activation function included in the audio encoder into a plurality of learnable activation functions, and retrain the neural network model by weighting a plurality of output values ​​output from the plurality of learnable activation functions based on text embedding.

[0078] For example, the processor (120) may input text embedding and conditioning parameters into a softmax function, and retrain the neural network model by weighting multiple output values ​​based on the softmax function. Here, the conditioning parameters may have updated values ​​during the retraining process, the preset activation function may include a ReLU function, and the multiple learnable activation functions may include at least one of an ELU function, a hard sigmoid function, a ReLU function, a softplus function, a swish function, or a tanh function.

[0079] Meanwhile, functions related to artificial intelligence according to the present disclosure can be operated through a processor (120) and a memory (110).

[0080] The processor (120) may be composed of one or more processors. In this case, one or more processors may be a general-purpose processor such as a CPU, AP, DSP, etc., a graphics-only processor such as a GPU or VPU (Vision Processing Unit), or an artificial intelligence-only processor such as an NPU.

[0081] One or more processors are controlled to process input data according to predefined operating rules or artificial intelligence models stored in the memory (110). Alternatively, if one or more processors are dedicated artificial intelligence processors, the dedicated artificial intelligence processors may be designed with a hardware structure specialized for processing a specific artificial intelligence model. The predefined operating rules or artificial intelligence models are characterized by being created through learning.

[0082] Here, "created through learning" means that a basic artificial intelligence model is learned using a learning algorithm using a plurality of learning data, thereby creating a predefined set of operating rules or an artificial intelligence model set to perform a desired characteristic (or purpose). This learning may be performed on the device itself on which the artificial intelligence according to the present disclosure is performed, or may be performed through a separate server and / or system. Examples of learning algorithms include, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning.

[0083] An artificial intelligence model may be composed of multiple neural network layers. Each of the multiple neural network layers has multiple weight values ​​and performs neural network operations by calculating the results of previous layers and the multiple weights. The multiple weights of the multiple neural network layers can be optimized based on the learning results of the artificial intelligence model. For example, the multiple weights may be updated during the learning process to reduce or minimize the loss or cost values ​​obtained by the artificial intelligence model.

[0084] Artificial neural networks may include deep neural networks (DNNs), such as, but not limited to, convolutional neural networks (CNNs), deep neural networks (DNNs), recurrent neural networks (RNNs), restricted boltzmann machines (RBMs), deep belief networks (DBNs), bidirectional recurrent deep neural networks (BRDNNs), generative adversarial networks (GANs), or deep Q-networks.

[0085] FIG. 3 is a block diagram showing a detailed configuration of an electronic device (100) according to one embodiment of the present disclosure.

[0086] The electronic device (100) may include a memory (110) and a processor (120). In addition, according to FIG. 3, the electronic device (100) may further include a microphone (130), a display (140), a speaker (150), a communication interface (160), a user interface (170), and a camera (180). Among the components illustrated in FIG. 3, a detailed description of the overlapping parts with the components illustrated in FIG. 2 will be omitted.

[0087] The microphone (130) is configured to receive sound and convert it into an audio signal. The microphone (130) is electrically connected to the processor (120) and can receive sound under the control of the processor (120).

[0088] For example, the microphone (130) may be formed as an integrated unit integrated into the upper side, front side, side side, etc. of the electronic device (100). Alternatively, the microphone (130) may be provided in a remote control, etc., separate from the electronic device (100). In this case, the remote control may receive sound through the microphone (130) and provide the received sound to the electronic device (100).

[0089] The microphone (130) may include various configurations such as a microphone that collects analog sound, an amplifier circuit that amplifies the collected sound, an A / D conversion circuit that samples the amplified sound and converts it into a digital signal, and a filter circuit that removes noise components from the converted digital signal.

[0090] Meanwhile, the microphone (130) may be implemented in the form of a sound sensor, and any method may be used as long as it has a configuration capable of collecting sound.

[0091] The display (140) is a configuration that displays an image and can be implemented as a display of various forms such as an LCD (Liquid Crystal Display), an OLED (Organic Light Emitting Diodes) display, a PDP (Plasma Display Panel), etc. The display (140) may also include a driving circuit, a backlight unit, etc. that can be implemented as a form such as an a-si TFT, an LTPS (low temperature poly silicon) TFT, an OTFT (organic TFT), etc. Meanwhile, the display (140) may be implemented as a touch screen combined with a touch sensor, a flexible display, a 3D display, etc.

[0092] The speaker (150) is a component that outputs various audio data processed by the processor (120) as well as various notification sounds and voice messages.

[0093] The communication interface (160) is a configuration that performs communication with various types of external devices according to various types of communication methods. For example, the electronic device (100) can perform communication with a server through the communication interface (160).

[0094] The communication interface (160) may include a Wi-Fi module, a Bluetooth module, an infrared communication module, a wireless communication module, etc. Here, each communication module may be implemented in the form of at least one hardware chip.

[0095] Wi-Fi and Bluetooth modules communicate via Wi-Fi and Bluetooth, respectively. When using a Wi-Fi or Bluetooth module, connection information, such as the SSID and session key, is first transmitted and received. This information is then used to establish a communication connection before various other information can be transmitted and received. Infrared communication modules use infrared data association (IrDA) technology, which wirelessly transmits data over short distances using infrared light, which lies between visible light and millimeter waves.

[0096] In addition to the above-described communication method, the wireless communication module may include at least one communication chip that performs communication according to various wireless communication standards such as zigbee, 3G (3rd Generation), 3GPP (3rd Generation Partnership Project), LTE (Long Term Evolution), LTE-A (LTE Advanced), 4G (4th Generation), 5G (5th Generation), etc.

[0097] Alternatively, the communication interface (160) may include a wired communication interface such as HDMI, DP, Thunderbolt, USB, RGB, D-SUB, DVI, etc.

[0098] In addition, the communication interface (160) may include at least one of a LAN (Local Area Network) module, an Ethernet module, or a wired communication module that performs communication using a pair cable, a coaxial cable, or an optical fiber cable.

[0099] The user interface (170) may be implemented with buttons, a touch pad, a mouse, a keyboard, etc., or may be implemented with a touch screen capable of performing both display and operation input functions. Here, the buttons may be various types of buttons, such as mechanical buttons, touch pads, wheels, etc., formed on any area of ​​the front, side, or back of the main body of the electronic device (100).

[0100] The camera (180) is configured to capture still images or moving images. The camera (180) can capture still images at a specific point in time, but can also capture still images continuously.

[0101] The camera (180) includes a lens, a shutter, an aperture, a solid-state image sensor, an AFE (Analog Front End), and a TG (Timing Generator). The shutter controls the time at which light reflected from a subject enters the camera (180), and the aperture mechanically increases or decreases the size of the opening through which light enters to control the amount of light incident on the lens. When the solid-state image sensor accumulates light reflected from a subject as a photocharge, the image generated by the photocharge is output as an electrical signal. The TG outputs a timing signal for reading out pixel data of the solid-state image sensor, and the AFE samples and digitizes the electrical signal output from the solid-state image sensor.

[0102] As described above, the electronic device (100) can obtain a final neural network model with enhanced target keyword recognition performance from the general-purpose neural network model. Furthermore, the electronic device (100) can retrain even with a small amount of sample data by updating only a portion of the general-purpose neural network model without adding a classification layer.

[0103] Hereinafter, the operation of the electronic device (100) will be described in more detail with reference to FIGS. 4 to 7. For convenience of explanation, individual embodiments are described in FIGS. 4 to 7. However, the individual embodiments of FIGS. 4 to 7 may be implemented in any combination.

[0104] FIG. 4 is a diagram illustrating the structure and learning method of a final neural network model according to one embodiment of the present disclosure.

[0105] The processor (120) may acquire a neural network model including an acoustic encoder and a text encoder that have learned the relationship between a plurality of first sample sounds and a plurality of sample texts. For example, the processor (120) may receive the neural network model from a server. Alternatively, the processor (120) may acquire a neural network model by learning a neural network model including an acoustic encoder and a text encoder based on a plurality of first sample sounds and a plurality of sample texts. Here, the neural network model may be a general-purpose neural network model.

[0106] When the processor (120) directly learns the neural network model, as illustrated in FIG. 4, the first sample sound can be input into the acoustic encoder (10) to obtain an acoustic embedding (11), and the sample text can be input into the text encoder (20) to obtain a text embedding (21). The processor (120) can identify the similarity of the acoustic embedding (11) and the text embedding (21) through the FC layer (30) and σ(·) (31), and can learn to reduce the difference between the acoustic embedding (11) and the text embedding (21) when the first sample sound corresponds to the sample text, and can learn to increase the difference between the acoustic embedding (11) and the text embedding (21) when the first sample sound does not correspond to the sample text. The processor (120) can learn the neural network model by repeating this operation for a plurality of first sample sounds and a plurality of sample texts. Here, the correspondence between the first sample sound and the sample text may be information provided to the processor (120) or identifiable by the processor (120). The neural network model acquired through this learning process may be a general-purpose neural network model. For convenience of explanation, the above learning process is referred to as primary learning below.

[0107] The processor (120) can retrain the neural network model to obtain a final neural network model with improved recognition performance of the target keyword. For example, as illustrated in FIG. 4, the processor (120) inputs target text corresponding to the target keyword into the text encoder (20) to obtain a text embedding (21), and retrains the neural network model based on a plurality of second sample sounds containing the target keyword and the text embedding, thereby obtaining a final neural network model with an updated audio encoder. For example, the processor (120) may input the second sample sound into the audio encoder (10) to obtain an audio embedding (11), identify the similarity of the audio embedding (11) and the text embedding (21) through the FC layer (30) and σ(·) (31), and retrain the neural network model to reduce the difference between the audio embedding (11) and the text embedding (21) if the second sample sound corresponds to the target text, and retrain the neural network model to increase the difference between the audio embedding (11) and the text embedding (21) if the second sample sound does not correspond to the target text. The processor (120) may retrain the neural network model by repeating this operation for a plurality of second sample sounds. Here, whether the second sample sound corresponds to the target text may be information that is provided to the processor (120) or that the processor (120) can identify.

[0108] The above method is called a TE classifier, and it is possible to obtain effects similar to those using a classification layer without using a classification layer.

[0109] For convenience of explanation, if retraining is described as secondary training, then in the first training process, the sample text may be changed and the text embedding may be changed accordingly, but in the second training process, the text embedding is obtained from the target text, and the text embedding may be used as is thereafter. The plurality of first sample sounds used in the first training process have a considerably large capacity, and therefore, the neural network model that has completed the first training may be a general-purpose neural network model, but the plurality of second sample sounds used in the second training process have a very small capacity, and therefore, the neural network model that has completed the second training may be a model with improved target keyword recognition performance. In addition, all the components of FIG. 4 are updated in the first training process, but only the audio encoder (10) may be updated in the second training process.

[0110] As described above, since only the audio encoder (10) is updated in the second learning process, learning is possible even with a small amount of second sample sound, and the recognition performance of the target keyword can be improved.

[0111] After relearning is completed, when a sound is received through a microphone (130), the processor (120) can obtain the probability that the sound x is the target keyword k by applying the sigmoid aictivaiton function (σ) to the cosine similarity (S(·, ·)) of the text embedding (TE) and the acoustic embedding (AE), as follows.

[0112] p(k|x) = σ(θk·AE) = σ(S(TE, AE))

[0113] FIGS. 5 to 7 are drawings for explaining a method for updating an acoustic encoder according to an embodiment of the present disclosure.

[0114] The processor (120) can obtain a final neural network model in which the acoustic encoder included in the neural network model is updated through a retraining process. In particular, the processor (120) may retrain only some layers of the acoustic encoder. Alternatively, the processor (120) may only perform retraining through conditioning. Alternatively, the processor (120) may perform retraining of some layers of the acoustic encoder and retraining through conditioning. That is, the retraining of some layers of the acoustic encoder and retraining through conditioning may be performed in parallel. In the following, the retraining of some layers of the acoustic encoder will be described first, and the retraining through conditioning will be described thereafter.

[0115] First, the retraining method of some layers of the acoustic encoder is called a feature weight adapter (FW-adapter). Since the acoustic encoder was trained using sufficient data for each keyword in the first training phase, the keyword features required for keyword classification may already have been learned. Therefore, in situations where a neural network model needs to adapt to target keywords with a small amount of sample data, it may be reasonable to modify only the way in which well-trained features from the first training phase are aggregated and propagated to higher layers.

[0116] The processor (120) can retrain the neural network model based on a plurality of second sample sounds and text embeddings to obtain a final neural network model in which at least one layer included in the audio encoder is updated. For example, the processor (120) can obtain a final neural network model by updating only 520 layers, as illustrated in FIG. 5 . For example, the processor (120) can obtain a final neural network model by updating at least one of a batch normalization (BN) layer that determines a feature distribution for each channel or a squeeze-and-excitation (SE) module that determines a feature importance for each channel as at least one layer.

[0117] The BN layer and SE module are layers with relatively few parameters, making them suitable for few-shot transfer learning. However, this is not a limitation, and any other type of layer may be retrained. For example, the processor (120) may identify at least one layer based on the number of parameters of each of the multiple layers included in the audio encoder.

[0118] While at least one layer is being trained, the parameters of other layers may remain fixed. That is, if only a retraining method of some layers of the acoustic encoder is used without retraining through conditioning, only the parameters of at least one layer (520) may be updated, and the parameters of the remaining layers (510, 530) may remain fixed.

[0119] The relearning method through conditioning is called text-conditioned feature modulation (TCFM), and can be a method of extracting features adapted to target keywords by conditioning text embeddings that contain information about target keywords.

[0120] For example, the processor (120) may convert a preset activation function (510) included in the audio encoder into a plurality of learnable activation functions, as illustrated in FIG. 5, and retrain the neural network model by weighting a plurality of output values ​​output from the plurality of learnable activation functions based on text embedding. For example, the processor (120) may input text embedding and conditioning parameters into a softmax function, and retrain the neural network model by weighting a plurality of output values ​​based on the softmax function. The conditioning parameters may have updated values ​​during the retraining process. Here, the preset activation function may include a ReLU function, and the plurality of learnable activation functions may include at least one of an ELU function, a hard sigmoid function, a ReLU function, a softplus function, a swish function, or a tanh function.

[0121] Figure 6 illustrates conditioning text embeddings containing target keyword information using Adaptive Instance Normalization (AdaIN). However, this method may be unsuitable for few-shot transfer learning, which requires a small amount of adaptive data, because it incurs a large number of additional parameters.

[0122] FIG. 7 is a diagram illustrating a process for converting a preset activation function into multiple learnable activation functions and conditioning text embeddings according to an embodiment of the present disclosure.

[0123] The processor (120) can perform conditioning and weighted sum operations of text embedding (TE) based on the following formula.

[0124] s = softmax(TE·w + b), y = Σsi * Ai(h)

[0125] A weighted sum of Ai, which is a set of multiple learnable activation functions, is output. The weights si of each Ai can be estimated from text embeddings, and w and b can be conditioning parameters. The conditioning parameters w and b are learned so that information about target keywords can be conditioned on the acoustic encoder, thereby extracting features adapted to the target keywords.

[0126] While a plurality of learnable activation functions transformed from a preset activation function (510) are learned, the parameters of other layers may remain fixed. That is, if only a relearning method through conditioning is used without relearning some layers of the acoustic encoder, only the parameters of the plurality of learnable activation functions transformed from a preset activation function (510) may be updated, and the parameters of the remaining layers (520, 530) may remain fixed.

[0127] For convenience of explanation, the retraining of some layers of the acoustic encoder and retraining through conditioning are described as being performed separately, but this is not limited thereto. For example, the retraining of some layers of the acoustic encoder and retraining through conditioning may be performed simultaneously. In this case, the parameters of at least one layer (520) and the parameters of multiple learnable activation functions converted from the preset activation function (510) may be updated, and the parameters of the remaining layers (530) may be fixed.

[0128] FIG. 8 is a flowchart for explaining a control method of an electronic device according to an embodiment of the present disclosure.

[0129] First, a neural network model including an audio encoder and a text encoder that have learned the relationship between a plurality of first sample sounds and a plurality of sample texts is obtained (S810). Then, a target text corresponding to a target keyword is input into the text encoder to obtain a text embedding (S820). Then, the neural network model is retrained based on a plurality of second sample sounds including the target keyword and the text embedding to obtain a final neural network model in which the audio encoder is updated (S830).

[0130] In addition, the step of obtaining a final neural network model (S830) may retrain the neural network model based on a plurality of second sample sounds and text embeddings to obtain a final neural network model in which at least one layer included in the audio encoder is updated.

[0131] And, the step of obtaining the final neural network model (S830) can identify at least one layer based on the number of parameters of each of the multiple layers included in the acoustic encoder.

[0132] Additionally, at least one layer may include at least one of a batch normalization (BN) layer that determines a channel-wise feature distribution or a squeeze-and-excitation (SE) module that determines a channel-wise feature importance.

[0133] And, the step (S830) of obtaining the final neural network model can retrain the neural network model by converting the preset activation function included in the audio encoder into a plurality of learnable activation functions and weighting a plurality of output values ​​output from the plurality of learnable activation functions based on the text embedding.

[0134] In addition, the step of obtaining the final neural network model (S830) inputs text embedding and conditioning parameters into a softmax function, and retrains the neural network model by weighting multiple output values ​​based on the softmax function, and the conditioning parameters may have values ​​updated during the retraining process.

[0135] And, the preset activation function includes a ReLU function, and the plurality of learnable activation functions may include at least one of an ELU function, a hard sigmoid function, a ReLU function, a softplus function, a swish function, or a tanh function.

[0136] In addition, the step of obtaining the final neural network model (S830) can retrain the neural network model by converting a preset activation function included in the audio encoder into a plurality of learnable activation functions and weighting a plurality of output values ​​output from the plurality of learnable activation functions based on text embedding.

[0137] And, the step (S830) of obtaining the final neural network model inputs text embedding and conditioning parameters into a softmax function, and retrains the neural network model by weighting multiple output values ​​based on the softmax function, and the conditioning parameters may have values ​​updated during the retraining process.

[0138] Additionally, the preset activation function may include a ReLU function, and the plurality of learnable activation functions may include at least one of an ELU function, a hard sigmoid function, a ReLU function, a softplus function, a swish function, or a tanh function.

[0139] And, the number of the plurality of second sample sounds may be less than the number of the plurality of first sample sounds.

[0140] According to the various embodiments of the present disclosure, an electronic device can obtain a final neural network model with enhanced target keyword recognition performance from a general-purpose neural network model. Furthermore, the electronic device can retrain even with a small amount of sample data by updating only a portion of the general-purpose neural network model without adding a classification layer.

[0141] Electronic devices according to one or more embodiments disclosed herein may take various forms. Electronic devices may include, for example, portable communication devices (e.g., smartphones), computer devices, portable multimedia devices, portable medical devices, cameras, wearable devices, or home appliances. Electronic devices according to embodiments disclosed herein are not limited to the aforementioned devices.

[0142] It should be understood that one or more embodiments of the present disclosure and terminology used therein are not intended to limit the technical features described in the present disclosure to specific embodiments, but rather to encompass various modifications, equivalents, or substitutes of the embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of the items, unless the context clearly dictates otherwise. In the present disclosure, phrases such as "A or B," "at least one of A and B," "at least one of A or B," "A, B, or C," "at least one of A, B, and C," and "at least one of A, B, or C" can each include any one of the items listed together in the corresponding phrase, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used merely to distinguish one component from another, and do not limit the components in any other respect (e.g., importance or order). When a component (e.g., a first component) is referred to as "coupled" or "connected" to another component (e.g., a second component), with or without the terms "functionally" or "communicatively," it means that the component can be connected to the other component directly (e.g., wired), wirelessly, or through a third component.

[0143] The term "module" as used in one or more embodiments of the present disclosure may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be an integral component, or a minimum unit or part of such a component that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0144] One or more embodiments of the present disclosure may be implemented as software (e.g., a program (140)) including one or more instructions stored in a storage medium (e.g., an internal memory (136) or an external memory (138)) readable by a machine (e.g., an electronic device (101)). For example, a processor (e.g., a processor (120)) of the machine (e.g., an electronic device (101)) may call at least one instruction among the one or more instructions stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' simply means that the storage medium is a tangible device and does not contain signals (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently or temporarily on the storage medium.

[0145] According to one embodiment, a method according to one or more embodiments disclosed in the present disclosure may be provided as a computer program product. The computer program product may be traded as a commodity between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) via an application store (e.g., Play Store™) or directly between two user devices (e.g., smart phones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily generated in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0146] According to one or more embodiments, each component (e.g., a module or a program) of the above-described components may include one or more entities, and some of the entities may be separated and placed in other components. According to one or more embodiments, one or more components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (e.g., a module or a program) may be integrated into a single component. In such a case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by the corresponding component of the plurality of components prior to the integration. According to one or more embodiments, the operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

Claims

1. In electronic devices, A neural network model including an acoustic encoder and a text encoder that have learned the relationship between a plurality of first sample sounds and a plurality of sample texts, and a memory that stores instructions; and one or more processors including processing circuitry; The above instructions, when individually or collectively executed by the one or more processors, Input the target text corresponding to the target keyword into the text encoder to obtain a text embedding, An electronic device that retrains the neural network model based on a plurality of second sample sounds including the target keyword and the text embedding, thereby obtaining an updated final neural network model of the audio encoder.

2. In paragraph 1, The above instructions, when individually or collectively executed by one or more processors, An electronic device that retrains the neural network model based on the plurality of second sample sounds and the text embedding to obtain the final neural network model in which at least one layer included in the audio encoder is updated.

3. In paragraph 2, The above instructions, when individually or collectively executed by one or more processors, An electronic device that identifies at least one layer based on the number of parameters of each of the plurality of layers included in the acoustic encoder.

4. In paragraph 2, At least one layer above, An electronic device comprising at least one of a batch normalization (BN) layer for determining a channel-specific feature distribution or a squeeze-and-excitation (SE) module for determining a channel-specific feature importance.

5. In paragraph 2, The above instructions, when individually or collectively executed by one or more processors, Converting the preset activation function included in the above acoustic encoder into multiple learnable activation functions, An electronic device that retrains the neural network model by weighting a plurality of output values ​​output from the plurality of learnable activation functions based on the text embedding.

6. In paragraph 5, The above instructions, when individually or collectively executed by one or more processors, Input the above text embedding and conditioning parameters into the softmax function, Retraining the neural network model by weighting multiple output values ​​based on the above softmax function, The above conditioning parameters are: An electronic device whose values ​​are updated during the above relearning process.

7. In paragraph 5, The above preset activation function is, Includes ReLU function, The above multiple learnable activation functions are, An electronic device comprising at least one of an ELU function, a hard sigmoid function, the ReLU function, a softplus function, a swish function, or a tanh function.

8. In paragraph 1, The above instructions, when individually or collectively executed by one or more processors, Converting the preset activation function included in the above acoustic encoder into multiple learnable activation functions, An electronic device that retrains the neural network model by weighting a plurality of output values ​​output from the plurality of learnable activation functions based on the text embedding.

9. In paragraph 8, The above instructions, when individually or collectively executed by one or more processors, Input the above text embedding and conditioning parameters into the softmax function, Retraining the neural network model by weighting multiple output values ​​based on the above softmax function, The above conditioning parameters are: An electronic device whose values ​​are updated during the above relearning process.

10. In paragraph 8, The above preset activation function is, Includes ReLU function, The above multiple learnable activation functions are, An electronic device comprising at least one of an ELU function, a hard sigmoid function, the ReLU function, a softplus function, a swish function, or a tanh function.

11. In paragraph 1, The above plurality of second sample sounds are, An electronic device having a number less than the plurality of first sample sounds.

12. In a method for controlling an electronic device, A step of obtaining a neural network model including an acoustic encoder and a text encoder that have learned a relationship between a plurality of first sample sounds and a plurality of sample texts; A step of obtaining a text embedding by inputting a target text corresponding to a target keyword into the text encoder; and A control method comprising: a step of retraining the neural network model based on a plurality of second sample sounds including the target keyword and the text embedding, thereby obtaining an updated final neural network model of the audio encoder.

13. In paragraph 12, The steps for obtaining the final neural network model are: A control method for retraining the neural network model based on the plurality of second sample sounds and the text embedding to obtain the final neural network model in which at least one layer included in the audio encoder is updated.

14. In paragraph 13, The steps for obtaining the final neural network model are: A control method for identifying at least one layer based on the number of parameters of each of the plurality of layers included in the above acoustic encoder.

15. In paragraph 13, At least one layer above, A control method comprising at least one of a batch normalization (BN) layer for determining a channel-specific feature distribution or a squeeze-and-excitation (SE) module for determining a channel-specific feature importance.

Images