Electronic device and control method thereof

Abstract

An electronic device and a method for controlling the electronic device are disclosed. The electronic device of the present disclosure includes a microphone, a memory storing at least one instruction, and a processor configured to execute the at least one instruction. The processor is configured to: obtain second voice data by inputting first voice data input via the microphone into a first model trained to enhance sound quality, obtain weights by inputting the first voice data and the second voice data into the second model, and identify input data to be input into a third model using the weights by executing the at least one instruction.

Description

Technical Field

[0001] The present disclosure relates to an electronic device and a method for controlling the electronic device, and for example, to an electronic device that determines data to be input to a specific model based on voice data and data obtained by applying an algorithm for enhancing sound quality to the voice data, and a method for controlling the electronic device. Background Art

[0002] Various algorithms (eg, algorithms for filtering noise contained in a speech signal, algorithms based on beamforming, algorithms for mapping features based on neural networks, etc.) have conventionally been developed and used to enhance the sound quality of speech.

[0003] When the algorithm for enhancing sound quality is applied to the user's voice input in a normal noisy environment, the sound quality of the voice can be enhanced. However, if the algorithm for enhancing sound quality is applied to the user's voice input in a clean environment with relatively little or no noise, the sound quality of the voice may deteriorate.

[0004] In other words, conventional enhancement algorithms map input data to target data to minimize or maximize a determined objective function, but do not measure the reliability of the mapped output data. Therefore, enhancement algorithms may degrade rather than enhance the sound quality of speech in certain situations. Summary of the invention

[0005] Technical issues

[0006] The present disclosure is made in view of the above needs, and provides an electronic device and a method for controlling the electronic device for determining input data to be input to a specific model based on voice data and output data obtained by inputting the voice data to a model for enhancing sound quality.

[0007] Solution to the problem

[0008] Embodiments of the present disclosure provide an electronic device and a method for controlling the electronic device for determining input data to be input to a specific model based on voice data and output data obtained by inputting the voice data to a model for enhancing sound quality.

[0009] According to an example embodiment of the present disclosure, an electronic device is provided, which includes: a microphone, a memory storing at least one instruction, and a processor configured to execute the at least one instruction, wherein the processor is configured by executing the at least one instruction to: obtain second voice data by inputting first voice data input via the microphone into a first model trained to enhance sound quality, obtain weights by inputting the first voice data and the second voice data into the second model, and use the weights to identify input data to be input into a third model.

[0010] The processor may obtain the weight using the first voice data, the second voice data, and an estimate of an error between the second voice data and clean voice data corresponding to the first voice data.

[0011] The processor may linearly combine the first voice data and the second voice data based on the weight; and identify a value of the linear combination as input data.

[0012] The processor may identify one of the first voice data and the second voice data as input data based on the weight.

[0013] The processor may identify the second voice data as input data based on the weight being greater than a threshold value, and identify the first voice data as input data based on the weight being less than the threshold value.

[0014] The processor can identify the second voice data as input data based on a weight greater than a first threshold, linearly combine the first voice data and the second voice data based on the weight and identify the value of the linear combination as input data based on a weight greater than a second threshold and less than the first threshold, and identify the first voice data as input data based on a weight less than a second threshold, wherein the first threshold can be a value greater than the second threshold.

[0015] The third model may be at least one of a wake-up model or an automatic speech recognition model.

[0016] The processor may train the first model and the second model based on the first voice data, clean voice data corresponding to the first voice data, and the second voice data.

[0017] The processor can obtain a first score corresponding to the first voice data and a second score corresponding to the second voice data by inputting each of the first voice data and the second voice data into the third model, and train the second model based on the first voice data, the second voice data, the first score and the second score.

[0018] The processor may obtain an estimated value of a difference between the first score and the second score by inputting the first voice data and the second voice data into the second model, and obtain a weight based on the estimated value of the difference between the first score and the second score.

[0019] According to another example embodiment of the present disclosure, a method for controlling an electronic device including a microphone includes: obtaining second voice data by inputting first voice data input via the microphone into a first model trained to enhance sound quality; obtaining weights by inputting the first voice data and the second voice data into the second model; and identifying input data to be input into a third model using the weights.

[0020] Obtaining may include obtaining a weight based on the first speech data, the second speech data, and an estimate of an error between the second speech data and clean speech data corresponding to the first speech data.

[0021] The recognizing may include linearly combining the first voice data and the second voice data based on a weight, and recognizing a value of the linear combination as input data.

[0022] The recognizing may include recognizing one of the first voice data and the second voice data as input data based on the weight.

[0023] The recognizing may include recognizing the second voice data as input data based on the weight being greater than a threshold value, and recognizing the first voice data as input data based on the weight being less than the threshold value.

[0024] The determination may include identifying the second voice data as input data based on a weight being greater than a first threshold, linearly combining the first voice data and the second voice data based on the weight and identifying the value of the linear combination as input data based on a weight being greater than a second threshold and less than a first threshold, and identifying the first voice data as input data based on a weight being less than a second threshold, wherein the first threshold is a value greater than the second threshold.

[0025] The third model may be at least one of a wake-up model or an automatic speech recognition model.

[0026] The method may also include training the first model and the second model based on the first speech data, clean speech data corresponding to the first speech data, and the second speech data.

[0027] The method may also include obtaining a first score corresponding to the first voice data and obtaining a second score corresponding to the second voice data by inputting each of the first voice data and the second voice data into a third model, and training a second model based on the first voice data, the second voice data, the first score, and the second score.

[0028] Obtaining the weight may include obtaining an estimated value of a difference between the first score and the second score by inputting the first voice data and the second voice data into the second model, and obtaining the weight based on the estimated value of the difference between the first score and the second score.

[0029] Advantageous Effects of the Invention

[0030] As described above, according to an embodiment of the present disclosure, a user can use a voice recognition system more efficiently because the electronic device determines (e.g., recognizes) data to be input into a specific model based on voice data and data obtained by applying an algorithm for enhancing sound quality to the voice data. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The above and other aspects, features and advantages of certain embodiments of the present disclosure will become more apparent from the following detailed description taken in conjunction with the accompanying drawings, in which:

[0032] Figure 1 is a block diagram illustrating an example configuration and operation of an electronic device according to an embodiment;

[0033] Figure 2A and Figure 2B is a block diagram illustrating an example process of an electronic device training a second model according to an embodiment;

[0034] Figure 3 and Figure 4 is a flowchart illustrating an example process of an electronic device determining input data using obtained weights according to an embodiment;

[0035] Figure 5 is a block diagram showing an example configuration of an electronic device according to an embodiment; and

[0036] Figure 6 is a flow chart illustrating an example method for controlling an electronic device according to an embodiment. DETAILED DESCRIPTION

[0037] The present disclosure is made in view of the above needs, and provides an electronic device and a method for controlling the electronic device for determining input data to be input to a specific model based on voice data and output data obtained by inputting the voice data to a model for enhancing sound quality.

[0038] Figure 1The electronic device 100 shown can determine (e.g., identify) input data to be input to a specific model based on first voice data input via microphone 110 and second voice data obtained by applying an algorithm for enhancing sound quality to the first voice data. If the algorithm for enhancing sound quality is applied to the first voice data input via microphone 110 in a clean environment, the sound quality of the first voice data may be relatively deteriorated. Therefore, the electronic device 100 can perform a process of determining the voice data to be input to a specific model based on the first voice data and the second voice data. The process of determining (e.g., identifying) the voice data to be input to a specific model by the electronic device 100 will be described in more detail below.

[0039] Hereinafter, example embodiments of the present disclosure will be described in more detail with reference to the accompanying drawings.

[0040] Figure 1 1 is a diagram showing an example configuration of an example electronic device according to an embodiment of the present disclosure and helping to describe an example operation of the electronic device. Figure 1 As shown, the electronic device 100 may include a microphone 110, a memory 120, and a processor (eg, including a processing circuit) 130. However, Figure 1 The illustrated configuration is an example for implementing an embodiment of the present disclosure, and appropriate hardware or software components apparent to those skilled in the art may be additionally added to the electronic device 100 .

[0041] The microphone 110 may be, for example, a component for receiving a voice from a user. The microphone 110 may receive first voice data from the user. The first voice data may be affected by the surrounding environment because it is voice data input by the user's speech from outside the electronic device 100. Therefore, the first voice data input in an environment with a lot of noise may have interference and reverberation effects due to the noise, and the first voice data input in a clean environment with relatively less noise may have relatively less interference and reverberation effects due to the noise. The term "user" may refer to a person using an electronic device or a device (e.g., an artificial intelligence electronic device) using an electronic device.

[0042] The microphone 110 may be provided in the electronic device 100, or may be provided outside the electronic device 100 and electrically connected to the electronic device 100. In addition, if the microphone 110 is provided externally, the microphone 110 may transmit a user voice signal generated via a wired or wireless interface (e.g., Wi-Fi or Bluetooth) to the processor 130.

[0043] The memory 120 may store instructions or data related to at least another component of the electronic device 100. The instruction may refer to, for example, an action statement directly executed by the processor 130 in a programming language and may include, for example, a minimum unit of execution or action of a program. The processor 130 may access the memory 120, and may perform reading, recording, editing, deleting, or updating of data by the processor 130.

[0044] The memory 120 may store models capable of performing various functions. For example, the memory 120 may store a first model 10, a second model 20, an input data determination model 30, and a third model 40, and each model may be controlled / executed by the processor 130. The various models disclosed and described herein may include, for example, processing circuits and / or executable program elements executed by processing circuits.

[0045] For example, the first model 10 may be a model that applies an algorithm for enhancing sound quality to first voice data input via the microphone 110. Therefore, the first model 10 may output second voice data by applying an algorithm for enhancing sound quality to the first voice data input via the microphone 110 under the control of the processor 130. The first model 10 may be implemented as a model that applies an algorithm for filtering noise or an algorithm based on beamforming to the first voice data. In another example, the first model 10 may be implemented as a neural network model (e.g., a Gaussian mixture model (GMM)) that executes an algorithm for removing various noises and adjusting reverberation.

[0046] For example, the second model 20 may be a neural network model trained to output weights when the second voice data and the first voice data are input under the control of the processor 130. The weights may refer to, for example, parameters for adjusting the effect of the input data applied to the output data, and may be elements capable of determining (e.g., identifying) input data to be input to the third model 40.

[0047] In an embodiment of the present disclosure, in response to inputting the first voice data and the second voice data, the trained second model 20 can obtain, for example, an estimated value of the mean square error between the second voice data and the clean voice data corresponding to the first voice data. The trained second model 20 can output weights based on the obtained estimated value of the mean square error, the first voice data, and the second voice data. Clean voice data can refer to, for example, voice data input in a clean environment without external influences such as noise.

[0048] The estimated value of the mean square error between the second voice data and the clean voice data corresponding to the first voice data can be inversely proportional to the reliability of the second voice data. The larger the estimated value of the mean square error, the greater the difference between the second voice data and the clean voice data corresponding to the first voice data. Therefore, the larger the estimated value of the error, the lower the reliability of the second voice data.

[0049] The second model 20 may be trained using the value of the mean square error between the clean speech data corresponding to the first speech data and the second speech data as learning data under the control of the processor 130. Figure 2A The process of training the second model 20 based on the value of the mean square error under the control of the processor 130 is described in more detail.

[0050] In another embodiment of the present disclosure, in response to inputting the first voice data and the second voice data, the second model 20 may obtain an estimated value of the difference between the first score corresponding to the first voice data and the second score corresponding to the second voice data. In addition, the second model 20 may output a weight based on the estimated value of the difference between the obtained first score and the second score. For example, the second model 20 may obtain the weight by applying a predefined mapping function to the estimated value of the difference between the first score and the second score.

[0051] The first score corresponding to the first voice data and the second score corresponding to the second voice data may include data to be output when the first voice data and the second voice data are respectively input to the third model 40. For example, the score corresponding to each voice data can be implemented in different ways according to the third model 40. For example, if the third model 40 includes an automatic speech recognition (ASR) model (e.g., including a processing circuit and / or an executable program element executed by the processing circuit), the score may be voice reliability or a word error rate (WER). For example, if the third model 40 is a wake-up model, the score may be the possibility of activating the dialogue system.

[0052] The second model 20 may be trained using the estimated value of the difference between the first score and the second score as learning data under the control of the processor 130. Figure 2B The process of training the second model 20 based on the estimated value of the difference between the first score and the second score under the control of the processor 130 is described in more detail.

[0053] For example, the input data determination model 30 may be a model that uses at least one of the first voice data, the second voice data, and the weight to determine (e.g., recognize) the input data. For example, the input data determination model 30 may linearly combine the first voice data and the second voice data based on the weight, and determine (e.g., recognize) the value of the linear combination as the input data. In another example, the input data determination model 30 may determine (e.g., recognize) one of the first voice data and the second voice data as the input data based on the weight.

[0054] In yet another example, if the weight is greater than the first threshold, the input data determination model 30 may determine (e.g., recognize) the second voice data as input data. If the weight is greater than the second threshold and less than the first threshold, the input data determination model 30 may determine (e.g., recognize) the value of a linear combination of the first voice data and the second voice data based on the weight as input data. If the weight is less than the second threshold, the input data determination model 30 may determine (e.g., recognize) the first voice data as input data. The first threshold and the second threshold may be values, for example, for determining which of the first voice data and the second voice data the data to be input into the third model 40 is similar to, and may be values, for example, predetermined by experiments, and may be, for example, changed by a user.

[0055] The third model 40 may be a model to which the input data determined (e.g., recognized) by the input data determination model 30 is input, for example. The third model 40 may be implemented as, for example, but not limited to, a wake-up model, an ASR model, and the like. However, this is merely an example, and the third model may be implemented differently depending on the type of electronic device 100, the type of input data, and user commands, and the like. For example, the wake-up model may be a model that wakes up or activates a dialogue system capable of performing natural language processing when the wake-up model determines (e.g., recognizes) that the user voice input via the microphone 110 includes a wake-up word. For example, an automatic speech recognition model may be a model that performs speech recognition on the user voice input via the microphone 110 and outputs text corresponding to the recognized voice.

[0056] The first model 10, the second model 20, the input data determination model 30, and the third model 40 may be stored in a non-volatile memory and may be loaded into a volatile memory under the control of the processor 130 when the dialogue system is activated. In another embodiment, when a user voice is input via the microphone 110, each model may be loaded into a volatile memory under the control of the processor 130. Loading may refer to, for example, the following operation: calling data stored in a non-volatile memory and storing it in a volatile memory so that the processor 130 can access the data. The case where the dialogue system is activated may include the case where the dialogue system is stored in a non-volatile memory and loaded into a volatile memory under the control of the processor 130.

[0057] Figure 1 The following example embodiment is shown, in which each model is loaded from a non-volatile memory to a volatile memory, and the volatile memory is included as a component of the processor 130. However, this is only an example, and the volatile memory may be implemented as a component separate from the processor 130. The volatile memory may include, for example, a memory that can retain stored information even if power is stopped. For example, the non-volatile memory 120 may include at least one of a flash memory, a programmable read-only memory (PROM), a magnetoresistive random access memory (MRAM), and a resistive RAM (RRAM). The volatile memory may include, for example, a memory that requires continuous power supply to retain stored information. For example, the volatile memory may include at least one of a dynamic random access memory (DRAM) and a static RAM (SRAM).

[0058] The processor 130 may include various processing circuits and is electrically connected to the memory 120, and may control general operations of the electronic device 100. For example, by executing at least one instruction stored in the memory 120, the processor 130 may obtain second voice data by inputting first voice data input via the microphone 110 to the first model 10 trained to enhance sound quality. For example, the processor 130 may obtain second voice data by applying an algorithm for enhancing sound quality to the first voice data via the first model 10.

[0059] In addition, the processor 130 may obtain the weight by inputting the first voice data and the second voice data into the pre-trained second model 20 .

[0060] In an embodiment of the present disclosure, the processor 130 may obtain an estimated value of the mean square error between the second speech data and the clean speech data corresponding to the first speech data through the second model 20. The estimated value of the mean square error between the clean speech data corresponding to the first speech data and the second speech data obtained by the second model 20 may be expressed as the following mathematical formula 1.

[0061]

Mathematical formula 1

[0062]

[0063] In mathematical formula 1, x represents clean speech data corresponding to the first speech data, hat{x} represents the second speech data, and v represents an estimated value of a mean square error. v, x, and hat{x} can be implemented in the form of a tensor of a multidimensional matrix, and mathematical formula 1 is a formula of the i-th component of the tensor.

[0064] The processor 130 may obtain the weight based on the first voice data, the second voice data, and the estimated value of the error. For example, the processor 130 may obtain the weight based on the first voice data, the second voice data, and the estimated value of the error through the second model 20. For example, the weight obtained by the processor 130 may be a weight value for minimizing and / or reducing the difference between the input data determined (e.g., identified) to be input to the third model 40 and the clean voice data corresponding to the first voice data, and the formula for obtaining the weight may be implemented as the following mathematical formula 2. The weight may be a value from 0 to 1.

[0065]

Mathematical formula 2

[0066]

[0067] In mathematical formula 2, w represents a weight value, y represents the first voice data, hat{x} represents the second voice data, and v represents an estimated error value. w, y, and hat{x} can be implemented in the form of a tensor, and mathematical formula 2 is a formula of the i-th component of the tensor. Through mathematical formula 3, the weight of mathematical formula 2 can be derived as a weight for minimizing and / or reducing the difference between the input data determined (e.g., identified) to be input to the third model and the clean voice data corresponding to the first voice data.

[0068]

Mathematical formula 3

[0069]

[0070] For example, referring to Mathematical Formula 3, ε represents the value of the mean square error between z, which is the input data determined (or, identified) to be input to the third model, and the clean speech data corresponding to the first speech data. The input data z can be expressed as shown in Mathematical Formula 8 to be described below. Figure 3 As shown, the input data z may be a value obtained by linearly combining the second voice data and the first voice data based on a weight. ε, z, x, y, and hat{x} may be implemented in the form of a tensor, and Mathematical Formula 3 is a formula of the i-th component of the tensor.

[0071] If an estimated value of the mean square error between the second speech data and the clean speech data corresponding to the first speech data is obtained by unbiased estimation, the value of ε can be expressed as the following mathematical formula 4.

[0072]

Mathematical formula 4

[0073]

[0074] Mathematical Formula 4 is a quadratic formula with respect to weights, and the weights for minimizing and / or reducing the value of ε can be expressed as Mathematical Formula 5.

[0075]

Mathematical formula 5

[0076]

[0077] If it is not the training process of the first model 10 and the second model 20, the clean voice data corresponding to the first voice data is not recognized, and therefore, the electronic device 100 may use the formula in the following mathematical formula 6 instead of the formula in mathematical formula 5. The following mathematical formula 6 may be expressed in the same manner as mathematical formula 2.

[0078]

Mathematical formula 6

[0079]

[0080] In order to derive Mathematical Formula 6, the following Mathematical Formula 7 is assumed.

[0081]

Mathematical formula 7

[0082]

[0083] In another embodiment of the present disclosure, the processor 130 may obtain an estimated value of the difference between the first score and the second score through the second model 20. Each of the first score and the second score is data obtained when each of the first voice data and the second voice data is input to the third model 40. The processor 130 may obtain a weight by applying a predefined mapping function to the estimated value of the difference between the first score and the second score.

[0084] For example, the third model 40 may be implemented as an ASR model and the score corresponding to the voice data may be, for example, a word error rate (WER). The processor 130 may obtain an estimated value of the difference between the first word error rate and the second word error rate by inputting the first voice data and the second voice data into the second model 20. The first word error rate and the second word error rate may be word error rates output when each of the first voice data and the second voice data is input into the ASR model. The processor 130 may obtain a weight via the second model 20 by applying a mapping function to an estimated value of the difference between the first word error rate and the second word error rate.

[0085] The processor 130 can determine (e.g., identify) the input data to be input to the third model 40 using the weights obtained by the input data determination model 30. In one embodiment, the processor 130 can linearly combine the first voice data and the second voice data based on the weights, and determine (e.g., identify) the linearly combined data as input data through the input data determination model 30. The value of the linear combination obtained by the processor 130 can be expressed as the following mathematical formula 8. In mathematical formula 8, Z represents the value of the linear combination, W represents the weight, y represents the first voice data, and hat{x} represents the second voice data. In other words, the processor 130 can use linear interpolation to obtain the optical input data to be input to the third model.

[0086]

Mathematical formula 8

[0087]

[0088] The electronic device 100 can determine (e.g., identify) which data of the input data to be input into the third model is more similar to the first voice data and the second voice data through Mathematical Formula 2 and Mathematical Formula 8. For example, when the estimated value (v) of the error increases, the weight value becomes close to 0, and the input data value may be data that is more similar to the first voice data. For example, if the reliability of the second voice data decreases due to the increase in the estimated error value, the electronic device 100 can determine (e.g., identify) data similar to the first voice data as input data. When the estimated error value decreases, the weight value becomes close to 1, and accordingly, the input data value may be data that is more similar to the second voice data. In other words, the reliability of the second voice data increases due to the decrease in the estimated error value, and accordingly, the electronic device 100 can determine (e.g., identify) data similar to the second voice data as input data.

[0089] In another example, the processor 130 may determine (e.g., recognize) one of the first voice data and the second voice data as input data based on the weight using the input data determination model 30. For example, if the weight is greater than a threshold, the processor 130 may determine (e.g., recognize) the second voice data as input data. In another example, if the weight is less than a threshold, the processor 130 may determine (e.g., recognize) the first voice data as input data.

[0090] In yet another example, if the weight is greater than the first threshold, the processor 130 may determine (e.g., recognize) the second voice data as input data, and if the weight is less than the second threshold, the processor 130 may determine (e.g., recognize) the first voice data as input data using the input data determination model 30. If the weight is greater than the second threshold and less than the first threshold, the processor 130 may linearly combine the first voice data and the second voice data based on the weight and determine (e.g., recognize) the value of the linear combination as input data.

[0091] The processor 130 may obtain a result that a specific function can be executed by inputting the determined (e.g., recognized) input data to the third model 40. For example, if the third model is a wake-up model that recognizes a trigger voice in the user's voice that can wake up the dialogue system and the first voice data is data corresponding to the trigger voice, the processor 130 may obtain a signal for waking up the dialogue system by inputting the determined (e.g., recognized) input data to the wake-up model. In another example, if the third model is an ASR model, the processor 130 may recognize the voice input to the user by inputting the determined (e.g., recognized) input data to the ASR model.

[0092] The processor 130 may use the learning data to train the second model 20. The learning data may be an estimate of the mean square error between the clean speech data corresponding to the first speech data and the second speech data. In another example, the learning data may be an estimate of the difference between the first score and the second score. Figure 2A and Figure 2B The process of the processor 130 training the second model 20 using the learning data is described in more detail.

[0093] In the present disclosure, the processor 130 may include one or more processors. The one or more processors 130 may be a general-purpose processor such as a central processing unit (CPU) or an application processor (AP), a graphics-specific processor such as a graphics processing unit (GPU) or a visual processing unit (VPU), or an artificial intelligence processor such as a neural processing unit (NPU), etc., but is not limited thereto.

[0094] The functions related to artificial intelligence according to the present disclosure may be operated by the memory 120 and the processor 130. One or more processors 130 may perform control to process input data according to predefined action rules or artificial intelligence models stored in the memory 120. The predefined action rules or artificial intelligence models are formed by training. Formation by training here may refer to, for example, forming an artificial intelligence model or predefined action rules with desired characteristics by applying a training algorithm to multiple learning data. Such training may be performed in a device displaying artificial intelligence according to the present disclosure, or by a separate server or system.

[0095] According to the functions related to artificial intelligence of the present disclosure, the processor and the memory can be operated. The processor may include one or more processors. One or more processors may be general-purpose processors such as CPUs, APs, or digital signal processors (DSPs), graphics-specific processors such as GPUs or VPUs, artificial intelligence processors such as NPUs, etc., but are not limited thereto. One or more processors may perform control to process input data according to predefined action rules or artificial intelligence models stored in the memory. In addition, if one or more processors are artificial intelligence-specific processors, the artificial intelligence-specific processors may be designed to have a hardware structure that specifically processes a specific artificial intelligence model.

[0096] Predefined action rules or artificial intelligence models can be formed by training. Formation by training here can refer to, for example, training a basic artificial intelligence model using multiple learning data through a training algorithm to form an artificial intelligence model or predefined action rules that are set to perform the desired features (or objects). Such training can be performed in a device that exhibits artificial intelligence according to the present disclosure, or by a separate server or system. Examples of learning algorithms include supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but are not limited to these examples.

[0097] The artificial intelligence model may include multiple neural network layers. The multiple neural network layers each have multiple weight values, and the neural network processing is performed by the processing result of the previous layer and the processing between the multiple weights. The multiple weights of the multiple neural network layers can be optimized and / or improved by the training results of the artificial intelligence model. For example, multiple weights can be updated to reduce or minimize and / or reduce the loss value or cost value obtained by the artificial intelligence model during the training process. The artificial neural network may include, for example, but not limited to, a convolutional neural network (CNN), a deep neural network (DNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q network, etc., but there is no limitation to these examples.

[0098] Figure 2A and Figure 2B is a block diagram illustrating an example process of training the second model 20 by the processor 130 of the electronic device 100 according to an embodiment of the present disclosure. Figure 2A and Figure 2B is a diagram showing a situation where each model and data are loaded into a volatile memory. Figure 2A and Figure 2B A case where the volatile memory is included as a component of the processor 130 is illustrated, but this is merely an example, and the volatile memory may be a component separate from the processor 130 .

[0099] In the embodiments of the present disclosure, Figure 2A As shown, the processor 130 can train the second model 20 based on an estimate of the mean square error between the clean voice data 50 corresponding to the first voice data and the second voice data. For example, the processor 130 can obtain the first voice data by inputting the clean voice data 50 into a simulation tool (e.g., including a processing circuit and / or an executable program element) 60. The clean voice data 50 can be predetermined data for training the second model 20, and can be voice data without any external influence (such as noise). For example, the simulation tool 60 can be a model that outputs the first voice data for training the second model 20 by applying an external influence such as noise to the clean voice data 50. Therefore, the first voice data output by the simulation tool 60 can be similar to the first voice data input via the microphone 110.

[0100] The processor 130 may obtain a value of a mean square error between the clean voice data corresponding to the first voice data and the second voice data through the second model 20. For example, the second model 20 may obtain a value of a mean square error using mathematical formula 1. The v in mathematical formula 1 may be a value of a mean square error between the clean voice data corresponding to the first voice data and the second voice data.

[0101] The processor 130 can obtain an estimated value of the mean square error between the clean speech data corresponding to the first speech data and the second speech data by inputting the first speech data and the second speech data into the second model 20. The processor 130 can obtain the difference between: the value of the mean square error obtained in advance; and the estimated value of the mean square error between the clean speech data corresponding to the first speech data and the second speech data. The processor 130 can train the second model 20 to output an optimal and / or improved estimate to minimize and / or reduce the difference between the value of the mean square error obtained and the estimated value of the mean square error between the clean speech data corresponding to the first speech data and the second speech data.

[0102] When the first voice data and the second voice data are input under the control of the processor 130, the pre-trained second model 20 can obtain an optimized and / or improved estimate of the mean square error between the clean voice data corresponding to the first voice data and the second voice data.

[0103] In another embodiment of the present disclosure, referring to Figure 2B , the processor 130 may train the second model 20 based on the difference between the first score and the second score. Figure 2A The process of the processor 130 obtaining the first voice data using the clean voice data through the simulation tool 60 is described, so repeated description will be omitted. For example, the processor 130 can obtain the first score and the second score corresponding to the first voice data and the second voice data, respectively, by inputting each of the first voice data and the second voice data separately into the third model 40. The processor 130 can obtain the difference between the first score and the second score.

[0104] The processor 130 may obtain an estimated value of the difference between the first score and the second score by inputting the first speech data and the second speech data into the second model 20. The processor 130 may train the second model 20 to output an optimal and / or improved estimated value to minimize and / or reduce the difference between: 1) the difference between the first score and the second score obtained in advance; and 2) the estimated value of the difference between the first score and the second score. Therefore, when the first speech data and the second speech data are input under the control of the processor 130, the trained second model 20 may obtain an optimized and / or improved estimated value of the difference between the first score and the second score.

[0105] For example, if the third model 40 is implemented as an ASR model, the processor 130 can obtain a voice reliability value of the first voice data and a second voice reliability value of the second voice data by inputting each of the first voice data and the second voice data into the ASR model. For example, the voice reliability value can be a value indicating the correct degree of recognition and conversion of the input voice data into text. The processor 130 can obtain the difference between the second voice reliability value and the first voice reliability value.

[0106] The processor 130 can obtain an estimated value of the difference between the first voice reliability value and the second voice reliability value by inputting the first voice data and the second voice data into the second model 20. The processor 130 can train the second model 20 to output an optimal and / or improved estimated value to minimize and / or reduce the difference between: 1) the difference between the first voice reliability value and the second voice reliability value obtained in advance; and 2) the estimated value of the difference between the first voice reliability value and the second voice reliability value. Therefore, when the processor 130 inputs the first voice data and the second voice data, the trained second model 20 can obtain an estimated value of the difference between the optimized and / or improved first voice reliability value and the second voice reliability value. The second model 20 can output a weight based on the estimated value of the difference between the optimal and / or improved first voice reliability value and the second voice reliability value. The weight can be a value obtained by applying a predefined mapping function to the estimated value.

[0107] The estimated value of the difference between the second voice reliability value and the first voice reliability value as a negative value may refer to, for example, a high first voice reliability value, and the estimated value of the difference between the second voice reliability value and the first voice reliability value as a positive value may refer to, for example, a high second voice reliability value. Therefore, when a negative estimated value is output and the second model 20 outputs a weight based on the output estimated value, the input data determination model 30 may determine (for example, recognize) the first voice data as data to be input to the third model 40. When a positive estimated value is output and the second model 20 outputs a weight based on the output estimated value, the input data determination model 30 may determine (for example, recognize) the second voice data as data to be input to the third model 40.

[0108] In another example, if the third model 40 is implemented as an ASR model, the second model 20 can be trained based on a first word error rate (e.g., WER) of the first voice data and a second word error rate of the second voice data under the control of the processor 130. For example, the word error rate can be a numerical value indicating the degree of incorrect recognition of the input voice by the ASR model. For example, the word error rate can be a numerical value indicating the degree to which word errors occur when the actual input voice is compared with the recognition result of the ASR model.

[0109] For example, the processor 130 may obtain a first word error rate of the first voice data and a second word error rate of the second voice data by inputting each of the first voice data and the second voice data into the ASR model. The processor 130 may obtain a difference between the first word error rate and the second word error rate. The processor 130 may obtain an estimated value of the difference between the first word error rate and the second word error rate by inputting the first voice data and the second voice data into the second model 20. The processor 130 may train the second model 20 to output an optimal and / or improved estimated value to minimize and / or reduce the difference between: 1) the difference between the first word error rate and the second word error rate obtained in advance; and 2) the estimated value of the difference between the first word error rate and the second word error rate. Accordingly, when the processor 130 inputs the first voice data and the second voice data, the trained second model 20 may obtain an estimated value of the difference between the first word error rate and the second word error rate that is optimized and / or improved. The processor 130 may output an optimal and / or improved weight based on the obtained estimated value of the optimized and / or improved difference.

[0110] An estimated value of the difference between the first word error rate and the second word error rate as a negative value may suggest a high second word error rate, while an estimated value of the difference between the first word error rate and the second word error rate as a positive value may suggest a high first word error rate. Therefore, when a negative estimated value is output and the second model 20 outputs a weight based on the estimated value, the input data determination model 30 can determine (e.g., recognize) the first speech data as data to be input to the third model 40. When a positive estimated value is output and the second model 20 outputs a weight based on the output estimated value, the input data determination model 30 can determine (e.g., recognize) the second speech data as data to be input to the third model 40.

[0111] In yet another example, if the third model 40 is implemented as a wake-up model, the processor 130 may obtain a first likelihood value of the wake-up model activating the dialogue system when the first voice data including the wake-up word is input to the wake-up model. The processor 130 may obtain a second likelihood value of the wake-up model activating the dialogue system when the second voice data including the wake-up word is input to the wake-up model. For example, the wake-up word may be a word capable of activating a dialogue system, which is an artificial intelligence model that provides an answer to the user's voice stored in the memory 120 and may be expressed as, for example, a trigger word. The first likelihood value and the second likelihood value may be numerical values ​​of the likelihood that the wake-up model recognizes the first voice data and the second voice data including the wake-up word and outputs a signal for activating the dialogue system.

[0112] The processor 130 may obtain the first likelihood value and the second likelihood value by inputting the first voice data and the second voice data, each including the wake-up word, into the wake-up model. The processor 130 may obtain the difference between the first likelihood value and the second likelihood value. The processor 130 may obtain an estimate of the difference between the first likelihood value and the second likelihood value by inputting the first voice data and the second voice data into the second model 20. The processor 130 may train the second model 20 to output an optimal and / or improved estimate to minimize and / or reduce the difference between: 1) the difference between the first likelihood value and the second likelihood value obtained in advance; and 2) the estimate of the difference between the first likelihood value and the second likelihood value. Therefore, the processor 130 may obtain an estimate of the difference between the best and / or improved first likelihood value and the second likelihood value by inputting the first voice data and the second voice data into the trained second model 20.

[0113] The estimated value of the difference between the first likelihood value and the second likelihood value as a negative value may suggest a high second likelihood value, while the estimated value of the difference between the first likelihood value and the second likelihood value as a positive value may suggest a high first likelihood value. Therefore, when a negative estimated value is output and the second model 20 outputs a weight based on the estimated value, the input data determination model 30 can determine (e.g., recognize) the first voice data as data to be input to the third model 40. When a positive estimated value is output and the second model 20 outputs a weight based on the estimated value, the input data determination model 30 can determine (e.g., recognize) the second voice data as data to be input to the third model 40.

[0114] In yet another example, if the third model 40 is implemented as a wake-up model, the processor 130 may obtain a third likelihood value for activating the dialogue system for the wake-up model when the first voice data not including the wake-up word is input to the wake-up model. When the second voice data not including the wake-up word is input to the wake-up model, the processor 130 may obtain a fourth likelihood value for activating the dialogue system for the wake-up model. A high third likelihood value or a high fourth likelihood value may imply that speech recognition is not correctly performed on the first voice data or the second voice data because the first voice data and the second voice data do not include the wake-up word.

[0115] The processor 130 may obtain the difference between the third likelihood value and the fourth likelihood value. The processor 130 may obtain an estimate of the difference between the third likelihood value and the fourth likelihood value by inputting the first voice data and the second voice data not including the wake-up word into the second model 20. The processor 130 may train the second model 20 to output an optimal and / or improved estimate to minimize and / or reduce the difference between: 1) the difference between the third likelihood value and the fourth likelihood value obtained in advance; and 2) the estimate of the difference between the third likelihood value and the fourth likelihood value. Therefore, when the processor 130 inputs the first voice data and the second voice data, the trained second model 20 may output an estimate of the difference between the optimal and / or improved third likelihood value and the fourth likelihood value.

[0116] The estimated value of the difference between the third likelihood value and the fourth likelihood value as a negative value may imply a high fourth likelihood value, and may imply that speech recognition has been correctly performed on the first speech data compared to the second speech data. The estimated value of the difference between the third likelihood value and the fourth likelihood value as a positive value may imply a high third likelihood value, and may imply that speech recognition has been correctly performed on the second speech data compared to the first speech data. Therefore, when a negative estimated value is output and the second model 20 outputs a weight based on the output estimated value, the input data determination model 30 can determine (e.g., recognize) the first speech data as data to be input to the third model 40. When a positive estimated value is output and the second model 20 outputs a weight based on the output estimated value, the input data determination model 30 can determine (e.g., recognize) the second speech data as data to be input to the third model 40.

[0117] For example, if the third model 40 is implemented as an ASR model or a wake-up model, the processor 130 may train the second model 20 based on a result obtained by inputting the first voice data and the second voice data to the third model 40 .

[0118] Figure 3 and Figure 4 is a flowchart illustrating an example process in which the electronic device 100 determines input data using the obtained weights according to an embodiment of the present disclosure.

[0119] Figure 3 1 is a flowchart showing an example process according to an embodiment of the present disclosure, in which the electronic device 100 determines (e.g., identifies) data to be input to the third model by comparing the weight with the threshold. The electronic device 100 may obtain the weight based on the first voice data, the second voice data, and the estimated value of the error (S310). Figure 1The process of obtaining weights for minimizing and / or reducing the mean square error between the clean speech data corresponding to the first speech data and the input data is described in detail, so the repeated description will not be repeated here.

[0120] The electronic device 100 may determine (eg, identify) whether the weight is greater than a threshold value (S320). The threshold value may be a predetermined value or a value obtained through experiments, but this is only an example, and the threshold value may be changed by a user command.

[0121] When it is determined (e.g., recognized) that the weight is greater than the threshold, the electronic device 100 can determine (e.g., recognize) the second voice data as input data (S320-Y). For example, a weight greater than the threshold means that the estimated value of the error is small, which may imply that the reliability value of the second voice data is large. Therefore, if the weight is greater than the threshold, the electronic device 100 can determine (e.g., recognize) the second voice data as input data to be input to the third model.

[0122] On the other hand, when it is determined (e.g., recognized) that the weight is less than the threshold, the electronic device 100 may determine (e.g., recognize) the first voice data as input data (S320-N). For example, a weight less than the threshold may imply that the estimated value of the error is large, which may imply that the reliability value of the second voice data is small. Therefore, if the weight is less than the threshold, the electronic device 100 may determine (e.g., recognize) the first voice data as input data to be input to the third model.

[0123] Figure 4 1 is a flowchart showing an example process according to an embodiment of the present disclosure, in which the electronic device 100 determines (e.g., identifies) data to be input to the third model by comparing the weights with the first threshold and the second threshold. The electronic device 100 can obtain a weight for minimizing and / or reducing the mean square error between the clean speech data corresponding to the first speech data and the input data (S410). Figure 1 An embodiment for obtaining weights has been described in detail, so the repeated description will not be repeated here.

[0124] The electronic device 100 may compare the weight value with a first threshold value and a second threshold value (S420). The first threshold value may be a predetermined value greater than the second threshold value. However, this is only an example, and the first threshold value and the second threshold value may be changed according to a user command.

[0125] If the weight value is greater than the first threshold value, the electronic device 100 may determine (e.g., recognize) the second voice data as input data to be input to the third model (S430-1). On the other hand, if the weight value is less than the second threshold value, the electronic device 100 may determine (e.g., recognize) the first voice data as input data to be input to the first model (S430-3).

[0126] If the weight is less than the first threshold and greater than the second threshold, the electronic device 100 can linearly combine the first voice data and the second voice data based on the weight, and determine (e.g., identify) the value of the linear combination as input data to be input to the third model (S430-2). In other words, except when the weight is greater than the first threshold or less than the second threshold, the electronic device 100 can linearly combine the first voice data and the second voice data and determine (e.g., identify) the value of the linear combination as input data.

[0127] Figure 5 is a block diagram showing an example configuration of an electronic device 100 according to an embodiment of the present disclosure. Figure 5 , the electronic device 100 may include a microphone 110, a memory 120, a processor (e.g., including a processing circuit) 130, a communicator (e.g., including a communication circuit) 140, a display 150, a camera 160, an input unit (e.g., including an input circuit) 170, and a speaker 180. Figure 1 The microphone 110 , the memory 120 , and the processor 130 are described, and thus a repeated description will not be repeated here.

[0128] The communicator 140 includes various communication circuits and may perform communication with a server (not shown) or an external device (not shown). For example, the processor 130 may send or receive various data or information to or from a server (not shown) or an external device (not shown) connected via the communicator 140. For example, the communicator 140 may receive learning data of various models stored in the memory 120 from an external server. Therefore, the processor 130 may train various models based on the learning data received via the communicator 140.

[0129] The communicator 140 may include various communication modules to perform communication with external devices. In an example, the communicator 140 may include wireless communication modules each including various communication circuits, and for example, but not limited to, include a cellular communication module that uses at least one of LTE, Advanced LTE (LTE-A), Code Division Multiple Access (CDMA), Wideband CDMA (WCDMA), Universal Mobile Telecommunications System (UMTS), Wireless Broadband (WiBro), 5th Generation (5G), Global System for Mobile Communications (GSM), etc. In another example, the wireless communication module may, for example, include at least one of Wireless Fidelity (Wi-Fi), Bluetooth, Bluetooth Low Energy (BLE), and Zigbee.

[0130] The display 150 may be implemented as, for example, a liquid crystal display panel (LCD), an organic light emitting diode (OLED), etc., and may also be implemented as a flexible display or a transparent display in some cases. The display 160 may be implemented as a touch screen having a touch panel. However, there is no limitation to the above embodiment, and the display 150 may be implemented differently according to the type of the electronic device 100.

[0131] The camera 160 may, for example, include a component for obtaining one or more pieces of image data of the surrounding environment of the electronic device 100. The processor 130 may determine (e.g., recognize) input data to be input to a specific model based on the one or more pieces of image data obtained by the camera 160. This embodiment will be described in more detail below. The camera may be implemented differently as an RGB camera, a 3D camera, or a depth camera, and may be located outside the electronic device 100 and electrically connected to the electronic device 100.

[0132] The input unit 170 may include various input circuits, and the processor 140 may receive a user command for controlling the operation of the electronic device 100 via the input unit 170. The input unit 170 may include various input circuits, such as, for example and without limitation, a touch sensor, a (digital) pen sensor, a pressure sensor, a key, a microphone, etc. The touch sensor may be, for example, at least one of an electrostatic type, a pressure-sensitive type, an infrared type, or an ultrasonic type.

[0133] The speaker 180 may be a component that outputs various alarms or voice messages in addition to various audio data subjected to various processing operations such as decoding, amplification, noise filtering, etc. by an audio processor (not shown). For example, the speaker 180 may output input data to be input to the third model under the control of the processor 130. In addition, the speaker 180 may output a result output by the third model (e.g., a response message corresponding to the first voice data) under the control of the processor 130.

[0134] Figure 6is a flowchart illustrating an example method for controlling the electronic device 100 according to an embodiment of the present disclosure.

[0135] The electronic device 100 can obtain the second voice data by inputting the first voice data input via the microphone 110 into the first model trained to enhance the sound quality (S610). The first model can be a model that executes an algorithm for filtering out noise, etc. of the first voice data and outputs the second voice data. In addition, the first model can be a neural network model trained based on the first voice data, the second voice data, and the clean voice data corresponding to the first voice data. For example, the first model can be a neural network model that is trained to output second voice data similar to the clean voice data corresponding to the first voice data when the first voice data is input.

[0136] The electronic device 100 may obtain a weight by inputting the first voice data and the second voice data into the second model (S620).

[0137] In an embodiment of the present disclosure, the electronic device 100 may obtain an estimated value of a mean square error between the second voice data and the clean voice data corresponding to the first voice data by inputting the first voice data and the second voice data into the second model. Figure 1 , Figure 2A and Figure 2B The formulas related thereto are described in detail, so the repeated descriptions are not repeated here. In addition, the electronic device 100 can obtain weights based on the first voice data, the second voice data, and the estimated value of the mean square error obtained by the second model. The weight can be a value for minimizing and / or reducing the mean square error between the clean voice data corresponding to the first voice data and the input data to be input to the third model.

[0138] In another embodiment of the present disclosure, the electronic device 100 may obtain an estimated value of the difference between the first score and the second score by inputting the first voice data and the second voice data into the second model. The first score and the second score may be data obtained when each of the first voice data and the second voice data is input into the third model. For example, if the third model is an ASR model, the score may be a voice reliability value or WER, and if the third model is a wake-up model, the score may be a likelihood value for activating the dialogue system. The electronic device 100 may obtain a weight by applying a predefined mapping function to the estimated value of the difference between the first score and the second score by the second model.

[0139] The electronic device 100 may use the obtained weight to determine (e.g., identify) input data to be input into the third model (S630). In an embodiment, the electronic device 100 may linearly combine the first voice data and the second voice data based on the weight and determine (e.g., identify) the value of the linear combination as input data. In another example, if the weight is less than a threshold value, the electronic device 100 may determine (e.g., identify) the first voice data as input data, and if the weight is greater than a threshold value, the electronic device 100 may determine (e.g., identify) the second voice data as input data.

[0140] In yet another example, if the weight is greater than the second threshold and less than the first threshold, the electronic device 100 may linearly combine the first voice data and the second voice data based on the weight and determine (e.g., recognize) the value of the linear combination as input data. If the weight is less than the second threshold, the electronic device 100 may determine (e.g., recognize) the first voice data as input data, and if the weight is greater than the first threshold, the electronic device 100 may determine (e.g., recognize) the second voice data as input data. The first threshold may be a value greater than the second threshold.

[0141] In the above, an embodiment has been described in which the electronic device 100 determines (e.g., recognizes) voice data to be input to a third model based on the first voice data and the second voice data, but this is only an example. For example, in another embodiment of the present disclosure, the electronic device 100 can determine (e.g., recognize) input data to be input to a specific model or image data to be displayed on a display based on first image data obtained via a camera and second image data obtained by applying an algorithm for enhancing image quality to the first image data.

[0142] For example, the electronic device 100 may obtain the second image data by inputting the first image data input via the camera into the first model trained to enhance the image quality. In addition, the electronic device 100 may obtain an estimated value of the mean square error between the second image data and the clean image data corresponding to the first image data by inputting the first image data and the second image data into the second model. The clean image data may include image data captured in a clean environment without receiving influence from the surrounding environment.

[0143] The electronic device 100 may obtain a weight based on the first image data, the second image data, and an estimated value of the error. The electronic device 100 may determine (e.g., identify) input data to be input to the third model using at least one of the first image data, the second image data, and the weight. The third model may be implemented as an image classifier model, etc., but this is only an example, and the third model may be implemented differently as a model capable of editing an image, etc.

[0144] The electronic device 100 may train a third model based on the clean image data corresponding to the first image data and the second image data. For example, the electronic device 100 may obtain a mean square error between the clean image data corresponding to the first image data and the second image data. In addition, the electronic device 100 may obtain an estimated value of the mean square error between the second image data and the clean image data corresponding to the first image data by inputting the first image data and the second image data into the second model. The electronic device 100 may train the second model to minimize and / or reduce the difference between: 1) the mean square error between the clean image data corresponding to the first image data and the second image data; and 2) the estimated value of the mean square error.

[0145] The drawings of the present disclosure are not intended to limit the technology disclosed in the present disclosure to specific embodiments, but should be interpreted as including all modifications, equivalents and / or substitutions of the embodiments of the present disclosure. Regarding the interpretation of the drawings, similar reference numerals may be used for similar elements.

[0146] In the present disclosure, terms such as "including", "may include", "consist of..." or "may consist of..." are used herein to specify the existence of corresponding features (e.g., constituent elements such as numbers, functions, operations, or parts), and do not exclude the existence of additional features.

[0147] In the present disclosure, expressions such as "A or B", "at least one of A [and / or] B", or "one or more of A [and / or] B" include all possible combinations of the listed items. For example, "A or B", "at least one of A and B", or "at least one of A or B" includes any of the following: (1) at least one A, (2) at least one B, or (3) at least one A and at least one B.

[0148] The expressions “first,” “second,” and the like used in the present disclosure may denote various elements regardless of order and / or importance and may be used to distinguish one element from another and not to limit the elements.

[0149] If a specific element (e.g., a first element) is described as being “operably or communicatively coupled to / operably or communicatively coupled to” another element (e.g., a second element) or “connected to” another element (e.g., a second element), it should be understood that the specific element may be connected to the other element directly or through another element (e.g., a third element). On the other hand, if a specific element (e.g., a first element) is described as being “directly coupled to” or “directly connected to” another element (e.g., a second element), it should be understood that there is no element (e.g., a third element) between the specific element and the other element.

[0150] In addition, as appropriate, the expression "configured to" used in the present disclosure may be used interchangeably with other expressions, such as "suitable for", "capable of", "designed to", "adapted to", "manufactured to", and "capable of". At the same time, the expression "configured to" does not necessarily refer to a device that is "specially designed to..." in terms of hardware. Rather, in some cases, the expression "device configured to..." may mean that the device is "capable of" performing operations together with another device or component. For example, the phrase "a unit or processor configured (or set) to perform A, B, and C" may refer to, for example, but not limited to, a dedicated processor (e.g., an embedded processor), a general-purpose processor (e.g., a central processing unit (CPU) or an application processor), etc., which is used to perform the corresponding operations by executing one or more software programs stored in a memory device.

[0151] The electronic device according to the embodiments of the present disclosure may include, for example, but not limited to, at least one of a smart phone, a tablet PC, a desktop PC, a laptop PC, a netbook computer, a server, a PDA, a medical device, a wearable device, etc. In some embodiments, the electronic device may include, for example, but not limited to, a television, a refrigerator, an air conditioner, an air purifier, a set-top box, a media box (e.g., Samsung HOMESYNCTM, APPLE TVTM, or GOOGLE TVTM), etc.

[0152] Various embodiments of the present disclosure may be implemented as software including instructions stored in a machine (e.g., computer) readable storage medium. A machine is a device that calls instructions stored in a storage medium and operates according to the called instructions, and may include an electronic device (e.g., electronic device 100) according to the disclosed embodiments. In the case where the instructions are executed by a processor, the processor may use other elements to perform functions corresponding to the instructions directly or under the control of the processor. The instructions may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, "non-transitory storage medium" is tangible and may not include signals, and does not distinguish whether the data is semi-permanent or temporarily stored in a storage medium. For example, a "non-transitory storage medium" may include a buffer for temporarily storing data.

[0153] According to an embodiment, the method according to various embodiments disclosed in the present disclosure may be provided to be included in a computer program product. The computer program product may be exchanged between a seller and a buyer as a commercially available product. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., a compact disc read-only memory (CD-ROM)) or distributed online through an application store (e.g., PlayStoreTM). In the case of online distribution, at least a portion of the computer program product (e.g., a downloadable application) may be at least temporarily stored or temporarily generated in a storage medium (e.g., a memory of a manufacturer's server, a server of an application store, or a relay server).

[0154] Each element (e.g., module or program) according to the various embodiments described above may include a single entity or multiple entities, and in various embodiments, some of the sub-elements described above may be omitted or other sub-elements may be further included. Alternatively or additionally, some elements (e.g., modules or programs) may be integrated into one entity to perform the same or similar functions performed by each corresponding element before integration. According to various embodiments, the operations performed by modules, programs or other elements may be performed sequentially, in parallel, repeatedly or heuristically, or at least some operations may be performed in different orders, omitted or different operations may be added.

[0155] Although the present disclosure has been illustrated and described with reference to various exemplary embodiments, it will be understood that the various exemplary embodiments are intended to be illustrative rather than restrictive. It will be understood by those skilled in the art that various changes may be made in form and details without departing from the true spirit and full scope of the present disclosure, including the appended claims and their equivalents.

Claims

1. An electronic device, comprising: microphone; a memory storing at least one instruction; as well as a processor configured to execute the at least one instruction, The processor is configured to: obtaining second voice data by inputting first voice data input via a microphone into a first model trained to enhance sound quality, The weights are obtained by inputting the first speech data and the second speech data into the second model, using the first speech data, the second speech data, and an estimated value of an error between the second speech data and clean speech data corresponding to the first speech data, and Using the weights, identify the input data to be fed into the third model, Wherein, the processor is further configured to: Based on the weight being greater than the first threshold, identifying the second speech data as input data, Based on the weight being greater than the second threshold and less than the first threshold, linearly combining the first voice data and the second voice data based on the weight, and identifying a value of the linear combination as input data, and Based on the weight being less than a second threshold, identifying the first speech data as input data, The first threshold is a value greater than the second threshold.

2. The device according to claim 1, wherein: The processor is further configured to: linearly combine the first voice data and the second voice data based on the weight; and identify a value of the linear combination as input data.

3. The device according to claim 1, wherein: The processor is further configured to identify one of the first voice data and the second voice data as input data based on the weight.

4. The device according to claim 3, wherein: The processor is further configured to: Based on the weight being greater than the threshold, recognizing the second voice data as input data, and Based on the weight being less than the threshold, the first voice data is recognized as input data.

5. The device according to claim 1, wherein: The third model is at least one of a wake-up model or an automatic speech recognition model.

6. The device according to claim 1, wherein: The processor is also configured to train the first model and the second model based on the first voice data, clean voice data corresponding to the first voice data, and the second voice data.

7. The device according to claim 1, wherein: The processor is further configured to: obtain a first score corresponding to the first voice data and obtain a second score corresponding to the second voice data by inputting each of the first voice data and the second voice data into the third model, and The second model is trained based on the first speech data, the second speech data, the first score, and the second score.

8. The device according to claim 7, wherein: The processor is further configured to obtain the weight by: obtaining an estimated value of a difference between the first score and the second score by inputting the first speech data and the second speech data into the second model, and The weight is obtained based on an estimate of the difference between the first score and the second score.

9. A method for controlling an electronic device including a microphone, the method comprising: obtaining second voice data by inputting first voice data input via a microphone into a first model trained to enhance sound quality; Obtaining a weight by inputting the first speech data and the second speech data into a second model, using the first speech data, the second speech data, and an estimated value of an error between the second speech data and clean speech data corresponding to the first speech data; as well as Using the weights, identify the input data to be fed into the third model, Wherein, the method further comprises: Based on the weight being greater than the first threshold, identifying the second speech data as input data, Based on the weight being greater than the second threshold and less than the first threshold, linearly combining the first voice data and the second voice data based on the weight, and identifying a value of the linear combination as input data, and Based on the weight being less than a second threshold, identifying the first speech data as input data, The first threshold is a value greater than the second threshold.

10. The method according to claim 9, wherein: The obtaining includes obtaining a weight based on the first speech data, the second speech data, and an estimated value of an error between the second speech data and clean speech data corresponding to the first speech data.

11. The method according to claim 9, wherein: The recognition includes linearly combining the first voice data and the second voice data based on a weight, and identifying a value of the linear combination as input data.

12. The method according to claim 9, wherein: The recognizing includes recognizing one of first voice data and second voice data as input data based on a weight.

13. The method according to claim 12, wherein: The identification includes: Based on the weight being greater than the threshold, recognizing the second voice data as input data, and Based on the weight being less than the threshold, the first voice data is recognized as input data.

Images