Electronic device and control method thereof

Abstract

An electronic device and a control method thereof are provided. The electronic device includes an input device and a processor, wherein the processor is configured to: based on receiving an audio signal through the input device, obtain speech clarity of the audio signal, and modify the audio signal so that the speech clarity becomes a target clarity, wherein the target clarity is set based on scene information related to an audio type included in the audio signal, wherein the audio type includes at least one of sound effects, shouting, music, and speech.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application is based on and claims the benefit of priority from Korean Patent Application No. 10-2019-0162644 filed in the Korean Intellectual Property Office on December 9, 2019, the disclosure of which is incorporated herein by reference in its entirety. Technical Field

[0003] The present invention relates to an electronic device and a control method thereof. More particularly, the present invention relates to an electronic device that performs an operation corresponding to a user's speech and a control method of the electronic device. Background Art

[0004] With the development of the Internet and streaming services, many contents are being consumed in multimedia devices. However, due to various factors, including environmental factors such as the content reproduction environment (ambient noise, etc.) or limitations of device speakers, content production factors such as outdoor recording or lack of editing time, it may happen that the speech included in the reproduced content cannot be heard.

[0005] Specifically, as a result of 24-hour analysis of audio signals provided by five broadcasters, it has been found that the ratio of speech content is about 91.8%, and the power of speech and the power of other signals other than speech are the same. Therefore, this analysis shows that the part where speech transmission is not good is 30% or more. As a result, more attention is paid to the speech transmission ability of the content.

[0006] In the related art, there is a technology that controls the power of each frequency band based on the non-speech volume and speech probability of an input audio signal, thereby improving the intelligibility of speech.

[0007] However, in the prior art, the non-speech volume is measured by tracking the minimum value of the power of each frequency band, and in addition to maintaining constant non-speech, it is impossible to properly measure a sudden increase in non-speech, and non-speech cannot be accurately measured due to sensitivity problems of recording microphones, post-correction, etc. In addition, since parameters related to the final output are adjusted by measuring the probability of speech by frequency band, speech and non-speech of the same frequency band are increased together and used for output.

[0008] Furthermore, in the related art, since only the operation of enhancing speech intelligibility is performed on all types of input content, there is a problem that the production intention of the producer according to the type of audio content is not reflected in the control of speech intelligibility. Summary of the invention

[0009] An electronic device that performs an operation corresponding to a user's speech and a control method thereof are provided.

[0010] Additional aspects will be set forth in part in the description which follows and, in part, will be obvious from the description, or may be learned by practice of the presented embodiments.

[0011] According to one aspect of the present disclosure, an electronic device includes an inputter and a processor, the processor being configured to: based on receiving an audio signal through the inputter, obtain speech clarity included in the audio signal, and modify the audio signal so that the speech clarity becomes a target clarity, wherein the target clarity is set based on scene information related to an audio type included in the audio signal, the audio type including at least one of sound effects, shouting, music, and speech.

[0012] The processor may be further configured to calculate speech intelligibility based on a speech signal included in the audio signal and a non-speech signal other than the speech signal.

[0013] The processor may also be configured to extract a speech signal included in the audio signal using an artificial intelligence model trained to extract a speech signal included in the audio signal, and extract one or more remaining signals other than the extracted speech signal from the audio signal as non-speech signals.

[0014] The speech intelligibility may be one of a signal-to-noise ratio (SNR) of a speech signal and a non-speech signal included in the audio signal and a speech intelligibility index (SII) based on the speech signal and the non-speech signal.

[0015] The speech intelligibility may be an SNR, and the processor may be further configured to adjust the gain of the speech signal by as much as the difference between the target intelligibility and the achieved speech intelligibility to modify the audio signal.

[0016] The speech intelligibility may be SII, and the processor may be further configured to: calculate a gain adjustment value, and adjust the gain of the speech signal by an amount equal to the calculated gain adjustment value to modify the audio signal, wherein the gain adjustment value may be calculated according to the following formula: Gain adjustment value=α*(SII 目标 -SII 测量 )+β, and SII 目标 Can indicate target clarity, SII 测量 may represent the speech intelligibility achieved, and α and β may represent constant values ​​experimentally calculated by varying the amount of SII according to the gain variation of the speech signal.

[0017] The processor may also be configured to obtain at least one audio feature about the audio signal, and obtain scene information based on the obtained at least one audio feature.

[0018] The processor may be further configured to obtain scene information using an artificial intelligence model trained to distinguish between audio types contained in the audio signal.

[0019] The target clarity can be set differently for different audio types.

[0020] Based on the audio type being a sound effect, the target clarity may be set higher than when the audio type is a shout.

[0021] According to another aspect of the present disclosure, a method for controlling an electronic device includes: obtaining an audio signal; obtaining speech clarity of the audio signal based on the audio signal; and modifying the audio signal so that the speech clarity becomes a target clarity, wherein the target clarity is set based on scene information related to an audio type included in the audio signal, and the audio type includes at least one of sound effects, shouting, music, or speech.

[0022] Obtaining the speech intelligibility may include calculating the speech intelligibility based on a speech signal included in the audio signal and a non-speech signal other than the speech signal.

[0023] Obtaining speech intelligibility may include extracting a speech signal included in the audio signal using an artificial intelligence model trained to extract a speech signal included in the audio signal, and extracting one or more remaining signals other than the extracted speech signal from the audio signal as non-speech signals.

[0024] The speech intelligibility may be one of a signal-to-noise ratio (SNR) of a speech signal and a non-speech signal included in the audio signal and a speech intelligibility index (SII) based on the speech signal and the non-speech signal.

[0025] The speech intelligibility may be an SNR, and modifying the audio signal may include adjusting a gain of the speech signal by as much as a difference between the target intelligibility and the achieved speech intelligibility to modify the audio signal.

[0026] The speech intelligibility may be SII, and modifying the audio signal may include: calculating a gain adjustment value, and adjusting the gain of the speech signal by an amount equal to the calculated gain adjustment value to modify the audio signal, wherein the gain adjustment value may be calculated according to the following formula: Gain adjustment value=α*(SII 目标 -SII 测量 )+β, and SII 目标 Can indicate target clarity. SII 测量 may represent the speech intelligibility achieved, and α and β may represent constant values ​​experimentally calculated by varying the amount of SII according to the gain variation of the speech signal.

[0027] The method of controlling an electronic device may further include obtaining at least one audio feature regarding an audio signal, and obtaining scene information based on the obtained at least one audio feature.

[0028] The method of controlling an electronic device may further include obtaining scene information using an artificial intelligence model trained to distinguish audio types included in the audio signal.

[0029] The target clarity can be set differently for different audio types.

[0030] According to another aspect of the present disclosure, an electronic device includes a memory and a processor, the memory storing instructions, and the processor being configured to execute the instructions to: obtain speech intelligibility of an audio signal, and modify the audio signal so that the speech intelligibility becomes a target intelligibility, wherein the target intelligibility is set based on a determined audio type included in the audio signal.

[0031] According to another aspect of the present invention, a non-transitory computer-readable recording medium has recorded thereon instructions executable by at least one processor to perform a method of controlling an electronic device. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The above and other aspects, features and advantages of certain embodiments of the present disclosure will become more apparent from the following description taken in conjunction with the accompanying drawings, in which:

[0033] Figure 1 is a diagram illustrating an environment in which audio content including an audio signal is provided to an electronic device through a network according to an embodiment;

[0034] Figure 2 is a block diagram of an electronic device according to an embodiment;

[0035] Figure 3 is a functional block diagram of a processor according to an embodiment;

[0036] Figure 4 is a graph showing speech recognition accuracy according to a speech intelligibility index;

[0037] Figure 5 is a detailed block diagram of an electronic device according to an embodiment; and

[0038] Figure 6 is a flowchart illustrating a method for controlling an electronic device according to an embodiment. DETAILED DESCRIPTION

[0039] An electronic device capable of more accurately controlling speech intelligibility and a control method thereof are provided. According to one or more embodiments, an electronic device capable of optimally controlling speech intelligibility based on, reflecting or considering the production intention of audio content and a control method thereof are provided.

[0040] It should be understood that the suffix “part” of components used in the following description is given or used in consideration of ease of writing of the specification, and has no obvious meaning or role.

[0041] The terms used in the specification are used to describe the embodiments, but may not be intended to limit the scope of other embodiments. Unless specifically defined otherwise, a singular expression may include a plural expression.

[0042] It is to be understood that terms such as “comprising” or “consisting of…” used herein indicate the presence of features, numbers, operations, elements, components or combinations thereof, but do not exclude the presence or possibility of adding one or more other features, numbers, operations, elements, components or combinations thereof.

[0043] As used herein, the terms "first," "second," and the like may identify corresponding components regardless of order and / or importance, and are used to distinguish a component from another component without otherwise limiting the component.

[0044] If it is described that an element (e.g., a first element) is operably or communicatively coupled or connected to another element (e.g., a second element), it is understood that the element may be directly connected to the other element or connected to the other element through another element (e.g., a third element). When it is mentioned that an element (e.g., a first element) is directly coupled or directly connected to another element (e.g., a second element), it is understood that there is no element (e.g., a third element) between the element and the other element.

[0045] Unless otherwise defined, the terms used in the examples may be interpreted according to meanings well known to those skilled in the art.

[0046] Various embodiments will be described in more detail with reference to the accompanying drawings.

[0047] Figure 1 is a diagram illustrating an environment 10 in which audio contents including audio signals are provided to electronic devices 100 - 1 to 100 - 4 through a network according to an embodiment.

[0048] like Figure 1 As shown, audio content (or sound source) may be provided to the electronic devices 100-1 to 100-4 from a broadcast transmission station 1, a satellite 2, a content providing server 3, etc. via a communication medium 5 (eg, a network or the Internet).

[0049] The audio content may be composed of a multi-channel audio signal such as a stereo channel audio signal or a 5.1 channel audio signal, but is not limited thereto, and may be composed of a mono channel audio signal, a 7.1 channel audio signal, a 5.2 channel audio signal, etc. Depending on the type of content and / or the type of electronic device, the audio content may be provided to the electronic devices 100-1 to 100-4 alone, and the audio content may be provided to the electronic devices 100-1 to 100-4 together with the video content.

[0050] The broadcast transmission station 1 may include a transmitter or a repeater for transmitting terrestrial broadcast content. The satellite 2 may include a satellite for transmitting data or satellite broadcast content. The content providing server 3 may be a server on a communication network, which provides broadcast content for Internet Protocol Television (IPTV), broadcast content for Cable Television (TV), various audio source content, video on demand (VOD) content, etc.

[0051] The communication medium 5 may include an air medium or a constructed communication network. The communication network may include a wireless cellular network, the Internet, a wide area network (WAN), a local area network (LAN), a wired telephone network, a cable network, and the like.

[0052] The electronic apparatuses 100 - 1 to 100 - 4 may include not only the audio device 100 - 3 capable of reproducing only audio content but also the display apparatuses 100 - 1 , 100 - 2 , and 100 - 4 capable of reproducing video and audio together.

[0053] The display devices 100 - 1 , 100 - 2 , and 100 - 4 are devices including a display for reproducing a video and outputting an audio through a speaker, such as a smart TV, a monitor, a smart phone, a desktop computer, a laptop computer, a tablet computer, a navigation device, a digital signage, and the like.

[0054] The audio device 100-3 is an electronic device configured to reproduce and output only audio, and for example, the audio device 100-3 may include a radio device, an audio device, a recorder, a speech recognition speaker device, a CD player equipped with a speaker, a digital audio player (DAP), an audio device for a vehicle, a home appliance equipped with a speaker, a sound bar, various devices capable of performing sound output operations, and the like.

[0055] When (or based on) receiving an audio signal constituting audio content through the communication medium 5, the electronic devices 100-1 to 100-4 may process the received audio signal to generate an output signal, and may output the generated output signal through at least one speaker. According to an embodiment, at least one speaker may be provided in the electronic devices 100-1 to 100-4, and / or may be separately provided outside the electronic devices 100-1 to 100-4.

[0056] According to an embodiment, the electronic devices 100-1 to 100-4 may recognize (or obtain, determine, calculate, etc.) the clarity of speech (e.g., speech clarity value) included in the received audio signal, and correct or modify the audio signal so that the recognized speech clarity becomes the target clarity (e.g., target clarity value), and generate an output signal. The target clarity may be set based on scene information about the audio type included in the received audio signal (e.g., sound effect, sound, music, speech, etc.).

[0057] Specifically, unlike the prior art that tracks the minimum value of the power of each frequency band to measure the non-speech volume, the electronic devices 100-1 to 100-4 according to the embodiments can separate speech signals and non-speech signals from the received audio signals, and identify the clarity of speech based on the separated speech signals and non-speech signals.

[0058] Electronic devices 100-1 to 100-4 may be different from the prior art, which measures speech probability through frequency bands to adjust parameters related to the final output. Electronic devices 100-1 to 100-4 may adjust speech clarity by performing gain adjustment on at least one of the separated speech signals and non-speech signals or performing various processing on the separated speech signals and non-speech signals.

[0059] The electronic devices 100-1 to 100-4 can set the target clarity based on the scene information related to the audio type included in the audio signal, which is different from the prior art that only increases the speech clarity for all types of content input. Therefore, the electronic devices 100-1 to 100-4 can correct the audio signal so that the clarity of the speech of the received audio signal is the set target clarity.

[0060] As described above, according to the embodiment, a speech signal and a non-speech signal may be separated from an audio signal to identify speech intelligibility, and at least one of the separated speech signal and non-speech signal is processed to adjust speech intelligibility, so that the speech intelligibility may be adjusted more accurately.

[0061] Furthermore, when the target intelligibility is set based on the scene information, adjustment of the speech intelligibility can be performed for each audio type to be different, and the production intention of the audio content producer can be reflected therethrough.

[0062] Although it has been described that audio content is provided from outside the electronic devices 100 - 1 to 100 - 4 through the communication medium 5 , it should be understood that one or more other embodiments are not limited thereto.

[0063] For example, the audio content may be provided to the electronic devices 100 - 1 to 100 - 4 through a portable storage medium such as a Universal Serial Bus (USB), a Secure Digital (SD) memory card, various optical storage media, or the like.

[0064] The audio content may be stored in a memory of electronic devices 100-1 to 100-4 (e.g., a hard disk drive (HDD), a solid state drive (SSD), a system memory (ROM, BIOS, etc.), etc.) and output by electronic devices 100-1 to 100-4 (e.g., in response to or based on a user's request).

[0065] Figure 2 is a block diagram of an electronic device 100 according to an embodiment. Figure 2 , the electronic device 100 includes an input device 110 and a processor 120 .

[0066] The inputter 110 may receive an audio signal and provide the received audio signal to the processor 120. As described above, the audio signal may be provided to the electronic device 100 through the communication medium 5 or through an external portable storage medium. Therefore, various wired and wireless communication interfaces for receiving audio signals may perform the function of the inputter 110.

[0067] The audio signal may be provided from a memory included in the electronic device 100 to the processor 120 , and in this case, the memory included in the electronic device 100 may perform the function of the inputter 110 .

[0068] The processor 120 controls the overall operation of the audio output device (ie, electronic apparatus) 100. Based on receiving the audio signal through the inputter 110, the processor 120 may recognize the articulation of speech included in the audio signal based on the received audio signal.

[0069] The processor 120 may recognize speech intelligibility based on a speech signal included in the audio signal and a non-speech signal from which the speech signal is excluded from the audio signal.

[0070] To this end, the processor 120 may extract a speech signal from the audio signal, and extract the remaining signal except the extracted speech signal as a non-speech signal. According to an embodiment, the processor 120 may extract a speech signal from the audio signal received through the inputter 110 by using an artificial intelligence model trained to extract a speech signal from the audio signal. However, it should be understood that one or more other embodiments are not limited thereto.

[0071] When extracting a speech signal and a non-speech signal from the audio signal, the processor 120 may identify speech intelligibility included in the audio signal based on the extracted speech signal and non-speech signal.

[0072] Specifically, the processor 120 may calculate a signal-to-noise ratio (SNR) of the extracted speech signal and non-speech signal, and may identify the calculated SNR as speech intelligibility. The processor 120 may calculate a speech intelligibility index (SII) based on the extracted speech signal and non-speech signal, and may identify the calculated speech intelligibility index as speech intelligibility.

[0073] Therefore, the processor 120 can correct the audio signal so that the identified speech intelligibility becomes the target intelligibility. In an embodiment where the speech intelligibility is identified using SNR, the target intelligibility also has an SNR value, and in an embodiment where the speech intelligibility is identified using a speech intelligibility index, the target intelligibility can also have a speech intelligibility index value.

[0074] Specifically, when the SNR is used to identify the speech intelligibility (or based thereon), the processor 120 may adjust the gain of the speech signal by as much as the difference between the target intelligibility and the identified speech intelligibility to correct the audio signal.

[0075] When speech intelligibility is identified by using the speech intelligibility index (or based thereon), the processor 120 may calculate a gain adjustment value based on the following Formula 1, and may adjust the gain of the speech signal by the calculated gain adjustment value to correct the audio signal.

[0076] [Formula 1] Gain adjustment value = α*(SII 目标 -SII 测量 )+β

[0077] Here, SII 目标 is the target intelligibility in the Speech Intelligibility Index format, SII 测量 is the recognized speech intelligibility in the speech intelligibility index format, and α and β are constant values ​​experimentally calculated by the numerical change of the speech intelligibility index according to the gain change of the speech signal.

[0078] It should be understood that the method for calculating the gain adjustment value is not limited to the above formula 1. For example, when the above formula 1 uses a first-order linear regression, the processor 120 can calculate the gain adjustment value by using a linear regression function such as α 1 *(SII 目标 -SII 测量 )2+α 2 *(SII 目标 -SII 测量 )+β or higher order linear regression to obtain more complex gain adjustment values.

[0079] Processor 120 may obtain the gain adjustment value from a specific index value (eg, SII, Speech Transmission Index (STI), etc., described below).

[0080] The audio signal whose speech intelligibility has been adjusted may be output through at least one speaker provided inside or outside the electronic device 100 .

[0081] In various embodiments, the above-mentioned target intelligibility may have a specific value for each audio type as a value set based on scene information about the audio type included in the audio signal. For example, in the case where speech intelligibility is identified as SII, a specific value may be directly set as the target intelligibility value, such as 0.6 for sound effects, 0.5 for shouts, and 0.4 for music.

[0082] However, it should be understood that one or more other embodiments are not limited thereto. For example, the target intelligibility may be set to a percentage value of the intelligibility to be adjusted for each audio type. In this case, the percentage value of the intelligibility to be adjusted may be set to the target intelligibility, such as +10% for sound effects, -10% for shouting, and 0% for music. The processor 120 may calculate the actual target intelligibility value by calculating the percentage value of the intelligibility to be adjusted in addition to the currently measured speech intelligibility.

[0083] The target definition may be stored as a mapping table in a memory preset by the audio type, and the processor 120 may refer to the mapping table to check a target definition value corresponding to the scene information.

[0084] Scene information is a sub-concept of genre information, and may include information about whether the audio type included in the audio signal corresponds to any of sound effects, shouts, music, and speech. For example, audio content in the style of "movie" may include various audios, such as sound effects, shouts, and music, and at this time, each audio type such as speech, sound effects, shouts, and music may be a scene included in the audio signal.

[0085] According to an embodiment, the processor 120 may obtain at least one audio feature of the audio signal and obtain scene information based on the at least one audio feature obtained. In addition, according to another embodiment, the processor 120 may obtain scene information using an artificial intelligence model trained to recognize the audio type included in the audio signal.

[0086] The target clarity may be set according to the scene information of the obtained audio signal. Specifically, the target clarity may be set differently for each audio type included in the scene information. For example, when the audio type is a sound effect, the target clarity may be set higher than when the audio type is a shout, but it is understood that one or more other embodiments are not limited thereto.

[0087] According to various embodiments, by setting a target clarity value based on scene information and adjusting an audio signal based thereon, the production intention of a content producer may be reflected or considered in clarity adjustment.

[0088] In the following, reference will be made to Figure 3 and Figure 4 Various embodiments are further described.

[0089] Figure 3 1 is a functional block diagram of the processor 120 according to an embodiment. Figure 3 , the processor 120 may include a speech / non-speech separator 121 , a speech clarity analyzer 122 , a scene analyzer 123 , and a speech clarity renderer 124 .

[0090] The speech / non-speech separator 121 may separate and / or extract a speech signal and a non-speech signal from the audio signal received through the input 110 .

[0091] The speech / non-speech separator 121 may extract a speech signal from the audio signal and extract the remaining signal except the extracted speech signal as a non-speech signal.

[0092] According to an embodiment, when the received audio signal is an audio signal including a separate speech channel, the speech / non-speech separator 121 may extract a signal of the corresponding speech channel as a speech signal and extract signals of the remaining channels as non-speech signals.

[0093] According to another embodiment, since the non-speech signal may be mixed in the signal of the speech channel, the speech / non-speech separator 121 may extract the speech signal from the signal of the speech channel, and may extract the remaining non-speech signal of the speech channel (excluding the extracted speech signal) and the signal of the remaining channel as the non-speech signal.

[0094] For example, many audio signals reproduced in electronic devices such as multimedia devices are 5.1 channel audio signals or stereo channel audio signals. In 5.1 channel audio signals, speech exists in the center channel, while in stereo channel audio signals, speech exists in the signal with a sound image angle of 0 degrees.

[0095] Therefore, when or based on the 5.1 channel, the speech / non-speech separator 121 can extract speech from the center channel signal. Since the center channel includes non-speech signals in addition to speech signals, the speech / non-speech separator 121 can extract the non-speech signal of the center channel except the extracted speech signal and the signals of the remaining channels (subwoofer channel, front left channel, front right channel, rear left channel, and rear optical channel) as non-speech signals.

[0096] When receiving a stereo channel audio signal (or based thereon), the speech / non-speech separator 121 may extract a speech signal from a signal having a sound phase angle of 0. In this example, the non-speech signal may be included in the signal having a sound image angle of 0, and therefore, the speech / non-speech separator 121 may extract the non-speech signal (excluding the extracted speech signal) from the signal having a sound image angle of 0 and signals of the remaining sound image angles (i.e., signals at different angles other than the zero degree angle) as the non-speech signal.

[0097] The speech / non-speech separator 121 may use various existing speech signal extraction algorithms to extract speech signals. According to an embodiment, the speech / non-speech separator 121 may use an artificial intelligence-based algorithm trained to extract speech signals to extract speech signals. Here, the artificial intelligence model may include at least one of a deep learning model, a convolutional neural network (CNN) model, and a recurrent neural network (RNN) model.

[0098] An artificial intelligence model trained to extract speech signals may be included in a memory of the electronic device 100 for use by the speech / non-speech separator 121, and / or may be included in a server located outside the electronic device 100 and used by the speech / non-speech separator 121 through communication between the server and the electronic device 100.

[0099] The speech / non-speech separator 121 may extract a speech signal from the audio signal using a simple noise removal method or various speech extraction methods based on audio features.

[0100] The audio features may include at least one of time domain features and frequency domain features, wherein the time domain features include short term energy (STE), zero crossing rate (ZCR), low short term energy ratio (LSTER), high zero crossing rate ratio (HZCRR); and the frequency domain features include Mel frequency cepstral coefficients (MFCC), total power spectrum, sub-band power, frequency centroid, bandwidth, pitch frequency, spectral flux (SF), etc.

[0101] According to various embodiments, the non-speech signal may represent a signal of the remaining portion of the entire audio signal except for the speech signal extracted as described above. As an example, the non-speech signal may be extracted by the following formula 2.

[0102] [Formula 2] Non-speech signal = the entire received audio signal - speech signal

[0103] The extracted speech signal and non-speech signal are used by the speech intelligibility analyzer 122 to identify the intelligibility of speech included in the audio signal.

[0104] For example, the speech intelligibility analyzer 122 may identify speech intelligibility included in the received audio signal based on at least one of SNR, SII, STI, etc.

[0105] Specifically, according to an embodiment, the speech intelligibility analyzer 122 may identify the SNR measured by the following Formula 3 as the intelligibility of the speech included in the received audio signal.

[0106] [Formula 3]

[0107] According to another embodiment, the speech intelligibility analyzer 122 may identify a sound intelligibility index (SII) as the intelligibility of speech included in the received audio signal, wherein the sound intelligibility index (SII) may be measured using a US standard method. Here, the sound intelligibility index is also measured based on a speech signal and a non-speech signal (remaining signal) separated from the audio signal.

[0108] Figure 4 is a graph showing speech recognition accuracy according to the speech intelligibility index. Specifically, Figure 4 The speech recognition accuracy of users on three audio data sets (eg, CID W-22, NU-6, and CST) is shown, where the horizontal axis of the graph indicates SII and the vertical axis indicates speech recognition accuracy.

[0109] refer to Figure 4 , when SII is greater than or equal to 0.6, a speech recognition accuracy of 90% or higher is exhibited for all three datasets. This value (0.6) can be used by the speech intelligibility renderer 124 as a target intelligibility level.

[0110] According to another embodiment, the speech intelligibility analyzer 122 may identify an objective number (eg, STI) reflecting the degree of speech recognition as the intelligibility of speech included in the received audio signal.

[0111] The scene analyzer 123 may analyze the audio signal to obtain scene information. Specifically, the scene analyzer 123 may obtain at least one audio feature of a predetermined number of audio frames among a plurality of audio frames included in the audio signal, and obtain scene information of the predetermined number of audio frames based on the obtained audio feature.

[0112] The audio features may include at least one time domain feature, such as short term energy (STE), zero crossing rate (ZCR), low short term energy ratio (LSTER), high zero crossing rate ratio (HZCRR); and / or at least one frequency domain feature, such as Mel-frequency cepstral coefficients (MFCC), total power spectrum, sub-band power, frequency centroid, bandwidth, pitch frequency, spectral flux (SF), etc.

[0113] For example, if the audio signal is a stereo audio signal and scene information is obtained for each pair of L, R audio frames, the scene analyzer 123 can analyze the pair of L, R audio frames to extract at least one of the audio features, and can identify whether the L, R audio frames include audio types (and which type) among sound effects, shouting, and music based on the extracted audio features.

[0114] According to another embodiment, the scene analyzer 123 may obtain scene information using an artificial intelligence model trained to identify an audio type included in an audio signal. The artificial intelligence model may include at least one of a deep learning model, a convolutional neural network (CNN) model, and a recurrent neural network (RNN) model.

[0115] For example, in the case where the audio signal is a stereo audio signal and scene information is obtained for each pair of L, R audio frames, the scene analyzer 123 can use a trained CNN model and a spectrogram in which a pair of L, R audio frames are converted into a two-dimensional axis to identify whether the L, R audio frames include audio types among sound effects, shouting, and music and which audio types are included by calculating a pair of matching probabilities for each audio type, and identify whether the L, R audio frames contain sounds, implicits, and music.

[0116] An artificial intelligence model trained to recognize audio types may also be included in the memory of the electronic device 100 for use by the scene analyzer 123, like an artificial intelligence model trained to extract speech signals, and / or may be included in a server existing outside the electronic device 100 and may be used by the scene analyzer 123 through communication between the server and the electronic device 100.

[0117] Although in the above example, the scene analyzer 123 directly analyzes or processes the audio signal to obtain scene information, it should be understood that one or more other embodiments are not limited thereto. For example, the scene analyzer 123 may receive scene information corresponding to the received audio signal from an external server that generates and manages scene information about audio content.

[0118] The speech intelligibility renderer 124 may control the speech intelligibility included in the audio signal by correcting at least one of the speech signal and the residual signal using the speech intelligibility recognized by the speech intelligibility analyzer 122 and the scene information obtained by the scene analyzer 123 .

[0119] Specifically, the speech clarity renderer 124 may control the gain of the speech signal to control the speech clarity. At this time, the degree of clarity control may be identified through the speech clarity information received from the speech clarity analyzer 122 and the scene information received from the scene analyzer 123.

[0120] For example, in the case where the clarity of speech is recognized as a speech clarity index, if the user's goal is to recognize speech at a level of 90%, as shown in the above reference Figure 4 As described above, the speech intelligibility index should be about 0.6. If the currently recognized speech intelligibility index is 0.4, it should be increased by 0.2 to obtain the desired level of speech intelligibility.

[0121] By experimenting with the numerical change of the speech intelligibility index according to the gain change of the speech signal, it is possible to predict or determine how much the gain value should be adjusted to increase the intelligibility index by 0.2. For example, whenever the gain of the speech signal increases by 1 dB for various audio signals, the change in the numerical value of the speech intelligibility index can be observed and calculated back to obtain α and β of the above formula 1, and the gain adjustment value of the speech signal can be calculated by formula 1 to increase the speech intelligibility index by 0.2.

[0122] According to various embodiments, the percentage of user-recognized speech will be the target (ie, target clarity) that is recognized by the scene information.

[0123] Furthermore, according to various embodiments, by setting the target intelligibility according to the audio type included in the audio signal in different ways, the speech intelligibility may be adjusted by reflecting the production intention of the audio content producer.

[0124] When the audio type is sound effects, although the audience considers speech to be the most important, the sound effects are statistically important. Therefore, it is necessary to fully compensate for the speech intelligibility that may be reduced due to the sound effects.

[0125] For example, if the target intelligibility is a speech intelligibility index of 0.6 and the measured speech intelligibility is an SII index of 0.5, the speech intelligibility renderer 124 may adjust the gain of the speech signal by the gain adjustment value calculated via Formula 1 to improve the user's speech recognition accuracy to 90%.

[0126] Shouting during the movement has a great influence on the sense of reality that the viewer may feel. Therefore, when the audio type is shouting, if the target clarity is set too high, the shouting signal becomes relatively small compared to the speech, and thus the sense of reality may decrease compared to before the speech clarity is adjusted. Therefore, the target clarity may be set to an appropriate number through experiments. At this time, when the audio type is sound effect, the appropriate value obtained through experiments may be a value smaller than the target clarity, although it should be understood that one or more other embodiments are not limited thereto.

[0127] For example, if the target clarity is set to a speech clarity index of 0.6, the speech of the commentator and the announcer may be clear, but the residual signal in which the sound exists may be relatively small, and therefore, the viewer may not be able to fully enjoy the realism. Therefore, the target clarity may be set to a speech clarity index of around 0.5 to maintain appropriate clarity and realism.

[0128] In music, it is important to maintain the balance between the vocals and the instrumental sounds that the producer wants. Therefore, if the audio type is music, even if the clarity value of speech measured by the speech clarity analyzer 122 is low, the gain of the speech signal can be adjusted to a maximum value of 3 dB; or by not adjusting the gain of the speech signal, the intention of the music content producer can be reflected as much as possible.

[0129] In the above example, the speech intelligibility renderer 124 controls the gain of the speech signal to control the speech intelligibility, although it should be understood that one or more other embodiments are not limited thereto. For example, the speech intelligibility renderer 124 may control the gain of the non-speech signal, or may control the intelligibility of the speech signal by utilizing techniques such as a dynamic range compressor, a linear prediction coefficient (LPC) filter, a harmonic enhancer, and the like.

[0130] As described above, the speech intelligibility renderer 124 may adjust the speech intelligibility included in the received audio signal, and may generate an audio signal having the adjusted speech intelligibility as an output signal. The generated output signal may be output through at least one speaker.

[0131] On the other hand, the processor 120 may include a central processing unit (CPU), a microcontroller unit (MCU), a microprocessor (microprocessor), an electronic control unit (ECU), an application processor (AP) and / or other electronic units (hereinafter referred to as "CPU", etc.), which can process various calculations and generate control signals to control the operations of the speech / non-speech separator 121, the speech clarity analyzer 122, the scene analyzer 123, and the speech clarity renderer 124. In this case, the CPU, etc. may be provided in a form integrated with at least one or a part of the speech / non-speech separator 121, the speech clarity analyzer 122, the scene analyzer 123, and the speech clarity renderer 124.

[0132] The speech / non-speech separator 121, the speech clarity analyzer 122, the scene analyzer 123, and the speech clarity renderer 124 may be integrated into one or more functional modules and form the processor 120. For example, the speech clarity analyzer 122, the scene analyzer 123, and the speech clarity renderer 124 may be integrated into a single signal processing module, or the speech / non-speech separator 121, the speech clarity analyzer 122, the scene analyzer 123, and the speech clarity renderer 124 may be integrated into a single signal processing module. The signal processing module may be, but is not limited to, a digital signal processor (DSP).

[0133] Figure 5 is a detailed block diagram of the electronic device 100 according to an embodiment. Figure 5 , the electronic device 100 may include a processor 120, a memory 130, a display 140, a user input device 150, a communicator 180, and an audio output device 170. According to an embodiment, the Figure 5 Some of the components of the electronic device 100 are shown and / or may include other components.

[0134] The audio outputter 170 is configured to output an audio signal as an output signal. As described above, the audio outputter 170 may output an audio signal conditioned by the processor 120. The audio outputter 170 may include at least one speaker and / or a terminal or interface for outputting an audio signal to an external speaker or audio output device.

[0135] The communicator 180 is configured to communicate with an external device. The communicator 180 may include a wireless communicator 181 , a wired communicator 182 , and an input interface 183 .

[0136] The wireless communicator 181 can use wireless communication technology and / or mobile communication technology to communicate with the external broadcast transmission station 1, satellite 2, content providing server 3 and other terminal devices. Wireless communication technology includes, for example, Bluetooth, Bluetooth low power, CAN communication, Wi-Fi, Wi-Fi direct connection, ultra-wideband (UWB), Zigbee, infrared data association (IrDA), near field communication (NFC), etc., and mobile communication technology may include 3GPP, Wi-Max, long term evolution (LTE), fifth generation (5G), etc.

[0137] The wireless communicator 181 may receive audio content from another terminal device or a server, and may transmit the received audio content to the processor 120. The wireless communicator 181 may be implemented using an antenna, a communication chip, a substrate, etc. that may transmit electromagnetic waves to the outside or receive electromagnetic waves transmitted from the outside.

[0138] The wired communicator 182 can communicate with the external broadcast transmission station 1, satellite 2, content providing server 3 and other terminal devices based on the wired communication network. Here, the wired communication network can be implemented using a physical cable, for example, a pair cable, a coaxial cable, an optical fiber cable or an Ethernet cable. The wired communicator 182 can receive audio content from another terminal device or server and send the received audio content to the processor 120.

[0139] According to an embodiment, any one of the wireless communicator 181 and the wired communicator 182 may be omitted. Therefore, the audio output device 100 may include only the wireless communicator 181 or only the wired communicator 182. In addition, the audio output device 100 may include an integrated communicator that supports wireless connection of the wireless communicator 181 and wired connection of the wired communicator 182.

[0140] The input interface 183 may be connected to another device provided separately from the audio output apparatus 100, such as an external storage device, and may receive audio content from the other device and transmit or provide the received audio content to the processor 120. For example, the input interface 183 may be a universal serial bus (USB) terminal, and may include at least one of various interface terminals, such as a high-definition multimedia interface (HDMI) terminal, a Thunderbolt terminal, etc.

[0141] On the other hand, Figure 5 In the embodiment, the audio outputter 170 including at least one speaker is directly connected to the processor 120 of the electronic device 100 (specifically, the speech intelligibility renderer 124 included in the processor 120) and embedded in the electronic device 100, but it should be understood that one or more other embodiments are not limited to this.

[0142] As described above, the output signal generated by the processor 120 may be output through a separate speaker installed or provided outside the electronic device 100. In this case, the separate speaker installed outside the electronic device 100 may be connected to the electronic device 100 through the communicator 180, and the output signal generated by the processor 120 may be output to the separate speaker installed outside the electronic device 100 through the communicator 180.

[0143] According to an embodiment, the communicator 180 can communicate with an external server that generates and manages scene information of audio content, an external server that generates and manages an artificial intelligence model trained to extract speech signals, and / or an external server that generates and manages an artificial intelligence model trained to identify audio types included in audio signals, and can receive scene information or various artificial intelligence models from the external server.

[0144] The memory 130 may store audio content temporarily or non-temporarily, and may forward the audio content to the processor 120 in the form of an audio signal according to a call of the processor 120. The memory 130 may store various information necessary for the operation, processing or control operation of the processor 120 in an electronic format.

[0145] For example, the memory 130 may store all or part of various data, applications, filters, algorithms, etc. necessary for the operation of the processor 120, and may provide these data, applications, filters, algorithms, etc. to the processor 120 as needed. Here, the application may be obtained through an electronic software distribution network accessible through the wireless communicator 181 or the wired communicator 182.

[0146] The memory 130 may include, for example, at least one of a main memory device and an auxiliary memory device. The main memory may be implemented using a semiconductor storage medium such as a read-only memory (ROM) and / or a random access memory (RAM). The ROM may include, for example, a conventional ROM, an EPROM, an EEPROM, and / or a mask ROM. The RAM may include, for example, a DRAM and / or an SRAM. The auxiliary memory device may be implemented using at least one storage medium capable of permanently or semi-permanently storing data, such as a flash memory device, a secure digital memory (SD) card, a solid-state drive (SSD), a solid-state drive (SSD), a hard disk drive (HDD), a magnetic drum, a compact disk (CD), an optical medium (such as a digital video disk (DVD) or a laser disk), a magnetic tape, a magneto-optical disk, and / or a floppy disk.

[0147] exist Figure 2 In the electronic device 100, the input 110 is configured to receive an audio signal and provide it to the processor 120. Figure 5, the audio signal can be provided to the processor 120 through the communicator 180 or the memory 130. Figure 5 , the communicator 180 and the memory 130 may correspond to Figure 2 The input device 110 described in .

[0148] The display 140 displays various images. In particular, when a video signal is received through the communicator 180 or the memory 130, the processor 120 can reproduce the video through the display 140. The display 140 may include various types of display panels, such as but not limited to a liquid crystal display (LCD) panel, an organic light emitting diode (OLED) panel, a plasma display panel (PDP) panel, an inorganic LED panel, a micro LED panel, etc. The display 140 may constitute a touch screen having a touch panel.

[0149] The user input 150 is configured to receive various user inputs. To this end, the user input 150 may include various buttons or a touch panel, but is not limited thereto.

[0150] The processor 120 controls the overall operation of the audio output device 100. The processor 120 may perform the operations described above with reference to Figures 1 to 4 The operation of the electronic device 100, the processor 120 or the functional blocks of the processor 120.

[0151] If the audio content is sent through the memory 130 or the communicator 180, the processor 120 can decode the audio content and convert the content into an uncompressed format. Here, decoding refers to the process of restoring an audio signal compressed by an audio compression format such as MPEG Layer 3 (MP3), Advanced Audio Coding (AAC), Audio Codec-3 (AC-3), Digital Theater System (DTS), Free Lossless Audio Codec (FLAC), Windows Media Audio (WMA), etc. to an uncompressed audio signal. If the audio content is not compressed, the decoding process can be omitted. The recovered audio signal may include one or more channels.

[0152] Figure 6 is a flowchart showing a method of controlling the electronic device 100 according to an embodiment. Figure 6 , in operation S610, the electronic device 100 may receive an audio signal, and in operation S620, identify speech intelligibility included in the audio signal based on the received audio signal.

[0153] For example, the electronic device 100 may calculate speech intelligibility based on a speech signal included in a received audio signal and a non-speech signal other than the speech signal. The electronic device 100 may extract the speech signal included in the audio signal using an artificial intelligence model trained to extract the speech signal included in the audio signal, and may extract the remaining signal other than the extracted speech signal from the audio signal as a non-speech signal.

[0154] Therefore, in operation S630, the electronic device 100 may adjust the audio signal so that the recognized speech clarity becomes the target clarity. Here, the target clarity is a value set based on scene information related to the audio type included in the audio signal, and the audio type included in the scene information may include at least one of sound effects, shouting, and music.

[0155] The electronic device 100 may obtain at least one audio feature for the audio signal and obtain scene information based on the obtained at least one audio feature. Alternatively, the electronic device 100 may obtain scene information using an artificial intelligence model trained to recognize the audio type included in the audio signal.

[0156] According to an embodiment, the target clarity may be set differently for each audio type. For example, if the audio type is a sound effect, the target clarity may be set relatively high, but is not limited thereto.

[0157] The above-mentioned intelligibility may be any one of a signal-to-noise ratio (SNR) of a speech signal and a non-speech signal included in the audio signal and a speech intelligibility index (SII) based on the speech signal and the non-speech signal.

[0158] Therefore, when the clarity of speech included in the audio signal is determined as (or based on) the signal-to-noise ratio, the electronic device 100 may adjust the gain of the speech signal by the difference between the target clarity and the recognized speech clarity to correct the audio signal.

[0159] When speech intelligibility included in an audio signal is identified as a speech intelligibility index (or based thereon), the electronic device 100 may calculate a gain adjustment value based on the following Formula 1, and adjust the gain of the speech signal by the calculated gain adjustment value to correct the audio signal.

[0160] [Formula 1] Gain adjustment value = α*(SII 目标 -SII 测量 )+β

[0161] SII 目标 is the target clarity, SII 测量is the recognized speech intelligibility, and α and β are constant values ​​experimentally calculated according to the amount of gain change of the speech signal.

[0162] According to various embodiments as described above, speech intelligibility can be controlled more accurately. In addition, the best speech intelligibility can be adjusted by reflecting the production intention of the audio content producer. Therefore, the best sound experience can be provided to the user.

[0163] Various embodiments may be implemented with software including instructions stored in a machine-readable storage medium (e.g., a computer). Here, the device is a device that calls the stored instructions from the storage medium and operates according to the called instructions, and may include the electronic devices 100, 100-1 to 100-4 according to the embodiments.

[0164] When the instruction is executed by the processor, the processor may perform the function corresponding to the instruction directly or by using other components under the control of the processor. The instruction may include a code generated by a compiler or a code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, the term "non-transitory" only means that the storage medium is not limited to a signal but is tangible, and does not distinguish between a case where data is semi-permanently stored in the storage medium and a case where data is temporarily stored in the storage medium.

[0165] The method according to the above-described embodiment may be provided as included in a computer program product. The computer program product may be traded between a seller and a consumer as a product. The computer program product may be distributed online in the form of a machine-readable storage medium (e.g., a compact disk read-only memory (CD-ROM)), or through an application store (e.g., a PLAY STORE). TM and APP STORE TM ) distribution, or directly distributed (e.g., downloaded or uploaded) between users (e.g., smart phones). In the case of online distribution, at least a portion of the computer program product may be at least temporarily stored in a server of the producer, a server of an application store, or a machine-readable storage medium (e.g., a memory of a relay server) or temporarily generated in them.

[0166] According to an embodiment, each element (for example, module or program) of the above-mentioned elements may include a single entity or multiple entities. In addition, at least one element or operation in the above-mentioned corresponding elements may be omitted, or at least one other element or operation may be added. Alternatively or additionally, multiple components (for example, module or program) may be combined to form a single entity. In this way, the integrated entity may perform at least one function of each of the multiple elements in the same manner or in a similar manner as the manner performed by the corresponding elements in the multiple elements before integration. Modules, program modules or operations performed by other elements according to various embodiments may be performed continuously, in parallel, repeatedly or tentatively, or at least some operations may be omitted, or other operations may be added thereto.

[0167] Although the present disclosure has been described by way of example, it will be understood by those skilled in the art that various changes in form and detail may be made therein without departing from its basic features. In addition, the above-described embodiments are not intended to limit the technical spirit of the present disclosure, but are used to describe technical aspects, and the scope of the technical aspects of the present disclosure is not limited to the embodiments herein. Therefore, the protection scope of the present disclosure should be interpreted as including at least the attached claims and all technical concepts falling within the scope of the present disclosure.

Claims

1. Electronic devices, including: Input device; as well as a processor configured to: based on receiving an audio signal via the input, acquire speech intelligibility based on a speech signal included in the audio signal and a non-speech signal other than the speech signal, and modify the audio signal so that the speech intelligibility becomes a target intelligibility, wherein the target definition is set based on scene information related to an audio type included in the audio signal, The audio type includes at least one of sound effects, shouting, music and speech. The speech intelligibility is one of a signal-to-noise ratio SNR of the speech signal and the non-speech signal included in the audio signal and a speech intelligibility index SII based on the speech signal and the non-speech signal.

2. The electronic device according to claim 1, wherein: The processor is also configured to extract the speech signal contained in the audio signal using an artificial intelligence model trained to extract the speech signal contained in the audio signal, and extract one or more remaining signals other than the extracted speech signal from the audio signal as the non-speech signal.

3. The electronic device as claimed in claim 1, wherein: The speech intelligibility is the signal-to-noise ratio; and The processor is also configured to adjust a gain of the speech signal by as much as a difference between the target intelligibility and the achieved speech intelligibility to modify the audio signal.

4. The electronic device as claimed in claim 1, wherein: The speech intelligibility is the speech intelligibility index; The processor is further configured to: calculate a gain adjustment value, and adjust the gain of the speech signal by as much as the calculated gain adjustment value to modify the audio signal; The gain adjustment value is calculated according to the following formula: Gain adjustment value = α * (SII 目标 -SII 测量 )+β; and The SII 目标 Indicates the target clarity, the SII 测量 represents the acquired speech intelligibility, and α and β represent constant values ​​experimentally calculated by changing the quantity of the speech intelligibility index as the gain of the speech signal changes.

5. The electronic device as claimed in claim 1, wherein: The processor is further configured to: obtain at least one audio feature about the audio signal, and obtain the scene information based on the obtained at least one audio feature.

6. The electronic device as claimed in claim 1, wherein: The processor is also configured to obtain the scene information using an artificial intelligence model trained to distinguish audio types included in an audio signal.

7. The electronic device as claimed in claim 1, wherein: The target intelligibility is set differently for different audio types.

8. The electronic device as claimed in claim 1, wherein: Based on the audio type being the sound effect, the target clarity is set higher than a case where the audio type is the shouting.

9. A method for controlling an electronic device, the method comprising: Get audio signal; acquiring speech intelligibility based on a speech signal included in the audio signal and a non-speech signal other than the speech signal; as well as modifying the audio signal so that the speech intelligibility reaches the target intelligibility, wherein the target definition is set based on scene information related to an audio type included in the audio signal, The audio type includes at least one of sound effects, shouting, music and speech. The speech intelligibility is one of a signal-to-noise ratio SNR of the speech signal and the non-speech signal included in the audio signal and a speech intelligibility index SII based on the speech signal and the non-speech signal.

10. The method of claim 9, wherein: Acquiring the speech intelligibility includes: extracting the speech signal included in the audio signal using an artificial intelligence model trained to extract the speech signal included in the audio signal; and One or more remaining signals other than the extracted speech signal are extracted from the audio signal as the non-speech signal.

11. The method of claim 9, wherein: The speech intelligibility is the signal-to-noise ratio; as well as Modifying the speech intelligibility includes adjusting a gain of the speech signal by as much as a difference between the target intelligibility and the achieved speech intelligibility to modify the audio signal.

Images