Sound pickup device, sound pickup method, and program

Abstract

In a pickup device (100), a phoneme information acquisition section (short-time phoneme analysis section (16)) acquires information related to a phoneme included in an audio signal. A band expansion section (34) generates a high-frequency component of the audio signal on the basis of the information related to the phoneme, and expands a frequency band of the audio signal by the high-frequency component.

Description

Technical Field

[0001] This invention relates to a technique for extending the bandwidth of narrowband audio signals. Background Technology

[0002] Patent Document 1 discloses a bandwidth-extending device that includes a nonlinear processing unit for nonlinear processing of a bandwidth-limited narrowband signal, thereby generating a bandwidth-extended signal. The device acquires the spectral envelope of the bandwidth-extended signal, generates a signal that flattens the acquired spectral envelope and assigns its characteristics to white noise, and then superimposes this signal onto the bandwidth-extended signal.

[0003] Existing technical documents Patent documents Patent document 1: Japanese Patent Application Publication No. 2012-27255. Summary of the Invention

[0004] However, the LPC analysis in Patent Document 1 represents the human voice, i.e., the constituent elements of sound—the throat and vocal tract. In the analysis at the bandwidth-limited stage, information equivalent to higher-order formants is lost, which may lead to the reconstruction of incorrect broadband audio components. Furthermore, in the technology of Patent Document 1, although unvoiced sounds are generated by LPC noise synthesis based on white noise, as mentioned above, the LPC information equivalent to the extended bandwidth is inaccurate. Therefore, depending on the type of unvoiced sound, i.e., the type of phoneme, unexpected noise components may be added.

[0005] The present invention was made in view of the above-mentioned situation, and its object is to provide a technique that can improve the sound quality of bandwidth-limited audio signals.

[0006] To address the aforementioned issues, one aspect of the sound pickup device of this embodiment is characterized by comprising: a phoneme information acquisition unit for acquiring phoneme-related information contained in an audio signal; and a bandwidth extension unit for generating a high-frequency component of the audio signal based on the phoneme-related information, and extending the bandwidth of the audio signal using the high-frequency component.

[0007] Another aspect of this embodiment is a sound pickup method. This method is characterized by a computer performing the following steps: acquiring phoneme-related information contained in an audio signal; generating high-frequency components of the audio signal based on the phoneme-related information; and extending the frequency band of the audio signal using the high-frequency components.

[0008] Another aspect of this embodiment is a program. This program is characterized by causing a computer to perform: acquiring phoneme-related information contained in an audio signal; generating high-frequency components of the audio signal based on the phoneme-related information; and extending the frequency band of the audio signal using the high-frequency components.

[0009] Furthermore, any combination of the above-mentioned constituent elements, as well as schemes for converting the expression of this embodiment into methods, apparatus, systems, recording media, computer programs, etc., are also effective aspects of this embodiment.

[0010] According to this embodiment, a technique can be provided that can improve the sound quality of bandwidth-limited audio signals. Attached Figure Description

[0011] Figure 1 This is a diagram illustrating the functional configuration of the pickup device according to the embodiment.

[0012] Figure 2 This is a graph showing the time waveforms of audio signals acquired by a bone conduction microphone and a regular microphone, respectively.

[0013] Figure 3 It is shown Figure 2 A graph showing the frequency characteristics of the vowel vocal range within a predetermined analysis width in the waveform.

[0014] Figure 4 It is shown Figure 2 A graph showing the frequency characteristics of the consonant vocalization range within a predetermined analysis width in the waveform.

[0015] Figure 5 (a) is a graph showing the spectrogram of the audio signal. Figure 5 (b) shows from Figure 5 A spectrogram of the harmonic structure components extracted from the audio signal (a). Figure 5 (c) is showing from Figure 5 Subtract from the audio signal of (a) Figure 5 The spectrum of the residual component obtained from the harmonic structure component of (b) is shown in the figure.

[0016] Figure 6 This is a diagram showing the functional structure of the short-time phoneme analysis unit.

[0017] Figure 7 This is another example of the functional configuration of the short-time phoneme analysis unit.

[0018] Figure 8 This is a diagram illustrating the functional structure of a machine learning device.

[0019] Figure 9 (a) is a diagram schematically showing the frequency characteristics of the output signal of the harmonic structure component extraction unit. Figure 9 (b) shows the generated overtones superimposed on the base. Figure 9 The graph shows the frequency characteristics of the audio signal on signal (a).

[0020] Figure 10 (a) shows the higher-order resonant peak superimposed on the frequency distribution correction target. Figure 9 The frequency response graph of (b) is shown. Figure 10 (b) is a graph showing the frequency characteristics of the corrected audio signal.

[0021] Figure 11 (a) is a graph showing the time waveform of the audio signal when the sound "shio" is pronounced. Figure 11 (b) is a graph showing the frequency characteristics of the consonant “s”.

[0022] Figure 12 (a) is a graph showing the frequency characteristics of the consonant "h". Figure 12 (b) is a graph showing the frequency characteristics of the consonant “k”.

[0023] Figure 13 This is a diagram showing the correspondence between the types of consonants and their waveform envelopes.

[0024] Figure 14 This is a flowchart illustrating the operational steps of the sound pickup device. Detailed Implementation

[0025] Before detailing the implementation method, the following is a brief overview. This embodiment relates to an audio signal pickup device (hereinafter referred to as the pickup device) for extending the frequency band of audio signals such as telephone signals (200-3400Hz) or the frequency band of narrow-band audio signals picked up by vibration sensors such as bone conduction microphones. Within the sound-producing range, the pickup device determines the types of vowels and consonants by analyzing the phonemes of the input signal. For vowels, harmonic structure components are interpolated within the frequency extension band, and correction filtering is performed to reproduce the higher-order formant characteristics corresponding to the determined vowel. For consonants, within the frequency extension band, high-frequency noise components that correspond to the waveform envelope of the consonant type are subjected to gain correction processing to ensure consistency with the sound pressure level of the input signal. Thus, the pickup device adds a signal generated according to the phoneme type within the frequency extension band to the input signal. This improves the perceived clarity of speech.

[0026] Figure 1This diagram illustrates the functional configuration of the sound pickup device 100 according to the embodiment. The sound pickup device 100 includes a microphone 10, an analog-to-digital converter 12, a sound range determination unit 14, a short-time phoneme analysis unit 16, a bandwidth extension unit 34, and a digital-to-analog converter 32. The bandwidth extension unit 34 includes a harmonic structure component extraction unit 18, a residual component calculation unit 20, a high-frequency overtone generation and superposition unit 22, a higher-order formant correction unit 24, a high-frequency noise generation and superposition unit 26, a high-frequency envelope correction unit 28, and an adder unit 30.

[0027] The blocks shown in the block diagrams of this disclosure can be implemented in hardware by any computer's CPU, memory, other LSIs, and in software by programs loaded into memory, but what is depicted here are functional blocks implemented through their cooperation. Therefore, those skilled in the art will understand that these functional blocks can be implemented in various forms by hardware alone, by software alone, or by a combination thereof.

[0028] Furthermore, at least some of the functional blocks shown in the block diagrams of this disclosure can be implemented as computer programs. These computer programs can be stored in a recording medium and installed into a computer via that medium. Alternatively, the computer programs can be downloaded and installed into a computer via a communication network. The computer's CPU can read the computer program into main memory and execute it, thereby enabling the functionality of the functional blocks implemented in the computer program.

[0029] Microphone 10, for example, is a bone conduction microphone, which primarily acquires the vocal signals emitted by a person as vibration signals propagating to the human body. Hereinafter, the vibration signal will be referred to as the audio signal. The bone conduction microphone can also be a vibration sensor that acquires the vibration signal. Microphone 10 outputs the acquired audio signal to AD converter 12.

[0030] Typically, the audio signal picked up by a bone conduction microphone does not contain background noise or audio signals from speakers other than the speaker. It has great advantages for the purpose of obtaining independent audio signals. However, it is different from the original speaker's own voice characteristics and is accompanied by the transmission characteristics of the human body, especially the reduction of high-frequency components, resulting in a muffled sound.

[0031] Figure 2 This is a graph showing the time waveforms of audio signals acquired by a bone conduction microphone and a regular microphone, respectively. The time waveform is approximately 20 seconds long. A regular microphone is a microphone that converts air vibrations into audio signals. Figure 3 It is shown Figure 2A graph showing the frequency characteristics of the vowel phonation interval within a predetermined analysis width in the waveform. The analysis width is tens of milliseconds. Figure 3 The frequency characteristics contain a large number of overtone components known as harmonic structures, which periodically exhibit high sound pressure levels along the frequency axis. Figure 4 It is shown Figure 2 A graph showing the frequency response of the consonant vocalization range within a predetermined analysis width in the waveform. Figure 4 In its frequency response, it contains a large amount of white noise in the high-frequency region. Figure 3 , Figure 4 In the diagram, the frequency characteristics of a standard microphone are represented by a solid line, while the frequency characteristics of a bone conduction microphone are represented by a dashed line. For example... Figure 3 , Figure 4 As clearly shown, bone conduction microphones attenuate high-frequency components compared to regular microphones, thus producing a muffled sound. In particular, in Figure 3 In the high-frequency region, the absence of the portion of vowels known as higher-order formants—characteristics of the speaker's voice (bands with sound pressure levels higher than other frequency bands)—is quite noticeable. Figure 4 In the high-frequency region, the missing feature portion (waveform envelope of noise component) representing the differences in consonant types is quite obvious.

[0032] In this embodiment, the audio signal acquired by the microphone 10 is an audio signal with attenuated high-frequency components, the purpose of which is to extend the bandwidth-limited audio signal to the bandwidth of the original sound. Furthermore, not limited to the audio signal acquired by the microphone 10, bandwidth-limited audio signals transmitted through communication circuits such as telephones, or low-rate encoded narrowband audio signals, can also be input to the AD converter 12 as input signals.

[0033] The AD converter 12 performs analog-to-digital conversion on the acquired audio signal, converting the audio signal into a digital audio signal. The AD converter 12 divides the digital audio signal into blocks with a predetermined unit time width and outputs the block audio signals to the sound range determination unit 14, the short-time phoneme analysis unit 16, the harmonic structure component extraction unit 18, and the residual component calculation unit 20, respectively. The predetermined unit time width is, for example, tens of milliseconds. The audio signal is divided into blocks with a predetermined number of processing samples equivalent to tens of milliseconds. The AD converter 12 outputs the block audio signals sequentially at time intervals equal to the unit time width. The sound range determination unit 14, the short-time phoneme analysis unit 16, and the bandwidth expansion unit 34 perform the following processing for each audio signal block.

[0034] The vocal interval determination unit 14 determines whether the unit time width of the current block is a vocal interval or a non-vocal interval based on the audio signal of the current block. A vocal interval is the time period during which the target's voice is detected; it can also be called a speech interval. A non-vocal interval can also be called a silent interval. The vocal interval determination unit 14 can observe the target audio signal, for example, detecting a vocal interval when the sound pressure level exceeds a predetermined level, or, to improve accuracy, detecting vocal intervals by detecting human voice characteristics through frequency analysis (for example, see Japanese Patent No. 5874344 and Japanese Patent No. 5948918). In particular, the technology disclosed in Japanese Patent No. 5874344 is suitable for detecting voiced sounds equivalent to vowels. The technology disclosed in Japanese Patent No. 5948918 is suitable for detecting voiceless sounds equivalent to consonants. The vocal interval determination unit 14 sends VAD (Voice Activity Detection) information, indicating whether it is a vocal interval or a non-vocal interval, to the short-time phoneme analysis unit 16. At this time, if detailed analysis results are obtained, the sound range determination unit 14 can also include the determination results of voiced and unvoiced sounds in it and send them.

[0035] The short-time phoneme analysis unit 16 analyzes the phonemes of the audio signal of the current block using the audio signal of the current block and previously acquired audio signals. For example, if the phonemes are vowels, they are "a", "i", "u", "e", and "o". That is, the short-time phoneme analysis unit 16 acquires the band-limited audio signal block by block and, based on the acquired audio signal, acquires the phoneme-related information contained in the audio signal block by block. When the VAD information received from the vocal interval determination unit 14 indicates a vocal interval, the short-time phoneme analysis unit 16 acquires phoneme-related information (phoneme information) containing information on at least one of vowels and consonants. When the audio signal of the vocal interval contains only vowels, the phoneme information contains vowel phoneme candidates as vowel information and background noise information. When the audio signal of the vocal interval contains only consonants, the phoneme information contains consonant phoneme candidates as consonant information and background noise information. When the audio signal of the vocal interval contains both vowels and consonants, the phoneme information contains information on both vowel phoneme candidates and consonant phoneme candidates. When the VAD information received from the vocal range determination unit 14 indicates a non-vocal range, the short-time phoneme analysis unit 16 acquires background noise information as phoneme-related information. In the case of a non-vocal range, the phoneme-related information does not include vowel and consonant information. The short-time phoneme analysis unit 16 is equivalent to a phoneme information acquisition unit. Furthermore, background noise refers to ambient noise other than the target sound. Background noise information is information indicating the probability that the sound contained in the audio signal is background noise.

[0036] If a lexical analysis method combining acoustic models, language models, and vocalization models commonly used in speech recognition is employed, very accurate phoneme unit analysis can be performed. However, the purpose of the pickup device 100 in this implementation is to improve the quality of audio signals in conversations, which necessitates a delay that is not uncomfortable for the listener during conversation, such as a delay of tens of milliseconds. Therefore, the short-time phoneme analysis unit 16 does not use the language models or vocalization models used in general speech recognition. Even in the acoustic model, only analysis methods based on the transition from past signals to the current signal can be used. The analysis method is not particularly limited here, but MFCC (Mel-Frequency Cepstrum Coefficient) parameters, which are used as methods for extracting sound features, and extremely low-delay phoneme analysis methods using forward predictive hidden Markov models, can be used. Vowels are relatively easy to analyze, but consonants have short pronunciation times, and their feature quantities deviate from vowels among phonemes in a less clear manner. Therefore, consonants can be classified as phoneme types rather than phonemes. Phoneme types include, for example, plosives, fricatives, approximations, and nasals, which are equivalent to consonant types. Since these phoneme types share the same generation process, their sound components also have similar relationships. The specific processing of the short-time phoneme analysis unit 16 will be described later. The short-time phoneme analysis unit 16 sends the acquired phoneme information to the higher-order formant correction unit 24 and the high-frequency envelope correction unit 28 of the bandwidth extension unit 34.

[0037] For each audio signal block, the bandwidth extension unit 34 generates a high-frequency component of the audio signal based on the phoneme-related information acquired by the short-time phoneme analysis unit 16, and extends the bandwidth of the audio signal using the generated high-frequency component. When the phoneme-related information includes vowel information, the bandwidth extension unit 34 generates a high-frequency component containing overtone components, which reproduce the higher-order formant characteristics corresponding to the vowel. When the phoneme-related information includes consonant information, the bandwidth extension unit 34 generates a high-frequency component containing noise components, which have frequency characteristics corresponding to the consonant and are adjusted in magnitude based on the sound pressure level of the audio signal. When the phoneme-related information includes background noise information, the bandwidth extension unit 34 generates a high-frequency component containing noise components, which have frequency characteristics corresponding to the background noise and are adjusted in magnitude based on the sound pressure level of the audio signal. These processes of the bandwidth extension unit 34 will be described in more detail below.

[0038] The harmonic structure component extraction unit 18 separates and extracts vowel components with harmonic structures from the audio signal of the current block. The harmonic structure can also be called the overtone structure. The harmonic structure component extraction unit 18 first detects the pitch information of the audio signal using methods such as autocorrelation. Then, it obtains linear prediction coefficients that partially model the human throat and oral vocal tract using an LPC coefficient acquisition method, such as the Levinson-Durbin method. Based on the pitch information, it prepares a continuous excitation signal as the input signal and passes it through a linear prediction filter to generate components with harmonic structures. Alternatively, it can use a portion of the speech analysis and synthesis method in representative speech coding schemes such as CELP (Code-Excited Linear Prediction) to extract the harmonic structure components.

[0039] Figure 5 (a) is a spectrogram showing the audio signal over a specific time period. Figure 5 (b) shows from Figure 5 The graph shows the spectrogram of the harmonic structure component extracted from the audio signal (a). This harmonic structure component is obtained by the harmonic structure component extraction unit 18 generating an excitation signal based on pitch information and inputting the excitation signal into a linear prediction filter composed of linear prediction coefficients obtained through speech analysis. Figure 5 As can be seen from (b), the overtone components of the original audio signal, which are the high-brightness components, are extracted. On the other hand, the noise components in the high-frequency band of the non-overtone components, i.e., the consonant components, are reduced. The harmonic structure component extraction unit 18 sends the extracted harmonic structure components to the high-frequency overtone generation and superposition unit 22 and the residual component calculation unit 20.

[0040] Next, the residual component calculation unit 20 calculates the residual component by subtracting the harmonic structure component obtained by the harmonic structure component extraction unit 18 from the audio signal of the current block that is aligned in time. If it is a vocal interval, the residual component can be considered to be equivalent to the consonant component.

[0041] Figure 5 (c) is showing from Figure 5 Subtract from the audio signal of (a) Figure 5 The spectrum of the residual components obtained from the harmonic structure components of (b) is shown in the figure. Figure 5 As shown in (c), since the harmonic structure components have been subtracted from the original audio signal and the high-brightness stripe pattern representing overtones in the frequency axis direction cannot be observed, it can be seen that the overtone components have been cleanly removed. The residual component calculation unit 20 sends the acquired residual components to the high-frequency noise generation and superposition unit 26.

[0042] Furthermore, as mentioned above, when dealing with bandwidth-limited audio signals acquired by bone conduction microphones or audio signals transmitted via narrowband telephone lines of 300Hz to 3400Hz, bandwidth expansion requires processing with an audio format that includes high-frequency sound components to improve the clarity of speech. Therefore, if the above-mentioned AD conversion or SRC (Sample Rate Converter) function has already digitized at, for example, a sampling frequency of 16kHz or 24kHz, then this audio format is used. If the processing up to this point is, for example, a narrowband audio format with a sampling frequency of 8kHz or less, then sampling rate conversion processing is performed to ensure the high-frequency band. Although not shown, if this SRC function is required, it is inserted after the harmonic structure component extraction unit 18 and the residual component calculation unit 20. In the vocal range determination unit 14 and the short-time phoneme analysis unit 16, the influence of broadband components on the determination accuracy is very slight, so a narrowband audio format can also be used. From the perspective of signal processing load, the point in time at which the broadband audio format is used can be arbitrarily selected.

[0043] Next, the high-frequency overtone generation and superposition unit 22 extends the frequency band by generating and superimposing high-frequency overtones from the harmonic structure components obtained from the harmonic structure component extraction unit 18. As a method for generating overtones, any known method can be selected, such as adding overtones by clipping the upper and lower peak portions of the audio signal on the time axis to make it closer to a square wave, or expanding in the frequency domain and shifting the low-frequency components to the frequency at which overtones are generated. The high-frequency overtone generation and superposition unit 22 sends the audio signal composed of harmonic structure components, superimposed with overtones and with an extended frequency band, to the higher-order formant correction unit 24.

[0044] When the phoneme information provided from the short-time phoneme analysis unit 16 includes vowel phoneme information, the higher-order formant correction unit 24 reproduces the higher-order formants as acoustic characteristic quantities of the extended overtone components based on the vowel phoneme information provided from the short-time phoneme analysis unit 16. Since the high-frequency overtone generation superposition unit 22 only adds overtone components without level adjustment, the higher-order formant correction unit 24 adjusts the formant shape of each vowel representing the frequency characteristics through a correction filter. The higher-order formant correction unit 24 pre-stores the formant information of each vowel, selects the formant corresponding to the vowel contained in the phoneme information, and adjusts the level of the overtone components in a manner that matches the selected formant.

[0045] The bandwidth expansion method in the aforementioned Patent Document 1 is not based on the generation of high-frequency overtones of phoneme information, and therefore may not be accompanied by the speaker's own voice quality. However, in the implementation, since bandwidth expansion including the formant shape as a sound feature is performed, it can be converted into a sound with natural voice quality and the speaker's own characteristics. The higher-order formant correction unit 24 sends an audio signal containing overtone components that reproduce the features of the higher-order formants to the adder unit 30.

[0046] On the other hand, when the phoneme information provided by the short-time phoneme analysis unit 16 is background noise information and does not contain vowel phoneme information, the higher-order formant correction unit 24 sends the audio signal containing the harmonic structure components extracted by the harmonic structure component extraction unit 18 to the addition unit 30 as is.

[0047] In addition, the high-frequency overtone generation and superposition unit 22 may not superimpose the generated overtones onto the original harmonic structure components, but the higher-order formant correction unit 24 may superimpose the level-adjusted overtones onto the original harmonic structure components.

[0048] Furthermore, the high-frequency noise generation and superposition unit 26 creates and superimposes a high-frequency noise component onto the residual component containing consonant components received from the residual component calculation unit 20, thereby extending the frequency band. The noise is, for example, white noise. The high-frequency noise generation and superposition unit 26 uses a noise generation circuit or a pre-generated noise signal to adjust the level of the high-frequency noise component to match the level of the frequency band connected to the low frequency before superposition. Alternatively, the high-frequency noise level may not be unified in the high-frequency noise generation and superposition unit 26; instead, the high-frequency envelope correction unit 28 in the subsequent stage can correct the level of the extended high-frequency noise component while correcting the high-frequency envelope. The high-frequency noise generation and superposition unit 26 sends the audio signal superimposed with high-frequency noise to the high-frequency envelope correction unit 28.

[0049] When the phoneme information provided from the short-time phoneme analysis unit 16 includes consonant phoneme information, the high-frequency envelope correction unit 28 reproduces the noise component with frequency characteristics corresponding to the consonant based on the consonant type phoneme information provided from the short-time phoneme analysis unit 16. The high-frequency envelope correction unit 28 corrects the high-frequency envelope of the expanded high-frequency noise component in the audio signal provided from the high-frequency noise generation and superposition unit 26. As described with respect to the short-time phoneme analysis unit 16, even if instantaneous consonant phoneme analysis is difficult, the type of consonant can be estimated with a certain probability using limited low-frequency information. The estimation method will be described later. The high-frequency envelope correction unit 28 performs level correction on the superimposed high-frequency noise component from the provided phoneme information representing the consonant type, using the high-frequency envelope prepared in advance for each consonant type.

[0050] The main component of a consonant is white noise, positioned before and after a vowel with a predetermined time width, and is perceived as a consonant. Key factors for recognition as a consonant are its timing and high-frequency envelope. Regarding timing, sufficient information can be maintained through the VAD information from the vocal range determination unit 14 and the low-frequency consonant component present in the acquired audio signal, despite its limited bandwidth. Furthermore, the high-frequency envelope of the emitted consonant can be reproduced based on the consonant type information; therefore, it is expected to obtain a sound quality that is perceived as a consonant with appropriate noise components, unlike the sound quality with additional noise added as in Patent Document 1. The high-frequency envelope correction unit 28 sends the audio signal with the corrected high-frequency envelope to the addition unit 30.

[0051] On the other hand, when the phoneme information provided by the short-time phoneme analysis unit 16 is background noise information and does not contain consonant phoneme information, the high-frequency envelope correction unit 28 does not correct the high-frequency envelope of the high-frequency noise component, but sends the audio signal provided by the high-frequency noise generation and superposition unit 26 to the addition unit 30 as is. That is, the frequency characteristics corresponding to the background noise are, for example, the frequency characteristics of white noise.

[0052] In addition, the high-frequency noise generation and superposition unit 26 may not superimpose the generated high-frequency noise component onto the residual component, but the high-frequency envelope correction unit 28 may superimpose the noise component after high-frequency envelope correction onto the residual component.

[0053] The adder 30 adds the audio signal received from the higher-order formant correction unit 24 to the audio signal received from the high-frequency envelope correction unit 28, and outputs the resulting audio signal to the DA converter 32. The bandwidth of the audio signal output from the adder 30 is extended by the high-frequency component. When the audio signal of the current block in the sound-producing interval contains only vowels, the high-frequency component of the audio signal output from the adder 30 includes an overtone component that reproduces the higher-order formant characteristics corresponding to the vowel, and a noise component with level-adjusted frequency characteristics corresponding to the background noise. When the audio signal of the current block in the sound-producing interval contains only consonants, the high-frequency component of the audio signal output from the adder 30 includes a noise component with level-adjusted frequency characteristics corresponding to the consonant. When the audio signal of the current block in the sound-producing interval contains both vowels and consonants, the high-frequency component of the audio signal output from the adder 30 includes an overtone component that reproduces the higher-order formant characteristics corresponding to the vowel, and a noise component with level-adjusted frequency characteristics corresponding to the consonant. In the non-sound-producing region, the high-frequency components of the audio signal output from the adder 30 include a noise component with frequency characteristics adjusted to match the background noise. Furthermore, in the sound-producing region, when the audio signal of the current block contains only consonants, the high-frequency components of the audio signal output from the adder 30 may include both a noise component with frequency characteristics adjusted to match the consonants and a noise component with frequency characteristics adjusted to match the background noise.

[0054] The DA converter 32 performs digital-to-analog conversion on the audio signal received from the adder 30 and outputs the converted analog audio signal as the output signal.

[0055] Next, an example of the specific processing of the short-time phoneme analysis unit 16 will be explained. Figure 6 This diagram illustrates the functional configuration of the short-time phoneme analysis unit 16. The short-time phoneme analysis unit 16 includes a frequency conversion unit 40, an acoustic feature extraction unit 42, an acoustic feature accumulation unit 44, a feature matching unit 46, an acoustic feature database 48, a phoneme candidate extraction unit 50, a phoneme candidate accumulation unit 52, a statistical probability model unit 54, and a phoneme estimation unit 56.

[0056] As described above, the audio signal input to the short-time phoneme analysis unit 16 has been segmented. Each part of the short-time phoneme analysis unit 16 performs the following processing for each audio signal segment. As described above, since the purpose of the pickup device 100 in this embodiment is to provide measures to improve the quality of audio signals in the form of calls, the allowable delay is required to be, for example, around tens of milliseconds, so that the time difference is imperceptible to the human. Therefore, the short-time phoneme analysis unit 16 determines and outputs short-time phoneme information as the final output for each audio signal segment.

[0057] The frequency conversion unit 40 receives the audio signal of the current block and converts the time information into frequency information. The frequency conversion unit 40 then sends the frequency information of the converted audio signal to the acoustic feature extraction unit 42.

[0058] The acoustic feature extraction unit 42 extracts acoustic feature information by parameterizing the features of the audio signal of the current block based on the received frequency information. The acoustic feature information serves as an indicator for classification into each phoneme. Examples of acoustic feature information include frequency characteristics, or MFCC information obtained by treating frequency characteristic information as a signal sequence and further performing frequency conversion. The extraction of acoustic feature information can utilize known techniques. The acoustic feature extraction unit 42 sends the extracted acoustic feature information to the acoustic feature accumulation unit 44 and the feature matching unit 46.

[0059] The acoustic feature accumulation unit 44 accumulates the acoustic features extracted by the acoustic feature extraction unit 42 from the previous few blocks in chronological order. The acoustic feature accumulation unit 44 also accumulates the acoustic features received for the current block.

[0060] There are significant differences in frequency characteristics between vowels and consonants, and each phoneme also exhibits specific features in its formants or waveform envelopes in the frequency domain. The acoustic feature database 48 pre-stores acoustic features by phoneme as a database.

[0061] The feature matching unit 46 compares the acoustic features in the acoustic feature database 48 with the acoustic features of the current block received from the acoustic feature extraction unit 42, and calculates the similarity of each phoneme as a numerical value or score. At this time, the feature matching unit 46 also refers to the acoustic features of the block preceding the current block in time, stored in the acoustic feature accumulation unit 44, and estimates which phoneme the acoustic feature of the current block following the previous block has a higher probability of belonging to, and reflects the estimation result in the similarity of each phoneme. The feature matching unit 46 can also refer to the acoustic features of the blocks two blocks preceding the current block in time, and estimate which phoneme the acoustic feature of the current block following the acoustic features of the blocks two blocks preceding and the previous block has a higher probability of belonging to. This processing can utilize known diphonetic or triphonetic techniques.

[0062] Furthermore, since the current block does indeed contain speech, the feature matching unit 46 can estimate phoneme candidates with high accuracy. Therefore, the feature matching unit 46 receives VAD information from the phonation interval determination unit 14, and when it is a phonation interval, calculates the similarity of each phoneme. For example, if the current block contains vowels and consonants, it can be assumed that one of the vowels has a high similarity, and one of the consonants also has a high similarity. On the other hand, when the VAD information indicates a non-phonation interval, the feature matching unit 46 outputs information indicating background noise instead of calculating the similarity of each phoneme. Furthermore, when the VAD information contains the aforementioned voiced / unvoiced sound determination results, the feature matching unit 46 can weight the referenced acoustic features based on the VAD information and calculate the similarity of each phoneme. The feature matching unit 46 sends the calculated similarity information of each phoneme to the phoneme candidate extraction unit 50.

[0063] The phoneme candidate extraction unit 50 narrows down the range of phoneme candidates based on the similarity of each received phoneme. When the audio signal of the vocal range contains both vowels and consonants, it is conceivable that the narrowed phoneme candidate range includes both vowel and consonant candidates. When the audio signal of the vocal range contains only vowels, it is conceivable that the narrowed phoneme candidate range includes vowel candidates and background noise information. The phoneme candidate range for each vowel and consonant is not limited to one, but can also include multiple candidates. The phoneme candidate extraction unit 50 extracts the probability information of the phonemes along with the phoneme candidates and sends the extracted phoneme candidates to the phoneme candidate accumulation unit 52 and the statistical probability model unit 54. When information indicating that it is background noise is received, the phoneme candidate extraction unit 50 sends, for example, information indicating that the probability of background noise is 100%.

[0064] Since speech is a temporally continuous signal sequence, there is continuity between adjacent blocks, and there are discrepancies in the probability of consecutive occurrences of speech words (phonemes). To detect this information, phoneme estimation methods based on acoustic models, used in speech recognition, can be employed. However, due to time delay limitations, processing is only performed by referring to past acoustic information that has already ended. The phoneme candidate accumulation unit 52 stores phoneme candidates for each block of past audio signals, updating the information while maintaining a referential state.

[0065] When the VAD information indicates a vocalized region, the statistical probability model unit 54 selects the phoneme with the highest probability from the received phoneme candidates of the current block. The statistical probability model unit 54, as needed, refers to the phoneme candidate accumulation unit 52 and narrows down the final phoneme candidates by correction based on occurrence probability, such as forward prediction weighting based on a Hidden Markov Model, or weighting of previously extracted phoneme information considering speech continuity. The statistical probability model unit 54 sends the phoneme information of the selected phoneme candidate to the phoneme estimation unit 56. When the VAD information indicates a non-vocalized region, the statistical probability model unit 54 outputs background noise information as phoneme information.

[0066] The phoneme estimation unit 56 finally determines the phoneme candidates narrowed down by the statistical probability model unit 54 and outputs them as the phoneme information for the current block. The phoneme information can also be called short-time phoneme information.

[0067] In the above description, the short-time phoneme analysis unit 16 acquires phoneme information according to a predetermined algorithm, but a deep learning model, i.e., a deep neural network, can also be used to acquire phoneme information. An example of the configuration of the short-time phoneme analysis unit 16 using a deep learning model to acquire phoneme information will be referred to... Figure 7 , Figure 8 Please provide an explanation.

[0068] A model is pre-trained using a large amount of speech data, such as a known speech corpus, to take acoustic features as input and output a phoneme occurrence probability vector. The short-time phoneme analysis unit 16 uses the trained model to obtain the phoneme occurrence probability vector based on the input acoustic features. The trained model can also be called an acoustic model.

[0069] Figure 7 This diagram illustrates another example of the functional configuration of the short-time phoneme analysis unit 16. The short-time phoneme analysis unit 16 includes a frequency conversion unit 60, an acoustic feature extraction unit 62, a trained parameter storage unit 64, a neural network 66, and a phoneme candidate output unit 68.

[0070] As described above, multiple blocks of the audio signal are input to the short-time phoneme analysis unit 16 in chronological order. The next block in time is input at time intervals equal to the unit time width. The frequency conversion unit 60 acquires the audio signal of the current block, converts the time information into frequency information, and sends the frequency information to the acoustic feature extraction unit 62. Based on the received frequency information of the audio signal of the current block, the acoustic feature extraction unit 62 extracts acoustic features of the audio signal of the current block. The acoustic features are multi-dimensional vectors. The acoustic feature extraction unit 62 sends the acoustic features to the neural network 66.

[0071] The trained parameter storage unit 64 stores the trained parameters trained by the machine learning device 70, which will be described later. The neural network 66 is a neural network capable of processing time-series data, such as an RNN (Recurrent Neural Network). Since neural networks can use known structures, detailed descriptions are omitted. The neural network 66 functions as a trained model by applying the trained parameters obtained from the trained parameter storage unit 64. As described below, the trained model performs machine learning on each of multiple blocks in the time-series order of the training audio signal, based on the acoustic features of the target block and the phoneme information of the target block.

[0072] When the VAD information received from the vocalization interval determination unit 14 indicates a vocalization interval, the neural network 66 receives the acoustic feature vector of the current block at its input layer and outputs the phoneme occurrence probability vector of the current block to the phoneme candidate output unit 68 from its output layer. The phoneme occurrence probability vector includes the occurrence probabilities of multiple vowels, multiple consonant types, and background noise. The occurrence probability vector may also include the occurrence probabilities of multiple vowels and multiple consonants. When the VAD information indicates a non-vocalization interval, the neural network 66 outputs a phoneme occurrence probability vector with the background noise probability set to, for example, 100%, regardless of the input acoustic feature vector.

[0073] The neural network 66 outputs a phoneme occurrence probability vector based not only on the acoustic features of the current block of the audio signal, but also on the acoustic features of multiple blocks preceding the current block that it internally retains. Therefore, it can obtain a high-precision phoneme occurrence probability vector that also reflects the past phoneme arrangement.

[0074] The phoneme candidate output unit 68 determines the phoneme information of the current block based on the received phoneme occurrence probability vector and outputs the determined phoneme information. For example, the phoneme candidate output unit 68 outputs the vowel with the highest probability and the consonant type with the highest probability or background noise as phoneme information in the phoneme occurrence probability vector. If the probability of background noise is 100%, the phoneme candidate output unit 68 only outputs the background noise information as phoneme information.

[0075] Figure 8 This diagram illustrates the functional configuration of the machine learning device 70. The machine learning device 70 includes a control unit 72, a first frequency conversion unit 74, a first acoustic feature extraction unit 76, a reference phoneme output unit 78, an equivalent filter 80, a second frequency conversion unit 82, a second acoustic feature extraction unit 84, a neural network 86, a phoneme candidate output unit 88, a comparison unit 90, and a trained parameter storage unit 92.

[0076] The machine learning device 70 performs machine learning on the neural network 86, which serves as the learning model, for each of the multiple blocks in the time-series sequence of the training audio signal, with the acoustic features of the target block as input and the phoneme information of the target block as the ground truth, through deep learning.

[0077] The control unit 72 synchronizes the processing of sending reference phonemes to the comparison unit 90 via the first frequency conversion unit 74, the first acoustic feature extraction unit 76, and the reference phoneme output unit 78 with the processing of outputting determined phoneme candidates to the comparison unit 90 via the equivalent filter 80, the second frequency conversion unit 82, the second acoustic feature extraction unit 84, the neural network 86, and the phoneme candidate output unit 88, and adjusts the processing target block of the speech corpus (teacher dataset of training audio signals and true phoneme information).

[0078] The first frequency conversion unit 74 and the equivalent filter 80, each under the control of the control unit 72, acquire multiple blocks of training audio signals from an external teacher dataset storage unit (not shown) storing teacher datasets such as speech corpora, one by one in chronological order. The unit time width of the block is the same as the unit time width in the pickup device 100. Since it is not necessary to complete the learning of a block within the unit time width of the block, the control unit 72 can change the processing target block to the next block at time intervals longer than the unit time width. The training audio signal, for example, contains speech information of about a few seconds in length.

[0079] The first frequency conversion unit 74, under the control of the control unit 72, acquires the audio signal of the processing target block, converts the time information into frequency information, and sends the frequency information to the first acoustic feature extraction unit 76. Based on the received frequency information, the first acoustic feature extraction unit 76 extracts the acoustic features of the audio signal of the processing target block and sends the extracted acoustic features to the reference phoneme output unit 78.

[0080] The reference phoneme output unit 78 performs speech recognition on acoustic features using known techniques and sends the obtained phonemes as reference phonemes to the comparison unit 90. The reference phonemes include at least one of vowels and consonants. The reference phonemes may also contain information representing background noise. The control unit 72 changes the processing block to the next block at a predetermined time interval longer than the time required for speech recognition by the reference phoneme output unit 78. Therefore, the reference phoneme output unit 78 can perform speech recognition using known techniques with processing times longer than the unit time width of the block. The reference phoneme output unit 78 can perform speech recognition using lexical analysis combining acoustic models, language models, and vocalization models. Since high-precision speech recognition techniques with relatively long processing times can be utilized, training accuracy can be improved.

[0081] The equivalent filter 80, under the control of the control unit 72, acquires the audio signal of the target block and band-limits the acquired audio signal to make it equivalent to the audio signal picked up by the bone conduction microphone. The equivalent filter 80 then sends the band-limited audio signal to the second frequency conversion unit 82.

[0082] The second frequency conversion unit 82 receives the bandwidth-limited audio signal, converts the time information into frequency information, and sends the frequency information to the second acoustic feature extraction unit 84. The second frequency conversion unit 82 has the same configuration as the first frequency conversion unit 74. Based on the received frequency information, the second acoustic feature extraction unit 84 extracts acoustic features of the audio signal of the processing target block and sends the extracted acoustic features as training data to the neural network 86. The second acoustic feature extraction unit 84 has the same configuration as the first acoustic feature extraction unit 76.

[0083] The neural network 86 takes the acoustic feature vector of the audio signal of the processing object block as input and outputs the phoneme occurrence probability vector to the phoneme candidate output unit 88. The audio signal of the processing object block is equivalent to the audio signal picked up by a bone conduction microphone. The neural network 86 includes... Figure 7 The neural network 66 has the same structure. The phoneme candidate output unit 88 determines phoneme candidates based on the received phoneme occurrence probability vector and outputs the determined phoneme candidates to the comparison unit 90. The phoneme candidate determination process is the same as... Figure 7 The phoneme information determination process performed by the phoneme candidate output unit 68 is the same.

[0084] The comparison unit 90 compares the phoneme candidates from the phoneme candidate output unit 88 with the reference phonemes from the reference phoneme output unit 78, and feeds back the consistency as a comparison result to the neural network 86.

[0085] The neural network 86 adjusts its internal parameters based on the comparison results from the comparison unit 90, using methods such as backpropagation of errors, to make the phoneme candidates of the current block closer to the reference phoneme. The neural network 86 stores the adjusted internal parameters in the trained parameter storage unit 92.

[0086] The machine learning device 70 uses various speech corpora from the teacher dataset as teacher data to repeatedly train the internal parameters of the neural network 86, thereby improving the inference accuracy of the neural network 86. The internal parameters stored in the trained parameter storage unit 92 are updated each time training is performed. In this way, a trained learning model can be obtained for narrowband audio signals corresponding to audio signals picked up by the bone conduction microphone.

[0087] Furthermore, when phoneme candidates are pre-assigned to the speech corpus dataset, the first frequency conversion unit 74, the first acoustic feature extraction unit 76, and the reference phoneme output unit 78 can be omitted. In this case, an acquisition unit (not shown) can be set up under the control of the control unit 72. This acquisition unit acquires phoneme candidates corresponding to the processing target block from the speech corpus dataset. The acquisition unit sends the acquired phoneme candidates as reference phonemes for the true values ​​to the comparison unit 90.

[0088] Furthermore, if the reference phoneme output unit 78 can complete speech recognition in a shorter time than the unit time width of the block, and can complete the learning of one block within the unit time width of the block, the control unit 72 may not control the timing of turning the processing target block into the next block, but instead input blocks continuously according to each unit time width without setting a time interval between temporally adjacent blocks. In this case, a delay unit that delays the audio signal of the processing target block can be inserted before the first frequency conversion unit 74. The delay amount of the delay unit can be adjusted so that the timing of the audio signal of the processing target block after being delayed by the delay unit input to the first frequency conversion unit 74 is the same as the timing of the audio signal of the processing target block after being band-limited by the equivalent filter 80 input to the second frequency conversion unit 82.

[0089] Next, the band extension methods for vowels (harmonic structure components) and consonants (residual components) using short-time phoneme information are explained. Figure 9 (a) is a diagram schematically showing the frequency characteristics of the output signal of the harmonic structure component extraction unit 18. Figure 9 As shown in (a), when the input audio signal is picked up by a bone conduction microphone, the high-frequency components of the audio signal are significantly attenuated due to the transmission characteristics of the human body.

[0090] Figure 9 (b) shows the overtones generated by the high-frequency overtone generation superposition unit 22 superimposed on the base. Figure 9 A graph showing the frequency characteristics of the audio signal on signal (a). The high-frequency overtone generation superposition unit 22 superimposes overtone components at the same level as the highest unattenuated frequency level in the underlying signal. Since the acoustic characteristics of the sound pressure level of the frequency components in the extended band are unknown, they are all superimposed at the same level at that point in time.

[0091] Here, if the type of vowel is determined by the phoneme information calculated by the short-time phoneme analysis unit 16, the acoustic characteristic quantity of each phoneme, namely the frequency distribution of the higher-order formants F3 and F4, is determined. Figure 10 (a) shows the superposition of higher-order resonant peaks as the target of frequency distribution correction onto... Figure 9The frequency response diagram of (b) is shown. The higher-order formant correction unit 24 corrects the level of the extended high-frequency overtone component according to the higher-order formant that becomes the correction target, thereby generating a signal as shown in the diagram. Figure 10 The original audio signal, as shown in (b), has had its acoustic characteristics restored. The levels of higher-order formants are adjusted to match the levels of the unattenuated frequency components in the underlying signal. Figure 10 (b) is a graph showing the frequency characteristics of the corrected audio signal. As a result, the dull sound is improved in terms of listenability due to the presence of natural frequency bands.

[0092] Next, the method for band extension of consonant components will be explained. Figure 11 (a) is a graph showing the time waveform of the audio signal when the sound “shio” is pronounced. In the vowels “i” and “o”, we can see the low-frequency components that represent the fundamental frequency of the pitch, but the typical consonant component “s” is a signal similar to white noise, with stronger high-frequency components. Figure 11 (b) is a graph showing the frequency characteristics of the consonant "s". In Japanese, "s" belongs to the "sa" row and is classified as a fricative (voiceless) consonant. Fricatives are characterized by a smooth increase in level as the frequency rises in the high-frequency region. Figure 12 (a) is a graph showing the frequency characteristics of the consonant "h" as a fricative sound. Figure 12 (b) is a graph showing the frequency characteristics of the consonant "k" classified as a voiceless plosive. Consonants classified according to their formation methods have different acoustic characteristics and high-frequency waveform envelopes, but if the consonant type is the same, the waveform envelopes tend to be similar.

[0093] For consonants, the bandwidth is extended based on the high-frequency waveform envelope corresponding to the consonant type in the residual component of the output signal of the residual component calculation unit 20. The consonant type is obtained from the short-time phoneme analysis unit 16. For the high-frequency attenuated residual component, the high-frequency noise generation and superposition unit 26 first generates white noise with no level information in the extended bandwidth and superimposes it onto the residual component signal, which is the original signal. Then, based on the short-time phoneme information, according to... Figure 13 The phoneme classification shown is based on, for example, Figure 11 of (b) Figure 12 of (a) Figure 12 The frequency characteristic type with high-frequency waveform envelope shown in (b) is adjusted by the high-frequency envelope correction unit 28.

[0094] Figure 13 This is a diagram showing the correspondence between consonant types and waveform envelopes. Figure 13The example shown illustrates a classification that can be further subdivided into consonant categories. Furthermore, for each consonant category, a waveform envelope type can be prepared for each phoneme. That is, for example, for fricatives, waveform envelopes can be set separately for the phonemes "s" and "h". Moreover, waveform envelopes are not limited to... Figure 11 of (b) Figure 12 of (a) Figure 12 The type shown in (b).

[0095] Explain the operation of the pickup device 100 constructed as described above. Figure 14 This is a flowchart illustrating the operation steps of the pickup device 100. (The following is omitted.) Figure 1 The processing of the AD converter 12 and DA converter 32 is described, illustrating the action steps related to the digitized audio signal.

[0096] First, the sound range determination unit 14, the short-time phoneme analysis unit 16, the harmonic structure component extraction unit 18, and the residual component calculation unit 20 each acquire the narrowband audio signal segmented by the AD converter 12 (S10). The harmonic structure component extraction unit 18 uses pitch detection and linear prediction analysis to extract the harmonic structure component, which is the main component of the sound, from the acquired audio signal (S12). The residual component calculation unit 20 calculates the residual component by subtracting the harmonic structure component extracted by the harmonic structure component extraction unit 18 from the audio signal acquired in S10 (S14).

[0097] The vocal range determination unit 14 determines whether a vocal range exists based on the audio signal segments in S10 (S16). If it is a vocal range (S16 "Yes"), the short-time phoneme analysis unit 16 estimates the vowel based on the audio signal segments in S10 through short-time phoneme analysis (S18). When the short-time phoneme analysis unit 16 determines that a vowel is included (S19 "Yes"), the high-frequency overtone generation and superposition unit 22 generates attenuated overtones in the high-frequency band based on the harmonic structure components extracted in S12 (S20). The higher-order formant correction unit 24 corrects the frequency characteristics of the restored high-frequency overtones according to the frequency characteristics of the higher-order formants that differ for each type of vowel (S22), and superimposes the high-frequency overtone components with corrected frequency characteristics from S22 onto the harmonic structure components extracted in S12 (S36), thus expanding the frequency band. When the short-time phoneme analysis unit 16 determines that a vowel is not included (S19 "No"), the process proceeds to S24.

[0098] Next, the short-time phoneme analysis unit 16 estimates the consonant type based on the segmented audio signal in S10 through short-time phoneme analysis (S24). When the short-time phoneme analysis unit 16 determines that a consonant is included ("Yes" in S25), the high-frequency noise generation and superposition unit 26 generates a noise component in the attenuated high-frequency band of the residual component obtained in S14 (S26), and the high-frequency envelope correction unit 28 corrects the frequency characteristics of the restored high-frequency noise according to the waveform envelope that is different according to each consonant type (S28), and superimposes the high-frequency noise with corrected frequency characteristics in S28 onto the residual component obtained in S14 (S38). When the short-time phoneme analysis unit 16 determines that no consonant is included ("No" in S25), the process proceeds to S30.

[0099] On the other hand, when it is determined to be a non-voiceable region in S16 ("No" in S16), the processing also proceeds to S30. The short-time phoneme analysis unit 16 estimates the audio signal as a background noise signal (S30), and the high-frequency noise generation and superposition unit 26 generates a noise component in the high-frequency band that is attenuated in the residual component (S32). It estimates the high-frequency background noise level that is consistent with the sound pressure level of the narrowband audio signal, which is the background noise signal. Based on the estimated high-frequency background noise level, it corrects the high-frequency noise level (S34), and superimposes the high-frequency noise whose level has been corrected in S34 onto the residual component obtained in S14 (S35). Thus, different processing is performed on the residual component in the case of consonants and in the case of non-voiceable regions.

[0100] Even in the non-sound-producing range, there are certain acoustic components. Therefore, by separating the audio signal into harmonic structural components and residual components, high-frequency components are generated and superimposed on the bandwidth-limited audio signal, regardless of whether it is a sound-producing or non-sound-producing range. This ensures the natural bandwidth of the original audio signal for any audio signal, and can be expected to improve the audio quality.

[0101] Next, the adder 30 adds the band-extended harmonic structure component and the residual component, and outputs the band-extended audio signal of the current block (S40). Specifically, in the sound-producing interval, when the audio signal contains only vowels, the adder 30 adds the harmonic structure component superimposed with high-frequency overtones in S36 to the residual component superimposed with high-frequency noise in S35. In the sound-producing interval, when the audio signal contains only consonants, the adder 30 adds the harmonic structure component extracted in S12 to the residual component superimposed with high-frequency noise in S38. In the sound-producing interval, when the audio signal contains both vowels and consonants, the adder 30 adds the harmonic structure component superimposed with high-frequency overtones in S36 to the residual component superimposed with high-frequency noise in S38. In the non-sound-producing interval, the adder 30 adds the harmonic structure component extracted in S12 to the residual component superimposed with high-frequency noise in S35.

[0102] Next, when the audio signal output action ends (S42 "Yes"), the processing ends. When the audio signal output action does not end (S42 "No"), the processing returns to S10. In S10, the audio signal of the next block of the time series is acquired.

[0103] Alternatively, the process of S36 can be inserted between S20 and S22, with correction processing applied to the desired frequency band in S22. Similarly, the process of S38 can be inserted between S26 and S28, with correction processing applied to the desired frequency band in S28. The process of S35 can also be inserted between S32 and S34. In this case, Figure 14 The processing in S36 and S38 can be omitted.

[0104] According to the implementation method, phoneme-related information contained in a bandwidth-limited audio signal is acquired, high-frequency components of the audio signal are generated based on the phoneme-related information, and the bandwidth of the audio signal is extended by the high-frequency components. Therefore, the high-frequency information of the original sound can be estimated and restored with high accuracy, thereby improving sound quality. At this time, by using a trained model based on deep learning to acquire phoneme-related information, high-precision phoneme information can be obtained in a short time. Therefore, higher-precision high-frequency components can be restored in a short time, achieving natural-sounding communication, etc.

[0105] Furthermore, when the phoneme-related information includes vowel information, a high-frequency component containing overtones is generated. This overtone component reproduces the high-order formant characteristics corresponding to the vowel, thus improving the sound quality of the vowel components in the audio signal. Additionally, when the phoneme-related information includes consonant information, a high-frequency component containing noise is generated. This noise component has frequency characteristics corresponding to the consonant and its magnitude is adjusted based on the sound pressure level of the audio signal, thus improving the sound quality of the consonant components in the audio signal. Further, when the phoneme-related information includes background noise information, a high-frequency component containing noise is generated. This noise component has frequency characteristics corresponding to the background noise and its magnitude is adjusted based on the sound pressure level of the audio signal, thus improving the sound quality of the background noise components in the audio signal.

[0106] The present invention has been described above based on embodiments. These embodiments are exemplary, and those skilled in the art will understand that various modifications can be made to the combination of the constituent elements and processing procedures, and such modifications are also within the scope of the present invention.

[0107] Industrial availability This invention can be used to extend the bandwidth of narrowband audio signals.

[0108] Symbol Explanation 10...Microphone, 12...AD Converter, 14...Sound Range Determination Unit, 16...Short-Time Phoneme Analysis Unit, 18...Harmonic Structure Component Extraction Unit, 20...Residual Component Calculation Unit, 22...High-Frequency Overtone Generation and Superposition Unit, 24...High-Order Formant Correction Unit, 26...High-Frequency Noise Generation and Superposition Unit, 28...High-Frequency Envelope Correction Unit, 30...Addition Unit, 32...DA Converter, 34...Bandwidth Extension Unit, 40...Frequency Conversion Unit, 42...Acoustic Feature Extraction Unit, 44...Acoustic Feature Accumulation Unit, 46...Feature Matching Unit, 48...Acoustic Feature Database, 50...Phonetic Element Candidate Extraction Unit, 52...Phonetic Element Candidate Accumulation unit, 54...Statistical probability model unit, 56...Phonetic estimation unit, 60...Frequency conversion unit, 62...Acoustic feature extraction unit, 64...Trained parameter storage unit, 66...Neural network, 68...Phonetic candidate output unit, 70...Machine learning device, 72...Control unit, 74...First frequency conversion unit, 76...First acoustic feature extraction unit, 78...Reference phoneme output unit, 80...Equivalent filter, 82...Second frequency conversion unit, 84...Second acoustic feature extraction unit, 86...Neural network, 88...Phonetic candidate output unit, 90...Comparison unit, 92...Trained parameter storage unit, 100...Pickup device.

Claims

1. A sound pickup device, characterized in that, include: The phoneme information acquisition unit acquires phoneme-related information contained in the audio signal; as well as The bandwidth extension unit generates high-frequency components of the audio signal based on information related to the phonemes, and extends the bandwidth of the audio signal using the high-frequency components.

2. The pickup device according to claim 1, characterized in that, The phoneme information acquisition unit uses a trained model to acquire information related to the phonemes contained in the audio signal.

3. The pickup device according to claim 1 or 2, characterized in that, When the information associated with the phoneme includes vowel information, the bandwidth extension generates the high-frequency component containing overtone components, which reproduce the characteristics of the higher-order formants corresponding to the vowel.

4. The pickup device according to any one of claims 1 to 3, characterized in that, When the information associated with the phoneme includes information about consonants, the bandwidth extension generates the high-frequency component that includes a noise component, the noise component having frequency characteristics corresponding to the consonant, and the magnitude is adjusted based on the sound pressure level of the audio signal.

5. A sound pickup method, characterized in that, Computer execution steps: To obtain phoneme-related information contained in the audio signal; and The high-frequency component of the audio signal is generated based on information related to the phonemes, and the frequency band of the audio signal is extended by the high-frequency component.

6. A program, characterized in that, To make the computer perform: To obtain phoneme-related information contained in the audio signal; and The high-frequency component of the audio signal is generated based on information related to the phonemes, and the frequency band of the audio signal is extended by the high-frequency component.

Images