Data augmentation method using fundamental freuqency obtained by dj transform

Abstract

A data augmentation method according to an embodiment of the present invention comprises, wherein each step is performed by a computer processor and is intended to generate training data for machine learning for speech recognition or speaker recognition, a step of extracting at least one fundamental frequency of speech data; and a step of generating harmonics of the fundamental frequency as training data.

Description

Technology Field

[0001] The present invention relates to a data augmentation method for generating training data for machine learning in the field of speech recognition or speaker recognition. Background Technology

[0003] Speech recognition is a technology in which a computer interprets spoken language and converts its content into text data, while speaker recognition refers to a technology that distinguishes speakers by analyzing voiceprints, which can be called 'voice fingerprints,' and is generally divided into speaker identification and speaker verification. Speaker identification is a technology that finds the speaker with the voice most similar to the input utterance from registered voiceprint data, and speaker verification is a technology that determines whether the voiceprint in the input utterance and the registered voiceprint belong to the same person. Recently, as the performance of speech and speaker recognition technologies has been improved through deep learning models, a type of machine learning, interest in technologies for more successful machine learning is deepening.

[0004] In general, data augmentation techniques are applied in machine learning to increase the volume of training datasets and enhance their diversity. Data augmentation in machine learning refers to methods for increasing the size of a training dataset, utilizing techniques such as adding modified copies of the original training dataset or adding newly synthesized data derived from the original dataset.

[0005] A widely used data augmentation technique in the speech domain is a method that utilizes the MUSAN dataset published in 2018 to add noise to audio and apply reverberation using RIR (Room Impulse Response). However, since this data augmentation method only augments data based on four simple patterns—reverberation, chatter, music, and noise—it shows limitations in performance improvement when noise from a new environment is added. Prior art literature

[0007] Registered Patent No. 10-2158743 The problem to be solved

[0008] Embodiments of the present invention aim to propose a new data augmentation method for a machine learning system robust to new environments. means of solving the problem

[0010] A data augmentation method according to an embodiment of the present invention is a data augmentation method for generating training data for machine learning for speech recognition or speaker recognition, wherein each step is performed by a computer processor.

[0011] (a) a step of extracting at least one fundamental frequency of voice data; and

[0012] (b) includes the step of generating harmonics of the fundamental frequency as training data.

[0013] The above step (a) is,

[0014] (a-1) For the input of voice data, the vibrational motion of a plurality of springs having different natural frequencies is modeled respectively to calculate the estimated pure tone amplitude and phase according to the natural frequencies, and a DJ transform spectrogram is generated that represents the estimated pure tone amplitude according to a frequency corresponding to the natural frequency of each of the plurality of springs and a plurality of time points; and

[0015] (a-2) The method may include the step of extracting the fundamental frequency based on the moving average of the estimated pure tone amplitude or the moving standard deviation of the estimated pure tone amplitude for each of the natural frequencies of the DJ transform spectrogram.

[0016] The above (a-1) step is,

[0017] A step of estimating a steady-state expected amplitude, which is the convergence value of the amplitude of each of the plurality of springs in a steady state, based on the amplitudes at two time points of a natural one-period interval of each of the plurality of springs; and

[0018] The method may include a step of calculating the estimated pure tone amplitude based on the pure tone predicted amplitude, which is the amplitude of the input sound estimated based on the above steady-state predicted amplitude.

[0019] The above estimated pure tone amplitude may be the above steady-state expected amplitude or the above pure tone predicted amplitude.

[0020] The above (a-2) step is,

[0021] (a-21) A step of calculating the fundamental frequency fit based on the moving average of the estimated pure tone amplitude or the moving standard deviation of the estimated pure tone amplitude in the above DJ transform spectrogram; and

[0022] (a-22) The method may include the step of calculating the maximum value of the fundamental frequency suitability at each of the plurality of time points and extracting the fundamental frequency based on the calculated maximum value of the fundamental frequency suitability.

[0023] The above fundamental frequency fit may be proportional to the moving average of the estimated pure tone amplitude or inversely proportional to the moving standard deviation of the estimated pure tone amplitude.

[0024] The above (a-22) step is,

[0025] A grayscale spectrogram generation step of, at each of the plurality of time points, extracting the top N (N is an integer greater than or equal to 2) of the fundamental frequency suitability, setting the value corresponding to the natural frequency of the N to "1" and the remaining values ​​to "0";

[0026] A step for generating an average black-and-white spectrogram, which calculates the average of the black-and-white spectrogram for an area of ​​equal size including each point of the black-and-white spectrogram; and

[0027] It may include a step of extracting the maximum value of the average grayscale spectrogram at each of the plurality of time points.

[0028] The above step (a-22) may further include the step of extracting a candidate fundamental frequency based on the difference in natural frequencies corresponding to adjacent maximum values ​​of the average grayscale spectrogram and the lowest frequency among the natural frequencies corresponding to the maximum values ​​of the average grayscale spectrogram at each of the plurality of time points.

[0029] The above step (a-22) may further include the step of calculating the time average of a black-and-white spectrogram-based fundamental frequency set for a predetermined time interval including adjacent time points, setting a first frequency set including frequencies near values ​​obtained by multiplying the time average by positive integers less than or equal to a predetermined value, and setting the frequency with the largest average black-and-white spectrogram among the frequencies belonging to the first frequency set as the black-and-white spectrogram-based fundamental frequency by dividing the value obtained by the positive integer multiplied when setting the first frequency set.

[0030] The above step (a-22) may further include the step of setting the candidate fundamental frequency of the time point with the smallest shifting variance of the difference between the candidate fundamental frequencies of adjacent time points among the candidate fundamental frequencies for the plurality of time points as the black-and-white spectrogram-based fundamental frequency of the time point with the smallest shifting variance.

[0031] The above step (a-22) may further include the step of, at each of the plurality of time points, setting a second frequency set comprising values ​​obtained by multiplying a black-and-white spectrogram-based fundamental frequency by positive integers less than or equal to a predetermined value, and setting the frequency with the greatest fundamental frequency suitability among the frequencies belonging to the second frequency set as the final fundamental frequency by dividing the value obtained by the positive integer multiplied when setting the second frequency set.

[0032] In step (b) above, the harmonic of the fundamental frequency may have the expected amplitude and phase of the stable state of the spring having a natural frequency corresponding to the harmonic of the fundamental frequency as the amplitude and phase of the harmonic of the fundamental frequency.

[0033] The harmonics of the above fundamental frequency may be contained in 50% to 75% of the total training data.

[0034] A computer-readable recording medium according to an embodiment of the present invention is recorded such that each step of the data augmentation method can be executed on a computer. Effects of the invention

[0036] According to an embodiment of the present invention, since training data is generated using harmonics of the fundamental frequency obtained by DJ transformation, machine learning performance can be improved even when noise from a new environment is added. Brief explanation of the drawing

[0038] FIG. 1 is a diagram illustrating how a data augmentation method according to an embodiment of the present invention is utilized in machine learning. FIG. 2 is a flowchart showing each step of a data augmentation method according to an embodiment of the present invention. Figure 3 is a flowchart showing an example of the fundamental frequency extraction step of Figure 2. Figure 4 is a flowchart showing an example of the fundamental frequency extraction step of Figure 3. Figure 5 is a flowchart showing an example of the fundamental frequency extraction step of Figure 4. Figure 6 is a diagram showing an experimental example of the present invention when a sound of harmonics that changes over time is input. Figure 7 is a table showing experimental results for an embodiment of the present invention. FIG. 8 is a structural diagram of a data augmentation system according to an embodiment of the present invention. Specific details for implementing the invention

[0039] The advantages and features of the present invention and the methods for achieving them will become clear by referring to the embodiments described below in detail together with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below but may be implemented in various different forms. These embodiments are provided merely to ensure that the disclosure of the present invention is complete and to fully inform those skilled in the art of the scope of the invention, and the present invention is defined only by the scope of the claims. Throughout the specification, the same reference numerals refer to the same components.

[0040] The terms used herein are for describing the embodiments and are not intended to limit the invention. In this specification, the singular form includes the plural form unless specifically stated otherwise in the text. As used herein, "comprises" and / or "comprising" do not exclude the presence or addition of one or more other components, steps, actions, and / or elements to the mentioned components, steps, actions, and / or elements.

[0041] Unless otherwise defined, all terms used in this specification (including technical and scientific terms) may be used in a meaning that is commonly understood by those skilled in the art to which the present invention belongs.

[0042] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

[0044] FIG. 1 is a diagram illustrating how a data augmentation method according to an embodiment of the present invention is utilized in machine learning.

[0045] Referring to FIG. 1, a dataset must be input to generate a machine learning algorithm for speaker recognition or speaker verification. The training dataset of the machine learning system of FIG. 1 is an original training dataset according to the prior art, and may be, for example, a dataset augmented by MUSAN and RIP (Room Impulse Response).

[0046] An embodiment of the present invention generates (augments) data consisting of harmonics of the fundamental frequency extracted through DJ transform of the original training dataset as a training dataset. Then, the original training dataset and the augmented data according to the embodiment of the present invention are input into a machine learning algorithm. At this time, each of the original training dataset and the augmented data according to the embodiment of the present invention may include speech data (e.g., WAV) and a label (a value indicating the speaker in the case of speaker recognition, or a value indicating whether the speaker is correct in the case of speaker verification).

[0047] Although FIG. 1 illustrates that both the original training dataset and the augmented data according to an embodiment of the present invention are input into the machine learning algorithm, only the augmented data according to an embodiment of the present invention may be input.

[0049] FIG. 2 is a flowchart illustrating each step of a data augmentation method according to an embodiment of the present invention. Each step is performed by a computer processor.

[0050] First, at least one fundamental frequency of the voice data is extracted (S10). The extraction of the fundamental frequency utilizes a DJ transform modeling the vibrational motion of multiple springs, and details will be described later.

[0051] Next, the harmonics of the above fundamental frequency are generated as training data (S20).

[0052] The generated training data is used as training data for machine learning for speech recognition or speaker recognition.

[0054] FIG. 3 is a flowchart showing an example of the fundamental frequency extraction step (S10) of FIG. 2.

[0055] Referring to FIG. 3, a fundamental frequency extraction method according to an embodiment of the present invention comprises: a step (S100) of modeling the vibrational motion of a plurality of springs having different natural frequencies for the input of voice data to calculate an estimated pure tone amplitude and phase according to the natural frequencies, and generating a DJ transform spectrogram representing the estimated pure tone amplitude according to the natural frequencies of each of the plurality of springs and the plurality of time points; and a step (S200) of calculating the fundamental frequency based on the moving average of the estimated pure tone amplitude or the moving standard deviation of the estimated pure tone amplitude for each of the natural frequencies of the DJ transform spectrogram.

[0056] The DJ transformation models the vibrational motion of multiple springs having different natural frequencies. Its purpose is to accurately represent the characteristics of actual sound by simulating the movement of hair cells in the cochlea of ​​the ear through the vibrational motion of the springs. Since frequency can be easily converted into vibration frequency or angular velocity, these terms are used interchangeably in this specification.

[0057] Multiple springs are set to have different natural frequencies. The natural frequencies of the multiple springs may have a predetermined frequency interval, such as 1 Hz, 2 Hz, or 10 Hz, within a frequency range corresponding to sound, for example, the audible frequency band of 20 Hz to 20 kHz.

[0058] spring constant k i person spring s i The displacement x of an object of mass M fixed at one end relative to its parallel position i The equation of motion for the external force F(t) at (t) is as follows.

[0059] (Equation 1)

[0060] Here, ωoi is the natural resonance angular velocity And, when the damping rate is ζ, Γ i is the damping constant per unit mass The model used M=1 and ζ=0.001, but these values ​​may be changed in the future to improve performance.

[0061] angular velocity is ω ext and F of constant amplitude ext external sound Let's assume that is input. In this case, x is the solution to the equation of motion of a spring with an initial condition of rest. i (t) is expressed as follows.

[0062]

[0063] (Equation 2)

[0064] Here However, in the model, if you use a very small value for ζ, for example, around 0.001 This becomes. And and It is as follows.

[0065] (Equation 3)

[0066] (Equation 4)

[0067] Angular velocity ω of the external force ext and the angular velocity ω of the spring's natural frequency 0i when it matches and It becomes as follows.

[0068] (Equation 5)

[0069] (Equation 6)

[0070] angular velocity ω of external sound ext When , the angular velocity ω of the natural frequency of the spring used in DJ conversion 0i go A spring satisfying the condition is called a spring with the resonance condition. At this time, Since it can be said that, the spring displacement x i (t) is expressed as follows.

[0071] (Equation 7)

[0072] In Equation 7 value when Equation 5 value when Since it is almost the same, it is used as the same value in the expression expansion.

[0073] τ n second Let's define . Equation 7 as t=τ n At the point in time, i.e., displacement x in the first period i If observed at the time when (t) is maximum, x i (t=τ n The value of ) is simply expressed as follows.

[0074] (Equation 8)

[0075] According to Equation 8, after sufficient time has elapsed (n→∞), the displacement x in the stabilized state i (t=τ n ) Is It converges to the value of.

[0076] Displacement x of the stabilized state after sufficient time has passed, at the point before convergence after external sound input begins to enter. i convergence value of (t) It can be obtained. The process is as follows.

[0077] First, transform Equation 8 as follows.

[0078] (Essence 9)

[0079] If we change the value of n in Equation 9 to n+1, the equation changes as follows.

[0080] (Equation 10)

[0081] If we divide Equation 9 into Equation 10 and rearrange the sides, we can obtain the following equation.

[0082] (Equation 11)

[0083] When, Equation 11 is x i (t=τ n ) and x i (t=τ n+1 If the value of ) is known, the displacement x of the stabilized state after sufficient time has passed i The convergence value of (t), i.e., the expected amplitude of the steady state It shows that it can be estimated. And the estimated value obtained at that point in time Using Equation 5, the intensity of the external sound F at that point in time ext The magnitude of (t) can be calculated as follows.

[0084] (Equation 12)

[0085] In this specification, displacement x in a stabilized state i convergence value of (t) External sound intensity F calculated based on ext (t) is called the pure tone predicted amplitude.

[0086] Meanwhile, to extract the phase, if the corresponding spring motion is viewed as a one-dimensional projection of uniform circular motion, the phase is as follows.

[0087] (Equation 13)

[0089] Next, let us assume that a harmonic input consisting of n components whose frequencies are positive integer multiples of the fundamental frequency f0 is given. Then, the set W of the angular velocities of the harmonics is as follows.

[0090] (Equation 14)

[0091] Let's arrange the elements of set W in order from smallest to largest and represent them as follows.

[0092] (Equation 15)

[0093] These harmonics are It can be represented as.

[0094] Given a harmonic F(t) as input, the spring displacement x i (t) can be expressed as the sum of the spring displacements for each angular velocity constituting the frequency set W, as follows.

[0095] (Equation 16)

[0096] Here and It is as follows.

[0097] (Equation 17)

[0098] (Equation 18)

[0099] The magnitude of the angular velocity of the spring's natural frequency, ω 0i x in the direction of increasing (or decreasing) i By observing (t), we can find each element of the set of angular velocities W contained in the harmonics and the springs that meet the resonance condition. When observed over an arbitrary short time interval, the displacement x of the spring meeting the resonance condition is given by Equations 16, 17, and 18. i The maximum value of (t) is the spring displacement x that does not constitute a resonance condition immediately adjacent to the spring's natural angular velocity. i It is greater than the maximum value of (t). Therefore, the spring displacement x i If we construct a spectrogram of the DJ transform result using Equations 11 and 12 for the maximum values ​​for each natural period of the spring at (t), the angular velocity value at the point where the maximum value is observed at a specific time corresponds one-to-one with an element of the set of angular velocities W of the harmonics.

[0100] That is, by modeling the oscillatory motion of the spring, the spring displacement x expressed by Equations 16 to 18 i (t) can be known, and the displacement x of this spring iBy applying Equations 11 and 12 to (t), the estimated pure tone amplitude can be calculated when sounds with various frequencies are input. The estimated pure tone amplitude can be the steady-state predicted amplitude of Equation 11 or the pure tone predicted amplitude of Equation 12. Accordingly, by plotting the estimated pure tone amplitude in the space of the time axis and the frequency axis corresponding to the spring's resonance frequency, a DJ transform spectrogram based on the estimated pure tone amplitude can be generated.

[0102] In this regard, the displacement x corresponding to a maximum value of the spectrogram i From Equations 16, 17, and 18, it can be seen that (t) is significantly influenced by sounds with angular velocity included in the harmonics that meet the resonance condition, but is also influenced by sounds with angular velocity that do not meet the resonance condition. Given a harmonic, the natural vibration angular velocity ω 0i ga ω ext,m and the resonance condition, i.e. ( ) spring s i displacement x i (t) As for the angular velocity ω that is not a resonance condition ext,n Displacement x with sound input i The rate at which the amplitude of (t) changes can be estimated from the following equation.

[0103] (Equation 19)

[0104] Observing Equations 17 and 18, near the resonance condition In places far removed from the resonance conditions is. Equation 19 is the result of selecting and comparing only the larger values ​​among them. F ext,n and F ext,m If the value of does not differ significantly, when ζ=0.001 The impact of the protest Equation 19 shows that it is much more dominant. The influence of frequencies other than the resonance condition of the harmonics is not significant enough to change the location of the maximum value generated by the resonance condition. Therefore, the maximum value can be observed in the DJ transform spectrogram at the frequency location included in the harmonics.

[0106] This time, the frequencies constituting the harmonics and the displacement x that becomes the resonance condition for each i We examine the relationship between the maximum value of (t). In the DJ transformation, the natural frequency f0 and the spring displacement x, which is the resonance condition. i The maximum value of (t) is calculated every period of 1 / f0. Displacement x i f, which is not the fundamental frequency included in the harmonics, at the maximum value of (t). i The influence of is reflected, but the period of these frequencies is 1 / f i becomes a divisor of 1 / f0, so when calculating with a period of 1 / f0, x i It is reflected as a periodic property in the maximum value of (t). In DJ transformation, f is not the fundamental frequency. i and the spring displacement x that is the resonance condition i The maximum value of (t) is also 1 / f i It is calculated every period. The period of the part affected by f0 is 1 / f0 (1 / f0 > 1 / f i Since ), 1 / f i If calculated with a period, displacement x i The maximum value of (t) cannot be reflected by periodic properties.

[0107] Therefore, x related to f0 i The maximum value of (t) has a small amplitude of oscillation because periodicity is not broken, and f i x related to i At the maximum value of (t), periodicity is broken, and the amplitude of the oscillation of that value appears large. Through Equations 11 and 12, x i The amplitude value of the estimated pure tone amplitude-based spectrogram calculated using the maximum value of (t) is x iThe characteristics of the maximum value of (t) are reflected as is. Therefore, when calculating the standard deviation of the spectrogram amplitude values, the value is small in the part related to f0, and f i The value is large in the part related to.

[0108] In summary, when a harmonic is given, a spring that resonates with the fundamental frequency of the harmonic can be seen to have 1) a small variance of amplitude over time when measuring the amplitude of the spectrogram of its fundamental frequency, and 2) a large maximum value of amplitude.

[0109] Based on these characteristics, an embodiment of the present invention extracts a fundamental frequency based on the moving average of the estimated pure tone amplitude or the moving standard deviation of the estimated pure tone amplitude for each natural frequency of the DJ transform spectrogram.

[0111] FIG. 4 is a flowchart illustrating an example of the fundamental frequency extraction step (200) of FIG. 3. Referring to FIG. 4, the fundamental frequency extraction step may include a step (S210) of calculating the fundamental frequency fit based on the moving average of the estimated pure tone amplitude or the moving standard deviation of the estimated pure tone amplitude in the DJ transform spectrogram, and a step (S220) of calculating the maximum value of the fundamental frequency fit at each of a plurality of time points and extracting the fundamental frequency based on the maximum value of the calculated fundamental frequency fit.

[0112] Let R(t, f) be the degree to which the natural frequency f of the spring at time t is similar to the fundamental frequency of the input harmonic. As previously mentioned, the displacement of a spring resonating with the fundamental frequency shows less change over time compared to the displacement of a spring resonating with other frequencies constituting the harmonic. Furthermore, the displacement of a spring resonating with each frequency constituting the harmonic has a larger amplitude than that of a spring with an adjacent natural frequency. Using these characteristics, the fundamental frequency fit R(t, f) of the spring can be calculated by the following Equation 20.

[0113] (Equation 20)

[0114] (Equation 21)

[0115] (Equation 22)

[0116] (Equation 23)

[0117] Here, N is an integer and ε is a very small value greater than 0. For example, at time t, ε is It can be done.

[0118] To reduce the influence of small amplitude values ​​in the spectrogram The other side Make it so that β is a small value, β=10 -12 You can use .

[0119] According to the embodiment, instead of (Equation 20) or You can also use . That is, the fundamental frequency fit may be proportional to the moving average M(t, f) of the DJ transform spectrogram S(t, f) representing the estimated pure tone amplitude, or inversely proportional to the moving standard deviation σ(t, f).

[0120] Next, the fundamental frequency is extracted based on the maximum value of the fundamental frequency fit according to the natural frequency at each time point (S220).

[0121] According to an embodiment, the fundamental frequency can be extracted as the lowest frequency among the frequencies corresponding to the maximum value of the fundamental frequency fit according to the natural frequency at each time point.

[0123] FIG. 5 is a flowchart showing an example of the fundamental frequency extraction step (S220) of FIG. 4.

[0124] Referring to FIG. 5, the fundamental frequency extraction step (S220) may include a black and white spectrogram generation step (S310), an average black and white spectrogram generation step (S320), a maximum value extraction step of the average black and white spectrogram (S330), a candidate fundamental frequency extraction step (S340), a step of setting the fundamental frequency based on the black and white spectrogram (S350), and a step of setting the final fundamental frequency (S360) to improve accuracy by excluding the influence of noise.

[0125] The fundamental frequency extraction step (S220) does not need to include all of the steps S310 to S360, and may include only some of them depending on the embodiment.

[0127] According to an embodiment, the fundamental frequency extraction step (S220) may include: a black and white spectrogram generation step (S310) that extracts the top N (N is an integer greater than or equal to 2) of the fundamental frequency suitability for each time point, sets the value corresponding to the natural frequency corresponding to the N as "1", and sets the remaining value as "0"; an average black and white spectrogram generation step (S320) that calculates the average of the black and white spectrogram for an area of ​​the same size including each point of the black and white spectrogram; and a step (S330) that extracts the maximum value of the average black and white spectrogram at each of the plurality of time points.

[0128] The grayscale spectrogram generation step (S310) is a point in time that constitutes the DJ transform spectrogram. Extract the top N values ​​from the fundamental frequency fit R(t,f). Construct a grayscale spectrogram with values ​​of 0 and 1 based on whether it falls into the top N. If R(t,f) falls into the top N at time t, set BW(t,f)=1; otherwise, set BW(t,f)=0.

[0129] The average grayscale spectrogram generation step (S320) calculates the average in a rectangular area relative to each point constituting the grayscale spectrogram BW(t,f) as shown in the equation below. The result thus constructed is the average grayscale spectrogram Let's say.

[0130] (Equation 24)

[0131] The step of extracting the maximum value of the average grayscale spectrogram (S330) involves a given threshold value among the maximum values ​​according to the change in natural frequency at each time point t in the average grayscale spectrogram. Extract larger maximum values. Here, the threshold is calculated for each time point t The maximum value of the maximum value of It is set to a value obtained by multiplying by a constant ratio γ (0≤γ≤1.0). For example, γ can be set to 0.2.

[0132] In other words, the extracted maximum values ​​satisfy the following conditions simultaneously.

[0133] , (Equation 25)

[0134] , (Equation 26)

[0135] , (0≤γ≤1.0) (Equation 27)

[0137] The fundamental frequency extraction step (S220) may further include a step (S340) of extracting a candidate fundamental frequency based on the difference between the natural frequencies corresponding to adjacent maximum values ​​of the average grayscale spectrogram at each of a plurality of time points and the lowest frequency among the natural frequencies corresponding to the maximum values ​​of the average grayscale spectrogram.

[0138] The frequency corresponding to the k-th maximum value among the maximum values ​​extracted from the average grayscale spectrogram at time t and sorted in ascending order of frequency Let's call it the spacing between adjacent frequencies. Calculate as follows.

[0139] (Equation 28)

[0140] middle Select larger values ​​and the smallest value among them and Compare and find the smaller value as the candidate fundamental frequency at time t. It is determined as follows. Here, the fact that the minimum value among the difference values ​​of adjacent frequencies of harmonics present in the sound of a voice or instrument is likely to be the fundamental frequency was utilized.

[0141] If the amplitudes of all frequencies constituting a noise-free harmonic are the same, for each k It becomes.

[0143] The fundamental frequency extraction step (S300) may include a black-and-white spectrogram-based fundamental frequency setting step (S350), and the black-and-white spectrogram-based fundamental frequency setting step may include calculating the time average of the black-and-white spectrogram-based fundamental frequency set for a predetermined time interval including adjacent time points, setting a first frequency set including frequencies near values ​​obtained by multiplying the time average by positive integers less than or equal to a predetermined value, and setting the frequency with the largest average black-and-white spectrogram among the frequencies belonging to the first frequency set as the black-and-white spectrogram-based fundamental frequency by dividing the value obtained by multiplying the first frequency set by the positive integer.

[0144] Here, for setting the initial value, the method may further include the step of setting the candidate fundamental frequency of the time point with the smallest shifting variance of the difference between the candidate fundamental frequencies of adjacent time points among the candidate fundamental frequencies for the plurality of time points as the black-and-white spectrogram-based fundamental frequency of the time point with the smallest shifting variance.

[0145] Candidate fundamental frequencies for each time point t Let us assume that we have found the grayscale spectrogram-based fundamental frequency BF0(t) for each time point t. First, we calculate the grayscale spectrogram-based fundamental frequency BF0(t) at a specific time point t0. Second, we calculate the grayscale spectrogram-based fundamental frequency as we increase time from time point t0. Third, we calculate the grayscale spectrogram-based fundamental frequency as we decrease time from time point t0.

[0146] First, the time point t0 for calculating the fundamental frequency based on the grayscale spectrogram is set as the time point where the variance of the change over time of the candidate fundamental frequency based on the grayscale spectrogram is smallest. The variance V(t) of the change in the candidate fundamental frequency based on the grayscale spectrogram at each time point t is calculated using the following formula.

[0147] (Equation 29)

[0148] (Equation 30)

[0149] (Equation 31)

[0150] The time point t0 when V(t) is smallest is The fundamental frequency BF0(t0) at time t0 is determined to be the same value as the candidate fundamental frequency as follows.

[0151] (Equation 32)

[0152] In the second step, the fundamental frequency based on the grayscale spectrogram is calculated while increasing time from time point t0. From time point t0 to time point t k Let us assume that the fundamental frequencies based on grayscale spectrograms have been determined up to this point. S(t) is the set of natural frequencies near the average frequency of the n grayscale spectrogram-based fundamental frequencies determined up to that point, and near frequencies that are positive integer multiples of this average frequency. k+1 Let's call it ).

[0153]

[0154] (Equation 33)

[0155] Here,

[0156] (Equation 34)

[0157] and, for example, Δf=20Hz, i max It can be set to =5.

[0158] set S(t k+1 Among the natural frequencies belonging to ), the frequency with the largest average grayscale spectrogram value is f max and f max is in the frequency domain Let's assume it belongs to . Then, time t k+1 Black and white spectrogram-based fundamental frequency BF0(t) k+1 ) is calculated using the following formula.

[0159] (Equation 35)

[0160] t k+1 Repeat the second step above while increasing k by 1 until the last time of the given spectrogram.

[0161] In the third step, a process similar to the second step is performed while decreasing time from time t0 until the fundamental frequency based on the black-and-white spectrogram at each time point is obtained.

[0163] Next, the step of extracting the fundamental frequency (S300) may further include the step of setting a second frequency set comprising values ​​obtained by multiplying the black-and-white spectrogram-based fundamental frequency by positive integers less than or equal to a predetermined value at each of the plurality of time points, and setting the value obtained by dividing the frequency with the greatest fundamental frequency suitability among the frequencies belonging to the second frequency set by the positive integer multiplied when setting the second frequency set as the final fundamental frequency.

[0164] We intend to extract the final fundamental frequency f0(t) using the grayscale spectrogram-based fundamental frequency BF0(t) at each time point t and the aforementioned fundamental frequency fit R(t, f).

[0165] At time t, the grayscale spectrogram-based fundamental frequency BF0(t) and the set of frequencies near positive integer multiples of BF0(t) Let's say.

[0166] (Equation 36)

[0167] Here, Δf=20Hz, i max It can be set to =5.

[0168] set at time t Among the frequencies belonging to , the frequency with the largest fundamental frequency fit R(t, f) is f max and f max is in the frequency domain Let's assume that it belongs to . Then, the final fundamental frequency f0(t) at time t is calculated using the following formula.

[0169] (Equation 37)

[0171] Next, harmonics of the final fundamental frequency f0(t) are generated as training data.

[0172] (Equation 38)

[0173] Here, is the amplitude of the frequency j times f0(t) at time t, and Satisfying, This can be seen from equation (11). Also, is the phase at the frequency j times f0(t) at time t, and And, This can be seen from equation (13).

[0174] For example, if you want to generate voice data (wav data) with a DJ conversion resolution of 1 msec and a sampling frequency of 48,000 Hz, you must generate 48 voice data at 1 msec intervals, so you generate F(t) by increasing time t by 1 / 48 msec. That is, the interval between multiple time points can be 1 / 48 msec.

[0176] Figure 6 is a diagram showing an experimental example of the present invention when a sound of harmonics that changes over time is input.

[0177] FIG. 6(a) represents a sound input, FIG. 6(b) represents a DJ-transformed spectrogram generated using the sound input of FIG. 6(a), FIG. 6(c) represents a grayscale spectrogram generated using the calculated fundamental frequency fit after calculating the fundamental frequency fit from the DJ-transformed spectrogram of FIG. 6(b), FIG. 6(d) represents a candidate fundamental frequency selected using the maximum value in the frequency direction of the average grayscale spectrogram and the frequency values ​​at that location after generating an average grayscale spectrogram using the grayscale spectrogram of FIG. 6(c), FIG. 6(e) represents a grayscale spectrogram-based fundamental frequency generated using the candidate fundamental frequency of FIG. 6(d), FIG. 6(f) represents the final fundamental frequency calculated using the grayscale spectrogram-based fundamental frequency and the fundamental frequency fit of FIG. 6(e), FIG 6(g) is a magnified view of part of 5(b).

[0178] As shown in Figures 6 (a) to (g), it can be seen that the final fundamental frequency approximately matches the value corresponding to the fundamental frequency of the sound input.

[0180] Figure 7 is a table showing experimental results for an embodiment of the present invention.

[0181] Referring to the top of Fig. 7, a dataset for speaker identification (VoxCeleb1 Identification Set) was input for 2.58 seconds, and the data augmented by MUSAN and RIR (Room Impulse Response), respectively, and the data augmented by generating a DJ transform spectrogram according to an embodiment of the present invention were applied to a thin ResNetSE50 network. At this time, softmax was used as the loss function.

[0182] In this experiment, the content of the augmented data (harmonics of the fundamental frequency) using the DJ transform spectrogram is 62.5%. The content of fundamental frequency harmonics within the total augmented data can range from 50% to 75%. If the content of fundamental frequency harmonics is lower than 50%, the essence of the original data may be compromised; on the other hand, if the content is higher than 75%, the training data becomes too close to the original data, which may reduce the effectiveness of data augmentation. In this experiment, the effect of performance improvement was demonstrated by conducting the experiment at 62.5%, which is the median value of this range.

[0183] According to conventional technology, when only data augmented by MUSAN and RIR was used, the accuracy of speaker identification was 94.91%, but when data augmented by DJ transform spectrogram was included, it was found that the accuracy improved to 95.55%.

[0184] Referring to the bottom of Fig. 7, a dataset for speaker verification (VoxCeleb1 Verification Set) was input for 2.58 seconds, and the data augmented by MUSAN and RIR (Room Impulse Response), respectively, and the data augmented by generating a DJ transform spectrogram according to an embodiment of the present invention (content 62.5%) were applied to a thin ResNetSE152 network. At this time, angle-proto was used as the loss function.

[0185] According to the prior art, when only data augmented by MUSAN and RIR was used, the error rate of speaker verification was 4.279%, but when data augmented by DJ transform spectrogram was included, the error rate was reduced to 4.040%, showing an improved effect.

[0186] Although various data augmentation methods are used in machine learning, a method of augmenting data by extracting harmonics of the fundamental frequency of the original data has never been used before, and as can be seen from the experimental results above, its effectiveness is excellent.

[0188] FIG. 8 is a structural diagram of a data augmentation system (1) according to an embodiment of the present invention.

[0189] Referring to FIG. 8, the data augmentation system (1) may include a processing unit (2) and a memory (3).

[0190] The processing device (2) executes a data augmentation method according to an embodiment of the present invention. The data augmentation method basically includes the step of extracting at least one fundamental frequency of voice data and the step of generating harmonics of the fundamental frequency as training data, and may include the additional steps described above according to the embodiment.

[0191] Data necessary for the execution of the data augmentation method is stored in the memory (3). For example, voice data may be stored in the memory (3) and provided to the processing device (2). Additionally, the memory (3) may store estimated pure tone amplitude, steady state predicted amplitude, pure tone predicted amplitude, phase, DJ transform spectrogram, fundamental frequency fit, black and white spectrogram, average black and white spectrogram, and black and white spectrogram-based fundamental frequency, etc., and the processing device (2) may use these to extract the final fundamental frequency.

[0192] Although not illustrated in FIG. 8, the data augmentation system (1) may additionally include a voice input device such as a microphone for receiving voice input and converting it into electronic voice data, a display device such as a monitor for displaying data stored in memory (3), and input / output devices such as a keyboard and mouse for receiving user input.

[0194] Although the present invention has been described in detail through preferred embodiments, the invention is not limited thereto, and it is obvious to those skilled in the art that various modifications and applications can be made within the scope of the technical concept of the invention. Accordingly, the true scope of protection of the present invention should be interpreted by the following claims, and all technical concepts within an equivalent scope should be interpreted as being included within the scope of rights of the present invention.

Claims

Claim 1 Each step is performed by a computer processor, and the data augmentation method for generating training data for machine learning for speech recognition or speaker recognition comprises: (a) a step of extracting at least one fundamental frequency of speech data; and (b) a step of generating harmonics of said fundamental frequency as training data, wherein the harmonics of said fundamental frequency are included in 50% to 75% of the total training data; wherein step (a) comprises: (a-1) a step of, for the input of speech data, modeling the vibrational motion of a plurality of springs having different natural frequencies to calculate the estimated pure tone amplitude and phase according to said natural frequencies, and generating a DJ transform spectrogram representing the estimated pure tone amplitude according to a frequency corresponding to the natural frequency of each of said plurality of springs and a plurality of time points; and (a-2) a step of extracting the fundamental frequency based on the moving average of the estimated pure tone amplitude or the moving standard deviation of the estimated pure tone amplitude for each of the natural frequencies of the DJ transform spectrogram; a data augmentation method comprising. Claim 2 delete Claim 3 A data augmentation method according to claim 1, wherein step (a-1) comprises: a step of estimating a steady-state expected amplitude, which is a convergence value of the amplitude of each of the plurality of springs in a steady state, based on the amplitudes of each of the plurality of springs at two time points of a natural one-period interval; and a step of calculating the estimated pure tone amplitude based on a pure tone predicted amplitude, which is the amplitude of the input sound estimated based on the steady-state expected amplitude. Claim 4 A data augmentation method characterized in that, in paragraph 3, the estimated pure tone amplitude is the steady-state expected amplitude or the pure tone predicted amplitude. Claim 5 A data augmentation method according to claim 1, wherein the step (a-2) comprises: (a-21) a step of calculating the fundamental frequency fit based on the moving average of the estimated pure tone amplitude or the moving standard deviation of the estimated pure tone amplitude in the DJ transform spectrogram; and (a-22) a step of calculating the maximum value of the fundamental frequency fit at each of the plurality of time points and extracting the fundamental frequency based on the calculated maximum value of the fundamental frequency fit. Claim 6 A data augmentation method according to claim 5, characterized in that the fundamental frequency fit is proportional to the moving average of the estimated pure tone amplitude or inversely proportional to the moving standard deviation of the estimated pure tone amplitude. Claim 7 In claim 5, the above step (a-22) comprises: a grayscale spectrogram generation step of extracting the top N (N is an integer greater than or equal to 2) of the fundamental frequency suitability at each of the plurality of time points, setting the value corresponding to the natural frequency corresponding to the N to "1" and the remaining value to "0"; an average grayscale spectrogram generation step of calculating the average of the grayscale spectrogram for an area of ​​the same size including each point of the grayscale spectrogram; and a step of extracting the maximum value of the average grayscale spectrogram at each of the plurality of time points. Claim 8 A data augmentation method according to claim 7, wherein the above step (a-22) further comprises the step of extracting a candidate fundamental frequency based on the difference of the natural frequency corresponding to the adjacent maximum value of the average grayscale spectrogram and the lowest frequency among the natural frequencies corresponding to the maximum value of the average grayscale spectrogram at each of the plurality of time points. Claim 9 A data augmentation method according to claim 8, wherein the above step (a-22) further comprises the step of calculating a time average of a black-and-white spectrogram-based fundamental frequency set for a predetermined time interval including adjacent time points, setting a first frequency set including frequencies near values ​​obtained by multiplying the time average by positive integers less than or equal to a predetermined value, and setting the frequency with the largest average black-and-white spectrogram among the frequencies belonging to the first frequency set as the black-and-white spectrogram-based fundamental frequency by dividing the value obtained by the positive integer multiplied when setting the first frequency set. Claim 10 A data augmentation method according to claim 9, wherein the above step (a-22) further comprises the step of setting the candidate fundamental frequency of the time point with the smallest shifting variance of the difference between the candidate fundamental frequencies of adjacent time points among the candidate fundamental frequencies for the plurality of time points as the black-and-white spectrogram-based fundamental frequency of the time point with the smallest shifting variance. Claim 11 A data augmentation method according to claim 9, wherein the step (a-22) further comprises the step of, at each of the plurality of time points, setting a second frequency set including values ​​obtained by multiplying a black-and-white spectrogram-based fundamental frequency by positive integers less than or equal to a predetermined value, and setting the frequency with the greatest fundamental frequency suitability among the frequencies belonging to the second frequency set as the final fundamental frequency by dividing the value obtained by the positive integer multiplied when setting the second frequency set. Claim 12 A data augmentation method according to claim 3, wherein in step (b) above, the harmonic of the fundamental frequency has the expected amplitude and phase of the stable state of a spring having a natural frequency corresponding to the harmonic of the fundamental frequency as the amplitude and phase of the harmonic of the fundamental frequency. Claim 13 delete Claim 14 A computer-readable recording medium in which each step of paragraph 1 is recorded so that it can be executed on a computer.

Images