Audio augmentation method and device, computer equipment and storage medium

Abstract

The invention discloses an audio augmentation method and device, computer equipment and a storage medium, and the method comprises the steps: carrying out the time-frequency feature extraction of a first audio signal and a second audio signal, and obtaining a first feature value and a second feature value of each time-frequency point; adjusting the smaller characteristic value in the first characteristic value and the second characteristic value to obtain a third characteristic value; obtaining a fourth characteristic value of each time frequency point according to a larger characteristic value in the first characteristic value and the second characteristic value and the third characteristic value; and generating a third audio signal according to the fourth characteristic value of each time frequency point. By comparing the characteristic values of different audio signals on each time frequency point and determining the secondary component on each time frequency point for subsequent disturbance, the interference of random disturbance on the audio characteristics is reduced as much as possible on the basis of ensuring the amplitude and phase diversity of the generated signals, and the accuracy of the audio characteristics is improved. And the synthesized signal can still maintain the audio characteristics of the original audio signal, so that the subsequent training effect on the model is improved.

Description

Technical Field

[0001] The present application relates to the field of audio signal processing, and in particular to an audio augmentation method, apparatus, computer device and storage medium. Background Art

[0002] During the training process of a speech recognition model, it is usually necessary to augment the signal, such as noise augmentation, to increase the richness and diversity of training samples so that the model can effectively process noisy input audio, thereby improving the robustness of the model.

[0003] However, related technologies usually use clean audio signals and noisy audio signals to superimpose in the time-frequency domain, or apply a random perturbation to an audio signal to generate more diverse augmented signals. However, applying random perturbations to audio signals can easily cause the synthesized augmented signals to lose some of the audio features of the original audio signals, thus affecting the subsequent training effect of the model. Summary of the invention

[0004] In view of the above technical problems, the embodiments of the present application provide an audio augmentation method, apparatus, computer device and storage medium, aiming to solve the problem of unsatisfactory audio augmentation effect.

[0005] In a first aspect, an embodiment of the present application provides an audio augmentation method, comprising:

[0006] Extracting time-frequency features of the first audio signal to obtain a first eigenvalue of the first audio signal at each time-frequency point, and extracting time-frequency features of the second audio signal to obtain a second eigenvalue of the second audio signal at each time-frequency point;

[0007] The smaller eigenvalue of the first eigenvalue and the second eigenvalue of each time-frequency point is adjusted according to a preset coefficient to obtain a third eigenvalue of each time-frequency point;

[0008] Obtaining a fourth eigenvalue of each time-frequency point according to the larger eigenvalue of the first eigenvalue and the second eigenvalue of each time-frequency point, and the third eigenvalue;

[0009] Generate a third audio signal according to the fourth eigenvalue of each time-frequency point

[0010] In a second aspect, an embodiment of the present application provides an audio augmentation device, comprising:

[0011] a processing module, configured to extract time-frequency features of the first audio signal to obtain a first eigenvalue of the first audio signal at each time-frequency point, and to extract time-frequency features of the second audio signal to obtain a second eigenvalue of the second audio signal at each time-frequency point;

[0012] An adjustment module, used to adjust the smaller eigenvalue of the first eigenvalue and the second eigenvalue of each time-frequency point according to a preset coefficient to obtain a third eigenvalue of each time-frequency point;

[0013] The determination module is used to obtain the fourth eigenvalue of each time-frequency point according to the larger eigenvalue of the first eigenvalue and the second eigenvalue of each time-frequency point, and the third eigenvalue.

[0014] A generating module is used to generate a third audio signal according to the fourth eigenvalue of each time-frequency point.

[0015] In a third aspect, an embodiment of the present application further provides a computer device, including a processor and a memory, wherein the memory stores a plurality of instructions; the processor loads instructions from the memory to execute the steps of audio augmentation as described above.

[0016] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a plurality of instructions, wherein the instructions are suitable for loading by a processor to execute the steps of audio augmentation as described above.

[0017] In a fifth aspect, an embodiment of the present application further provides a computer program product, wherein the computer program product comprises a computer program, and the computer program is executed by a processor to perform the steps of audio augmentation as described above.

[0018] When adopting the scheme of the embodiment of the present application, the first audio signal and the second audio signal are first processed to obtain the first eigenvalue and the second eigenvalue at the first time-frequency point. The first eigenvalue and the second eigenvalue can describe the signal component of the audio signal at the time-frequency point. At this time, by screening out the smaller eigenvalue of the first eigenvalue and the second eigenvalue, that is, screening out the secondary component, and then adjusting the third eigenvalue as the secondary component to obtain the disturbed eigenvalue, and further combining it with the larger eigenvalue, that is, the main component, to generate the fourth eigenvalue of the time-frequency point, so as to retain the feature information at each video point as much as possible, so that the third audio signal generated by the fourth eigenvalue can better integrate the signal characteristics of the first audio signal and the second audio signal. By adjusting the smaller component of the signal at the time-frequency point, the present application can minimize the interference to the time-frequency characteristics caused by the adjustment of the signal component on the basis of ensuring the diversity of the generated augmented audio signal, and ensure that the synthesized signal can still retain the audio characteristics of the original audio signal, thereby facilitating the improvement of the subsequent training effect of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0020] Figure 1 A schematic diagram of the steps of an audio augmentation method provided in an embodiment of the present application;

[0021] Figure 2 A schematic flow chart of a step of fusing time-frequency feature information at adjacent time-frequency points to determine a feature value at each time-frequency point provided in an embodiment of the present application;

[0022] Figure 3 A schematic flow chart of a step of setting weights for neighboring time-frequency points to determine characteristic values ​​at the time-frequency points provided in an embodiment of the present application;

[0023] Figure 4 A schematic diagram of a process flow for determining a preset coefficient according to a characteristic value difference provided in an embodiment of the present application;

[0024] Figure 5 A schematic diagram of a process flow of generating a third audio signal based on a fourth eigenvalue provided in an embodiment of the present application;

[0025] Figure 6 A schematic diagram of a flow chart of steps for implementing a speech model through a third audio signal provided in an embodiment of the present application;

[0026] Figure 7 A schematic diagram of the structure of an audio amplification device provided in an embodiment of the present application;

[0027] Figure 8 A schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0028] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present application. At the same time, in the description of the embodiments of the present application, the terms "first", "second", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more features. In the description of the embodiments of the present application, the meaning of "multiple" is two or more, unless otherwise clearly and specifically defined.

[0029] In order to better understand the audio augmentation method, device, computer equipment and storage medium provided by the present application, the relevant application background of the audio augmentation method is first explained. Among them, the audio augmentation method mainly refers to the process of training a speech recognition model, by augmenting the training samples, for example, synthesizing with noise signals to achieve noise augmentation, to increase the richness and diversity of the training samples, thereby improving the training effect of the model, for example, enabling the model to still effectively process the input audio with noise, thereby improving the robustness of the model.

[0030] In the related art, the more common audio augmentation is to simply superimpose the clean audio signal and the noisy audio signal. However, due to the limitation that the amplitude and phase of the synthesized audio signal are only related to the clean audio signal and the noisy audio signal, the augmented audio synthesized in this way is limited in the diversity of amplitude and phase. In order to improve the diversity of synthesized audio, the related art further proposes to apply random perturbations to an audio signal in the frequency domain to improve the diversity of amplitude and phase of the synthesized audio. However, in actual application, it is found that applying random perturbations to audio signals can easily cause the synthesized augmented signal to lose some audio features of the original audio signal, thereby affecting the subsequent training effect of the model.

[0031] In order to solve the above problems, the present application provides an audio augmentation method, device, computer equipment and storage medium, which compares the features of different audio signals at different time-frequency points and determines the smaller eigenvalue with a lower component at each time-frequency point, so that in the subsequent process of adding random disturbances, the smaller eigenvalue is disturbed to reduce the interference of random disturbances on the audio features, and merged with the larger eigenvalue to ensure that the final synthesized signal can still retain the audio features of the original audio signal, thereby improving the subsequent training effect of the model. For details, please refer to Figure 1 , Figure 1 A schematic flow chart of a method for audio augmentation provided in an embodiment of the present application, specifically, comprising steps S110 to S140:

[0032] S110, extracting time-frequency features of the first audio signal to obtain a first eigenvalue of the first audio signal at each time-frequency point, and extracting time-frequency features of the second audio signal to obtain a second eigenvalue of the second audio signal at each time-frequency point.

[0033] In the embodiment of the present application, combined with the above-mentioned related description, it can be known that the first audio signal and the second audio signal can be generally understood as the clean audio signal and the noise audio signal required for the audio augmentation process, wherein the definition of the clean audio signal and the noise audio signal is not limited in the embodiment of the present application, and any audio signal that can synthesize audio to achieve audio augmentation can be understood as one of the first audio signal and the second audio signal provided in the embodiment of the present application. Specifically, for ease of understanding, taking the model training process as an example, the first audio signal as a clean audio signal can be extracted from the training sample set of the model training, and the second audio signal as a noise audio signal can be read from the noise sample database based on network technology. The specific implementation scheme can be further referred to the relevant explanations of the subsequent embodiments.

[0034] Further, by performing time-frequency feature processing on the first audio signal, the first eigenvalue of the first audio signal at each time-frequency point can be obtained, that is, the first eigenvalue here is usually related to the time-frequency feature of the audio signal. Wherein, time-frequency feature processing usually refers to extracting frequency domain features from the audio signal in the time domain. For example, as a common feasible implementation scheme, the time-frequency feature information of the first audio signal can be obtained by extracting frequency domain features from the audio signal in the time domain, which includes the features of the first audio signal at each time-frequency point. The frequency domain feature extraction here can usually be obtained by extracting frequency domain features such as Mel frequency cepstrum coefficients or Mel filter bank coefficients. Of course, it is also feasible to extract other frequency domain features, but considering that the present application needs to perform certain disturbance changes on the extracted time-frequency feature values ​​to synthesize augmented audio, it is found in actual applications that, compared with other frequency domain features, frequency domain features such as Mel frequency cepstrum coefficients (MFCC, Mel Frequency Cepstrum Coefficient) or Mel filter bank coefficients (Fbank, Mel filter bank) can achieve better augmentation effects as subsequent feature value comparisons. Therefore, as a feasible implementation scheme of the present application, it is possible to process the signal in the time domain to obtain Mel-frequency cepstrum coefficients or Mel-frequency filter bank coefficients as the characteristic values ​​of the signal at the time-frequency point.

[0035] On the basis of the foregoing, by performing similar processing on the second audio signal, the characteristic value of the second audio signal at each time-frequency point can also be obtained, that is, the time-frequency characteristic information of the second audio signal can be obtained. On this basis, based on the association between the characteristic value and the time-frequency point included in the time-frequency characteristic information, a corresponding relationship between the characteristic value of the first audio signal and the second audio signal at each time-frequency point can also be constructed to compare the first characteristic value and the second characteristic value of the first audio signal and the second audio signal at the same time-frequency point.

[0036] Specifically, to facilitate understanding of the above content, taking the first audio signal s(t) and the second audio signal n(t) as an example, after extracting the Mel filter bank coefficient (Fbank) and performing an inverse transform in the logarithmic domain, the time-frequency feature information S(t, f) of the first audio signal and the time-frequency feature information N(t, f) of the second audio signal can be obtained, which respectively include the characteristic values ​​at each time-frequency point (t, f), thereby facilitating determination of the first characteristic value of the first audio signal and the second audio value of the second audio signal at the same time-frequency point.

[0037] Of course, in addition to the aforementioned implementation scheme of using the individual characteristic information of the audio signal at each time-frequency point as the characteristic value, that is, in addition to considering the characteristic information of each time-frequency point separately, the characteristic value at each time-frequency point can also be determined by the characteristic information of the first audio signal at multiple time-frequency points within a certain time-frequency neighborhood range. For details, please refer to Figure 2 , Figure 2 A schematic flow chart of the steps of fusing time-frequency feature information at adjacent time-frequency points to determine feature values ​​at each time-frequency point is provided in an embodiment of the present application. Specifically, in an embodiment of the present application, a first audio signal is used as an example for explanation. The processing process of the second audio signal can refer to the following, which is not repeated in this application. Specifically, steps S210 to S220 are included:

[0038] S210: Perform frequency domain feature extraction on the first audio signal to obtain time-frequency feature information of the first audio signal at each time-frequency point.

[0039] S220: Determine a first eigenvalue of the first audio signal at the first time-frequency point according to first time-frequency feature information of the first audio signal at the first time-frequency point and second time-frequency feature information of the first audio signal at the second time-frequency point.

[0040] In this embodiment of the present application, the time interval and / or frequency difference between the first time-frequency point and the second time-frequency point meets a preset threshold.

[0041] Unlike the aforementioned frequency domain feature extraction of the first audio signal and directly using the obtained time-frequency feature information of each time-frequency point as the characteristic value at the time-frequency point, in the present embodiment, the first characteristic value of the first audio signal at the first time-frequency point is determined based on the first time-frequency feature information of the first time-frequency point and the second time-frequency feature information at the second time-frequency point, wherein the time interval and / or frequency difference between the first time-frequency point and the second time-frequency point satisfies the preset threshold, that is, the second time-frequency point can be understood as the time-frequency point of the first time-frequency point in the time-frequency neighborhood, that is, the characteristic value at each time-frequency point is jointly determined by the video feature information at the time-frequency point and the neighboring time-frequency points, thereby avoiding the occurrence of erroneous noise points in the audio signal at a certain time-frequency point, which leads to the screening of erroneous main components, thereby affecting the subsequent signal fusion effect.

[0042] Of course, in order to further accurately obtain the characteristic values ​​of the audio signal at each time-frequency point to identify the main signal components and secondary signal components at the time-frequency point, the influence of the time-frequency neighborhood on the current time-frequency point will also be fully considered, that is, corresponding weight coefficients can be set for different neighboring time-frequency points of the time-frequency point to weight the time-frequency characteristic information. For details, please refer to Figure 3 , Figure 3 A schematic flow chart of a step of setting weights for neighboring time-frequency points to determine characteristic values ​​at the time-frequency points provided in an embodiment of the present application, specifically, comprising steps S310 to S320:

[0043] S310: Determine a weight coefficient corresponding to each of the second time-frequency points according to a time interval and / or a frequency difference between the second time-frequency point and the first time-frequency point.

[0044] In the embodiment of the present application, the larger the time interval and / or frequency difference between the second time-frequency point and the first time-frequency point, the smaller the time-frequency characteristic information of the second time-frequency point interferes with the time-frequency characteristic information of the first time-frequency point. At this time, a smaller weight coefficient can be set for such time-frequency point. Conversely, the smaller the time interval and / or frequency difference between the second time-frequency point and the first time-frequency point, that is, the closer they are in the time-frequency domain, the larger the weight coefficient can be set for such time-frequency point. In addition, the weight of the first time-frequency point can be set to 1 by default.

[0045] S320: Weight the first time-frequency feature information and a plurality of second time-frequency feature information according to a weight coefficient corresponding to each second time-frequency point to obtain a first feature value of the first audio signal at the first time-frequency point.

[0046] In the embodiment of the present application, by weighting the first time-frequency feature information and the second time-frequency feature information using the weight coefficient corresponding to each time-frequency point, a first eigenvalue of the first audio signal at the first time-frequency point can be obtained that more accurately reflects the strength of the signal component.

[0047] In addition, it should be noted that in the aforementioned process, especially in the noise augmentation process, it is usually required that the synthesized audio signal has a certain signal-to-noise ratio. Therefore, before extracting the time-frequency feature information of the first audio signal and the second audio signal, the first audio signal and the second audio signal are usually weighted based on the weight coefficient corresponding to the signal-to-noise ratio. That is, the time-frequency feature extraction of the first audio signal to obtain the first eigenvalue of the first audio signal at each time-frequency point, and the time-frequency feature extraction of the second audio signal to obtain the second eigenvalue of the second audio signal at each time-frequency point include:

[0048] The first audio signal and the second audio signal are weighted according to weight information corresponding to a plurality of preset signal-to-noise ratios to obtain weighted first audio signals and second audio signals.

[0049] In order to facilitate the understanding of the above content, the following will be specifically explained, taking the training model as an example, wherein the signal-to-noise ratio snr of the mixed noise during training can usually be randomly sampled in the range of [10dB, 30dB], that is, snr∈[10dB, 30dB].

[0050] Considering that the Mel-frequency cepstrum coefficients or Mel-filter bank coefficients usually need to use the magnitude spectrum in the logarithmic domain when they are extracted, it is usually necessary to convert the signal-to-noise ratio snr into the exponential domain, that is, to obtain ratio = exp(snr) / 10. Specifically, the obtained ratio is the signal weight coefficient corresponding to the signal-to-noise ratio and used to describe the signal ratio of the first audio signal and the second audio signal. For example, the signal weight ratio of the first audio signal and the second audio signal is ratio: 1. Therefore, the first audio signal and the second audio signal can be weighted according to the signal weight ratio provided above, that is, the weight information, so as to obtain the weighted first audio signal and the second audio signal for further use in subsequent processing.

[0051] Of course, considering that the sound intensity of the synthesized audio signal is usually changed in the above process, in the actual process of processing the time-frequency feature information, it is usually also possible to normalize based on the signal weight coefficient to obtain the weight coefficients of the first audio signal and the second audio signal respectively, for example, the weight coefficient α of the first audio signal is ratio / (ratio+1), and the weight coefficient β of the second audio signal is 1 / (ratio+1), so as to weight the first audio signal and the second audio signal respectively and obtain the weighted result for subsequent processing.

[0052] S120: Adjust the smaller eigenvalue between the first eigenvalue and the second eigenvalue of each time-frequency point according to a preset coefficient to obtain a third eigenvalue of each time-frequency point.

[0053] Compared with the implementation scheme in the related art by applying random perturbations to the time-frequency characteristics of a signal, in the embodiment of the present application, the smaller eigenvalue is further determined from the first eigenvalue and the second eigenvalue at each time-frequency point, that is, the eigenvalues ​​of the first audio signal and the second audio signal at the time-frequency point respectively, so as to adjust the smaller eigenvalue according to a preset coefficient, so as to obtain a richer augmented signal through random perturbations while minimizing the change in the audio feature information of the obtained augmented information.

[0054] Furthermore, the scheme for perturbing the smaller eigenvalues ​​at different time-frequency points can be based on a preset fixed perturbation coefficient, for example, the eigenvalue is perturbed at a ratio of 0.1. Of course, the randomly generated preset coefficient can also be used for each time-frequency point to adjust the smaller eigenvalue, where the preset coefficient can be randomly generated between -0.5 and 0.5. Of course, as a further optional implementation scheme of the present application, an implementation scheme for determining the preset coefficient based on the eigenvalue difference between the first eigenvalue and the second eigenvalue is also provided. For details, please refer to Figure 4 , Figure 4 A schematic flow chart of a step of determining a preset coefficient according to a characteristic value difference provided in an embodiment of the present application, specifically, comprising steps S410 to S420:

[0055] S410, obtaining an eigenvalue difference between a first eigenvalue and a second eigenvalue at each time-frequency point.

[0056] In the embodiment of the present application, by subtracting the first eigenvalue from the second eigenvalue at each time-frequency point, the eigenvalue difference at each time-frequency point can be obtained.

[0057] S420: Determine a preset coefficient corresponding to each of the time-frequency points according to the eigenvalue difference.

[0058] In an embodiment of the present application, a preset coefficient corresponding to each time-frequency point is determined based on the eigenvalue difference between the first eigenvalue and the second eigenvalue. For example, as a common feasible implementation scheme, the eigenvalue difference and the preset coefficient may be positively correlated. For example, when the eigenvalue difference between the first eigenvalue and the second eigenvalue is large, that is, when the gap between the main component and the secondary component in the signal is too large, a larger preset coefficient may be set, for example, it may be set to 0.4, so that the secondary component is adjusted to a larger value, thereby improving the richness of the audio signal generated by subsequent fusion without affecting the main component. On the contrary, when the eigenvalue difference between the first eigenvalue and the second eigenvalue is small, a smaller preset coefficient may be set, for example, 0.1, so as to avoid adjusting the secondary component to a larger value and affecting the main component, so that the fused audio signal retains the characteristic information of the original audio signal as much as possible.

[0059] S130: Obtain a fourth eigenvalue of each time-frequency point according to a larger eigenvalue among the first eigenvalue and the second eigenvalue of each time-frequency point, and the third eigenvalue.

[0060] In the embodiment of the present application, after completing the perturbation of the smaller eigenvalue as mentioned above to obtain the third eigenvalue, the larger eigenvalue of the first eigenvalue and the second eigenvalue of each time-frequency point can be further combined with the third eigenvalue to obtain the fourth eigenvalue of each time-frequency point, so that the fourth eigenvalue of each time-frequency point retains the audio signal characteristics of the main signal component and part of the audio signal characteristics of the secondary signal component.

[0061] Specifically, to facilitate understanding of the above content, the following will describe the perturbation process of the feature value in the form of an audio signal feature mask to obtain the final feature information. The details are as follows.

[0062] In the embodiment of the present application, by comparing the eigenvalues ​​at each time-frequency point, the main component and the secondary component can be determined, thereby obtaining a mask at each time-frequency point. For example, when the first eigenvalue is greater than the second eigenvalue, the mask can be set to 1. Conversely, when the first eigenvalue is less than or equal to the second eigenvalue, the mask can be set to 0.

[0063] Furthermore, when the mask is 1, it indicates that the first audio signal is the main component at this time-frequency point. At this time, the eigenvalue of the second audio signal is perturbed. Finally, the fourth eigenvalue FakeFbank(t,f) obtained at this time-frequency point satisfies FakeFbank(t,f)=S(t,f)+random_weight(t,f)*N(t,f);

[0064] When the mask is 0, it means that the second audio signal is the main component at this time-frequency point. At this time, the audio eigenvalue of the first audio signal will be perturbed. Finally, the fused eigenvalue FakeFbank(t,f) obtained at this time-frequency point satisfies FakeFbank(t,f)=random_weight(t,f)*S(t,f)+N(t,f).

[0065] Among them, FakeFbank(t,f) is the fourth eigenvalue after fusion, S(t,f) represents the first eigenvalue of the first audio signal S at the time-frequency point (t,f), N(t,f) represents the second eigenvalue of the second audio signal N at the time-frequency point (t,f), and random_weight(t,f) represents the preset coefficient at the time-frequency point, which can be a random value or obtained based on the difference between the first eigenvalue S(t,f) and the second eigenvalue N(t,f).

[0066] S140: Generate a third audio signal according to the fourth eigenvalue at each time-frequency point.

[0067] In the embodiment of the present application, in the aforementioned fourth eigenvalue that retains the audio signal characteristics of the main signal component and part of the audio signal characteristics of the secondary signal component, the fourth eigenvalue in the time-frequency domain is converted to the time domain space by inverse transforming the fourth eigenvalue in the time-frequency domain. The third audio signal in the time domain space obtained is the audio signal obtained by fusing the time-frequency characteristics of the first audio signal and the time-frequency characteristics of the second audio signal, and by changing the audio characteristics of the secondary component, the richness of the generated audio signal in terms of phase and amplitude is guaranteed, and the audio characteristics of the main component are retained as much as possible, thereby improving the application value of the fused audio signal in the subsequent model training process.

[0068] For details, please refer to Figure 5 , Figure 5 A schematic flow chart of a step of generating a third audio signal based on a fourth eigenvalue provided in an embodiment of the present application, specifically, comprising steps S510 to S520:

[0069] S510, performing an inverse logarithmic domain transform on the fourth eigenvalue of each time-frequency point to obtain a time-frequency characteristic signal.

[0070] In the embodiment of the present application, by performing an inverse logarithmic domain transform on the fourth eigenvalue of each time-frequency point, a characteristic signal in the complete time-frequency domain space can be obtained.

[0071] S520: Perform inverse transformation on the time-frequency feature signal to obtain the third audio signal.

[0072] In the embodiment of the present application, by further performing an inverse logarithmic transformation on the fourth eigenvalue of each time-frequency point, an audio signal in the time domain space can be finally fused and generated, and the third audio signal is the fusion result of the first audio signal and the second audio signal in the time domain space.

[0073] Specifically, taking the fourth eigenvalue FakeFbank(t,f) at the obtained time-frequency point as an example, by transforming each obtained FakeFbank(t,f) back to the logarithmic domain, the characteristic signal in the complete time-frequency domain space can be obtained, and the final third audio signal can be further generated based on the characteristic signal.

[0074] In addition, it should be noted that the audio augmentation method provided in the embodiment of the present application is particularly suitable for the training process of the speech model. Figure 6 , Figure 6 The schematic flow chart of the steps of implementing the speech model through the third audio signal provided in the embodiment of the present application specifically includes steps S610 to S620:

[0075] S610: Determine a label corresponding to the first audio signal as a label of the third audio signal.

[0076] S620: Train a model according to the third audio signal and the label of the third audio signal.

[0077] In the embodiment of the present application, in combination with the above-mentioned related description, the first audio signal generally refers to a type of clean audio signal. When this type of audio signal is used to train a speech model, an audio label is usually pre-set. For example, more commonly, when used to train a speech classification model, the audio label can be a classification label, such as a description of the type of device that emits the audio signal. Of course, when training other speech models, the first audio signal will also be set with a corresponding label, where the label can usually be obtained by manual annotation, and the accuracy of the label is relatively high.

[0078] Furthermore, since the third audio signal obtained by fusion in the aforementioned manner is obtained by adding a noise signal, that is, the second audio signal, to the first audio signal, the third audio signal can retain most of the features of the original audio signal due to adjustments made on the secondary components. Therefore, it usually has the same label information as the first audio signal. Therefore, the label corresponding to the first audio signal can be determined as the label of the third audio signal, thereby quickly enriching the training samples used for model training without introducing obvious erroneous labeling results, thereby improving the training effect of the model.

[0079] When adopting the scheme of the embodiment of the present application, the first audio signal and the second audio signal are first processed to obtain the first eigenvalue and the second eigenvalue at the first time-frequency point. The first eigenvalue and the second eigenvalue can describe the signal component of the audio signal at the time-frequency point. At this time, by screening out the smaller eigenvalue of the first eigenvalue and the second eigenvalue, that is, screening out the secondary component, and then adjusting the third eigenvalue as the secondary component to obtain the disturbed eigenvalue, and further combining it with the larger eigenvalue, that is, the main component, to generate the fourth eigenvalue of the time-frequency point, so as to retain the feature information at each video point as much as possible, so that the third audio signal generated by the fourth eigenvalue can well integrate the signal characteristics of the first audio signal and the second audio signal. By adjusting the smaller component of the signal at the time-frequency point, the present application can minimize the interference to the time-frequency characteristics caused by the adjustment of the signal component on the basis of ensuring the diversity of the generated augmented audio signal, and ensure that the synthesized signal can still retain the audio characteristics of the original audio signal, thereby facilitating the improvement of the subsequent training effect of the model.

[0080] Specifically, in order to facilitate understanding of the complete implementation process of the audio augmentation method provided in the embodiment of the present application, the following will be combined with the above Figure 1 to Figure 6 The content provides a specific implementation step of an audio augmentation method, which includes the following steps:

[0081] (1) Determine the initial speech model to be trained, for example, a common one may be an Automatic Speech Recognition (ASR) model, and collect training samples for training the initial speech model, which are generally clean audio signals without noise and their corresponding sample labels, and then extract several noisy audio signals from a noise sample database;

[0082] (2) Determine the signal-to-noise ratio of the mixed noise during training. For example, you can randomly sample within the range of [10dB, 30dB] each time to obtain the required signal-to-noise ratio snr of the mixed noise, and then use snr to calculate the mixing ratio of the clean audio signal and the noisy audio signal. Taking the use of the Mel filter bank coefficient Fbank as an example, since the Mel filter bank coefficient uses the amplitude spectrum in the logarithmic domain when extracting frequency domain features, it is necessary to convert the signal-to-noise ratio snr into the exponential domain to obtain the signal weight ratio of the clean audio signal and the noisy audio signal, that is, ratio = exp(snr) / 10. Considering normalization, the signal weight ratio of the clean audio signal α = ratio / (ratio+1) and the weight coefficient of the noisy audio signal β = 1 / (ratio+1) can be further calculated;

[0083] (3) For the clean audio signal s(t) and the noisy audio signal n(t) in the time domain, extract the Mel filter bank coefficients respectively, then perform an inverse transformation on the frequency features in the logarithmic domain, and multiply them by the aforementioned α or β respectively, so as to obtain the time-frequency features S(t, f) of the clean audio signal at each time-frequency point, and the time-frequency features N(t, f) of the noisy audio signal at each time-frequency point;

[0084] (4) Compare the eigenvalues ​​at each time-frequency point to determine the main component and the secondary component, in preparation for the subsequent random perturbation of the secondary component. Specifically, the signal component can be obtained by masking the main component mask = (S (t, f) > = N (t, f)), that is, at a certain time-frequency point, when the time-frequency characteristics of the clean audio signal are greater than the time-frequency characteristics of the noisy audio signal, set mask = 1, otherwise, set mask = 0;

[0085] (5) Using the mask obtained above, when mask=1, it means that the clean audio signal is the main component. At this time, the noise audio signal needs to be disturbed at this time-frequency point, that is, FakeFbank(t,f)=S(t,f)+random_weight(t,f)*N(t,f). On the contrary, when mask=0, it means that the noise audio signal is the main component. At this time, the clean audio signal needs to be disturbed, that is, FakeFbank(t,f)=random_weight(t,f)*S(t,f)+N(t,f), where random_weight is the disturbance weight, and random_weight(t,f) is the disturbance weight at each time-frequency point. These disturbance weights can be randomly generated within a preset range, such as -0.5 to 0.5, to reflect the characteristics of simulating different phase difference mixed noise. Of course, they can also be determined based on the feature difference between the two time-frequency features S(t,f) and N(t,f) at the time-frequency point;

[0086] (6) After determining the time-frequency feature FakeFbank(t,f) at each time-frequency point after the disturbance, the time-frequency feature is transformed back to the logarithmic domain to obtain the time-frequency feature information of the final mixed noise signal, so as to further determine the fused noise augmented signal;

[0087] (7) Due to the random perturbation on the secondary component through the aforementioned processing, the feature similarity between the noise-augmented signal and the clean noise signal is ensured as much as possible on the basis of ensuring the amplitude and phase diversity of the generated noise-augmented signal. That is, the sample label of the clean audio signal can be assigned to the noise-augmented signal and added to the training sample. In the subsequent training sample process, the robustness of the model can be improved. For example, the trained speech recognition model can effectively process the audio signal with noise and obtain accurate speech recognition results.

[0088] In order to better implement the audio augmentation method provided in the embodiment of the present application, the present embodiment also provides an audio augmentation device, which can be integrated in a server. Figure 7 As shown, the audio augmentation device may include:

[0089] The processing module 710 is configured to extract the time-frequency features of the first audio signal to obtain a first eigenvalue of the first audio signal at each time-frequency point, and to extract the time-frequency features of the second audio signal to obtain a second eigenvalue of the second audio signal at each time-frequency point;

[0090] An adjustment module 720 is used to adjust the smaller eigenvalue of the first eigenvalue and the second eigenvalue of each time-frequency point according to a preset coefficient to obtain a third eigenvalue of each time-frequency point;

[0091] The determination module 730 is used to obtain the fourth eigenvalue of each time-frequency point according to the larger eigenvalue of the first eigenvalue and the second eigenvalue of each time-frequency point, and the third eigenvalue.

[0092] The generating module 740 is configured to generate a third audio signal according to the fourth eigenvalue at each time-frequency point.

[0093] In some embodiments of the present application, the generating module 740 is further configured to perform an inverse logarithmic domain transform on the fourth eigenvalue of each time-frequency point to obtain a time-frequency characteristic signal; and perform an inverse transform on the time-frequency characteristic signal to obtain the third audio signal.

[0094] In some embodiments of the present application, the adjustment module 720 is further used to obtain the eigenvalue difference between the first eigenvalue and the second eigenvalue of each time-frequency point; and determine the preset coefficient corresponding to each of the time-frequency points according to the eigenvalue difference.

[0095] In some embodiments of the present application, the processing module 710 is also used to perform frequency domain feature extraction on the first audio signal to obtain time-frequency feature information of the first audio signal at each time-frequency point; determine the first characteristic value of the first audio signal at the first time-frequency point according to the first time-frequency feature information of the first audio signal at the first time-frequency point and the second time-frequency feature information of the first audio signal at the second time-frequency point, wherein the time interval and / or frequency difference between the first time-frequency point and the second time-frequency point meets a preset threshold.

[0096] In some embodiments of the present application, the processing module 710 is also used to determine the weight coefficient corresponding to each of the second time-frequency points based on the time interval and / or frequency difference between the second time-frequency point and the first time-frequency point; according to the weight coefficient corresponding to each of the second time-frequency points, the first time-frequency feature information and multiple second time-frequency feature information are weighted to obtain the first characteristic value of the first audio signal at the first time-frequency point.

[0097] In some embodiments of the present application, the generating module 740 is further configured to determine a label corresponding to the first audio signal as a label of the third audio signal; and train a model according to the third audio signal and the label of the third audio signal.

[0098] In some embodiments of the present application, the processing module 710 is further configured to perform weighted processing on the first audio signal and the second audio signal according to weight information corresponding to multiple preset signal-to-noise ratios to obtain the weighted first audio signal and the second audio signal.

[0099] When adopting the scheme of the embodiment of the present application, the first audio signal and the second audio signal are first processed to obtain the first eigenvalue and the second eigenvalue at the first time-frequency point. The first eigenvalue and the second eigenvalue can describe the signal component of the audio signal at the time-frequency point. At this time, by screening out the smaller eigenvalue of the first eigenvalue and the second eigenvalue, that is, screening out the secondary component, and then adjusting the third eigenvalue as the secondary component to obtain the disturbed eigenvalue, and further combining it with the larger eigenvalue, that is, the main component, to generate the fourth eigenvalue of the time-frequency point, so as to retain the feature information at each video point as much as possible, so that the third audio signal generated by the fourth eigenvalue can well integrate the signal characteristics of the first audio signal and the second audio signal. By adjusting the smaller component of the signal at the time-frequency point, the present application can minimize the interference to the time-frequency characteristics caused by the adjustment of the signal component on the basis of ensuring the diversity of the generated augmented audio signal, and ensure that the synthesized signal can still retain the audio characteristics of the original audio signal, thereby facilitating the improvement of the subsequent training effect of the model.

[0100] Based on the same inventive concept, the embodiment of the present application also provides a computer device, which may be a server or a terminal device, and includes a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the steps of the above-mentioned model evaluation method or model training method are implemented. Various functions are implemented, such as executing the following steps:

[0101] Extracting time-frequency features of the first audio signal to obtain a first eigenvalue of the first audio signal at each time-frequency point, and extracting time-frequency features of the second audio signal to obtain a second eigenvalue of the second audio signal at each time-frequency point;

[0102] The smaller eigenvalue of the first eigenvalue and the second eigenvalue of each time-frequency point is adjusted according to a preset coefficient to obtain a third eigenvalue of each time-frequency point;

[0103] Obtaining a fourth eigenvalue of each time-frequency point according to the larger eigenvalue of the first eigenvalue and the second eigenvalue of each time-frequency point, and the third eigenvalue;

[0104] A third audio signal is generated according to the fourth eigenvalue at each time-frequency point.

[0105] When adopting the scheme of the embodiment of the present application, the first audio signal and the second audio signal are first processed to obtain the first eigenvalue and the second eigenvalue at the first time-frequency point. The first eigenvalue and the second eigenvalue can describe the signal component of the audio signal at the time-frequency point. At this time, by screening out the smaller eigenvalue of the first eigenvalue and the second eigenvalue, that is, screening out the secondary component, and then adjusting the third eigenvalue as the secondary component to obtain the disturbed eigenvalue, and further combining it with the larger eigenvalue, that is, the main component, to generate the fourth eigenvalue of the time-frequency point, so as to retain the feature information at each video point as much as possible, so that the third audio signal generated by the fourth eigenvalue can well integrate the signal characteristics of the first audio signal and the second audio signal. By adjusting the smaller component of the signal at the time-frequency point, the present application can minimize the interference to the time-frequency characteristics caused by the adjustment of the signal component on the basis of ensuring the diversity of the generated augmented audio signal, and ensure that the synthesized signal can still retain the audio characteristics of the original audio signal, thereby facilitating the improvement of the subsequent training effect of the model.

[0106] In one embodiment, the computer device is a terminal device, for example, its internal structure diagram can be as follows: Figure 8 As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. The processor, the memory and the input / output interface are connected via a system bus, and the communication interface, the display unit and the input device are connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, an audio augmentation method is implemented. The display unit of the computer device is used to form a visually visible image, and can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer covered on the display screen, or a button, trackball or touchpad set on the computer device casing, or an external keyboard, touchpad or mouse, etc.

[0107] Those skilled in the art will understand that Figure 8The structure shown is only a block diagram of a portion of the structure related to the present application scheme, and does not constitute a limitation on the computer device to which the present application scheme is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different component arrangement.

[0108] Based on the same inventive concept, an embodiment of the present application also provides a computer-readable storage medium, which may include: a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, etc.

[0109] Since the computer program stored in the computer-readable storage medium can execute any audio augmentation method provided in the embodiments of the present application, the beneficial effects that can be achieved by any audio augmentation method provided in the embodiments of the present application can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0110] When adopting the scheme of the embodiment of the present application, the first audio signal and the second audio signal are first processed to obtain the first eigenvalue and the second eigenvalue at the first time-frequency point. The first eigenvalue and the second eigenvalue can describe the signal component of the audio signal at the time-frequency point. At this time, by screening out the smaller eigenvalue of the first eigenvalue and the second eigenvalue, that is, screening out the secondary component, and then adjusting the third eigenvalue as the secondary component to obtain the disturbed eigenvalue, and further combining it with the larger eigenvalue, that is, the main component, to generate the fourth eigenvalue of the time-frequency point, so as to retain the feature information at each video point as much as possible, so that the third audio signal generated by the fourth eigenvalue can well integrate the signal characteristics of the first audio signal and the second audio signal. By adjusting the smaller component of the signal at the time-frequency point, the present application can minimize the interference to the time-frequency characteristics caused by the adjustment of the signal component on the basis of ensuring the diversity of the generated augmented audio signal, and ensure that the synthesized signal can still retain the audio characteristics of the original audio signal, thereby facilitating the improvement of the subsequent training effect of the model.

[0111] Based on the same inventive concept, the embodiment of the present application further provides a computer program product or a computer program, which includes a computer instruction, and the computer instruction is stored in a computer-readable storage medium. The processor of the computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the method provided in various optional implementations in the above embodiments.

[0112] It should be noted that the object data (including but not limited to user device information, user personal information, etc.) and conversation data involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions. A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods.

[0113] Any reference to the memory, database or other media used in the embodiments provided in the present application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (Re RAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory may include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM may be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0114] The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. The non-relational database may include a distributed database based on blockchain, etc., but is not limited thereto. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but is not limited thereto.

[0115] In the above-mentioned audio augmentation device, computer device, computer-readable storage medium, and computer program product embodiments, the description of each embodiment has its own emphasis. For parts not described in detail in a certain embodiment, reference can be made to the relevant description of other embodiments. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working process and beneficial effects of the above-mentioned audio augmentation device, computer-readable storage medium, computer program product, computer device and its corresponding units can refer to the description of the audio augmentation method in the above embodiment, and will not be repeated here.

[0116] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0117] The above is a detailed introduction to an audio augmentation method, device, computer equipment and storage medium provided in the embodiments of the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for technicians in this field, according to the ideas of the present application, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as limiting the present application.

Claims

1. An audio augmentation method, characterized in that: include: Extracting time-frequency features of the first audio signal to obtain a first eigenvalue of the first audio signal at each time-frequency point, and extracting time-frequency features of the second audio signal to obtain a second eigenvalue of the second audio signal at each time-frequency point; The smaller eigenvalue of the first eigenvalue and the second eigenvalue of each time-frequency point is adjusted according to a preset coefficient to obtain a third eigenvalue of each time-frequency point; Obtaining a fourth eigenvalue of each time-frequency point according to the larger eigenvalue of the first eigenvalue and the second eigenvalue of each time-frequency point, and the third eigenvalue; A third audio signal is generated according to the fourth eigenvalue at each time-frequency point.

2. The method according to claim 1, characterized in that Generating a third audio signal according to the fourth eigenvalue of each time-frequency point includes: Performing an inverse logarithmic domain transform on the fourth eigenvalue of each time-frequency point to obtain a time-frequency characteristic signal; Perform an inverse transformation on the time-frequency characteristic signal to obtain the third audio signal.

3. The method according to claim 1, characterized in that Before the step of adjusting the smaller eigenvalue of the first eigenvalue and the second eigenvalue of each time-frequency point according to the preset coefficient, the method further includes: Obtain the eigenvalue difference between the first eigenvalue and the second eigenvalue of each time-frequency point; The preset coefficient corresponding to each of the time-frequency points is determined according to the eigenvalue difference.

4. The method according to claim 1, characterized in that: The first audio signal is a time domain signal; and extracting the time-frequency feature of the first audio signal to obtain a first feature value of the first audio signal at each time-frequency point includes: Performing frequency domain feature extraction on the first audio signal to obtain time-frequency feature information of the first audio signal at each time-frequency point; Determine a first characteristic value of the first audio signal at the first time-frequency point based on first time-frequency feature information of the first audio signal at a first time-frequency point and second time-frequency feature information of the first audio signal at a second time-frequency point, wherein the time interval and / or frequency difference between the first time-frequency point and the second time-frequency point satisfies a preset threshold.

5. The method according to claim 4, characterized in that The second time-frequency point includes a plurality of points; and determining, according to the first time-frequency feature information of the first audio signal at the first time-frequency point and the second time-frequency feature information of the first audio signal at the second time-frequency point, a first feature value of the first audio signal at the first time-frequency point includes: Determine a weight coefficient corresponding to each of the second time-frequency points according to a time interval and / or a frequency difference between the second time-frequency point and the first time-frequency point; The first time-frequency feature information and a plurality of second time-frequency feature information are weighted according to a weight coefficient corresponding to each second time-frequency point to obtain a first feature value of the first audio signal at the first time-frequency point.

6. The method according to claim 1, characterized in that After generating the third audio signal according to the fourth eigenvalue of each time-frequency point, the method further includes: Determine the label corresponding to the first audio signal as the label of the third audio signal; A model is trained according to the third audio signal and the label of the third audio signal.

7. The method according to any one of claims 1 to 6, characterized in that: The method further comprises: extracting the time-frequency features of the first audio signal to obtain a first eigenvalue of the first audio signal at each time-frequency point, and extracting the time-frequency features of the second audio signal to obtain a second eigenvalue of the second audio signal at each time-frequency point. The first audio signal and the second audio signal are weighted according to weight information corresponding to a plurality of preset signal-to-noise ratios to obtain weighted first audio signals and second audio signals.

8. An audio amplification device, characterized in that: include: a processing module, configured to extract time-frequency features of the first audio signal to obtain a first eigenvalue of the first audio signal at each time-frequency point, and to extract time-frequency features of the second audio signal to obtain a second eigenvalue of the second audio signal at each time-frequency point; An adjustment module, used to adjust the smaller eigenvalue of the first eigenvalue and the second eigenvalue of each time-frequency point according to a preset coefficient to obtain a third eigenvalue of each time-frequency point; A determination module, configured to obtain a fourth eigenvalue of each time-frequency point according to a larger eigenvalue among the first eigenvalue and the second eigenvalue of each time-frequency point, and the third eigenvalue; A generating module is used to generate a third audio signal according to the fourth eigenvalue of each time-frequency point.

9. A computer device, characterized in that: The method comprises a processor and a memory, wherein the memory stores a plurality of instructions; the processor loads instructions from the memory to execute the steps of the audio augmentation method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor to execute the steps of the audio augmentation method according to any one of claims 1 to 7.

11. A computer program product, characterized in that The computer program product comprises a computer program, and the computer program is used by a processor to execute the steps of the audio augmentation method according to any one of claims 1 to 7.

Images