Audio augmentation method, device, computer equipment and storage medium

Abstract

The present application discloses an audio augmentation method, apparatus, computer equipment and storage medium, the method comprising: extracting time-frequency features from a first audio signal and a second audio signal to obtain a first eigenvalue and a second eigenvalue of each time-frequency point; adjusting the smaller eigenvalue of the first eigenvalue and the second eigenvalue to obtain a third eigenvalue; obtaining a fourth eigenvalue of each time-frequency point based on the larger eigenvalue of the first eigenvalue and the second eigenvalue, and the third eigenvalue; and generating a third audio signal based on the fourth eigenvalue of each time-frequency point. The present application compares the eigenvalues ​​of different audio signals at each time-frequency point and determines the secondary components at each time-frequency point for subsequent perturbation. On the basis of ensuring the diversity of the amplitude and phase of the generated signal, the application minimizes the interference of random perturbations on the audio features as much as possible, ensuring that the synthesized signal can still retain the audio features of the original audio signal, thereby improving the subsequent training effect of the model.

Description

Technical Field

[0001] The present application relates to the field of audio signal processing, and in particular to an audio augmentation method, apparatus, computer device, and storage medium. Background Art

[0002] During the training process of a speech recognition model, it is usually necessary to augment the signal, such as noise augmentation, to increase the richness and diversity of training samples so that the model can effectively process noisy input audio, thereby improving the robustness of the model.

[0003] However, related technologies typically use a superposition of clean and noisy audio signals in the time-frequency domain, or apply random perturbations to an audio signal, to generate more diverse augmented signals. However, applying random perturbations to audio signals can easily cause the synthesized augmented signals to lose some of the audio features of the original audio signal, thus affecting the subsequent model training results. Summary of the Invention

[0004] In response to the above technical problems, embodiments of the present application provide an audio augmentation method, apparatus, computer device, and storage medium, aiming to solve the problem of unsatisfactory audio augmentation effects.

[0005] In a first aspect, an embodiment of the present application provides an audio augmentation method, comprising:

[0006] performing time-frequency feature extraction on the first audio signal to obtain a first eigenvalue of the first audio signal at each time-frequency point, and performing time-frequency feature extraction on the second audio signal to obtain a second eigenvalue of the second audio signal at each time-frequency point;

[0007] Adjusting the smaller eigenvalue of the first eigenvalue and the second eigenvalue of each time-frequency point according to a preset coefficient to obtain a third eigenvalue of each time-frequency point;

[0008] Obtaining a fourth eigenvalue of each time-frequency point according to the larger eigenvalue of the first eigenvalue and the second eigenvalue of each time-frequency point, and the third eigenvalue;

[0009] Generate a third audio signal according to the fourth eigenvalue of each time-frequency point

[0010] In a second aspect, an embodiment of the present application provides an audio augmentation device, comprising:

[0011] a processing module, configured to perform time-frequency feature extraction on the first audio signal to obtain a first eigenvalue of the first audio signal at each time-frequency point, and to perform time-frequency feature extraction on the second audio signal to obtain a second eigenvalue of the second audio signal at each time-frequency point;

[0012] an adjustment module, configured to adjust the smaller eigenvalue of the first eigenvalue and the second eigenvalue of each time-frequency point according to a preset coefficient to obtain a third eigenvalue of each time-frequency point;

[0013] The determination module is configured to obtain a fourth eigenvalue of each time-frequency point based on the larger eigenvalue of the first eigenvalue and the second eigenvalue of each time-frequency point, and the third eigenvalue.

[0014] A generating module is configured to generate a third audio signal according to the fourth eigenvalue at each time-frequency point.

[0015] In a third aspect, an embodiment of the present application further provides a computer device, comprising a processor and a memory, wherein the memory stores a plurality of instructions; the processor loads instructions from the memory to execute the audio augmentation steps described above.

[0016] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a plurality of instructions, wherein the instructions are suitable for loading by a processor to execute the audio augmentation steps described above.

[0017] In a fifth aspect, an embodiment of the present application further provides a computer program product, which includes a computer program, and the computer program is executed by a processor to perform the audio augmentation steps described above.

[0018] When using the solution of the embodiment of the present application, the first audio signal and the second audio signal are first processed to obtain the first eigenvalue and the second eigenvalue at the first time-frequency point. The first eigenvalue and the second eigenvalue can describe the signal component of the audio signal at the time-frequency point. At this time, by screening out the smaller eigenvalue of the first eigenvalue and the second eigenvalue, that is, screening out the secondary component, and then adjusting the third eigenvalue as the secondary component to obtain the disturbed eigenvalue, and further combining it with the larger eigenvalue, that is, the main component, to generate the fourth eigenvalue of the time-frequency point, so as to retain the feature information at each video point as much as possible, so that the third audio signal generated by the fourth eigenvalue can better integrate the signal characteristics of the first audio signal and the second audio signal. By adjusting the smaller component of the signal at the time-frequency point, the present application can minimize the interference on the time-frequency characteristics caused by the adjustment of the signal component on the basis of ensuring the diversity of the generated augmented audio signal, and ensure that the synthesized signal can still retain the audio characteristics of the original audio signal, thereby facilitating the improvement of the subsequent training effect of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0020] Figure 1 A schematic flow chart of the steps of an audio augmentation method provided in an embodiment of the present application;

[0021] Figure 2 A schematic flow chart of the steps for fusing time-frequency feature information of adjacent time-frequency points to determine the feature value of each time-frequency point provided in an embodiment of the present application;

[0022] Figure 3 A flowchart of the steps for setting weights for neighboring time-frequency points to determine characteristic values ​​at the time-frequency points provided in an embodiment of the present application;

[0023] Figure 4 A schematic diagram of a flow chart of steps for determining a preset coefficient based on a characteristic value difference provided in an embodiment of the present application;

[0024] Figure 5 A schematic flow chart of steps for generating a third audio signal based on a fourth eigenvalue provided in an embodiment of the present application;

[0025] Figure 6 A schematic flow chart of the steps for implementing a speech model using a third audio signal provided in an embodiment of the present application;

[0026] Figure 7 A schematic structural diagram of an audio amplification device provided in an embodiment of the present application;

[0027] Figure 8 A schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0028] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of this application. At the same time, in the description of the embodiments of the present application, the terms "first", "second", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more features. In the description of the embodiments of the present application, the meaning of "multiple" is two or more, unless otherwise clearly and specifically defined.

[0029] To better understand the audio augmentation method, apparatus, computer device, and storage medium provided in this application, the relevant application background of the audio augmentation method is first described. The audio augmentation method primarily refers to increasing the diversity of training samples during speech recognition model training by augmenting them, for example, by synthesizing them with noise signals to achieve noise augmentation, thereby improving the model's training effectiveness. For example, this allows the model to effectively process noisy input audio, thereby improving the model's robustness.

[0030] In related technologies, a common audio augmentation method is to simply superimpose clean audio signals and noisy audio signals. However, due to the limitation that the amplitude and phase of the synthesized audio signal are only related to the clean audio signal and the noisy audio signal, the diversity of the amplitude and phase of the augmented audio synthesized in this way is limited. In order to improve the diversity of the synthesized audio, related technologies further propose to apply random perturbations to an audio signal in the frequency domain to improve the diversity of the amplitude and phase of the synthesized audio. However, in actual applications, it is found that applying random perturbations to audio signals can easily cause the synthesized augmented signal to lose some of the audio features of the original audio signal, thereby affecting the subsequent training of the model.

[0031] In order to solve the above problems, this application provides an audio augmentation method, device, computer equipment and storage medium. By comparing the features of different audio signals at different time-frequency points and determining the smaller eigenvalue with a lower component at each time-frequency point, the smaller eigenvalue is disturbed in the subsequent random perturbation process to reduce the interference of random perturbation on the audio features and merge it with the larger eigenvalue to ensure that the final synthesized signal can still retain the audio features of the original audio signal, thereby improving the subsequent training effect of the model. For details, please refer to Figure 1 , Figure 1 A flowchart of an audio augmentation method provided in an embodiment of the present application, specifically comprising steps S110 to S140:

[0032] S110: Perform time-frequency feature extraction on the first audio signal to obtain a first eigenvalue of the first audio signal at each time-frequency point, and perform time-frequency feature extraction on the second audio signal to obtain a second eigenvalue of the second audio signal at each time-frequency point.

[0033] In the embodiment of the present application, combined with the above-mentioned relevant description, it can be known that the first audio signal and the second audio signal can be generally understood as the clean audio signal and the noise audio signal required for the audio augmentation process, wherein the definition of the clean audio signal and the noise audio signal is not limited in the embodiment of the present application. Any audio signal that can be synthesized to achieve audio augmentation can be understood as one of the first audio signal and the second audio signal provided in the embodiment of the present application. Specifically, for ease of understanding, taking the model training process as an example, the first audio signal as the clean audio signal can be extracted from the training sample set of the model training, and the second audio signal as the noise audio signal can be read from the noise sample database based on network technology. For the specific implementation scheme, please refer to the relevant explanations of the subsequent embodiments.

[0034] Furthermore, by performing time-frequency feature processing on the first audio signal, the first eigenvalue of the first audio signal at each time-frequency point can be obtained, that is, the first eigenvalue here is usually related to the time-frequency feature of the audio signal. Among them, time-frequency feature processing usually refers to extracting frequency domain features from the audio signal in the time domain. For example, as a common feasible implementation scheme, the time-frequency feature information of the first audio signal can be obtained by extracting frequency domain features from the audio signal in the time domain, which includes the features of the first audio signal at each time-frequency point. The frequency domain feature extraction here can usually be obtained by extracting frequency domain features such as Mel frequency cepstrum coefficients or Mel filter bank coefficients. Of course, other frequency domain feature extractions are also feasible, but considering that the present application requires a certain disturbance change to the extracted time-frequency feature values ​​to synthesize augmented audio, it is found in actual applications that compared with other frequency domain features, frequency domain features such as Mel frequency cepstrum coefficients (MFCC) or Mel filter bank coefficients (Fbank) can achieve better augmentation effects as subsequent feature value comparisons. Therefore, as a feasible implementation scheme of the present application, we can choose to process the signal in the time domain to obtain Mel-frequency cepstral coefficients or Mel-frequency filter bank coefficients as the characteristic values ​​of the signal at the time-frequency point.

[0035] On the basis of the foregoing, by performing similar processing on the second audio signal, the eigenvalue of the second audio signal at each time-frequency point can also be obtained, that is, the time-frequency feature information of the second audio signal can be obtained. On this basis, based on the association between the eigenvalues ​​contained in the time-frequency feature information and the time-frequency points, a correspondence between the eigenvalues ​​of the first audio signal and the second audio signal at each time-frequency point can also be established, so as to compare the first eigenvalue and the second eigenvalue of the first audio signal and the second audio signal at the same time-frequency point.

[0036] Specifically, to facilitate understanding of the above content, taking the first audio signal s(t) and the second audio signal n(t) as an example, after extracting the Mel filter bank coefficients (Fbank) and performing an inverse transform in the logarithmic domain, the time-frequency feature information S(t, f) of the first audio signal and the time-frequency feature information N(t, f) of the second audio signal can be obtained, which respectively include the eigenvalues ​​at each time-frequency point (t, f), thereby facilitating determination of the first eigenvalue of the first audio signal and the second audio value of the second audio signal at the same time-frequency point.

[0037] Of course, in addition to the aforementioned implementation scheme of using the individual characteristic information of the audio signal at each time-frequency point as the characteristic value, that is, in addition to considering the characteristic information of each time-frequency point separately, the characteristic value at each time-frequency point can also be determined by the characteristic information of the first audio signal at multiple time-frequency points within a certain time-frequency neighborhood range. For details, please refer to Figure 2 , Figure 2 A flowchart of the steps for fusing time-frequency feature information at adjacent time-frequency points to determine the feature values ​​at each time-frequency point is provided in an embodiment of the present application. Specifically, in the embodiment of the present application, the first audio signal is used as an example for description. The processing process of the second audio signal can refer to the following, which is not repeated here in this application. Specifically, steps S210 to S220 are included:

[0038] S210: Perform frequency domain feature extraction on the first audio signal to obtain time-frequency feature information of the first audio signal at each time-frequency point.

[0039] S220: Determine a first eigenvalue of the first audio signal at the first time-frequency point based on first time-frequency feature information of the first audio signal at the first time-frequency point and second time-frequency feature information of the first audio signal at the second time-frequency point.

[0040] In this embodiment of the present application, the time interval and / or frequency difference between the first time-frequency point and the second time-frequency point meets a preset threshold.

[0041] Unlike the aforementioned frequency domain feature extraction of the first audio signal and directly using the obtained time-frequency feature information of each time-frequency point as the characteristic value at the time-frequency point, in this embodiment, the first characteristic value of the first audio signal at the first time-frequency point is determined based on the first time-frequency feature information of the first time-frequency point and the second time-frequency feature information of the second time-frequency point, wherein the time interval and / or frequency difference between the first time-frequency point and the second time-frequency point meets the preset threshold, that is, the second time-frequency point can be understood as the time-frequency point of the first time-frequency point in the time-frequency neighborhood, that is, the characteristic value at each time-frequency point is jointly determined by the video feature information at the time-frequency point and the neighboring time-frequency points, thereby avoiding the occurrence of erroneous noise points in the audio signal at a certain time-frequency point, resulting in the screening out of the wrong main component, thereby affecting the subsequent signal fusion effect.

[0042] Of course, in order to further accurately obtain the characteristic values ​​of the audio signal at each time-frequency point to identify the main signal components and secondary signal components at the time-frequency point, the influence of the time-frequency neighborhood on the current time-frequency point will also be fully considered. That is, corresponding weight coefficients can be set for different neighboring time-frequency points of the time-frequency point to weight the time-frequency feature information. For details, please refer to Figure 3 , Figure 3 The present invention provides a flowchart of a step of setting weights for neighboring time-frequency points to determine characteristic values ​​at the time-frequency points, specifically including steps S310 to S320:

[0043] S310: Determine a weight coefficient corresponding to each second time-frequency point according to a time interval and / or a frequency difference between the second time-frequency point and the first time-frequency point.

[0044] In this embodiment of the present application, the larger the time interval and / or frequency difference between the second time-frequency point and the first time-frequency point, the less interference the time-frequency feature information of the second time-frequency point has on the time-frequency feature information of the first time-frequency point. In this case, a smaller weight coefficient can be set for such time-frequency point. Conversely, the smaller the time interval and / or frequency difference between the second time-frequency point and the first time-frequency point, that is, the closer they are in the time-frequency domain, the larger the weight coefficient can be set for such time-frequency point. In addition, the weight of the first time-frequency point can be set to 1 by default.

[0045] S320: Weight the first time-frequency feature information and multiple pieces of second time-frequency feature information according to a weight coefficient corresponding to each second time-frequency point to obtain a first eigenvalue of the first audio signal at the first time-frequency point.

[0046] In the embodiment of the present application, the first time-frequency feature information and the second time-frequency feature information are weighted using the weight coefficient corresponding to each time-frequency point, so as to obtain a first eigenvalue of the first audio signal at the first time-frequency point that more accurately reflects the signal component strength.

[0047] In addition, it should be noted that in the aforementioned process, especially in the noise augmentation process, the synthesized audio signal is usually required to have a certain signal-to-noise ratio. Therefore, before extracting time-frequency feature information from the first audio signal and the second audio signal, the first audio signal and the second audio signal are usually weighted based on a weight coefficient corresponding to the signal-to-noise ratio. That is, extracting the time-frequency features of the first audio signal to obtain the first eigenvalue of the first audio signal at each time-frequency point, and extracting the time-frequency features of the second audio signal to obtain the second eigenvalue of the second audio signal at each time-frequency point, include:

[0048] The first audio signal and the second audio signal are weighted according to weight information corresponding to a plurality of preset signal-to-noise ratios to obtain weighted first audio signals and second audio signals.

[0049] In order to facilitate understanding of the above content, the following will be specifically explained, taking the training model as an example. During training, the signal-to-noise ratio (SNR) of the mixed noise (SNR) can usually be randomly sampled in the range of [10 dB, 30 dB], that is, SNR∈[10 dB, 30 dB].

[0050] Taking into account that the Mel-frequency cepstral coefficients or Mel-filter bank coefficients usually need to use the magnitude spectrum in the logarithmic domain when extracting, it is usually necessary to convert the signal-to-noise ratio snr into the exponential domain, that is, to obtain ratio = exp(snr) / 10. Specifically, the obtained ratio is the signal weight coefficient corresponding to the signal-to-noise ratio, which is used to describe the signal ratio of the first audio signal and the second audio signal. For example, the signal weight ratio of the first audio signal and the second audio signal is ratio: 1. Therefore, the first audio signal and the second audio signal can be weighted according to the signal weight ratio provided above, that is, the weight information, to obtain the weighted first audio signal and the second audio signal for further use in subsequent processing.

[0051] Of course, considering that the sound intensity of the synthesized audio signal is usually changed in the above process, in the actual process of processing the time-frequency feature information, normalization can usually be performed based on the signal weight coefficient to obtain weight coefficients of the first audio signal and the second audio signal, for example, the weight coefficient α of the first audio signal is ratio / (ratio+1), and the weight coefficient β of the second audio signal is 1 / (ratio+1), so as to weight the first audio signal and the second audio signal respectively and obtain weighted results for subsequent processing.

[0052] S120 , adjusting the smaller eigenvalue between the first eigenvalue and the second eigenvalue of each time-frequency point according to a preset coefficient to obtain a third eigenvalue of each time-frequency point.

[0053] Compared with the implementation scheme in the related art by applying random perturbations to the time-frequency characteristics of a signal, in the embodiment of the present application, the smaller eigenvalue is further determined from the first eigenvalue and the second eigenvalue at each time-frequency point, that is, the eigenvalues ​​of the first audio signal and the second audio signal at the time-frequency point respectively, and the smaller eigenvalue is adjusted according to the preset coefficient, so as to obtain a richer augmented signal through random perturbations while minimizing the changes in the audio feature information of the obtained augmented information.

[0054] Furthermore, the scheme for perturbing the smaller eigenvalues ​​at different time-frequency points can be based on a preset fixed perturbation coefficient, for example, the eigenvalue is perturbed at a ratio of 0.1. Of course, the adjustment of the smaller eigenvalue can also be achieved by randomly using a randomly generated preset coefficient for each time-frequency point, wherein the preset coefficient here can be randomly generated between -0.5 and 0.5. Of course, as a further optional implementation scheme of the present application, an implementation scheme for determining the preset coefficient based on the eigenvalue difference between the first eigenvalue and the second eigenvalue is also provided. For details, please refer to Figure 4 , Figure 4 The present invention provides a flowchart of a method for determining a preset coefficient based on a characteristic value difference, specifically including steps S410 to S420:

[0055] S410: Obtain an eigenvalue difference between a first eigenvalue and a second eigenvalue at each time-frequency point.

[0056] In the embodiment of the present application, the eigenvalue difference at each time-frequency point can be obtained by taking the difference between the first eigenvalue and the second eigenvalue at each time-frequency point.

[0057] S420: Determine a preset coefficient corresponding to each of the time-frequency points according to the eigenvalue difference.

[0058] In an embodiment of the present application, the preset coefficient corresponding to each time-frequency point is determined based on the eigenvalue difference between the first eigenvalue and the second eigenvalue. For example, as a common feasible implementation scheme, the eigenvalue difference and the preset coefficient can be positively correlated. For example, when the eigenvalue difference between the first eigenvalue and the second eigenvalue is large, that is, when the gap between the main component and the secondary component in the signal is too large, a larger preset coefficient can be set. For example, it can be set to 0.4 to make the secondary component more adjusted, thereby improving the richness of the audio signal generated by subsequent fusion without affecting the main component. Conversely, when the eigenvalue difference between the first eigenvalue and the second eigenvalue is small, a smaller preset coefficient can be set, such as 0.1, to avoid adjusting the secondary component more and affecting the main component, so that the fused audio signal retains the characteristic information of the original audio signal as much as possible.

[0059] S130 , obtaining a fourth eigenvalue of each time-frequency point according to the larger eigenvalue of the first eigenvalue and the second eigenvalue of each time-frequency point, and the third eigenvalue.

[0060] In an embodiment of the present application, after completing the perturbation of the smaller eigenvalue and obtaining the third eigenvalue as mentioned above, the larger eigenvalue of the first eigenvalue and the second eigenvalue of each time-frequency point can be further combined with the third eigenvalue to obtain the fourth eigenvalue of each time-frequency point, so that the fourth eigenvalue of each time-frequency point retains the audio signal characteristics of the main signal component and part of the audio signal characteristics of the secondary signal component.

[0061] Specifically, to facilitate understanding of the above content, the following will describe the perturbation process of the eigenvalues ​​in the form of an audio signal feature mask to obtain the final feature information. Detailed description is as follows.

[0062] In an embodiment of the present application, by comparing the eigenvalues ​​at each time-frequency point, the main component and the secondary component can be determined, thereby obtaining a mask at each time-frequency point. For example, when the first eigenvalue is greater than the second eigenvalue, the mask can be set to 1; conversely, when the first eigenvalue is less than or equal to the second eigenvalue, the mask can be set to 0.

[0063] Furthermore, when the mask is 1, it indicates that the first audio signal is the main component at this time-frequency point. In this case, the eigenvalue of the second audio signal will be perturbed. Finally, the fourth eigenvalue FakeFbank(t,f) obtained at this time-frequency point satisfies FakeFbank(t,f)=S(t,f)+random_weight(t,f)*N(t,f);

[0064] When the mask is 0, it means that the second audio signal is the main component at this time-frequency point. At this time, the audio eigenvalue of the first audio signal will be perturbed. Finally, the fused eigenvalue FakeFbank(t,f) obtained at this time-frequency point satisfies FakeFbank(t,f)=random_weight(t,f)*S(t,f)+N(t,f).

[0065] Among them, FakeFbank(t,f) is the fourth eigenvalue after fusion, S(t,f) represents the first eigenvalue of the first audio signal S at the time-frequency point (t,f), N(t,f) represents the second eigenvalue of the second audio signal N at the time-frequency point (t,f), and random_weight(t,f) represents the preset coefficient at the time-frequency point, which can be a random value or obtained based on the difference between the first eigenvalue S(t,f) and the second eigenvalue N(t,f).

[0066] S140: Generate a third audio signal according to the fourth eigenvalue at each time-frequency point.

[0067] In an embodiment of the present application, in the aforementioned fourth eigenvalue of the audio signal features that retains the main signal component and part of the audio signal features of the secondary signal component, the fourth eigenvalue in the time-frequency domain is converted to the time domain space by performing an inverse transform. The third audio signal obtained in the time domain space is the audio signal obtained by fusing the time-frequency features of the first audio signal and the time-frequency features of the second audio signal, and by changing the audio features of the secondary component, the richness of the generated audio signal in phase and amplitude is ensured, while the audio features of the main component are retained as much as possible, thereby improving the application value of the fused audio signal in the subsequent model training process.

[0068] For details, please refer to Figure 5 , Figure 5 A schematic flow chart of a step of generating a third audio signal based on a fourth eigenvalue provided in an embodiment of the present application, specifically comprising steps S510 to S520:

[0069] S510 , performing an inverse logarithmic domain transform on the fourth eigenvalue of each time-frequency point to obtain a time-frequency characteristic signal.

[0070] In the embodiment of the present application, by performing an inverse logarithmic domain transform on the fourth eigenvalue of each time-frequency point, a characteristic signal in the complete time-frequency domain space can be obtained.

[0071] S520: Perform inverse transformation on the time-frequency feature signal to obtain the third audio signal.

[0072] In the embodiment of the present application, by further performing an inverse logarithmic transformation on the fourth eigenvalue of each time-frequency point, an audio signal in the time domain space can be finally fused and generated. The third audio signal is the fusion result of the first audio signal and the second audio signal in the time domain space.

[0073] Specifically, taking the fourth eigenvalue FakeFbank(t,f) obtained at the time-frequency point as an example, by transforming each obtained FakeFbank(t,f) back to the logarithmic domain, the characteristic signal in the complete time-frequency domain space can be obtained, and the final third audio signal can be further generated based on the characteristic signal.

[0074] In addition, it should be noted that the audio augmentation method provided in the embodiment of the present application is particularly suitable for the training process of the speech model. Figure 6 , Figure 6 The flowchart of the steps for implementing the speech model through the third audio signal provided in the embodiment of the present application specifically includes steps S610 to S620:

[0075] S610: Determine a label corresponding to the first audio signal as a label of the third audio signal.

[0076] S620: Train a model according to the third audio signal and the label of the third audio signal.

[0077] In an embodiment of the present application, in combination with the aforementioned related description, the first audio signal generally refers to a type of clean audio signal. When this type of audio signal is used to train a speech model, an audio label is usually pre-set. For example, more commonly, when used to train a speech classification model, the audio label can be a classification label, such as a description of the device that emits the audio signal. Of course, when training other speech models, the first audio signal will also be correspondingly labeled, where the label can usually be obtained by manual annotation, and the accuracy of the label is relatively high.

[0078] Furthermore, since the third audio signal obtained by fusion in the aforementioned manner is obtained by adding a noise signal, that is, the second audio signal, to the first audio signal, the third audio signal can retain most of the characteristics of the original audio signal due to adjustments made on the secondary components. Therefore, it usually has the same label information as the first audio signal. Therefore, the label corresponding to the first audio signal can be determined as the label of the third audio signal, thereby quickly enriching the training samples used for model training without introducing obvious erroneous labeling results, thereby improving the training effect of the model.

[0079] When using the solution of the embodiment of the present application, the first audio signal and the second audio signal are first processed to obtain the first eigenvalue and the second eigenvalue at the first time-frequency point. The first eigenvalue and the second eigenvalue can describe the signal component of the audio signal at the time-frequency point. At this time, by screening out the smaller eigenvalue of the first eigenvalue and the second eigenvalue, that is, screening out the secondary component, and then adjusting the third eigenvalue as the secondary component to obtain the disturbed eigenvalue, and further combining it with the larger eigenvalue, that is, the main component, to generate the fourth eigenvalue of the time-frequency point, so as to retain the feature information at each video point as much as possible, so that the third audio signal generated by the fourth eigenvalue can better integrate the signal features of the first audio signal and the second audio signal. By adjusting the smaller component of the signal at the time-frequency point, the present application can minimize the interference with the time-frequency features caused by the adjustment of the signal component on the basis of ensuring the diversity of the generated augmented audio signal, and ensure that the synthesized signal can still retain the audio features of the original audio signal, thereby facilitating the improvement of the subsequent training effect of the model.

[0080] Specifically, in order to facilitate understanding of the complete implementation process of the audio augmentation method provided in the embodiment of the present application, the following will be combined with the aforementioned Figures 1 to 6 The content provides a specific implementation step of an audio augmentation method, which includes the following steps:

[0081] (1) Determine the initial speech model to be trained, for example, the more common one may be an Automatic Speech Recognition (ASR) model, and collect training samples for training the initial speech model, which are generally clean audio signals without noise and their corresponding sample labels, and then extract several noisy audio signals from the noise sample database;

[0082] (2) Determine the signal-to-noise ratio of the mixed noise during training. For example, you can randomly sample within the range of [10dB, 30dB] each time to obtain the signal-to-noise ratio snr of the required mixed noise, and then calculate the mixing ratio of the clean audio signal and the noisy audio signal through snr. Taking the use of the Mel filter bank coefficient Fbank as an example, since the Mel filter bank coefficient uses the amplitude spectrum in the logarithmic domain when extracting frequency domain features, it is necessary to convert the signal-to-noise ratio snr into the exponential domain to obtain the signal weight ratio of the clean audio signal and the noisy audio signal, that is, ratio = exp(snr) / 10. Considering normalization, we can further calculate the signal weight ratio of the clean audio signal α = ratio / (ratio+1), and the weight coefficient of the noisy audio signal β = 1 / (ratio+1);

[0083] (3) For the clean audio signal s(t) and the noisy audio signal n(t) in the time domain, extract the Mel filter bank coefficients respectively, then perform an inverse transform on the frequency features in the logarithmic domain, and multiply them by the aforementioned α or β, thereby obtaining the time-frequency features S(t, f) of the clean audio signal at each time-frequency point, and the time-frequency features N(t, f) of the noisy audio signal at each time-frequency point;

[0084] (4) Compare the eigenvalues ​​at each time-frequency point to determine the main component and the secondary component, in preparation for the subsequent random perturbation of the secondary component. Specifically, the signal component can be obtained by obtaining the mask mask = (S(t, f)> = N(t, f)) of the main component. That is, at a certain time-frequency point, when the time-frequency characteristics of the clean audio signal are greater than the time-frequency characteristics of the noisy audio signal, set the mask mask = 1, otherwise, set the mask mask = 0;

[0085] (5) Using the mask mask obtained above, when mask mask = 1, it indicates that the clean audio signal is the main component. At this time, the noise audio signal needs to be perturbed at this time-frequency point, that is, FakeFbank(t,f) = S(t,f) + random_weight(t,f) * N(t,f). Conversely, when mask mask = 0, it indicates that the noise audio signal is the main component. At this time, the clean audio signal needs to be perturbed, that is, FakeFbank(t,f) = random_weight(t,f) * S(t,f) + N(t,f), where random_weight is the perturbation weight, and random_weight(t,f) is the perturbation weight at each time-frequency point. These perturbation weights can be randomly generated within a preset range, such as -0.5 to 0.5, to reflect the characteristics of simulating different phase difference noise. Of course, they can also be determined based on the feature difference between the two time-frequency features S(t,f) and N(t,f) at the time-frequency point.

[0086] (6) After determining the time-frequency feature FakeFbank(t,f) at each time-frequency point after the disturbance, the time-frequency feature is transformed back to the logarithmic domain to obtain the time-frequency feature information of the final mixed noise signal, so as to further determine the noise augmented signal after fusion;

[0087] (7) Due to the random perturbation on the secondary component through the above-mentioned processing, the feature similarity between the noise-augmented signal and the clean noise signal is guaranteed as much as possible on the basis of ensuring the amplitude and phase diversity of the generated noise-augmented signal. That is, the sample label of the clean audio signal can be assigned to the noise-augmented signal and added to the training sample, so that in the subsequent training sample process, the robustness of the model can be improved. For example, the trained speech recognition model can effectively process the audio signal with noise and obtain accurate speech recognition results.

[0088] In order to better implement the audio augmentation method provided in the embodiment of the present application, the present embodiment also provides an audio augmentation device, which can be integrated into a server. Figure 7 As shown, the audio augmentation device may include:

[0089] a processing module 710 configured to perform time-frequency feature extraction on the first audio signal to obtain a first eigenvalue of the first audio signal at each time-frequency point, and to perform time-frequency feature extraction on the second audio signal to obtain a second eigenvalue of the second audio signal at each time-frequency point;

[0090] An adjustment module 720 is configured to adjust the smaller eigenvalue between the first eigenvalue and the second eigenvalue of each time-frequency point according to a preset coefficient to obtain a third eigenvalue of each time-frequency point;

[0091] The determination module 730 is configured to obtain a fourth eigenvalue of each time-frequency point according to the larger eigenvalue of the first eigenvalue and the second eigenvalue of each time-frequency point, and the third eigenvalue.

[0092] The generating module 740 is configured to generate a third audio signal according to the fourth eigenvalue of each time-frequency point.

[0093] In some embodiments of the present application, the generating module 740 is further configured to perform an inverse logarithmic domain transform on the fourth eigenvalue of each time-frequency point to obtain a time-frequency feature signal; and perform an inverse transform on the time-frequency feature signal to obtain the third audio signal.

[0094] In some embodiments of the present application, the adjustment module 720 is further configured to obtain an eigenvalue difference between a first eigenvalue and a second eigenvalue of each time-frequency point; and determine a preset coefficient corresponding to each time-frequency point based on the eigenvalue difference.

[0095] In some embodiments of the present application, the processing module 710 is further used to perform frequency domain feature extraction on the first audio signal to obtain time-frequency feature information of the first audio signal at each time-frequency point; based on the first time-frequency feature information of the first audio signal at the first time-frequency point and the second time-frequency feature information of the first audio signal at the second time-frequency point, determine the first feature value of the first audio signal at the first time-frequency point, wherein the time interval and / or frequency difference between the first time-frequency point and the second time-frequency point meets a preset threshold.

[0096] In some embodiments of the present application, the processing module 710 is further used to determine the weight coefficient corresponding to each second time-frequency point based on the time interval and / or frequency difference between the second time-frequency point and the first time-frequency point; according to the weight coefficient corresponding to each second time-frequency point, the first time-frequency feature information and multiple second time-frequency feature information are weighted to obtain the first feature value of the first audio signal at the first time-frequency point.

[0097] In some embodiments of the present application, the generating module 740 is further configured to determine the label corresponding to the first audio signal as the label of the third audio signal; and train a model according to the third audio signal and the label of the third audio signal.

[0098] In some embodiments of the present application, the processing module 710 is further configured to perform weighted processing on the first audio signal and the second audio signal according to weight information corresponding to multiple preset signal-to-noise ratios to obtain weighted first audio signal and second audio signal.

[0099] When using the solution of the embodiment of the present application, the first audio signal and the second audio signal are first processed to obtain the first eigenvalue and the second eigenvalue at the first time-frequency point. The first eigenvalue and the second eigenvalue can describe the signal component of the audio signal at the time-frequency point. At this time, by screening out the smaller eigenvalue of the first eigenvalue and the second eigenvalue, that is, screening out the secondary component, and then adjusting the third eigenvalue as the secondary component to obtain the disturbed eigenvalue, and further combining it with the larger eigenvalue, that is, the main component, to generate the fourth eigenvalue of the time-frequency point, so as to retain the feature information at each video point as much as possible, so that the third audio signal generated by the fourth eigenvalue can better integrate the signal features of the first audio signal and the second audio signal. By adjusting the smaller component of the signal at the time-frequency point, the present application can minimize the interference with the time-frequency features caused by the adjustment of the signal component on the basis of ensuring the diversity of the generated augmented audio signal, and ensure that the synthesized signal can still retain the audio features of the original audio signal, thereby facilitating the improvement of the subsequent training effect of the model.

[0100] Based on the same inventive concept, an embodiment of the present application further provides a computer device, which may be a server or terminal device. The computer device includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, it implements the steps of the above-mentioned model evaluation method or model training method. This implements various functions, such as executing the following steps:

[0101] performing time-frequency feature extraction on the first audio signal to obtain a first eigenvalue of the first audio signal at each time-frequency point, and performing time-frequency feature extraction on the second audio signal to obtain a second eigenvalue of the second audio signal at each time-frequency point;

[0102] Adjusting the smaller eigenvalue of the first eigenvalue and the second eigenvalue of each time-frequency point according to a preset coefficient to obtain a third eigenvalue of each time-frequency point;

[0103] Obtaining a fourth eigenvalue of each time-frequency point according to the larger eigenvalue of the first eigenvalue and the second eigenvalue of each time-frequency point, and the third eigenvalue;

[0104] A third audio signal is generated according to the fourth eigenvalue of each time-frequency point.

[0105] When using the solution of the embodiment of the present application, the first audio signal and the second audio signal are first processed to obtain the first eigenvalue and the second eigenvalue at the first time-frequency point. The first eigenvalue and the second eigenvalue can describe the signal component of the audio signal at the time-frequency point. At this time, by screening out the smaller eigenvalue of the first eigenvalue and the second eigenvalue, that is, screening out the secondary component, and then adjusting the third eigenvalue as the secondary component to obtain the disturbed eigenvalue, and further combining it with the larger eigenvalue, that is, the main component, to generate the fourth eigenvalue of the time-frequency point, so as to retain the feature information at each video point as much as possible, so that the third audio signal generated by the fourth eigenvalue can better integrate the signal features of the first audio signal and the second audio signal. By adjusting the smaller component of the signal at the time-frequency point, the present application can minimize the interference with the time-frequency features caused by the adjustment of the signal component on the basis of ensuring the diversity of the generated augmented audio signal, and ensure that the synthesized signal can still retain the audio features of the original audio signal, thereby facilitating the improvement of the subsequent training effect of the model.

[0106] In one embodiment, the computer device is a terminal device, for example, its internal structure diagram can be as follows Figure 8 As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. The processor, the memory and the input / output interface are connected via a system bus, and the communication interface, the display unit and the input device are connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, an audio augmentation method is implemented. The display unit of the computer device is used to form a visually visible image, and can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device casing, or an external keyboard, touchpad or mouse, etc.

[0107] Those skilled in the art will understand that Figure 8The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different component arrangement.

[0108] Based on the same inventive concept, an embodiment of the present application also provides a computer-readable storage medium, which may include: a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, etc.

[0109] Since the computer program stored in the computer-readable storage medium can execute any of the audio augmentation methods provided in the embodiments of the present application, the beneficial effects that can be achieved by any of the audio augmentation methods provided in the embodiments of the present application can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0110] When using the solution of the embodiment of the present application, the first audio signal and the second audio signal are first processed to obtain the first eigenvalue and the second eigenvalue at the first time-frequency point. The first eigenvalue and the second eigenvalue can describe the signal component of the audio signal at the time-frequency point. At this time, by screening out the smaller eigenvalue of the first eigenvalue and the second eigenvalue, that is, screening out the secondary component, and then adjusting the third eigenvalue as the secondary component to obtain the disturbed eigenvalue, and further combining it with the larger eigenvalue, that is, the main component, to generate the fourth eigenvalue of the time-frequency point, so as to retain the feature information at each video point as much as possible, so that the third audio signal generated by the fourth eigenvalue can better integrate the signal features of the first audio signal and the second audio signal. By adjusting the smaller component of the signal at the time-frequency point, the present application can minimize the interference with the time-frequency features caused by the adjustment of the signal component on the basis of ensuring the diversity of the generated augmented audio signal, and ensure that the synthesized signal can still retain the audio features of the original audio signal, thereby facilitating the improvement of the subsequent training effect of the model.

[0111] Based on the same inventive concept, embodiments of the present application further provide a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods provided in various optional implementations of the above embodiments.

[0112] It should be noted that the object data (including but not limited to user device information, user personal information, etc.) and conversation data involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions. Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods.

[0113] Any reference to the memory, database or other media used in the various embodiments provided herein may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory may include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0114] The databases involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processors involved in the various embodiments provided herein may include, but are not limited to, general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic units, data processing logic units based on quantum computing, and the like.

[0115] In the aforementioned embodiments of the audio augmentation device, computer device, computer-readable storage medium, and computer program product, the descriptions of each embodiment have their respective emphases. For portions not detailed in a particular embodiment, reference can be made to the relevant descriptions of other embodiments. Those skilled in the art will clearly understand that, for ease and brevity of description, the specific operating processes and beneficial effects of the aforementioned audio augmentation device, computer-readable storage medium, computer program product, computer device, and their corresponding units can be referred to in the description of the audio augmentation method in the aforementioned embodiments, and will not be further elaborated here.

[0116] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0117] The above describes in detail an audio augmentation method, apparatus, computer device, and storage medium provided in the embodiments of the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is intended only to help understand the method and core concept of the present application. Furthermore, those skilled in the art may vary the specific implementation methods and scope of application based on the concepts of the present application. In summary, the contents of this specification should not be construed as limiting the present application.

Claims

1. An audio augmentation method, characterized in that: include: performing time-frequency feature extraction on the first audio signal to obtain a first eigenvalue of the first audio signal at each time-frequency point, and performing time-frequency feature extraction on the second audio signal to obtain a second eigenvalue of the second audio signal at each time-frequency point; Adjusting the smaller eigenvalue of the first eigenvalue and the second eigenvalue of each time-frequency point according to a preset coefficient to obtain a third eigenvalue of each time-frequency point; Obtaining a fourth eigenvalue of each time-frequency point according to the larger eigenvalue of the first eigenvalue and the second eigenvalue of each time-frequency point, and the third eigenvalue; A third audio signal is generated according to the fourth eigenvalue of each time-frequency point.

2. The method according to claim 1, characterized in that Generating a third audio signal according to the fourth eigenvalue of each time-frequency point includes: Performing an inverse logarithmic domain transform on the fourth eigenvalue of each time-frequency point to obtain a time-frequency characteristic signal; Perform an inverse transform on the time-frequency feature signal to obtain the third audio signal.

3. The method according to claim 1, characterized in that Before the step of adjusting the smaller eigenvalue of the first eigenvalue and the second eigenvalue of each time-frequency point according to the preset coefficient, the method further includes: Obtain the eigenvalue difference between the first eigenvalue and the second eigenvalue of each time-frequency point; The preset coefficient corresponding to each of the time-frequency points is determined according to the eigenvalue difference.

4. The method according to claim 1, wherein The first audio signal is a time domain signal; and extracting time-frequency features from the first audio signal to obtain first eigenvalues ​​of the first audio signal at each time-frequency point includes: Performing frequency domain feature extraction on the first audio signal to obtain time-frequency feature information of the first audio signal at each time-frequency point; Determine, based on first time-frequency feature information of the first audio signal at a first time-frequency point, and second time-frequency feature information of the first audio signal at a second time-frequency point, a first eigenvalue of the first audio signal at the first time-frequency point, wherein a time interval and / or a frequency difference between the first time-frequency point and the second time-frequency point satisfies a preset threshold.

5. The method according to claim 4, characterized in that The second time-frequency points include a plurality of points; and determining, based on the first time-frequency feature information of the first audio signal at the first time-frequency point and the second time-frequency feature information of the first audio signal at the second time-frequency point, a first eigenvalue of the first audio signal at the first time-frequency point includes: determining a weight coefficient corresponding to each second time-frequency point according to a time interval and / or a frequency difference between the second time-frequency point and the first time-frequency point; The first time-frequency feature information and a plurality of second time-frequency feature information are weighted according to a weight coefficient corresponding to each second time-frequency point to obtain a first eigenvalue of the first audio signal at the first time-frequency point.

6. The method according to claim 1, characterized in that After generating the third audio signal according to the fourth eigenvalue at each time-frequency point, the method further includes: determining a label corresponding to the first audio signal as a label of the third audio signal; A model is trained based on the third audio signal and the label of the third audio signal.

7. The method according to any one of claims 1 to 6, characterized in that The method further comprises: extracting the time-frequency features of the first audio signal to obtain a first eigenvalue of the first audio signal at each time-frequency point, and extracting the time-frequency features of the second audio signal to obtain a second eigenvalue of the second audio signal at each time-frequency point. The first audio signal and the second audio signal are weighted according to weight information corresponding to a plurality of preset signal-to-noise ratios to obtain weighted first audio signals and second audio signals.

8. An audio amplification device, characterized in that: include: a processing module, configured to perform time-frequency feature extraction on the first audio signal to obtain a first eigenvalue of the first audio signal at each time-frequency point, and to perform time-frequency feature extraction on the second audio signal to obtain a second eigenvalue of the second audio signal at each time-frequency point; an adjustment module, configured to adjust the smaller eigenvalue of the first eigenvalue and the second eigenvalue of each time-frequency point according to a preset coefficient to obtain a third eigenvalue of each time-frequency point; a determination module, configured to obtain a fourth eigenvalue of each time-frequency point based on the larger eigenvalue of the first eigenvalue and the second eigenvalue of each time-frequency point, and the third eigenvalue; A generating module is configured to generate a third audio signal according to the fourth eigenvalue at each time-frequency point.

9. A computer device, characterized in that: The system comprises a processor and a memory, wherein the memory stores a plurality of instructions; the processor loads instructions from the memory to execute the steps of the audio augmentation method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor to execute the steps of the audio augmentation method according to any one of claims 1 to 7.

11. A computer program product, characterized in that The computer program product comprises a computer program, and the computer program is used by a processor to execute the steps of the audio augmentation method according to any one of claims 1 to 7.

Images