A discrete audio feature generation method and device, and an audio data tokenizer training method and device

Abstract

The application provides a discrete audio feature generation method and device, and an audio data tokenizer training method and device. The discrete audio feature generation method is implemented based on an audio data tokenizer. A vector quantization module includes a plurality of codebooks, and each codebook includes a plurality of codebook vectors. The generation method includes: inputting initial audio data into the encoder to obtain an acoustic feature vector; and for each piece of initial audio data, matching a target codebook vector from the vector quantization module based on the acoustic feature vector to obtain the target codebook vector as a discrete audio feature corresponding to the initial audio data. In this way, different target codebook vectors are adaptively selected as discrete audio features according to acoustic feature vectors of initial audio data, resource waste or insufficient quality caused by a traditional fixed codebook can be better balanced, and the generation quality of subsequent audio generation using the discrete audio feature is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of audio processing, in particular to a discrete audio feature generation method and device, and an audio data tokenizer training method and device. BACKGROUND

[0002] With the rapid development of deep learning technology, the field of audio signal processing has made significant progress. Audio tokenizers, as key components, are widely used in speech-related tasks. Traditional audio tokenizers are mainly based on vector quantization (VQ) technology, and the core goal is to achieve high-quality signal reconstruction by discretizing continuous audio signals into acoustic unit sequences.

[0003] In the prior art, the vector quantization process is usually implemented through a codebook. Existing codebook designs usually adopt a fixed strategy: the same set of codebooks is forced to be applied to different types of audio (such as pure speech, noisy speech, music, etc.), while the acoustic feature distributions of different types of audio differ significantly. This quantization method can lead to a decrease in audio reconstruction quality, further restricting the generalization ability of audio tokenizers in multi-modal tasks. SUMMARY

[0004] Therefore, the purpose of the present application is to provide a discrete audio feature generation method and device, and an audio data tokenizer training method and device. The vector quantization module includes multiple codebooks, each codebook includes multiple codebook vectors, and different target codebook vectors are adaptively selected as discrete audio features according to the acoustic feature vectors of the initial audio data. This can better balance resources and improve the generation quality of subsequent audio generation using discrete audio features, solving the problem of resource waste or insufficient quality caused by traditional fixed codebooks.

[0005] In a first aspect, the embodiments of the present application provide a discrete audio feature generation method, which is implemented based on an audio data tokenizer. The audio data tokenizer includes an encoder and a vector quantization module. The vector quantization module includes multiple codebooks, each codebook including multiple codebook vectors. The discrete audio feature generation method includes:

[0006] inputting initial audio data into the encoder to obtain acoustic feature vectors;

[0007] For each piece of initial audio data, a target codebook vector is matched from the vector quantization module based on the acoustic feature vector, and is used as the discrete audio feature corresponding to the initial audio data.

[0008] Furthermore, an audio feature extraction module is also included between the encoder and the vector quantization module, and the discrete audio feature generation method further includes:

[0009] The acoustic feature vector is input into the audio feature extraction module to obtain the audio statistics corresponding to the initial audio data;

[0010] Based on the first preset matching rule and the audio statistics, a target codebook that matches the initial audio data is determined.

[0011] In the target codebook, a codebook vector matching the acoustic feature vector is determined based on a second preset matching rule, which serves as the discrete audio feature corresponding to the initial audio data.

[0012] Furthermore, the vector quantization module includes a gating module;

[0013] The first preset matching rule includes: inputting the audio statistics into the gating module, the gating module determining the matching probability of each codebook with the acoustic feature vector based on the value of the audio statistics, and taking the codebook with the highest matching probability among multiple codebooks as the target codebook.

[0014] Furthermore, the step of determining the codebook vector matching the acoustic feature vector based on the second preset matching rule, as the audio feature corresponding to the initial audio data, includes:

[0015] Calculate the spatial distance between the acoustic feature vector and all codebook vectors in the target codebook;

[0016] Obtain the codebook vector with the smallest spatial distance among multiple codebook vectors, and use this codebook vector as the discrete audio feature corresponding to the initial audio data.

[0017] Secondly, embodiments of this application also provide a training method for an audio data segmenter. The training method is used to train the audio data segmenter, which is used to implement a discrete audio feature generation method. The training method includes:

[0018] The audio sample data is input into the encoder to obtain the acoustic sample feature vector;

[0019] Based on the vector quantization module, the acoustic feature vector is transformed into discrete audio sample features;

[0020] The discrete audio sample features are input into the decoder to obtain the reconstructed audio data;

[0021] Establish at least one loss function, and iteratively train at least one parameter of the audio data segmenter and the decoder based on the result of the at least one loss function.

[0022] Furthermore, establishing at least one loss function includes:

[0023] Obtain the corresponding text sample data of the audio sample data, convert the text sample data into a discrete text feature vector, and establish a first loss function between the discrete audio sample features and the discrete text feature vector;

[0024] A second loss function is established between the acoustic sample feature vector and the discrete audio sample features;

[0025] A third loss function and a fourth loss function are established between the audio sample data and the reconstructed audio data;

[0026] Based on a weighted fusion strategy, the first loss function, the second loss function, the third loss function, and the fourth loss function are fused together to obtain a fusion loss function. Based on the result of the fusion loss function, at least one parameter of the audio data segmenter and the decoder is iteratively trained.

[0027] Furthermore, the weighted fusion strategy includes:

[0028] A first weight, a second weight, a third weight, and a fourth weight are respectively configured for the first loss function, the second loss function, the third loss function, and the fourth loss function, wherein the first weight, the second weight, the third weight, and the fourth weight are learnable weight parameters;

[0029] The iterative process of the learnable weight parameters includes:

[0030] Obtain the rate of change of loss for each loss function per unit time, and dynamically adjust the corresponding weights based on the rate of change of loss for each loss function.

[0031] Furthermore, the training phase of each codebook in the vector quantization module includes:

[0032] At the start of training, the acoustic sample feature vectors are allowed to be mapped to at least two codebook vectors from at least two codebooks; during training, the gating function is gradually adjusted so that the output vector of the gating function approaches the one-hot vector.

[0033] Furthermore, the gradual adjustment of the gating function to make its output vector approach the one-hot vector includes:

[0034] The gating function includes a temperature parameter, which is preset to a maximum value so that the matching probability of the acoustic sample feature vector with each codebook and each codebook vector is approximately the same when training begins, thereby optimizing all codebook vectors.

[0035] The temperature parameter is gradually reduced to make the matching probability of different codebook vectors differ, until the temperature parameter approaches 0, at which point the output vector of the gating function approaches the one-hot vector.

[0036] Thirdly, this application also provides a discrete audio feature generation device, which is implemented based on an audio data segmenter. The audio data segmenter includes an encoder and a vector quantization module. The vector quantization module includes multiple codebooks, each codebook including multiple codebook vectors. The discrete audio feature generation device includes:

[0037] An acoustic feature vector determination module is used to input initial audio data into the encoder to obtain acoustic feature vectors;

[0038] The discrete audio feature generation module is used to obtain the target codebook vector from the vector quantization module based on the acoustic feature vector for each initial audio data, and use it as the discrete audio feature corresponding to the initial audio data.

[0039] Fourthly, embodiments of this application also provide a training apparatus for an audio data segmenter. The training apparatus is used to train the audio data segmenter, which is used to implement a discrete audio feature generation method. The training apparatus includes:

[0040] The sample feature vector determination module is used to input audio sample data into the encoder to obtain acoustic sample feature vectors.

[0041] The audio sample feature determination module is used to convert the acoustic feature vector into discrete audio sample features based on the vector quantization module.

[0042] The audio reconstruction module is used to input the features of the discrete audio samples into the decoder to obtain reconstructed audio data.

[0043] The model training module is used to establish at least one loss function and to iteratively train at least one parameter of the audio data segmenter and the decoder based on the result of the at least one loss function.

[0044] This application provides a discrete audio feature generation method and apparatus, and an audio data segmenter training method and apparatus. The discrete audio feature generation method is implemented based on an audio data segmenter, which includes an encoder and a vector quantization module. The vector quantization module includes multiple codebooks, and each codebook includes multiple codebook vectors. First, initial audio data is input into the encoder to obtain acoustic feature vectors. Then, for each piece of initial audio data, a target codebook vector is matched from the vector quantization module based on the acoustic feature vector, which serves as the discrete audio feature corresponding to the initial audio data.

[0045] In this way, since the vector quantization module includes multiple codebooks, and each codebook includes multiple codebook vectors, different target codebook vectors can be adaptively selected as discrete audio features based on the acoustic feature vectors of the initial audio data. This can better balance resources and improve the generation quality of subsequent audio generation using discrete audio features, thus solving the problem of resource waste or insufficient quality caused by traditional fixed codebooks.

[0046] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0047] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0048] Figure 1 A flowchart illustrating a discrete audio feature generation method provided in an embodiment of this application;

[0049] Figure 2 A flowchart illustrating a training method for an audio data segmenter, provided in another embodiment of this application;

[0050] Figure 3 This is a schematic diagram of the structure of a discrete audio feature generation device provided in an embodiment of this application;

[0051] Figure 4 A schematic diagram of the structure of a training device for an audio data segmenter provided in another embodiment of this application;

[0052] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0053] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of this application. Based on the embodiments of this application, every other embodiment obtained by those skilled in the art without inventive effort falls within the scope of protection of this application.

[0054] First, the applicable scenarios for this application will be introduced. This application can be applied to the field of audio processing technology.

[0055] With the rapid development of deep learning technology, significant progress has been made in the field of audio signal processing, and audio word segmenters, as a key component, are widely used in speech-related tasks. Traditional audio word segmenters are mainly based on vector quantization (VQ) technology, whose core goal is to achieve high-quality signal reconstruction by discretizing continuous audio signals into a sequence of acoustic units.

[0056] Research has revealed that in existing technologies, vector quantization is typically implemented using codebooks. Current codebook designs often employ a fixed strategy: the same set of codebooks is forcibly applied to different types of audio (such as clean speech, noisy speech, and music), while the acoustic feature distributions of different audio types differ significantly. This quantization method leads to a decrease in audio reconstruction quality, further limiting the generalization ability of audio segmenters in multimodal tasks.

[0057] Based on this, embodiments of this application provide a discrete audio feature generation method and apparatus, and an audio data segmenter training method and apparatus. By adaptively selecting different target codebook vectors as discrete audio features based on the acoustic feature vectors of the initial audio data, resources can be better balanced, and the generation quality of subsequent audio generation using discrete audio features can be improved, thus solving the problems of resource waste or insufficient quality caused by traditional fixed codebooks.

[0058] Please see Figure 1 , Figure 1 This is a flowchart illustrating a discrete audio feature generation method provided in an embodiment of this application. The discrete audio feature generation method provided in this application is based on an audio data segmenter, which includes an encoder and a vector quantization module. The vector quantization module includes multiple codebooks, and each codebook includes multiple codebook vectors, such as... Figure 1As shown in the embodiments of this application, the discrete audio feature generation method includes:

[0059] S101, input the initial audio data into the encoder to obtain the acoustic feature vector.

[0060] Here, the encoder is used to extract the acoustic features of the audio data and transform the audio data into several continuous vectors.

[0061] Regarding step S101 above, in specific implementation, initial audio data is acquired. Here, the initial audio data can be an audio signal or a Mel spectrum; this application does not specifically limit this. Then, the initial audio data is input to the encoder to extract acoustic features, obtaining an acoustic feature vector.

[0062] S102, for each initial audio data, a target codebook vector is obtained from the vector quantization module based on the acoustic feature vector, which serves as the discrete audio feature corresponding to the initial audio data.

[0063] Here, the Vector Quantization (VQ) module is a key component in deep learning, particularly in autoencoder frameworks and generative models, used to discretize continuous vector representations. Its core idea is to approximate the input vector using a finite, predefined "codebook." In the embodiments provided in this application, the VQ module maps the continuous latent space of the encoder output to a discrete sequence of audio tokens. Each latent vector is matched with the most similar vector in the codebook, and its index (Token ID) is used to represent the latent vector, thus forming discrete audio features. The VQ module includes multiple codebooks, each containing multiple codebook vectors. The core role of the codebook in the VQ module is to convert continuous input feature vectors into discrete symbols (indices). The number of codebook vectors in each codebook is predefined. For example, with three codebooks, the first codebook contains 1024 codebook vectors, the second codebook contains 4096 vectors, and the third codebook contains 8192 vectors. The codebook incorporates a gating network that selects different codebooks for vector quantization based on continuous feature vectors representing the dynamic range of the audio. Specifically, for audio with a small dynamic range (relatively flat or even silent), a smaller codebook is needed, such as one with 1024 vectors; for audio with a large dynamic range (e.g., the chorus of a song), a larger codebook is chosen, such as one with 8192 vectors; and for ordinary speech audio, a medium-sized codebook with 4096 vectors is selected. It should be understood that the above codebook data is merely an example; in practice, more or fewer codebooks can be trained based on model resources, and the examples in this application should not be construed as limiting the scope of protection of this application.

[0064] Regarding step S102 above, in specific implementation, for each initial audio data, the acoustic feature vector corresponding to the initial audio data is input into the vector quantization module. Based on the acoustic feature vector, the target codebook vector is matched from the multiple codebooks included in the vector quantization module, and the target codebook vector is used as the discrete audio feature corresponding to the initial audio data.

[0065] As an optional embodiment, an audio feature extraction module is further included between the encoder and the vector quantization module. Furthermore, the discrete audio feature generation method provided in this application embodiment also includes:

[0066] I: Input the acoustic feature vector into the audio feature extraction module to obtain the audio statistics corresponding to the initial audio data.

[0067] Here, an audio feature extraction module is introduced between the encoder and the vector quantization module. This module extracts statistical features related to the audio dynamic range from the encoder output as input to the gating module. The audio feature extraction module can be implemented using a neural network (such as an MLP or multi-layer LSTM / GRU), which processes the global features of the encoder and outputs a dynamic feature vector.

[0068] Regarding step I above, in specific implementation, the encoder output is received, the acoustic feature vector is input into the audio feature extraction module, the statistical features of the current audio segment are calculated, and an audio statistic representing the "activity level" of the audio is output. This audio statistic is then used for subsequent codebook selection decisions, enhancing the model's understanding of the audio content. The statistical features are used to indicate the dispersion of the audio, such as variance, standard deviation, peak-to-valley difference, etc., and this invention does not impose limitations on these aspects.

[0069] II: Determine the target codebook that matches the initial audio data based on the first preset matching rule and the audio statistics.

[0070] Regarding step II above, in specific implementation, the target codebook that matches the initial audio data is determined based on the first preset matching rule and the audio statistics output by the audio feature extraction module.

[0071] According to the embodiments provided in this application, the vector quantization module includes a gating module. The first preset matching rule includes: inputting the audio statistics into the gating module, the gating module determining the matching probability of each codebook with the acoustic feature vector based on the value of the audio statistics, and selecting the codebook with the highest matching probability among multiple codebooks as the target codebook.

[0072] Here, the gating module can be a multilayer perceptron or a lightweight neural network based on an attention mechanism, with its output layer dimension equal to the total number of codebooks. The gating module receives audio statistics representing the dynamic range of the audio. The gating function within the module determines the matching probability between each codebook and the acoustic feature vector based on the audio statistics data and outputs a result vector L, which is transformed into the matching probability distribution P={p1,p2,p3,……p} for each codebook. n}, where n is the number of codebooks. Then, the codebook with the highest matching probability among the multiple codebooks is selected as the target codebook. Preferably, before obtaining the matching probability distribution of each codebook, the result vector L is normalized to obtain the matching probability distribution P. The normalization method can use common normalization algorithms such as softmax, and this application does not impose any restrictions on it.

[0073] III: In the target codebook, a codebook vector matching the acoustic feature vector is determined based on the second preset matching rule, which serves as the discrete audio feature corresponding to the initial audio data.

[0074] Regarding step III above, in specific implementation, after the target codebook is determined in step II above, a codebook vector matching the acoustic feature vector is determined in the target codebook based on the second preset matching rule, so as to serve as the discrete audio feature corresponding to the initial audio data.

[0075] Specifically, regarding step III above, determining the codebook vector matching the acoustic feature vector based on the second preset matching rule, as the audio feature corresponding to the initial audio data, includes:

[0076] Calculate the spatial distance between the acoustic feature vector and all codebook vectors in the target codebook; obtain the codebook vector with the smallest spatial distance among multiple codebook vectors, and use this codebook vector as the discrete audio feature corresponding to the initial audio data.

[0077] Regarding the two steps mentioned above, in specific implementation, the spatial distance between the acoustic feature vector and each codebook vector in the target codebook is calculated, and the codebook vector with the smallest spatial distance among multiple codebook vectors is taken as the discrete audio feature corresponding to the initial audio data.

[0078] The discrete audio feature generation method provided in this application is based on an audio data segmenter, which includes an encoder and a vector quantization module. The vector quantization module includes multiple codebooks, each containing multiple codebook vectors. First, initial audio data is input to the encoder to obtain acoustic feature vectors. Then, for each piece of initial audio data, a target codebook vector is matched from the vector quantization module based on the acoustic feature vector, serving as the discrete audio feature corresponding to that initial audio data. Thus, because the vector quantization module includes multiple codebooks, each containing multiple codebook vectors, different target codebook vectors are adaptively selected as discrete audio features based on the acoustic feature vectors of the initial audio data. This better balances resources and improves the generation quality of subsequent audio using discrete audio features, solving the problem of resource waste or insufficient quality caused by traditional fixed codebooks.

[0079] Please see Figure 2 , Figure 2 This is a flowchart illustrating a training method for an audio data segmenter according to another embodiment of this application. The training method for the audio data segmenter provided in this embodiment is used to train an audio data segmenter, which is used to implement a discrete audio feature generation method, such as... Figure 2 As shown in the embodiments of this application, the training method includes:

[0080] S201, input the audio sample data into the encoder to obtain the acoustic sample feature vector.

[0081] S202, based on the vector quantization module, the acoustic feature vector is converted into discrete audio sample features.

[0082] Regarding steps S201-S202 above, in specific implementation, audio sample data is acquired and input into the encoder to obtain acoustic sample feature vectors. The acoustic sample feature vectors are then input into the vector quantization module, which converts them into discrete audio sample features. The methods for obtaining acoustic sample feature vectors and discrete audio sample features can refer to the descriptions of steps S101-S102 above, and achieve the same technical effect; therefore, they will not be elaborated further.

[0083] S203, input the discrete audio sample features into the decoder to obtain reconstructed audio data.

[0084] Here, the decoder is used to restore discrete audio feature sequences to audio data or Mel spectra, depending on whether the input encoder is audio data or Mel spectra.

[0085] In specific implementation of step S203, the discrete audio sample features output by the vector quantization module are input to the decoder for data restoration to obtain reconstructed audio data.

[0086] S204, establish at least one loss function, and iteratively train at least one parameter of the audio data segmenter and the decoder based on the result of at least one loss function.

[0087] Regarding step S204 above, in specific implementation, at least one loss function is established using audio sample data, acoustic sample feature vectors, discrete audio sample features, and reconstructed audio data. Based on the result of at least one loss function, at least one parameter of the audio data segmenter and decoder is iteratively trained.

[0088] As an optional embodiment, regarding step S204 above, establishing at least one loss function includes:

[0089] Step 2041: Obtain the text sample data corresponding to the audio sample data, convert the text sample data into a discrete text feature vector, and establish a first loss function between the discrete audio sample features and the discrete text feature vector.

[0090] Here, the first loss function is the Connectionist Temporal Classification (CTC) loss function. The core of the CTC loss function lies in its ability to solve the problems of inconsistent lengths between the input and output sequences and / or the lack of strict temporal alignment between the two sequences. In this application, it is established between the discrete audio sample features predicted by the vector quantization module and the discrete text feature vector processed by the text encoder. It can measure the probability of obtaining text sample data from discrete audio sample features under all possible alignment methods, thereby enhancing the semantic capabilities of the overall model and enabling the content in the output audio feature sequence to better correspond to the text. Each codebook in the vector quantization module can be composed of a single-layer VQ or multiple-layer VQ. When using multiple-layer VQ, the input vector is decomposed into multiple parts during the quantization stage, and each part is quantized using a codebook. This method can significantly increase the representational power while keeping the codebook size controllable.

[0091] Regarding step 2041 above, in specific implementation, the corresponding text sample data of the audio sample data is obtained, and the text sample data is converted into discrete text feature vectors. The discrete audio sample features and discrete text feature vectors are compared, and a first loss function is established between the discrete audio sample features and discrete text feature vectors.

[0092] Step 2042: Establish a second loss function between the acoustic sample feature vector and the discrete audio sample features.

[0093] Here, the second loss function is the codebook learning loss (VQ loss) function, which measures the difference between the acoustic sample feature vector output by the encoder and the sample target codebook corresponding to the discrete audio sample features.

[0094] Regarding step 2042 above, in specific implementation, a second loss function is established between the acoustic sample feature vector and the discrete audio sample features.

[0095] Step 2043: Establish a third loss function and a fourth loss function between the audio sample data and the reconstructed audio data.

[0096] Here, the third loss function is the adversarial loss function of the GAN model. A discriminator network is introduced, established between the initial audio sample data and the reconstructed audio data obtained by the decoder. The discriminator measures which audio is real and which is reconstructed, ensuring that the reconstructed audio is as close as possible to the audio initially input to the encoder. The fourth loss function is the reconstruction loss function, also established between the initial audio sample data and the reconstructed audio data obtained by the decoder. It measures the difference between the audio data reconstructed by the decoder and the audio sample data. This is a common loss category and calculation method in the model. Based on this reconstruction loss, the difference between the original input and the reconstructed output can be minimized, ensuring that the encoder, vector quantization module, and decoder can output higher-quality audio information during inference.

[0097] In specific implementation of step 2043 above, a third loss function and a fourth loss function are established between the audio sample data and the reconstructed audio data.

[0098] Step 2044: Based on the weighted fusion strategy, the first loss function, the second loss function, the third loss function, and the fourth loss function are fused together to obtain a fusion loss function. Based on the result of the fusion loss function, at least one parameter of the audio data segmenter and the decoder is iteratively trained.

[0099] Regarding step 2044 above, in its specific implementation, based on a weighted fusion strategy, the first loss function, the second loss function, the third loss function, and the fourth loss function are fused to obtain a fusion loss function. Then, based on the result of the fusion loss function, at least one parameter of the audio data segmenter and decoder is iteratively trained, and backpropagation is performed based on the fusion loss function to update the parameters of the audio data segmenter and decoder.

[0100] Here, the weighted fusion strategy includes: configuring a first weight, a second weight, a third weight, and a fourth weight for the first loss function, the second loss function, the third loss function, and the fourth loss function, respectively.

[0101] Here, the first, second, third, and fourth weights are assigned respectively for the first, second, third, and fourth loss functions. The fusion loss function is the weighted sum of the first, second, third, and fourth loss functions. The weights can represent the model's various capabilities to some extent. For example, the larger the weight of the CTC loss function, the stronger the semantic capability; the larger the weight of the adversarial loss function, the higher the model's acoustic performance, and so on.

[0102] Furthermore, the first, second, third, and fourth weights are learnable weight parameters. The iterative process of the learnable weight parameters includes: obtaining the rate of change of loss for each loss function per unit time, and dynamically adjusting the corresponding weights based on the rate of change of loss for each loss function.

[0103] Here, the weights of each loss function can be dynamically adjusted based on the model's performance and the desired model performance. For example, if the current audio task requires stronger semantic capabilities, the weight of the CTC loss can be increased accordingly. Specifically, the rate of change of loss for each loss function per unit time is obtained, and the corresponding weights are dynamically adjusted based on the rate of change of loss for each loss function.

[0104] Here, the unit of time is preferably defined as a fixed number of iterations, such as 5 iterations or 10 iterations per unit of time, etc., and this application does not impose specific limitations on this. For each loss function, the rate of change of loss is calculated using the following formula:

[0105]

[0106] in, This represents the average loss function value per unit time. This represents the average loss function value over the previous unit of time.

[0107] Dynamically adjusting weights based on the rate of change of loss includes: ,in, The weight is the weight within the current unit of time. This represents the weight from the previous unit of time. When When the value is greater than 1, it indicates that the loss has actually increased. At this time, the value of the exponent term is greater than 1, and the weight... This will be added, and the task will be emphasized; when When the value is 1, it means the loss remains unchanged, and the weight... Unchanged; when When the value is less than 1, it indicates a decrease in loss; at this time, the value of the exponent term is less than 1, and the weight... It will decrease. It is a sensitivity parameter, if Increasing the weight will result in a larger adjustment range. Too large a weight, or excessive weight fluctuations, can lead to unstable training. Conversely, if... If the value is small, the adjustment range of the weights will be small, resulting in a slower adjustment of the weights. Preferably, let... =0.5, and then gradually fine-tune it based on the results of each round of training. Size.

[0108] Preferably, when dynamically adjusting the weights of the loss functions, the weights of each loss function are adjusted simultaneously. More preferably, after adjusting the weight of any one loss function, the overall probability needs to be normalized. In this way, by weighted fusion of multiple loss functions and dynamic adjustment of the weights of each loss function, a balance can be found in the capabilities represented by each loss function in the model. This solves the problem that if the model only iterates parameters for one or two capabilities, the capabilities represented by other loss functions will become weak.

[0109] To reduce training costs and difficulty, each codebook in the vector quantization module can be trained independently. As an optional embodiment, the training phase for each codebook in the vector quantization module includes:

[0110] At the start of training, the acoustic sample feature vectors are allowed to be mapped to at least two codebook vectors from at least two codebooks; during training, the gating function is gradually adjusted so that the output vector of the gating function approaches the one-hot vector.

[0111] Furthermore, the gradual adjustment of the gating function to make its output vector approach the one-hot vector includes:

[0112] The gating function includes a temperature parameter. The temperature parameter is preset to a maximum value so that the matching probability of the acoustic sample feature vector with each codebook and each codebook vector is close to the same at the beginning of training, thus optimizing all codebook vectors. The value of the temperature parameter is gradually reduced so that the matching probability of different codebook vectors differs, until the temperature parameter approaches 0, at which point the output vector of the gating function approaches the one-hot vector.

[0113] Here, the gating function is expressed by the following formula:

[0114]

[0115] in, For gated functions, Represents the acoustic sample feature vector and the first Similarity between codebook vectors This represents the temperature parameter.

[0116] Here, the gating function mentioned above adds an extra layer to the ordinary normalization function. , That is, the temperature parameter. >0, The maximum value of T refers to the maximum value within its possible range. For example, it can be set to 2.0 at the beginning of training; towards the end of training, the value of T can be a decimal close to 0, such as 0.001.

[0117] in, When it is larger, for example As the value approaches positive infinity, the exponent of each codebook vector approaches 1. Therefore, for all codebooks and codebook vectors, the value approaches 1 / N. Thus, the probability of all codebook vectors being selected is similar, ensuring that all codebook vectors have the potential to be trained.

[0118] when As the value of gradually decreases, the probabilities of the various codebook vectors begin to differ, until... When the value approaches 0, the vector output by the gating function will approach the form [0,0,0,1,0,0,0…], where only one dimension is 1 and all other dimensions are 0, i.e., a one-hot vector. Here, 1 represents a selected codebook or selected codebook vector, and 0 represents an unselected codebook or unselected codebook vector. The decay of the temperature parameter can be linear or exponential.

[0119] For each codebook vector, in addition to training with VQ loss, an exponential moving average (EMA) update can be used to help codebook vectors that are selected too infrequently receive update signals indirectly. In this way, by training each codebook in the vector quantization module through the above steps, it is ensured that all codebook vectors can be trained, preventing the selection of the same or similar codebook vectors in each training iteration, which could result in some codebook vectors not being trained.

[0120] This application provides a training method for an audio data segmenter. The training method trains the audio data segmenter to implement a discrete audio feature generation method. First, audio sample data is input into an encoder to obtain acoustic sample feature vectors. Then, based on a vector quantization module, the acoustic feature vectors are converted into discrete audio sample features. The discrete audio sample features are input into a decoder to obtain reconstructed audio data. Finally, at least one loss function is established, and at least one parameter of the audio data segmenter and decoder is iteratively trained based on the results of the at least one loss function. By setting at least one loss function, the learning capabilities of the model in different aspects, such as reconstruction capability, semantic capability, and acoustic performance capability, can be considered simultaneously, thereby improving the quality of subsequent audio generation.

[0121] Please see Figure 3 , Figure 3 This is a schematic diagram of a discrete audio feature generation device provided in an embodiment of this application. The discrete audio feature generation device is implemented based on an audio data segmenter, which includes an encoder and a vector quantization module. The vector quantization module includes multiple codebooks, and each codebook includes multiple codebook vectors, such as... Figure 3 As shown, the discrete audio feature generation device 300 includes:

[0122] The acoustic feature vector determination module 301 is used to input the initial audio data into the encoder to obtain the acoustic feature vector;

[0123] The discrete audio feature generation module 302 is used to obtain the target codebook vector from the vector quantization module based on the acoustic feature vector for each initial audio data, and use it as the discrete audio feature corresponding to the initial audio data.

[0124] Furthermore, an audio feature extraction module is also included between the encoder and the vector quantization module, and the discrete audio feature generation module 302 is further used for:

[0125] The acoustic feature vector is input into the audio feature extraction module to obtain the audio statistics corresponding to the initial audio data;

[0126] Based on the first preset matching rule and the audio statistics, a target codebook that matches the initial audio data is determined.

[0127] In the target codebook, a codebook vector matching the acoustic feature vector is determined based on a second preset matching rule, which serves as the discrete audio feature corresponding to the initial audio data.

[0128] Furthermore, the vector quantization module includes a gating module;

[0129] The first preset matching rule includes: inputting the audio statistics into the gating module, the gating module determining the matching probability of each codebook with the acoustic feature vector based on the value of the audio statistics, and taking the codebook with the highest matching probability among multiple codebooks as the target codebook.

[0130] Furthermore, when the discrete audio feature generation module 302 is used to determine the codebook vector matching the acoustic feature vector based on the second preset matching rule as the audio feature corresponding to the initial audio data, the discrete audio feature generation module 302 is also used to:

[0131] Calculate the spatial distance between the acoustic feature vector and all codebook vectors in the target codebook;

[0132] Obtain the codebook vector with the smallest spatial distance among multiple codebook vectors, and use this codebook vector as the discrete audio feature corresponding to the initial audio data.

[0133] Please see Figure 4 , Figure 4 This is a schematic diagram of a training device for an audio data segmenter according to another embodiment of this application. The training device is used to train the audio data segmenter, which is used to implement a discrete audio feature generation method, such as... Figure 4 As shown, the training device 400 includes:

[0134] The sample feature vector determination module 401 is used to input audio sample data into the encoder to obtain acoustic sample feature vectors.

[0135] The audio sample feature determination module 402 is used to convert the acoustic feature vector into discrete audio sample features based on the vector quantization module.

[0136] The audio reconstruction module 403 is used to input the features of the discrete audio samples into the decoder to obtain reconstructed audio data;

[0137] The model training module 404 is used to establish at least one loss function and iteratively train at least one parameter of the audio data segmenter and the decoder based on the result of the at least one loss function.

[0138] Furthermore, when the model training module 404 is used to establish at least one loss function, the model training module 404 is also used to:

[0139] Obtain the corresponding text sample data of the audio sample data, convert the text sample data into a discrete text feature vector, and establish a first loss function between the discrete audio sample features and the discrete text feature vector;

[0140] A second loss function is established between the acoustic sample feature vector and the discrete audio sample features;

[0141] A third loss function and a fourth loss function are established between the audio sample data and the reconstructed audio data;

[0142] Based on a weighted fusion strategy, the first loss function, the second loss function, the third loss function, and the fourth loss function are fused together to obtain a fusion loss function. Based on the result of the fusion loss function, at least one parameter of the audio data segmenter and the decoder is iteratively trained.

[0143] Furthermore, the weighted fusion strategy includes:

[0144] A first weight, a second weight, a third weight, and a fourth weight are respectively configured for the first loss function, the second loss function, the third loss function, and the fourth loss function, wherein the first weight, the second weight, the third weight, and the fourth weight are learnable weight parameters;

[0145] The iterative process of the learnable weight parameters includes:

[0146] Obtain the rate of change of loss for each loss function per unit time, and dynamically adjust the corresponding weights based on the rate of change of loss for each loss function.

[0147] Furthermore, the training device 400 also includes a vector quantization module training module, the training phase of which for each codebook in the vector quantization module includes:

[0148] At the start of training, the acoustic sample feature vectors are allowed to be mapped to at least two codebook vectors from at least two codebooks; during training, the gating function is gradually adjusted so that the output vector of the gating function approaches the one-hot vector.

[0149] Furthermore, when the vector quantization module training module is used to gradually adjust the gating function so that the output vector of the gating function approaches the one-hot vector, the vector quantization module training module is also used for:

[0150] The gating function includes a temperature parameter, which is preset to a maximum value so that the matching probability of the acoustic sample feature vector with each codebook and each codebook vector is approximately the same when training begins, thereby optimizing all codebook vectors.

[0151] The temperature parameter is gradually reduced to make the matching probability of different codebook vectors differ, until the temperature parameter approaches 0, at which point the output vector of the gating function approaches the one-hot vector.

[0152] Please see Figure 5 , Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 5 As shown, the electronic device 500 includes a processor 510, a memory 520, and a bus 530.

[0153] The memory 520 stores machine-readable instructions executable by the processor 510. When the electronic device 500 is running, the processor 510 and the memory 520 communicate via the bus 530. When the machine-readable instructions are executed by the processor 510, they can perform the operations described above. Figure 1 as well as Figure 2 The steps of the discrete audio feature generation method and the audio data segmenter training method in the method embodiment shown are described in detail in the method embodiment, and will not be repeated here.

[0154] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, can perform the above-described actions. Figure 1 as well as Figure 2 The steps of the discrete audio feature generation method and the audio data segmenter training method in the method embodiment shown are described in detail in the method embodiment, and will not be repeated here.

[0155] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0156] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the shown or discussed mutual couplings, direct couplings, or communication connections may be through some communication interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0157] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0158] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0159] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a processor-executable, non-volatile, computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0160] Finally, it should be noted that the above-described embodiments are merely specific implementations of this application, used to illustrate the technical solutions of this application, and not to limit them. The scope of protection of this application is not limited thereto. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features, within the scope of the technology disclosed in this application. Such modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for generating discrete audio features, characterized in that, The discrete audio feature generation method is implemented based on an audio data segmenter, which includes an encoder and a vector quantization module. The vector quantization module includes multiple codebooks, each codebook including multiple codebook vectors. The discrete audio feature generation method includes: The initial audio data is input into the encoder to obtain the acoustic feature vector; For each initial audio data, a target codebook vector is obtained from the vector quantization module based on the acoustic feature vector, which serves as the discrete audio feature corresponding to the initial audio data. An audio feature extraction module is further included between the encoder and the vector quantization module, and the discrete audio feature generation method further includes: The acoustic feature vector is input into the audio feature extraction module to obtain the audio statistics corresponding to the initial audio data; Based on the first preset matching rule and the audio statistics, a target codebook that matches the initial audio data is determined. In the target codebook, a codebook vector matching the acoustic feature vector is determined based on a second preset matching rule, which serves as the discrete audio feature corresponding to the initial audio data.

2. The discrete audio feature generation method according to claim 1, characterized in that, The vector quantization module includes a gating module; The first preset matching rule includes: inputting the audio statistics into the gating module, the gating module determining the matching probability of each codebook with the acoustic feature vector based on the value of the audio statistics, and taking the codebook with the highest matching probability among multiple codebooks as the target codebook.

3. The discrete audio feature generation method according to claim 2, characterized in that, The step of determining the codebook vector that matches the acoustic feature vector based on the second preset matching rule, as the audio feature corresponding to the initial audio data, includes: Calculate the spatial distance between the acoustic feature vector and all codebook vectors in the target codebook; Obtain the codebook vector with the smallest spatial distance among multiple codebook vectors, and use this codebook vector as the discrete audio feature corresponding to the initial audio data.

4. A training method for an audio data segmenter, characterized in that, The training method is used to train an audio data segmenter, which is used to implement the discrete audio feature generation method as described in any one of claims 1-3. The training method includes: The audio sample data is input into the encoder to obtain the acoustic sample feature vector; Based on the vector quantization module, the acoustic feature vector is transformed into discrete audio sample features; The discrete audio sample features are input into the decoder to obtain the reconstructed audio data; Establish at least one loss function, and iteratively train at least one parameter of the audio data segmenter and the decoder based on the result of at least one loss function; The establishment of at least one loss function includes: Obtain the corresponding text sample data of the audio sample data, convert the text sample data into a discrete text feature vector, and establish a first loss function between the discrete audio sample features and the discrete text feature vector; A second loss function is established between the acoustic sample feature vector and the discrete audio sample features; A third loss function and a fourth loss function are established between the audio sample data and the reconstructed audio data; Based on a weighted fusion strategy, the first loss function, the second loss function, the third loss function, and the fourth loss function are fused together to obtain a fusion loss function. Based on the result of the fusion loss function, at least one parameter of the audio data segmenter and the decoder is iteratively trained.

5. The training method according to claim 4, characterized in that, The weighted fusion strategy includes: A first weight, a second weight, a third weight, and a fourth weight are respectively configured for the first loss function, the second loss function, the third loss function, and the fourth loss function, wherein the first weight, the second weight, the third weight, and the fourth weight are learnable weight parameters; The iterative process of the learnable weight parameters includes: Obtain the rate of change of loss for each loss function per unit time, and dynamically adjust the corresponding weights based on the rate of change of loss for each loss function.

6. The training method according to claim 4, characterized in that, The training phase of each codebook in the vector quantization module includes: At the start of training, the acoustic sample feature vectors are allowed to be mapped to at least two codebook vectors from at least two codebooks; during training, the gating function is gradually adjusted so that the output vector of the gating function approaches the one-hot vector.

7. The training method according to claim 6, characterized in that, The gradual adjustment of the gating function to make its output vector approach the one-hot vector includes: The gating function includes a temperature parameter, which is preset to a maximum value so that the matching probability of the acoustic sample feature vector with each codebook and each codebook vector is approximately the same when training begins, thereby optimizing all codebook vectors. The temperature parameter is gradually reduced to make the matching probability of different codebook vectors differ, until the temperature parameter approaches 0, at which point the output vector of the gating function approaches the one-hot vector.

8. A discrete audio feature generation device, characterized in that, The discrete audio feature generation device is implemented based on an audio data segmenter, which includes an encoder and a vector quantization module. The vector quantization module includes multiple codebooks, each codebook including multiple codebook vectors. The discrete audio feature generation device includes: An acoustic feature vector determination module is used to input initial audio data into the encoder to obtain acoustic feature vectors; The discrete audio feature generation module is used to obtain the target codebook vector from the vector quantization module based on the acoustic feature vector for each initial audio data, and use it as the discrete audio feature corresponding to the initial audio data. The encoder and the vector quantization module further include an audio feature extraction module, and the discrete audio feature generation module is further used for: The acoustic feature vector is input into the audio feature extraction module to obtain the audio statistics corresponding to the initial audio data; Based on the first preset matching rule and the audio statistics, a target codebook that matches the initial audio data is determined. In the target codebook, a codebook vector matching the acoustic feature vector is determined based on a second preset matching rule, which serves as the discrete audio feature corresponding to the initial audio data.

9. A training device for an audio data segmentation tool, characterized in that, The training device is used to train an audio data segmenter, the audio data segmenter being used to implement the discrete audio feature generation method as described in any one of claims 1-3, and the training device comprises: The sample feature vector determination module is used to input audio sample data into the encoder to obtain acoustic sample feature vectors. The audio sample feature determination module is used to convert the acoustic feature vector into discrete audio sample features based on the vector quantization module. The audio reconstruction module is used to input the features of the discrete audio samples into the decoder to obtain reconstructed audio data. The model training module is used to establish at least one loss function and iteratively train at least one parameter of the audio data segmenter and the decoder based on the result of the at least one loss function. When the model training module is used to establish at least one loss function, the model training module is also used for: Obtain the corresponding text sample data of the audio sample data, convert the text sample data into a discrete text feature vector, and establish a first loss function between the discrete audio sample features and the discrete text feature vector; A second loss function is established between the acoustic sample feature vector and the discrete audio sample features; A third loss function and a fourth loss function are established between the audio sample data and the reconstructed audio data; Based on a weighted fusion strategy, the first loss function, the second loss function, the third loss function, and the fourth loss function are fused together to obtain a fusion loss function. Based on the result of the fusion loss function, at least one parameter of the audio data segmenter and the decoder is iteratively trained.

Images