A speech feature processing method and a speech processing method

Abstract

The application provides a speech feature processing method and a speech processing method, relates to the technical field of speech token extraction, and comprises the following steps: acquiring audio data, pre-processing the audio data to obtain frequency domain features; introducing the frequency domain features into a speech token extraction model to extract speech tokens, and obtaining a plurality of groups of speech tokens; the speech token extraction model comprises an audio coding layer and a residual vector quantization layer; the audio coding layer is configured to perform feature coding according to the frequency domain features to obtain coded features; the residual vector quantization layer is configured to perform segmented residual quantization processing according to the coded features; the audio coding layer comprises a first convolution unit, a Resnet network unit and a second convolution unit; the Resnet network unit comprises a plurality of residual blocks connected with the first convolution unit. The above method solves the problem of high complexity of speech tokens extracted by the existing speech feature technology.

Description

[0001] This application is a divisional application. The original application has the application number 202411966054.3 and the original application date is December 30, 2024. The entire contents of the original application are incorporated herein by reference. Technical Field

[0002] This invention relates to the field of voice token extraction technology, and in particular to a voice feature processing method and a voice processing method. Background Technology

[0003] In recent years, with the rapid development of artificial intelligence technology, Large Language Models (LLMs) have made groundbreaking progress in the field of natural language processing. LLMs, represented by ChatGPT, GPT-4, and some advanced domestic models, have propelled the application of natural language processing to new heights with their powerful language understanding and generation capabilities. These models can not only perform complex dialogue interactions but also complete various tasks such as text creation and code generation, greatly expanding the application scenarios of natural language processing technology.

[0004] However, despite the significant achievements of LLMs in natural language processing, research and applications in the audio field are relatively limited. Audio, as an important information carrier, is crucial for processing and analysis in many applications, such as speech recognition, audio classification, and music generation. To extend the powerful capabilities of LLMs to the audio domain, a key problem needs to be solved: how to convert audio data into a format that LLMs can process. In natural language processing, LLMs typically receive and process discrete text tokens (i.e., basic units in text, such as words, characters, or sub-words). Therefore, to apply LLMs to the audio domain, we need to completely discretize the audio data to obtain a form similar to text tokens. In this process, audio data compression and encoding techniques play a crucial role.

[0005] Currently, the main technologies for compressing audio into tokens include Google's SoundStream and Meta's encoder. Both technologies utilize a residual quantization layer combined with an autoencoder framework to compress audio signals into a finite set of tokens. Specifically, SoundStream and encoder can compress audio signals into 12 or 8 sets of tokens, representing basic units or features in the audio signal. However, despite their achievements in audio compression, these technologies still face some challenges. Since these technologies are based on audio temporal domain modeling, a large number of tokens are needed to complete the full modeling of the audio signal. This means that LLMs need to process a large number of tokens during subsequent audio modeling, which not only increases computational complexity but may also negatively impact modeling results. Furthermore, too many tokens can lead to overfitting or underfitting during model training, further affecting the accuracy and efficiency of audio processing. Summary of the Invention

[0006] This application provides a speech feature processing method and a speech processing method to solve the problem of excessive speech token extraction groups in existing speech token extraction technologies.

[0007] Firstly, this application provides a speech feature processing method, the processing method comprising:

[0008] Acquire audio data, preprocess the audio data to obtain frequency domain features;

[0009] The frequency domain features are imported into a speech token extraction model to extract speech tokens, resulting in several sets of speech tokens. The speech token extraction model includes an audio coding layer and a residual vector quantization layer. The audio coding layer is configured to perform feature encoding based on the frequency domain features to obtain coded features. The residual vector quantization layer is configured to perform segmented residual quantization processing based on the coded features.

[0010] The audio coding layer consists of a first convolutional unit, a ResNet network unit, and a second convolutional unit; the ResNet network unit includes a plurality of residual blocks; the first convolutional unit is connected to each of the residual blocks in the ResNet network unit.

[0011] In the ResNet network unit, each residual block includes a fifth convolutional unit, a first activation function unit, a sixth convolutional unit, and a second activation function unit connected in sequence; the input port of the fifth convolutional unit and the output port of the second activation function unit are connected.

[0012] Preferably, the step of preprocessing the audio data includes:

[0013] The audio data is frequency-domain converted using the Fourier transform algorithm to obtain the frequency domain features.

[0014] Preferably, the audio coding layer is further configured as follows:

[0015] The coding features are obtained by performing Mel-spectrum transformation based on the frequency domain features.

[0016] Preferably, the residual vector quantization layer is further configured as follows:

[0017] A first operation is performed based on a first feature to obtain a second feature, wherein the first feature includes the encoded feature or a third feature obtained through the second operation;

[0018] Perform a second operation based on the second feature to obtain the third feature, and repeat the first operation n times and the second operation n-1 times.

[0019] Perform a third operation on n second features that have completed the first operation to obtain several sets of the voice tokens.

[0020] Preferably, the first operation includes:

[0021] The encoded feature or the third feature is segmented to obtain the first segmentation feature;

[0022] VQ quantization is used to perform vector quantization on all the first segmentation features to obtain several first quantized features, and all the first quantized features are concatenated to obtain the second feature;

[0023] The second operation includes:

[0024] Subtracting the first feature from the second feature yields the third feature;

[0025] The third operation includes:

[0026] The second feature of each completed first operation is summed to obtain several sets of the voice tokens.

[0027] Preferably, in the ResNet network unit, each residual block further includes a spatial convolution unit and a first feature fusion unit; in the residual block, the fifth convolution unit, the spatial convolution unit, the first activation function unit, the first feature fusion unit, the sixth convolution unit, and the second activation function unit are connected in sequence in a stacked structure;

[0028] The fifth convolutional unit is composed of multiple first convolutional kernels, wherein the first convolutional kernels are point convolutions, and the number of the first convolutional kernels is greater than the number of channels of the convolutional kernels in the first convolutional unit.

[0029] The spatial convolutional unit is composed of multiple second convolutional kernels, the size of which is larger than that of the first convolutional kernel; the spatial convolutional unit is used to divide the output of the fifth convolutional unit into multiple subsequences, and to perform convolution processing on each subsequence through a second convolutional kernel.

[0030] The first feature fusion unit includes an attention layer, which is used to perform feature fusion processing on the output of each second convolutional kernel based on the calculation of attention weights;

[0031] The sixth convolutional unit is composed of multiple third convolutional kernels, each of which is a point convolution. The number of third convolutional kernels is the same as the number of channels of the convolutional kernels in the first convolutional unit.

[0032] Preferably, the ResNet network unit further includes a downsampling unit and a second feature fusion unit, which are set independently of the residual blocks, wherein,

[0033] The downsampling unit is used to perform frequency domain downsampling processing on the output of the first convolutional unit;

[0034] The second feature fusion unit includes an attention layer, which is used to perform feature fusion processing on the output of the downsampling unit and the output of the ResNet network unit based on the calculation of attention weights, and input the fused features into the second convolution unit.

[0035] Preferably, the voice token extraction model further includes an audio decoding layer; the audio decoding layer is applied to the model training process of the voice token extraction model; the audio decoding layer is configured to perform feature decoding based on the voice token to obtain target audio data; the target audio data is used to verify the integrity of the voice token, thereby training the voice token extraction model.

[0036] Preferably, the audio decoding layer includes a first block, a second block, and a third block; the first block includes a deconvolution unit, a ResNet network unit, and a third convolution unit; the second block includes a deconvolution unit, an MRF unit, and a fourth convolution unit; and the third block includes a PQMF unit.

[0037] The processing method further includes:

[0038] Acquire audio data, preprocess the audio data to obtain frequency domain features;

[0039] The frequency domain features are imported into a speech token extraction model to extract speech tokens, resulting in several sets of speech tokens. The speech token extraction model includes an audio coding layer and a residual vector quantization layer. The audio coding layer is configured to perform feature encoding based on the frequency domain features to obtain coded features. The residual vector quantization layer is configured to perform segmented quantization processing based on the coded features. The segmented quantization processing includes:

[0040] A fourth operation is performed based on the fourth feature to obtain several second segmentation features; the fourth feature includes the encoded feature or the sixth feature obtained through the fifth operation; the fourth operation includes segmenting the encoded feature or the sixth feature.

[0041] A fifth operation is performed based on all the second segmentation features to obtain a sixth feature; the fifth operation includes using VQ quantization to perform vector quantization on all the second segmentation features to obtain several second quantized features, and concatenating all the second quantized features to obtain the sixth feature;

[0042] Repeat the fourth and fifth operations n times, and the sixth feature obtained from the nth fifth operation is the voice token.

[0043] The processing method further includes:

[0044] Acquire audio data, preprocess the audio data to obtain frequency domain features;

[0045] The frequency domain features are imported into a speech token extraction model to extract speech tokens, resulting in several sets of speech tokens. The speech token extraction model includes an audio coding layer and a residual vector quantization layer. The audio coding layer is configured to perform feature encoding based on the frequency domain features to obtain coded features. The residual vector quantization layer is configured to perform residual quantization processing based on the coded features. The residual quantization processing includes:

[0046] The sixth operation is performed based on the seventh feature to obtain the eighth feature; the seventh feature includes an encoded feature or a ninth feature obtained through the seventh operation; the sixth operation includes vector quantization of the seventh feature using VQ quantization to obtain the third quantized feature;

[0047] The seventh operation is performed based on the eighth feature to obtain the ninth feature; the seventh operation includes adding the eighth feature and the seventh feature to obtain the tenth feature;

[0048] Repeat the sixth and seventh operations n times, and the tenth feature obtained from the nth seventh operation is the voice token.

[0049] The processing method includes:

[0050] Acquire audio data, preprocess the audio data to obtain frequency domain features;

[0051] The frequency domain features are imported into a speech token extraction model to extract speech tokens, resulting in at least one set of speech tokens. The speech token extraction model includes an audio coding layer, a residual vector quantization layer, and an audio decoding layer. The audio coding layer is configured to perform enhanced feature encoding based on the frequency domain features to obtain enhanced coded features. The residual vector quantization layer is configured to perform weakened segmentation residual quantization processing based on the enhanced coded features. The audio decoding layer is configured to perform enhanced feature decoding on the speech tokens to obtain target audio data. The audio decoding layer is applied to the model training process of the speech token extraction model, and the target audio data is used to verify the integrity of the speech tokens, thereby training the speech token extraction model.

[0052] Secondly, this application also provides a speech processing method, which is applied to the above-mentioned speech feature processing method for feature extraction.

[0053] As described above, this application provides a speech feature processing method and a speech processing method. The processing method includes acquiring audio data, preprocessing the audio data to obtain frequency domain features, and importing the frequency domain features into a speech token extraction model to extract speech tokens, obtaining several sets of speech tokens. The speech token extraction model includes an audio coding layer and a residual vector quantization layer. The audio coding layer is configured to perform feature encoding based on the frequency domain features to obtain coded features. The residual vector quantization layer is configured to perform segmented residual quantization processing based on the coded features. This application solves the problem of excessive speech token extraction sets in existing speech token extraction technologies through the above method. Attached Figure Description

[0054] To more clearly illustrate the technical solution of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0055] Figure 1 This is a flowchart of a speech feature processing method according to this application;

[0056] Figure 2 This is a flowchart of the preprocessing step in a speech feature processing method according to this application;

[0057] Figure 3 This is a functional flowchart of the audio coding layer in a speech feature processing method of this application;

[0058] Figure 4 This is a functional flowchart of the residual vector quantization layer in a speech feature processing method of this application;

[0059] Figure 5 This is a flowchart of the first operation in a speech feature processing method according to this application;

[0060] Figure 6 This is a flowchart of the second operation in a speech feature processing method according to this application;

[0061] Figure 7 This is a flowchart of the third operation in a speech feature processing method according to this application;

[0062] Figure 8 This is a schematic diagram of a speech token extraction model in a speech feature processing method according to this application;

[0063] Figure 9 This is a schematic diagram of the audio coding layer in a speech feature processing method according to this application;

[0064] Figure 10 This is a schematic diagram of the audio decoding layer in a speech feature processing method of this application;

[0065] Figure 11 This is a schematic diagram of the residual block in a speech feature processing method according to this application. Detailed Implementation

[0066] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0067] It should be noted that the brief descriptions of terms in this application are only for the convenience of understanding the embodiments described below, and are not intended to limit the embodiments of this application. Unless otherwise stated, these terms should be understood in their ordinary and common meaning.

[0068] It should be noted that, in this application, the terms "exemplary" or "for example" are used to indicate that something is being described as an example, illustration, or illustration. Any embodiment or design described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or design solutions. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0069] Figure 1 This is a flowchart of a speech feature processing method according to this application.

[0070] Figure 8 This is a schematic diagram of a speech token extraction model in a speech feature processing method according to this application.

[0071] See Figure 1 and Figure 8 As can be seen, this embodiment provides a speech feature processing method, wherein the processing method includes a complete scheme, a first scheme, and a second scheme; wherein the difference between the complete scheme, the first scheme, and the second scheme lies in the different steps of model processing.

[0072] The complete solution, the first solution, and the second solution all include:

[0073] S100, acquire audio data, preprocess the audio data to obtain frequency domain features;

[0074] S200, the frequency domain features are imported into the speech token extraction model to extract speech tokens, resulting in several sets of speech tokens.

[0075] Specifically, in this embodiment, the complete scheme, the first scheme, and the second scheme all require first acquiring audio data, performing preprocessing, and then importing the preprocessed data into the voice token extraction model for voice token extraction. The difference lies in the fact that the voice token extraction model performs different voice token extraction methods in different schemes.

[0076] The complete solution, the first solution, and the second solution all require importing the voice token extraction model. The specific architecture of the voice token extraction model remains unchanged. Therefore, the difference between the complete solution, the first solution, and the second solution lies in performing different processing using the same architecture in the voice token extraction model.

[0077] The voice token extraction model includes an audio coding layer and a residual vector quantization layer. In the complete scheme, the first scheme, and the second scheme, the audio coding layer is used to encode features based on the frequency domain features to obtain coded features. The difference lies in the residual vector quantization layer, which performs different processing to achieve the complete scheme, the first scheme, and the second scheme.

[0078] In the complete solution:

[0079] The residual vector quantization layer is configured to perform segmented residual quantization based on the encoded features.

[0080] In the first scheme:

[0081] The residual vector quantization layer is configured to perform segmented quantization based on the encoded features; the segmented quantization includes:

[0082] A fourth operation is performed based on the fourth feature to obtain several second segmentation features; the fourth feature includes the encoded feature or the sixth feature obtained through the fifth operation; the fourth operation includes segmenting the encoded feature or the sixth feature.

[0083] A fifth operation is performed based on all the second segmentation features to obtain a sixth feature; the fifth operation includes using VQ quantization to perform vector quantization on all the second segmentation features to obtain several second quantized features, and concatenating all the second quantized features to obtain the sixth feature;

[0084] Repeat the fourth and fifth operations n times, and the sixth feature obtained from the nth fifth operation is the voice token.

[0085] Specifically, in the first scheme, since residual calculation is not performed and the scheme needs to meet the standard, it is necessary to make up for the residual calculation part by segmentation. This can be understood as using segmentation processing to replace the residual calculation part and achieving the effect of both segmentation processing and residual calculation.

[0086] It should be noted that, since the segmentation process is used to replace the residual calculation, the number of segmentation processes in the first scheme needs to be increased adaptively.

[0087] Since the first scheme abandons residual calculation, the first half of the specific implementation of the scheme is similar to the complete scheme, which is to repeatedly segment and quantize the features. The difference between the first scheme and the complete scheme is that the features after VQ quantization and concatenation are directly segmented and then repeatedly segmented, VQ quantized and concatenated to achieve the same technical effect as the complete scheme, that is, to extract the same number of voice tokens.

[0088] In the second option:

[0089] The residual vector quantization layer is configured to perform residual quantization processing based on the encoded features; the residual quantization processing includes:

[0090] The sixth operation is performed based on the seventh feature to obtain the eighth feature; the seventh feature includes an encoded feature or a ninth feature obtained through the seventh operation; the sixth operation includes vector quantization of the seventh feature using VQ quantization to obtain the third quantized feature;

[0091] The seventh operation is performed based on the eighth feature to obtain the ninth feature; the seventh operation includes adding the eighth feature and the seventh feature to obtain the tenth feature;

[0092] Repeat the sixth and seventh operations n times, and the tenth feature obtained from the nth seventh operation is the voice token.

[0093] Specifically, in the second scheme, since no segmentation process is performed and the scheme needs to meet the standard, it is necessary to compensate for the segmentation process through residual calculation. This can be understood as using residual calculation to replace the segmentation process and achieving the effect of both segmentation and residual calculation.

[0094] It should be noted that since residual calculation is used instead of segmentation, the number of residual calculations in the second scheme needs to be increased adaptively.

[0095] Since the second scheme omits segmentation processing, the latter part of its implementation is similar to the complete scheme, both involving repeated VQ quantization and residual calculation of the features. The difference between the second scheme and the complete scheme is that the preprocessed features are directly VQ quantized, and by repeatedly performing VQ quantization and residual calculation, the same technical effect as the complete scheme is achieved, namely, extracting the same number of voice tokens.

[0096] It should be noted that VQ quantization is an essential step in all three schemes: the complete scheme, the first scheme, and the second scheme.

[0097] Based on the above complete scheme, first scheme, and second scheme, it can be seen that the complete scheme is an integration of the first scheme and the second scheme; it can be understood that the first scheme only performs segmentation and VQ quantization, but does not perform residual calculation; the second scheme only performs VQ quantization and residual calculation; while the complete scheme performs segmentation, VQ quantization, and residual calculation in a complete manner.

[0098] It should be noted that the complete solution, the first solution, and the second solution described above can all extract voice tokens, differing only in the components extracted from the voice tokens. Therefore, the complete solution, the first solution, or the second solution can be selected depending on different practical needs. This embodiment mainly focuses on the complete solution; therefore, the following embodiments are all based on the complete solution.

[0099] The VQ quantization process maps points in the vector space to a finite number of representative points, also known as codebook vectors or cluster centers. By quantizing the input data to these representative points, the model representation can be significantly simplified, thereby reducing computational and storage requirements. Simultaneously, this quantization process helps reduce overfitting and improve the model's generalization ability.

[0100] Figure 9 This is a schematic diagram of the audio coding layer in a speech feature processing method according to this application.

[0101] See Figure 9 As can be seen, the audio coding layer consists of a first convolutional unit, a ResNet network unit, and a second convolutional unit; the ResNet network unit includes a plurality of residual blocks; the first convolutional unit is connected to each of the residual blocks in the ResNet network unit.

[0102] Figure 10 This is a schematic diagram of the audio decoding layer in a speech feature processing method of this application.

[0103] See Figure 10 It is known that the audio decoding layer includes a first block, a second block, and a third block; the first block includes a deconvolution unit, a ResNet network unit, and a third convolution unit; the second block includes a deconvolution unit, an MRF unit, and a fourth convolution unit; and the third block includes a PQMF unit.

[0104] Among them, the PQMF unit stands for Perfect Reconstruction Quadrature Mirror Filter Bank, which is an advanced signal processing technology that is widely used in audio encoding and decoding, digital communication, image processing and multimedia compression and other fields.

[0105] PQMF technology offers several advantages, such as high efficiency, flexibility, and ease of implementation. However, it also faces some challenges in practical applications. For example, the design and optimization of filter banks require advanced mathematical and signal processing knowledge, and the implementation complexity of filter banks is also high, necessitating the use of advanced digital signal processing techniques to reduce computational and storage requirements.

[0106] Furthermore, this embodiment also includes a third solution.

[0107] In the third scheme, the voice token extraction model includes an audio encoding layer, a residual vector quantization layer, and an audio decoding layer. The third scheme shares similarities with the complete scheme but also has differences. The similarities between the third scheme and the complete scheme are:

[0108] In both the complete scheme and the third scheme, the trained speech token extraction model is used to extract speech tokens, and the speech token extraction model has the functions of encoding, segmentation, VQ quantization, residual calculation and decoding.

[0109] The difference between the third option and the complete option is:

[0110] If the encoding parameter of the feature encoding in the complete scheme is denoted as 1, the encoding parameter in the third scheme will be 2 or 4. It can be seen that the feature encoding strength in the third scheme is enhanced, so the feature encoding in the third scheme is called enhanced feature encoding. If the parameter of the segmented residual quantization in the complete scheme is denoted as 2, the parameter of the segmented residual quantization in the third scheme will be 1. Specifically, the number of segments and the number of residuals are reduced, which is equivalent to the reduction in the complete scheme. It can be seen that the segmented residual quantization in the third scheme is weakened, so the segmented residual quantization in the third scheme is called weakened segmented residual quantization.

[0111] The differences between the third option and the complete option also include:

[0112] The third scheme emphasizes the training of the speech token extraction model, while the complete scheme tends to utilize the trained speech token extraction model. Therefore, the third scheme must emphasize the audio decoding layer required during model training. Similar to the complete scheme, the audio decoding layer is used for model convergence iteration. However, the difference lies in the fact that the intensity of feature encoding and segmentation residual quantization in the third scheme is adjusted. Correspondingly, the feature decoding in the third scheme also needs adaptive adjustment to achieve the desired effect. If the feature decoding parameter in the complete scheme is denoted as 1, the feature decoding parameter in the third scheme is 2 or 4. It can be seen that the feature decoding in the third scheme is enhanced, so the feature decoding in the third scheme is called enhanced feature decoding.

[0113] The following conclusions were drawn from the experiment on the third scheme:

[0114] When both the number of segments and the number of residuals are set to 1 (i.e., reducing the number of segments and residuals), by increasing the scale of the encoding and decoding parameters, for example, by increasing the encoding parameters to 2 or even 4 times the original size, the audio can be compressed into a single token. This not only achieves the effect of having 2-4 sets of speech tokens, but also further reduces the number of tokens, greatly simplifying the model training process. It allows direct use of the existing model training process, making model training more efficient and stable.

[0115] Figure 11 This is a schematic diagram of the residual block in a speech feature processing method according to this application.

[0116] See Figure 11 It is known that, in the ResNet network unit, each residual block includes a fifth convolutional unit, a first activation function unit, a sixth convolutional unit, and a second activation function unit connected in sequence; the input port of the fifth convolutional unit and the output port of the second activation function unit are connected.

[0117] Furthermore, in some embodiments, each residual block in the ResNet network unit further includes a spatial convolution unit and a first feature fusion unit; in the residual block, the fifth convolution unit, the spatial convolution unit, the first activation function unit, the first feature fusion unit, the sixth convolution unit, and the second activation function unit are connected sequentially in a stacked structure;

[0118] The fifth convolutional unit is composed of multiple first convolutional kernels, wherein the first convolutional kernels are point convolutions, and the number of the first convolutional kernels is greater than the number of channels of the convolutional kernels in the first convolutional unit.

[0119] The spatial convolutional unit is composed of multiple second convolutional kernels, the size of which is larger than that of the first convolutional kernel; the spatial convolutional unit is used to divide the output of the fifth convolutional unit into multiple subsequences, and to perform convolution processing on each subsequence through a second convolutional kernel.

[0120] The first feature fusion unit includes an attention layer, which is used to perform feature fusion processing on the output of each second convolutional kernel based on the calculation of attention weights;

[0121] The sixth convolutional unit is composed of multiple third convolutional kernels, each of which is a point convolution. The number of third convolutional kernels is the same as the number of channels of the convolutional kernels in the first convolutional unit.

[0122] Specifically, in this embodiment, features are compressed, segmented, enhanced, and fused in each residual block of the ResNet network unit, thereby significantly enhancing the ResNet network's ability to extract local features.

[0123] Specifically, the fifth convolutional unit performs feature compression along the feature channel dimension; after feature compression, the spatial convolutional unit segments the output of the fifth convolutional unit to obtain several sub-sequences, thus completing feature segmentation; after feature segmentation, the first feature fusion unit performs attention-weight-based feature fusion on the sub-sequences.

[0124] Furthermore, in some embodiments, the ResNet network unit further includes a downsampling unit and a second feature fusion unit set independently of the residual block, wherein,

[0125] The downsampling unit is used to perform frequency domain downsampling processing on the output of the first convolutional unit;

[0126] The second feature fusion unit includes an attention layer, which is used to perform feature fusion processing on the output of the downsampling unit and the output of the ResNet network unit based on the calculation of attention weights, and input the fused features into the second convolution unit.

[0127] Specifically, in this embodiment, based on the ResNet network, it is fused with features from other stages in the audio encoding process to process global features at multiple scales and levels, thereby enhancing the grasp of details and global information.

[0128] Figure 2 This is a flowchart of the preprocessing step in a speech feature processing method according to this application.

[0129] See Figure 2 It can be seen that, further, in some embodiments, the step of preprocessing the audio data includes:

[0130] S110, The audio data is frequency-domain converted using the Fourier transform algorithm to obtain the frequency domain features.

[0131] Specifically, in this embodiment, the preprocessing scheme for the audio data is to use the Short-Time Fourier Transform (STFT) algorithm to perform frequency domain transformation on the audio data, thereby obtaining the frequency domain features.

[0132] It should be noted that, depending on different needs, the preprocessing of the audio data can be integrated into the voice token extraction model.

[0133] Figure 3 This is a functional flowchart of the audio coding layer in a speech feature processing method of this application.

[0134] See Figure 3 Furthermore, in some embodiments, the audio encoding layer is further configured as follows:

[0135] S210, perform Mel-frequency conversion based on the frequency domain features to obtain the coding features.

[0136] Specifically, in this embodiment, in step S210, the frequency domain features are subjected to Mel-spectrum transformation through the audio coding layer to obtain the coded features.

[0137] Mel spectrum is a spectral representation method based on the Mel scale. The Mel scale is a non-linear frequency scale that is based on the human ear's sensory perception of equidistant pitch changes. Under this scale, the low-frequency range has higher resolution, capturing more detail, while the high-frequency range has lower resolution, reducing unnecessary complexity. This characteristic allows the Mel spectrum to more effectively reflect the human ear's perceptual characteristics of audio signals.

[0138] The calculation of the Mel spectrum typically involves the following steps:

[0139] Pre-emphasis: The original audio signal is pre-emphasized to enhance high-frequency components and reduce the impact of low-frequency noise.

[0140] Frame-based windowing: The audio signal is divided into multiple short frames, and each frame is processed by a windowing function to reduce spectral leakage and improve the accuracy of the analysis.

[0141] FFT Transform: Perform a Fast Fourier Transform (FFT) on each frame of signal to convert the time-domain signal into a frequency-domain signal, which is convenient for further analysis.

[0142] Mel filter bank processing: The FFT-transformed spectrum is processed through a set of bandpass filters distributed on the Mel scale. These filters are typically triangular and uniformly distributed on the Mel scale. The response of each filter represents the contribution of that frequency to the Mel spectrum.

[0143] Mel spectrum analysis has wide applications in speech recognition, music classification, and voiceprint recognition. Its advantages include:

[0144] It conforms to the characteristics of human hearing: Mel spectrum can well simulate the human ear's perception of frequency and has a good ability to represent audio signals.

[0145] Rich in content: Through Mel filter bank processing, Mel spectrum can capture the energy distribution of audio signals at different frequencies, providing rich feature information.

[0146] Low correlation of coefficients: Mel frequency cepstral coefficients (MFCC) are features extracted from the Mel spectrum. They have low correlation and are easier to establish statistical models such as Gaussian mixture models.

[0147] Figure 4 This is a flowchart illustrating the function of the residual vector quantization layer in a speech feature processing method according to this application.

[0148] See Figure 4 Furthermore, in some embodiments, the residual vector quantization layer is further configured as follows:

[0149] S220, perform a first operation based on the first feature to obtain a second feature, wherein the first feature includes the encoded feature or a third feature obtained through the second operation;

[0150] S230, perform a second operation based on the second feature to obtain the third feature, and repeat the first operation n times and the second operation n-1 times;

[0151] S240, perform a third operation on the n second features that have completed the first operation to obtain several sets of the voice tokens.

[0152] Specifically, in this embodiment, it can be understood that a number of first and second operations are performed, and the features obtained after each first operation are performed on a third operation to obtain the voice token, thereby completing the extraction of the voice token.

[0153] In step S200, the first operation can be the first start of the step or the nth start of the step cycle. Therefore, the feature for performing the first operation needs to be the coded feature or the third feature obtained through the second operation.

[0154] In step S230, the second operation can be the first end of the step or the nth end of the step cycle. Regardless of whether it is the first step or the nth step, the feature processed by step S230 is the feature processed by step S220, thereby realizing the closed loop between step S220 and step S230.

[0155] In step S240, before performing the third operation, it is necessary to confirm in advance whether the loop between steps S220 and S230 has met the target requirements. Only when the target requirements are met, that is, when the number of loops reaches a certain value, can the second feature that has completed the first operation in each step loop be performed on the third operation, so as to realize the overall process and obtain the implementation voice token.

[0156] Figure 5 This is a flowchart of the first operation in a speech feature processing method according to this application.

[0157] See Figure 5 It can be seen that the first operation includes:

[0158] S221, the encoded feature or the third feature is segmented to obtain the first segmentation feature;

[0159] S222, VQ quantization is used to perform vector quantization on all the first segmentation features to obtain several first quantized features;

[0160] S223, concatenate all the first quantization features to obtain the second feature.

[0161] Specifically, in this embodiment, the features are first segmented to obtain several segmented features. VQ quantization is then performed on all segmented features, and the VQ quantized features are then concatenated to complete one of the first operations.

[0162] Figure 6 This is a flowchart of the second operation in a speech feature processing method according to this application.

[0163] See Figure 6 It can be seen that the second operation includes:

[0164] S231, Subtract the first feature from the second feature to obtain the third feature.

[0165] Specifically, in this embodiment, the second operation can be completed by subtracting the input features of the first operation from the features obtained after completing the first operation.

[0166] Figure 7This is a flowchart of the third operation in a speech feature processing method of this application.

[0167] See Figure 7 It can be seen that the third operation includes:

[0168] S241, add up the second features of each completed first operation to obtain several sets of the voice tokens.

[0169] Specifically, in this embodiment, the features obtained after each first operation are added together, and the third operation can be performed to extract the voice token.

[0170] The second step and the third operation can be understood as residual calculation.

[0171] Furthermore, in some embodiments, the voice token extraction model further includes an audio decoding layer; the audio decoding layer is applied to the model training process of the voice token extraction model; the audio decoding layer is configured to perform feature decoding based on the voice token to obtain target audio data; the target audio data is used to verify the integrity of the voice token, thereby training the voice token extraction model.

[0172] Specifically, in this embodiment, the audio decoding layer is mainly used in the training process of the voice token extraction model. During the training process of the voice token extraction model, the audio is output through the audio decoding layer, and the audio is compared with the voice token to verify the performance of the model extraction. The process is iterated continuously to achieve the training of the voice token extraction model.

[0173] For example, a 1-second 16kHz audio file has 16,000 float points, occupying 16,000 × 4 = 64,000 bits. After compression, the feature is shortened by a factor of 320. The shortened feature is then quantized to obtain four tokens, resulting in 16,000 / 320 = 50 points. The feature corresponding to each point is further compressed into four tokens, each of which is an integer type. Therefore, the tokens obtained after compressing a 1-second audio file occupy 50 × 4 × 2 = 400 bits, showing that the overall audio compression is 160 times.

[0174] Furthermore, this embodiment also provides a speech processing method, which uses the speech feature processing method in any of the above embodiments to extract features.

[0175] Among them, speech processing methods can include speech recognition, voice cloning, and all speech-based feature extraction methods.

[0176] The advantages of this embodiment include:

[0177] The voice token extraction method described in this embodiment can compress audio into a small number of tokens (1-4 groups), reducing the total number of tokens without significant audio loss. During audio restoration, the tokens can be input into the decoding network to obtain the audio output. Furthermore, the introduction of a PDMF decoder during decoding accelerates audio compression by approximately four times, enabling all modules to run in streaming mode.

[0178] The aforementioned voice token extraction method demonstrates excellent application performance in audio transmission. Client A encodes the audio and extracts the token using an encoder, ultimately compressing it into a feature Z. This Z is then transmitted over the network to the target client B, where it is decoded and used by the decoder. Taking a 1-second audio clip as an example, the compressed data yields approximately 0.4kb of tokens, which can be quickly transmitted over the network for client B to decode and reconstruct. This significantly improves transmission quality without compromising audio quality.

[0179] For ease of explanation, the above description has been provided in conjunction with specific embodiments. However, the discussion in some embodiments is not intended to be exhaustive or to limit the embodiments to the specific forms disclosed above. Various modifications and variations can be obtained based on the above teachings. The selection and description of the above embodiments are for the purpose of better explaining the contents of this disclosure, thereby enabling those skilled in the art to better utilize the embodiments.

Claims

1. A speech feature processing method, characterized in that, The processing method includes: Acquire audio data, preprocess the audio data to obtain frequency domain features; The frequency domain features are imported into a speech token extraction model to extract speech tokens, resulting in several sets of speech tokens. The speech token extraction model includes an audio coding layer and a residual vector quantization layer. The audio coding layer is configured to perform feature encoding based on the frequency domain features to obtain coded features. The residual vector quantization layer is configured to perform segmented residual quantization processing based on the coded features. The audio coding layer consists of a first convolutional unit, a ResNet network unit, and a second convolutional unit; the ResNet network unit includes a plurality of residual blocks; the first convolutional unit is connected to each of the residual blocks in the ResNet network unit. In the ResNet network unit, each residual block includes a fifth convolutional unit, a first activation function unit, a sixth convolutional unit, and a second activation function unit connected in sequence; the input port of the fifth convolutional unit and the output port of the second activation function unit are connected. The voice token extraction model further includes an audio decoding layer; the audio decoding layer is applied to the model training process of the voice token extraction model; the audio decoding layer is configured to perform feature decoding based on the voice token to obtain target audio data; the target audio data is used to verify the integrity of the voice token, thereby training the voice token extraction model; The audio decoding layer includes a first block, a second block, and a third block; the first block includes a deconvolution unit, a ResNet network unit, and a third convolution unit; the second block includes a deconvolution unit, an MRF unit, and a fourth convolution unit; and the third block includes a PQMF unit.

2. The speech feature processing method according to claim 1, characterized in that, The preprocessing steps for the audio data include: The audio data is frequency-domain converted using the Fourier transform algorithm to obtain the frequency domain features; The audio coding layer is also configured to: The coding features are obtained by performing Mel-spectrum transformation based on the frequency domain features.

3. The speech feature processing method according to claim 1, characterized in that, The residual vector quantization layer is also configured to: A first operation is performed based on a first feature to obtain a second feature, wherein the first feature includes the encoded feature or a third feature obtained through the second operation; Perform a second operation based on the second feature to obtain the third feature, and repeat the first operation n times and the second operation n-1 times. Perform a third operation on n second features that have completed the first operation to obtain several sets of the voice tokens.

4. The speech feature processing method according to claim 3, characterized in that, The first operation includes: The encoded feature or the third feature is segmented to obtain the first segmentation feature; VQ quantization is used to perform vector quantization on all the first segmentation features to obtain several first quantized features, and all the first quantized features are concatenated to obtain the second feature; The second operation includes: Subtracting the first feature from the second feature yields the third feature; The third operation includes: The second feature of each completed first operation is summed to obtain several sets of the voice tokens.

5. The speech feature processing method according to claim 1, characterized in that, In the ResNet network unit, each residual block further includes a spatial convolution unit and a first feature fusion unit; in the residual block, the fifth convolution unit, the spatial convolution unit, the first activation function unit, the first feature fusion unit, the sixth convolution unit, and the second activation function unit are connected in sequence in a stacked structure; The fifth convolutional unit is composed of multiple first convolutional kernels, wherein the first convolutional kernels are point convolutions, and the number of the first convolutional kernels is greater than the number of channels of the convolutional kernels in the first convolutional unit. The spatial convolutional unit is composed of multiple second convolutional kernels, the size of which is larger than that of the first convolutional kernel; the spatial convolutional unit is used to divide the output of the fifth convolutional unit into multiple subsequences, and to perform convolution processing on each subsequence through a second convolutional kernel. The first feature fusion unit includes an attention layer, which is used to perform feature fusion processing on the output of each second convolutional kernel based on the calculation of attention weights; The sixth convolutional unit is composed of multiple third convolutional kernels, each of which is a point convolution. The number of third convolutional kernels is the same as the number of channels of the convolutional kernels in the first convolutional unit.

6. The speech feature processing method according to claim 5, characterized in that, The ResNet network unit further includes a downsampling unit and a second feature fusion unit, which are independent of the residual blocks. The downsampling unit is used to perform frequency domain downsampling processing on the output of the first convolutional unit; The second feature fusion unit includes an attention layer, which is used to perform feature fusion processing on the output of the downsampling unit and the output of the ResNet network unit based on the calculation of attention weights, and input the fused features into the second convolution unit.

7. A speech processing method, characterized in that, The speech processing method uses the speech feature processing method according to any one of claims 1 to 6 to extract features.

Images