A speech token extraction method and a speech processing method

Abstract

The application provides a speech token extraction method and a speech processing method, and relates to the technical field of speech token extraction. The extraction method comprises the following steps: acquiring audio data, pre-processing the audio data to obtain frequency domain features; introducing the frequency domain features into a speech token extraction model to extract speech tokens, and obtaining a plurality of groups of speech tokens; the speech token extraction model comprises an audio coding layer and a residual vector quantization layer; the audio coding layer is configured to perform feature coding according to the frequency domain features to obtain coded features; and the residual vector quantization layer is configured to perform segmented residual quantization processing according to the coded features. The application solves the problem of excessive extraction groups of speech tokens extracted by the existing speech token extraction technology through the above method.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of speech token extraction, and particularly relates to a speech token extraction method and a speech processing method. BACKGROUND

[0002] In recent years, with the rapid development of artificial intelligence technology, large models (LLMs) have made breakthrough progress in natural language processing. LLMs represented by ChatGPT, GPT-4 and some domestic advanced models have pushed the application of natural language processing to a new height with their powerful language understanding and generation capabilities. These models not only can conduct complex dialogue interaction, but also can complete text creation, code generation and other tasks, greatly broadening the application scenarios of natural language processing technology.

[0003] However, although LLMs have made remarkable achievements in natural language processing, their research and application in the field of audio are relatively less. As an important information carrier, audio processing and analysis are crucial for many application scenarios, such as speech recognition, audio classification, music generation, etc. In order to expand the powerful capabilities of LLMs to the field of audio, a key problem needs to be solved: how to convert audio data into a format that LLMs can process. In natural language processing, LLMs usually receive and process discrete text tokens (i.e. basic units in text, such as words, characters or subwords). Therefore, in order to apply LLMs to the field of audio, we need to completely discretize audio data to obtain a similar form to text tokens. In this process, audio data compression and encoding technology plays a crucial role.

[0004] Currently, the techniques for compressing audio into tokens mainly include the SoundStream technology proposed by Google and the encodec technology proposed by Meta. Both of these technologies use the technical framework of combining residual quantization layers with autoencoders to compress audio signals into a limited set of tokens. Specifically, the SoundStream and encodec technologies can compress audio signals into 12 or 8 sets of tokens, which represent the basic units or features in the audio signal. However, although these technologies have achieved certain results in audio compression, they still face some challenges. Since these technologies are based on modeling in the time domain of audio, a large number of tokens are needed to complete the complete modeling of the audio signal. This means that in the subsequent audio modeling process, LLMs need to process a large number of tokens, which not only increases the computational complexity but also can have a negative impact on the modeling effect. In addition, too many tokens can also cause the model to overfit or underfit during training, further affecting the accuracy and efficiency of audio processing. SUMMARY

[0005] The present application provides a voice token extraction method and a voice processing method to solve the problem of excessive voice token extraction groups in existing voice token extraction technology.

[0006] In a first aspect, the present application provides a voice token extraction method, which comprises:

[0007] Obtaining audio data, pre-processing the audio data to obtain frequency domain features;

[0008] Importing the frequency domain features into a voice token extraction model for voice token extraction to obtain several sets of voice tokens; the voice token extraction model comprises an audio encoding layer and a residual vector quantization layer; the audio encoding layer is configured to perform feature encoding according to the frequency domain features to obtain encoded features; the residual vector quantization layer is configured to perform segmented residual quantization processing according to the encoded features.

[0009] Preferably, the step of pre-processing the audio data comprises:

[0010] Using a Fourier transform algorithm to frequency-domainize the audio data to obtain the frequency domain features.

[0011] Preferably, the audio encoding layer is further configured to:

[0012] Performing Mel-spectrogram conversion according to the frequency domain features to obtain the encoded features.

[0013] Preferably, the residual vector quantization layer is further configured to:

[0014] performing a first operation according to a first feature to obtain a second feature, the first feature comprising the encoded feature or a third feature obtained by a second operation;

[0015] performing the second operation according to the second feature to obtain the third feature, and repeating the first operation n times and the second operation n-1 times;

[0016] performing a third operation on n second features that have completed the first operation to obtain a plurality of groups of speech tokens.

[0017] Preferably, the first operation comprises:

[0018] segmenting the encoded feature or the third feature to obtain a first segmented feature;

[0019] performing vector quantization on all the first segmented features by VQ quantization to obtain a plurality of first quantized features, and splicing all the first quantized features to obtain the second feature;

[0020] the second operation comprises:

[0021] subtracting the first feature from the second feature to obtain the third feature;

[0022] the third operation comprises:

[0023] adding each second feature that has completed the first operation to obtain a plurality of groups of speech tokens.

[0024] Preferably, the audio encoding layer is composed of a first convolution unit, a Resnet network unit and a second convolution unit; the Resnet network unit comprises a plurality of residual blocks; the first convolution unit is connected to each of the residual blocks in the Resnet network unit; each of the residual blocks in the Resnet network unit comprises a fifth convolution unit, a first activation function unit, a sixth convolution unit and a second activation function unit connected in sequence; and an input port of the fifth convolution unit is connected to an output port of the second activation function unit.

[0025] Preferably, each of the residual blocks in the Resnet network unit further comprises a spatial convolution unit and a first feature fusion unit; in the residual block, the fifth convolution unit, the spatial convolution unit, the first activation function unit, the first feature fusion unit, the sixth convolution unit and the second activation function unit are connected in a layer-by-layer structure in sequence.

[0026] The fifth convolution unit is composed of a plurality of first convolution kernels, the first convolution kernel is a point convolution, and the number of the first convolution kernels is greater than the number of channels of the convolution kernel in the first convolution unit.

[0027] The spatial convolution unit is composed of a plurality of second convolution kernels, the size of the second convolution kernel is greater than that of the first convolution kernel, the spatial convolution unit is used for dividing the output of the fifth convolution unit into a plurality of sub-sequences, and each sub-sequence is respectively subjected to convolution processing through a second convolution kernel.

[0028] The first feature fusion unit includes an attention layer, and the first feature fusion layer is used for performing feature fusion processing on the output of each second convolution kernel based on the calculation of attention weight.

[0029] The sixth convolution unit is composed of a plurality of third convolution kernels, the third convolution kernel is a point convolution, and the number of the third convolution kernels is the same as the number of channels of the convolution kernel in the first convolution unit.

[0030] Preferably, the Resnet network unit further includes a downsampling unit and a second feature fusion unit which are independent of the residual block;

[0031] The downsampling unit is used for performing frequency domain downsampling processing on the output of the first convolution unit.

[0032] The second feature fusion unit includes an attention layer, and the second feature fusion layer is used for performing feature fusion processing on the output of the downsampling unit and the output of the Resnet network unit based on the calculation of attention weight, and inputting the fused features to the second convolution unit.

[0033] Preferably, the speech token extraction model further includes an audio decoding layer; the audio decoding layer is applied to the model training process of the speech token extraction model; the audio decoding layer is configured to perform feature decoding according to the speech token to obtain target audio data; the target audio data is used to verify the integrity of the speech token, so as to train the speech token extraction model.

[0034] Preferably, the audio decoding layer includes a first block, a second block and a third block; the first block includes a deconvolution unit, a Resnet network unit and a third convolution unit; the second block includes a deconvolution unit, an MRF unit and a fourth convolution unit; and the third block includes a PQMF unit.

[0035] The extraction method further includes:

[0036] Obtaining audio data, pre-processing the audio data to obtain frequency domain features;

[0037] The frequency domain features are introduced into a speech token extraction model to perform speech token extraction, and a plurality of groups of speech tokens are obtained; the speech token extraction model comprises an audio coding layer and a residual vector quantization layer; the audio coding layer is configured to perform feature coding according to the frequency domain features to obtain coded features; the residual vector quantization layer is configured to perform segmented quantization processing according to the coded features; the segmented quantization processing comprises:

[0038] A fourth operation is performed according to the fourth features, and a plurality of second segmented features are obtained; the fourth features comprise the coded features or sixth features obtained through a fifth operation; the fourth operation comprises segmented processing on the coded features or the sixth features;

[0039] A fifth operation is performed according to all the second segmented features, and sixth features are obtained; the fifth operation comprises vector quantization of all the second segmented features by VQ quantization to obtain a plurality of second quantized features, and splicing of all the second quantized features to obtain the sixth features;

[0040] The fourth operation and the fifth operation are repeated n times, and the sixth features obtained through the fifth operation in the n th time are the speech tokens.

[0041] The extraction method further comprises:

[0042] Obtaining audio data, and performing preprocessing on the audio data to obtain frequency domain features;

[0043] The frequency domain features are introduced into a speech token extraction model to perform speech token extraction, and a plurality of groups of speech tokens are obtained; the speech token extraction model comprises an audio coding layer and a residual vector quantization layer; the audio coding layer is configured to perform feature coding according to the frequency domain features to obtain coded features; the residual vector quantization layer is configured to perform residual quantization processing according to the coded features; the residual quantization processing comprises:

[0044] A sixth operation is performed according to seventh features, and eighth features are obtained; the seventh features comprise coded features or ninth features obtained through a seventh operation; the sixth operation comprises vector quantization of the seventh features by VQ quantization to obtain third quantized features;

[0045] The seventh operation is performed according to the eighth features, and the ninth features are obtained; the seventh operation comprises adding the eighth features and the seventh features to obtain tenth features;

[0046] The sixth operation and the seventh operation are repeated n times, and the tenth feature obtained by the n-th seventh operation is the speech token.

[0047] The extraction method comprises:

[0048] Obtaining audio data, pre-processing the audio data to obtain frequency domain features;

[0049] The frequency domain features are input into a speech token extraction model to extract speech tokens, and at least one group of speech tokens is obtained; the speech token extraction model comprises an audio encoding layer, a residual vector quantization layer, and an audio decoding layer; the audio encoding layer is configured to perform enhanced feature coding according to the frequency domain features to obtain enhanced coding features; the residual vector quantization layer is configured to perform weakened partitioned residual quantization processing according to the enhanced coding features; and the audio decoding layer is configured to perform enhanced feature decoding on the speech tokens to obtain target audio data, the audio decoding layer is applied to a model training process of the speech token extraction model, and the target audio data is used to verify the integrity of the speech tokens, so as to train the speech token extraction model.

[0050] In a second aspect, the present application also provides a speech processing method, which is applied to the above-mentioned speech token extraction method for feature extraction.

[0051] As can be seen from the above, the present application provides a speech token extraction method and a speech processing method, the extraction method comprising obtaining audio data, pre-processing the audio data to obtain frequency domain features; inputting the frequency domain features into a speech token extraction model to extract speech tokens, and obtaining several groups of speech tokens; the speech token extraction model comprises an audio encoding layer and a residual vector quantization layer; the audio encoding layer is configured to perform feature coding according to the frequency domain features to obtain coding features; and the residual vector quantization layer is configured to perform partitioned residual quantization processing according to the coding features. The present application solves the problem of excessive extraction groups of speech tokens extracted by the existing speech token extraction technology through the above-mentioned method. BRIEF DESCRIPTION OF DRAWINGS

[0052] In order to more clearly illustrate the technical solutions of the present application, the drawings needed in the embodiments will be briefly introduced as follows, and obviously, other drawings can also be obtained by those skilled in the art without creative labor on the premise of not deviating from the concept of the present application.

[0053] Figure 1 A flowchart of a speech token extraction method of the present application;

[0054] Figure 2 A flow chart of a pre-processing procedure in a speech token extraction method of the present application;

[0055] Figure 3 A functional flow chart of an audio encoding layer in a speech token extraction method of the present application;

[0056] Figure 4 A functional flow chart of a residual vector quantization layer in a speech token extraction method of the present application;

[0057] Figure 5 A flow chart of a first operation in a speech token extraction method of the present application;

[0058] Figure 6 A flow chart of a second operation in a speech token extraction method of the present application;

[0059] Figure 7 A flow chart of a third operation in a speech token extraction method of the present application;

[0060] Figure 8 A schematic diagram of a speech token extraction model in a speech token extraction method of the present application;

[0061] Figure 9 A schematic diagram of an audio encoding layer in a speech token extraction method of the present application;

[0062] Figure 10 A schematic diagram of an audio decoding layer in a speech token extraction method of the present application;

[0063] Figure 11 A schematic diagram of a residual block in a speech token extraction method of the present application. DETAILED DESCRIPTION

[0064] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present application.

[0065] It should be noted that the brief description of the terms in the present application is only for the convenience of understanding the subsequently described embodiments, and is not intended to limit the embodiments of the present application. Unless otherwise specified, these terms should be understood according to their ordinary and general meanings.

[0066] It should be noted that the terms "exemplary" and "for example" are used herein to mean "an example of" rather than "an ideal". Any implementation or design scheme described herein as "exemplary" or "for example" should not be construed as preferred or advantageous over other implementations or design schemes. Rather, the use of the terms "exemplary" and "for example" is intended to present concepts in a particular manner.

[0067] Figure 1 A flowchart of a speech token extraction method according to the present application.

[0068] Figure 8 A schematic diagram of a speech token extraction model in a speech token extraction method according to the present application.

[0069] Referring to Figure 1 and Figure 8 It can be seen that the present embodiment provides a speech token extraction method, wherein the extraction method comprises a complete scheme, a first scheme and a second scheme; wherein the complete scheme, the first scheme and the second scheme differ in the steps of model processing.

[0070] The complete scheme, the first scheme and the second scheme all comprise:

[0071] S100, obtaining audio data, pre-processing the audio data to obtain frequency domain features;

[0072] S200, importing the frequency domain features into a speech token extraction model to extract speech tokens, to obtain a plurality of groups of speech tokens.

[0073] Specifically, in the present embodiment, the complete scheme, the first scheme and the second scheme all need to first obtain audio data, pre-process and import the pre-processed data into the speech token extraction model to extract speech tokens, wherein the difference lies in that the speech token extraction model extracts speech tokens in different ways in different schemes.

[0074] The complete scheme, the first scheme and the second scheme all need to import the speech token extraction model, and the specific architecture in the speech token extraction model is unchanged, so the difference between the complete scheme, the first scheme and the second scheme is that the same architecture in the speech token extraction model is used for different processing.

[0075] The voice token extraction model includes an audio coding layer and a residual vector quantization layer. In the complete scheme, the first scheme, and the second scheme, the audio coding layer is used to encode features based on the frequency domain features to obtain coded features. The difference lies in the residual vector quantization layer, which performs different processing to achieve the complete scheme, the first scheme, and the second scheme.

[0076] In the complete solution:

[0077] The residual vector quantization layer is configured to perform segmented residual quantization based on the encoded features.

[0078] In the first scheme:

[0079] The residual vector quantization layer is configured to perform segmented quantization based on the encoded features; the segmented quantization includes:

[0080] A fourth operation is performed based on the fourth feature to obtain several second segmentation features; the fourth feature includes the encoded feature or the sixth feature obtained through the fifth operation; the fourth operation includes segmenting the encoded feature or the sixth feature.

[0081] A fifth operation is performed based on all the second segmentation features to obtain a sixth feature; the fifth operation includes using VQ quantization to perform vector quantization on all the second segmentation features to obtain several second quantized features, and concatenating all the second quantized features to obtain the sixth feature;

[0082] Repeat the fourth and fifth operations n times, and the sixth feature obtained from the nth fifth operation is the voice token.

[0083] Specifically, in the first scheme, since residual calculation is not performed and the scheme needs to meet the standard, it is necessary to make up for the residual calculation part by segmentation. This can be understood as using segmentation processing to replace the residual calculation part and achieving the effect of both segmentation processing and residual calculation.

[0084] It should be noted that, since the segmentation process is used to replace the residual calculation, the number of segmentation processes in the first scheme needs to be increased adaptively.

[0085] In the first scheme, the discarded part is the residual calculation. Therefore, in the first half of the implementation of the scheme, the feature is repeatedly segmented and VQ quantized, which is similar to the complete scheme. The difference between the first scheme and the complete scheme is that the feature after VQ quantization and splicing is directly segmented, and through repeated segmentation, VQ quantization and splicing, the same technical effect as the complete scheme is finally achieved, that is, the same number of speech tokens are extracted.

[0086] In the second scheme, the discarded part is the segmentation processing.

[0087] The residual vector quantization layer is configured to perform residual quantization processing on the encoded feature. The residual quantization processing includes:

[0088] According to the seventh feature, the sixth operation is performed to obtain the eighth feature. The seventh feature includes an encoded feature or a ninth feature obtained by the seventh operation. The sixth operation includes performing vector quantization on the seventh feature by using VQ quantization to obtain a third quantized feature.

[0089] According to the eighth feature, the seventh operation is performed to obtain the ninth feature. The seventh operation includes adding the eighth feature and the seventh feature to obtain a tenth feature.

[0090] The sixth operation and the seventh operation are repeated n times. The tenth feature obtained by the seventh operation in the nth time is the speech token.

[0091] Specifically, in the second scheme, since no segmentation processing is performed and the scheme needs to meet the standard, residual calculation is needed to make up for the part of the segmentation processing. It can be understood that the part of the segmentation processing is replaced by residual calculation, and the scheme effect of having both segmentation processing and residual calculation is achieved.

[0092] It should be noted that since the segmentation processing is replaced by residual calculation, the number of residual calculations in the second scheme needs to be adaptively increased.

[0093] In the second scheme, the discarded part is the segmentation processing. Therefore, in the second half of the implementation of the scheme, the feature is repeatedly VQ quantized and residual calculated, which is similar to the complete scheme. The difference between the second scheme and the complete scheme is that the preprocessed feature is directly VQ quantized, and through repeated VQ quantization and residual calculation, the same technical effect as the complete scheme is finally achieved, that is, the same number of speech tokens are extracted.

[0094] It should be noted that in the complete scheme, the first scheme or the second scheme, VQ quantization is an essential step.

[0095] According to the complete scheme, the first scheme and the second scheme, the complete scheme is the integration of the first scheme and the second scheme; it can be understood that only segmentation and VQ quantization are performed in the first scheme, but no residual calculation is performed; only VQ quantization and residual calculation are performed in the second scheme; and in the complete scheme, segmentation, VQ quantization and residual calculation are all performed.

[0096] It should be noted that the complete scheme, the first scheme and the second scheme can all extract speech tokens, and only the components of speech token extraction are different, so according to different actual needs, the complete scheme, the first scheme or the second scheme can be selected. In this embodiment, the complete scheme is mainly introduced, so the following embodiments are all based on the complete scheme.

[0097] The VQ quantization is to map points in a vector space to a limited number of representative points, which are also called codebook vectors or cluster centers. By quantizing input data to these representative points, the representation of the model can be significantly simplified, thereby reducing the amount of calculation and storage requirements. At the same time, this quantization process helps to reduce overfitting and improve the generalization ability of the model.

[0098] Figure 9 A schematic diagram of an audio encoding layer in a speech token extraction method of the application.

[0099] Referring to Figure 9 It can be seen that the audio encoding layer is composed of a first convolution unit, a Resnet network unit and a second convolution unit; the Resnet network unit includes a plurality of residual blocks; the first convolution unit is connected with each of the residual blocks in the Resnet network unit respectively;

[0100] Figure 10 A schematic diagram of an audio decoding layer in a speech token extraction method of the application.

[0101] Referring to Figure 10 It can be seen that the audio decoding layer includes a first block, a second block and a third block; the first block includes an inverse convolution unit, a Resnet network unit and a third convolution unit; the second block includes an inverse convolution unit, an MRF unit and a fourth convolution unit; and the third block includes a PQMF unit.

[0102] The PQMF unit, which is short for Perfect Reconstruction Quadrature Mirror Filter Bank, is an advanced signal processing technology, which is widely used in audio encoding and decoding, digital communication, image processing and multimedia compression and other fields.

[0103] PQMF technology has many advantages, such as high efficiency, flexibility, easy to implement, etc. However, in practical applications, PQMF also faces some challenges. For example, the design and optimization of the filter bank require higher mathematical and signal processing knowledge, at the same time, the implementation complexity of the filter bank is also higher, advanced digital signal processing technology needs to be used to reduce the amount of calculation and storage requirements.

[0104] Further, the embodiment also includes a third scheme.

[0105] In the third scheme, the speech token extraction model includes an audio encoding layer, a residual vector quantization layer, and an audio decoding layer; the third scheme has the same and different with the complete scheme, the same of the third scheme and the complete scheme is:

[0106] No matter in the complete scheme or in the third scheme, the speech token extraction model trained is used to extract the speech token, and the speech token extraction model has the functions of encoding, segmentation, VQ quantization, residual calculation and decoding.

[0107] The difference between the third scheme and the complete scheme is:

[0108] If the encoding parameter of the feature encoding in the complete scheme is recorded as 1, then the encoding parameter in the third scheme is 2 or 4, so it can be seen that the feature encoding in the third scheme is enhanced, so the feature encoding in the third scheme is called enhanced feature encoding; If the parameter of the segmented residual quantization processing in the complete scheme is recorded as 2, then the parameter of the segmented residual quantization processing in the third scheme is 1, wherein the specific change is that the segmentation number and the residual number are equivalent to the complete scheme Decrease, so it can be seen that the segmented residual quantization processing in the third scheme is weakened, so the segmented residual quantization processing in the third scheme is called weakened segmented residual quantization processing.

[0109] Among them, the difference between the third scheme and the complete scheme also includes:

[0110] The third scheme emphasizes the training of the speech token extraction model, while the complete scheme is more inclined to use the trained speech token extraction model. Therefore, in the third scheme, the audio decoding layer required in the model training process must be emphasized. Similar to the complete scheme, the audio decoding layer is used for convergent iteration of the model, but the difference is that the strength of the feature encoding and the segmented residual quantization processing in the third scheme is adjusted, and the corresponding feature decoding in the third scheme also needs to be adaptively adjusted to achieve the expected effect. If the parameter of the feature decoding in the complete scheme is 1, the feature decoding in the third scheme is 2 or 4. As can be seen, the feature decoding in the third scheme is enhanced, so the feature decoding in the third scheme is called enhanced feature decoding.

[0111] In the experiment of the third scheme, the following conclusions are drawn:

[0112] When the segmentation number and the residual number are both 1 (i.e., the segmentation number and the residual number are reduced), by enlarging the parameter scale of encoding and decoding, for example, by changing the encoding parameter to 2 times or even 4 times the original, the audio can be compressed to 1 token group. Not only can the number of speech token groups be 2-4 groups, but also the token number can be further reduced, greatly simplifying the model training process, directly using the existing model training process, making the model training more efficient and stable.

[0113] Figure 11 A schematic diagram of a residual block in a speech token extraction method of the present application.

[0114] Referring to Figure 11 It can be seen that in the Resnet network unit, each residual block includes a fifth convolution unit, a first activation function unit, a sixth convolution unit and a second activation function unit connected in turn; the input port of the fifth convolution unit and the output port of the second activation function unit are connected.

[0115] Further, in some embodiments, in the Resnet network unit, each residual block further includes a spatial convolution unit and a first feature fusion unit; in the residual block, the fifth convolution unit, the spatial convolution unit, the first activation function unit, the first feature fusion unit, the sixth convolution unit and the second activation function unit are connected in a layer-by-layer structure in turn.

[0116] The fifth convolution unit is composed of a plurality of first convolution kernels, the first convolution kernel is a point convolution, and the number of the first convolution kernel is greater than the number of channels of the convolution kernel in the first convolution unit.

[0117] The spatial convolution unit is composed of a plurality of second convolution kernels, and the size of the second convolution kernel is greater than that of the first convolution kernel; the spatial convolution unit is used for dividing the output of the fifth convolution unit into a plurality of subsequences, and each subsequence is respectively processed by a second convolution kernel.

[0118] The first feature fusion unit includes an attention layer, and the first feature fusion layer is used for performing feature fusion processing on the output of each second convolution kernel based on the calculation of attention weights.

[0119] The sixth convolution unit is composed of a plurality of third convolution kernels, and the third convolution kernel is a point convolution, and the number of the third convolution kernel is the same as the number of channels of the convolution kernel in the first convolution unit.

[0120] Specifically, in the embodiment, the features are compressed, divided, enhanced and fused in each residual block in the Resnet network unit, so that the extraction ability of the Resnet network for local features is significantly enhanced.

[0121] The fifth convolution unit is used to compress the features in the feature channel dimension; after the compression of the features is completed, the spatial convolution unit is used to divide the output result of the fifth convolution unit to obtain a plurality of subsequences, so as to complete the division of the features; after the division of the features is completed, the first feature fusion unit is used to fuse the subsequences based on the attention weights.

[0122] Further, in some embodiments, the Resnet network unit further includes a down-sampling unit and a second feature fusion unit which are independent of the residual block, wherein,

[0123] The down-sampling unit is used for performing frequency domain down-sampling processing on the output of the first convolution unit.

[0124] The second feature fusion unit includes an attention layer, and the second feature fusion layer is used for performing feature fusion processing on the output of the down-sampling unit and the output of the Resnet network unit based on the calculation of attention weights, and inputting the fused features to the second convolution unit.

[0125] Specifically, in the embodiment, on the basis of the Resnet network, the features in other stages in the audio coding process are fused to process the global features in multiple scales and multiple levels, and the grasp of details and global information is enhanced.

[0126] Figure 2 A flowchart of a pre-processing process in a speech token extraction method.

[0127] Referring to Figure 2 It can be known that, further, in some embodiments, the step of preprocessing the audio data comprises:

[0128] S110, frequency domainizing the audio data by using a Fourier transform algorithm to obtain the frequency domain feature.

[0129] Specifically, in the present embodiment, the specific scheme of preprocessing the audio data is to frequency domainize the audio data by using a Fourier transform algorithm (stft) to obtain the frequency domain feature.

[0130] It should be noted that, according to different needs, preprocessing the audio data can be integrated into the speech token extraction model.

[0131] Figure 3 A functional flowchart of an audio encoding layer in a speech token extraction method of the present application.

[0132] Referring to Figure 3 It can be known that, further, in some embodiments, the audio encoding layer is further configured to:

[0133] S210, performing Mel-spectrum conversion according to the frequency domain feature to obtain the encoding feature.

[0134] Specifically, in the present embodiment, the Mel-spectrum conversion of the frequency domain feature is performed by the audio encoding layer in step S210 to obtain the encoding feature.

[0135] The Mel-spectrum is a kind of spectrum representation method based on Mel scale. Mel scale is a kind of nonlinear frequency scale, which is based on the sensory judgment of human ears to equal-interval pitch changes. In this scale, the resolution of the low-frequency part is higher, which can capture more details, while the resolution of the high-frequency part is lower, which reduces unnecessary complexity. This feature makes the Mel-spectrum more effectively reflect the perceptual characteristics of human ears to audio signals.

[0136] The calculation of Mel-spectrum usually includes the following steps:

[0137] Pre-emphasis: pre-emphasis processing is performed on the original audio signal to enhance the high-frequency component and reduce the influence of low-frequency noise.

[0138] Frame and windowing: the audio signal is divided into multiple short-time frames, and each frame signal is processed by a windowing function to reduce spectral leakage and improve the accuracy of analysis.

[0139] FFT transform: Fast Fourier transform (FFT) is performed on each frame of signal to convert time-domain signal to frequency-domain signal for further analysis.

[0140] Mel filter bank processing: The frequency spectrum after FFT transform is processed by a set of band-pass filters distributed according to Mel scale. These filters are usually triangular and uniformly distributed in Mel scale. The response of each filter represents the contribution of the frequency to the Mel spectrum.

[0141] Mel spectrum has wide application in speech recognition, music classification, voiceprint recognition, etc. Its advantages are:

[0142] In line with human auditory characteristics: Mel spectrum can well simulate the perception characteristics of human ear to frequency and has good representation ability to audio signal.

[0143] Rich expression of content: Through Mel filter bank processing, Mel spectrum can capture the energy distribution of audio signal at different frequencies and provide rich feature information.

[0144] Small correlation of coefficients: Mel frequency cepstral coefficients (MFCC) are extracted from Mel spectrum and have smaller correlation, which is easier to establish statistical models such as Gaussian mixture model.

[0145] Figure 4 A functional flowchart of a residual vector quantization layer in a speech token extraction method of the present application.

[0146] It can be seen that, further, in some embodiments, the residual vector quantization layer is further configured to: Figure 4

[0147] S220, performing a first operation according to a first feature to obtain a second feature, the first feature including the encoding feature or a third feature obtained by a second operation;

[0148] S230, performing a second operation according to the second feature to obtain the third feature, and repeating the first operation n times and the second operation n-1 times;

[0149] S240, performing a third operation on n second features completing the first operation to obtain a plurality of groups of speech tokens.

[0150] Specifically, in this embodiment, it can be understood that a plurality of times of first operation and second operation are performed, and the feature obtained by completing the first operation each time is subjected to a third operation, so that the speech token is obtained, and the extraction of the speech token is completed.

[0151] ​Wherein, in step S200, the first operation can be the first time of the step or the n time of the step, and the feature processed by the first operation is the encoding feature or the third feature obtained by the second operation.

[0152] Wherein, in step S230, the second operation can be the first time of the step or the n time of the step, and the feature processed by step S230 is the feature processed by step S220, so as to realize the loop between step S220 and step S230.

[0153] Wherein, in step S240, before the third operation, it is necessary to confirm whether the loop between step S220 and step S230 meets the target requirement, and only when the target requirement is met, that is, the number of loops reaches a certain value, the second feature processed by the first operation in each step of the step loop can be subjected to the third operation, so as to realize the whole process and obtain the speech token.

[0154] Figure 5 The flow chart of the first operation in the speech token extraction method.

[0155] Referring to Figure 5 It can be seen that the first operation comprises:

[0156] S221, the encoding feature or the third feature is subjected to segmentation processing to obtain a first segmented feature;

[0157] S222, all the first segmented features are subjected to vector quantization by VQ quantization to obtain a plurality of first quantized features;

[0158] S223, all the first quantized features are spliced to obtain the second feature.

[0159] Specifically, in this embodiment, the features are segmented to obtain a plurality of segmented features, all the segmented features are subjected to VQ quantization, and the features subjected to VQ quantization are spliced, so as to complete the first operation.

[0160] Figure 6 The flow chart of the second operation in the speech token extraction method.

[0161] Referring to Figure 6 It can be seen that the second operation comprises:

[0162] The first feature and the second feature are subtracted to obtain the third feature.

[0163] Specifically, in the embodiment, the input feature of the first operation is subtracted from the feature obtained by completing the first operation, and the second operation described above can be completed.

[0164] Figure 7 A flowchart of a third operation in a voice token extraction method of the application.

[0165] Referring to Figure 7 It can be seen that the third operation includes:

[0166] Each second feature obtained by completing the first operation is added to obtain a plurality of groups of voice tokens.

[0167] Specifically, in the embodiment, the features obtained after each completion of the first operation are added, and the third operation can be completed, so that the voice tokens can be extracted.

[0168] The second step and the third operation can be understood as residual calculation.

[0169] Further, in some embodiments, the voice token extraction model further includes an audio decoding layer; the audio decoding layer is applied to a model training process of the voice token extraction model; the audio decoding layer is configured to perform feature decoding according to the voice token to obtain target audio data; the target audio data is used to verify the integrity of the voice token, so as to train the voice token extraction model.

[0170] Specifically, in the embodiment, the audio decoding layer is mainly applied to the training process of the voice token extraction model. In the training process of the voice token extraction model, audio is output through the audio decoding layer, and the audio is compared with the voice token, so as to verify the performance of the model extraction, and continuously iterate, so as to realize the training of the voice token extraction model.

[0171] For example, 1s16k audio has 16000 float points, occupying 16000*4=64000 bits, and after compression, the feature is shortened by 320 times. The shortened feature is quantized to obtain four groups of tokens, that is, 16000 / 320=50 points are first obtained, and each point corresponds to a feature further compressed into 4 groups of tokens, and each token is an int type. Therefore, the token obtained after compression of the 1s audio occupies 50*4*2=400 bits, and it can be seen that the audio is compressed by 160 times in total.

[0172] Further, the embodiment also provides a speech processing method, which applies the speech token extraction method in any one of the above embodiments to perform feature extraction.

[0173] The speech processing method can include speech recognition, voice cloning, and all feature extraction based on speech.

[0174] Advantages of the embodiment include:

[0175] Through the speech token extraction method in the embodiment, the audio can be compressed into several groups of a small number of tokens (1-4 groups), not only the total number of tokens is reduced, but also there is no obvious audio loss. In the process of audio restoration, the token can be input into the decoding network, and the audio output can be obtained; and the PDMF decoder is introduced in the decoding process, so that the speed of audio compression can be increased by about 4 times, and all modules can realize streaming operation.

[0176] The above speech token extraction has good application effect in the audio transmission process. The client A encodes the audio through the encoder and token extraction, and finally compresses the obtained features into Z, and then transmits Z through the network to the target client B. Z is decoded and used on the client B through the decoder. Taking 1s audio as an example, the features obtained after compression are about 0.4kb, which can be quickly transmitted through the network for decoding and restoration by the client B. The transmission effect is significantly improved, and the audio effect is not affected.

[0177] The above description has been made in combination with specific embodiments for the convenience of explanation. However, the above discussion in some embodiments is not intended to exhaust or limit the embodiments to the specific forms disclosed above. Various modifications and variations can be obtained according to the above teachings. The selection and description of the above embodiments are to better explain the content of the disclosure, so that those skilled in the art can better use the embodiments.

Claims

1. A speech token extraction method, characterized by, The extraction method comprises: obtaining audio data, pre-processing the audio data to obtain frequency domain features; the frequency domain features are introduced into a speech token extraction model to extract speech tokens, and a plurality of groups of speech tokens are obtained; the speech token extraction model comprises an audio coding layer and a residual vector quantization layer; the audio coding layer is configured to perform feature coding according to the frequency domain features to obtain coded features; and the residual vector quantization layer is configured to perform segmented residual quantization processing according to the coded features; the residual vector quantization layer is further configured to: perform a first operation according to a first feature to obtain a second feature, wherein the first feature comprises the coded feature or a third feature obtained by a second operation; perform the first operation n times and the second operation n-1 times to obtain the third feature according to the second feature, and repeat the first operation and the second operation n times; perform a third operation on the second features obtained by the first operation to obtain a plurality of groups of speech tokens; the first operation comprises: performing segmentation processing on the coded features or the third features to obtain first segmented features; performing vector quantization on all the first segmented features by VQ quantization to obtain a plurality of first quantized features, and splicing all the first quantized features to obtain the second feature; the second operation comprises: subtracting the first feature from the second feature to obtain the third feature; the third operation comprises: adding each second feature obtained by the first operation to obtain a plurality of groups of speech tokens.

2. The speech token extraction method of claim 1, wherein, The step of pre-processing the audio data comprises: performing frequency domain processing on the audio data by a Fourier transform algorithm to obtain the frequency domain features; the audio coding layer is further configured to: performing Mel spectrum conversion on the frequency domain features to obtain the coded features.

3. The method of claim 1, wherein, The audio coding layer is composed of a first convolution unit, a Resnet network unit and a second convolution unit; the Resnet network unit comprises a plurality of residual blocks; and the first convolution unit is connected to each residual block in the Resnet network unit. In the Resnet network unit, each residual block comprises a fifth convolution unit, a first activation function unit, a sixth convolution unit and a second activation function unit connected in sequence; and an input port of the fifth convolution unit is connected to an output port of the second activation function unit.

4. The speech token extraction method of claim 3, wherein, In the Resnet network unit, each residual block further comprises a spatial convolution unit and a first feature fusion unit; and in the residual block, the fifth convolution unit, the spatial convolution unit, the first activation function unit, the first feature fusion unit, the sixth convolution unit and the second activation function unit are connected in a layer-by-layer structure in sequence. The fifth convolution unit is composed of a plurality of first convolution kernels; the first convolution kernel is a point convolution; and the number of the first convolution kernels is greater than the number of channels of the convolution kernels in the first convolution unit. The spatial convolution unit is composed of a plurality of second convolution kernels, and a size of the second convolution kernel is greater than that of the first convolution kernel; the spatial convolution unit is configured to divide the output of the fifth convolution unit into a plurality of subsequences, and perform convolution processing on each subsequence through a second convolution kernel respectively; The first feature fusion unit includes an attention layer, and the first feature fusion layer is configured to perform feature fusion processing on the output of each second convolution kernel based on calculation of attention weights; The sixth convolution unit is composed of a plurality of third convolution kernels, and the third convolution kernel is a point convolution; and the number of the third convolution kernels is the same as the number of channels of the convolution kernel in the first convolution unit.

5. The speech token extraction method of claim 4, wherein, The Resnet network unit further includes a downsampling unit and a second feature fusion unit which are independent of the residual block; The downsampling unit is configured to perform frequency domain downsampling processing on the output of the first convolution unit; The second feature fusion unit includes an attention layer, and the second feature fusion layer is configured to perform feature fusion processing on the output of the downsampling unit and the output of the Resnet network unit based on calculation of attention weights, and input the fused features to the second convolution unit.

6. The speech token extraction method of claim 3, wherein, The audio decoding layer includes a first block, a second block and a third block; the first block includes a deconvolution unit, a Resnet network unit and a third convolution unit; the second block includes a deconvolution unit, an MRF unit and a fourth convolution unit; and the third block includes a PQMF unit.

7. The speech token extraction method of claim 6, wherein, The extraction method includes:

8. A speech token extraction method characterized by, obtaining audio data, and performing preprocessing on the audio data to obtain frequency domain features; introducing the frequency domain features into a speech token extraction model to extract speech tokens, to obtain a plurality of groups of speech tokens; the speech token extraction model includes an audio encoding layer and a residual vector quantization layer; the audio encoding layer is configured to perform feature encoding on the frequency domain features to obtain encoded features; and the residual vector quantization layer is configured to perform segmented quantization processing on the encoded features; the segmented quantization processing includes: performing a fourth operation on a fourth feature to obtain a plurality of second segmented features; the fourth feature includes the encoded features or sixth features obtained through a fifth operation; and the fourth operation includes performing segmentation processing on the encoded features or the sixth features; performing a fifth operation on all the second segmented features to obtain sixth features; the fifth operation includes performing vector quantization on all the second segmented features by using VQ quantization to obtain a plurality of second quantized features, and splicing all the second quantized features to obtain the sixth features; and the audio decoding layer is configured to perform feature decoding on the speech tokens to obtain target audio data; and the target audio data is used to verify the integrity of the speech tokens, so as to train the speech token extraction model. The fourth operation and the fifth operation are repeated n times, and a sixth feature obtained by the fifth operation in the n-th time is the voice token.

9. A voice processing method, characterized by, The voice processing method applies the voice token extraction method of any one of claims 1 to 8 to feature extraction.

Images