Speech token extraction method and speech processing method

Abstract

The invention provides a voice token extraction method and a voice processing method, and relates to the technical field of voice token extraction, and the extraction method comprises the steps: obtaining audio data, carrying out the preprocessing of the audio data, and obtaining a frequency domain feature; importing the frequency domain features into a voice token extraction model for voice token extraction to obtain a plurality of groups of voice tokens; the speech token extraction model comprises an audio coding layer and a residual vector quantization layer; the audio coding layer is configured to perform feature coding according to the frequency domain features to obtain coding features; the residual vector quantization layer is configured to perform segmented residual quantization processing according to the coding features. According to the method, the problem of excessive voice token extraction groups extracted by an existing voice token extraction technology is solved.

Description

Technical Field

[0001] The present invention relates to the technical field of speech token extraction, and in particular to a speech token extraction method and a speech processing method. Background Art

[0002] In recent years, with the rapid development of artificial intelligence technology, large language models (LLMs) have made breakthrough progress in the field of natural language processing. LLMs, represented by ChatGPT, GPT-4 and some domestic advanced models, have pushed the application of natural language processing to a new level with their powerful language understanding and generation capabilities. These models can not only conduct complex conversational interactions, but also complete a variety of tasks such as text creation and code generation, greatly broadening the application scenarios of natural language processing technology.

[0003] However, despite the remarkable achievements of LLMs in the field of natural language processing, there are relatively few studies and applications in the audio field. As an important information carrier, the processing and analysis of audio is crucial for many application scenarios, such as speech recognition, audio classification, music generation, etc. In order to extend the powerful capabilities of LLMs to the audio field, a key problem needs to be solved: how to convert audio data into a format that LLMs can process. In natural language processing, LLMs usually receive and process discrete text tokens (i.e., basic units in text, such as words, characters, or subwords). Therefore, in order to apply LLMs to the audio field, we need to completely discretize the audio data and obtain a form similar to text tokens. In this process, the compression and encoding technology of audio data plays a vital role.

[0004] At present, the technologies for compressing audio into tokens mainly include the SoundStream technology proposed by Google and the encoder technology proposed by Meta. Both technologies use the residual quantization layer combined with the autoencoder technical framework to compress the audio signal into a limited set of tokens. Specifically, SoundStream and encoder technologies can compress audio signals into 12 or 8 groups of tokens, which represent the basic units or features in the audio signal. However, although these technologies have achieved certain results in audio compression, they still face some challenges. Since these technologies are modeled based on the audio time domain, many groups of tokens are required to complete the complete modeling of the audio signal. This means that in the subsequent audio modeling process, LLMs need to process a large number of tokens, which not only increases the computational complexity, but may also have a negative impact on the modeling effect. In addition, too many tokens may also cause the model to have overfitting or underfitting problems during training, further affecting the accuracy and efficiency of audio processing. Summary of the invention

[0005] The present application provides a voice token extraction method and a voice processing method to solve the problem of too many voice token extraction groups extracted by existing voice token extraction technologies.

[0006] In a first aspect, the present application provides a voice token extraction method, the extraction method comprising:

[0007] Acquire audio data, and preprocess the audio data to obtain frequency domain features;

[0008] The frequency domain features are imported into a speech token extraction model to extract speech tokens, and obtain a plurality of groups of speech tokens; the speech token extraction model includes an audio coding layer and a residual vector quantization layer; the audio coding layer is configured to perform feature encoding according to the frequency domain features to obtain coding features; the residual vector quantization layer is configured to perform segmented residual quantization processing according to the coding features.

[0009] Preferably, the step of preprocessing the audio data comprises:

[0010] The audio data is converted into frequency domain using a Fourier transform algorithm to obtain the frequency domain features.

[0011] Preferably, the audio coding layer is further configured as:

[0012] Mel spectrum conversion is performed according to the frequency domain features to obtain the coding features.

[0013] Preferably, the residual vector quantization layer is further configured as:

[0014] Perform a first operation according to the first feature to obtain a second feature, where the first feature includes the coding feature or a third feature obtained by the second operation;

[0015] Performing a second operation according to the second feature to obtain the third feature, and repeating the first operation n times and the second operation n-1 times;

[0016] Perform a third operation on the n second features that complete the first operation to obtain several groups of the voice tokens.

[0017] Preferably, the first operation includes:

[0018] Segmenting the coding feature or the third feature to obtain a first segmentation feature;

[0019] Performing vector quantization on all the first segmentation features by using VQ quantization to obtain a plurality of first quantized features, and concatenating all the first quantized features to obtain the second feature;

[0020] The second operation includes:

[0021] Subtracting the first feature from the second feature to obtain the third feature;

[0022] The third operation includes:

[0023] The second features of each of the first operations are added together to obtain several groups of the voice tokens.

[0024] Preferably, the audio coding layer is composed of a first convolution unit, a Resnet network unit and a second convolution unit; the Resnet network unit includes a plurality of residual blocks; the first convolution unit is respectively connected to each of the residual blocks in the Resnet network unit; in the Resnet network unit, each residual block includes a fifth convolution unit, a first excitation function unit, a sixth convolution unit and a second excitation function unit connected in sequence; the input port of the fifth convolution unit is connected to the output port of the second excitation function unit.

[0025] Preferably, in the Resnet network unit, each of the residual blocks further includes a spatial convolution unit and a first feature fusion unit; in the residual block, the fifth convolution unit, the spatial convolution unit, the first excitation function unit, the first feature fusion unit, the sixth convolution unit and the second excitation function unit are sequentially connected in a stacked structure;

[0026] The fifth convolution unit is composed of a plurality of first convolution kernels, the first convolution kernels are point convolutions, and the number of the first convolution kernels is greater than the number of channels of the convolution kernels in the first convolution unit;

[0027] The spatial convolution unit is composed of a plurality of second convolution kernels, and the size of the second convolution kernel is larger than the first convolution kernel; the spatial convolution unit is used to divide the output of the fifth convolution unit into a plurality of subsequences, and convolve each subsequence through a second convolution kernel respectively;

[0028] The first feature fusion unit includes an attention layer, and the first feature fusion layer is used to perform feature fusion processing on the output of each second convolution kernel based on the calculation of attention weight;

[0029] The sixth convolution unit is composed of multiple third convolution kernels, the third convolution kernels are point convolutions, and the number of the third convolution kernels is the same as the number of channels of the convolution kernels in the first convolution unit.

[0030] Preferably, the Resnet network unit further includes a downsampling unit and a second feature fusion unit independently provided from the residual block, wherein:

[0031] The down-sampling unit is used to perform frequency domain down-sampling processing on the output of the first convolution unit;

[0032] The second feature fusion unit includes an attention layer, which is used to perform feature fusion processing on the output of the downsampling unit and the output of the Resnet network unit based on the calculation of attention weights, and input the fused features into the second convolution unit.

[0033] Preferably, the speech token extraction model also includes an audio decoding layer; the audio decoding layer is applied to the model training process of the speech token extraction model; the audio decoding layer is configured to perform feature decoding according to the speech token to obtain target audio data; the target audio data is used to verify the integrity of the speech token, thereby training the speech token extraction model.

[0034] Preferably, the audio decoding layer includes a first block, a second block and a third block; the first block includes a deconvolution unit, a Resnet network unit and a third convolution unit; the second block includes a deconvolution unit, an MRF unit and a fourth convolution unit; the third block includes a PQMF unit.

[0035] The extraction method further comprises:

[0036] Acquire audio data, and preprocess the audio data to obtain frequency domain features;

[0037] The frequency domain features are introduced into a speech token extraction model to extract speech tokens, and obtain a plurality of groups of speech tokens; the speech token extraction model includes an audio coding layer and a residual vector quantization layer; the audio coding layer is configured to perform feature coding according to the frequency domain features to obtain coding features; the residual vector quantization layer is configured to perform segmentation quantization processing according to the coding features; the segmentation quantization processing includes:

[0038] Performing a fourth operation according to the fourth feature to obtain a plurality of second segmentation features; the fourth feature includes the coding feature or the sixth feature obtained by the fifth operation; the fourth operation includes performing segmentation processing on the coding feature or the sixth feature;

[0039] Performing a fifth operation on all the second segmentation features to obtain a sixth feature; the fifth operation includes performing vector quantization on all the second segmentation features using VQ quantization to obtain a plurality of second quantized features, and concatenating all the second quantized features to obtain the sixth feature;

[0040] The fourth operation and the fifth operation are repeated n times, and the sixth feature obtained by the nth operation of the fifth operation is the voice token.

[0041] The extraction method further comprises:

[0042] Acquire audio data, and preprocess the audio data to obtain frequency domain features;

[0043] The frequency domain features are introduced into a speech token extraction model to extract speech tokens, and obtain a plurality of groups of speech tokens; the speech token extraction model includes an audio coding layer and a residual vector quantization layer; the audio coding layer is configured to perform feature coding according to the frequency domain features to obtain coding features; the residual vector quantization layer is configured to perform residual quantization processing according to the coding features; the residual quantization processing includes:

[0044] Performing a sixth operation according to the seventh feature to obtain an eighth feature; the seventh feature includes a coding feature or a ninth feature obtained by the seventh operation; the sixth operation includes performing vector quantization on the seventh feature using VQ quantization to obtain a third quantized feature;

[0045] The seventh operation is performed according to the eighth feature to obtain the ninth feature; the seventh operation includes adding the eighth feature and the seventh feature to obtain the tenth feature;

[0046] The sixth operation and the seventh operation are repeated n times, and the tenth feature obtained by the nth operation of the seventh operation is the voice token.

[0047] The extraction method comprises:

[0048] Acquire audio data, and preprocess the audio data to obtain frequency domain features;

[0049] The frequency domain features are imported into a speech token extraction model for speech token extraction to obtain at least one group of speech tokens; the speech token extraction model includes an audio coding layer, a residual vector quantization layer and an audio decoding layer; the audio coding layer is configured to perform enhanced feature encoding according to the frequency domain features to obtain enhanced coding features; the residual vector quantization layer is configured to perform weakened segmentation residual quantization processing according to the enhanced coding features; the audio decoding layer is configured to perform enhanced feature decoding on the speech token to obtain target audio data, and the audio decoding layer is applied to the model training process of the speech token extraction model, and the target audio data is used to verify the integrity of the speech token, so as to train the speech token extraction model.

[0050] In a second aspect, the present application also provides a speech processing method, which is applied to the above-mentioned speech token extraction method for feature extraction.

[0051] As can be seen from the above content, the present application provides a speech token extraction method and a speech processing method, the extraction method includes obtaining audio data, preprocessing the audio data, and obtaining frequency domain features; importing the frequency domain features into a speech token extraction model to extract speech tokens and obtain a number of groups of speech tokens; the speech token extraction model includes an audio coding layer and a residual vector quantization layer; the audio coding layer is configured to perform feature encoding according to the frequency domain features to obtain coding features; the residual vector quantization layer is configured to perform segmented residual quantization processing according to the coding features. The present application solves the problem of too many speech token extraction groups extracted by existing speech token extraction technologies through the above method. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the technical solution of the present application, the drawings required for use in the embodiments are briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0053] Figure 1 A flowchart of a speech token extraction method for this application;

[0054] Figure 2 This is a flowchart of preprocessing in a speech token extraction method for this application;

[0055] Figure 3 This is a functional flow chart of the audio coding layer in a voice token extraction method for this application;

[0056] Figure 4 This is a functional flow chart of the residual vector quantization layer in a speech token extraction method for this application;

[0057] Figure 5 This is a flowchart of the first operation in a voice token extraction method of the present application;

[0058] Figure 6 This is a flowchart of the second operation in a voice token extraction method of the present application;

[0059] Figure 7 This is a flowchart of the third operation in a voice token extraction method of this application;

[0060] Figure 8 A schematic diagram of a speech token extraction model in a speech token extraction method of this application;

[0061] Fig. 9 This is a schematic diagram of the audio coding layer in a voice token extraction method of this application;

[0062] Fig.10 This is a schematic diagram of the audio decoding layer in a voice token extraction method of this application;

[0063] Fig.11 This is a schematic diagram of a residual block in a speech token extraction method of this application. DETAILED DESCRIPTION

[0064] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0065] It should be noted that the brief description of terms in this application is only for the convenience of understanding the embodiments described below, and is not intended to limit the embodiments of this application. Unless otherwise specified, these terms should be understood according to their common and usual meanings.

[0066] It should be noted that, in this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in this application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0067] Figure 1 This is a flowchart of a speech token extraction method for this application.

[0068] Figure 8 This is a schematic diagram of a speech token extraction model in a speech token extraction method of the present application.

[0069] See also Figure 1 and Figure 8 It can be seen that this embodiment provides a speech token extraction method, wherein the extraction method includes a complete scheme, a first scheme and a second scheme; wherein the difference between the complete scheme, the first scheme and the second scheme lies in the different steps of model processing.

[0070] The complete solution, the first solution and the second solution all include:

[0071] S100, acquiring audio data, and preprocessing the audio data to obtain frequency domain features;

[0072] S200, importing the frequency domain features into a speech token extraction model to extract speech tokens, and obtaining several groups of speech tokens.

[0073] Specifically, in this embodiment, the complete solution, the first solution and the second solution all need to first obtain audio data, perform preprocessing, and import the preprocessed data into the speech token extraction model for speech token extraction. The difference is that the speech token extraction model performs different speech token extraction methods in different solutions.

[0074] Among them, the complete solution, the first solution and the second solution all need to import the speech token extraction model, and the specific structure in the speech token extraction model remains unchanged. Therefore, using the same structure in the speech token extraction model for different processing is the difference between the complete solution, the first solution and the second solution.

[0075] Among them, the speech token extraction model includes an audio coding layer and a residual vector quantization layer. In the complete scheme, the first scheme and the second scheme, the audio coding layer is used to perform feature encoding according to the frequency domain features to obtain coding features; the difference lies in the residual vector quantization layer, and different processing is performed through the residual vector quantization layer to realize the complete scheme, the first scheme and the second scheme.

[0076] Among them, in the complete plan:

[0077] The residual vector quantization layer is configured to perform segmented residual quantization processing according to the coding feature.

[0078] Among them, in the first solution:

[0079] The residual vector quantization layer is configured to perform split quantization processing according to the coding feature; the split quantization processing includes:

[0080] Performing a fourth operation according to the fourth feature to obtain a plurality of second segmentation features; the fourth feature includes the coding feature or the sixth feature obtained by the fifth operation; the fourth operation includes performing segmentation processing on the coding feature or the sixth feature;

[0081] Performing a fifth operation on all the second segmentation features to obtain a sixth feature; the fifth operation includes performing vector quantization on all the second segmentation features using VQ quantization to obtain a plurality of second quantized features, and concatenating all the second quantized features to obtain the sixth feature;

[0082] The fourth operation and the fifth operation are repeated n times, and the sixth feature obtained by the nth operation of the fifth operation is the voice token.

[0083] Specifically, in the first solution, since residual calculation is not performed and the solution needs to meet the standard, it is necessary to make up for the residual calculation part through segmentation. It can be understood that the segmentation processing is used to replace the residual calculation part, and the solution effect of both segmentation processing and residual calculation is achieved.

[0084] It should be noted that since segmentation processing is used to replace residual calculation, the number of segmentation processing in the first solution needs to be adaptively increased.

[0085] Among them, since the residual calculation is abandoned in the first scheme, the first half of the specific implementation of the scheme is similar to the complete scheme, and both of them repeatedly segment and VQ quantize the features; the difference between the first scheme and the complete scheme is that the features after VQ quantization and splicing are directly segmented, and through repeated segmentation, VQ quantization and splicing, the same technical effect as the complete scheme is finally achieved, that is, the same number of voice tokens are extracted.

[0086] Among them, in the second scheme:

[0087] The residual vector quantization layer is configured to perform residual quantization processing according to the coding feature; the residual quantization processing includes:

[0088] Performing a sixth operation according to the seventh feature to obtain an eighth feature; the seventh feature includes a coding feature or a ninth feature obtained by the seventh operation; the sixth operation includes performing vector quantization on the seventh feature using VQ quantization to obtain a third quantized feature;

[0089] The seventh operation is performed according to the eighth feature to obtain the ninth feature; the seventh operation includes adding the eighth feature and the seventh feature to obtain the tenth feature;

[0090] The sixth operation and the seventh operation are repeated n times, and the tenth feature obtained by the nth operation of the seventh operation is the voice token.

[0091] Specifically, in the second solution, since segmentation processing is not performed and the solution needs to meet the standard, it is necessary to use residual calculation to make up for the segmentation processing part. It can be understood that residual calculation is used to replace the segmentation processing part and achieve the effect of a solution with both segmentation processing and residual calculation.

[0092] It should be noted that, since the segmentation process is replaced by residual calculation, the number of residual calculations in the second solution needs to be adaptively increased.

[0093] Among them, since the second solution abandons the segmentation processing, the second half of the specific implementation of the solution is similar to the complete solution, both of which repeatedly perform VQ quantization and residual calculation on the features; the difference between the second solution and the complete solution is that the preprocessed features are directly VQ quantized, and by repeating VQ quantization and residual calculation, the same technical effect as the complete solution is finally achieved, that is, extracting the same number of speech tokens.

[0094] It should be noted that VQ quantification is an essential step whether in the complete plan, the first plan or the second plan.

[0095] According to the above complete solution, the first solution and the second solution, the complete solution is an integration of the first solution and the second solution; it can be understood that in the first solution, only segmentation and VQ quantization are performed, but no residual calculation is performed; in the second solution, only VQ quantization and residual calculation are performed; and in the complete solution, segmentation, VQ quantization and residual calculation are performed completely.

[0096] It should be noted that the above complete solution, the first solution and the second solution can all realize the extraction of voice tokens, but there are certain differences in the components of voice token extraction. Therefore, according to different actual needs, the complete solution, the first solution or the second solution can be selected. In this embodiment, the complete solution is mainly introduced, so the following embodiments are all based on the complete solution as a technical basis.

[0097] The VQ quantization is to map points in the vector space to a limited number of representative points, which are also called codebook vectors or cluster centers. By quantizing the input data to these representative points, the representation of the model can be significantly simplified, thereby reducing the amount of calculation and storage requirements. At the same time, this quantization process helps to reduce overfitting and improve the generalization ability of the model.

[0098] Fig. 9 This is a schematic diagram of the audio coding layer in a voice token extraction method of this application.

[0099] See also Fig. 9 It can be seen that the audio coding layer is composed of a first convolution unit, a Resnet network unit and a second convolution unit; the Resnet network unit includes a plurality of residual blocks; the first convolution unit is respectively connected to each of the residual blocks in the Resnet network unit;

[0100] Fig.10 This is a schematic diagram of the audio decoding layer in a voice token extraction method of this application.

[0101] See also Fig.10 It can be seen that the audio decoding layer includes a first block, a second block and a third block; the first block includes a deconvolution unit, a Resnet network unit and a third convolution unit; the second block includes a deconvolution unit, an MRF unit and a fourth convolution unit; the third block includes a PQMF unit.

[0102] Among them, the full name of the PQMF unit is Perfect Reconstruction QuadratureMirror Filter Bank, which is an advanced signal processing technology that is widely used in many fields such as audio encoding and decoding, digital communication, image processing and multimedia compression.

[0103] PQMF technology has many advantages, such as high efficiency, flexibility, and easy implementation. However, in practical applications, PQMF also faces some challenges. For example, the design and optimization of filter banks require high-level mathematical and signal processing knowledge. At the same time, the implementation complexity of filter banks is also high, and advanced digital signal processing technology is needed to reduce the amount of calculation and storage requirements.

[0104] Furthermore, this embodiment also includes a third solution.

[0105] In the third scheme, the speech token extraction model includes an audio coding layer, a residual vector quantization layer, and an audio decoding layer; the third scheme has similarities and differences with the complete scheme, and the third scheme has similarities with the complete scheme in that:

[0106] Whether in the complete solution or the third solution, the trained speech token extraction model is used to extract speech tokens, and the speech token extraction model has the functions of encoding, segmentation, VQ quantization, residual calculation and decoding.

[0107] The third solution differs from the complete solution in that:

[0108] If the encoding parameter of the feature coding in the complete scheme is recorded as 1, the encoding parameter in the third scheme is 2 or 4. It can be seen that the feature coding strength in the third scheme is enhanced, so the feature coding in the third scheme is called enhanced feature coding; if the parameter of the segmented residual quantization processing in the complete scheme is recorded as 2, the parameter of the segmented residual quantization processing in the third scheme is 1, where the specific changes are that the number of segmentations and the number of residuals are reduced, which is equivalent to the reduction of the complete scheme. It can be seen that the segmented residual quantization processing in the third scheme is weakened, so the segmented residual quantization processing in the third scheme is called weakened segmented residual quantization processing.

[0109] The differences between the third plan and the complete plan also include:

[0110] The third scheme emphasizes the training of the speech token extraction model, while the complete scheme is more inclined to use the trained speech token extraction model. Therefore, the third scheme must emphasize the audio decoding layer required in the model training process; similar to the complete scheme, the audio decoding layer is used to converge and iterate the model, but the difference is that the feature encoding and the intensity of the split residual quantization processing in the third scheme have certain adjustments, and correspondingly, the feature decoding in the third scheme also needs adaptive adjustment to achieve the expected effect. If the parameter of the feature decoding in the complete scheme is recorded as 1, the feature decoding in the third scheme is 2 or 4. It can be seen that the feature decoding in the third scheme is enhanced, so the feature decoding in the third scheme is called enhanced feature decoding.

[0111] In the experiment of the third solution, the following conclusions were drawn:

[0112] When the number of segmentation and the number of residuals are both 1 (i.e., the number of segmentation and the number of residuals are reduced), by enlarging the scale of encoding and decoding parameters, for example, by increasing the encoding parameters to 2 or even 4 times of the original, the audio can be compressed to 1 group of tokens. Not only can the effect of 2-4 groups of voice tokens be achieved, but the number of tokens can also be further reduced, which greatly simplifies the training process of the model. The training process of the existing model can be directly used, making the model training more efficient and stable.

[0113] Fig.11 This is a schematic diagram of a residual block in a speech token extraction method of this application.

[0114] See also Fig.11 It can be seen that, in the Resnet network unit, each of the residual blocks includes a fifth convolution unit, a first excitation function unit, a sixth convolution unit and a second excitation function unit connected in sequence; the input port of the fifth convolution unit is connected to the output port of the second excitation function unit.

[0115] Further, in some embodiments, in the Resnet network unit, each of the residual blocks further includes a spatial convolution unit and a first feature fusion unit; in the residual block, the fifth convolution unit, the spatial convolution unit, the first excitation function unit, the first feature fusion unit, the sixth convolution unit and the second excitation function unit are sequentially connected in a stacked structure;

[0116] The fifth convolution unit is composed of a plurality of first convolution kernels, the first convolution kernels are point convolutions, and the number of the first convolution kernels is greater than the number of channels of the convolution kernels in the first convolution unit;

[0117] The spatial convolution unit is composed of a plurality of second convolution kernels, and the size of the second convolution kernel is larger than the first convolution kernel; the spatial convolution unit is used to divide the output of the fifth convolution unit into a plurality of subsequences, and convolve each subsequence through a second convolution kernel respectively;

[0118] The first feature fusion unit includes an attention layer, and the first feature fusion layer is used to perform feature fusion processing on the output of each second convolution kernel based on the calculation of attention weight;

[0119] The sixth convolution unit is composed of multiple third convolution kernels, the third convolution kernels are point convolutions, and the number of the third convolution kernels is the same as the number of channels of the convolution kernels in the first convolution unit.

[0120] Specifically, in this embodiment, features are compressed, segmented, enhanced and fused in each residual block in the Resnet network unit, thereby significantly enhancing the ability of the Resnet network to extract local features.

[0121] Among them, the fifth convolution unit is used to complete the compression processing of the features in the feature channel dimension; after the feature compression processing is completed, the spatial convolution unit is used to perform feature segmentation on the output result of the fifth convolution unit to obtain a plurality of subsequences, thereby completing the segmentation of the features; after the feature segmentation processing is completed, the first feature fusion unit is used to perform feature fusion on the subsequences based on the attention weight.

[0122] Furthermore, in some embodiments, the Resnet network unit further includes a downsampling unit and a second feature fusion unit independently provided from the residual block, wherein:

[0123] The down-sampling unit is used to perform frequency domain down-sampling processing on the output of the first convolution unit;

[0124] The second feature fusion unit includes an attention layer, which is used to perform feature fusion processing on the output of the downsampling unit and the output of the Resnet network unit based on the calculation of attention weights, and input the fused features into the second convolution unit.

[0125] Specifically, in this embodiment, based on the Resnet network, it is fused with the features of other stages in the audio encoding process to process global features at multiple scales and levels to enhance the grasp of details and global information.

[0126] Figure 2 This is a flowchart of the preprocessing in a speech token extraction method of this application.

[0127] See also Figure 2 It can be seen that, further, in some embodiments, the step of preprocessing the audio data includes:

[0128] S110, converting the audio data into frequency domain using a Fourier transform algorithm to obtain the frequency domain features.

[0129] Specifically, in this embodiment, the specific scheme for preprocessing the audio data is to use the Fourier transform algorithm (stft) (short time Fourier transform) to convert the audio data into the frequency domain, so as to obtain the frequency domain features.

[0130] It should be noted that, according to different requirements, the preprocessing of the audio data can be integrated into the speech token extraction model.

[0131] Figure 3 This is a functional flow chart of the audio coding layer in a voice token extraction method for this application.

[0132] See also Figure 3 It can be seen that, further, in some embodiments, the audio coding layer is also configured as follows:

[0133] S210, performing Mel spectrum conversion according to the frequency domain feature to obtain the coding feature.

[0134] Specifically, in this embodiment, in step S210, the frequency domain feature is converted into a Mel spectrum through the audio coding layer to obtain the coding feature.

[0135] Mel spectrum is a spectrum representation method based on the Mel scale. The Mel scale is a nonlinear frequency scale based on the human ear's sensory judgment of equally spaced pitch changes. Under this scale, the resolution of the low-frequency part is higher and can capture more details, while the resolution of the high-frequency part is lower, reducing unnecessary complexity. This feature enables the Mel spectrum to more effectively reflect the human ear's perception of audio signals.

[0136] The calculation of Mel spectrum usually includes the following steps:

[0137] Pre-emphasis: Pre-emphasize the original audio signal to enhance the high-frequency components and reduce the impact of low-frequency noise.

[0138] Frame Windowing: The audio signal is divided into multiple short-time frames, and each frame signal is processed through a windowing function to reduce spectrum leakage and improve the accuracy of analysis.

[0139] FFT transformation: Perform fast Fourier transform (FFT) on each frame of signal to convert the time domain signal into a frequency domain signal for further analysis.

[0140] Mel filter bank processing: The spectrum after FFT transformation is processed through a set of bandpass filters distributed on the Mel scale. These filters are usually triangular and evenly distributed on the Mel scale. The response of each filter represents the contribution of that frequency to the Mel spectrum.

[0141] Mel spectrum is widely used in speech recognition, music classification, voiceprint recognition and other fields. Its advantages are:

[0142] Conform to human hearing characteristics: Mel spectrum can well simulate the human ear's perception of frequency and has good characterization capabilities for audio signals.

[0143] Rich expression content: Through Mel filter bank processing, the Mel spectrum can capture the energy distribution of the audio signal at different frequencies and provide rich feature information.

[0144] Small coefficient correlation: Mel-frequency cepstral coefficients (MFCC) are features extracted from the Mel spectrum. They have small correlation and are easier to build statistical models such as Gaussian mixture models.

[0145] Figure 4 This is a functional flow chart of the residual vector quantization layer in a speech token extraction method of this application.

[0146] See also Figure 4 It can be seen that, further, in some embodiments, the residual vector quantization layer is further configured as:

[0147] S220, performing a first operation according to the first feature to obtain a second feature, where the first feature includes the coding feature or a third feature obtained by the second operation;

[0148] S230, performing a second operation according to the second feature to obtain the third feature, and repeating the first operation n times and the second operation n-1 times;

[0149] S240, performing a third operation on the n second features that complete the first operation to obtain several groups of the voice tokens.

[0150] Specifically, in this embodiment, it can be understood that the first operation and the second operation are performed several times, and the features obtained by completing the first operation each time are subjected to the third operation, so as to obtain the voice token, thereby completing the extraction of the voice token.

[0151] Among them, in step S200, the first operation can be the first start of the step or the nth start in the step cycle, so the feature of the first operation needs to be the implementation coding feature or the third feature obtained by the second operation.

[0152] Among them, in step S230, the second operation can be the first end of the step or the nth end in the step cycle. Regardless of whether it is the first step or the nth step, the features processed by step S230 are the features processed by step S220, thereby realizing the closed loop between step S220 and step S230.

[0153] Among them, in step S240, before performing the third operation, it is necessary to confirm in advance whether the loop between step S220 and step S230 meets the target requirements. Only when the target requirements are met, that is, the number of loops reaches a certain value, can the second feature of the first operation completed each time in the step loop be performed on the third operation, so as to realize the overall process and obtain the implementation voice token.

[0154] Figure 5 This is a flowchart of the first operation in a voice token extraction method of the present application.

[0155] See also Figure 5 It can be seen that, the first operation includes:

[0156] S221, performing segmentation processing on the coding feature or the third feature to obtain a first segmentation feature;

[0157] S222, performing vector quantization on all the first segmentation features by using VQ quantization to obtain a plurality of first quantized features;

[0158] S223: Concatenate all the first quantitative features to obtain the second feature.

[0159] Specifically, in this embodiment, the features are first segmented to obtain a number of segmented features, all the segmented features are VQ quantized, and the features after VQ quantization are spliced, thereby completing the first operation once.

[0160] Figure 6 This is a flowchart of the second operation in a voice token extraction method of the present application.

[0161] See also Figure 6 It can be seen that, the second operation includes:

[0162] The first feature is subtracted from the second feature to obtain the third feature.

[0163] Specifically, in this embodiment, the second operation can be completed by subtracting the input feature of the first operation from the feature obtained by completing the first operation.

[0164] Figure 7 This is a flowchart of the third operation in a voice token extraction method of the present application.

[0165] See also Figure 7 It can be seen that, the third operation includes:

[0166] The second features of each of the first operations are added together to obtain several groups of the voice tokens.

[0167] Specifically, in this embodiment, the features obtained after each completion of the first operation are added, and the third operation can be completed, so that the voice token can be extracted.

[0168] The second step and the third operation may be understood as residual calculation.

[0169] Furthermore, in some embodiments, the speech token extraction model also includes an audio decoding layer; the audio decoding layer is applied to the model training process of the speech token extraction model; the audio decoding layer is configured to perform feature decoding according to the speech token to obtain target audio data; the target audio data is used to verify the integrity of the speech token, thereby training the speech token extraction model.

[0170] Specifically, in this embodiment, the audio decoding layer is mainly used in the training process of the speech token extraction model. During the training process of the speech token extraction model, the audio is output through the audio decoding layer, and the audio is compared with the speech token to verify the performance of the model extraction, and the training is realized by continuous iteration.

[0171] For example, 1s 16k audio has 16000 float points, occupying 16000*4=64000 bits. After compression, the features are shortened by 320 times, and the shortened features are quantized to obtain four groups of tokens, that is, 16000 / 320=50 points are first obtained, and the features corresponding to each point are further compressed into 4 groups of tokens, each of which is of int type. Therefore, the tokens obtained after 1s audio compression occupy 50×4×2=400 bits, which shows that the audio is compressed 160 times as a whole.

[0172] Furthermore, this embodiment also provides a speech processing method, which applies the speech token extraction method in any one of the above embodiments to perform feature extraction.

[0173] Among them, speech processing methods can include speech recognition, sound cloning and all other speech-based feature extraction methods.

[0174] The advantages of this embodiment include:

[0175] By using the voice token extraction method in the embodiment, the audio can be compressed into several groups of small tokens (1-4 groups), which not only reduces the total number of tokens, but also eliminates obvious audio loss. During audio restoration, the token can be input into the decoding network to obtain the audio output; and the PDMF decoder is introduced in the decoding process, so that the speed of audio compression can be increased by about 4 times, and all modules can realize streaming operation.

[0176] The above-mentioned speech token extraction has a good application effect in the audio transmission process. Client A encodes and extracts the token through the encoder, and the final compressed feature is Z, which is then transmitted to the target client B through the network, and Z is decoded and used by the decoder on client B. Taking 1s audio as an example, after compression, a feature of about 0.4kb is obtained, which can be quickly transmitted through the network for decoding and restoration by client B. While significantly improving the transmission effect, it does not affect the audio effect.

[0177] For the convenience of explanation, the above description has been made in conjunction with specific embodiments. However, the above discussion in some embodiments is not intended to be exhaustive or limit the embodiments to the specific forms disclosed above. According to the above teachings, various modifications and variations can be obtained. The selection and description of the above embodiments are intended to better explain the content of this disclosure, so that those skilled in the art can better use the embodiments.

Claims

1. A speech token extraction method, characterized in that: The extraction method comprises: Acquire audio data, and preprocess the audio data to obtain frequency domain features; The frequency domain features are imported into a speech token extraction model to extract speech tokens, and obtain a plurality of groups of speech tokens; the speech token extraction model includes an audio coding layer and a residual vector quantization layer; the audio coding layer is configured to perform feature encoding according to the frequency domain features to obtain coding features; the residual vector quantization layer is configured to perform segmented residual quantization processing according to the coding features.

2. A speech token extraction method according to claim 1, characterized in that: The step of preprocessing the audio data comprises: Convert the audio data into frequency domain using Fourier transform algorithm to obtain the frequency domain features; The audio coding layer is further configured to: Mel spectrum conversion is performed according to the frequency domain features to obtain the coding features.

3. A speech token extraction method according to claim 1, characterized in that: The residual vector quantization layer is further configured as: Perform a first operation according to the first feature to obtain a second feature, where the first feature includes the coding feature or a third feature obtained by the second operation; Performing a second operation according to the second feature to obtain the third feature, and repeating the first operation n times and the second operation n-1 times; Perform a third operation on the n second features that complete the first operation to obtain several groups of the voice tokens.

4. A speech token extraction method according to claim 3, characterized in that: The first operation includes: Segmenting the coding feature or the third feature to obtain a first segmentation feature; Performing vector quantization on all the first segmentation features by using VQ quantization to obtain a plurality of first quantized features, and concatenating all the first quantized features to obtain the second feature; The second operation includes: Subtracting the first feature from the second feature to obtain the third feature; The third operation includes: The second features of each of the first operations are added together to obtain several groups of the voice tokens.

5. A speech token extraction method according to claim 1, characterized in that: The audio coding layer is composed of a first convolution unit, a Resnet network unit and a second convolution unit; the Resnet network unit includes a plurality of residual blocks; the first convolution unit is respectively connected to each of the residual blocks in the Resnet network unit; In the Resnet network unit, each of the residual blocks includes a fifth convolution unit, a first excitation function unit, a sixth convolution unit and a second excitation function unit connected in sequence; the input port of the fifth convolution unit is connected to the output port of the second excitation function unit.

6. A speech token extraction method according to claim 5, characterized in that: In the Resnet network unit, each of the residual blocks further includes a spatial convolution unit and a first feature fusion unit; in the residual block, the fifth convolution unit, the spatial convolution unit, the first excitation function unit, the first feature fusion unit, the sixth convolution unit and the second excitation function unit are sequentially connected in a stacked structure; The fifth convolution unit is composed of a plurality of first convolution kernels, the first convolution kernels are point convolutions, and the number of the first convolution kernels is greater than the number of channels of the convolution kernels in the first convolution unit; The spatial convolution unit is composed of a plurality of second convolution kernels, and the size of the second convolution kernel is larger than the first convolution kernel; the spatial convolution unit is used to divide the output of the fifth convolution unit into a plurality of subsequences, and convolve each subsequence through a second convolution kernel respectively; The first feature fusion unit includes an attention layer, and the first feature fusion layer is used to perform feature fusion processing on the output of each second convolution kernel based on the calculation of attention weight; The sixth convolution unit is composed of multiple third convolution kernels, the third convolution kernels are point convolutions, and the number of the third convolution kernels is the same as the number of channels of the convolution kernels in the first convolution unit.

7. A speech token extraction method according to claim 6, characterized in that: The Resnet network unit also includes a downsampling unit and a second feature fusion unit independently set from the residual block, wherein: The down-sampling unit is used to perform frequency domain down-sampling processing on the output of the first convolution unit; The second feature fusion unit includes an attention layer, which is used to perform feature fusion processing on the output of the downsampling unit and the output of the Resnet network unit based on the calculation of attention weights, and input the fused features into the second convolution unit.

8. A speech token extraction method according to claim 5, characterized in that: The speech token extraction model also includes an audio decoding layer; the audio decoding layer is applied to the model training process of the speech token extraction model; the audio decoding layer is configured to perform feature decoding according to the speech token to obtain target audio data; the target audio data is used to verify the integrity of the speech token, thereby training the speech token extraction model.

9. A speech token extraction method according to claim 8, characterized in that: The audio decoding layer includes a first block, a second block and a third block; the first block includes a deconvolution unit, a Resnet network unit and a third convolution unit; the second block includes a deconvolution unit, an MRF unit and a fourth convolution unit; the third block includes a PQMF unit.

10. A speech token extraction method, characterized in that: The extraction method comprises: Acquire audio data, and preprocess the audio data to obtain frequency domain features; The frequency domain features are introduced into a speech token extraction model to extract speech tokens, and obtain a plurality of groups of speech tokens; the speech token extraction model includes an audio coding layer and a residual vector quantization layer; the audio coding layer is configured to perform feature coding according to the frequency domain features to obtain coding features; the residual vector quantization layer is configured to perform segmentation quantization processing according to the coding features; the segmentation quantization processing includes: Performing a fourth operation according to the fourth feature to obtain a plurality of second segmentation features; the fourth feature includes the coding feature or the sixth feature obtained by the fifth operation; the fourth operation includes performing segmentation processing on the coding feature or the sixth feature; Performing a fifth operation on all the second segmentation features to obtain a sixth feature; the fifth operation includes performing vector quantization on all the second segmentation features using VQ quantization to obtain a plurality of second quantized features, and concatenating all the second quantized features to obtain the sixth feature; The fourth operation and the fifth operation are repeated n times, and the sixth feature obtained by the nth operation of the fifth operation is the voice token.

11. A speech token extraction method, characterized in that: The extraction method comprises: Acquire audio data, and preprocess the audio data to obtain frequency domain features; The frequency domain features are introduced into a speech token extraction model to extract speech tokens, and obtain a plurality of groups of speech tokens; the speech token extraction model includes an audio coding layer and a residual vector quantization layer; the audio coding layer is configured to perform feature coding according to the frequency domain features to obtain coding features; the residual vector quantization layer is configured to perform residual quantization processing according to the coding features; the residual quantization processing includes: Performing a sixth operation according to the seventh feature to obtain an eighth feature; the seventh feature includes a coding feature or a ninth feature obtained by the seventh operation; the sixth operation includes performing vector quantization on the seventh feature using VQ quantization to obtain a third quantized feature; The seventh operation is performed according to the eighth feature to obtain the ninth feature; the seventh operation includes adding the eighth feature and the seventh feature to obtain the tenth feature; The sixth operation and the seventh operation are repeated n times, and the tenth feature obtained by the nth operation of the seventh operation is the voice token.

12. A speech token extraction method, characterized in that: The extraction method comprises: Acquire audio data, and preprocess the audio data to obtain frequency domain features; The frequency domain features are imported into a speech token extraction model for speech token extraction to obtain at least one group of speech tokens; the speech token extraction model includes an audio coding layer, a residual vector quantization layer and an audio decoding layer; the audio coding layer is configured to perform enhanced feature encoding according to the frequency domain features to obtain enhanced coding features; the residual vector quantization layer is configured to perform weakened segmentation residual quantization processing according to the enhanced coding features; the audio decoding layer is configured to perform enhanced feature decoding on the speech token to obtain target audio data, and the audio decoding layer is applied to the model training process of the speech token extraction model, and the target audio data is used to verify the integrity of the speech token, so as to train the speech token extraction model.

13. A speech processing method, characterized in that: The speech processing method applies the speech token extraction method described in any one of claims 1 to 12 to perform feature extraction.

Images