Neural speech compression method and system

Abstract

The invention provides a neural speech compression method and system, and the method comprises the steps: obtaining original speech data, and carrying out the encoding of the original speech data through an encoder, and obtaining a hidden vector; inputting the hidden vector into a layered residual vector quantization module, and quantizing the hidden vector into a discrete code based on a multi-scale parallel token stream; inputting the discretized code into a decoder for decoding so as to reconstruct a language signal and obtain a target language signal; wherein the encoder is configured to introduce a cross-modal distillation technology to express high-level semantics extracted from a teacher network model when the original voice data is encoded; the decoder is configured as a neural decoder that performs adversarial training enhancement with the multi-scale discriminator. According to the method, the potential structure of the signal can be learned by utilizing deep learning, breakthrough progress is achieved in a low-bit-rate scene, the modeling capability and the compression quality are improved, and the method can be widely applied to scenes such as voice communication, audio storage and edge intelligent equipment.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of speech signal processing, and in particular to a neural speech compression method and system. BACKGROUND

[0002] Due to the demand for efficient storage and transmission of speech data, with the development of communication technology and artificial intelligence technology, the importance of speech compression is increasingly prominent. In the prior art, speech codecs are generally used to efficiently compress speech signals while maximizing perceptual quality. However, traditional codecs rely on rule-based vocoder pipelines, and compression performance is limited. Neural speech coding technology is a rapidly developing technology in recent years, which compresses and reconstructs speech signals through a deep learning model, and has shown great application potential in the field of low bit rate communication. Existing neural speech coding models usually use an encoder-decoder architecture, where the encoder is responsible for mapping the speech signal to a compact hidden representation, and the decoder reconstructs the speech signal based on the hidden representation.

[0003] However, in a real noisy environment, background noise, breathing sounds and other non-speech components in the input speech can seriously interfere with the encoding process, not only reducing the encoding efficiency, but also easily causing unpleasant artifacts in the reconstructed speech. Secondly, in the pursuit of extremely low bit rate scenarios, the model often loses high-level semantic information in the speech, such as natural fluctuations in tone, emotional color, and smooth pause rhythm, resulting in monotonous, mechanical and lack of expressive reconstructed speech, affecting the quality of communication. SUMMARY

[0004] To this end, the present application provides a neural speech compression method and system, aiming to solve the technical problems of the prior art in noise robustness and high-level semantic fidelity.

[0005] To achieve the above purpose, the present application adopts the following technical solutions:

[0006] According to the first aspect of the present application, the present application provides a neural speech compression method, which comprises:

[0007] obtaining original speech data, encoding the original speech data through an encoder to obtain a hidden vector;

[0008] inputting the hidden vector into a hierarchical residual vector quantization module, and quantizing the hidden vector into a discrete code based on a multi-scale parallel token stream;

[0009] inputting the discrete code into a decoder for decoding to reconstruct a language signal and obtain a target language signal;

[0010] The encoder is configured to introduce high-level semantic representation extracted in a teacher network model through a cross-modal distillation technology when encoding the original speech data.

[0011] The decoder is configured as a neural decoder enhanced by adversarial training with a multi-scale discriminator.

[0012] Further, before the original speech data is encoded by the encoder, the method further comprises:

[0013] The noise-aware gating module is configured to gate a convolutional network structure or a gated recurrent unit network structure.

[0014] The noise-aware gating module is configured to gate a convolutional network structure or a gated recurrent unit network structure.

[0015] Further, the quantization of the original speech data into a discrete encoding based on a multi-scale parallel token stream comprises:

[0016] The hierarchical residual vector quantization module utilizes a plurality of serial VQ layers to encode the speech frame, specifically comprising:

[0017] The continuous hidden vector generated by the encoder is input into the first VQ layer for quantization to generate the first layer quantization encoding.

[0018] The first layer quantization encoding is decoded to obtain the first layer quantization decoding, and the difference between the first layer quantization decoding and the continuous hidden vector is calculated to obtain the first layer residual.

[0019] The first layer residual is input into the next VQ layer for continuous quantization, and the last VQ layer is reached by analogy to form a multi-layer quantization encoding sequence.

[0020] The mathematical expression of the quantization loss of the hierarchical residual vector quantization module is:

[0021]

[0022] Wherein, represents the quantization loss; x represents the hidden vector; RVQ(x) represents the multi-layer quantization encoding sequence; q i represents the i-th layer quantization encoding; x i-1 represents the i-1 layer quantization decoding; ||·||2 represents the L2 norm; L represents the total number of VQ layers.

[0023] Further, the multi-scale discriminator comprises three sub-discriminators respectively acting on speech segments of different resolutions; the sub-discriminators utilize convolutional networks to extract local features.

[0024] The adversarial training enhancement with the multi-scale discriminator includes:

[0025] The multi-scale discriminator is alternately trained with the decoder, and an optimizer is updated respectively until a perceptual loss function is minimized.

[0026] Further, the perceptual loss function is configured as a weighted sum of a reconstruction loss and an adversarial loss; the reconstruction loss includes at least one of a time-domain waveform loss, a short-time Fourier transform-based frequency-domain loss, and a mel-spectrum-based frequency-domain loss.

[0027] The mathematical expression of the time-domain waveform loss is:

[0028]

[0029] Wherein, represents the time-domain waveform loss; x represents an original waveform; represents a reconstructed waveform; represents the L1 distance between the reconstructed waveform and the original waveform; represents the L2 distance between the reconstructed waveform and the original waveform; λ t1 , λ t2 respectively represent the hyperparameters of the L1 norm and the L2 norm, used to control the weight occupied;

[0030] and / or,

[0031] The mathematical expression of the short-time Fourier transform-based frequency-domain loss is:

[0032]

[0033] Wherein, represents the short-time Fourier transform-based frequency-domain loss; S i represents the log-compressed power spectrum calculated using a window length of 2 i and a shift length of ; and α represents a hyperparameter;

[0034] and / or,

[0035] The mathematical expression of the mel-spectrum-based frequency-domain loss is:

[0036]

[0037] Wherein, represents the mel-spectrum-based frequency-domain loss; M i represents the mel-spectral feature calculated using a window length of 2 i and a shift length of ; and α represents a hyperparameter;

[0038] and / or,

[0039] The mathematical expression of the adversarial loss is:

[0040]

[0041] wherein, denotes the adversarial loss; denotes the output of the discriminator k at time step l.

[0042] Further, the teacher network module comprises at least one of a pre-trained ASR automatic speech recognition model and a TTS text-to-speech model;

[0043] The high-level semantic representation extracted from the teacher network model is introduced when encoding the original language data through the cross-modal distillation technology, comprising:

[0044] The encoder-the hierarchical residual vector quantization module-the decoder are used as a student model, and the high-level semantic representation extracted from the teacher network module is input to the student model through the cross-modal distillation technology, so that the student model is simultaneously trained to optimize the speech reconstruction quality and the consistency with the semantic structure extracted from the teacher network module.

[0045] Further, the distillation training process of the student model comprises:

[0046] For a speech segment, a first intermediate output representation is obtained through the ASR automatic speech recognition model; for text data corresponding to the speech segment, a second intermediate output representation is obtained through the TTS text-to-speech model;

[0047] The output representation of the encoder is aligned with the first intermediate output representation and the second intermediate output representation to realize inter-modal mapping, and the student model is trained using a distillation loss function;

[0048] The mathematical expression of the distillation loss function is:

[0049]

[0050] wherein, denotes the distillation loss function; x denotes a speech segment; x asr denotes the first intermediate output representation; x tts denotes the second intermediate output representation; λ d1 ||x-x asr ||2 denotes the L2 distance between the speech segment and the first intermediate output representation; λ d2 ||x-x tts ||2 denotes the L2 distance between the speech segment and the second intermediate output representation; λd1 , λ d2 respectively represent hyperparameters of L2 norm, used to control the weight occupied.

[0051] Further, the method further comprises:

[0052] introducing a code rate adaptive control module to dynamically adjust the number of quantization layers and / or token reservation strategy according to the content complexity level of the language segment;

[0053] The code rate adaptive control module comprises a complexity estimator, a pre-trained network decision maker and a quantization scheduler; the complexity estimator is configured to calculate the content complexity level according to the silence ratio, energy spectrum distribution, pronunciation rate and / or entropy value of the current speech frame; the network decision maker is configured to generate a control signal of the corresponding quantization strategy based on the light-weight Transformer structure according to the content complexity level; and the quantization scheduler is configured to control the number of RVQ layers or the number of tokens reserved in each layer based on the control signal.

[0054] According to the second aspect of the present application, the present application provides a neural speech compression system, which is applied to the neural speech compression method of any one of the first aspect of the present application, and the system comprises:

[0055] an encoder configured to encode the original speech data to obtain a hidden vector;

[0056] a hierarchical residual vector quantization module configured to quantize the hidden vector into a discretized code based on a multi-scale parallel token stream;

[0057] a decoder configured to decode the discretized code to reconstruct a language signal to obtain a target language signal;

[0058] The encoder is configured to introduce high-level semantic representation extracted from a teacher network model through a cross-modal distillation technique when encoding the original language data.

[0059] The decoder is configured as a neural decoder enhanced by adversarial training with a multi-scale discriminator.

[0060] Further, the system further comprises:

[0061] a noise-aware gating module configured to perform noise masking processing on the original language data before the encoder encodes the original speech data;

[0062] The noise-aware gating module is configured as a gated convolutional network structure or a gated recurrent unit network structure.

[0063] and / or,

[0064] The code rate self-adaptive control module is used for dynamically adjusting the number of quantization layers and / or the token reservation strategy according to the content complexity level of the language segment.

[0065] The code rate self-adaptive control module comprises a complexity estimator, a pre-trained network decision maker and a quantization scheduler; the complexity estimator is used for calculating the content complexity level according to the silence ratio, energy spectrum distribution, pronunciation rate and / or entropy value of the current speech frame; the network decision maker is used for generating a control signal of the corresponding quantization strategy according to the content complexity level based on a lightweight Transformer structure; and the quantization scheduler is used for controlling the number of RVQ layers or the number of tokens reserved in each layer based on the control signal.

[0066] The present application has at least the following beneficial effects:

[0067] Through the technical scheme, a neural speech compression method with high compression ratio, high fidelity and low delay is provided, multiple innovative functional modules are fused, the potential structure of a learnable signal is utilized by deep learning, breakthrough progress is achieved in a low bit rate scene, and the modeling capability and compression quality are further improved, so that the method can be widely applied to speech communication, audio storage, edge intelligent device and other scenes.

[0068] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present application. BRIEF DESCRIPTION OF DRAWINGS

[0069] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0070] Figure 1 A flowchart of a neural speech compression method provided by an embodiment of the present application is shown;

[0071] Figure 2 A structural schematic diagram of a neural speech compression system provided by an embodiment of the present application is shown;

[0072] Figure 3 A structural schematic diagram of a neural speech compression system provided by another embodiment of the present application is shown. DETAILED DESCRIPTION

[0073] Exemplary embodiments of the present disclosure will be described in greater detail below with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it is understood that the present disclosure can be embodied in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the present disclosure to those skilled in the art.

[0074] It should be noted that, in this document, relational terms such as first and second and the like can be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises", "comprising", or any other variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element preceded by "comprises... a" does not, without more constraints, foreclose the existence of additional identical elements in the process, method, article, or apparatus that comprises the listed element.

[0075] The embodiment of the present application provides a neural speech compression method, which can at least include the following steps S101-S103 as shown in the figure: Figure 1 The embodiment of the present application provides a neural speech compression method, which can at least include the following steps S101-S103 as shown in the figure:

[0076] Step S101, obtaining original speech data, encoding the original speech data through an encoder to obtain a hidden vector.

[0077] The neural speech compression method provided by the embodiment of the present application is based on a neural speech compression system at least comprising an encoder, a hierarchical residual vector quantization module and a decoder connected in sequence. The encoder is configured to introduce high-level semantic representation extracted from a teacher network model through a cross-modal distillation technology when encoding the original speech data.

[0078] The cross-modal distillation (Cross-modal Knowledge Distillation) technology refers to migrating semantic or structural knowledge learned by other modal models to the neural speech compression system to improve the perceptual quality and semantic consistency of the compressed speech. The embodiment of the present application uses high-level representation layers extracted by a pre-trained teacher network module, such as phoneme embedding, attention context, semantic state, etc., as a knowledge source. By distilling training the neural speech compression system as a student model, the modeling ability of the neural speech compression system for high-order language information is improved.

[0079] In an optional embodiment, the teacher network module can be a pre-trained ASR automatic speech recognition model and / or a TTS text-to-speech model. The ASR automatic speech recognition model is, for example, Wav2Vec 2.0, Conformer, etc., which is used to extract phoneme-level semantic embedding and speech content structure; the TTS text-to-speech model is, for example, FastSpeech2, Tacotron2, which is used to extract text-aligned attention context and acoustic embedding. The neural speech compression system composed of an encoder-layer residual vector quantization module-decoder in the embodiment of the present application is used as a student model, and the high-level semantic representation extracted by the teacher network module is input to the student model through the cross-modal distillation technology, so that the student model is trained and optimized to simultaneously improve the speech reconstruction quality and the consistency with the semantic structure extracted by the teacher network module. In actual operation, the high-level representation (such as phoneme embedding, semantic embedding, and acoustic feature) can be transmitted to the “student network” of the neural speech compression system. The connection mechanism between the models is to insert a distillation interface (referred to as an Alignment Module) at the output layer of the encoder Encoder, align the output of some layers of the student model to the representation dimension of the teacher network model, and use a projection layer for inter-modal mapping, so as to realize inter-modal mapping. The distillation training process of the student model in the embodiment of the present application can be as follows: obtaining a first intermediate output representation by using an ASR automatic speech recognition model for a speech segment; obtaining a second intermediate output representation by using a TTS text-to-speech model for text data corresponding to the speech segment; aligning the output representation of the encoder with the first intermediate output representation and the second intermediate output representation to realize inter-modal mapping, and training the student model by using a distillation loss function; and the mathematical expression of the distillation loss function is:

[0080]

[0081] wherein, represents the distillation loss function; x represents a speech segment; x asr represents the first intermediate output representation; x tts represents the second intermediate output representation; λ d1 ||x-x asr ||2represents the L2 distance between the speech segment and the first intermediate output representation; λ d2 ||x-x tts ||2represents the L2 distance between the speech segment and the second intermediate output representation; λ d1 , λ d2 respectively represent the hyperparameters of the L2 norm, which are used to control the weight.

[0082] In practical applications, the following training strategy can be adopted: freezing the teacher network in the initial stage, training the student model; introducing an alternating freezing mechanism in the middle stage, fine-tuning the key layers in the student Encoder.

[0083] In step S102, the hidden vector is input into the hierarchical residual vector quantization module to quantize the hidden vector into a discrete coding based on a multi-scale parallel token stream.

[0084] The encoder output is quantized into a multi-scale token stream by a hierarchical residual vector quantization (RVQ) module, each layer capturing speech information at different frequencies or time resolutions, effectively improving compression efficiency and modeling capability. The parallel token stream design in the embodiment of the present application takes into account decoding speed and encoding fidelity, and adapts to the hardware parallel computing characteristics.

[0085] The hierarchical residual vector quantization module is composed of multiple VQ layers connected in series, each layer encoding the reconstruction residual of the previous layer. Specifically, the continuous hidden vector x generated by the encoder is input into the first VQ layer for quantization to generate the first layer quantization coding token q1; the first layer quantization coding q1 is decoded to obtain the first layer quantization decoding x1, and the difference between the continuous hidden vector x and the first layer quantization decoding x1 is calculated to obtain the first layer residual r1 = x - x1; the first layer residual r1 is input into the next VQ layer for continuous quantization, and so on until the last VQ layer, forming a multi-layer quantization coding sequence {q1, q2, q3, …, q l}. The quantization loss of the hierarchical residual vector quantization module takes into account the quantization error of the entire hierarchical residual vector quantization module and the sub-quantizers, and the mathematical expression can be:

[0086]

[0087] wherein, represents the quantization loss; x represents the hidden vector; PVQ(x) represents the multi-layer quantization coding sequence; q i represents the i-th layer quantization coding; x i-1 represents the i-1 layer quantization decoding; ||·||2 represents the L2 norm; L represents the total number of VQ layers.

[0088] The embodiment of the present application sets a hierarchical residual vector quantization module between the encoder and the decoder, and the multi-layer RVQ can approximate the original signal layer by layer to improve the compression expression capability; each layer of token has an interpretable frequency band or semantic hierarchical perception ability; and supports step-by-step precision improvement at low bit rate.

[0089] In step S103, the discrete coding is input into the decoder for decoding to reconstruct the language signal and obtain the target language signal.

[0090] In the embodiments of the present application, the decoder is configured as a neural decoder enhanced by adversarial training with the multi-scale discriminator. That is, the multi-scale discriminator and the perceptual loss (such as STFT Loss, Mel Loss) are introduced in the training process, so that the decoder can generate a reconstructed signal closer to the real speech, especially in the restoration of timbre and prosody.

[0091] In an optional embodiment, the multi-scale discriminator includes three sub-discriminators respectively acting on speech segments of different resolutions (such as full frame, frame level, and sub-frame level); the sub-discriminators use convolutional networks to extract local features to enhance the discrimination ability of sound quality, prosody, and high-frequency details. In actual application, the decoder and the multi-scale discriminator are enhanced by adversarial training, including: the multi-scale discriminator and the decoder are alternately trained, the optimizer is updated respectively, the decoder tries to "deceive" the multi-scale discriminator, and the perceptual loss function is minimized until the perceptual quality is improved. Specifically, the perceptual loss function is configured as the weighted sum of the reconstruction loss and the adversarial loss; the reconstruction loss includes the time-domain waveform loss, the frequency-domain loss based on the short-time Fourier transform, and / or the frequency-domain loss based on the mel spectrum.

[0092] The time-domain waveform loss directly constrains the error between the reconstructed waveform and the original waveform, and the mathematical expression is:

[0093]

[0094] wherein, represents the time-domain waveform loss; x represents the original waveform; represents the reconstructed waveform; represents the L1 distance between the reconstructed waveform and the original waveform; represents the L2 distance between the reconstructed waveform and the original waveform; λ t1 , λ t2 respectively represent the hyperparameters of the L1 norm and the L2 norm, used to control the weight occupied.

[0095] The frequency-domain loss based on the short-time Fourier transform is used to compare the short-time Fourier transform results of the input and the reconstructed signal, and the mathematical expression is:

[0096]

[0097] wherein, represents the frequency-domain loss based on the short-time Fourier transform; S i represents the log-compressed power spectrum calculated using the window length of 2 i and the shift length of ; and α represents the hyperparameter.

[0098] The frequency domain loss based on the Mel-spectrum is used to constrain the spectral similarity on the Mel-spectrum domain, and the mathematical expression is:

[0099]

[0100] wherein, represents the frequency domain loss based on the Mel-spectrum; M i represents the Mel-spectrum feature calculated using a window length of 2 i and a shift length of ; and a represents a hyperparameter.

[0101] The adversarial loss is used to improve the naturalness and diversity of the synthesized speech, and the mathematical expression is:

[0102]

[0103] wherein, represents the adversarial loss; represents the output of the discriminator k at the time step l.

[0104] In an optional embodiment, before the original speech data is encoded by the encoder, the method can further comprise: performing noise shielding processing on the original language data by using a noise perception gating module. The noise perception gating module is configured as a gating convolutional network structure or a gating recurrent unit network structure. That is, a noise perception gating module based on a gating neural network (such as a Gated CNN or a GRU) is added in front of the encoder, which adaptively filters out redundant components such as background noise and breathing sound. Compared with traditional VAD or pre-processing filters, this mechanism is integrated in the neural network backbone, which can realize end-to-end optimization and preserve semantic integrity.

[0105] Taking the front-end structure as a gating convolutional network structure (Gated CNN) as an example, the structure of the noise perception gating module is described as follows given an input feature X:

[0106] The general convolutional channel is represented as: A = Conv1D(X; W A )

[0107] The gating convolutional channel (Sigmoid gating) is represented as: B = σ(Conv1D(X; W B ))

[0108] The gating fusion and residual connection are represented as: Y = X + A ⊙ B

[0109] Wherein, σ is the Sigmoid function; and ⊙ is the point-by-point multiplication (gating), which can be regarded as “information passage x selective switch”.

[0110] Based on the noise perception gating module, the input original speech waveform can output the weight mask for suppressing the non-structural noise (such as breathing, keyboard sound) in the speech, and finally output the adaptive filtered speech waveform. The embodiment of the application sets the noise perception gating module at the original speech input end before the encoder Encoder, adaptively shields the noise part in the input through the gating mechanism, and sends the clean speech area to the subsequent encoder; the speech component modeling of the original input is strengthened, and the redundant background sound information is removed; the trainable parameter end-to-end fine-tuning is combined with the encoder to optimize and improve the robustness under the condition of low signal-to-noise ratio.

[0111] In an optional embodiment, the application can also introduce a code rate adaptive control module to dynamically adjust the number of quantization layers and / or token reservation strategies according to the content complexity level of the speech segment.

[0112] Among them, the code rate adaptive control module includes a complexity estimator, a pre-trained network decision maker and a quantization scheduler; the complexity estimator is used to calculate the content complexity level according to the silence ratio, energy spectrum distribution, pronunciation rate and / or entropy value of the current speech frame; the network decision maker is used to generate a control signal of the corresponding quantization strategy based on the light weight Transformer structure according to the content complexity level; the quantization scheduler is used to control the number of RVQ layers or the number of tokens reserved in each layer (such as Top-k reservation) based on the control signal. In the training process, a pseudo label "speech complexity level" can be added to supervise the scheduler to output different quantization strategies; during inference, the compression depth is dynamically determined according to the real-time input to realize the adaptive adjustment of the code rate, that is, "on-demand compression". In practical application, both fixed code rate and variable code rate working modes can be supported to improve the communication stability and energy saving ability.

[0113] The embodiment of the application performs end-to-end optimization through a carefully designed combined loss function. The total loss function is the weighted sum of multiple sub-loss terms, which can be expressed as:

[0114]

[0115] Among them, represents the reconstruction loss; represents the adversarial loss; represents the quantization loss; represents the distillation loss; λ rec , λ gan , λ rvq , λ distill respectively represent the corresponding weight hyperparameters, which are used to balance the importance of different optimization objectives.

[0116] The embodiment of the present application provides a neural speech compression method, comprising: obtaining original speech data, encoding the original speech data through an encoder to obtain a hidden vector; inputting the hidden vector into a hierarchical residual vector quantization module to quantize the hidden vector into a discretization code based on a multi-scale parallel token stream; inputting the discretization code into a decoder for decoding to reconstruct a language signal to obtain a target language signal; wherein the encoder is configured to introduce high-level semantic representation extracted in a teacher network model through a cross-modal distillation technology when encoding the original speech data; and the decoder is configured as a neural decoder enhanced by adversarial training with a multi-scale discriminator. Through the present application, at least the following beneficial effects are achieved:

[0117] 1) At a bit rate as low as 0.5 kbps, clear, natural and distortion-free speech reconstruction can still be maintained;

[0118] 2) Multi-layer RVQ compression and token parallelization design significantly improve throughput and real-time processing capability;

[0119] 3) Adversarial training and semantic distillation mechanism are superior to existing mainstream neural encoders and decoders in subjective listening evaluation (such as MOS);

[0120] 4) Supports multiple code rate strategies to adapt to different transmission bandwidths and storage constraints;

[0121] 5) Modular architecture facilitates integration with TTS, ASR, speech synthesis and other systems, and has high scalability.

[0122] Further, as a specific implementation of Figure 1 , the embodiment of the present application provides a neural speech compression system, as shown in Figure 2 , which can comprise an encoder 210, a hierarchical residual vector quantization module 220, and a decoder 230.

[0123] The encoder 210 can be used to encode the original speech data to obtain a hidden vector;

[0124] The hierarchical residual vector quantization module 220 can be used to quantize the hidden vector into a discretization code based on a multi-scale parallel token stream;

[0125] The decoder 230 can be used to decode the discretization code to reconstruct a language signal to obtain a target language signal;

[0126] Wherein, the encoder 210 is configured to introduce high-level semantic representation extracted in a teacher network model 240 through a cross-modal distillation technology when encoding the original language data;

[0127] The decoder 230 is configured as a neural decoder enhanced by adversarial training with the multi-scale discriminator 250.

[0128] Optionally, as shown in Figure 3 Another embodiment of the present application provides a neural speech compression system, which further comprises a noise-aware gating module 260 and a rate-adaptive control module 270 (not shown in the figure).

[0129] The noise-aware gating module 260 can be used to perform noise masking processing on the original speech data before the encoder 210 encodes the original speech data.

[0130] The noise-aware gating module 260 is configured as a gating convolutional network structure or a gating recurrent unit network structure.

[0131] The rate-adaptive control module 270 can be used to dynamically adjust the number of quantization layers and / or the token reservation strategy according to the content complexity level of the speech segment.

[0132] The rate-adaptive control module 270 comprises a complexity estimator, a pre-trained network decision maker and a quantization scheduler. The complexity estimator is used to calculate the content complexity level according to the silence ratio, energy spectrum distribution, pronunciation rate and / or entropy value of the current speech frame. The network decision maker is used to generate a control signal of the corresponding quantization strategy based on the light-weight Transformer structure according to the content complexity level. The quantization scheduler is used to control the number of RVQ layers or the number of tokens reserved in each layer based on the control signal.

[0133] It should be noted that other corresponding descriptions of the various functional modules involved in the neural speech compression system provided by the embodiments of the present application can refer to the corresponding descriptions of the method provided by the embodiments of the present application shown in Figure 1 The corresponding descriptions of the method provided by the embodiments of the present application are not repeated here.

[0134] Those skilled in the art can clearly understand the specific working process of the system, device, module and unit described above, which can refer to the corresponding process in the foregoing method embodiments. For the sake of brevity, it is not repeated here.

[0135] In addition, each functional unit in each embodiment of the present application can be physically independent of each other, or two or more functional units can be integrated together, and all functional units can be integrated in one processing unit. The integrated functional units can be realized in the form of hardware, software or firmware.

[0136] Those skilled in the art can understand that the integrated functional units, if realized in the form of software and sold or used as independent products, can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application can be embodied in the form of software product essentially or in the form of all or part of the technical solutions. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computing device (such as a personal computer, a server, or a network device) to execute all or part of the steps of the method described in the embodiments of the present application when the instructions are executed. The aforementioned storage medium includes a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various program code storage media.

[0137] Alternatively, all or part of the steps of the foregoing method embodiments can be completed by program instruction related hardware (such as a computing device of a personal computer, a server, or a network device). The program instruction can be stored in a computer readable storage medium, and when the program instruction is executed by a processor of the computing device, the computing device executes all or part of the steps of the method described in the embodiments of the present application.

[0138] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them. Although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that, within the spirit and principles of the present application, the technical solutions described in the foregoing embodiments can still be modified, or some or all of the technical features can be replaced by equivalents. Such modifications or replacements do not cause the corresponding technical solutions to deviate from the protection scope of the present application.

Claims

1. A neural speech compression method, characterized by, The method comprises: obtaining original speech data, encoding the original speech data through an encoder to obtain a hidden vector; inputting the hidden vector into a hierarchical residual vector quantization module, and quantizing the hidden vector into a discretization code based on a multi-scale parallel token stream; inputting the discretization code into a decoder for decoding to reconstruct a language signal to obtain a target language signal; wherein the encoder is configured to introduce high-level semantic representations extracted from a teacher network model through a cross-modal distillation technique when encoding the original speech data; the decoder is configured as a neural decoder enhanced by adversarial training with a multi-scale discriminator.

2. The method of claim 1, wherein, Before the original speech data is encoded by the encoder, the method further comprises: using a noise perception gating module to perform noise shielding processing on the original language data; wherein the noise perception gating module is configured as a gating convolutional network structure or a gating recurrent unit network structure.

3. The method of claim 1, wherein, The quantization of the original speech data into a discretization code based on a multi-scale parallel token stream comprises: The hierarchical residual vector quantization module encodes the speech frame using multiple VQ layers in series, specifically comprising: inputting the continuous hidden vector generated by the encoder into the first VQ layer for quantization to generate the first layer quantization code; decoding the first layer quantization code to obtain the first layer quantization decoding, calculating the difference between the first layer quantization decoding and the continuous hidden vector to obtain the first layer residual; inputting the first layer residual into the next VQ layer for continuous quantization, and similarly to the last VQ layer to form a multi-layer quantization code sequence; The mathematical expression of the quantization loss of the hierarchical residual vector quantization module is: wherein, denotes quantization loss; x denotes a hidden vector; RVQ(x) denotes a multi-layer quantization encoded sequence; q i denotes i-th layer quantization encoding; x i-1 denotes i-1-th layer quantization decoding; ||·||2denotes L2norm; L denotes total number of VQ layers.

4. The method of claim 1, wherein, The multi-scale discriminator includes three sub-discriminators acting on different resolution speech segments; the sub-discriminators use convolutional networks to extract local features; The adversarial training enhancement with the multi-scale discriminator comprises: The multi-scale discriminator and the decoder are alternately trained, and the optimizer is updated respectively until the perceptual loss function is minimized.

5. The method of claim 4, wherein, The perceptual loss function is configured as a weighted sum of reconstruction loss and adversarial loss; the reconstruction loss includes at least one of time domain waveform loss, frequency domain loss based on short-time Fourier transform, and frequency domain loss based on mel spectrum; The mathematical expression of the time domain waveform loss is: wherein, denotes the time-domain waveform loss; x denotes the original waveform; denotes the reconstructed waveform; denotes the L1 distance between the reconstructed waveform and the original waveform; denotes the L2 distance between the reconstructed waveform and the original waveform; λ t1 , λ t2 denote the hyperparameters of the L1 norm and the L2 norm, respectively, for controlling the occupied weight; and / or, The mathematical expression of the frequency domain loss based on short-time Fourier transform is: wherein, denotes a frequency-domain loss based on a short-time Fourier transform; S i denotes a log-compressed power spectrum calculated using a window length of 2 i and a shift length of denotes a log-compressed power spectrum calculated using a window length of 2 α denotes a hyperparameter; and / or, The mathematical expression of the frequency domain loss based on mel spectrum is: wherein, represents a frequency domain loss based on a mel-spectrum; M i represents a mel-spectral feature calculated using a window length of 2 i and a shift length of represents a mel-spectral feature calculated using a window length of 2 α represents a hyperparameter; and / or, The mathematical expression of the adversarial loss is: wherein, represents the adversarial loss; represents the output of the discriminator k at time step l.

6. The method of claim 1, wherein, The teacher network module includes at least one of a pre-trained ASR automatic speech recognition model and a TTS text-to-speech model; The introduction of high-level semantic representations extracted from the teacher network model through the cross-modal distillation technique when encoding the original language data comprises: The encoder, the hierarchical residual vector quantization module, and the decoder are taken as a student model, and the high-level semantic representation extracted by the teacher network module is input to the student model through the cross-modal distillation technology, so that the student model is simultaneously trained to optimize the voice reconstruction quality and the consistency with the semantic structure extracted by the teacher network module.

7. The method of claim 6, wherein, The distillation training process of the student model includes: For a voice segment, a first intermediate output representation is obtained through the ASR automatic speech recognition model; for text data corresponding to the voice segment, a second intermediate output representation is obtained through the TTS text-to-speech model; The output representation of the encoder is aligned with the first intermediate output representation and the second intermediate output representation to realize inter-modal mapping, and the student model is trained using a distillation loss function; The mathematical expression of the distillation loss function is: wherein, represents a distillation loss function; x represents a speech segment; x asr represents a first intermediate output representation; x tts represents a second intermediate output representation; λ d1 ||x-x asr ||2 represents an L2 distance between the speech segment and the first intermediate output representation; λ d2 ||x-x tts ||2 represents an L2 distance between the speech segment and the second intermediate output representation; λ d1 , λ d2 respectively represent hyperparameters of the L2 norm for controlling the weight taken.

8. The method according to any one of claims 1 to 7, characterized in that, The method further includes: A code rate adaptive control module is introduced to dynamically adjust the number of quantization layers and / or token reservation strategies according to the content complexity level of the language segment; The code rate adaptive control module includes a complexity estimator, a pre-trained network decision maker, and a quantization scheduler; the complexity estimator is used to calculate the content complexity level according to the silence ratio, energy spectrum distribution, pronunciation rate, and / or entropy value of the current voice frame; the network decision maker is used to generate a control signal of the corresponding quantization strategy based on the content complexity level according to the light-weight Transformer structure; and the quantization scheduler is used to control the number of RVQ layers or the number of tokens reserved in each layer based on the control signal.

9. A neural speech compression system, characterized by, The system is applied to the neural voice compression method of any one of claims 1-8, and the system includes: An encoder configured to encode the original voice data to obtain a hidden vector; A hierarchical residual vector quantization module configured to quantize the hidden vector into a discretized code based on a multi-scale parallel token stream; A decoder configured to decode the discretized code to reconstruct a language signal to obtain a target language signal; The encoder is configured to introduce a high-level semantic representation extracted by a teacher network model through a cross-modal distillation technology when encoding the original language data. The decoder is configured as a neural decoder enhanced by adversarial training with a multi-scale discriminator.

10. The system of claim 9, wherein, The system further includes: A noise perception gating module configured to perform noise masking processing on the original language data before the encoder encodes the original voice data; The noise perception gating module is configured as a gated convolutional network structure or a gated recurrent unit network structure; And / or, A code rate adaptive control module configured to dynamically adjust the number of quantization layers and / or token reservation strategies according to the content complexity level of the language segment. The code rate self-adaptive control module comprises a complexity estimator, a pre-trained network decision maker and a quantization scheduler; the complexity estimator is configured to calculate a content complexity level according to a silence ratio, an energy spectrum distribution, a pronunciation rate and / or an entropy value of a current speech frame; the network decision maker is configured to generate a control signal of a corresponding quantization strategy based on the content complexity level and a light-weight Transformer structure; and the quantization scheduler is configured to control an RVQ layer number or a number of tokens reserved in each layer based on the control signal.

Images