Voice signal decoding method and apparatus, and electronic device

Abstract

The present application is applied to the technical field of audio encoding and decoding. Provided are a voice signal decoding method and apparatus, and an electronic device. In the method, an encoding apparatus encodes an original voice signal, so as to obtain an encoded bitstream, wherein the encoded bitstream comprises an acoustic feature encoding result; and a decoding apparatus acquires the acoustic feature encoding result from the coded bitstream, acquires a style feature from the acoustic feature encoding result, acquires an excitation feature, uses the excitation feature and the style feature to perform style fusion processing, so as to obtain a fused voice feature, and reconstructs a decoded voice signal by means of the voice feature. Since a style feature indicates the voice style of an original voice signal, a voice feature obtained by means of performing style fusion on an excitation feature and the style feature can restore the voice feature in the original voice signal, and with regard to a decoded voice signal obtained by means of performing reconstruction on the basis of the fused voice feature, compared with a decoded voice signal obtained without performing style fusion, the quality of the decoded voice signal can be effectively improved.

Description

Voice signal decoding method and device and electronic equipment

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office on August 14, 2023, with application number 202311022626.8 and invention name “Speech Signal Decoding Method and Device and Electronic Device”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The embodiments of the present application relate to the field of audio decoding technology, and in particular to a method and device for decoding a speech signal, and an electronic device. Background Art

[0003] The quality of synthesized speech at low bit rates using traditional speech coding and decoding algorithms has reached a bottleneck. Algorithms based on deep neural networks (DNNs) have been applied to the field of speech signal processing. This is the case with a DNN-based speech coding and decoding system. Its main goal is to leverage the powerful learning capabilities of DNNs to improve the performance of traditional speech coding and decoding algorithms.

[0004] DNN-based speech codec systems can be broadly categorized into two main categories: neural vocoder-based systems and end-to-end neural encoder systems. Neural vocoder-based systems employ an encoder to extract and quantize acoustic features, then use a neural vocoder to synthesize the quantized acoustic features into speech, resulting in an encoded speech signal. These systems offer the advantage of low bitrates, typically as low as 1.6 kilobits per second (kbps). However, because both the encoder and quantization steps directly reuse traditional codec algorithms, big data training can only adjust the decoder's parameters, limiting the quality of the decoded speech.

[0005] In contrast, neural encoder systems utilize a DNN to implement both the encoder and decoder, eliminating the need for a specific encoder feature extraction solution. Network parameters are adjusted through end-to-end joint training. However, when the bitrate of the speech encoding output is low, the performance of the decoded speech signal is poor.

[0006] Summary of the Invention

[0007] The present application provides a speech signal decoding method and device, and an electronic device.

[0008] In a first aspect, an embodiment of the present application provides a speech signal decoding method, the method comprising: obtaining an acoustic feature encoding result in a coded bitstream; obtaining a style feature from the acoustic feature encoding result, the style feature being used to indicate the speech style of the original speech signal; in response to the obtained excitation feature, performing style fusion processing on the style feature and the excitation feature to obtain a fused speech feature; and generating a decoded speech signal based on the fused speech feature.

[0009] In the above-described implementation, an encoding device encodes an original speech signal to obtain an encoded bitstream, which includes acoustic feature encoding results. A decoding device obtains the acoustic feature encoding results from the encoded bitstream, obtains style features from the acoustic feature encoding results, obtains excitation features, and performs style fusion processing using the excitation features and style features to obtain fused speech features. A decoded speech signal is reconstructed using these speech features. Because the style features indicate the speech style of the original speech signal, the speech features obtained by style fusion of the excitation features and style features can restore the speech features in the original speech signal. The decoded speech signal reconstructed based on the fused speech features can effectively improve the quality of the decoded speech signal compared to a decoded speech signal obtained without style fusion.

[0010] In one possible manner, the method further includes: obtaining the excitation feature through an excitation network. In the above implementation, the excitation network can generate the excitation feature, solving the problem of obtaining the excitation feature.

[0011] In one possible approach, the excitation network includes an excitation signal generation model and a first convolution module. Acquiring the excitation feature through the excitation network includes generating an excitation signal through the excitation signal generation model and performing convolution processing on the excitation signal through the first convolution module to obtain the excitation feature. In this implementation, the excitation signal generation model and the first convolution module enable the generation of the excitation feature using a simple device structure, simplifying the excitation feature generation process.

[0012] In one possible embodiment, the excitation network further includes a short-time Fourier transform module; obtaining the excitation features through the excitation network further includes extracting time-frequency domain features from the excitation signal through the short-time Fourier transform module; and performing convolution processing on the excitation signal through the first convolution module includes performing convolution processing on the time-frequency domain features through the first convolution module. In the above implementation, the first convolution module convolves the time-frequency domain features and outputs excitation features. The excitation features have the same dimension as the acoustic features, thereby simplifying calculations.

[0013] In one possible approach, obtaining style features from the acoustic feature encoding results includes obtaining the style features from the acoustic feature encoding results using a style network. In the above implementation, the style network can obtain style features, and the decoding device can input the style feature network into a generator within the decoding device, which then reconstructs the speech signal. This solves the problem of obtaining style features.

[0014] In one possible approach, the style network includes: a first upsampling module and a second convolution module; obtaining the style features from the acoustic feature encoding results through the style network includes: upsampling the acoustic feature encoding results through the first upsampling module to obtain an acoustic feature upsampling result; and convolving the acoustic feature upsampling result through the second convolution module to obtain the style features. In this implementation, the first upsampling module and the second convolution module enable style features to be generated using a simple device structure, simplifying the style feature generation process.

[0015] In one possible embodiment, the style network includes: a first upsampling module, a first convolutional layer, and a second convolutional layer; obtaining the style features from the acoustic feature encoding results through the style network includes: upsampling the acoustic feature encoding results through the first upsampling module to obtain acoustic feature upsampling results; inputting the acoustic feature upsampling results into the first convolutional layer for convolution processing to obtain style sub-features; inputting the style sub-features into the first upsampling module for upsampling to obtain style upsampling results; and inputting the style upsampling results into the second convolutional layer for convolution processing to obtain the style features. In the above implementation, the first and second convolutional layers allow style features to be generated using a simple device structure, simplifying the style feature generation process and solving the problem of generating style features when multiple layers of convolutional modules are stacked.

[0016] In one possible approach, performing style fusion processing on the style features and the excitation features to obtain fused speech features includes: inputting the style features and the excitation features into a generator, respectively, and performing style fusion processing on the style features and the excitation features by the generator to obtain fused speech features. In the above implementation, the generator in the decoding device receives the style features and the excitation features, respectively, and then performs style fusion processing on the style features and the excitation features to obtain fused speech features. The fused speech features carry the speech style of the original speech signal, which can improve the signal quality of the reconstructed speech signal.

[0017] In one possible embodiment, the generator includes: a second upsampling module, a style fusion module, a gating module, and a third convolution module; the generator performs style fusion processing on the style feature and the excitation feature to obtain a fused speech feature, including: upsampling the excitation feature through the second upsampling module to obtain a first signal feature; linearly modulating the first signal feature and the style feature through the style fusion module to obtain a second signal feature, and processing the second signal feature through the gating module to obtain a third signal feature; and convolution processing on the third signal feature through the third convolution module to obtain the fused speech feature. In the above implementation scheme, through the second upsampling module, the style fusion module, the gating module, and the third convolution module, style fusion can be achieved through a simple device structure, which simplifies the style fusion processing process and solves the problem of the generator synthesizing the excitation feature into a high-quality decoded speech signal.

[0018] In one possible approach, the style fusion module includes: a first style fusion submodule and a second style fusion submodule; the gating module includes: a first gating submodule and a second gating submodule; the linear modulation of the first signal feature and the style feature by the style fusion module to obtain the second signal feature, and the processing of the second signal feature by the gating module to obtain the third signal feature, includes: linear modulation of the first signal feature and the style feature by the first style fusion submodule to obtain a first modulation result; processing the first modulation result by the first gating submodule to obtain a first gating result; linear modulation of the first gating result and the style feature by the second style fusion submodule to obtain a second modulation result; and processing the second modulation result by the second gating submodule to obtain the third signal feature. In the above implementation, the first and second style fusion submodules, the first and second gating submodules, and the second gating submodules enable the generation of the third signal feature using a simple device structure, simplifying the signal feature generation process and solving the problem of generating the third signal feature when multiple layers of style fusion modules and gating modules are stacked.

[0019] In one possible embodiment, the method further includes: inputting the decoded speech signal and the original speech signal into a discriminator, and distinguishing the decoded speech signal from the original speech signal via the discriminator. In the above solution, when the speech codec system uses a generative adversarial network for model training, the discriminator is used during training to distinguish between the original speech signal and the decoded speech signal, thereby improving training accuracy.

[0020] In a second aspect, an embodiment of the present application provides a speech signal decoding device, the device comprising:

[0021] An acquisition module is used to obtain the acoustic feature encoding result in the encoded bitstream;

[0022] a style feature acquisition module, configured to acquire style features from the acoustic feature encoding result, wherein the style features are used to indicate the speech style of the original speech signal;

[0023] a style fusion module, configured to perform style fusion processing on the style feature and the excitation feature in response to the acquired excitation feature to obtain a fused speech feature;

[0024] The signal generating module is used to generate a decoded speech signal according to the fused speech features.

[0025] In the above-described implementation, an encoding device encodes an original speech signal to obtain an encoded bitstream, which includes acoustic feature encoding results. A decoding device obtains the acoustic feature encoding results from the encoded bitstream, obtains style features from the acoustic feature encoding results, obtains excitation features, and performs style fusion processing using the excitation features and style features to obtain fused speech features. A decoded speech signal is reconstructed using these speech features. Because the style features indicate the speech style of the original speech signal, the speech features obtained by style fusion of the excitation features and style features can restore the speech features in the original speech signal. The decoded speech signal reconstructed based on the fused speech features can effectively improve the quality of the decoded speech signal compared to a decoded speech signal obtained without style fusion.

[0026] The speech signal decoding device of the second aspect can execute the steps in the first aspect and any implementation manner of the first aspect, which will not be repeated here.

[0027] The second aspect and any implementation of the second aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the second aspect and any implementation of the second aspect can be referred to the technical effects corresponding to the first aspect and any implementation of the first aspect, and will not be repeated here.

[0028] In a third aspect, an embodiment of the present application provides an electronic device comprising: a memory and a processor, the memory being coupled to the processor; the memory storing program instructions, which, when executed by the processor, enables the electronic device to execute the speech signal decoding method in the first aspect or any possible implementation of the first aspect.

[0029] The third aspect and any implementation of the third aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the third aspect and any implementation of the third aspect can be referred to the technical effects corresponding to the first aspect and any implementation of the first aspect, and will not be repeated here.

[0030] In a fourth aspect, an embodiment of the present application provides a chip comprising one or more interface circuits and one or more processors; the interface circuit is used to receive a signal from a memory of an electronic device and send a signal to the processor, the signal including a computer instruction stored in the memory; when the processor executes the computer instruction, the electronic device executes the speech signal decoding method in the first aspect or any possible implementation of the first aspect.

[0031] The fourth aspect and any implementation of the fourth aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the fourth aspect and any implementation of the fourth aspect can be referred to the technical effects corresponding to the first aspect and any implementation of the first aspect, and will not be repeated here.

[0032] In a fifth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program runs on a computer or a processor, the computer or the processor executes the speech signal decoding method in the first aspect or any possible implementation of the first aspect.

[0033] The fifth aspect and any implementation of the fifth aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the fifth aspect and any implementation of the fifth aspect can be referred to the technical effects corresponding to the first aspect and any implementation of the first aspect, and will not be repeated here.

[0034] In a sixth aspect, an embodiment of the present application provides a computer program product, which includes a software program. When the software program is executed by a computer or a processor, the computer or the processor executes the speech signal decoding method in the first aspect or any possible implementation of the first aspect.

[0035] The sixth aspect and any implementation of the sixth aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the sixth aspect and any implementation of the sixth aspect can be referred to the technical effects corresponding to the first aspect and any implementation of the first aspect, and will not be repeated here.

[0036] In a seventh aspect, an embodiment of the present application provides a device for storing a code stream, the device comprising: a receiver and at least one storage medium, the receiver being used to receive the code stream; the at least one storage medium being used to store the code stream; the code stream being generated according to the first aspect and any one of the implementation methods of the first aspect.

[0037] The seventh aspect and any implementation of the seventh aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the seventh aspect and any implementation of the seventh aspect can be referred to the technical effects corresponding to the first aspect and any implementation of the first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] FIG1a is a schematic diagram of an exemplary application scenario;

[0039] FIG1b is a schematic diagram illustrating an exemplary application scenario;

[0040] FIG2 is a schematic diagram showing the structure of a speech coding and decoding system;

[0041] FIG3 is a schematic structural diagram of another speech coding and decoding system;

[0042] FIG4 is a schematic structural diagram of another speech coding and decoding system;

[0043] FIG5 is a schematic diagram illustrating an exemplary decoding process;

[0044] FIG6 is a schematic diagram showing the structure of an exemplary excitation network;

[0045] FIG7 is a schematic diagram showing the structure of a style network;

[0046] FIG8 is a schematic diagram showing the structure of a second convolution module;

[0047] FIG9 is a schematic diagram showing the structure of a generator;

[0048] FIG10 is a schematic structural diagram illustrating an exemplary decoding device;

[0049] FIG11 is a schematic structural diagram of another exemplary speech coding and decoding system;

[0050] FIG12a is a schematic structural diagram of a first style network module and a first generator module in an exemplary decoding device;

[0051] FIG12 b is a schematic diagram illustrating the structures of an N-th style network module and an N-th generator module in an exemplary decoding device;

[0052] FIG13 is a schematic structural diagram illustrating an exemplary speech signal decoding apparatus;

[0053] FIG14 is a schematic structural diagram of an exemplary speech signal decoding apparatus. DETAILED DESCRIPTION

[0054] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0055] The term "and / or" in this article is merely a description of the association relationship between associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone.

[0056] In the description and claims of the embodiments of this application, the terms "first" and "second" are used to distinguish different objects, rather than to describe a specific order of objects. For example, the terms "first target object" and "second target object" are used to distinguish different objects, rather than to describe a specific order of objects.

[0057] In the embodiments of this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of this application should not be interpreted as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0058] In the description of the embodiments of this application, unless otherwise specified, "multiple" means two or more. For example, "multiple processing units" means two or more processing units; "multiple systems" means two or more systems.

[0059] In order to make the description of the following embodiments clear and concise, a brief introduction to the relevant technology is first given.

[0060] Sound is a continuous wave produced by the vibration of an object. The object that vibrates and emits sound waves is called the sound source. As sound waves propagate through a medium (such as air, solid, or liquid), they are detected by the human or animal auditory system.

[0061] The characteristics of sound waves include pitch, intensity, and timbre. Pitch indicates the highness of a sound. Intensity indicates how loud a sound is. Intensity can also be called loudness or volume. The unit of intensity is decibel (dB). Timbre is also called timbre quality.

[0062] The frequency of a sound wave determines its pitch. The higher the frequency, the higher the pitch. The frequency is the number of times an object vibrates in one second, measured in hertz (Hz). The human ear can detect sounds with frequencies between 20Hz and 20,000Hz.

[0063] The amplitude of a sound wave determines its intensity. The greater the amplitude, the greater the intensity. The closer to the sound source, the greater the intensity.

[0064] The waveform of the sound wave determines the timbre, which includes square wave, sawtooth wave, sine wave and pulse wave.

[0065] Based on the characteristics of sound waves, sounds can be divided into regular sounds and irregular sounds. Irregular sounds refer to sounds produced by the irregular vibration of a sound source. Irregular sounds are, for example, noise that affects people's work, study, and rest. Regular sounds refer to sounds produced by the regular vibration of a sound source. Regular sounds include speech and music. When sound is represented electrically, regular sounds are analog signals that continuously change in the time-frequency domain. This analog signal can be called an audio signal. An audio signal is an information carrier that carries speech, music, and sound effects. For example, audio signals include speech signals. Since human hearing has the ability to distinguish the positional distribution of sound sources in space, when listeners hear sounds in space, in addition to perceiving the pitch, intensity, and timbre of the sound, they can also perceive the direction of the sound.

[0066] The speech signal involved in the embodiment of the present application may include a mono signal or a multi-channel signal. For example, the original speech signal obtained by the encoding device of the embodiment of the present application may include a scene audio signal, and the scene audio signal may refer to a signal used to describe the sound field; wherein the speech signal may include: a higher order ambisonics (HOA) signal (wherein the HOA signal may include a three-dimensional HOA signal and a two-dimensional HOA signal (also referred to as a planar HOA signal)) and a three-dimensional audio signal; the three-dimensional audio signal may refer to other audio signals in the speech signal except the HOA signal.

[0067] Acoustic features are hidden acoustic features obtained by mapping the original speech signal through the encoding device. The encoding device inputs the acoustic features into the quantizer, and compresses the acoustic features through the quantizer. The compression result generated by the quantizer is called the acoustic feature encoding result.

[0068] A style feature refers to the characteristics of the style information of a speech signal. For example, a style feature is used to indicate the speech style of the original speech signal. In one implementation, the style feature may reflect a phonetic feature, such as at least one characteristic of the speech signal, such as the speech content, speech energy, or speech timbre.

[0069] In one possible implementation, the style features are generated by a style network. In an embodiment of the present application, a decoding device may include a style network. The acoustic feature encoding results output by the quantizer may be sent to the decoding device. The acoustic feature encoding results are input to the style network included in the decoding device. The style network may include multiple convolutional layers, and the style features are obtained through mapping through the multiple convolutional layers. In an embodiment of the present application, the acoustic feature encoding results are no longer directly used as input to the generator in the decoding device. Instead, the input of the generator is a constant. The style network uses the style features to convert the constant input into a target output, which is a decoded speech signal.

[0070] The excitation feature is used for style fusion processing with the style feature to obtain a fused speech feature. The excitation feature can be generated by an excitation network. For example, the excitation network generates the excitation feature and then inputs it to the generator in the decoding device. In another example, the decoding device includes an excitation network and a generator, and the excitation network generates the excitation feature and then sends it to the generator.

[0071] Figure 1a is a schematic diagram of an exemplary application scenario, which illustrates a speech signal encoding and decoding scenario.

[0072] 1a, illustratively, the first electronic device may include a first audio acquisition module, a first voice signal decoding module, a first channel encoding module, a first channel decoding module, a first voice signal decoding module, and a first audio playback module. It should be understood that the first electronic device may include more or fewer modules than those shown in FIG1a, and this application does not limit this.

[0073] 1a, illustratively, the second electronic device may include a second audio acquisition module, a second voice signal decoding module, a second channel encoding module, a second channel decoding module, a second voice signal decoding module, and a second audio playback module. It should be understood that the second electronic device may include more or fewer modules than those shown in FIG1a, and this application does not limit this.

[0074] Exemplarily, the process of encoding and transmitting a voice signal to a second electronic device by a first electronic device, and decoding and audio playback by the second electronic device can be as follows: the first audio acquisition module can perform audio acquisition and output the voice signal to the first voice signal decoding module. Then, the first voice signal decoding module can encode the voice signal and output the code stream to the first channel encoding module. Afterwards, the first channel encoding module can channel encode the code stream and transmit the channel-encoded code stream to the second electronic device via a wireless or wired network communication device. Then, the second channel decoding module of the second electronic device can channel decode the received data to obtain a code stream and output the code stream to the second voice signal decoding module. Then, the second voice signal decoding module can decode the code stream to obtain a reconstructed voice signal; then the reconstructed voice signal is output to the second audio playback module, and the second audio playback module performs audio playback.

[0075] It should be understood that the process of the second electronic device encoding and transmitting the voice signal to the first electronic device, which is decoded and the audio is played back by the first electronic device, is similar to the above-mentioned process of the first electronic device transmitting the voice signal to the second electronic device, which is played back by the second electronic device, and will not be repeated here.

[0076] Exemplarily, the first electronic device and the second electronic device may include, but are not limited to: personal computers, computer workstations, smart phones, tablet computers, servers, smart cameras, smart cars or other types of cellular phones, media consumption devices, wearable devices, set-top boxes, game consoles, etc.

[0077] For example, the present application can be specifically applied to virtual reality (VR) / augmented reality (AR) scenarios. In one possible embodiment, the first electronic device is a server and the second electronic device is a VR / AR device. In another possible embodiment, the second electronic device is a server and the first electronic device is a VR / AR device.

[0078] Exemplarily, the first voice signal decoding module and the second voice signal decoding module may be voice signal decoders. The first voice signal decoding module and the second voice signal decoding module may be voice signal decoders.

[0079] For example, when a first electronic device encodes a voice signal and a second electronic device reconstructs the voice signal, the first electronic device may be referred to as an encoding end or encoding device, and the second electronic device may be referred to as a decoding end or decoding device. When a second electronic device encodes a voice signal and a first electronic device reconstructs the voice signal, the second electronic device may be referred to as an encoding end or encoding device, and the first electronic device may be referred to as a decoding end or decoding device.

[0080] Figure 1b is a schematic diagram of an exemplary application scenario, which shows a transcoding scenario of a speech signal.

[0081] 1b(1), illustratively, a wireless or core network device may include: a channel decoding module, other audio decoding modules, a voice signal decoding module, and a channel coding module. The wireless or core network device may be used for audio transcoding.

[0082] Exemplarily, the specific application scenario of Figure 1b (1) may be: when the first electronic device is not provided with a voice signal decoding module but only with other audio encoding modules; and the second electronic device is only provided with a voice signal decoding module but not with other audio decoding modules, in order to enable the second electronic device to decode and play back the voice signal encoded by the first electronic device using other audio encoding modules, wireless or core network equipment may be used for transcoding.

[0083] Specifically, the first electronic device uses another audio encoding module to encode the voice signal to obtain a first code stream; and then channel-encodes the first code stream and sends it to the wireless or core network device. Next, the channel decoding module of the wireless or core network device can perform channel decoding and output the first code stream obtained by channel decoding to the other audio decoding module. Thereafter, the other audio decoding module decodes the first code stream to obtain a voice signal and outputs the voice signal to the voice signal decoding module. The voice signal decoding module can then encode the voice signal to obtain a second code stream and output the second code stream to the channel encoding module. The channel encoding module then channel-encodes the second code stream and sends it to the second electronic device. In this way, the second electronic device can call the voice signal decoding module to decode the second code stream obtained by channel decoding to obtain a reconstructed voice signal; the reconstructed voice signal can then be played back for audio.

[0084] 1b(2), illustratively, a wireless or core network device may include: a channel decoding module, a voice signal decoding module, other audio encoding modules, and a channel encoding module. The wireless or core network device may be used for audio transcoding.

[0085] Exemplarily, the specific application scenario of Figure 1b (2) may be: when the first electronic device is only provided with a voice signal decoding module and no other audio encoding modules; and the second electronic device is not provided with a voice signal decoding module and only provided with other audio decoding modules, in order to enable the second electronic device to decode and play back the voice signal encoded by the voice signal decoding module of the first electronic device, wireless or core network equipment may be used for transcoding.

[0086] Specifically, the first electronic device uses a voice signal decoding module to encode the voice signal to obtain a first code stream; the first code stream is channel-encoded and transmitted to a wireless or core network device. The channel decoding module of the wireless or core network device can then perform channel decoding and output the first code stream obtained by channel decoding to the voice signal decoding module. The voice signal decoding module then decodes the first code stream to obtain a voice signal, which is then output to another audio encoding module. The other audio encoding module can then encode the voice signal to obtain a second code stream, which is then output to the channel encoding module. The channel encoding module then performs channel encoding on the second code stream and transmits it to the second electronic device. The second electronic device can then call another audio decoding module to decode the second code stream obtained by channel decoding to obtain a reconstructed voice signal; the reconstructed voice signal can then be played back for audio playback.

[0087] Among end-to-end neural codec systems, there is a framework based on a vector quantization generative adversarial network (VQGAN). VQGAN-based speech coding systems produce low bitrates during speech encoding, resulting in poor performance of the decoded speech signal. For example, VQGAN-based speech coding systems aim to minimize the sample point error between the original and decoded speech signals. However, at low bitrates (less than 6 kbps), decoded signal recovery can be poor.

[0088] Please refer to Figure 2, which is a schematic diagram of the structure of a speech signal encoding and decoding system provided in an embodiment of the present application. The speech signal encoding and decoding system can specifically be a style-fusion-based vector quantization generative adversarial network (StyleVQGAN) speech encoding and decoding system. Compared to the aforementioned VQGAN framework, the speech signal encoding and decoding system provided in an embodiment of the present application includes a decoding device based on style fusion, which can utilize quantized latent acoustic features and excitation features to better reconstruct the decoded signal, reconstructing a decoded speech signal that is as close to the original signal as possible in an adversarial generation manner.

[0089] As shown in FIG2 , an embodiment of the present application provides a speech signal encoding and decoding system, including:

[0090] An encoding device for mapping an original speech signal into acoustic features of a hidden layer, wherein the original speech signal may include a time domain signal; the acoustic features are input into a quantizer;

[0091] A quantizer is used to compress the acoustic features to obtain a coded stream, which includes a compressed acoustic feature result; and transmit the compressed acoustic feature result in the coded stream to a decoding device via a transmission network;

[0092] A decoding device, used to obtain an acoustic feature encoding result in the encoded bit stream, the acoustic feature encoding result being obtained by the encoding device performing feature extraction on the original speech signal;

[0093] The decoding device is further used to obtain style features from the acoustic feature encoding result, where the style features are used to indicate the speech style of the original speech signal;

[0094] The decoding device is further configured to perform style fusion processing on the style feature and the excitation feature in response to the acquired excitation feature to obtain a fused speech feature;

[0095] The decoding device is used to generate a decoded speech signal based on the fused speech features.

[0096] Please refer to Figure 3, which is a schematic diagram of the structure of another speech signal encoding and decoding system provided in an embodiment of the present application. The speech signal encoding and decoding system also includes: an excitation network and a style network. Specifically, the style network can be part of the decoding device, or the style network is a separate functional component independent of the decoding device. The decoding device also includes a generator. The excitation network can be part of the decoding device, or the excitation network is a separate functional component independent of the decoding device.

[0097] The style network is used to obtain style features from the acoustic feature encoding results and input the style features into the generator;

[0098] The excitation network is used to obtain excitation features and input the excitation features into the generator;

[0099] For example, the excitation network may include a speech generation model, which generates an excitation signal through the speech generation model, and then extracts excitation features from the excitation signal. Specifically, the excitation network may include a sine wave-noise model, which generates an excitation signal through the sine wave-noise model, and then extracts excitation features from the excitation signal. For another example, the excitation network may include a Gaussian white noise generator and a constant sequence generator, which generates an excitation signal through the Gaussian white noise generator and the constant sequence generator, and then extracts excitation features from the excitation signal.

[0100] The generator is used to synthesize the excitation features into the decoded speech signal under the guidance of the style features as conditions.

[0101] For example, the excitation features generated by the excitation network are used as the input excitation of the generator. The excitation features include: time-frequency domain feature maps. The generator uses the time-frequency domain feature maps as conditions to synthesize the excitation features into a decoded speech signal.

[0102] Please refer to Figure 4, which is a schematic diagram of the composition structure of another speech signal encoding and decoding system provided in an embodiment of the present application. The speech signal encoding and decoding system also includes: a discriminator, which is connected to the decoding device. After the generator generates a decoded speech signal, the decoded speech signal is input into the discriminator.

[0103] The discriminator is configured to receive the decoded speech signal and the original speech signal, and identify the decoded speech signal and the original speech signal.

[0104] For example, the discriminator may specifically be a multi-scale time-frequency domain discriminator.

[0105] In the embodiments of the present application, the decoding device and the discriminator reconstruct a decoded speech signal that is as close to the original speech signal as possible in an adversarial generation manner. The discriminator in the speech codec system is an optional device. After the model training of the speech codec system is completed, the discriminator can no longer be used to recognize speech signals.

[0106] The following describes the encoding and decoding process of the speech signal.

[0107] First, the speech encoding method on the encoding side provided in the embodiment of the present application is described.

[0108] The encoder's primary purpose is to map the input time-domain signal into a hidden acoustic representation. The encoder downsamples the time-domain signal and ultimately outputs an acoustic representation containing speech information. This acoustic representation is not fed directly into the generator. Instead, it passes through the style network to generate style features, which are then fed into the generator as auxiliary input.

[0109] After the encoding device maps the acoustic features, the acoustic features are input into the quantizer. The purpose of the quantizer is to compress the acoustic representation output by the encoding device to a specific bit rate to obtain the acoustic feature encoding result.

[0110] Next, the speech decoding method on the decoding side provided in the embodiment of the present application is described.

[0111] Figure 5 is a schematic diagram of an exemplary decoding process. Figure 5 is a decoding process corresponding to the aforementioned encoding process, which mainly includes the following steps:

[0112] 501. Obtain acoustic feature encoding results in the encoded bitstream.

[0113] The encoding device may extract features from the original speech signal to obtain an acoustic feature encoding result, and the encoding device and the decoding device may transmit a code stream, and the decoding device may obtain the encoded code stream from the encoding device. It is not limited to the fact that the acoustic feature encoding result may also be obtained after being compressed by a quantizer.

[0114] In some embodiments of the present application, the main part of the encoder is composed of a plurality of encoder blocks stacked together, and the decoding device performs a decoding process inverse to that of the encoder. The decoding device may also include a plurality of decoding blocks. An example is given below:

[0115] The encoding device includes four encoding blocks that downsample the original speech signal at downsampling rates of 2, 4, 5, and 8, respectively, ultimately outputting a latent acoustic representation containing speech information. The decoding device includes four decoding modules consisting of a style network module and a generator module, with each decoding module upsampling at rates of 8, 5, 4, and 2, respectively.

[0116] 502. Obtain a style feature from the acoustic feature encoding result, where the style feature is used to indicate the speech style of the original speech signal.

[0117] After the decoding device obtains the acoustic feature encoding result from the encoded code stream, it performs speech style extraction based on the acoustic feature encoding result to obtain a style feature. The style feature indicates the speech style of the original speech signal. The style feature can be used as an input condition of the generator of the decoding device to guide the generator to reconstruct the speech signal.

[0118] 503. In response to the acquired excitation feature, perform style fusion processing on the style feature and the excitation feature to obtain a fused speech feature.

[0119] The decoding device may obtain the excitation feature, for example, the decoding device obtains the excitation feature before obtaining the style feature, or obtains the excitation feature after obtaining the style feature, or the decoding device obtains the style feature and the excitation feature at the same time.

[0120] In some embodiments of the present application, in addition to performing the aforementioned steps, the speech signal decoding method provided in the embodiments of the present application further includes the following steps:

[0121] A1. Obtain incentive features through incentive network.

[0122] As shown in Figure 3 or Figure 4, the excitation network can generate excitation features, and the decoding device can receive the excitation features input by the excitation network into the decoding device, or the decoding device can include an excitation network, and the excitation network can generate excitation features, thereby solving the problem of obtaining excitation features.

[0123] In some embodiments of the present application, as shown in FIG6 , the excitation network includes: an excitation signal generation model and a first convolution (ConV) module;

[0124] The aforementioned step A1 obtains the incentive features through the incentive network, including:

[0125] A11. Generate an excitation signal through an excitation signal generation model;

[0126] Among them, the excitation network includes an excitation signal generation model, which can generate an excitation signal. For example, the excitation signal generation model can include a Gaussian white noise generator, a constant sequence generator, or a traditional speech generation model, such as a pulse wave-noise model and a sine wave-noise model.

[0127] Taking the sine wave-noise model as an example of the excitation signal generation model, the acoustic characteristics are used to judge the state of unvoiced and voiced sounds, and the corresponding excitation signal is generated according to whether a frame signal is unvoiced or voiced. Gaussian noise excitation is used as the excitation signal in the unvoiced part, and sine wave excitation is used as the excitation signal in the voiced part.

[0128] A12. Perform convolution processing on the excitation signal through the first convolution module to obtain an excitation feature.

[0129] The first convolution module inputs an excitation signal, performs convolution on the excitation signal, and outputs an excitation feature. The function of the first convolution module is to obtain an excitation feature with the same dimension as the acoustic feature.

[0130] In the embodiment of the present application, the excitation signal generation model and the first convolution module are used to generate excitation features through a simple device structure, thereby simplifying the excitation feature generation process.

[0131] In some embodiments of the present application, as shown in FIG6 , the excitation network further includes: a short-time Fourier transform module (STFT);

[0132] Step A1 obtains the incentive features through the incentive network, and also includes:

[0133] A13. Extract time-frequency domain features from the excitation signal through the short-time Fourier transform module.

[0134] The time-frequency domain features extracted from the excitation signal may be a time-frequency domain feature map, and the time-frequency domain feature map generated by the excitation network is used as an input excitation of the generator.

[0135] Step A12 performs convolution processing on the excitation signal through a first convolution module, including:

[0136] A121. Perform convolution processing on the time-frequency domain features through the first convolution module.

[0137] Among them, the first convolution module convolves the time-frequency domain features and outputs the excitation features, which have the same dimension as the acoustic features, thereby simplifying the calculation.

[0138] In some embodiments of the present application, step 502 of obtaining style features from the acoustic feature encoding result includes:

[0139] B1. Obtain style features from the acoustic feature encoding results through the style network.

[0140] As shown in Figure 3 or Figure 4, the style network can obtain style features, and the decoding device can input the style feature network into the generator in the decoding device to reconstruct the speech signal through the generator. This solves the problem of obtaining style features.

[0141] In some embodiments of the present application, as shown in FIG7 , the style network includes: a first upsampling module (Upsampler) and a second convolution module;

[0142] The aforementioned step B1 obtains style features from the acoustic feature encoding results through the style network, including:

[0143] B11. Upsampling the acoustic feature encoding result through a first upsampling module to obtain an acoustic feature upsampling result;

[0144] The acoustic feature encoding result may include quantized acoustic features, and the first upsampling module upsamples the acoustic feature encoding result to obtain an acoustic feature upsampling result. The upsampling operation performed by the first upsampling module corresponds to the downsampling performed by the encoding device.

[0145] B12. Perform convolution processing on the acoustic feature upsampling result through the second convolution module to obtain style features.

[0146] The first upsampling module inputs the acoustic feature upsampling results into the second convolution module, which performs convolution processing on the acoustic feature upsampling results to generate style features. The style features indicate the speech style of the original speech signal. Style features can also be called speech style features.

[0147] In the embodiment of the present application, the first upsampling module and the second convolution module are used to generate style features through a simple device structure, thereby simplifying the process of generating style features.

[0148] In some embodiments of the present application, the second convolution module may include two or more convolution layers, wherein the input of the current layer is the output of the previous layer, and the last layer outputs the convolution result. For example, the second convolution module includes two convolution layers. As shown in FIG7 , the second convolution module includes: a first convolution layer and a second convolution layer.

[0149] In some embodiments of the present application, the style network includes: a first upsampling module, a first convolutional layer, and a second convolutional layer. For example, the style network includes: a first upsampling module (Upsampler) and a second convolutional module, as shown in Figure 8, where the second convolutional module includes the first convolutional layer and the second convolutional layer.

[0150] The aforementioned step B1 obtains style features from the acoustic feature encoding results through the style network, including:

[0151] B13. Upsampling the acoustic feature encoding result by the first upsampling module to obtain an acoustic feature upsampling result;

[0152] The implementation of step B13 is similar to that of step B11, and will not be repeated here.

[0153] B14. Input the acoustic feature upsampling result into the first convolutional layer for convolution processing to obtain style sub-features;

[0154] Among them, the first convolution layer is the top layer in the second convolution module. The input of the first convolution layer is the acoustic feature upsampling result. The first convolution layer performs convolution processing on the acoustic feature upsampling result and outputs the style sub-feature. The style sub-feature represents the speech style of the original speech signal extracted by the first convolution layer.

[0155] B15. Input the style sub-feature into the first upsampling module for upsampling to obtain a style upsampling result.

[0156] The first convolutional layer inputs the style sub-feature into the first upsampling module, which then upsamples the style sub-feature to produce a style upsampling result. The style upsampling result represents the result of upsampling the style sub-feature. Furthermore, the first convolutional layer inputs the style sub-feature into the generator.

[0157] It is understood that the first upsampling module in both step B11 and step B15 performs upsampling, which can be specifically performed by different layers included in the first upsampling module. For example, as shown in Figure 10, the first upsampling module includes: a first upsampling layer and a second upsampling layer, the first upsampling layer performs the aforementioned step B11, and the second upsampling layer performs the aforementioned step B15.

[0158] B16. Input the style upsampling result into the second convolutional layer for convolution processing to obtain style features.

[0159] The first upsampling module inputs the style upsampling result into the second convolutional layer, which then performs convolution processing on the style upsampling result to obtain the style feature. In this example, the second convolutional module includes two convolutional layers, with the second convolutional layer being the last layer. In this case, the second convolutional layer inputs the style sub-feature into the generator. It is understood that if the second convolutional module includes more layers, the second convolutional layer still needs to send the style feature to the next layer.

[0160] In the embodiment of the present application, the first convolution layer and the second convolution layer are used to generate style features through a simple device structure, which simplifies the style feature generation process and solves the problem of generating style features when multiple layers of convolution modules are stacked.

[0161] In some embodiments of the present application, as shown in FIG3 or FIG4 , the aforementioned step 503 performs style fusion processing on the style feature and the excitation feature to obtain a fused speech feature, including:

[0162] C1. Input the style features and the excitation features into the generator respectively, and perform style fusion processing on the style features and the excitation features through the generator to obtain the fused speech features.

[0163] The decoding device includes a style network and a generator, and can also be referred to as a decoder based on style fusion. The generator in the decoding device receives style features and excitation features separately, and then performs style fusion processing on the style features and excitation features to obtain fused speech features. The fused speech features carry the speech style of the original speech signal, which can improve the signal quality of the reconstructed speech signal.

[0164] In some embodiments of the present application, as shown in FIG9 , the generator includes: a second upsampling module, a style fusion module (StyleBlock), a gating module (Gated Function) and a third convolution module;

[0165] Step C1 performs style fusion processing on the style features and the excitation features through the generator to obtain the fused speech features, including:

[0166] C11. Perform upsampling processing on the excitation feature through a second upsampling module to obtain a first signal feature.

[0167] Among them, the excitation network inputs the excitation feature into the second upsampling module, and the second upsampling module performs upsampling processing on the excitation feature to obtain the first signal feature. The upsampling operation performed by the second upsampling module corresponds to the downsampling performed by the encoding device.

[0168] It is understandable that the second upsampling module and the first upsampling module have the same number of sampling layers. For example, as shown in FIG10 , the second upsampling module includes: a third upsampling layer and a fourth upsampling layer, and the third upsampling layer performs the aforementioned step C11.

[0169] C12. Linearly modulate the first signal feature and the style feature through a style fusion module to obtain a second signal feature, and process the second signal feature through a gating module to obtain a third signal feature.

[0170] The style network inputs the style features into the style fusion module. The second upsampling module inputs the first signal features into the style fusion module. The style fusion module linearly modulates the first signal features and the style features to obtain the second signal features. For example, the style fusion module uses the Temporal Adaptive DE-normalization (TADE) algorithm for style fusion. The style fusion module uses the TADE algorithm to linearly modulate the first signal features and the style features to obtain the second signal features.

[0171] After the style fusion module generates the second signal feature, the second signal feature is input into the gating module, which processes the second signal feature to obtain the third signal feature. The gating module can reduce the quantization noise in the second signal feature and improve the quality of the decoded speech signal.

[0172] C13. Perform convolution processing on the third signal feature through a third convolution module to obtain a fused speech feature.

[0173] After generating the third signal feature, the gating module inputs the third signal feature into the third convolution module. The third convolution module performs convolution processing on the third signal feature to obtain a fused speech feature. The convolution processing is performed by the third convolution module so that the fused speech feature can have the same dimension as the acoustic feature of the original speech signal, thereby improving the quality of the decoded speech signal.

[0174] In the embodiment of the present application, through the second upsampling module, the style fusion module, the gating module and the third convolution module, style fusion can be achieved through a simple device structure, which simplifies the processing process of style fusion and solves the problem of the generator synthesizing the excitation features into a high-quality decoded speech signal.

[0175] In some embodiments of the present application, the second upsampling module may include two or more upsampling layers, wherein the input of the current layer is the output of the previous layer, and the last layer outputs the upsampling result. For example, the second upsampling module includes two upsampling layers. As shown in FIG10 , the second upsampling module includes a third upsampling layer and a fourth upsampling layer.

[0176] Similarly, the style fusion module included in the generator includes two or more style fusion sub-modules. In some embodiments of the present application, as shown in FIG10 , the style fusion module includes a first style fusion sub-module and a second style fusion sub-module for example.

[0177] The gating module included in the generator includes two or more gating sub-modules. In some embodiments of the present application, as shown in FIG10 , the gating module includes a first gating sub-module and a second gating sub-module for illustrative purposes.

[0178] Based on the style network and generator shown in FIG10 , step C12 linearly modulates the first signal feature and the style feature through the style fusion module to obtain a second signal feature, and processes the second signal feature through the gating module to obtain a third signal feature, including:

[0179] C121. Linearly modulate the first signal feature and the style feature through a first style fusion submodule to obtain a first modulation result.

[0180] The excitation network inputs the excitation features into the third upsampling layer in the second upsampling module. The third upsampling layer upsamples the excitation features to obtain first signal features. The third upsampling layer inputs the first signal features into the first style fusion submodule. When the first style fusion submodule performs style fusion, it may linearly modulate the first signal features and the style features to obtain a first modulation result. The first style fusion submodule inputs the first modulation result into the first gating submodule.

[0181] As shown in FIG10 , the style feature input into the first style fusion submodule is specifically the style sub-feature output by the first convolution layer in the second convolution module.

[0182] C122 . Process the first modulation result through a first gating submodule to obtain a first gating result.

[0183] The first gating submodule processes the first modulation result to obtain a first gating result. The first gating submodule can reduce quantization noise in the first modulation result and improve the quality of the decoded speech signal.

[0184] After obtaining the first gating result, the first gating submodule inputs the first gating result to the fourth upsampling layer in the second upsampling module. The fourth upsampling layer upsamples the first gating result to obtain an upsampled first gating result. The fourth upsampling layer inputs the upsampled first gating result to the second style fusion submodule.

[0185] C123. Linearly modulate the first gating result and the style feature through a second style fusion submodule to obtain a second modulation result.

[0186] When the second style fusion submodule performs style fusion, it can fuse the first gating result and the style feature by linearly modulating them to obtain a second modulated result. The second style fusion submodule inputs the second modulated result into the second gating submodule.

[0187] As shown in FIG10 , the style features input into the second style fusion submodule are specifically the style features output by the second convolutional layer in the second convolutional module.

[0188] C124. Process the second modulation result through the second gating submodule to obtain a third signal feature.

[0189] The second gating submodule processes the second modulation result to obtain a third signal feature. The second gating submodule can reduce quantization noise in the second modulation result and improve the quality of the decoded speech signal.

[0190] In an embodiment of the present application, through the first style fusion submodule and the second style fusion submodule, the first gating submodule and the second gating submodule, the third signal feature can be generated through a simple device structure, which simplifies the signal feature generation process and solves the problem of generating the third signal feature when the style fusion module and the gating module are stacked in multiple layers.

[0191] 504. Generate a decoded speech signal according to the fused speech features.

[0192] After the decoding device generates the fused speech features in step 503, it can reconstruct the speech signal based on the fused speech features, thereby generating a decoded speech signal. In the embodiment of the present application, since the decoding side uses a decoding device based on style fusion, it can simultaneously utilize acoustic features and excitation features to better reconstruct the decoded speech signal, thereby improving the quality of the decoded speech signal.

[0193] In some embodiments of the present application, in addition to performing the aforementioned steps, the speech signal decoding method provided in the embodiments of the present application further includes the following steps:

[0194] D1. Input the decoded speech signal and the original speech signal into the discriminator, and use the discriminator to identify the decoded speech signal and the original speech signal.

[0195] As shown in Figure 4, the discriminator is connected to the decoding device. After the generator generates a decoded speech signal, it is input into the discriminator. The decoding device and the discriminator use an adversarial generative approach to reconstruct a decoded speech signal that is as close to the original as possible. When a speech codec system uses a generative adversarial network for model training, the discriminator is used during training to distinguish between the original and decoded speech signals to improve training accuracy. After the speech codec system completes training, the discriminator may be removed from the system.

[0196] As can be seen from the examples in the aforementioned embodiments, the encoding device encodes the original speech signal to obtain an encoded bitstream, which includes an acoustic feature encoding result. The decoding device obtains the acoustic feature encoding result in the encoded bitstream, obtains style features from the acoustic feature encoding result, obtains excitation features, and performs style fusion processing using the excitation features and style features to obtain fused speech features. The decoded speech signal is reconstructed using these speech features. Because the style features indicate the speech style of the original speech signal, the speech features obtained by style fusion of the excitation features and style features can restore the speech features in the original speech signal. The decoded speech signal reconstructed based on the fused speech features can effectively improve the quality of the decoded speech signal compared to the decoded speech signal obtained without style fusion.

[0197] To facilitate a better understanding and implementation of the above solutions in the embodiments of the present application, the following examples are given for specific explanation using corresponding application scenarios.

[0198] The quality of voice communication in scenarios such as online conferencing, online education, and satellite communications is a key concern for users. The embodiment of the present application is aimed at such scenarios. Please refer to Figure 11, which is a voice codec system of the embodiment of the present application. The voice codec system mainly includes the following: an encoding device, a quantizer, an excitation network, a decoding device based on style fusion, and a discriminator. Among them, the discriminator is only used to distinguish between the original voice signal and the decoded voice signal during the model training process, so as to improve the training accuracy. It does not need to be configured in the voice codec system when the voice codec system actually decodes the voice signal.

[0199] The speech codec system uses the VQGAN (StyleVQGAN) coding framework based on style fusion. The speech codec system mainly includes:

[0200] The encoding device is used to downsample the original speech signal and ultimately extract the hidden layer acoustic representation.

[0201] The excitation network is used to generate an excitation signal through a speech generation model and convert it into an excitation feature through the network. The excitation feature serves as an input parameter of a decoding device based on style fusion.

[0202] The quantizer is used to compress the hidden layer acoustic representation obtained by the encoding device to obtain a quantized acoustic representation.

[0203] A decoding device based on style fusion, including: a style network and a generator;

[0204] The style network is used to upsample the quantized acoustic representation to obtain style features, which serve as conditional input to the generator;

[0205] The generator is used to synthesize the excitation features into the decoded speech signal under the guidance of the style features.

[0206] The speech coding and decoding system provided in the embodiment of the present application includes: an excitation network and a decoding device based on style fusion (Style-based decoder). Compared with the decoding device that only uses the quantized latent acoustic representation to reconstruct the speech signal, the embodiment of the present application uses an excitation network to generate an excitation feature containing prior knowledge of speech as the input parameter of the generator, and then uses the style network to extract the style features in the original speech signal from the latent acoustic representation. Finally, under the guidance of the style features, the generator synthesizes the excitation features into a decoded speech signal. At the lowest possible bit rate consumption, a high-quality speech signal is reconstructed to improve speech coding efficiency and user communication experience.

[0207] As shown in FIG11 , the encoding device, quantizer, excitation network, style fusion-based decoding device, and discriminator are respectively illustrated with examples.

[0208] 1. Encoding device

[0209] The main purpose of the encoding device is to map the input time domain signal x into the hidden acoustic representation z.

[0210] The main part of the encoding device is composed of multiple stacked encoder blocks. Each encoder block consists of a residual unit composed of two layers of one-dimensional convolution and a downsampling convolution layer. The speech signal is continuously downsampled through these encoder blocks, for example, with downsampling rates of 2, 4, 5, and 8, and the final output is an acoustic representation containing the speech information: z = E(x);

[0211] Unlike the encoding framework of VQGAN, the acoustic representation is not sent directly to the generator as input. Instead, it passes through the style network to obtain style features, which are then auxiliary input to the generator. The excitation features are then converted into decoded speech signals through a decoding device.

[0212] 2. Quantizer

[0213] The purpose of the quantizer is to compress the acoustic representation z output by the encoding device to a specific bit rate. The embodiment of the present application uses a residual vector quantizer (RVQ) to learn a set of codebooks to further quantize the acoustic representation output by the encoding device. In the following formula, Z represents the codebook. The quantization algorithm is to calculate the value of each element z in z. ij Find the code vector z that is closest to it in the code number k The initial value of this codebook is obtained by clustering, and then the value of the codebook is adjusted together through neural network training.

[0214] 3. Incentivizing the Network

[0215] As shown in Figure 6, the main goal of the excitation network is to generate an excitation feature as an input parameter of the generator. The excitation network used in this embodiment mainly includes an excitation signal generation model, a short-time Fourier transform (STFT) module and a first convolution module. The role of convolution is to obtain an excitation feature with the same dimension as the hidden acoustic feature. The excitation signal generation model can adopt a speech generation model, such as a sine wave-noise model, and use the hidden acoustic feature z q The unvoiced and voiced sound states are judged to generate corresponding signals, that is, Gaussian noise excitation is used as the excitation feature in the unvoiced part, and sine wave excitation is used as the excitation feature in the voiced part.

[0216] It is understandable that the dimension of the hidden layer acoustic features is determined by the encoding device. To facilitate subsequent style fusion operations, the feature dimension of the final output of the excitation network needs to be the same as the dimension of the hidden layer acoustic features.

[0217] 4. Decoding device based on style fusion

[0218] As shown in Figures 12a and 12b, a decoding device based on style fusion according to an embodiment of the present application is provided. The decoding device has multiple layers. For example, a decoding device having N layers is used as an example, and the value of N is not limited. Each layer of the decoding device includes a style network submodule and a generator module.

[0219] For example, as shown in FIG12a, the first layer of the decoding device includes a first style network module and a first generator module. The first style network module includes: a first upsampling module and a first convolution module. The input of the first upsampling module is the hidden layer acoustic feature z qThe output of the first convolution module is k0, which is input to the second style fusion module and the second upsampling module in the first generator module. The first generator module includes: a second upsampling module, a first style fusion module, and a first gating module. The input of the second upsampling module is the excitation feature y0, and the output of the first gating module is y'0. The first gating module inputs y'0 to the fourth upsampling module in the second generator module.

[0220] As shown in FIG12b, the Nth layer of the decoding device includes the Nth style network submodule and the Nth generator module. The Nth style network submodule includes: the (2N-1)th upsampling module and the Nth convolution module. The input of the (2N-1)th upsampling module is the hidden layer acoustic feature k N-1 , the output of the Nth convolutional module is k N , the Nth convolution module will k N Input to the (2N)th upsampling module in the Nth generator module. The Nth generator module includes: the (2N)th upsampling module, the Nth style fusion module and the Nth gating module. The input of the (2N)th upsampling module is the y output of the (N-1)th gating module. N-1 The output of the Nth gating module is y', which is input to the (N+1)th convolution module. The (N+1)th convolution module performs convolution on y' and outputs the decoded speech signal.

[0221] The decoding device based on style fusion consists of two parts: a style network and a generator. Its main function is to establish a mapping between the input excitation feature y and the target speech x under the guidance of the hidden acoustic features output by the encoding device as conditional information, and the decoded speech signal output by the decoding device is Satisfies the following relationship:

[0222] Among them, G stands for generator, E stands for decoder, and q stands for quantizer.

[0223] The decoding device includes N style network modules and N generator modules. One style network module and one generator module constitute a decoding submodule. For example, if N is 4, the upsampling rates of the four layers in the decoding device are 8, 5, 4, and 2, respectively.

[0224] The initial input of the style network module is the hidden layer acoustic feature z q , and generates style features through upsampling and convolution, and inputs them into the generator submodule.

[0225] The initial input of the generator module is the stimulus feature y. After corresponding upsampling, the two features are fused in the style fusion module (StyleBlock). The fusion algorithm used in the style fusion module is Temporal Adaptive DE-normalization (TADE), which fuses the input stimulus feature and style feature according to linear modulation. The specific implementation method is as follows:

[0226] First, the input excitation features are normalized, and then the style features are mapped into two modulation parameters γ and β through the convolutional network. Finally, the normalized excitation features are linearly modulated using the following formula: γ*y+β, where * represents a multiplication operation.

[0227] After that, the input of each layer of decoding submodule is the fused excitation features and the upsampled hidden layer acoustic features.

[0228] The output fusion signal y' of the decoding submodule of the last layer is passed through a convolution layer to obtain the decoded speech signal

[0229] 5. Discriminator

[0230] The embodiment of the present application adopts a multi-scale STFT discriminator, whose input is the real and imaginary parts of the time-frequency domain signal of the original speech signal and the decoded speech signal. By using different window lengths and the number of Fourier transform points, multiple STFT sub-discriminators of different scales are obtained.

[0231] This embodiment uses an end-to-end approach to train the speech codec system, which is implemented based on vector quantization (VQ)-generative adversarial network (GAN), where VQ is the vector quantization algorithm used by the quantizer, and GAN is a network architecture composed of a codec device and a discriminator. The network parameters of the encoding device and the decoding device in the system, as well as the codebook parameters, are optimized by minimizing the objective function. The objective function used includes the adversarial loss of the generative adversarial network, the difference between the feature maps output by the discriminator, and the quantization loss of RVQ. In specific application scenarios, only the encoding device, quantizer, and generator are used to implement an end-to-end speech coding process.

[0232] Next, the quality of the decoded speech signal of the speech coding and decoding system provided in the embodiment of the present application is tested. This embodiment is tested under an open source speech database (such as the LibriTTS dataset). The dataset is an English corpus of multiple speakers with a total of 585 hours of English data and a sampling rate of 24kHz. Among them, 100 hours of speech data are used for training and 300 speech data are used for testing. The tested bit rate is 6kbps, and the VQ-GAN speech coding system that does not use style features and excitation features is compared. The evaluation algorithms are perceptual evaluation of speech quality (PESQ) and short-time objective intelligibility (STOI). Among them, PESQ mainly evaluates the objective perceptual quality of speech, and STOI mainly focuses on the intelligibility of speech.

[0233] As shown in Table 1 below, the VQ-GAN system based on style fusion provided in the embodiment of the present application has higher objective scores under both evaluation indicators than the VQ-GAN system serving as the baseline. Therefore, in the embodiment of the present application, better benefits will be obtained by simultaneously utilizing the excitation features obtained by the prior knowledge of the excitation signal and the latent acoustic representation information of the original speech signal to reconstruct the decoded speech signal.

[0234] Table 1 Comparison of objective quality scores of coded speech in different systems

[0235] This embodiment adds an excitation network to the end-to-end neural coding framework and adopts a decoding device based on style fusion. The excitation features output by the excitation network provide the generator with prior knowledge of the speech model, and the decoding device based on style fusion better utilizes the hidden acoustic features of the speech signal to restore a higher quality decoded speech signal.

[0236] In the above-mentioned embodiment of the present application, the excitation signal generation model adopts a speech generation model, and the time-frequency domain features are obtained by the STFT module. However, the selection of the excitation signal can also be more diverse, for example, a Gaussian white noise generator, a constant sequence generator, or a time domain signal of a speech generation model (such as a sine wave-noise model).

[0237] As shown in Table 2, the inputs generated by different excitation signals affect the final results. The system performs best when the input is a sine wave-noise model with frequency-domain features, because the input signal contains the most useful information. However, compared to the baseline VQ-GAN system without style fusion in Table 1, the style fusion-based VQ-GAN system proposed in this embodiment of the application performs better under different excitation signals.

[0238] Table 2 Comparison of objective quality scores of coded speech under different excitation signals

[0239] As shown in the examples above, the excitation network uses a speech generation model to generate an excitation signal, and a convolutional network extracts the excitation features of the excitation signal as input to the decoding device. The decoding device based on style fusion uses the style network to upsample the quantized latent acoustic representation to obtain style features. These style features serve as conditional information to guide the generator in synthesizing the excitation features into a decoded speech signal.

[0240] The speech encoding and decoding system provided in the embodiments of the present application can adopt a VQ-GAN framework based on style fusion, adding an excitation network to the audio and video field. The excitation features generated by the excitation network use the prior knowledge of the speech model to better reconstruct the decoded speech signal. The decoding device based on style fusion uses the style features as conditions to guide the excitation features in the generator to synthesize the decoded speech signal. The embodiments of the present application can simultaneously utilize both the excitation signal and the quantized hidden layer representation, thereby obtaining higher quality decoded speech.

[0241] It should be noted that for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that this application is not limited by the order of the actions described, because according to this application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by this application.

[0242] In order to better implement the above-mentioned solutions of the embodiments of the present application, relevant devices for implementing the above-mentioned solutions are also provided below.

[0243] FIG13 is a schematic diagram illustrating the structure of an exemplary speech signal decoding device. The speech signal decoding device in FIG13 can be used to implement the speech signal decoding method of the aforementioned embodiment. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects of the corresponding method provided above, and will not be repeated here. The speech signal decoding device may include:

[0244] An acquisition module 1301 is configured to acquire an acoustic feature encoding result in an encoded bitstream;

[0245] A style feature acquisition module 1302 is configured to acquire a style feature from the acoustic feature encoding result, wherein the style feature is used to indicate the speech style of the original speech signal;

[0246] The style fusion module 1303 is configured to perform style fusion processing on the style feature and the excitation feature in response to the acquired excitation feature to obtain a fused speech feature;

[0247] The signal generating module 1304 is configured to generate a decoded speech signal according to the fused speech features.

[0248] In an example, FIG14 shows a schematic block diagram of a device 1400 according to an embodiment of the present application. The device 1400 may include: a processor 1401 and a transceiver / transceiver pin 1402 , and optionally, a memory 1403 .

[0249] The various components of the device 1400 are coupled together via a bus 1404, wherein the bus 1404 includes, in addition to a data bus, a power bus, a control bus, and a status signal bus. However, for the sake of clarity, all buses are referred to as bus 1404 in the figure.

[0250] Optionally, the memory 1403 may be used to store instructions in the aforementioned method embodiment. The processor 1401 may be used to execute the instructions in the memory 1403 and control the receiving pin to receive signals and the transmitting pin to send signals. Specifically, the processor 1401 executes the aforementioned voice signal decoding method.

[0251] The apparatus 1400 may be the electronic device or a chip of the electronic device in the above method embodiment.

[0252] Among them, all relevant contents of each step involved in the above method embodiment can be referred to the functional description of the corresponding functional module and will not be repeated here.

[0253] This embodiment further provides a chip comprising one or more interface circuits and one or more processors. The interface circuits are configured to receive signals from a memory of an electronic device and send signals to the processors, the signals comprising computer instructions stored in the memory. When the processors execute the computer instructions, the electronic device executes the method of the above embodiment. The interface circuits may refer to transceiver 1402 in FIG. 14 .

[0254] This embodiment further provides a computer-readable storage medium, which stores computer instructions. When the computer instructions are executed on an electronic device, the electronic device executes the above-mentioned related method steps to implement the speech signal decoding method in the above-mentioned embodiment.

[0255] This embodiment further provides a computer program product. When the computer program product is run on a computer, the computer is caused to execute the above-mentioned related steps to implement the speech signal decoding method in the above-mentioned embodiment.

[0256] In addition, an embodiment of the present application also provides a device, which can specifically be a chip, component or module, and the device may include a connected processor and memory; wherein the memory is used to store computer-executable instructions, and when the device is running, the processor can execute the computer-executable instructions stored in the memory to enable the chip to execute the speech signal decoding method in the above-mentioned method embodiments.

[0257] Among them, the electronic device, computer-readable storage medium, computer program product or chip provided in this embodiment are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods provided above, and will not be repeated here.

[0258] Through the description of the above implementation methods, technical personnel in the relevant field can understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0259] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of modules or units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0260] Units described as separate components may or may not be physically separate, and components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple places. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0261] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0262] Any content of each embodiment of this application, as well as any content of the same embodiment, can be freely combined. Any combination of the above content is within the scope of this application.

[0263] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to execute all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0264] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of this application, ordinary technicians in this field can also make many forms without departing from the purpose of this application and the scope of protection of the claims, all of which are within the protection of this application.

[0265] The steps of the method or algorithm described in conjunction with the disclosure of the embodiments of the present application can be implemented in a hardware manner, or can be implemented by a processor executing a software instruction. The software instruction can be composed of corresponding software modules, and the software module can be stored in a random access memory (Random Access Memory, RAM), a flash memory, a read-only memory (Read Only Memory, ROM), an erasable programmable read-only memory (Erasable Programmable ROM, EPROM), an electrically erasable programmable read-only memory (Electrically EPROM, EEPROM), a register, a hard disk, a mobile hard disk, a read-only compact disc (CD-ROM) or any other form of storage medium well known in the art. An exemplary storage medium is coupled to the processor so that the processor can read information from the storage medium and can write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an ASIC.

[0266] Those skilled in the art will appreciate that in one or more of the above examples, the functions described in the embodiments of the present application can be implemented using hardware, software, firmware, or any combination thereof. When implemented using software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium. Computer-readable media include computer-readable storage media and communication media, wherein communication media include any media that facilitates the transmission of computer programs from one place to another. The storage medium can be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0267] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of this application, ordinary technicians in this field can also make many forms without departing from the purpose of this application and the scope of protection of the claims, all of which are within the protection of this application.

Claims

1. A speech signal decoding method, characterized in that: The method comprises: Obtain the acoustic feature encoding result in the encoded bitstream; Acquiring a style feature from the acoustic feature encoding result, wherein the style feature is used to indicate the speech style of the original speech signal; In response to the acquired excitation feature, performing style fusion processing on the style feature and the excitation feature to obtain a fused speech feature; A decoded speech signal is generated according to the fused speech features.

2. The method according to claim 1, characterized in that The method further comprises: The excitation feature is obtained through an excitation network.

3. The method according to claim 2, characterized in that The excitation network includes: an excitation signal generation model and a first convolution module; The step of obtaining the excitation feature through an excitation network includes: Generate an excitation signal by using the excitation signal generation model; The excitation signal is convolved by the first convolution module to obtain the excitation feature.

4. The method according to claim 3, characterized in that The excitation network also includes: a short-time Fourier transform module; The obtaining of the excitation feature through the excitation network further includes: extracting time-frequency domain features from the excitation signal through the short-time Fourier transform module; The performing convolution processing on the excitation signal by using the first convolution module includes: performing convolution processing on the time-frequency domain features by using the first convolution module.

5. The method according to any one of claims 1 to 4, characterized in that The acquiring of style features from the acoustic feature encoding result comprises: The style feature is obtained from the acoustic feature encoding result through a style network.

6. The method according to claim 5, characterized in that The style network includes: a first upsampling module and a second convolution module; The obtaining the style feature from the acoustic feature encoding result through a style network includes: Upsampling the acoustic feature encoding result by the first upsampling module to obtain an acoustic feature upsampling result; The second convolution module performs convolution processing on the acoustic feature upsampling result to obtain the style feature.

7. The method according to claim 5, characterized in that The style network includes: a first upsampling module, a first convolutional layer and a second convolutional layer; The obtaining the style feature from the acoustic feature encoding result through a style network includes: Upsampling the acoustic feature encoding result by the first upsampling module to obtain an acoustic feature upsampling result; Inputting the acoustic feature upsampling result into the first convolution layer for convolution processing to obtain style sub-features; Inputting the style sub-feature into the first upsampling module for upsampling to obtain a style upsampling result; The style upsampling result is input into the second convolution layer for convolution processing to obtain the style feature.

8. The method according to any one of claims 1 to 7, characterized in that The performing style fusion processing on the style feature and the excitation feature to obtain a fused speech feature includes: The style feature and the excitation feature are respectively input into a generator, and the generator performs style fusion processing on the style feature and the excitation feature to obtain fused speech features.

9. The method according to claim 8, characterized in that The generator includes: a second upsampling module, a style fusion module, a gating module and a third convolution module; The generator performs style fusion processing on the style feature and the excitation feature to obtain a fused speech feature, including: Performing up-sampling processing on the excitation feature through the second up-sampling module to obtain a first signal feature; Linearly modulating the first signal feature and the style feature through the style fusion module to obtain a second signal feature, and processing the second signal feature through the gating module to obtain a third signal feature; The third signal feature is convolved by the third convolution module to obtain the fused speech feature.

10. The method according to claim 9, characterized in that The style fusion module includes: a first style fusion submodule and a second style fusion submodule; The gating module comprises: a first gating submodule and a second gating submodule; The linearly modulating the first signal feature and the style feature by the style fusion module to obtain a second signal feature, and processing the second signal feature by the gating module to obtain a third signal feature, includes: Linearly modulating the first signal feature and the style feature through the first style fusion submodule to obtain a first modulation result; Processing the first modulation result by the first gating submodule to obtain a first gating result; Linearly modulating the first gating result and the style feature through the second style fusion submodule to obtain a second modulation result; The second modulation result is processed by the second gating submodule to obtain the third signal feature.

11. The method according to any one of claims 1 to 10, characterized in that The method further comprises: The decoded speech signal and the original speech signal are input into a discriminator, and the decoded speech signal and the original speech signal are identified by the discriminator.

12. A speech signal decoding device, characterized in that: The device comprises: An acquisition module, used to obtain the acoustic feature encoding result in the encoding code stream; A style feature acquisition module, used to acquire style features from the acoustic feature encoding result, wherein the style features are used to indicate the speech style of the original speech signal; A style fusion module, configured to perform style fusion processing on the style feature and the excitation feature in response to the acquired excitation feature, so as to obtain a fused speech feature; The signal generating module is used to generate a decoded speech signal according to the fused speech features.

13. An electronic device, characterized in that: include: a memory and a processor, the memory being coupled to the processor; The memory stores program instructions, and when the program instructions are executed by the processor, the electronic device executes the speech signal decoding method according to any one of claims 1 to 11.

14. A chip, characterized in that: The invention comprises one or more interface circuits and one or more processors; the interface circuit is used to receive a signal from a memory of an electronic device and send the signal to the processor, wherein the signal includes a computer instruction stored in the memory; when the processor executes the computer instruction, the electronic device executes the speech signal decoding method according to any one of claims 1 to 11.

15. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program runs on a computer or a processor, the computer or the processor executes the speech signal decoding method according to any one of claims 1 to 11.

16. A computer program product, characterized in that The computer program product comprises a software program, and when the software program is executed by a computer or a processor, the steps of the method according to any one of claims 1 to 11 are performed.

Images