Voice signal decoding method and device and electronic equipment

Abstract

The invention provides a voice signal decoding method and device and electronic equipment, and is applied to the technical field of audio encoding and decoding, in the method, an encoding device encodes an original voice signal to obtain an encoded code stream, the encoded code stream comprises an acoustic feature encoding result, and a decoding device obtains the acoustic feature encoding result in the encoded code stream and sends the acoustic feature encoding result to the audio decoding device. And obtaining a style feature from the acoustic feature coding result, obtaining an excitation feature, carrying out style fusion processing by using the excitation feature and the style feature to obtain a fused voice feature, and reconstructing a decoded voice signal through the voice feature. Because the style feature indicates the voice style of the original voice signal, the voice feature obtained by carrying out style fusion on the excitation feature and the style feature can restore the voice feature in the original voice signal, and the decoded voice signal is reconstructed based on the fused voice feature. Compared with a decoded voice signal obtained without style fusion, the quality of the decoded voice signal can be effectively improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of audio decoding technology, and more particularly to a method and device for decoding a speech signal and an electronic device. Background Art

[0002] The quality of synthesized speech at low bit rates of traditional speech coding and decoding algorithms has reached a bottleneck period. Algorithms based on deep neural networks (DNNs) are applied to the field of speech signal processing, that is, there is a DNN-based speech coding and decoding system, the main goal of which is to use the powerful learning ability of DNN to improve the performance of traditional speech coding and decoding algorithms.

[0003] DNN-based speech codec systems can be mainly divided into two categories: neural vocoder-based systems and end-to-end neural encoder systems. Among them, the neural vocoder-based system uses an encoder to extract acoustic features and quantize them, and then uses a neural vocoder to perform speech synthesis on the quantized acoustic features to obtain an encoded speech signal, which has the advantage of low bit rate, usually as low as 1.6 kilobits per second (kbps). However, since the encoder and quantization steps directly reuse traditional encoding and decoding algorithms, big data training can only adjust the parameters of the decoder, so the quality of the decoded speech is limited.

[0004] In the neural encoder system, the encoder and decoder are all implemented by DNN, so there is no need to design a specific implementation plan for encoder feature extraction, and the network parameters are adjusted through end-to-end joint training. However, when the speech bit rate of the speech encoding output is low, the performance of the decoded speech signal is poor. Summary of the invention

[0005] The present application provides a speech signal decoding method and device and an electronic device.

[0006] In a first aspect, an embodiment of the present application provides a speech signal decoding method, the method comprising: obtaining an acoustic feature encoding result in a coded bitstream; obtaining a style feature from the acoustic feature encoding result, the style feature being used to indicate the speech style of the original speech signal; in response to the acquired excitation feature, performing style fusion processing on the style feature and the excitation feature to obtain a fused speech feature; and generating a decoded speech signal based on the fused speech feature.

[0007] In the above implementation scheme, the encoding device encodes the original speech signal to obtain an encoded code stream, which includes an acoustic feature encoding result, the decoding device obtains the acoustic feature encoding result in the encoded code stream, obtains the style feature from the acoustic feature encoding result, and obtains the excitation feature. The excitation feature and the style feature are used to perform style fusion processing to obtain a fused speech feature, and the decoded speech signal is reconstructed by the speech feature. Since the style feature indicates the speech style of the original speech signal, the speech feature obtained by style fusion of the excitation feature and the style feature can restore the speech feature in the original speech signal. The decoded speech signal reconstructed based on the fused speech feature can effectively improve the quality of the decoded speech signal compared to the decoded speech signal obtained without style fusion.

[0008] In a possible manner, the method further includes: obtaining the excitation feature through an excitation network. In the above implementation, the excitation network can generate the excitation feature, which solves the problem of obtaining the excitation feature.

[0009] In one possible manner, the excitation network includes: an excitation signal generation model and a first convolution module; the obtaining of the excitation feature through the excitation network includes: generating an excitation signal through the excitation signal generation model; and performing convolution processing on the excitation signal through the first convolution module to obtain the excitation feature. In the above implementation, the excitation feature can be generated through a simple device structure through the excitation signal generation model and the first convolution module, thereby simplifying the generation process of the excitation feature.

[0010] In one possible manner, the excitation network further includes: a short-time Fourier transform module; the obtaining of the excitation feature through the excitation network further includes: extracting time-frequency domain features from the excitation signal through the short-time Fourier transform module; the convolution processing of the excitation signal through the first convolution module includes: convolution processing of the time-frequency domain features through the first convolution module. In the above implementation, the first convolution module convolves the time-frequency domain features and outputs the excitation feature, which has the same dimension as the acoustic feature, thereby simplifying the calculation.

[0011] In one possible manner, the obtaining of style features from the acoustic feature encoding result includes: obtaining the style features from the acoustic feature encoding result through a style network. In the above implementation, the style network can obtain style features, and the decoding device can input the style feature network into a generator in the decoding device, and reconstruct the speech signal through the generator. The problem of obtaining style features is solved.

[0012] In one possible manner, the style network includes: a first upsampling module and a second convolution module; the obtaining of the style feature from the acoustic feature encoding result through the style network includes: upsampling the acoustic feature encoding result through the first upsampling module to obtain an acoustic feature upsampling result; and convolution processing the acoustic feature upsampling result through the second convolution module to obtain the style feature. In the above implementation, the first upsampling module and the second convolution module are used to generate the style feature through a simple device structure, thereby simplifying the generation process of the style feature.

[0013] In one possible manner, the style network includes: a first upsampling module, a first convolutional layer, and a second convolutional layer; the obtaining of the style feature from the acoustic feature encoding result through the style network includes: upsampling the acoustic feature encoding result through the first upsampling module to obtain an acoustic feature upsampling result; inputting the acoustic feature upsampling result into the first convolutional layer for convolution processing to obtain a style sub-feature; inputting the style sub-feature into the first upsampling module for upsampling to obtain a style upsampling result; inputting the style upsampling result into the second convolutional layer for convolution processing to obtain the style feature. In the above implementation, the first convolutional layer and the second convolutional layer can generate style features through a simple device structure, simplifying the style feature generation process and solving the problem of generating style features when multiple layers of convolutional modules are stacked.

[0014] In one possible manner, the style feature and the excitation feature are subjected to style fusion processing to obtain a fused speech feature, including: inputting the style feature and the excitation feature into a generator respectively, and performing style fusion processing on the style feature and the excitation feature by the generator to obtain a fused speech feature. In the above implementation scheme, the generator in the decoding device receives the style feature and the excitation feature respectively, and then the generator performs style fusion processing on the style feature and the excitation feature to obtain a fused speech feature. The fused speech feature carries the speech style of the original speech signal, which can improve the signal quality of the reconstructed speech signal.

[0015] In one possible manner, the generator includes: a second upsampling module, a style fusion module, a gating module and a third convolution module; the generator performs style fusion processing on the style feature and the excitation feature to obtain the fused speech feature, including: upsampling the excitation feature through the second upsampling module to obtain the first signal feature; linearly modulating the first signal feature and the style feature through the style fusion module to obtain the second signal feature, and processing the second signal feature through the gating module to obtain the third signal feature; and convolution processing the third signal feature through the third convolution module to obtain the fused speech feature. In the above implementation scheme, through the second upsampling module, the style fusion module, the gating module and the third convolution module, style fusion can be achieved through a simple device structure, which simplifies the style fusion process and solves the problem of the generator synthesizing the excitation feature into a high-quality decoded speech signal.

[0016] In a possible manner, the style fusion module includes: a first style fusion submodule and a second style fusion submodule; the gating module includes: a first gating submodule and a second gating submodule; the linear modulation of the first signal feature and the style feature by the style fusion module to obtain the second signal feature, and the processing of the second signal feature by the gating module to obtain the third signal feature, includes: linear modulation of the first signal feature and the style feature by the first style fusion submodule to obtain a first modulation result; processing the first modulation result by the first gating submodule to obtain a first gating result; linear modulation of the first gating result and the style feature by the second style fusion submodule to obtain a second modulation result; processing the second modulation result by the second gating submodule to obtain the third signal feature. In the above implementation scheme, the third signal feature can be generated by a simple device structure through the first style fusion submodule and the second style fusion submodule, the first gating submodule and the second gating submodule, which simplifies the signal feature generation process and solves the problem of generating the third signal feature when the style fusion module and the gating module are stacked in multiple layers.

[0017] In one possible manner, the method further includes: inputting the decoded speech signal and the original speech signal into a discriminator, and identifying the decoded speech signal and the original speech signal through the discriminator. In the above scheme, when the speech codec system adopts a generative adversarial network for model training, the discriminator is used to distinguish the original speech signal from the decoded speech signal during the training process to improve the training accuracy.

[0018] In a second aspect, an embodiment of the present application provides a speech signal decoding device, the device comprising:

[0019] An acquisition module, used to obtain the acoustic feature encoding result in the encoding code stream;

[0020] A style feature acquisition module, used to acquire style features from the acoustic feature encoding result, wherein the style features are used to indicate the speech style of the original speech signal;

[0021] A style fusion module, configured to perform style fusion processing on the style feature and the excitation feature in response to the acquired excitation feature, so as to obtain a fused speech feature;

[0022] The signal generating module is used to generate a decoded speech signal according to the fused speech features.

[0023] In the above implementation scheme, the encoding device encodes the original speech signal to obtain an encoded code stream, which includes an acoustic feature encoding result, the decoding device obtains the acoustic feature encoding result in the encoded code stream, obtains the style feature from the acoustic feature encoding result, and obtains the excitation feature. The excitation feature and the style feature are used to perform style fusion processing to obtain a fused speech feature, and the decoded speech signal is reconstructed by the speech feature. Since the style feature indicates the speech style of the original speech signal, the speech feature obtained by style fusion of the excitation feature and the style feature can restore the speech feature in the original speech signal. The decoded speech signal reconstructed based on the fused speech feature can effectively improve the quality of the decoded speech signal compared to the decoded speech signal obtained without style fusion.

[0024] The speech signal decoding device of the second aspect can execute the steps in the first aspect and any implementation method of the first aspect, which will not be described in detail here.

[0025] The second aspect and any implementation of the second aspect correspond to the first aspect and any implementation of the first aspect respectively. The technical effects corresponding to the second aspect and any implementation of the second aspect can refer to the technical effects corresponding to the above-mentioned first aspect and any implementation of the first aspect, which will not be repeated here.

[0026] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a memory and a processor, the memory being coupled to the processor; the memory storing program instructions, and when the program instructions are executed by the processor, the electronic device executes the speech signal decoding method in the first aspect or any possible implementation of the first aspect.

[0027] The third aspect and any implementation of the third aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the third aspect and any implementation of the third aspect can refer to the technical effects corresponding to the first aspect and any implementation of the first aspect, which will not be repeated here.

[0028] In a fourth aspect, an embodiment of the present application provides a chip comprising one or more interface circuits and one or more processors; the interface circuit is used to receive a signal from a memory of an electronic device and send a signal to the processor, the signal comprising a computer instruction stored in the memory; when the processor executes the computer instruction, the electronic device executes the speech signal decoding method in the first aspect or any possible implementation of the first aspect.

[0029] The fourth aspect and any implementation of the fourth aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the fourth aspect and any implementation of the fourth aspect can refer to the technical effects corresponding to the above-mentioned first aspect and any implementation of the first aspect, which will not be repeated here.

[0030] In a fifth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program runs on a computer or a processor, the computer or the processor executes the speech signal decoding method in the first aspect or any possible implementation of the first aspect.

[0031] The fifth aspect and any implementation of the fifth aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the fifth aspect and any implementation of the fifth aspect can refer to the technical effects corresponding to the first aspect and any implementation of the first aspect, which will not be repeated here.

[0032] In a sixth aspect, an embodiment of the present application provides a computer program product, which includes a software program. When the software program is executed by a computer or a processor, the computer or the processor executes the speech signal decoding method in the first aspect or any possible implementation of the first aspect.

[0033] The sixth aspect and any implementation of the sixth aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the sixth aspect and any implementation of the sixth aspect can refer to the technical effects corresponding to the first aspect and any implementation of the first aspect, which will not be repeated here.

[0034] In a seventh aspect, an embodiment of the present application provides a device for storing a code stream, the device comprising: a receiver and at least one storage medium, the receiver is used to receive the code stream; the at least one storage medium is used to store the code stream; the code stream is generated according to the first aspect and any one implementation method of the first aspect.

[0035] The seventh aspect and any implementation of the seventh aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the seventh aspect and any implementation of the seventh aspect can refer to the technical effects corresponding to the above-mentioned first aspect and any implementation of the first aspect, which will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1a is a schematic diagram of an application scenario shown as an example;

[0037] Figure 1b is a schematic diagram of an application scenario shown as an example;

[0038] Figure 2 The figure is a schematic diagram showing the structure of a speech coding and decoding system as an example;

[0039] Figure 3 is a schematic diagram of the structure of another speech coding and decoding system shown as an example;

[0040] Figure 4 is a schematic diagram of the structure of another speech coding and decoding system shown as an example;

[0041] Figure 5 A schematic diagram of a decoding process is shown as an example;

[0042] Figure 6 is a schematic diagram of the structure of an excitation network shown as an example;

[0043] Figure 7 is a schematic diagram of the structure of a style network shown as an example;

[0044] Figure 8 is a schematic diagram of the structure of a second convolution module shown as an example;

[0045] Fig. 9 is a schematic diagram of the structure of a generator shown as an example;

[0046] Fig.10 is a schematic diagram of the structure of a decoding device shown as an example;

[0047] Fig.11 is a schematic diagram of the structure of another speech coding and decoding system shown as an example;

[0048] Fig.12aSchematic diagram of the structure of a first style network module and a first generator module in a decoding device shown as an example;

[0049] Figure 12b is a structural schematic diagram of a structural schematic diagram of an N-th style network module and an N-th generator module in a decoding device shown as an example;

[0050] Fig.13 is a schematic structural diagram of an exemplary speech signal decoding device;

[0051] Fig.14 FIG. 4 is a schematic diagram of the structure of an exemplary speech signal decoding device. DETAILED DESCRIPTION

[0052] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0053] The term "and / or" in this article is merely a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone.

[0054] The terms "first" and "second" in the description and claims of the embodiments of the present application are used to distinguish different objects rather than to describe a specific order of objects. For example, a first target object and a second target object are used to distinguish different target objects rather than to describe a specific order of target objects.

[0055] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0056] In the description of the embodiments of the present application, unless otherwise specified, the meaning of "multiple" refers to two or more than two. For example, multiple processing units refer to two or more processing units; multiple systems refer to two or more systems.

[0057] In order to make the description of the following embodiments clear and concise, a brief introduction to the related technology is first given.

[0058] Sound is a continuous wave generated by the vibration of an object. The object that vibrates and emits sound waves is called a sound source. When sound waves propagate through a medium (such as air, solid or liquid), the auditory organs of humans or animals can perceive the sound.

[0059] The characteristics of sound waves include pitch, intensity and timbre. Pitch refers to the high or low pitch of a sound. Intensity refers to the size of a sound. Intensity can also be called loudness or volume. The unit of intensity is decibel (dB). Timbre is also called timbre quality.

[0060] The frequency of sound waves determines the pitch of the sound. The higher the frequency, the higher the pitch. The number of times an object vibrates in one second is called frequency, and the unit of frequency is hertz (Hz). The frequency of sound that the human ear can recognize is between 20Hz and 20,000Hz.

[0061] The amplitude of the sound wave determines the intensity of the sound. The greater the amplitude, the greater the intensity of the sound. The closer to the sound source, the greater the intensity of the sound.

[0062] The waveform of the sound wave determines the timbre, and the waveforms of the sound wave include square wave, sawtooth wave, sine wave and pulse wave.

[0063] According to the characteristics of sound waves, sounds can be divided into regular sounds and irregular sounds. Irregular sounds refer to sounds produced by irregular vibrations of sound sources. Irregular sounds are, for example, noises that affect people's work, study, and rest. Regular sounds refer to sounds produced by regular vibrations of sound sources. Regular sounds include speech and music. When sound is represented by electricity, regular sounds are analog signals that change continuously in the time and frequency domains. This analog signal can be called an audio signal. An audio signal is an information carrier that carries speech, music, and sound effects. For example, audio signals include speech signals. Since human hearing has the ability to distinguish the positional distribution of sound sources in space, when listeners hear sounds in space, in addition to being able to feel the pitch, intensity, and timbre of the sound, they can also feel the direction of the sound.

[0064] The speech signal involved in the embodiment of the present application may include a mono signal or a multi-channel signal. For example, the original speech signal acquired by the encoding device in the embodiment of the present application may include a scene audio signal, and the scene audio signal may refer to a signal used to describe the sound field; wherein the speech signal may include: a higher order ambisonics (HOA) signal (wherein the HOA signal may include a three-dimensional HOA signal and a two-dimensional HOA signal (also referred to as a planar HOA signal)) and a three-dimensional audio signal; the three-dimensional audio signal may refer to other audio signals in the speech signal except the HOA signal.

[0065] Acoustic features are hidden acoustic features obtained by mapping the original speech signal through the encoding device. The encoding device inputs the acoustic features into the quantizer, and compresses the acoustic features through the quantizer. The compression result generated by the quantizer is called the acoustic feature encoding result.

[0066] The style feature refers to the feature of the style information of the speech signal, for example, the style feature is used to indicate the speech style of the original speech signal. In one implementation, the style feature can reflect the phonetic feature, for example, the style feature can reflect at least one feature of the speech content, speech energy, speech timbre, etc. of the speech signal.

[0067] In one possible implementation, the style feature is generated by a style network. In the embodiment of the present application, the decoding device may include a style network, and the acoustic feature encoding result output by the quantizer may be sent to the decoding device. The acoustic feature encoding result is input into the style network included in the decoding device. The style network may include multiple convolutional layers, and the style feature is obtained by mapping the multiple convolutional layers. In the embodiment of the present application, the acoustic feature encoding result is no longer directly used as the input of the generator in the decoding device, but the input of the generator is a constant, and the constant input is converted into a target output using the style feature through the style network, and the target is a decoded speech signal.

[0068] The excitation feature is used to perform style fusion processing with the style feature to obtain the fused speech feature. The excitation feature can be generated by an excitation network, for example, the excitation network generates the excitation feature and then inputs it into the generator in the decoding device. For another example, the decoding device includes an excitation network and a generator, and the excitation network generates the excitation feature and then sends it to the generator.

[0069] Figure 1a FIG. 1 is a schematic diagram of an exemplary application scenario. Figure 1a The diagram shows the encoding and decoding scenario of a speech signal.

[0070] Reference Figure 1a , exemplarily, the first electronic device may include a first audio acquisition module, a first voice signal decoding module, a first channel encoding module, a first channel decoding module, a first voice signal decoding module and a first audio playback module. It should be understood that the first electronic device may include more Figure 1a The present application is not limited to more or fewer modules as shown.

[0071] Reference Figure 1a , exemplarily, the second electronic device may include a second audio acquisition module, a second voice signal decoding module, a second channel encoding module, a second channel decoding module, a second voice signal decoding module, and a second audio playback module. It should be understood that the second electronic device may include Figure 1aThe present application is not limited to more or fewer modules as shown.

[0072] Exemplarily, the first electronic device encodes and transmits a voice signal to the second electronic device, and the process of decoding and audio playback by the second electronic device can be as follows: the first audio acquisition module can perform audio acquisition and output the voice signal to the first voice signal decoding module. Then, the first voice signal decoding module can encode the voice signal and output the code stream to the first channel encoding module. After that, the first channel encoding module can channel encode the code stream and transmit the channel-encoded code stream to the second electronic device through a wireless or wired network communication device. Then, the second channel decoding module of the second electronic device can channel decode the received data to obtain the code stream and output the code stream to the second voice signal decoding module. Then, the second voice signal decoding module can decode the code stream to obtain a reconstructed voice signal; then the reconstructed voice signal is output to the second audio playback module, and the second audio playback module performs audio playback.

[0073] It should be understood that the process of the second electronic device encoding and transmitting the voice signal to the first electronic device, decoding and audio playback by the first electronic device is similar to the above-mentioned process of the first electronic device transmitting the voice signal to the second electronic device, and audio playback by the second electronic device, and will not be repeated here.

[0074] Exemplarily, the first electronic device and the second electronic device may include, but are not limited to: personal computers, computer workstations, smart phones, tablet computers, servers, smart cameras, smart cars or other types of cellular phones, media consumption devices, wearable devices, set-top boxes, game consoles, etc.

[0075] Exemplarily, the present application can be specifically applied to virtual reality (VR) / augmented reality (AR) scenarios. In one possible manner, the first electronic device is a server, and the second electronic device is a VR / AR device. In one possible manner, the second electronic device is a server, and the first electronic device is a VR / AR device.

[0076] Exemplarily, the first voice signal decoding module and the second voice signal decoding module may be voice signal decoders. The first voice signal decoding module and the second voice signal decoding module may be voice signal decoders.

[0077] Exemplarily, when a first electronic device encodes a voice signal and a second electronic device reconstructs the voice signal, the first electronic device may be referred to as an encoding end or encoding device, and the second electronic device may be referred to as a decoding end or decoding device. When a second electronic device encodes a voice signal and a first electronic device reconstructs the voice signal, the second electronic device may be referred to as an encoding end or encoding device, and the first electronic device may be referred to as a decoding end or decoding device.

[0078] Figure 1b FIG. 1 is a schematic diagram of an exemplary application scenario. Figure 1b The figure shows a transcoding scenario of a speech signal.

[0079] Reference Figure 1b (1) Exemplarily, a wireless or core network device may include: a channel decoding module, other audio decoding modules, a voice signal decoding module and a channel coding module. The wireless or core network device may be used for audio transcoding.

[0080] For example, Figure 1b A specific application scenario of (1) may be: when the first electronic device is not provided with a voice signal decoding module but only with other audio encoding modules; and the second electronic device is only provided with a voice signal decoding module but not with other audio decoding modules, in order to enable the second electronic device to decode and play back the voice signal encoded by the first electronic device using other audio encoding modules, wireless or core network equipment may be used for transcoding.

[0081] Specifically, the first electronic device uses other audio encoding modules to encode the voice signal to obtain a first code stream; and the first code stream is channel-encoded and sent to the wireless or core network device. Then, the channel decoding module of the wireless or core network device can perform channel decoding and output the first code stream obtained by channel decoding to other audio decoding modules. After that, other audio decoding modules decode the first code stream to obtain a voice signal and output the voice signal to the voice signal decoding module. Then, the voice signal decoding module can encode the voice signal to obtain a second code stream and output the second code stream to the channel encoding module, and the channel encoding module performs channel encoding on the second code stream and sends it to the second electronic device. In this way, the second electronic device can call the voice signal decoding module, decode the second code stream obtained by channel decoding, and obtain a reconstructed voice signal; the reconstructed voice signal can be subsequently played back for audio.

[0082] Reference Figure 1b (2) Exemplarily, the wireless or core network device may include: a channel decoding module, a voice signal decoding module, other audio encoding modules and a channel encoding module. The wireless or core network device may be used for audio transcoding.

[0083] For example, Figure 1b The specific application scenario of (2) may be: when the first electronic device is only provided with a voice signal decoding module but not other audio encoding modules; and the second electronic device is not provided with a voice signal decoding module but only other audio decoding modules, in order to enable the second electronic device to decode and play back the voice signal encoded by the voice signal decoding module of the first electronic device, wireless or core network equipment may be used for transcoding.

[0084] Specifically, the first electronic device uses a voice signal decoding module to encode the voice signal to obtain a first code stream; and the first code stream is channel-encoded and sent to a wireless or core network device. Then, the channel decoding module of the wireless or core network device can perform channel decoding and output the first code stream obtained by channel decoding to the voice signal decoding module. After that, the voice signal decoding module decodes the first code stream to obtain a voice signal and outputs the voice signal to other audio encoding modules. Then, other audio encoding modules can encode the voice signal to obtain a second code stream and output the second code stream to the channel encoding module, and the channel encoding module performs channel encoding on the second code stream and sends it to the second electronic device. In this way, the second electronic device can call other audio decoding modules to decode the second code stream obtained by channel decoding to obtain a reconstructed voice signal; the reconstructed voice signal can be subsequently played back for audio.

[0085] In an end-to-end neural codec system, there is a framework based on a vector quantization generative adversarial network (VQGAN). The speech coding system based on VQGAN outputs a low speech bit rate during speech coding, resulting in poor performance of the speech signal decoded by the decoding device. For example, the goal of the speech coding system based on VQGAN is to minimize the sample point error between the original speech signal and the decoded speech signal, and the problem of poor recovery of the decoded signal will occur at a low bit rate (less than 6kbps).

[0086] See also Figure 2As shown, it is a schematic diagram of the composition structure of a speech signal coding and decoding system provided in an embodiment of the present application. The speech signal coding and decoding system can specifically be a style-fusion-based vector quantization-generative adversarial network (StyleVQGAN) speech coding and decoding system. Compared with the aforementioned VQGAN framework, the speech signal coding and decoding system provided in an embodiment of the present application includes a decoding end that adopts a decoding device based on style fusion, which can use the quantized hidden acoustic features and excitation features to better reconstruct the decoded signal, and reconstruct the decoded speech signal as close to the original signal as possible in an adversarial generation manner.

[0087] like Figure 2 As shown, a speech signal encoding and decoding system provided in an embodiment of the present application includes:

[0088] The encoding device is used to map the original speech signal into the acoustic features of the hidden layer, wherein the original speech signal may include a time domain signal; the acoustic features are input into the quantizer;

[0089] A quantizer is used to compress the acoustic features to obtain a coded stream, which includes the acoustic feature compression result; and transmit the acoustic feature compression result in the coded stream to a decoding device through a transmission network;

[0090] A decoding device, used to obtain an acoustic feature encoding result in the encoded code stream, where the acoustic feature encoding result is obtained by extracting features from the original speech signal by the encoding device;

[0091] The decoding device is further used to obtain style features from the acoustic feature encoding result, where the style features are used to indicate the speech style of the original speech signal;

[0092] The decoding device is further used to perform style fusion processing on the style feature and the excitation feature in response to the acquired excitation feature to obtain a fused speech feature;

[0093] The decoding device is used to generate a decoded speech signal according to the fused speech features.

[0094] See also Figure 3 As shown, it is a schematic diagram of the composition structure of another speech signal encoding and decoding system provided by an embodiment of the present application, and the speech signal encoding and decoding system also includes: an excitation network and a style network. Specifically, the style network may belong to the decoding device, or the style network is a separate functional device independent of the decoding device, and the decoding device also includes a generator. The excitation network may belong to the decoding device, or the excitation network is a separate functional device independent of the decoding device.

[0095] The style network is used to obtain style features from the acoustic feature encoding results and input the style features into the generator;

[0096] The excitation network is used to obtain excitation features and input the excitation features into the generator;

[0097] For example, the excitation network may include a speech generation model, an excitation signal is generated by the speech generation model, and then the excitation feature is extracted from the excitation signal. Specifically, the excitation network may include a sine wave-noise model, an excitation signal is generated by the sine wave-noise model, and then the excitation feature is extracted from the excitation signal. For another example, the excitation network may include a Gaussian white noise generator and a constant sequence generator, an excitation signal is generated by the Gaussian white noise generator and the constant sequence generator, and then the excitation feature is extracted from the excitation signal.

[0098] The generator is used to synthesize the excitation features into a decoded speech signal under the guidance of the style features as conditions.

[0099] For example, the excitation feature generated by the excitation network is used as the input excitation of the generator, and the excitation feature includes: a time-frequency domain feature map. The generator uses the time-frequency domain feature map as a condition to synthesize the excitation feature into a decoded speech signal.

[0100] See also Figure 4 As shown, it is a schematic diagram of the composition structure of another speech signal encoding and decoding system provided in an embodiment of the present application. The speech signal encoding and decoding system also includes: a discriminator, which is connected to the decoding device. After the generator generates a decoded speech signal, the decoded speech signal is input into the discriminator.

[0101] The discriminator is used for receiving the decoded speech signal and the original speech signal, and identifying the decoded speech signal and the original speech signal.

[0102] For example, the discriminator may specifically be a multi-scale time-frequency domain discriminator.

[0103] In the embodiment of the present application, the decoding device and the discriminator reconstruct a decoded speech signal as close as possible to the original speech signal in an adversarial generation manner. The discriminator in the speech coding and decoding system is an optional device. After the model training of the speech coding and decoding system is completed, the discriminator can no longer be used to recognize the speech signal.

[0104] The encoding and decoding process of the speech signal is described below.

[0105] First, the speech encoding method on the encoding side provided in an embodiment of the present application is described.

[0106] The main purpose of the encoding device is to map the input time domain signal into a hidden acoustic representation. The encoding device downsamples the time domain signal and finally outputs an acoustic representation containing speech information. This acoustic representation is not sent to the generator as a direct input, but is passed through the style network to obtain style features, which are sent to the generator as auxiliary input.

[0107] After the encoding device maps the acoustic features, the acoustic features are input into a quantizer. The purpose of the quantizer is to compress the acoustic representation output by the encoding device to a specific bit rate to obtain an acoustic feature encoding result.

[0108] Next, the speech decoding method on the decoding side provided in an embodiment of the present application is described.

[0109] Figure 5 FIG. 4 is a schematic diagram showing an exemplary decoding process. Figure 5 The decoding process corresponding to the aforementioned encoding process mainly includes the following steps:

[0110] 501. Obtain the acoustic feature encoding result in the encoded bitstream.

[0111] The encoding device can extract features from the original speech signal to obtain an acoustic feature encoding result, the encoding device and the decoding device transmit a code stream, and the decoding device obtains the encoded code stream from the encoding device. It is not limited that the acoustic feature encoding result can also be obtained after being compressed by a quantizer.

[0112] In some embodiments of the present application, the main part of the encoder is composed of a plurality of encoder blocks (Encoder block) stacked together, and the decoding device performs a decoding process inverse to that of the encoder, and the decoding device may also include a plurality of decoding blocks. An example is given below:

[0113] The encoding device includes four encoding blocks, which downsample the original speech signal, and the downsampling rates are 2, 4, 5, and 8, respectively, and finally output a hidden acoustic representation containing speech information. The decoding device may include four decoding modules composed of a style network module and a generator module, and the upsampling rates of each decoding module are 8, 5, 4, and 2, respectively.

[0114] 502. Obtain style features from the acoustic feature encoding result, where the style features are used to indicate the speech style of the original speech signal.

[0115] After the decoding device obtains the acoustic feature encoding result from the encoded bit stream, it performs speech style extraction based on the acoustic feature encoding result to obtain a style feature. The style feature indicates the speech style of the original speech signal. The style feature can be used as an input condition of the generator of the decoding device to guide the generator to reconstruct the speech signal.

[0116] 503. In response to the acquired excitation feature, style fusion processing is performed on the style feature and the excitation feature to obtain a fused speech feature.

[0117] The decoding device may acquire the excitation feature, for example, the decoding device acquires the excitation feature before acquiring the style feature, or acquires the excitation feature after acquiring the style feature, or the decoding device acquires the style feature and the excitation feature at the same time.

[0118] In some embodiments of the present application, in addition to performing the aforementioned steps, the speech signal decoding method provided in the embodiments of the present application further includes the following steps:

[0119] A1. Obtain incentive features through incentive network.

[0120] like Figure 3 or Figure 4 As shown, the excitation network can generate excitation features, and the decoding device can receive the excitation features input by the excitation network into the decoding device, or the decoding device can include the excitation network, and the excitation network can generate excitation features, thereby solving the problem of obtaining excitation features.

[0121] In some embodiments of the present application, Figure 6 As shown, the excitation network includes: an excitation signal generation model and a first convolution (ConV) module;

[0122] The aforementioned step A1 obtains the excitation feature through the excitation network, including:

[0123] A11. Generate an excitation signal through an excitation signal generation model;

[0124] Among them, the excitation network includes an excitation signal generation model, which can generate an excitation signal. For example, the excitation signal generation model can include a Gaussian white noise generator, a constant sequence generator, or a traditional speech generation model, such as a pulse wave-noise model and a sine wave-noise model.

[0125] Taking the sine wave-noise model as an example of the excitation signal generation model, the acoustic characteristics are used to determine the state of unvoiced and voiced sounds, and the corresponding excitation signal is generated according to whether a frame of signal is unvoiced or voiced. Gaussian noise excitation is used as the excitation signal in the unvoiced part, and sine wave excitation is used as the excitation signal in the voiced part.

[0126] A12. Perform convolution processing on the excitation signal through the first convolution module to obtain an excitation feature.

[0127] The first convolution module inputs an excitation signal, performs convolution on the excitation signal, and outputs an excitation feature. The first convolution module is used to obtain an excitation feature of the same dimension as the acoustic feature.

[0128] In the embodiment of the present application, the excitation signal generation model and the first convolution module are used to generate excitation features through a simple device structure, thereby simplifying the generation process of the excitation features.

[0129] In some embodiments of the present application, Figure 6 As shown, the excitation network also includes: a short-time Fourier transform module (STFT);

[0130] Step A1 obtains the excitation feature through the excitation network, and also includes:

[0131] A13. Extract time-frequency domain features from the excitation signal through the short-time Fourier transform module.

[0132] The time-frequency domain features extracted from the excitation signal may be a time-frequency domain feature map. The time-frequency domain feature map generated by the excitation network is used as an input excitation of the generator.

[0133] Step A12 performs convolution processing on the excitation signal through a first convolution module, including:

[0134] A121. Perform convolution processing on the time-frequency domain features through the first convolution module.

[0135] Among them, the first convolution module convolves the time-frequency domain features and outputs excitation features, which have the same dimension as the acoustic features, thereby simplifying the calculation.

[0136] In some embodiments of the present application, step 502 of acquiring style features from the acoustic feature encoding result includes:

[0137] B1. Obtain style features from acoustic feature encoding results through style network.

[0138] like Figure 3 or Figure 4 As shown, the style network can obtain style features, and the decoding device can input the style feature network into the generator in the decoding device, and reconstruct the speech signal through the generator. The problem of obtaining style features is solved.

[0139] In some embodiments of the present application, Figure 7 As shown, the style network includes: a first upsampling module (Upsampler) and a second convolution module;

[0140] The aforementioned step B1 obtains style features from the acoustic feature encoding result through the style network, including:

[0141] B11. Upsampling the acoustic feature encoding result through a first upsampling module to obtain an acoustic feature upsampling result;

[0142] The acoustic feature encoding result may include quantized acoustic features, and the first upsampling module upsamples the acoustic feature encoding result to obtain an acoustic feature upsampling result. The upsampling operation performed by the first upsampling module corresponds to the downsampling performed by the encoding device.

[0143] B12. Perform convolution processing on the acoustic feature upsampling result through the second convolution module to obtain the style feature.

[0144] The first upsampling module inputs the acoustic feature upsampling result into the second convolution module, and the second convolution module performs convolution processing on the acoustic feature upsampling result, thereby generating a style feature, which indicates the speech style of the original speech signal. The style feature can also be called a speech style feature.

[0145] In the embodiment of the present application, through the first upsampling module and the second convolution module, style features can be generated through a simple device structure, which simplifies the process of generating style features.

[0146] In some embodiments of the present application, the second convolution module may include two or more convolution layers, wherein the input of the current layer is the output of the previous layer, and the last layer outputs the convolution result. Figure 7 As shown, the second convolution module includes: a first convolution layer and a second convolution layer.

[0147] In some embodiments of the present application, the style network includes: a first upsampling module, a first convolutional layer, and a second convolutional layer. For example, the style network includes: a first upsampling module (Upsampler) and a second convolutional module, such as Figure 8 As shown, the second convolution module includes a first convolution layer and a second convolution layer.

[0148] The aforementioned step B1 obtains style features from the acoustic feature encoding result through the style network, including:

[0149] B13. Upsampling the acoustic feature encoding result through the first upsampling module to obtain an acoustic feature upsampling result;

[0150] Among them, the implementation method of the above step B13 is similar to that of the above step B11, and will not be repeated here.

[0151] B14, input the acoustic feature upsampling result into the first convolution layer for convolution processing to obtain style sub-features;

[0152] Among them, the first convolution layer is the top layer in the second convolution module. The input of the first convolution layer is the acoustic feature upsampling result. The first convolution layer performs convolution processing on the acoustic feature upsampling result and outputs a style sub-feature. The style sub-feature represents the speech style of the original speech signal extracted by the first convolution layer.

[0153] B15. Input the style sub-feature into the first upsampling module for upsampling to obtain a style upsampling result.

[0154] The first convolution layer inputs the style sub-feature into the first upsampling module, and the first upsampling module continues to upsample the style sub-feature to obtain a style upsampling result, which represents the result of upsampling the style sub-feature. In addition, the first convolution layer inputs the style sub-feature into the generator.

[0155] It can be understood that the first upsampling module in step B11 and step B15 performs upsampling, which can be specifically performed by different layers included in the first upsampling module. Fig.10 As shown, the first upsampling module includes: a first upsampling layer and a second upsampling layer. The first upsampling layer executes the aforementioned step B11, and the second upsampling layer executes the aforementioned step B15.

[0156] B16. Input the style upsampling result into the second convolutional layer for convolution processing to obtain style features.

[0157] Among them, the first upsampling module inputs the style upsampling result into the second convolution layer, and the second convolution layer continues to perform convolution processing on the style upsampling result to obtain the style feature. In this example, the second convolution module includes two convolution layers as an example, and the second convolution layer is the last layer. At this time, the second convolution layer inputs the style sub-feature into the generator. It can be understood that if the second convolution module includes more layers, the second convolution layer still needs to send the style feature to the next layer.

[0158] In the embodiment of the present application, through the first convolution layer and the second convolution layer, style features can be generated through a simple device structure, which simplifies the style feature generation process and solves the problem of generating style features when multiple layers of convolution modules are stacked.

[0159] In some embodiments of the present application, Figure 3 or Figure 4 As shown, the aforementioned step 503 performs style fusion processing on the style feature and the excitation feature to obtain the fused speech feature, including:

[0160] C1. Input the style features and the excitation features into the generator respectively, and perform style fusion processing on the style features and the excitation features through the generator to obtain the fused speech features.

[0161] The decoding device includes a style network and a generator, and the decoding device can also be called a decoder based on style fusion. The generator in the decoding device receives the style feature and the excitation feature respectively, and then performs style fusion processing on the style feature and the excitation feature to obtain a fused speech feature. The fused speech feature carries the speech style of the original speech signal, which can improve the signal quality of the reconstructed speech signal.

[0162] In some embodiments of the present application, Fig. 9 As shown, the generator includes: a second upsampling module, a style fusion module (StyleBlock), a gating module (Gated Function) and a third convolution module;

[0163] Step C1 performs style fusion processing on the style feature and the excitation feature through the generator to obtain the fused speech feature, including:

[0164] C11. Perform upsampling processing on the excitation feature through a second upsampling module to obtain a first signal feature.

[0165] Among them, the excitation network inputs the excitation feature to the second upsampling module, and the second upsampling module upsamples the excitation feature to obtain the first signal feature. The upsampling operation performed by the second upsampling module corresponds to the downsampling performed by the encoding device.

[0166] It is understandable that the second upsampling module and the first upsampling module have the same number of sampling layers. Fig.10 As shown, the second upsampling module includes: a third upsampling layer and a fourth upsampling layer, and the third upsampling layer performs the aforementioned step C11.

[0167] C12. Linearly modulate the first signal feature and the style feature through a style fusion module to obtain a second signal feature, and process the second signal feature through a gating module to obtain a third signal feature.

[0168] The style network inputs the style feature into the style fusion module, the second upsampling module inputs the first signal feature into the style fusion module, and the style fusion module linearly modulates the first signal feature and the style feature to obtain the second signal feature. An example is given below, where the style fusion module uses the Temporal Adaptive DE-normalization (TADE) algorithm for style fusion, and the style fusion module uses the TADE algorithm to linearly modulate the first signal feature and the style feature to obtain the second signal feature.

[0169] After the style fusion module generates the second signal feature, the second signal feature is input into the gating module, and the gating module processes the second signal feature to obtain the third signal feature. The gating module can reduce the quantization noise in the second signal feature and improve the quality of the decoded speech signal.

[0170] C13. Perform convolution processing on the third signal feature through a third convolution module to obtain a fused speech feature.

[0171] After generating the third signal feature, the gating module inputs the third signal feature into the third convolution module, and the third convolution module performs convolution processing on the third signal feature to obtain a fused speech feature. The convolution processing is performed by the third convolution module so that the fused speech feature can have the same dimension as the acoustic feature of the original speech signal, thereby improving the quality of the decoded speech signal.

[0172] In the embodiment of the present application, through the second upsampling module, the style fusion module, the gating module and the third convolution module, style fusion can be achieved through a simple device structure, the processing process of style fusion is simplified, and the problem of the generator synthesizing the excitation features into a high-quality decoded speech signal is solved.

[0173] In some embodiments of the present application, the second upsampling module may include two upsampling layers or more upsampling layers, wherein the input of the current layer is the output of the previous layer, and the last layer outputs the upsampling result. Fig.10 As shown, the second upsampling module includes: a third upsampling layer and a fourth upsampling layer.

[0174] Similarly, the style fusion module included in the generator includes two or more style fusion sub-modules. In some embodiments of the present application, for example Fig.10 As shown, the style fusion module includes a first style fusion submodule and a second style fusion submodule for example description.

[0175] The gating module included in the generator includes two or more gating sub-modules. In some embodiments of the present application, for example Fig.10 As shown, the gating module includes a first gating submodule and a second gating submodule for example description.

[0176] based on Fig.10 The style network and generator shown in step C12 linearly modulate the first signal feature and the style feature through the style fusion module to obtain the second signal feature, and process the second signal feature through the gating module to obtain the third signal feature, including:

[0177] C121. Linearly modulate the first signal feature and the style feature through a first style fusion submodule to obtain a first modulation result.

[0178] The excitation network inputs the excitation feature to the third upsampling layer in the second upsampling module, the third upsampling layer performs upsampling processing on the excitation feature to obtain the first signal feature, the third upsampling layer inputs the first signal feature to the first style fusion submodule, and when the first style fusion submodule performs style fusion, the first signal feature and the style feature may be linearly modulated to obtain a first modulation result. The first style fusion submodule inputs the first modulation result to the first gating submodule.

[0179] like Fig.10 As shown, the style feature input into the first style fusion submodule is specifically the style sub-feature output by the first convolution layer in the second convolution module.

[0180] C122. Process the first modulation result through a first gating submodule to obtain a first gating result.

[0181] The first gating submodule processes the first modulation result to obtain a first gating result. The first gating submodule can reduce quantization noise in the first modulation result and improve the quality of the decoded speech signal.

[0182] After obtaining the first gating result, the first gating submodule inputs the first gating result to the fourth upsampling layer in the second upsampling module, and the fourth upsampling layer upsamples the first gating result to obtain the upsampled first gating result, and the fourth upsampling layer inputs the upsampled first gating result to the second style fusion submodule.

[0183] C123. Linearly modulate the first gating result and the style feature through a second style fusion submodule to obtain a second modulation result.

[0184] When the second style fusion submodule performs style fusion, it may fuse the first gating result and the style feature by linear modulation to obtain a second modulation result. The second style fusion submodule inputs the second modulation result into the second gating submodule.

[0185] like Fig.10 As shown, the style features input into the second style fusion submodule are specifically the style features output by the second convolutional layer in the second convolutional module.

[0186] C124. Process the second modulation result through a second gating submodule to obtain a third signal feature.

[0187] The second gating submodule processes the second modulation result to obtain a third signal feature. The second gating submodule can reduce the quantization noise in the second modulation result and improve the quality of the decoded speech signal.

[0188] In the embodiment of the present application, through the first style fusion sub-module and the second style fusion sub-module, the first gating sub-module and the second gating sub-module, the third signal feature can be generated through a simple device structure, which simplifies the signal feature generation process and solves the problem of generating the third signal feature when the style fusion module and the gating module are stacked in multiple layers.

[0189] 504. Generate a decoded speech signal according to the fused speech features.

[0190] After the decoding device generates the fused speech features through the above step 503, the decoding device can reconstruct the speech signal according to the fused speech features, thereby generating a decoded speech signal. In the embodiment of the present application, since the decoding side adopts a decoding device based on style fusion, the acoustic features and the excitation features can be used at the same time to better reconstruct the decoded speech signal, thereby improving the quality of the decoded speech signal.

[0191] In some embodiments of the present application, in addition to performing the aforementioned steps, the speech signal decoding method provided in the embodiments of the present application further includes the following steps:

[0192] D1. Input the decoded speech signal and the original speech signal into the discriminator, and use the discriminator to identify the decoded speech signal and the original speech signal.

[0193] See also Figure 4 As shown, the discriminator is connected to the decoding device, and after the generator generates a decoded speech signal, the decoded speech signal is input into the discriminator. The decoding device and the discriminator reconstruct a decoded speech signal as close to the original speech signal as possible in an adversarial generation manner. When the speech codec system uses a generative adversarial network for model training, the discriminator is used to distinguish the original speech signal from the decoded speech signal during the training process to improve the training accuracy. After the speech codec system completes the training, the speech codec system may not include the discriminator.

[0194] It can be seen from the examples of the above embodiments that the encoding device encodes the original speech signal to obtain a coded stream, which includes an acoustic feature coding result, the decoding device obtains the acoustic feature coding result in the coded stream, obtains the style feature from the acoustic feature coding result, and obtains the excitation feature. The excitation feature and the style feature are used to perform style fusion processing to obtain a fused speech feature, and the decoded speech signal is reconstructed by the speech feature. Since the style feature indicates the speech style of the original speech signal, the speech feature obtained by style fusion of the excitation feature and the style feature can restore the speech feature in the original speech signal. The decoded speech signal reconstructed based on the fused speech feature can effectively improve the quality of the decoded speech signal compared to the decoded speech signal obtained without style fusion.

[0195] In order to better understand and implement the above-mentioned solutions in the embodiments of the present application, the following examples are given for specific explanation by using corresponding application scenarios.

[0196] The quality of voice communication in scenarios such as online conferences, online education, and satellite communications is the focus of user attention. The embodiments of this application are aimed at such scenarios. Fig.11 As shown, a speech codec system of an embodiment of the present application is shown, and the speech codec system mainly includes the following: a coding device, a quantizer, an excitation network, a decoding device based on style fusion, and a discriminator. Among them, the discriminator is only used to distinguish the original speech signal from the decoded speech signal during the model training process to improve the training accuracy, and does not need to be configured in the speech codec system when the speech codec system actually decodes the speech signal.

[0197] The speech codec system adopts the VQGAN (StyleVQGAN) coding framework based on style fusion. The speech codec system mainly includes:

[0198] The encoding device is used to downsample the original speech signal and finally extract the hidden layer acoustic representation.

[0199] The excitation network is used to generate an excitation signal through a speech generation model and convert it into an excitation feature through the network. The excitation feature is used as an input parameter of a decoding device based on style fusion.

[0200] The quantizer is used to compress the hidden layer acoustic representation obtained by the encoding device to obtain a quantized acoustic representation.

[0201] A decoding device based on style fusion, comprising: a style network and a generator;

[0202] The style network is used to upsample the quantized acoustic representation to obtain style features, which are used as conditional inputs of the generator.

[0203] The generator is used to synthesize the excitation features into a decoded speech signal under the guidance of the style features.

[0204] The speech coding and decoding system provided in the embodiment of the present application includes: an excitation network and a decoding device based on style fusion (Style-based decoder). Compared with the decoding device that only uses the quantized latent acoustic representation to reconstruct the speech signal, the embodiment of the present application uses an excitation network to generate an excitation feature containing speech prior knowledge as an input parameter of the generator, and then uses the style network to extract the style features in the original speech signal from the latent acoustic representation. Finally, under the guidance of the style features, the excitation features are synthesized into a decoded speech signal through the generator. At the lowest possible bit rate consumption, a high-quality speech signal is reconstructed to improve speech coding efficiency and user communication experience.

[0205] like Fig.11 As shown, the above-mentioned encoding device, quantizer, excitation network, style fusion-based decoding device, and discriminator are respectively illustrated with examples.

[0206] 1. Encoding device

[0207] The main purpose of the encoding device is to map the input time domain signal x into a hidden acoustic representation z.

[0208] The main part of the encoding device is composed of multiple encoder blocks stacked together, each of which consists of a residual unit consisting of two layers of one-dimensional convolution and a downsampling convolution layer. The speech signal is continuously downsampled through these encoder blocks, for example, the downsampling rates are 2, 4, 5, and 8 in sequence, and the final output is an acoustic representation containing speech information:

[0209] z=E(x);

[0210] Different from the encoding framework of VQGAN, the acoustic representation is not sent to the generator as a direct input, but the style features are obtained after passing through the style network, which are then input into the generator as an auxiliary input, and then the excitation features are converted into decoded speech signals through the decoding device.

[0211] 2. Quantizer

[0212] The purpose of the quantizer is to compress the acoustic representation z output by the encoding device to a specific bit rate. The embodiment of the present application uses a residual vector quantizer (RVQ) to learn a set of codebooks to further quantize the acoustic representation output by the encoding device. In the following formula, Z represents the codebook. The quantization algorithm is to calculate the value of the codebook for each element z in z. ij Find the code vector z that is closest to it in the code number kThe initial value of this codebook is obtained by clustering, and then the value of the codebook is adjusted through neural network training.

[0213]

[0214] 3. Incentivizing the network

[0215] like Figure 6 As shown in FIG. 1 , the main goal of the excitation network is to generate an excitation feature as an input parameter of the generator. The excitation network used in this embodiment mainly includes an excitation signal generation model, a short-time Fourier transform (STFT) module and a first convolution module. The role of convolution is to obtain an excitation feature with the same dimension as the hidden acoustic feature. The excitation signal generation model can adopt a speech generation model, such as a sine wave-noise model, and use the hidden acoustic feature z q The state of unvoiced and voiced sounds is determined to generate corresponding signals, that is, Gaussian noise excitation is used as the excitation feature in the unvoiced part, and sinusoidal wave excitation is used as the excitation feature in the voiced part.

[0216] It is understandable that the dimension of the hidden layer acoustic features is determined by the encoding device. In order to facilitate the subsequent style fusion operation, the feature dimension of the final output of the excitation network needs to be the same as the dimension of the hidden layer acoustic features.

[0217] 4. Decoding device based on style fusion

[0218] like Fig.12a and 12b As shown, a decoding device based on style fusion provided in an embodiment of the present application is provided, and the decoding device has multiple layers. For example, the decoding device has N layers, and the value of N is not limited. Each layer of the decoding device includes a style network submodule and a generator module.

[0219] For example, Fig.12a As shown, the first layer of the decoding device includes a first style network module and a first generator module. The first style network module includes: a first upsampling module and a first convolution module. The input of the first upsampling module is the hidden acoustic feature z q , the output of the first convolutional module is k 0 , the first convolutional module converts k 0 Input to the second upsampling module in the second style fusion module and the first generator module. The first generator module includes: a second upsampling module, a first style fusion module and a first gating module. The input of the second upsampling module is the excitation feature y 0 , the output of the first gating module is y' 0 , the first gating module will y' 0Input to the fourth upsampling module in the second generator module.

[0220] Figure 12b As shown, the Nth layer of the decoding device includes the Nth style network submodule and the Nth generator module, and the Nth style network submodule includes: the (2N-1)th upsampling module and the Nth convolution module, and the input of the (2N-1)th upsampling module is the hidden layer acoustic feature k N-1 , the output of the Nth convolutional module is k N , the Nth convolution module will k N Input to the (2N)th upsampling module in the Nth generator module. The Nth generator module includes: the (2N)th upsampling module, the Nth style fusion module and the Nth gating module. The input of the (2N)th upsampling module is the y output of the (N-1)th gating module. N-1 The output of the Nth gating module is y', and the Nth gating module inputs y' into the (N+1)th convolution module. The (N+1)th convolution module performs convolution processing on y' and outputs the decoded speech signal

[0221] The decoding device based on style fusion includes two parts: style network and generator. Its main functions are: under the guidance of the hidden acoustic features output by the encoding device as conditional information, a mapping between the input excitation feature y and the target speech x is established, and the decoded speech signal output by the decoding device is Satisfies the following relationship:

[0222]

[0223] Among them, G stands for generator, E stands for decoder, and q stands for quantizer.

[0224] The decoding device includes N style network modules and N generator modules, and one style network module and one generator module constitute a decoding submodule. For example, N is equal to 4, and the upsampling rates of the four layers in the decoding device are 8, 5, 4, and 2 respectively.

[0225] The initial input of the style network module is the hidden acoustic feature z q , and generates style features through upsampling and convolution, and inputs them into the generator submodule.

[0226] The initial input of the generator module is the excitation feature y. After the corresponding upsampling, the two features are fused in the style fusion module (Style Block). The fusion algorithm used in the style fusion module is Temporal Adaptive DE-normalization (TADE), which fuses the input excitation feature and style feature according to linear modulation. The specific implementation method is as follows:

[0227] First, the input excitation features are normalized, and then the style features are mapped into two modulation parameters γ and β through a convolutional network. Finally, the normalized excitation features are linearly modulated using the following formula γ*y+β, where * represents a multiplication operation.

[0228] After that, the input of each layer of decoding submodule is the fused excitation features and the upsampled hidden acoustic features.

[0229] The output fusion signal y' of the decoding submodule of the last layer is passed through a convolution layer to obtain the decoded speech signal

[0230] 5. Discriminator

[0231] The embodiment of the present application adopts a multi-scale STFT discriminator, the input is the real part and imaginary part of the time-frequency domain signal of the original speech signal and the decoded speech signal, and multiple STFT sub-discriminators of different scales are obtained by adopting different window lengths and Fourier transform points.

[0232] This embodiment uses an end-to-end approach to train the speech codec system, which is implemented based on vector quantization (VQ)-generative adversarial network (GAN), where VQ is the vector quantization algorithm used by the quantizer, and GAN is a network architecture composed of a codec device and a discriminator. The network parameters of the encoding device and the decoding device in the system, as well as the codebook parameters are optimized by minimizing the objective function. The objective function used includes the adversarial loss of the generative adversarial network, the difference between the feature maps output by the discriminator, and the quantization loss of RVQ. In specific application scenarios, only the encoding device, quantizer, and generator will be used to achieve an end-to-end speech coding process.

[0233] Next, the quality of the decoded speech signal of the speech coding and decoding system provided in the embodiment of the present application is tested. This embodiment is tested under an open source speech database (such as the LibriTTS data set), which is an English corpus of multiple speakers, with a total of 585 hours of English data and a sampling rate of 24kHz. Among them, 100 hours of speech data are used for training, and the test data is 300 voices. The tested bit rate is 6kbps, and the VQ-GAN speech coding system without style features and excitation features is compared. The evaluation algorithms are perceptual evaluation of speech quality (PESQ) and short-time objective intelligibility (STOI), among which PESQ mainly evaluates the objective perceptual quality of speech, and STOI mainly focuses on the intelligibility of speech.

[0234] As shown in Table 1 below, the VQ-GAN system based on style fusion provided in the embodiment of the present application has higher objective scores under both evaluation indicators than the VQ-GAN system serving as the baseline. Therefore, in the embodiment of the present application, better benefits will be obtained by simultaneously utilizing the excitation features obtained by the prior knowledge of the excitation signal and the latent acoustic representation information of the original speech signal to reconstruct the decoded speech signal.

[0235] Table 1 Comparison of objective quality scores of coded speech in different systems

[0236]

[0237] This embodiment adds an excitation network to the end-to-end neural coding framework and adopts a decoding device based on style fusion. The excitation features output by the excitation network provide the generator with prior knowledge of the speech model, and the decoding device based on style fusion better utilizes the hidden acoustic features of the speech signal to restore a decoded speech signal with higher quality.

[0238] In the above-mentioned embodiment of the present application, the excitation signal generation model adopts the speech generation model, and the time-frequency domain features are obtained through the STFT module. The selection of the excitation signal can also be more diversified, such as using a Gaussian white noise generator, a constant sequence generator, or a time domain signal of a speech generation model (such as a sine wave-noise model).

[0239] As shown in Table 2, the input generated by different excitation signals will affect the final result. When the input is a sine wave-noise model with frequency domain features, the system performs best because the input signal contains the most useful information. However, compared with the VQ-GAN system without style fusion as the baseline in Table 1, the VQ-GAN system based on style fusion proposed in the embodiment of the present application has better results under different excitation signals.

[0240] Table 2 Comparison of objective quality scores of coded speech under different excitation signals

[0241]

[0242] From the above examples, it can be seen that the above excitation network uses the speech generation model to generate an excitation signal, and extracts the excitation features of the excitation signal through the convolutional network as the input of the decoding device. The decoding device based on style fusion uses the style network to upsample the quantized hidden acoustic representation to obtain the style features, and uses the style features as conditional information to guide the generator to synthesize the excitation features into a decoded speech signal.

[0243] The speech encoding and decoding system provided in the embodiment of the present application can adopt the VQ-GAN framework based on style fusion, add an excitation network in the audio and video field, and the excitation features generated by the excitation network use the prior knowledge of the speech model to better reconstruct the decoded speech signal, while the decoding device based on style fusion uses the style features as conditions to guide the excitation features to synthesize the decoded speech signal in the generator. The embodiment of the present application can simultaneously use the two parts of information, the excitation signal and the quantized hidden layer representation, to obtain a higher quality decoded speech.

[0244] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0245] In order to better implement the above-mentioned solution of the embodiment of the present application, relevant devices for implementing the above-mentioned solution are also provided below.

[0246] Fig.13 FIG. 4 is a schematic diagram of the structure of an exemplary speech signal decoding device. Fig.13 The speech signal decoding device in the embodiment can be used to execute the speech signal decoding method of the aforementioned embodiment. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects of the corresponding method provided above, which will not be repeated here. The speech signal decoding device may include:

[0247] The acquisition module 1301 is used to obtain the acoustic feature encoding result in the encoding bit stream;

[0248] A style feature acquisition module 1302, used to acquire style features from the acoustic feature encoding result, wherein the style features are used to indicate the speech style of the original speech signal;

[0249] A style fusion module 1303 is used to perform style fusion processing on the style feature and the excitation feature in response to the acquired excitation feature to obtain a fused speech feature;

[0250] The signal generating module 1304 is configured to generate a decoded speech signal according to the fused speech features.

[0251] In one example, Fig.14 A schematic block diagram of a device 1400 according to an embodiment of the present application is shown. The device 1400 may include: a processor 1401 and a transceiver / transceiver pin 1402 , and optionally, a memory 1403 .

[0252] The components of the device 1400 are coupled together via a bus 1404, wherein the bus 1404 includes a power bus, a control bus, and a status signal bus in addition to a data bus. However, for the sake of clarity, all buses are referred to as bus 1404 in the figure.

[0253] Optionally, the memory 1403 may be used to store instructions in the aforementioned method embodiment. The processor 1401 may be used to execute instructions in the memory 1403, and control the receiving pin to receive a signal, and control the sending pin to send a signal. Specifically, the processor 1401 executes the aforementioned voice signal decoding method.

[0254] The apparatus 1400 may be the electronic device or a chip of the electronic device in the above method embodiment.

[0255] Among them, all relevant contents of each step involved in the above method embodiment can be referred to the functional description of the corresponding functional module, and will not be repeated here.

[0256] This embodiment also provides a chip, which includes one or more interface circuits and one or more processors; the interface circuit is used to receive signals from the memory of the electronic device and send signals to the processor, the signals including computer instructions stored in the memory; when the processor executes the computer instructions, the electronic device executes the method in the above embodiment. The interface circuit may refer to Fig.14 The transceiver 1402 in FIG.

[0257] This embodiment further provides a computer-readable storage medium, in which computer instructions are stored. When the computer instructions are executed on an electronic device, the electronic device executes the above-mentioned related method steps to implement the speech signal decoding method in the above-mentioned embodiment.

[0258] This embodiment further provides a computer program product. When the computer program product is run on a computer, the computer is enabled to execute the above-mentioned related steps to implement the speech signal decoding method in the above-mentioned embodiment.

[0259] In addition, an embodiment of the present application also provides a device, which can specifically be a chip, component or module, and the device may include a connected processor and memory; wherein the memory is used to store computer-executable instructions, and when the device is running, the processor can execute the computer-executable instructions stored in the memory so that the chip executes the speech signal decoding method in the above-mentioned method embodiments.

[0260] Among them, the electronic device, computer-readable storage medium, computer program product or chip provided in this embodiment are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods provided above and will not be repeated here.

[0261] Through the description of the above implementation methods, technical personnel in the relevant field can understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0262] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0263] The units described as separate components may or may not be physically separated, and the components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple different places. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0264] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0265] Any content of each embodiment of the present application, as well as any content of the same embodiment, can be freely combined. Any combination of the above content is within the scope of the present application.

[0266] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium, including several instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to perform all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program code.

[0267] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

[0268] The steps of the method or algorithm described in conjunction with the disclosure of the embodiments of the present application can be implemented in a hardware manner, or can be implemented by a processor executing a software instruction. The software instruction can be composed of corresponding software modules, and the software module can be stored in a random access memory (Random Access Memory, RAM), a flash memory, a read-only memory (Read Only Memory, ROM), an erasable programmable read-only memory (Erasable Programmable ROM, EPROM), an electrically erasable programmable read-only memory (Electrically EPROM, EEPROM), a register, a hard disk, a mobile hard disk, a read-only compact disk (CD-ROM) or any other form of storage medium known in the art. An exemplary storage medium is coupled to a processor so that the processor can read information from the storage medium and can write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an ASIC.

[0269] Those skilled in the art should be aware that in one or more of the above examples, the functions described in the embodiments of the present application can be implemented with hardware, software, firmware, or any combination thereof. When implemented using software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium. Computer-readable media include computer-readable storage media and communication media, wherein the communication media include any media that facilitates the transmission of a computer program from one place to another. The storage medium can be any available medium that a general or special-purpose computer can access.

[0270] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A speech signal decoding method, characterized in that: The method comprises: Obtain the acoustic feature encoding result in the encoded bitstream; Acquiring a style feature from the acoustic feature encoding result, wherein the style feature is used to indicate the speech style of the original speech signal; In response to the acquired excitation feature, performing style fusion processing on the style feature and the excitation feature to obtain a fused speech feature; A decoded speech signal is generated according to the fused speech features.

2. The method according to claim 1, characterized in that The method further comprises: The excitation feature is obtained through an excitation network.

3. The method according to claim 2, characterized in that The excitation network includes: an excitation signal generation model and a first convolution module; The step of obtaining the excitation feature through an excitation network includes: Generate an excitation signal by using the excitation signal generation model; The excitation signal is convolved by the first convolution module to obtain the excitation feature.

4. The method according to claim 3, characterized in that The excitation network also includes: a short-time Fourier transform module; The obtaining of the excitation feature through the excitation network further includes: extracting time-frequency domain features from the excitation signal through the short-time Fourier transform module; The performing convolution processing on the excitation signal by using the first convolution module includes: performing convolution processing on the time-frequency domain features by using the first convolution module.

5. The method according to any one of claims 1 to 4, characterized in that The acquiring of style features from the acoustic feature encoding result comprises: The style feature is obtained from the acoustic feature encoding result through a style network.

6. The method according to claim 5, characterized in that The style network includes: a first upsampling module and a second convolution module; The obtaining the style feature from the acoustic feature encoding result through a style network includes: Upsampling the acoustic feature encoding result by the first upsampling module to obtain an acoustic feature upsampling result; The second convolution module performs convolution processing on the acoustic feature upsampling result to obtain the style feature.

7. The method according to claim 5, characterized in that The style network includes: a first upsampling module, a first convolutional layer and a second convolutional layer; The obtaining the style feature from the acoustic feature encoding result through a style network includes: Upsampling the acoustic feature encoding result by the first upsampling module to obtain an acoustic feature upsampling result; Inputting the acoustic feature upsampling result into the first convolution layer for convolution processing to obtain style sub-features; Inputting the style sub-feature into the first upsampling module for upsampling to obtain a style upsampling result; The style upsampling result is input into the second convolution layer for convolution processing to obtain the style feature.

8. The method according to any one of claims 1 to 7, characterized in that The performing style fusion processing on the style feature and the excitation feature to obtain a fused speech feature includes: The style feature and the excitation feature are respectively input into a generator, and the generator performs style fusion processing on the style feature and the excitation feature to obtain fused speech features.

9. The method according to claim 8, characterized in that The generator includes: a second upsampling module, a style fusion module, a gating module and a third convolution module; The generator performs style fusion processing on the style feature and the excitation feature to obtain a fused speech feature, including: Performing up-sampling processing on the excitation feature through the second up-sampling module to obtain a first signal feature; Linearly modulating the first signal feature and the style feature through the style fusion module to obtain a second signal feature, and processing the second signal feature through the gating module to obtain a third signal feature; The third signal feature is convolved by the third convolution module to obtain the fused speech feature.

10. The method according to claim 9, characterized in that The style fusion module includes: a first style fusion submodule and a second style fusion submodule; The gating module comprises: a first gating submodule and a second gating submodule; The linearly modulating the first signal feature and the style feature by the style fusion module to obtain a second signal feature, and processing the second signal feature by the gating module to obtain a third signal feature, includes: Linearly modulating the first signal feature and the style feature through the first style fusion submodule to obtain a first modulation result; Processing the first modulation result by the first gating submodule to obtain a first gating result; Linearly modulating the first gating result and the style feature through the second style fusion submodule to obtain a second modulation result; The second modulation result is processed by the second gating submodule to obtain the third signal feature.

11. The method according to any one of claims 1 to 10, characterized in that The method further comprises: The decoded speech signal and the original speech signal are input into a discriminator, and the decoded speech signal and the original speech signal are identified by the discriminator.

12. A speech signal decoding device, characterized in that: The device comprises: An acquisition module, used to obtain the acoustic feature encoding result in the encoding code stream; A style feature acquisition module, used to acquire style features from the acoustic feature encoding result, wherein the style features are used to indicate the speech style of the original speech signal; A style fusion module, configured to perform style fusion processing on the style feature and the excitation feature in response to the acquired excitation feature, so as to obtain a fused speech feature; The signal generating module is used to generate a decoded speech signal according to the fused speech features.

13. An electronic device, characterized in that: include: a memory and a processor, the memory being coupled to the processor; The memory stores program instructions, and when the program instructions are executed by the processor, the electronic device executes the speech signal decoding method according to any one of claims 1 to 11.

14. A chip, characterized in that: The invention comprises one or more interface circuits and one or more processors; the interface circuit is used to receive a signal from a memory of an electronic device and send the signal to the processor, wherein the signal includes a computer instruction stored in the memory; when the processor executes the computer instruction, the electronic device executes the speech signal decoding method according to any one of claims 1 to 11.

15. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program runs on a computer or a processor, the computer or the processor executes the speech signal decoding method according to any one of claims 1 to 11.

16. A computer program product, characterized in that The computer program product comprises a software program, and when the software program is executed by a computer or a processor, the steps of the method according to any one of claims 1 to 11 are performed.

Images