A voice communication system and method

Abstract

The application discloses a voice communication system and method, and relates to the technical field of voice communication.The system comprises a first electronic device and a second electronic device.The first electronic device is used for acquiring the waveform of a voice signal, extracting a first log amplitude spectrum and a first phase spectrum from the waveform of the voice signal through short-time Fourier transform, generating a continuous code according to the first log amplitude spectrum and the first phase spectrum, performing a discretization process on the continuous code to obtain an index vector, and sending the index vector to the second electronic device.The second electronic device is used for generating a quantization code according to the index vector, generating a second log amplitude spectrum and a second phase spectrum according to the quantization code, and restoring the second log amplitude spectrum and the second phase spectrum to the waveform of the voice signal through inverse short-time Fourier transform.Thus, the voice signal can be stored or transmitted at a low coding bit rate while the performance of the decoded voice signal is ensured to be good, thereby improving the efficiency and restoration degree of voice communication.

Description

Technical Field

[0001] This application relates to the field of voice communication technology, and in particular to a voice communication system and method. Background Technology

[0002] Currently, speech codec technology has been widely applied in fields such as voice communication. Speech codec technology refers to the technique of compressing speech signals into discrete codes and then reconstructing the speech signals using these discrete codes. Speech codecs can be implemented using speech codecs.

[0003] In related technologies, speech codecs are mainly divided into parametric codecs and waveform codecs. Parametric codecs typically decode and encode the characteristic parameters of the speech signal. Due to the short-time stationary nature of speech signals, parametric codecs have the advantage of lower encoding bit rates. However, they also suffer from lower quality decoded speech signals and sensitivity to noise. Waveform codecs typically encode and decode the waveform of the speech signal. Therefore, waveform codecs produce speech signals with high fidelity, but this requires higher encoding bit rates, thus increasing the storage and transmission costs of the speech signal.

[0004] Therefore, how to store or transmit voice signals with as few bits as possible (i.e., low coding bit rate) while ensuring good performance of the decoded voice signals has become an urgent technical problem to be solved. Summary of the Invention

[0005] This application provides a voice communication system and method that can ensure good performance of the decoded voice signal while storing or transmitting voice signals at a low coding bit rate.

[0006] This application discloses the following technical solution:

[0007] In a first aspect, this application provides a voice communication system, the system comprising: a first electronic device and a second electronic device;

[0008] The first electronic device is configured to acquire the waveform of a speech signal; extract a first logarithmic amplitude spectrum and a first phase spectrum from the waveform of the speech signal through a short-time Fourier transform; generate a continuous code based on the first logarithmic amplitude spectrum and the first phase spectrum; discretize the continuous code to obtain an index vector; and send the index vector to the second electronic device.

[0009] The second electronic device is configured to generate a quantization code based on the index vector; generate a second logarithmic amplitude spectrum and a second phase spectrum based on the quantization code; and recover the waveform of the speech signal by using an inverse short-time Fourier transform to the second logarithmic amplitude spectrum and the second phase spectrum.

[0010] Optionally, the first electronic device includes: an encoder module and a quantizer module;

[0011] The encoder module is used to acquire the waveform of the speech signal; extract a first logarithmic amplitude spectrum and a first phase spectrum from the waveform of the speech signal through short-time Fourier transform; generate a continuous code based on the first logarithmic amplitude spectrum and the first phase spectrum; and send the continuous code to the quantizer module.

[0012] The quantizer module is used to discretize the continuous code to obtain an index vector; and send the index vector to the second electronic device.

[0013] Optionally, the encoder module is specifically used to: encode the first logarithmic amplitude spectrum to obtain an amplitude code; encode the first phase spectrum to obtain a phase code; and concatenate the amplitude code and the phase code to generate a continuous code.

[0014] Optionally, the encoder module includes: an amplitude sub-encoder and a phase sub-encoder;

[0015] The amplitude sub-encoder is used to encode the first logarithmic amplitude spectrum to obtain an amplitude code;

[0016] The phase sub-encoder is used to encode the first phase spectrum to obtain a phase code.

[0017] Optionally, the second electronic device includes: a decoder module;

[0018] The decoder module is used to generate a quantization code based on the index vector; generate a second logarithmic amplitude spectrum and a second phase spectrum based on the quantization code; and recover the waveform of the speech signal by using an inverse short-time Fourier transform to the second logarithmic amplitude spectrum and the second phase spectrum.

[0019] Optionally, the second electronic device is also used for:

[0020] A waveform reconstruction model is generated based on the quantization code, the second logarithmic amplitude spectrum, the second phase spectrum, and the waveform of the speech signal. The waveform reconstruction model is a neural network model that generates the waveform of the speech signal through the quantization code.

[0021] Optionally, the second electronic device is further configured to update the waveform recovery model based on the value of one or more loss functions selected from the amplitude spectrum loss function, phase spectrum loss function, short-time spectrum loss function, and waveform loss function.

[0022] Optionally, the phase spectrum loss function is a linear combination of the instantaneous phase loss function, the group delay loss function, and the instantaneous angular frequency loss function; the short-time spectrum loss function is a linear combination of the real part loss function, the imaginary part loss function, and the short-time spectrum consistency loss function; and the waveform loss function is a linear combination of the loss function of the generative adversarial network, the feature matching loss function, and the Mel spectrum loss function.

[0023] Secondly, this application provides a voice communication method applied to a first electronic device, the method comprising:

[0024] Obtain the waveform of the speech signal;

[0025] The first logarithmic amplitude spectrum and the first phase spectrum are extracted from the waveform of the speech signal by using short-time Fourier transform.

[0026] A continuous code is generated based on the first logarithmic amplitude spectrum and the first phase spectrum;

[0027] The continuous code is discretized to obtain an index vector;

[0028] The index vector is sent to the second electronic device.

[0029] Thirdly, this application provides a voice communication method applied to a second electronic device, the method comprising:

[0030] Receive the index vector sent by the first electronic device;

[0031] Based on the index vector, generate quantization codes;

[0032] Based on the quantization code, a second logarithmic amplitude spectrum and a second phase spectrum are generated;

[0033] The second logarithmic amplitude spectrum and the second phase spectrum are recovered into the waveform of the speech signal by inverse short-time Fourier transform.

[0034] Compared with the prior art, this application has the following beneficial effects:

[0035] This application provides a voice communication system and method. The system includes: a first electronic device and a second electronic device. The first electronic device is used to acquire the waveform of a voice signal; extract a first logarithmic amplitude spectrum and a first phase spectrum from the waveform of the voice signal through a short-time Fourier transform; generate a continuous code based on the first logarithmic amplitude spectrum and the first phase spectrum; discretize the continuous code to obtain an index vector; and send the index vector to the second electronic device. The second electronic device is used to generate a quantization code based on the index vector; generate a second logarithmic amplitude spectrum and a second phase spectrum based on the quantization code; and recover the waveform of the voice signal from the second logarithmic amplitude spectrum and the second phase spectrum through an inverse short-time Fourier transform. Thus, by using a short-time Fourier transform, the waveform of a voice signal with high fidelity is converted into a voice amplitude spectrum and a phase spectrum, and then the voice amplitude spectrum and phase spectrum are used as voice parameter features for parallel encoding. Similarly, the encoded quantized code can be decoded in parallel, and the decoded speech amplitude spectrum and phase spectrum can be restored to the waveform of the speech signal through inverse short-time Fourier transform. While storing or transmitting the speech signal at a low encoding bit rate, the performance of the decoded speech signal is guaranteed to be good, thereby improving the efficiency and fidelity of speech communication. Attached Figure Description

[0036] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0037] Figure 1 A schematic diagram of a voice communication system provided in an embodiment of this application;

[0038] Figure 2 A signaling diagram of a voice communication system provided in an embodiment of this application;

[0039] Figure 3 A flowchart illustrating a voice communication method provided in an embodiment of this application;

[0040] Figure 4 A flowchart of another voice communication method provided in an embodiment of this application. Detailed Implementation

[0041] As described earlier, speech codec technology has been widely applied in fields such as voice communication and signal processing. Speech codec technology refers to the technique of compressing speech signals into discrete codes and then reconstructing the speech signals using these discrete codes. Speech codecs can be implemented using speech codecs.

[0042] In related technologies, speech codecs are generally divided into three categories: parametric codecs, waveform codecs, and hybrid codecs.

[0043] Parametric codecs typically decode and encode the feature parameters of speech signals, such as Linear Predictive Coding (LPC). Due to the short-time stationary nature of speech signals, parametric codecs have the advantage of low bit rates. However, they also suffer from poorer decoded speech signal quality and sensitivity to noise. With the development of deep learning, neural network vocoders can be used to further improve the quality of decoded speech signals. These vocoders convert the encoded feature parameters into speech waveforms through a neural network model, thus achieving decoding. However, there is still room for improvement in the quality of speech signals decoded by parametric codecs.

[0044] Waveform codecs typically encode and decode the waveform of a speech signal. Therefore, the speech signal decoded by a waveform codec has the advantage of high fidelity. However, this also requires a higher encoding bit rate, which increases the storage and transmission cost of the speech signal.

[0045] With the development of deep learning, some end-to-end neural network codecs have been proposed, which can effectively balance decoded speech quality and coding bit rate. For example, the SoundStream codec employs residual vector quantization to reduce the coding bit rate while using generative adversarial loss proposed by the HiFi-GAN vocoder to ensure the quality of the decoded speech signal. However, because these end-to-end neural network codecs directly encode and decode waveforms, they require upsampling and downsampling operations of hundreds of times, which not only makes modeling the neural network model extremely difficult but also potentially reduces the efficiency of encoding and decoding.

[0046] Hybrid codecs, such as the Regular-Pulse Excited Linear Prediction Codec, combine the advantages of parametric codecs and waveform codecs. However, research on implementing hybrid codecs using neural network models is still lacking.

[0047] Therefore, how to store or transmit speech signals with as few bits as possible (i.e., low coding bit rate) while ensuring that the performance of the decoded speech signal does not degrade significantly has become an urgent technical problem to be solved.

[0048] In view of this, this application provides a voice communication system and method. The system includes: a first electronic device and a second electronic device. The first electronic device is used to acquire the waveform of a voice signal; extract a first logarithmic amplitude spectrum and a first phase spectrum from the waveform of the voice signal through a short-time Fourier transform; generate a continuous code based on the first logarithmic amplitude spectrum and the first phase spectrum; discretize the continuous code to obtain an index vector; and send the index vector to the second electronic device. The second electronic device is used to generate a quantization code based on the index vector; generate a second logarithmic amplitude spectrum and a second phase spectrum based on the quantization code; and recover the waveform of the voice signal from the second logarithmic amplitude spectrum and the second phase spectrum through an inverse short-time Fourier transform. Thus, by using a short-time Fourier transform, the waveform of a voice signal with high fidelity is converted into a voice amplitude spectrum and a phase spectrum, and then the voice amplitude spectrum and phase spectrum are used as voice parameter features for parallel encoding. Similarly, the encoded quantized code can be decoded in parallel, and the decoded speech amplitude spectrum and phase spectrum can be restored to the waveform of the speech signal through inverse short-time Fourier transform. While storing or transmitting the speech signal at a low encoding bit rate, the performance of the decoded speech signal is guaranteed to be good, thereby improving the efficiency and fidelity of speech communication.

[0049] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application.

[0050] See Figure 1 This figure is a schematic diagram of a voice communication system provided in an embodiment of this application. Figure 1 It is known that the voice communication system 10 includes a first electronic device 11 and a second electronic device 12. The first electronic device 11 includes an encoder module 101 and a quantizer module 102; the second electronic device 12 includes a decoder module 103.

[0051] The encoder module 101 consists of an amplitude sub-encoder and a phase sub-encoder. Specifically, both the amplitude sub-encoder and the phase sub-encoder consist of an input convolutional layer, a ConvNeXt network, and a downsampling convolutional layer. The ConvNeXt network is a feature processing structure composed of convolutional layers, feedforward layers, Gaussian error linear units, and residual connections. The downsampling convolutional layer is used to further amplify the frame shift of the encoded features, thereby reducing the encoding bit rate.

[0052] The quantizer module 102 employs a residual vector quantization strategy. It consists of multiple vector quantizers connected in a residual manner to discretize continuous codes and generate quantized codes. It should be noted that this application does not limit the specific number of quantizers.

[0053] The decoder module 103 consists of an amplitude sub-decoder and a phase sub-decoder. Specifically, the amplitude sub-decoder consists of an upsampled deconvolutional layer, a ConvNeXt network, and an output convolutional layer. The phase sub-decoder differs from the amplitude sub-decoder in that it replaces the output convolutional layer with a phase parallel estimation architecture. This architecture consists of two parallel output convolutional layers and a phase calculation formula. This phase parallel estimation architecture can simulate the calculation process from the real and imaginary parts of the short-time complex spectrum to the phase spectrum, strictly limiting the predicted phase value to the principal phase value range, and achieving direct prediction of the coiled phase.

[0054] See Figure 2 This figure is a signaling diagram of a voice communication system provided in an embodiment of this application.

[0055] S201: The encoder module acquires the waveform of the voice signal.

[0056] First, the encoder module 101 of the first electronic device 11 needs to acquire the waveform of the voice signal. Where x is the waveform, Let T be the set of real numbers, and T be the length of the waveform points.

[0057] S202: The encoder module extracts the first logarithmic amplitude spectrum and the first phase spectrum from the waveform of the speech signal through short-time Fourier transform.

[0058] The Short-Time Fourier Transform (STFT) is a mathematical calculation used to determine the frequency and phase of a local sinusoidal wave in a time-varying signal. Through the STFT, the waveform of a speech signal can be analyzed. Extract the corresponding first logarithmic amplitude spectrum and the first phase spectrum

[0059] Where A is the first logarithmic amplitude spectrum. Let F be the set of real numbers, F be the number of frames, N be the number of frequency points, and P be the first phase spectrum. It can be understood that both the first logarithmic amplitude spectrum A and the first phase spectrum P are F×N matrices, with all elements being real numbers.

[0060] In some examples, a short-time Fourier transform can be performed on the waveform of a speech signal with a sampling rate of 16 kHz (kilohertz) to obtain a first logarithmic amplitude spectrum and a first phase spectrum with a sampling rate of 400 Hz. It should be noted that this application does not limit the specific sampling rate.

[0061] S203: The encoder module generates a continuous code based on the first logarithmic amplitude spectrum and the first phase spectrum.

[0062] The encoder module 101 includes an amplitude sub-encoder and a phase sub-encoder. The amplitude sub-encoder and the phase sub-encoder are parallel and have the same structure.

[0063] Specifically, the amplitude sub-encoder and the phase sub-encoder respectively use the first logarithmic amplitude spectrum and the first phase spectrum As input, parallel to the first logarithmic amplitude spectrum and the first phase spectrum Encoding is performed to generate amplitude and phase codes separately. Then, the generated amplitude and phase codes are concatenated and fused, and passed through a dimensionality-reducing convolutional layer to generate a long-frame-shift, low-dimensional continuous code. Where C represents a consecutive code, F is the set of real numbers. c N is the number of frames in a continuous code. c is the dimension of the continuous code.

[0064] In some examples, the sampling rate of the continuous code generated based on the logarithmic amplitude spectrum and the phase spectrum with a sampling rate of 400 Hz can be 50 Hz.

[0065] It should be noted that both the amplitude sub-encoder and the phase sub-encoder described above consist of an input convolutional layer, a ConvNeXt network, and a downsampling convolutional layer. Therefore, the first logarithmic amplitude spectrum of the input... and the first phase spectrum First, the input is normalized using a convolutional layer with C channels. Then, it undergoes depth processing using a ConvNeXt network. Finally, it is passed through a downsampling convolutional layer with C / 2 channels and a stride of D to reduce the dimensionality of the ConvNeXt network's output to half its original size, ultimately generating an amplitude-continuous code / phase-continuous code.

[0066] More specifically, the ConvNeXt network consists of multiple cascaded identical ConvNeXt blocks. Within each ConvNeXt block, the normalized logarithmic magnitude and phase spectra output from the input convolutional layer are sequentially passed through a linear convolutional layer with C channels and a layer with C nodes. HThe implementation maps features to a higher-dimensional feedforward layer (C). H >C), a Gaussian error linear unit (GELU) and a feedforward layer with C nodes that maps features to the original low-dimensional layer, and the output of the feedforward layer is added to the input (i.e., residual connection) as the final output of the ConvNeXt block.

[0067] S204: The encoder module sends the continuous code to the quantizer module.

[0068] S205: The quantizer module discretizes the continuous code to obtain the index vector.

[0069] Quantizer module 102 converts the continuous code generated by encoder module 101 into a quantizer module. After discretization, the index vector m can be obtained. 1 ,m 2 ,…,m Q .

[0070] Specifically, the quantizer module 102 employs a residual vector quantization (RVQ) strategy, consisting of Q vector quantizers (VQ), each with a trainable codebook. Among them, B q For codebook, N c Let M be the dimension of the continuous code, and M be the number of vectors in the codebook. The index vector m... 1 ,m 2 ,…,m Q It is then stored or transmitted in binary form. Therefore, the encoding bit rate (in kilobits per second, i.e., kbps) of the codec proposed in this application can be calculated by the following formula (1):

[0071]

[0072] Where Bitrate is the encoding bit rate, f s w is the sampling rate of the speech signal waveform. s The frame shift is used to extract the first logarithmic amplitude spectrum and the first phase spectrum from the waveform of the speech signal. D is the downsampling factor of the downsampling convolutional layer, Q is the number of vector quantizers, and M is the number of vectors in the codebook.

[0073] S206: The quantizer module sends the index vector to the decoder module.

[0074] S207: The decoder module generates quantized codes based on the index vector.

[0075] The decoder module 103 of the second electronic device 12 can be based on the index vector m 1 ,m 2 ,…,m Q Further generate quantization codes

[0076] S208: The decoder module generates a second logarithmic amplitude spectrum and a second phase spectrum based on the quantization code.

[0077] Quantization code First, the original dimension is restored by passing the data through an up-dimensional convolutional layer with C / 2 channels. Then, the data is input into the amplitude sub-decoder and the phase sub-decoder, which decode the second logarithmic amplitude spectrum in parallel. Second phase spectrum

[0078] Specifically, for the amplitude decoder module, the quantized code of the original input dimension is first upsampled by a factor of D through an upsampling deconvolution layer with C channels and a stride of D, then processed by a ConvNeXt network depthwise, and finally predicted by an output convolutional layer with N channels to obtain the second logarithmic amplitude spectrum.

[0079] The only difference between the phase sub-decoder and the amplitude sub-encoder is that the phase sub-decoder's output is a phase-parallel estimation architecture. This architecture guarantees direct prediction of the wound phase spectrum and consists of two parallel output linear convolutional layers, each with N channels, and a phase calculation formula Φ. It can be understood that the outputs of the two output convolutional layers are... and Then the second phase spectrum Need to pass calculate.

[0080] It should be noted that the function Φ is calculated element by element. For example, the formula for the function Φ can be shown in formulas (2) and (3) below:

[0081]

[0082] Where R is the first output of the first output convolutional layer, and I is the second output of the second output convolutional layer. When x≥0, Sgn * (x) = 1; when x < 0, Sgn * (x) = -1.

[0083] S209: The decoder module recovers the waveform of the speech signal from the second logarithmic amplitude spectrum and the second phase spectrum through inverse short-time Fourier transform.

[0084] After obtaining the second logarithmic amplitude spectrum Second phase spectrum The waveform of the speech signal can then be reconstructed using the Inverse Short-Time Fourier Transform (ISTFT). It should be noted that the waveform of the speech signal here... It is the same as the original speech signal waveform. Waveforms that are highly similar or completely identical.

[0085] In other specific implementations, the waveform of the speech signal can be transformed not only through steps S202-S209 as described above. Restore to It can also be obtained through quantization codes, first logarithmic amplitude spectrum, second logarithmic amplitude spectrum, first phase spectrum, second phase spectrum, and the waveform of the speech signal. and A waveform reconstruction model is generated. This waveform reconstruction model is a neural network model used to reconstruct the waveform of a speech signal. After the waveform reconstruction model is constructed, the waveform of the corresponding speech signal can be obtained by directly inputting the quantization code into the waveform reconstruction model. Furthermore, loss functions can be constructed based on the second logarithmic amplitude spectrum, the second phase spectrum, and the waveform of the speech signal to optimize and update the waveform reconstruction model. It should be noted that the waveform reconstruction model can be included not only in the second electronic device 12 but also in the first electronic device 11. This application does not limit the specific location of the waveform reconstruction model.

[0086] In some examples, the loss function described above may include the amplitude spectrum loss function. The amplitude spectrum loss function Defined as the second logarithmic amplitude spectrum The mean square error loss between the first logarithmic amplitude spectrum A and the second logarithmic amplitude spectrum is used to reduce the decoded second logarithmic amplitude spectrum. The difference between the first logarithmic amplitude spectrum A and the first logarithmic amplitude spectrum A. Specifically, the amplitude spectrum loss function. The formula can be shown in formula (4) below:

[0087]

[0088] in, For amplitude spectrum loss function, To take the average, A is the second logarithmic amplitude spectrum, and A is the first logarithmic amplitude spectrum.

[0089] It should be noted that when calculating the first logarithmic amplitude spectrum A, it is necessary to first start with the waveform of the original speech signal. The natural short-time complex spectrum S is extracted by short-time Fourier transform, and then calculated by the following equation (5):

[0090]

[0091] Where A is the first logarithmic amplitude spectrum, Re and Im represent taking the real part and taking the imaginary part, respectively, and S is the short-time complex spectrum.

[0092] In other examples, the loss function described above may include a phase spectrum loss function. The phase spectrum loss function Defined as the second phase spectrum The loss between the first phase spectrum P and the second phase spectrum is used to reduce the second phase spectrum. The difference between the first phase spectrum P and the phase spectrum loss function. It can contain three sub-loss functions, namely the instantaneous phase loss function. Group delay loss function and instantaneous angular frequency loss function

[0093] Specifically, instantaneous phase loss function Defined as the second phase spectrum The anti-winding loss of the first phase spectrum P can be expressed by the following formula (6):

[0094]

[0095] in, Let be the instantaneous phase loss function. To obtain the mean, f AW It is a linear anti-winding function. P represents the second phase spectrum, and P represents the first pair of phase spectra.

[0096] Specifically, group delay loss function Defined as decoding group delay and natural group delay Δ DF The resistance of P to winding loss can be expressed by the following formula (7):

[0097]

[0098] in, Let be the group delay loss function. To take the average, For the decoding group delay, Δ DF P is the natural group delay, Δ DF For differential along the frequency axis, Δ DT For the time-axis difference, f AW It is a linear anti-winding function.

[0099] Specifically, the instantaneous angular frequency loss function Defined as decoding instantaneous angular frequency and natural instantaneous angular frequency Δ DT The resistance of P to winding loss can be expressed by the following formula (8):

[0100]

[0101] in, The instantaneous angular frequency loss function, To take the average, To decode the instantaneous angular frequency, Δ DT P is the natural instantaneous angular frequency, f AW It is a linear anti-winding function.

[0102] It should be noted that the linear anti-winding function f is involved in all of the above formulas (6) to (8). AW This can avoid the problem of training error amplification caused by phase winding, as shown in the following formula (9):

[0103]

[0104] Among them, f AW It is a linear anti-winding function. It is the set of real numbers.

[0105] It should be noted that the first phase spectrum P involved in formulas (6) to (8) above can be represented by the following formula (10):

[0106] P = Φ(Re(S), Im(S)) (10)

[0107] Where P is the first phase spectrum, S is the short-time complex spectrum, and Re and Im represent taking the real part and taking the imaginary part, respectively.

[0108] Finally, the phase spectrum loss function This can be expressed as an instantaneous phase loss function. Group delay loss function and instantaneous angular frequency loss function The linear combination of can be represented by the following formula (11):

[0109]

[0110] in, For phase spectrum loss function, Let be the instantaneous phase loss function. Let be the group delay loss function. This is the instantaneous angular frequency loss function.

[0111] In some examples, short-time spectra were also obtained during the above process. The short-time complex spectrum S can also be based on the short-time spectral loss function. Update the waveform recovery model. Use the short-time spectral loss function. Used to improve the second logarithmic amplitude spectrum of decoding Second phase spectrum The degree of matching between them, and the short-time spectra that guarantee their reconstruction. Consistency. Among them, short-time spectrum. It contains three sub-loss functions, namely the real part loss function. Imaginary part loss function and short-time spectral consistency loss function

[0112] Specifically, the real part loss function Defined as reconstructing the real part The absolute error loss between the real part Re(S) and the imaginary part loss is defined by the function. To reconstruct the imaginary part The absolute error loss between the imaginary part Im(S) and the natural imaginary part can be represented by the following formulas (12) and (13):

[0113]

[0114]

[0115] in, Let be the real part loss function. Define a function for the imaginary part loss. To take the average, To reconstruct the real part, Re(S) is the natural real part. To reconstruct the imaginary part, Im(S) is the natural imaginary part. S is the short-time spectrum, and S is the short-time complex spectrum.

[0116] Since the second logarithmic amplitude spectrum and the second phase spectrum mentioned above are predicted, and the short-time spectral domain is only a subset of the complex domain, their reconstructed short-time spectrum... It is not necessarily a real short-time spectrum. A real short-time spectrum is represented as... Short-time spectral consistency loss function Short-time spectrum used for reconstructing and real short-time spectrum The gap between them. And the real short-time spectrum. It is achieved by first reconstructing the short-time spectrum. Performing the inverse short-time Fourier transform yields a waveform. The waveform is then subjected to a Fourier transform, as shown in the following formula (14):

[0117]

[0118] in, For real-world short-time spectra, STFT is the Short-Time Fourier Transform, and ISTFT is the Inverse Short-Time Fourier Transform. The short-time spectrum is reconstructed.

[0119] Short-time spectral consistency loss function Defined as reconstructed short-time spectrum With real-world short-time spectrum The L2 norm between them, written in the form of real and imaginary parts, can be represented by the following formula (15):

[0120]

[0121] in, Let E be the short-time spectral consistency loss function, and E be the mean value. To reconstruct the real part, For real parts, To reconstruct the imaginary part, For real and imaginary parts, For short-time spectrum, It is a real short-time spectrum.

[0122] Finally, the short-time spectral loss function It can be represented as a real part loss function. Imaginary part loss function and short-time spectral consistency loss function A linear combination according to a certain proportion can be represented by the following formula (16):

[0123]

[0124] in, Let λ be the short-time spectral loss function. RI It's a hyperparameter. Let be the real part loss function. The loss function is the imaginary part. This is the short-time spectral consistency loss function.

[0125] In some examples, the loss function described above may include a waveform loss function. Waveform loss function Used to reduce the decoded waveform The difference between the natural waveform x (i.e., the waveform input to encoder module 101 and the waveform output by decoder module 103) and the natural waveform x (i.e., the waveform input to encoder module 101 and the waveform output by decoder module 103) includes the generator loss function of the generative adversarial network. Feature matching loss function Mel spectrum loss function Waveform loss function The above sub-loss functions are linearly combined in a certain proportion, as shown in the following formula (16):

[0126]

[0127] in, The waveform loss function is... To generate the generator loss function for the adversarial network, Let λ be the feature matching loss function. Mel It's a hyperparameter. Let be the Mel spectrum loss function.

[0128] It should be noted that the waveform recovery model described above adopts a standard generative adversarial training model, and the final generator loss is... A linear combination of all the above losses in a certain proportion can be represented by the following formula (17):

[0129]

[0130] in, For generator loss, For amplitude spectrum loss function, For phase spectrum loss function, For short-time spectral loss function, Let λ be the waveform loss function. A , λ P and λ S All of these are hyperparameters.

[0131] During training, loss is used. The decision loss function of generative adversarial networks Train the generator and the decision generator alternately until the model converges.

[0132] In some specific scenarios, taking the actual application of voice communication as an example, the first electronic device (sender) encodes the voice signal and sends the encoded voice signal to the second electronic device (receiver). The receiver then decodes the encoded voice signal to obtain the voice signal.

[0133] In some specific implementations, the encoder of the first electronic device first extracts the logarithmic amplitude spectrum A and phase spectrum P from the waveform of the speech signal through a short-time Fourier transform, and generates a continuous code C based on the logarithmic amplitude spectrum A and phase spectrum P. Then, the quantizer of the first electronic device discretizes the continuous code C and generates an index vector m. 1 m 2 , ..., mQ Finally, the index vector m 1 m 2 , ..., m Q It is transmitted in binary form and sent to a second electronic device.

[0134] After the decoder of the second electronic device receives the index vector, it first uses the codebook B. 1 B 2 B Q Generate quantization code And according to the quantization code Generate logarithmic amplitude spectrum and phase spectrum Finally, the logarithmic amplitude spectrum and phase spectrum The speech signal waveform is recovered by inverse short-time Fourier transform. This enables a complete voice communication process.

[0135] In summary, this application discloses a voice communication system, comprising: a first electronic device and a second electronic device; the first electronic device is used to acquire the waveform of a voice signal; extract a first logarithmic amplitude spectrum and a first phase spectrum from the waveform of the voice signal through a short-time Fourier transform; generate a continuous code based on the first logarithmic amplitude spectrum and the first phase spectrum; discretize the continuous code to obtain an index vector; and send the index vector to the second electronic device; the second electronic device is used to generate a quantization code based on the index vector; generate a second logarithmic amplitude spectrum and a second phase spectrum based on the quantization code; and recover the waveform of the voice signal from the second logarithmic amplitude spectrum and the second phase spectrum through an inverse short-time Fourier transform. Thus, by using short-time Fourier transform and inverse short-time Fourier transform, the waveform of a voice signal with high fidelity is converted into a voice amplitude spectrum and a phase spectrum. Then, the voice amplitude spectrum and phase spectrum are used as voice parameter features for parallel encoding and decoding. This ensures good performance of the decoded voice signal while storing or transmitting the voice signal at a low encoding bit rate, thereby improving the efficiency and fidelity of voice communication.

[0136] See Figure 3 This figure is a flowchart of a voice communication method provided in an embodiment of this application. The method is applied to a first electronic device 11 and includes:

[0137] S301: Acquire the waveform of the voice signal.

[0138] S302: Extract the first logarithmic amplitude spectrum and the first phase spectrum from the waveform of the speech signal through short-time Fourier transform.

[0139] S303: Generate a continuous code based on the first logarithmic amplitude spectrum and the first phase spectrum.

[0140] S304: Discretize the continuous code to obtain an index vector.

[0141] S305: Send the index vector to the second electronic device.

[0142] In summary, this application provides a voice communication method that can ensure good performance of the decoded voice signal while storing or transmitting voice signals at a low encoding bit rate, thereby improving the efficiency and fidelity of voice communication.

[0143] See Figure 4 This figure is a flowchart of another voice communication method provided in an embodiment of this application. The method is applied to a second electronic device 12 and includes:

[0144] S401: Receive the index vector sent by the first electronic device.

[0145] S402: Generate quantization code based on the index vector.

[0146] S403: Generate a second logarithmic amplitude spectrum and a second phase spectrum according to the quantization code.

[0147] S404: By using inverse short-time Fourier transform, the second logarithmic amplitude spectrum and the second phase spectrum are restored to the waveform of the speech signal.

[0148] In summary, this application provides a voice communication method that can ensure good performance of the decoded voice signal while storing or transmitting voice signals at a low encoding bit rate, thereby improving the efficiency and fidelity of voice communication.

[0149] In the embodiments of this application, the terms "first" and "second" (if they exist) are used only as name identifiers and do not represent the order of first and second.

[0150] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that all or part of the steps in the methods of the above embodiments can be implemented by means of software plus a general-purpose hardware platform. Based on this understanding, the technical solution of this application can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as a read-only memory (ROM) / RAM, magnetic disk, optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, a server, or a network communication device such as a router) to execute the methods described in various embodiments or some parts of the embodiments of this application.

[0151] It should be noted that the various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for the device and medium embodiments, since they are basically similar to the system and method embodiments, the description is relatively simple, and the relevant parts can be referred to the description of the method embodiments. The device and medium embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components indicated as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0152] The above description is merely one specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A voice communication system, characterized in that, The system includes: a first electronic device and a second electronic device; The first electronic device is configured to acquire the waveform of a speech signal; extract a first logarithmic amplitude spectrum and a first phase spectrum from the waveform of the speech signal through a short-time Fourier transform; generate a continuous code based on the first logarithmic amplitude spectrum and the first phase spectrum; discretize the continuous code to obtain an index vector; and send the index vector to the second electronic device. The second electronic device is configured to generate a quantization code based on the index vector; generate a second logarithmic amplitude spectrum and a second phase spectrum based on the quantization code; and recover the waveform of the speech signal by using an inverse short-time Fourier transform to the second logarithmic amplitude spectrum and the second phase spectrum.

2. The system according to claim 1, characterized in that, The first electronic device includes: an encoder module and a quantizer module; The encoder module is used to acquire the waveform of the speech signal; extract a first logarithmic amplitude spectrum and a first phase spectrum from the waveform of the speech signal through short-time Fourier transform; generate a continuous code based on the first logarithmic amplitude spectrum and the first phase spectrum; and send the continuous code to the quantizer module. The quantizer module is used to discretize the continuous code to obtain an index vector; and send the index vector to the second electronic device.

3. The system according to claim 2, characterized in that, The encoder module is specifically used for: encoding the first logarithmic amplitude spectrum to obtain an amplitude code; encoding the first phase spectrum to obtain a phase code; and concatenating the amplitude code and the phase code to generate a continuous code.

4. The system according to claim 3, characterized in that, The encoder module includes: an amplitude sub-encoder and a phase sub-encoder; The amplitude sub-encoder is used to encode the first logarithmic amplitude spectrum to obtain an amplitude code; The phase sub-encoder is used to encode the first phase spectrum to obtain a phase code.

5. The system according to claim 1, characterized in that, The second electronic device includes: a decoder module; The decoder module is used to generate a quantization code based on the index vector; generate a second logarithmic amplitude spectrum and a second phase spectrum based on the quantization code; and recover the waveform of the speech signal by using an inverse short-time Fourier transform to the second logarithmic amplitude spectrum and the second phase spectrum.

6. The system according to claim 1, characterized in that, The second electronic device is also used for: A waveform reconstruction model is generated based on the quantization code, the second logarithmic amplitude spectrum, the second phase spectrum, and the waveform of the speech signal. The waveform reconstruction model is a neural network model that generates the waveform of the speech signal through the quantization code.

7. The system according to claim 6, characterized in that, The second electronic device is also used to update the waveform recovery model based on the value of one or more loss functions selected from the amplitude spectrum loss function, phase spectrum loss function, short-time spectrum loss function, and waveform loss function.

8. The system according to claim 7, characterized in that, The phase spectrum loss function is a linear combination of the instantaneous phase loss function, the group delay loss function, and the instantaneous angular frequency loss function; the short-time spectrum loss function is a linear combination of the real part loss function, the imaginary part loss function, and the short-time spectrum consistency loss function; and the waveform loss function is a linear combination of the loss function of the generative adversarial network, the feature matching loss function, and the Mel spectrum loss function.

9. A voice communication method, characterized in that, Applied to a first electronic device, the method includes: Obtain the waveform of the speech signal; The first logarithmic amplitude spectrum and the first phase spectrum are extracted from the waveform of the speech signal by using short-time Fourier transform. A continuous code is generated based on the first logarithmic amplitude spectrum and the first phase spectrum; The continuous code is discretized to obtain an index vector; The index vector is sent to the second electronic device, so that the second electronic device generates a quantization code based on the index vector, generates a second logarithmic amplitude spectrum and a second phase spectrum based on the quantization code, and recovers the waveform of the speech signal by the second logarithmic amplitude spectrum and the second phase spectrum through inverse short-time Fourier transform.

10. A voice communication method, characterized in that, Applied to a second electronic device, the method includes: The system receives an index vector sent by a first electronic device, wherein the index vector is obtained by the first electronic device from the waveform of the speech signal, extracts a first logarithmic amplitude spectrum and a first phase spectrum from the waveform of the speech signal through a short-time Fourier transform, generates a continuous code based on the first logarithmic amplitude spectrum and the first phase spectrum, and then discretizes the continuous code to obtain the result. Based on the index vector, generate quantization codes; Based on the quantization code, a second logarithmic amplitude spectrum and a second phase spectrum are generated; The second logarithmic amplitude spectrum and the second phase spectrum are recovered into the waveform of the speech signal by inverse short-time Fourier transform.

Images