Ultra low bit rate speech coding and decoding system based on text semantic information fidelity

Abstract

The application provides a kind of based on text semantic information fidelity super low code rate speech coding system, it is related to speech coding technical field.The system includes: through multimodal text-speech joint encoder, speech feature extraction and text feature extraction are carried out to original speech, speech feature and text feature are obtained, and text feature is embedded into speech feature, text-speech feature of original speech is obtained, and text-speech feature of original speech is sent to receiving end in semantic communication system;The text-speech feature received by receiving end is input into text semantic fidelity decoder based on attention mechanism, to obtain reconstructed speech feature and reconstructed text feature, and the reconstructed speech feature is decoded layer by layer and attention calculation is carried out with the reconstructed text feature, to obtain decoded speech.Through the speech coding technology of the application, good speech perceptual quality can still be maintained under super low code rate, and the speech intelligibility is guaranteed.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of speech coding, and in particular, relate to an ultra-low code rate speech coding system based on text semantic information fidelity. BACKGROUND

[0002] Speech compression coding technology transmits by compressing the sending signal into a bit stream and reconstructing at the receiving end, and realizes communication in a lossy source compression manner, which is an indispensable technology in information transmission and communication systems. With the continuous development of communication technology, semantic communication systems, as a new communication mode, have gradually attracted attention, focusing on the semantic content carried by information, aiming to realize accurate understanding and interaction of information meaning between the two parties of communication.

[0003] In bandwidth severely limited scenarios such as satellite communication and emergency communication, voice services urgently need to realize high compression rate transmission at ultra-low bit rate. However, the traditional speech compression method based on waveform and spectrum restoration fails to fully consider the semantic content of speech information, and under the condition of ultra-low code rate, the content is often severely distorted, the intelligibility is significantly reduced, and it is difficult to meet the expectations of modern communication users for voice quality. SUMMARY

[0004] Embodiments of the present application aim to provide an ultra-low code rate speech coding system based on text semantic information fidelity, which aims to overcome the above problems or at least partially solve the above problems.

[0005] The first aspect of the embodiments of the present application provides an ultra-low code rate speech coding system based on text semantic information fidelity, comprising:

[0006] The sending end in the semantic communication system extracts speech features and text features of the original speech through a multi-modal text-speech joint encoder, and embeds the text features of the original speech into the speech features of the original speech to obtain text-speech features of the original speech, and sends the text-speech features of the original speech to the receiving end in the semantic communication system;

[0007] The receiving end inputs the received text-speech features into a text semantic fidelity decoder based on attention mechanism to obtain reconstructed speech features and reconstructed text features, decodes the reconstructed speech features layer by layer and performs attention calculation with the reconstructed text features to obtain decoded speech, and the attention calculation is used to realize the display guidance of the reconstructed text features obtained by each layer of decoded speech features.

[0008] In an alternative implementation, the attention mechanism-based text semantic fidelity decoder comprises a first convolutional layer, N decoding modules, N text attention modules, a first activation function layer, a first output projection layer, and a second activation function layer;

[0009] The receiving end processes the received text-speech features through the first convolutional layer to obtain reconstructed speech features and reconstructed text features;

[0010] The receiving end performs layer-by-layer up-sampling on the reconstructed speech features through the N decoding modules;

[0011] The receiving end takes the reconstructed text features as K and V of the nth text attention module, takes the up-sampled speech features output by the nth decoding module as Q, performs attention calculation, obtains enhanced speech features output by the nth text attention module after adding text semantic information, and takes the enhanced speech features as the input of the (n+1)th decoding module;

[0012] The receiving end processes the output of the last text attention module through the first activation function layer, the first output projection layer, and the second activation function layer to obtain decoded speech.

[0013] In an alternative implementation, the multi-modal text-speech joint encoder comprises a speech encoder, a text encoder, and a text-speech alignment splicing module;

[0014] The speech encoder comprises a second convolutional layer, N encoding modules, a first activation function layer, and a second output projection layer; the sending end maps the original speech into a multi-dimensional feature space through the second convolutional layer while keeping the time dimension of the original speech unchanged; the sending end performs layer-by-layer down-sampling on the output of the second convolutional layer through the N encoding modules to realize local-to-global speech feature extraction; the sending end models the periodic features in the original speech through the first activation function layer and the second output projection layer to obtain the speech features of the original speech;

[0015] The sending end performs log-mel feature extraction on the original speech through the text encoder to obtain the mel spectrogram of the original speech, and maps the original speech into a multi-dimensional phoneme space according to the mel spectrogram of the original speech to obtain the text features of the original speech;

[0016] The sending end performs time dimension alignment on the text features of the original speech through the relative position keeping manner through the text-speech alignment splicing module, splices the aligned text features and the speech features of the original speech in the channel dimension to obtain the text-speech features.

[0017] In an optional implementation, the method performed by the text semantic information fidelity based super low bit rate speech coding system is implemented by a text semantic information fidelity based super low bit rate speech coding model, which at least comprises the multi-modal text-speech joint encoder and the attention mechanism based text semantic fidelity decoder.

[0018] The loss function value of the text semantic information fidelity based super low bit rate speech coding model at least comprises a time domain loss function value and a frequency domain loss function value.

[0019] The time domain loss function value is used to promote the consistency in the time domain between the sample decoded speech output by the attention mechanism based text semantic fidelity decoder for the sample original speech and the sample original speech input to the multi-modal text-speech joint encoder.

[0020] The frequency domain loss function value is used to promote the consistency in the frequency domain between the mel-frequency spectrum feature of the sample decoded speech and the mel-frequency spectrum feature of the sample original speech.

[0021] In an optional implementation, the text semantic information fidelity based super low bit rate speech coding model further comprises a residual vector quantizer; and the sending end sends the text-speech feature of the original speech to a receiving end in the semantic communication system, comprising:

[0022] The sending end quantizes the text-speech feature of the original speech according to a codebook set by the residual vector quantizer to obtain a codebook index, and sends the codebook index to the receiving end.

[0023] The receiving end dequantizes the received codebook index according to the codebook set to obtain the received text-speech feature before inputting the received text-speech feature to the attention mechanism based text semantic fidelity decoder.

[0024] The loss function value of the text semantic information fidelity based super low bit rate speech coding model further comprises a commitment loss function value and a codebook loss function value corresponding to the residual vector quantizer.

[0025] The commitment loss function value is used to promote the output of the multi-modal text-speech joint encoder to be close to the nearest codebook vector.

[0026] The codebook loss function value is used to promote the update of the codebook vector to match the output of the multi-modal text-speech joint encoder.

[0027] In an optional embodiment, the training phase of the super-low code rate speech coding model based on text semantic information fidelity comprises a first training phase and a second training phase.

[0028] The first training phase is used for pre-training a text encoder to be trained in a multi-modal text-speech joint encoder to be trained, and specifically comprises the following steps:

[0029] inputting a sample original speech into the text encoder to be trained to obtain recognized text corresponding to the sample original speech;

[0030] obtaining a first CTC loss function value based on a real text label corresponding to the sample original speech and the recognized text corresponding to the sample original speech;

[0031] updating parameters of the text encoder to be trained based on the first CTC loss function value to obtain a pre-trained text encoder, and freezing parameters of the pre-trained text encoder;

[0032] The second training phase is used for jointly training a text semantic fidelity decoder based on an attention mechanism to be trained, a residual vector quantizer to be trained, and a speech encoder to be trained in the multi-modal text-speech joint encoder to be trained.

[0033] In an optional embodiment, the text semantic fidelity decoder based on the attention mechanism to be trained, the residual vector quantizer to be trained, and the multi-modal text-speech joint encoder to be trained are generators, a discriminator is connected after the text semantic fidelity decoder based on the attention mechanism to be trained, and the training of the second training phase is performed in an adversarial training manner; wherein the discriminator is used to distinguish between real speech and synthesized speech.

[0034] The loss function value of the super-low code rate speech coding model based on text semantic information fidelity further comprises a generator loss function value, which is used to make the speech generated by the generator more real until the discriminator cannot distinguish between real speech and synthesized speech.

[0035] In an optional embodiment, the discriminator at least comprises:

[0036] The multi-scale STFT discriminator is used for processing sample decoded speech output by the to-be-trained attention mechanism-based text semantic fidelity decoder for a sample original speech, obtaining a complex STFT spectrum feature map, performing sub-band division on the complex STFT spectrum feature map to capture acoustic characteristics of different frequency ranges, and obtaining a discrimination result based on the STFT spectrum feature map of each sub-band; the loss function value of the text semantic information fidelity-based super-low code rate speech coding and decoding model further comprises a multi-scale STFT discriminator loss function value, which is used for prompting the speech generated by the generator to be consistent with the spectrum features of the real speech at multi-scale and multi-resolution.

[0037] In an optional implementation, the discriminator further comprises:

[0038] The multi-period discriminator is used for rearranging the sample decoded speech according to different periods to generate a two-dimensional representation with different time granularities, and obtaining a discrimination result based on the two-dimensional representation with different time granularities; the loss function value of the text semantic information fidelity-based super-low code rate speech coding and decoding model further comprises a multi-period discriminator loss function value, which is used for prompting the time-domain periodicity structure of the speech generated by the generator to be consistent with the time-domain periodicity structure of the real speech.

[0039] In an optional implementation, a pre-trained recognizer is connected after the to-be-trained attention mechanism-based text semantic fidelity decoder;

[0040] The loss function value of the text semantic information fidelity-based super-low code rate speech coding and decoding model further comprises a second CTC loss function value, which is used for prompting the recognition text corresponding to the speech generated by the generator to be consistent with the real text label corresponding to the sample original speech input into the generator in terms of semantics.

[0041] In the text semantic information fidelity-based super-low code rate speech coding and decoding system provided in the application, a sending end in a semantic communication system performs speech feature extraction and text feature extraction on an original speech through a multi-modal text-speech joint encoder to obtain speech features of the original speech and text features of the original speech, embeds the text features of the original speech into the speech features of the original speech to obtain text-speech features of the original speech, and sends the text-speech features of the original speech to a receiving end in the semantic communication system; the receiving end inputs the received text-speech features into an attention mechanism-based text semantic fidelity decoder to obtain reconstructed speech features and reconstructed text features, decodes the reconstructed speech features layer by layer and performs attention calculation with the reconstructed text features to obtain decoded speech, and the attention calculation is used to realize display guidance of the reconstructed text features obtained by each layer of decoded speech features.

[0042] Thus, in the super low bit rate speech coding system based on text semantic information fidelity, the text semantic information is integrated into the speech feature as an auxiliary modality, the effect of modality complementary cooperation is realized, the foundation for maintaining semantic integrity under super low bit rate is laid; through the text attention mechanism, the decoder of the receiving end can guide the generation of speech features according to the text features, selectively enhance the speech features, and still maintain good speech perceptual quality under super low bit rate, guarantee the speech intelligibility, and meet the speech transmission demand of bandwidth limited scene (such as satellite communication, emergency communication, etc.); an effective solution is provided for speech transmission of semantic communication system in low resource environment. BRIEF DESCRIPTION OF DRAWINGS

[0043] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed to be used in the description of the embodiments of the present application will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0044] Figure 1 is a structural schematic diagram of a super low bit rate speech coding system based on text semantic information fidelity according to an embodiment of the present application;

[0045] Figure 2 is a structural schematic diagram of a text semantic fidelity decoder based on attention mechanism of a super low bit rate speech coding system based on text semantic information fidelity according to an embodiment of the present application;

[0046] Figure 3 is a structural schematic diagram of a speech encoder of a super low bit rate speech coding system based on text semantic information fidelity according to an embodiment of the present application;

[0047] Figure 4 is a structural schematic diagram of a text encoder of a super low bit rate speech coding system based on text semantic information fidelity according to an embodiment of the present application;

[0048] Figure 5 is a structural schematic diagram of a residual vector quantizer of a super low bit rate speech coding system based on text semantic information fidelity according to an embodiment of the present application;

[0049] Figure 6 is a structural schematic diagram of a discriminator and a recognizer used in a joint optimization method based on multi-task adversarial training of a super low bit rate speech coding system based on text semantic information fidelity according to an embodiment of the present application;

[0050] Figure 7Fig. 1 is a schematic diagram of an overall structure of an ultra-low code rate speech coding and decoding system based on text semantic information fidelity according to an embodiment of the present application. DETAILED DESCRIPTION

[0051] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some of the embodiments of the present application, but not all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of the present application.

[0052] In the drawings, the size of the constituent elements, the thickness of the layers, or the area may be exaggerated for the sake of clearness and convenience in some cases. Therefore, the present disclosure is not necessarily limited to the size, shape, and values shown in the drawings. The shapes and sizes of the components in the drawings do not reflect the actual proportions of the components. Furthermore, the drawings schematically show ideal examples, and the present disclosure is not limited to the shapes or values shown in the drawings.

[0053] Reference Signs List Figure 1 , Figure 1 Fig. 1 is a schematic diagram of an overall structure of an ultra-low code rate speech coding and decoding system based on text semantic information fidelity according to an embodiment of the present application. As shown in Fig. 1, the system includes a multi-modal text-speech joint encoder 10 and an attention mechanism-based text semantic fidelity decoder 20. Figure 1

[0054] The sending end in the semantic communication system performs speech feature extraction and text feature extraction on the original speech through the multi-modal text-speech joint encoder 10, obtains the speech features of the original speech and the text features of the original speech, embeds the text features of the original speech into the speech features of the original speech, obtains the text-speech features of the original speech, and sends the text-speech features of the original speech to the receiving end in the semantic communication system.

[0055] ​In this embodiment, at the sending end in the semantic communication system, the original speech is encoded by a multi-modal text-speech joint encoder in a text semantic information fidelity-based super-low code rate speech coding and decoding system. In the encoding process, speech feature extraction and text feature extraction are performed on the input original speech to obtain speech features of the original speech and text features (such as the phoneme "ni hao") of the original speech; the text features of the original speech are embedded into the speech features in a text-speech alignment splicing manner to obtain text-speech features of the original speech; finally, the text-speech features of the original speech are sent to the receiving end in the semantic communication system. Before sending the text-speech features, the text-speech features can be quantized and compressed to reduce bandwidth consumption. By fusing the semantic information of the text features, the semantic consistency of the system is enhanced, and the generated speech can be more natural and fluent.

[0056] The receiving end inputs the received text-speech features into a text semantic fidelity decoder 20 based on an attention mechanism to obtain reconstructed speech features and reconstructed text features, decodes the reconstructed speech features layer by layer and performs attention calculation with the reconstructed text features to obtain decoded speech, and the attention calculation is used to realize display guidance of the reconstructed text features obtained by each layer of decoded speech features.

[0057] In this embodiment, at the receiving end in the semantic communication system, the received text-speech features are decoded by a text semantic fidelity decoder based on an attention mechanism in a text semantic information fidelity-based super-low code rate speech coding and decoding system. In the decoding process, the text-speech features are separated according to an agreed modal splicing strategy to obtain reconstructed speech features and reconstructed text features; the reconstructed speech features are decoded layer by layer, and attention calculation is performed with the reconstructed text features to enhance the speech features according to the text semantic information, and finally high-quality and semantically consistent reconstructed speech, i.e., decoded speech, is output; and the attention calculation is used to realize display guidance of the reconstructed text features obtained by each layer of decoded speech features. By using the text attention mechanism, the text features are fully utilized to guide speech reconstruction, which can effectively recover easily lost key speech details and obtain high-quality and semantically consistent reconstructed speech, and can meet the expectations of modern communication users for speech quality.

[0058] Therefore, in the super low bit rate speech coding system based on text semantic information fidelity, the text semantic information is integrated into the speech feature as an auxiliary modality, the effect of modality complementary cooperation is realized, the foundation for maintaining semantic integrity under the condition of super low bit rate is laid, the text feature is received by the decoder of the receiving end to guide the generation of the speech feature, the speech feature is selectively enhanced, the good speech perceptual quality is maintained under the condition of super low bit rate, the speech intelligibility is ensured, and the speech transmission demand of the bandwidth limited scene (such as satellite communication, emergency communication and the like) can be met. An effective solution is provided for the speech transmission of the semantic communication system in the low resource environment.

[0059] In an implementation mode, the application further provides a super low bit rate speech coding system based on text semantic information fidelity, wherein the text semantic fidelity decoder based on the attention mechanism includes a first convolutional layer, N decoding modules, N text attention modules, a first activation function layer, a first output projection layer and a second activation function layer.

[0060] The receiving end processes the received text-speech feature through the first convolutional layer to obtain reconstructed speech features and reconstructed text features.

[0061] The receiving end performs layer-by-layer up-sampling on the reconstructed speech features through the N decoding modules.

[0062] The receiving end takes the reconstructed text features as K and V of the nth text attention module, takes the up-sampled speech features output by the nth decoding module as Q, performs attention calculation, obtains the enhanced speech features output by the nth text attention module after the text semantic information is added, and takes the enhanced speech features as the input of the (n+1)th decoding module.

[0063] The receiving end processes the output of the last text attention module through the first activation function layer, the first output projection layer and the second activation function layer to obtain the decoded speech.

[0064] In the embodiment, the text semantic fidelity decoder based on the attention mechanism mainly includes a speech decoder and a text attention module, and the core is to fuse the text semantic information in the decoding process through the text attention module. The decoder specifically includes a first convolutional layer, N decoding modules, N text attention modules, a first activation function layer, a first output projection layer and a second activation function layer. As shown in FIG. 2, the speech decoder is used to decode the speech feature to obtain the reconstructed speech feature, and the text attention module is used to fuse the text semantic information into the reconstructed speech feature to obtain the enhanced speech feature. Figure 2As shown, the text semantic fidelity decoder based on the attention mechanism proposed in the present application specifically comprises: an initial convolutional layer (first convolutional layer), four layers of up-sampling decoding blocks (decoding modules), four text attention modules, a Snake activation function (first activation function layer), a first output projection layer, a Tanh activation function (second activation function layer).

[0065] In the present embodiment, at the receiving end in the semantic communication system, the received text-speech features are processed by the first convolutional layer in the text semantic fidelity decoder based on the attention mechanism, the speech and text features are separated according to the agreed modal concatenation strategy, and the reconstructed speech features and the reconstructed text features are obtained. For example, the text-speech features are , after separation, the reconstructed speech features and the reconstructed text features are obtained: . The agreed modal concatenation strategy refers to the rules or methods agreed by the sending end and the receiving end in advance for embedding the text features into the speech features (encoding stage) and reverse separation (decoding stage). For example, the text-speech alignment concatenation method adopted during encoding is frame-level concatenation, i.e., the text features are concatenated after being aligned with the speech frames, and the corresponding modal concatenation strategy is direct cutting according to the feature dimension.

[0066] In the present embodiment, the reconstructed speech features are processed by the N decoding modules for step-by-step up-sampling, and each decoding module is composed of three residual units, a Snake activation and an up-sampling unit convolution. The text attention module is introduced after the decoding layer where each decoding module is located, to realize the display guidance of the speech features by the text features. For the nth text attention module, the reconstructed text features are taken as K and V, and the up-sampled speech features output by the nth decoding module are taken as Q, for attention calculation, to obtain the enhanced speech features output by the nth text attention module after adding the text semantic information, and the enhanced speech features are taken as the input of the (n+1)th decoding module. Through this attention mechanism, the text features can provide targeted guidance at each stage of speech decoding, helping to recover the lost speech details (such as in the quantization process), especially the key phoneme features related to the semantic content.

[0067] For example, referring to Figure 2 , for the first text attention module, the decoded speech features of the previous layer (the first decoding module) are received as the input, after attention calculation, the enhanced speech features after adding the text semantic information are obtained, and the enhanced speech features are taken as the input of the next decoding block (the second decoding module), until the up-sampled speech features are output by the last decoding module (the fourth decoding module).

[0068] The text attention module follows the principle of multi-head attention mechanism, taking the speech features as the query Q and the text features as the key K and the value V:

[0069]

[0070] where, represents the speech feature output by the i-th decoding module; is the reconstructed text feature separated from the joint feature (text-speech feature), 、 and are the linear projection matrices of the learnable query, key and value, respectively, for mapping the features to a dimensional space suitable for attention calculation.

[0071] The attention weight is calculated by scaling dot product attention and the information is aggregated:

[0072]

[0073] where, is a scaling factor, where is the dimension of the key vector, used to stabilize the gradient; The expression of the function is:

[0074]

[0075] After concatenating the multi-head attention results and processing them through a linear layer, the original speech information is retained through a residual connection:

[0076]

[0077] where, represents the feature obtained by concatenating the results of multiple attention heads and passing them through an output projection, each head focusing on a different text-speech feature subspace; is the enhanced speech feature after adding text semantic information, which will be used as the input of the next decoding block.

[0078] In this embodiment, the output of the last text attention module is processed through a first activation function layer, a first output projection layer and a second activation function layer to obtain the decoded speech. As shown in Figure 2 , the Snake activation function and output projection layer can be used, and the Tanh function is used to map the features to the range of [-1, 1] to generate the final waveform output, i.e. the decoded speech:

[0079]

[0080] where, W outdenotes a learnable linear projection matrix; the Snake activation function is a periodic activation function used to enhance the high-frequency and periodic information of speech features; and the Tanh function is used to map the linear projection result to the range of [-1, 1], which conforms to the standardized representation of audio waveforms.

[0081] In an embodiment, the application further provides an ultra-low code rate speech coding and decoding system based on text semantic information fidelity, wherein the multi-modal text-speech joint encoder 10 comprises a speech encoder 101, a text encoder 102, and a text-speech alignment splicing module 103.

[0082] The speech encoder 101 comprises a second convolutional layer, N encoding modules, a first activation function layer, and a second output projection layer. The sender maps the original speech into a multi-dimensional feature space through the second convolutional layer while keeping the time dimension of the original speech unchanged. The sender performs layer-by-layer down-sampling on the output of the second convolutional layer through the N encoding modules to realize local-to-global speech feature extraction. The sender models the periodic features in the original speech through the first activation function layer and the second output projection layer to obtain the speech features of the original speech.

[0083] In this embodiment, as shown in Figure 3 The speech encoder specifically comprises a second convolutional layer (initial convolutional layer), N encoding modules (four encoding blocks), a first activation function layer (Snake activation function), and a second output projection layer. The original speech is mapped into a multi-dimensional feature space through the second convolutional layer while keeping the time dimension of the original speech unchanged. For example, the initial convolutional layer has a convolution kernel size of 7, an output channel number of 64, and a step size of 1. In this way, the input original speech signal x n is mapped into a 64-dimensional feature space while keeping the time dimension unchanged, that is, the shape of the input tensor is mapped from [B, 1, T] to [B, 64, T], where B is the batch size and T is the length of the original speech signal.

[0084] In this embodiment, the output of the second convolutional layer is layer-by-layer down-sampled through the N encoding modules to realize local-to-global speech feature extraction. As shown in Figure 3As shown, each encoding module in the speech encoder is composed of three layers of residual units, a Snake activation function, and a down-sampling convolution. The step size of the down-sampling convolution is a control parameter, which can be set in advance. The convolution kernel size is 2 times the step size, and the output channel is 2 times the input channel. By controlling the step size of the last down-sampling convolution of each encoding module, the down-sampling rate of the encoder can be controlled, thereby controlling the overall down-sampling rate of the speech encoding. Each residual unit is composed of a Snake activation, a dilated convolution, and a point convolution. The dilated convolution is a commonly used convolution method in speech signal processing, which can increase the receptive field of the convolution kernel while keeping the feature map size unchanged, and capture the periodic relationship in the speech signal. The three layers of residual units can have dilated rates of [1, 3, 9] respectively, corresponding to short-term, medium-term, and long-term speech periods, to capture speech features from local to global.

[0085] In this embodiment, the periodic characteristics in the original speech are modeled through the first activation function layer, and the features are mapped to the target dimension through the second output projection layer to obtain the speech features of the original speech. The basic formula of the Snake activation function is as follows:

[0086]

[0087] wherein a is a learnable parameter, and different frequency responses can be adaptively learned for different channels of signals. The Snake activation function can better model the periodic characteristics in the original speech waveform by introducing a learnable periodic induction bias, and can significantly improve the naturalness of the synthesized speech.

[0088] The sending end performs log-Mel feature extraction on the original speech through the text encoder 102 to obtain a Mel spectrogram of the original speech, and maps the original speech into a multi-dimensional phoneme space according to the Mel spectrogram of the original speech to obtain text features of the original speech.

[0089] In this embodiment, as shown in FIG. 2, the text encoder 102 is composed of a first activation function layer, a second output projection layer, and a third output projection layer. Figure 4As shown, the text encoder includes two convolutional layers, five bidirectional long short-term memory (LSTM) layers, and a fully connected layer. First, the input raw speech is subjected to log-mel feature extraction to obtain a mel spectrogram. Then, the mel spectrogram of the raw speech is subjected to two-dimensional convolution in two layers. The first layer of convolution locally models in the time and frequency dimensions to capture the time-frequency local correlation of the speech, with a convolution kernel size of (11, 41) and a convolution step size of (2, 2). The second layer of convolution further abstracts high-order time-frequency features, reduces the frequency dimension, and retains more time information, with a convolution kernel size of (11, 21) and a step size of (1.2). Subsequently, the five bidirectional long short-term memory layers are used to capture long-time dependencies, with a hidden layer size of 512 for the LSTM. Finally, a linear projection layer is used to map the features output by the LSTM to a 221-dimensional phoneme space. In this way, the input raw speech signal x n is mapped to a 221-dimensional feature space, that is, the shape of the input tensor is mapped from [B, 1, T] to [B, 221, T / 4], where B is the batch size and T is the length of the raw speech signal. For the output tensor, the log probability can be calculated, and the text features z n .

[0090] The sending end aligns the text features of the raw speech in the time dimension by the relative position preserving manner through the text-speech alignment splicing module 103, splices the aligned text features and the speech features of the raw speech in the channel dimension, and obtains the text-speech features.

[0091] In this embodiment, the length (T / 4) of the text features and the length (T) of the speech features are not equal, and therefore, the text features z n extracted by the text encoder are aligned in the time dimension by the relative position preserving manner. Specifically, the relative position of each non-zero feature value in the text features in the original sequence is calculated one by one and is mapped to the target time dimension. If multiple non-zero feature values of the text features are mapped to the same target time dimension, the evasive strategy of left or right shifting the non-zero feature value of the later-mapped text feature ensures the information integrity. For example, the original sequence is 0102, but it needs to be expanded to 16 lengths, the non-zero feature 1 still needs to be kept at the original relative position of 2 / 4, and the non-zero feature 2 still needs to be kept at the original relative position of 4 / 4, and the middle padding can be filled with 0, to obtain the target sequence 00000001 0000 0002.

[0092] In this embodiment, the aligned text features are then reshaped to have dimensions [B, D t , L aRemodeling features of , so that the dimensions of text features and speech features are always concatenable. Finally, the text-speech features are obtained by directly concatenating the text features and the speech features in the channel dimension . , and the text features z t that are aligned with the speech features z a , i.e. the agreed modal concatenation strategy. The accurate alignment and fusion of text features and speech features are realized, and the concatenated features can accurately reflect the temporal correlation of text and speech, which is helpful for subsequent generation of natural and fluent speech.

[0093] In an embodiment, the application also provides a text semantic information fidelity based super low bit rate speech coding and decoding system, in which the method performed is realized by a text semantic information fidelity based super low bit rate speech coding and decoding model, which at least includes the multi-modal text-speech joint encoder and the attention mechanism based text semantic fidelity decoder.

[0094] The loss function value of the text semantic information fidelity based super low bit rate speech coding and decoding model at least includes a time domain loss function value and a frequency domain loss function value.

[0095] The time domain loss function value is used to promote the consistency in the time domain between the sample decoded speech output by the attention mechanism based text semantic fidelity decoder for the sample original speech and the sample original speech input to the multi-modal text-speech joint encoder.

[0096] The frequency domain loss function value is used to promote the consistency in the frequency domain between the mel-frequency spectrum features of the sample decoded speech and the mel-frequency spectrum features of the sample original speech.

[0097] In this embodiment, the core goal of the text semantic information fidelity based super low bit rate speech coding and decoding system is to realize high-quality speech reconstruction at an extremely low bit rate by combining the joint representation of text semantic information and speech signals. The method performed by the system is realized by a text semantic information fidelity based super low bit rate speech coding and decoding model. In order to ensure that the decoded speech is highly consistent with the original speech in the time domain and the frequency domain, the mean square error loss functions in the time domain and the frequency domain are used to constrain the difference between the generated decoded speech and the input original speech, and then the loss function value of the model at least includes a time domain loss function value and a frequency domain loss function value. The time domain loss function (time domain mean square error loss function) is as follows:

[0098] ​

[0099] The frequency domain loss function (frequency domain mean square error loss function) is as follows:

[0100]

[0101] wherein, and are the mel-spectral features of the input original speech and the generated decoded speech respectively, indicates a mean square error loss calculation. By minimizing the mean square error loss function in the time domain and the frequency domain, the generated decoded speech can be constrained to be as consistent as possible with the input original speech in the time domain and the frequency domain.

[0102] In an embodiment, the application also provides a text semantic information fidelity-based super-low code rate speech coding system, wherein the text semantic information fidelity-based super-low code rate speech coding model further comprises a residual vector quantizer; the sending end sends the text-speech features of the original speech to a receiving end in the semantic communication system, comprising:

[0103] The sending end quantizes the text-speech features of the original speech according to the codebook set through the residual vector quantizer to obtain a codebook index, and sends the codebook index to the receiving end;

[0104] The receiving end dequantizes the received codebook index according to the codebook set before inputting the received text-speech features into the text semantic fidelity decoder based on the attention mechanism to obtain the received text-speech features;

[0105] The loss function value of the text semantic information fidelity-based super-low code rate speech coding model further comprises a commitment loss function value and a codebook loss function value corresponding to the residual vector quantizer;

[0106] The commitment loss function value is used to make the output of the multi-modal text-speech joint encoder close to the nearest codebook vector;

[0107] The codebook loss function value is used to update the codebook vector to match the output of the multi-modal text-speech joint encoder.

[0108] In the embodiment, the text semantic information fidelity-based super-low code rate speech coding model described above further comprises a residual vector quantizer, which is used to realize efficient data compression by quantizing residual signals step by step. Therefore, the sending end can transmit the text-speech features of the original speech obtained by the multi-modal text-speech joint encoder into the residual vector quantizer, generate a compressed bit stream, and then transmit it to the receiving end in the semantic communication system.

[0109] As Figure 5 shown, residual vector quantization adopts a step-by-step quantization method, and performs hierarchical quantization through N quantizers in the residual vector quantizer. Specifically, for the input text-speech feature , a coarse quantization is first performed using a quantizer to obtain a coarse quantization codebook index, and then a residual is calculated by inputting the text-speech feature and the codebook feature corresponding to the coarse quantization codebook index to obtain a residual feature. Next, another quantizer is used to quantize the residual feature in fine granularity to obtain a fine-grained codebook index, until all quantizers are passed to obtain the total codebook index . Among them, the steps of coarse quantization and fine quantization are to search a codebook feature with the closest Euclidean distance to the input feature z i (such as text-speech feature, residual feature) from the pre-trained codebook set , the residual feature z r is the difference between the codebook feature and the input feature, and the codebook feature in the codebook set corresponds to the codebook index. When dequantizing, the input feature is reconstructed according to the codebook index selected by quantization, and the reconstruction process is to search the corresponding codebook feature from the pre-trained codebook set according to the codebook index , and add all the codebook features to obtain the dequantized text-speech feature .

[0110] Among them, for the pre-trained codebook set, the training process includes initialization and update strategy. The initialization strategy adopts random initialization, which is sampled from uniform distribution or standard normal distribution. The update of the codebook can use the exponential moving average (EMA), and its calculation formula can be expressed as:

[0111]

[0112] Among them, represents the updated codebook vector (codebook feature vector); represents the current codebook vector; represents the current input feature vector; is a decay factor, and the codebook vector selected for each input is updated using an exponential moving average with a decay of 0.99, and the unused codebook vector is replaced by a randomly sampled input frame from the current batch.

[0113] In this embodiment, the performance of the residual vector quantizer directly affects the amount of information in the bit stream transmitted to the decoder and significantly affects the quality of the decoded speech. The training process adopts a codebook random inactivation strategy, which randomly uses different numbers of codebooks for training to enable the model to adapt to different code rate scenarios. And it is optimized by commitment loss and codebook loss. Therefore, the loss function value of the corresponding low-code-rate speech coding and decoding model based on text semantic information fidelity also includes the commitment loss function value and the codebook loss function value of the residual vector quantizer. Among them, the commitment loss function value is used to: promote the output of the multi-modal text-speech joint encoder to be close to the nearest codebook vector, and improve the quantization efficiency. The commitment loss function is:

[0114]

[0115] Wherein, is the feature of the encoder output projected into the i-th quantization space, is the nearest codebook vector, denotes the gradient stop operator, which is used to pass the gradient of the decoder directly to the encoder to realize end-to-end optimization.

[0116] The codebook loss function value is used to: promote the codebook vector update to match the output of the multi-modal text-speech joint encoder, and improve the codebook utilization rate. The codebook loss function is:

[0117]

[0118] The commitment loss function value and the codebook loss function value can be combined according to certain weights to obtain the loss L of the residual vector quantizer = αL commitment +βL codebook , wherein α and β are weight coefficients for balancing the importance of the two losses.

[0119] In one embodiment, the application also provides a low-code-rate speech coding and decoding system based on text semantic information fidelity. In this system, the training phase of the low-code-rate speech coding and decoding model based on text semantic information fidelity includes a first training phase and a second training phase.

[0120] The first training phase is used to pre-train the text encoder in the multi-modal text-speech joint encoder to be trained, which specifically includes the following steps:

[0121] Inputting the sample original speech into the text encoder to be trained to obtain the recognized text corresponding to the sample original speech;

[0122] obtain a first CTC loss function value based on the real text label corresponding to the sample original speech and the recognized text corresponding to the sample original speech;

[0123] update parameters of the text encoder to be trained based on the first CTC loss function value, obtain a pre-trained text encoder, and freeze parameters of the pre-trained text encoder;

[0124] The second training stage is used for joint training of the text semantic fidelity decoding encoder to be trained based on the attention mechanism, the residual vector quantizer to be trained, and the speech encoder to be trained in the multi-modal text-speech joint encoder to be trained.

[0125] In the embodiment, the training process of the text semantic information fidelity-based super-low code rate speech coding and decoding model is divided into two stages: the first training stage is the pre-training of the text encoder, and the second training stage is the joint training of the speech encoder, the residual vector quantizer, and the text semantic fidelity decoding encoder. In the text encoder training of the first training stage, the sample original speech is input into the text encoder to be trained, and the recognized text corresponding to the sample original speech, i.e., the text feature (such as the phoneme "ni hao"), is output. Based on the CTC loss function, the first CTC loss function value is calculated according to the real text label corresponding to the sample original speech and the recognized text, and the model parameters are optimized by back propagation, and finally the pre-trained text encoder is obtained, and the parameters of the pre-trained text encoder are frozen. The formula of the first CTC loss function is as follows:

[0126]

[0127] wherein x1 is an input sequence (the recognized text corresponding to the sample original speech), y is a target sequence (the real text label), and Y is a set of all possible target sequences. The CTC loss function calculates the similarity between the input sequence and the target sequence by summing all possible alignments of the input sequence x to the target sequence y.

[0128] In one embodiment, the application also provides a text semantic information fidelity-based super-low code rate speech coding and decoding system. In the system, the text semantic fidelity decoding encoder based on the attention mechanism to be trained, the residual vector quantizer to be trained, and the multi-modal text-speech joint encoder to be trained are used as generators, a discriminator is connected after the text semantic fidelity decoding encoder based on the attention mechanism to be trained, and the training in the second training stage is performed in an adversarial training manner; wherein the discriminator is used to distinguish between real speech and synthesized speech.

[0129] The loss function value of the text semantic information fidelity-based super low bit rate speech coding and decoding model further includes a generator loss function value, which is used to make the speech generated by the generator more real until the discriminator cannot distinguish between real speech and synthesized speech.

[0130] In the joint training of the speech encoder, the residual vector quantizer and the text semantic fidelity decoder in the second training stage, the multi-task adversarial training is adopted for training in the embodiment. Specifically, the text semantic fidelity decoder based on the attention mechanism to be trained, the residual vector quantizer to be trained and the multi-modal text-speech joint encoder to be trained are taken as the generator. The discriminator is connected after the text semantic fidelity decoder based on the attention mechanism to be trained, and the discriminator is used to distinguish between real speech and synthesized speech (decoded speech) to perform adversarial training with the generator. Therefore, the loss function value of the text semantic information fidelity-based super low bit rate speech coding and decoding model further includes a generator loss function value. The loss function of the discriminator is as follows:

[0131]

[0132] wherein, is the prediction probability of the discriminator for real speech, is the prediction probability of the discriminator for synthesized speech, and E represents the mathematical expectation, which is approximated by using the average value of the small batch samples during training.

[0133] In the adversarial training, the goal of the generator is to deceive the discriminator so that the discriminator cannot distinguish between real speech and synthesized speech, that is, the speech generated by the generator is more real. By minimizing the loss function of the generator, the speech generated by the generator can be made more real, wherein the loss function of the generator is as follows:

[0134]

[0135] wherein, is the prediction probability of the discriminator for synthesized speech.

[0136] In an embodiment, the application further provides a text semantic information fidelity-based super low bit rate speech coding and decoding system, in which the discriminator at least includes:

[0137] The multi-scale STFT discriminator is configured to process the sample decoded speech output by the attention-based text semantic fidelity decoder for the sample original speech to obtain a complex STFT spectrum feature map, and perform sub-band division on the complex STFT spectrum feature map to capture acoustic characteristics in different frequency ranges, and obtain a discrimination result based on the STFT spectrum feature map of each sub-band. The loss function value of the text semantic information fidelity-based super-low code rate speech coding and decoding model further includes a multi-scale STFT discriminator loss function value configured to promote the generator to generate speech whose spectrum features at multiple scales and multiple resolutions are consistent with those of real speech.

[0138] In this embodiment, the discriminator includes at least a multi-scale STFT discriminator. During the second-stage adversarial training, the multi-scale STFT discriminator further feeds back a feature matching loss to the generator to ensure that the spectrum features of the synthesized speech (decoded speech) at multiple scales and multiple resolutions are consistent with those of real speech. Therefore, the loss function value of the text semantic information fidelity-based super-low code rate speech coding and decoding model further includes a multi-scale STFT discriminator loss function value. As shown in Figure 6 For the multi-scale STFT discriminator, different FFT window lengths are used to perform short-time Fourier transform analysis on the input sample decoded speech to generate a complex spectrum representation at multiple resolutions. Specifically, the discriminator first concatenates the real and imaginary parts to form a complex STFT spectrum feature map, and then performs sub-band division on the spectrum to capture acoustic characteristics in different frequency ranges. The STFT spectrum feature map of each sub-band is independently processed by a five-layer weight-normalized two-dimensional convolutional network, and the features of each layer are merged and then passed through a final discriminant convolution layer to obtain a discrimination result. The discriminator averages the discrimination results of different resolutions and different sub-bands to form the final discrimination result. In addition, the feature maps of each hidden layer of the discriminator are also retained, and the average absolute error loss is calculated between the feature maps and the corresponding features of real speech to form a multi-scale STFT discriminator loss function value (feature matching loss), where the multi-scale STFT discriminator loss function is as follows:

[0139]

[0140] wherein, and are the output features of the multi-scale STFT discriminator at the i-th layer for real speech and synthesized speech, respectively, M i is the feature dimension of the i-th layer. By minimizing the multi-scale STFT discriminator loss function, the spectrum features of the synthesized speech at multiple scales and multiple resolutions can be made consistent with those of real speech.

[0141] In one embodiment, this application also provides an ultra-low bitrate speech codec system based on text semantic information fidelity, wherein the discriminator in this system further includes:

[0142] A multi-period discriminator is used to rearrange the sample decoded speech according to different periods to generate two-dimensional representations with different time granularities. Based on the two-dimensional representations with different time granularities, the discrimination result is obtained. The loss function value of the ultra-low bitrate speech codec model based on text semantic information fidelity also includes: the multi-period discriminator loss function value, which is used to ensure that the temporal periodic structure of the speech generated by the generator is consistent with the temporal periodic structure of the real speech.

[0143] In this embodiment, the discriminator further includes a multi-period discriminator. During the second stage of adversarial training, the multi-period discriminator also feeds back a feature matching loss to the generator, ensuring that the synthesized speech (decoded speech) maintains consistency with the real speech in its temporal periodic structure. Therefore, the loss function value of the ultra-low bitrate speech codec model based on text semantic information fidelity also includes the multi-period discriminator loss function value. For example... Figure 6 As shown, this discriminator rearranges the speech waveform according to different period values ​​(e.g., 2, 3, 5, 7, 11) to generate two-dimensional representations with different temporal granularities. Each periodically rearranged feature map is processed by a five-layer two-dimensional convolutional network, and then the final discriminative convolutional layer outputs the discrimination result. The multi-period discriminator forms the final discrimination result by averaging the discrimination results of each period, achieving a comprehensive evaluation of the temporal periodicity of speech and effectively capturing prosodic and tonal features at different time scales. The output feature map of each convolutional layer is also retained, and the average absolute value error loss is calculated with the corresponding features of the real speech, constituting the multi-period discriminator loss function (feature matching loss). The expression is consistent with that of the multi-scale STFT discriminator loss function. By minimizing the multi-period discriminator loss function, the synthesized speech can maintain consistency with real speech in terms of temporal periodic structure.

[0144] In one embodiment, this application also provides an ultra-low bitrate speech codec system based on text semantic information fidelity, in which a pre-trained recognizer is connected after the text semantic information fidelity decoder based on the attention mechanism to be trained.

[0145] The loss function value of the ultra-low bitrate speech codec model based on text semantic information fidelity also includes: a second CTC loss function value, which is used to ensure that the recognized text corresponding to the speech generated by the generator maintains semantic consistency with the real text label corresponding to the original sample speech input to the generator.

[0146] In this embodiment, in order to prioritize the recognizability of the content, that is, to ensure that the generated decoded speech is consistent with the input text features in semantics, a pre-trained recognizer (text recognizer) is connected after the attention mechanism-based text semantic fidelity decoder to be trained to optimize the performance of the generator at a low bit rate. Therefore, the loss function value of the text semantic information fidelity-based super-low code rate speech coding and decoding model also includes a second CTC loss function value. For the pre-trained recognizer, the Whisper model (a pre-trained recognition model) can be used. The Whisper model is a Transformer-based speech recognition model with strong speech feature extraction and modeling capabilities. As shown in Figure 6 the second CTC loss function value is calculated by comparing the recognition text corresponding to the speech (sample decoded speech) generated by the generator output by the pre-trained recognition model with the true text label corresponding to the sample original speech of the generator, and the parameters of the model are optimized by backpropagation. The formula of the second CTC loss function is as follows:

[0147]

[0148] where x2 is the input sequence (recognition text corresponding to the sample decoded speech), y is the target sequence (true text label), is the set of all possible target sequences.

[0149] In combination with the above embodiments, the overall loss function of the generator can be composed by weighting the above loss functions, as follows:

[0150]

[0151]

[0152] wherein are the weight coefficients of the respective loss functions.

[0153] This multi-task joint optimization strategy enables the text semantic information fidelity-based super-low code rate speech coding and decoding model to simultaneously consider acoustic quality and semantic integrity under low code rate conditions. The frequency domain loss function and the time domain loss function ensure that the generated decoded speech matches the overall input original speech in the frequency domain and the time domain; the loss function of the generator, the multi-cycle discriminator loss function, and the multi-scale STFT discriminator loss function promote the naturalness and fine structure of the generated decoded speech; the commitment loss function and the codebook loss function of the residual vector quantizer ensure the effective representation of the encoded information; and the second CTC loss function strengthens the semantic consistency between the decoded speech and the text content. This enables the model to achieve balance in multiple performance indicators.

[0154] In an embodiment, as shown in Figure 7 Figure 7 is the overall structure diagram of the super low bit rate speech coding system based on text semantic information fidelity shown in an embodiment of the present application. The system mainly consists of three main modules of multi-modal text-speech joint encoder, residual vector quantizer and text semantic fidelity decoder based on attention mechanism, and is optimized by combining multi-task adversarial training method. In the encoding process, the input original speech signal passes through the speech encoder and text encoder in the multi-modal text-speech joint encoder at the same time, and the speech features and text features are extracted respectively, and after the text-speech alignment splicing, the residual vector quantizer is transmitted to the residual vector quantizer, and the compressed bit stream is used for transmission. In the decoding process, the received bit stream is recovered by the text semantic fidelity decoder based on the attention mechanism. The decoder selectively enhances the key acoustic features layer by layer through the text attention mechanism, so as to reconstruct the high-quality speech signal.

[0155] The super low bit rate speech coding system based on text semantic information fidelity proposed in the present application has the following significant advantages:

[0156] 1. Semantic fidelity under super low bit rate: In the extremely low bit rate range of 250-780bps, the character recognition accuracy can be maintained at 82.5%-93.2%, which is significantly better than other coding schemes under the same bit rate. This enables accurate transmission of voice information in bandwidth-limited scenarios such as satellite communication and emergency communication.

[0157] 2. Significant reduction in model parameter quantity: The model parameter quantity under 250bps bit rate is only 650MB, which is about 9% of the similar codec Semanticodec (310bps), greatly reducing the storage requirement and facilitating deployment on resource-limited devices.

[0158] 3. Excellent noise robustness: In a 10dB signal-to-noise ratio noise environment, the performance decline is controlled within 30%, which is better than similar codecs, showing excellent environmental adaptability.

[0159] 4. Text semantic enhanced acoustic feature recovery: Through the text attention mechanism, the decoder can selectively enhance the key acoustic features according to the text content, and still maintain good speech perception quality under super low bit rate, with MOS-LQO score reaching 1.87-2.50 in the range of 250-780bps, which is much higher than traditional methods.

[0160] 5. Cross-modal feature fusion: The text semantic information is integrated into the coding process as an auxiliary modality, enabling the model to prioritize the preservation of semantic key information during compression, achieving the effect of modal complementary cooperation.

[0161] ​In summary, the super low bit rate speech coding and decoding system based on text semantic information fidelity proposed in the application successfully solves the key technical problems of speech coding and decoding under super low bit rate, and through the text semantic information fidelity technology, the bit rate and model complexity are significantly reduced while the speech intelligibility is ensured, thereby providing a new solution for speech communication in bandwidth limited scenarios.

[0162] Each of the embodiments in the specification is described in a progressive manner, and each embodiment focuses on the difference from other embodiments. The same or similar parts between the embodiments can be referred to each other.

[0163] Those skilled in the art should understand that the embodiments of the application can be provided as a method, device or computer program product. Therefore, the embodiments of the application can be in the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware aspects. Moreover, the embodiments of the application can be in the form of a computer program product implemented on one or more computer usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer usable program code.

[0164] The embodiments of the application are described with reference to flowcharts and / or block diagrams according to the method, terminal device (system) and computer program product of the embodiments of the application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of the flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to a general purpose computer, a special purpose computer, an embedded processor or other programmable data processing terminal device to produce a machine, so that the instructions executed by the computer or other programmable data processing terminal device produce a device that implements the functions specified in the flowcharts and / or block diagrams. Figure 1 The functions specified in one or more flows and / or blocks Figure 1 The device that implements the functions specified in one or more flows and / or blocks.

[0165] These computer program instructions can also be stored in a computer readable storage medium that can guide the computer or other programmable data processing terminal device to work in a specific way, so that the instructions stored in the computer readable storage medium produce a product including instruction devices that implement the functions specified in the flowcharts and / or block diagrams. Figure 1 The functions specified in one or more flows and / or blocks Figure 1 The device that implements the functions specified in one or more flows and / or blocks.

[0166] These computer program instructions can also be loaded into a computer or other programmable data processing terminal device, so that a series of operation steps are performed on the computer or other programmable terminal device to produce a computer implemented process, so that the instructions executed on the computer or other programmable terminal device provide a process for implementing the functions specified in the flowcharts and / or block diagrams.Figure 1 one or more processes and / or blocks Figure 1 the steps of a function specified in one or more processes or blocks.

[0167] While preferred embodiments of the application have been described, a wide variety of modifications, alterations, and permutations can be made to the described embodiments by those of ordinary skill in the art now that the general nature of the application is known. That is, although particular preferred embodiments of the present application have been described, various modifications and changes can be made by those skilled in the art without departing from the scope of the present application as set forth in the claims below. Accordingly, the appended claims are intended to embrace all such alterations, permutations, and modifications as fall within the scope of the application, including the patent claims as interpreted by the judicial or patent office officials, administrators, and representatives.

[0168] Finally, it should be noted that the terms "comprises", "comprising", or other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article, or apparatus. An element proceeded by "comprises... a" does not, without more constraints, exclude the presence of additional identical elements in the process, method, article, or apparatus that comprises the element.

[0169] The above provides a kind of based on text semantic information fidelity of the present application, the super low code rate speech coding system, is introduced in detail, the principle and implementation mode of the present application are described in this paper with specific examples, the above example is only for helping to understand the method of the present application and its core idea;For the general technical personnel of the field, according to the idea of the present application, there will be changes in specific implementation mode and application range, the above-mentioned content of the specification should not be understood as the limitation of the present application.

Claims

1. A low-bitrate speech codec system based on text semantic information fidelity, characterized in that, include: In the semantic communication system, the sending end extracts speech features and text features from the original speech through a multimodal text-speech joint encoder to obtain the speech features and text features of the original speech. The text features of the original speech are then embedded into the speech features of the original speech to obtain the text-speech features of the original speech. Finally, the text-speech features of the original speech are sent to the receiving end in the semantic communication system. The receiving end inputs the received text-speech features into a text semantic fidelity decoder based on an attention mechanism to obtain reconstructed speech features and reconstructed text features. It then decodes the reconstructed speech features layer by layer and performs attention calculations with the reconstructed text features to obtain decoded speech. The attention calculations are used to provide display guidance for obtaining the reconstructed text features from the speech features decoded at each layer. The attention-based text semantic fidelity decoder includes: a first convolutional layer, N decoding modules, N text attention modules, a first activation function layer, a first output projection layer, and a second activation function layer; The receiving end processes the received text-speech features through the first convolutional layer to obtain reconstructed speech features and reconstructed text features. The receiving end upsamples the reconstructed speech features layer by layer through the N decoding modules; The receiving end uses the reconstructed text features as K and V of the nth text attention module, and the upsampled speech features output by the nth decoding module as Q, to perform attention calculation, thereby obtaining the enhanced speech features output by the nth text attention module after adding text semantic information, and uses it as the input of the (n+1)th decoding module. The receiving end uses the first activation function layer, the first output projection layer, and the second activation function layer to process the output of the last text attention module to obtain decoded speech.

2. The ultra-low bitrate speech codec system based on text semantic information fidelity according to claim 1, characterized in that, The multimodal text-speech co-encoder includes: a speech encoder, a text encoder, and a text-speech alignment and splicing module; The speech encoder includes a second convolutional layer, N encoding modules, a first activation function layer, and a second output projection layer. The transmitting end maps the original speech into a multi-dimensional feature space through the second convolutional layer while maintaining the time dimension of the original speech. The transmitting end downsamples the output of the second convolutional layer layer by layer through the N encoding modules to achieve speech feature extraction from local to global. The transmitting end models the periodic features in the original speech through the first activation function layer and the second output projection layer to obtain the speech features of the original speech. The transmitting end extracts logarithmic Mel features from the original speech using the text encoder to obtain the Mel spectrogram of the original speech, and maps the original speech into a multidimensional phoneme space based on the Mel spectrogram of the original speech to obtain the text features of the original speech. The sending end uses the text-speech alignment and splicing module to align the text features of the original speech in the time dimension by maintaining the relative position, and then splices the aligned text features with the speech features of the original speech in the channel dimension to obtain the text-speech features.

3. The ultra-low bitrate speech codec system based on text semantic information fidelity according to claim 1, characterized in that, The method executed by the ultra-low bitrate speech codec system based on text semantic information fidelity is implemented through an ultra-low bitrate speech codec model based on text semantic information fidelity. The ultra-low bitrate speech codec model based on text semantic information fidelity includes at least: the multimodal text-speech joint encoder and the text semantic fidelity decoder based on the attention mechanism. The loss function values ​​of the ultra-low bitrate speech codec model based on text semantic information fidelity include at least: time-domain loss function values ​​and frequency-domain loss function values; The temporal loss function value is used to: ensure that the sample decoded speech output by the attention-based text semantic fidelity decoder for the original sample speech is consistent with the original sample speech input to the multimodal text-speech joint encoder in the temporal domain; The frequency domain loss function value is used to: promote consistency between the Mel spectral features of the decoded speech sample and the Mel spectral features of the original speech sample in the frequency domain.

4. The ultra-low bitrate speech codec system based on text semantic information fidelity according to claim 3, characterized in that, The ultra-low bitrate speech codec model based on text semantic information fidelity further includes: a residual vector quantizer; the sending end sends the text-speech features of the original speech to the receiving end in the semantic communication system, including: The transmitting end uses the residual vector quantizer to quantize the text-speech features of the original speech according to the codebook set, obtains the codebook index, and sends the codebook index to the receiving end. Before inputting the received text-speech features into the attention-based text semantic fidelity decoder, the receiving end dequantizes the received codebook index according to the codebook set to obtain the received text-speech features. The loss function value of the ultra-low bitrate speech codec model based on text semantic information fidelity also includes: the commitment loss function value and the codebook loss function value corresponding to the residual vector quantizer; The committed loss function value is used to: make the output of the multimodal text-speech joint encoder closer to the nearest codebook vector; The codebook loss function value is used to: prompt the codebook vector to be updated to match the output of the multimodal text-speech co-encoder.

5. The ultra-low bitrate speech codec system based on text semantic information fidelity according to claim 3, characterized in that, The training phase of the ultra-low bitrate speech codec model based on text semantic information fidelity includes a first training phase and a second training phase. The first training phase is used to pre-train the text encoder to be trained in the multimodal text-speech joint encoder to be trained, and specifically includes the following steps: The original speech sample is input into the text encoder to be trained to obtain the recognized text corresponding to the original speech sample. Based on the real text labels corresponding to the original speech of the sample and the recognized text corresponding to the original speech of the sample, the first CTC loss function value is obtained; Based on the first CTC loss function value, the parameters of the text encoder to be trained are updated to obtain a pre-trained text encoder, and the parameters of the pre-trained text encoder are frozen. The second training phase is used to jointly train the attention-based text semantic fidelity decoder to be trained, the residual vector quantizer to be trained, and the speech encoder to be trained in the multimodal text-speech joint encoder to be trained.

6. The ultra-low bitrate speech codec system based on text semantic information fidelity according to claim 5, characterized in that, Using the attention-based text semantic fidelity decoder, the residual vector quantizer, and the multimodal text-speech joint encoder to be trained as generators, a discriminator is connected after the attention-based text semantic fidelity decoder to be trained, and the second training stage is trained using an adversarial training method; wherein, the discriminator is used to distinguish between real speech and synthesized speech. The loss function value of the ultra-low bitrate speech codec model based on text semantic information fidelity also includes: a generator loss function value, which is used to make the speech generated by the generator more realistic until the discriminator can no longer distinguish between real speech and synthesized speech.

7. The ultra-low bitrate speech codec system based on text semantic information fidelity according to claim 6, characterized in that, The discriminator includes at least: A multi-scale STFT discriminator is used to process the sample decoded speech output by the attention-based text semantic fidelity decoder for the original sample speech, to obtain a complex STFT spectral feature map, and to divide the complex STFT spectral feature map into sub-bands to capture the acoustic characteristics of different frequency ranges. Furthermore, a discrimination result is obtained based on the STFT spectral feature map of each sub-band. The loss function value of the ultra-low bitrate speech codec model based on text semantic information fidelity also includes a multi-scale STFT discriminator loss function value, used to ensure that the spectral features of the speech generated by the generator at multiple scales and resolutions are consistent with the spectral features of real speech at multiple scales and resolutions.

8. The ultra-low bitrate speech codec system based on text semantic information fidelity according to claim 7, characterized in that, The discriminator further includes: A multi-period discriminator is used to rearrange the sample decoded speech according to different periods to generate two-dimensional representations with different time granularities. Based on the two-dimensional representations with different time granularities, the discrimination result is obtained. The loss function value of the ultra-low bitrate speech codec model based on text semantic information fidelity also includes: the multi-period discriminator loss function value, which is used to ensure that the temporal periodic structure of the speech generated by the generator is consistent with the temporal periodic structure of the real speech.

9. The ultra-low bitrate speech codec system based on text semantic information fidelity according to claim 6, characterized in that, A pre-trained recognizer is connected after the attention-based text semantic fidelity decoder to be trained; The loss function value of the ultra-low bitrate speech codec model based on text semantic information fidelity also includes: a second CTC loss function value, which is used to ensure that the recognized text corresponding to the speech generated by the generator maintains semantic consistency with the real text label corresponding to the original sample speech input to the generator.

Images