Semantic communication method and system, computer equipment and storage medium

Abstract

The invention belongs to the technical field of audio processing, and relates to a semantic communication method and system, computer equipment and a storage medium. The method comprises the following steps: a sending terminal carries out vector conversion operation on input text data according to a text word segmentation device, carries out feature extraction operation on voice sample data according to a sending semantic knowledge base, and carries out residual feature extraction operation according to a Transform encoder; obtaining a first input text mark vector, a first voice sample residual feature and a voice sample index code; sending the first input text mark vector t, the first voice sample residual feature r and the voice sample index code to a receiving terminal; and the receiving terminal performs channel decoding operation and voice synthesis operation to obtain target voice data corresponding to the input text data. According to the method, the semantic knowledge bases are constructed at the sending end and the receiving end respectively, the sending end is responsible for efficient semantic extraction, and the receiving end is used for realizing natural and smooth speech synthesis, so that speech data with remarkable fidelity is generated.

Description

Technical Field

[0001] The present application relates to the field of audio processing technology, and in particular to a semantic communication method, system, computer device and storage medium. Background Art

[0002] In recent years, the rise of generative artificial intelligence techniques has stimulated people's interest in generative tasks such as image-to-video generation and text-to-speech (TTS) synthesis. To meet the diverse needs of generative tasks, a new generative semantic communication paradigm is proposed, which aims to generate the required content based on the received signal and the semantic knowledge base. Specifically, the semantic knowledge base is able to learn the intrinsic structure of the data and generate new content that does not appear in the training data. Due to the significant advantages of the semantic knowledge base, generative semantic communication provides a novel solution to perform various tasks such as image generation and remote monitoring.

[0003] This application focuses on the text-to-speech (TTS) synthesis task in a point-to-point communication environment, aiming to generate natural and fluent speech at the receiving end, which requires the sending end to provide speech samples and text information. There are usually two ways to achieve this goal: one is to generate speech directly at the sending end and transmit it to the receiving end; the other is to encode the speech samples and text into digital signals for transmission, and then generate speech at the receiving end based on the received signals.

[0004] However, the applicant has found that both of the above two methods have the problem of high communication overhead. Summary of the invention

[0005] The purpose of the embodiments of the present application is to propose a semantic communication method, system, computer device and storage medium to solve the problem of high communication overhead in traditional methods.

[0006] In order to solve the above technical problems, the present application embodiment provides a semantic communication method, which adopts the following technical solution:

[0007] The sending terminal receives a semantic communication request sent by a user terminal, wherein the semantic communication request includes input text data and voice sample data;

[0008] The sending terminal performs a vector conversion operation on the input text data according to the text word segmenter to obtain a first input text tag vector Among them, d t represents the dimension of the label vector;

[0009] The sending terminal performs a feature extraction operation on the voice sample data according to the sending semantic knowledge base to obtain a first voice sample feature code And the speech sample index code, where d ω Indicates the dimension of the feature code;

[0010] The sending terminal generates the first speech sample feature code according to the Transformer encoder Perform residual feature extraction to obtain the residual features of the first speech sample Among them, d r represents the dimension of the residual vector;

[0011] The transmitting terminal integrates the first input text tag vector t, the first voice sample residual feature r and the voice sample index code to obtain a communication transmission signal Among them, d x represents the dimension of the channel encoder output;

[0012] The transmitting terminal transmits the communication transmission signal x to the receiving terminal;

[0013] The receiving terminal performs a channel decoding operation on the communication transmission signal x to obtain a decoded second input text tag vector The residual features of the second speech sample and the voice sample index code;

[0014] The receiving terminal performs an inverse feature extraction operation on the voice sample index code according to the received semantic knowledge base to obtain a second voice sample feature code.

[0015] The receiving terminal converts the second input text label vector The second speech sample residual feature And the second voice sample feature code The text data is input to a semantic decoder for speech synthesis to obtain target speech data corresponding to the input text data.

[0016] Furthermore, the sending semantic knowledge base includes a WavLM speech model, a converter encoder, and a residual vector quantization encoder. The sending terminal performs a feature extraction operation on the speech sample data according to the sending semantic knowledge base to obtain a first speech sample feature code. And the step of obtaining the voice sample index code specifically includes the following steps:

[0017] Perform a feature extraction operation on the speech sample data according to the WavLM speech model to obtain the first speech sample feature code ω;

[0018] According to the converter encoder, the first speech sample feature code ω is subjected to a vector conversion operation to obtain a source vector

[0019] The source vector z1 is sequentially quantized according to the residual vector quantization encoder to obtain the speech sample index code z i+1 .

[0020] Furthermore, the transmission semantic knowledge base also includes a feature codebook corresponding to the residual vector quantization encoder, and the speech sample index code z i+1 It is expressed as:

[0021]

[0022] Here, ‖·‖ represents the Euclidean distance.

[0023] Furthermore, the semantic decoder includes a priori encoder, a diffusion model and a SoundStream module, and the receiving terminal converts the second input text tag vector The second speech sample residual feature And the second voice sample feature code The step of inputting the data into a semantic decoder for speech synthesis operation to obtain target speech data corresponding to the input text data specifically comprises the following steps:

[0024] The second input text token vector Input to the a priori encoder to obtain the adjustment information μ for controlling the diffusion model speech synthesis process;

[0025] Inputting the adjustment information μ into the diffusion model to obtain SoundStream features;

[0026] The SoundStream features are converted into the target voice data according to the SoundStream module.

[0027] Further, the a priori encoder includes a text encoder, an expansion module and a pitch predictor, and the second input text tag vector The step of inputting the information into the a priori encoder to obtain the adjustment information μ for controlling the diffusion model speech synthesis process specifically includes the following steps:

[0028] The second input text token vector Input to the text encoder to obtain the phoneme-level hidden sequence h p and duration vector d1;

[0029] The phoneme-level hidden sequence h is transformed into a hidden sequence according to the expansion module and the duration vector d1. p Perform expansion operation to obtain the frame-level hidden sequence h f ;

[0030] The pitch predictor is used to predict the residual features of the second speech sample The second voice sample feature code And the frame level hidden sequence h f Perform a pitch prediction operation to obtain the pitch information p1 of the target speech;

[0031] The frame-level hidden sequence h f And the pitch information p1 is spliced ​​to obtain the adjustment information μ.

[0032] Furthermore, the step of inputting the adjustment information μ into the diffusion model to obtain the SoundStream feature specifically includes the following steps:

[0033] According to the first stochastic differential equation, the adjustment information μ is forward diffused to obtain Gaussian noise S T ;

[0034] According to the second stochastic differential equation, the Gaussian noise S T A backward diffusion operation is performed to obtain the SoundStream feature.

[0035] In order to solve the above technical problems, the embodiment of the present application also provides a semantic communication system, which adopts the following technical solution:

[0036] The sending terminal is used to receive a semantic communication request sent by a user terminal, wherein the semantic communication request includes input text data and voice sample data;

[0037] The sending terminal is further used to perform a vector conversion operation on the input text data according to the text word segmenter to obtain a first input text tag vector Among them, d t represents the dimension of the label vector;

[0038] The sending terminal is further used to perform a feature extraction operation on the voice sample data according to the sending semantic knowledge base to obtain a first voice sample feature code And the speech sample index code, where d ω Indicates the dimension of the feature code;

[0039] The sending terminal is further configured to generate a feature code of the first speech sample according to a Transformer encoder. Perform residual feature extraction to obtain the residual features of the first speech sample Among them, d r represents the dimension of the residual vector;

[0040] The sending terminal is further used to integrate the first input text tag vector t, the first voice sample residual feature r and the voice sample index code to obtain a communication transmission signal;

[0041] The sending terminal is further used to send the communication transmission signal to the receiving terminal;

[0042] The receiving terminal is used to perform a channel decoding operation on the communication transmission signal to obtain a decoded second input text tag vector The residual features of the second speech sample and the voice sample index code;

[0043] The receiving terminal is further used to perform an inverse feature extraction operation on the voice sample index code according to the received semantic knowledge base to obtain a second voice sample feature code

[0044] The receiving terminal is further configured to convert the second input text mark vector The second speech sample residual feature And the second voice sample feature code The text data is input to a semantic decoder for speech synthesis to obtain target speech data corresponding to the input text data.

[0045] Further, the sending semantic knowledge base includes a WavLM speech model, a converter encoder, and a residual vector quantization encoder, and the calculation module includes:

[0046] The sending terminal is further used to perform a feature extraction operation on the voice sample data according to the WavLM voice model to obtain the first voice sample feature code ω;

[0047] The transmitting terminal is further used to perform a vector conversion operation on the first speech sample feature code ω according to the converter encoder to obtain a source vector

[0048] The transmitting terminal is further used to perform a sequential quantization operation on the source vector z1 according to the residual vector quantization encoder to obtain the speech sample index code z i+1 .

[0049] In order to solve the above technical problems, the embodiment of the present application further provides a computer device, which adopts the following technical solution:

[0050] The method comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the semantic communication method as described above when executing the computer-readable instructions.

[0051] In order to solve the above technical problems, the embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solution:

[0052] The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the steps of the semantic communication method described above are implemented.

[0053] The present application provides a semantic communication method, comprising: a sending terminal receiving a semantic communication request sent by a user terminal, wherein the semantic communication request includes input text data and voice sample data; the sending terminal performing a vector conversion operation on the input text data according to a text segmenter to obtain a first input text tag vector Among them, d t Represents the dimension of the tag vector; the sending terminal performs a feature extraction operation on the voice sample data according to the sending semantic knowledge base to obtain a first voice sample feature code And the speech sample index code, where d ω The transmitting terminal encodes the first speech sample feature code according to the Transformer encoder Perform residual feature extraction to obtain the residual features of the first speech sample Among them, d r The transmitting terminal integrates the first input text tag vector t, the first voice sample residual feature r and the voice sample index code to obtain a communication transmission signal; the transmitting terminal sends the communication transmission signal to the receiving terminal; the receiving terminal performs a channel decoding operation on the communication transmission signal to obtain a decoded second input text tag vector The residual features of the second speech sample and the voice sample index code; the receiving terminal performs an inverse feature extraction operation on the voice sample index code according to the received semantic knowledge base to obtain a second voice sample feature code The receiving terminal converts the second input text label vector The second speech sample residual feature And the second voice sample feature code The target speech data corresponding to the input text data is input to the semantic decoder for speech synthesis operation. Compared with the prior art, the present application firstly uses a pre-trained large speech model and residual vector quantization technology to build a semantic knowledge base at the sending end and the receiving end respectively. The sending end is responsible for efficient semantic extraction, while the receiving end is used to achieve natural and fluent speech synthesis, thereby generating speech data with significant fidelity. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0055] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;

[0056] Figure 2 is a flowchart of the implementation of the semantic communication method provided in the embodiment of the present application;

[0057] Figure 3 It is a structural diagram of a model of a point-to-point communication system provided in an embodiment of the present application;

[0058] Figure 4 It is a schematic diagram of the structure of the multi-scale hierarchical model provided in the embodiment of the present application;

[0059] Figure 5 is a structural diagram of a semantic communication system provided in an embodiment of the present application;

[0060] Figure 6 It is a structural diagram of an embodiment of a computer device according to the present application. DETAILED DESCRIPTION

[0061] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0062] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0063] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0064] like Figure 1 As shown, the system architecture 100 may include a terminal device 101, a network 102 and a server 103. The terminal device 101 may be a laptop 1011, a tablet computer 1012 or a mobile phone 1013. The network 102 is used to provide a medium for a communication link between the terminal device 101 and the server 103. The network 102 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0065] The user can use the terminal device 101 to interact with the server 103 through the network 102 to receive or send messages, etc. Various communication client applications can be installed on the terminal device 101, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0066] The terminal device 101 can be any electronic device with a display screen and supporting web browsing. In addition to the laptop computer 1011, tablet computer 1012 or mobile phone 1013, the terminal device 101 can also be an e-book reader, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 (Moving Picture Experts Group Audio Layer IV), a laptop computer, a desktop computer, etc.

[0067] The server 103 may be a server that provides various services, such as a background server that provides support for web pages displayed on the terminal device 101 .

[0068] It should be noted that the semantic communication method provided in the embodiment of the present application is generally executed by a server / terminal device, and accordingly, the semantic communication system is generally set in the server / terminal device.

[0069] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0070] Continue to refer Figure 2 , showing a flow chart of an embodiment of a semantic communication method according to the present application. The semantic communication method comprises: step S201, step S202, step S203, step S204, step S205, step S206 and step S207.

[0071] In step S201, a sending terminal receives a semantic communication request sent by a user terminal, wherein the semantic communication request includes input text data and voice sample data.

[0072] In the embodiments of the present application, the user terminal refers to a terminal device used to execute the image processing method for preventing document abuse provided by the present application. The user terminal can be a mobile terminal such as a mobile phone, a smart phone, a laptop computer, a digital broadcast receiver, a PDA (personal digital assistant), a PAD (tablet computer), a PMP (portable multimedia player), a navigation device, etc., as well as a fixed terminal such as a digital TV, a desktop computer, etc. It should be understood that the examples of user terminals here are only for the convenience of understanding and are not used to limit the present application.

[0073] In the embodiments of this application, see Figure 3 The model structure diagram of the point-to-point communication system shown in the figure, the sending end and the receiving end each deploy a semantic knowledge base for semantic extraction and speech synthesis respectively.

[0074] In step S202, the sending terminal performs a vector conversion operation on the input text data according to the text segmenter to obtain a first input text tag vector Among them, d t Represents the dimension of the label vector.

[0075] In the embodiment of the present application, the text segmenter converts the input text data into a token vector Among them, d t Represents the dimension of the label vector.

[0076] In step S203, the sending terminal performs a feature extraction operation on the voice sample data according to the sending semantic knowledge base to obtain a first voice sample feature code And the speech sample index code, where d ω Indicates the dimension of the feature code.

[0077] In the embodiment of the present application, a speech sample is input into a transmitting end, and the transmitting end outputs a corresponding feature code and a code index.

[0078] In step S204, the sending terminal generates a first speech sample feature code according to the Transformer encoder. Perform residual feature extraction to obtain the residual features of the first speech sample Among them, d r Represents the dimension of the residual vector.

[0079] In the embodiment of the present application, the feature code contains the key information required for speech synthesis, which is then input into the Transformer encoder. The Transformer encoder extracts the residual information not covered by the feature code from the speech sample. Among them, d r Represents the dimension of the residual vector.

[0080] In the embodiment of the present application, the combination of the Transformer encoder and the text segmenter is defined as a semantic encoder.

[0081] In step S205, the sending terminal integrates the first input text tag vector t, the first voice sample residual feature r and the voice sample index code to obtain a communication transmission signal Among them, d x Represents the dimension of the channel encoder output.

[0082] In the embodiment of the present application, after the semantic encoding is completed, the tag vector t, the residual vector r and the code index are integrated into a data packet.

[0083] In step S206, the transmitting terminal sends the communication transmission signal x to the receiving terminal.

[0084] In some optional implementations of the embodiments of the present application, the data packet is encoded by a channel encoder before being sent to the receiving end. The present application can use a neural network to complete the channel coding task. The network consists of two modules connected in series, each module including a convolutional layer and a Transformer encoder following it. is the transmission signal, where d x Represents the dimension of the channel encoder output. The signal received by the receiver can be expressed as:

[0085] y=h(x)+n

[0086] Where y represents the received signal, h(x) represents the channel transfer function, and n represents the additive noise. The role of the channel encoder is to enhance the robustness of the signal so that it can resist the noise and interference in the channel, thereby ensuring that the receiving end can accurately receive the information sent by the sending end.

[0087] In step S207, the receiving terminal performs a channel decoding operation on the communication transmission signal x to obtain a decoded second input text tag vector The residual features of the second speech sample and the voice sample index code.

[0088] In an embodiment of the present application, at the receiving end, the received signal y will first be processed by a channel decoder to restore the original semantic coding information, and then this information will be used in the subsequent speech synthesis process.

[0089] In the embodiment of the present application, when the receiver receives the signal y, it performs a channel decoding process to extract the decoded label vector Residual vector and code index.

[0090] In step S208, the receiving terminal performs an inverse feature extraction operation on the voice sample index code according to the received semantic knowledge base to obtain a second voice sample feature code

[0091] In the embodiment of the present application, similar to the encoder, the decoder is also composed of two consecutive modules, each of which contains a transposed convolution layer and a transformer encoder. Next, the code index is fed into the receiver's knowledge base to reconstruct the speech features corresponding to the speech example, i.e., the dimension is d w of

[0092] In step S209, the receiving terminal marks the second input text vector The residual features of the second speech sample And the second voice sample feature code The data is input to the semantic decoder for speech synthesis operation to obtain target speech data corresponding to the input text data.

[0093] In the embodiments of the present application, is input into a semantic decoder to generate the target speech.

[0094] In an embodiment of the present application, a semantic communication method is provided, comprising: a sending terminal receiving a semantic communication request sent by a user terminal, wherein the semantic communication request includes input text data and voice sample data; the sending terminal performs a vector conversion operation on the input text data according to a text segmenter to obtain a first input text tag vector Among them, d t Represents the dimension of the tag vector; the sending terminal performs a feature extraction operation on the voice sample data according to the sending semantic knowledge base to obtain the first voice sample feature code And the speech sample index code, where d ω The dimension of the feature code is represented; the sending terminal generates the feature code of the first speech sample according to the Transformer encoder Perform residual feature extraction to obtain the residual features of the first speech sample Among them, d rrepresents the dimension of the residual vector; the sending terminal integrates the first input text tag vector t, the first voice sample residual feature r and the voice sample index code to obtain a communication transmission signal; the sending terminal sends the communication transmission signal to the receiving terminal; the receiving terminal performs a channel decoding operation on the communication transmission signal to obtain a decoded second input text tag vector The residual features of the second speech sample and a voice sample index code; the receiving terminal performs an inverse feature extraction operation on the voice sample index code according to the received semantic knowledge base to obtain a second voice sample feature code The receiving terminal converts the second input text tag vector The residual features of the second speech sample And the second voice sample feature code The data is input to the semantic decoder for speech synthesis operation to obtain the target speech data corresponding to the input text data. Compared with the prior art, the present application first uses a pre-trained large speech model and residual vector quantization technology to build a semantic knowledge base at the sending end and the receiving end respectively. The sending end is responsible for efficient semantic extraction, while the receiving end is used to achieve natural and fluent speech synthesis, thereby generating speech data with significant fidelity.

[0095] In some optional implementations of the embodiments of the present application, the above-mentioned sending semantic knowledge base includes a WavLM speech model, a converter encoder, and a residual vector quantization encoder. The sending terminal performs a feature extraction operation on the speech sample data according to the sending semantic knowledge base to obtain a first speech sample feature code. And the step of obtaining the voice sample index code specifically includes the following steps:

[0096] Perform feature extraction on the speech sample data according to the WavLM speech model to obtain a first speech sample feature code ω;

[0097] According to the converter encoder, the first speech sample feature code ω is subjected to a vector conversion operation to obtain a source vector

[0098] According to the residual vector quantization encoder, the source vector z1 is sequentially quantized to obtain the speech sample index code z i+1 .

[0099] In the embodiment of the present application, the transmission semantic knowledge base includes a large speech model named WavLM, a transformer encoder, N residual vector quantization (RVQ) encoders and N feature codebooks, where:

[0100] WavLM is a pre-trained large-scale speech model that can extract key information from speech and has been widely used to solve full-stack downstream speech tasks. Therefore, it serves as a key component of the transmitter knowledge base. Specifically, WavLM extracts a speech feature vector from the speech demonstration and uses It means that its dimension is d ω For convenience, this application refers to ω as the WavLM feature, which reflects the characteristics of the speech demonstration, such as pitch, timbre, and loudness.

[0101] In the embodiment of the present application, since the dimension of ω is usually large, the present application compresses it to improve communication efficiency. Since the speech feature ω has a combined hierarchical structure, the present application proposes a multi-scale hierarchical model to compress it. Figure 4 As shown, first, the converter encoder converts ω into a source vector Next, z1 is sequentially quantized by N RVQ encoders, where each RVQ encoder extracts the remaining speech information from the output of the previous RVQ encoder. i represents the input of the i-th RVQ encoder. Then, its output is defined as:

[0102] z i =Q i (z i )

[0103] Among them, Q i represents the quantization function of the i-th RVQ encoder.

[0104] In the embodiment of the present application, in each RVQ encoder, the quantization process can be further decomposed into multiple steps, each step corresponding to a specific feature codebook. i represents the feature codebook of the i-th RVQ encoder, which contains K i codewords. Then, the quantization process can be expressed as:

[0105]

[0106] Here, ‖·‖ represents the Euclidean distance.

[0107] In this embodiment of the present application, in this way, the original WavLM feature ω is compressed into a more compact representation z N+1 , which retains the key information of the original features while significantly reducing the amount of data. This compressed representation is then used for subsequent communication and task processing.

[0108] In some optional implementations of the embodiments of the present application, in order to restore the original WavLM feature ω, the present application needs to perform a corresponding inverse process at the receiving end. First, use N RVQ decoders to decode zN+1 Perform inverse quantization, and each decoder corresponds to an RVQ encoder. represents the output of the i-th RVQ decoder. Then, its input is defined as:

[0109]

[0110] Among them, D i represents the inverse quantization function of the i-th RVQ decoder.

[0111] Finally, the transformer decoder recovers the source vector Convert back to original WavLM features Complete the entire compression and decompression process.

[0112] With this design, the semantic knowledge base can not only effectively compress and transmit speech features but also maintain its semantic information, thus supporting various downstream speech tasks.

[0113] In addition, each RVQ encoder is connected to a specific feature codebook, which consists of M feature codes for speech representation. i =[C i ,1,C i ,2,...,C i ,M] represents the feature codebook associated with the i-th RVQ encoder, where C i , M represents the mth feature code. Next, the output z of each RVQ encoder is e Mapping to a feature code through the nearest neighbor search method is given by the following formula:

[0114]

[0115] Among them, ‖·‖ represents the Euclidean distance, z e+1 It is the feature code after mapping.

[0116] In some optional implementations of the embodiments of the present application, in order to further explain the above process, the present application may consider how each RVQ encoder uses its feature codebook for quantization. Assume that z e is the input vector of the e-th RVQ encoder. The quantization process includes the following steps:

[0117] 1. Calculate the distance: For the feature code book C i Each feature code C in i ,m, calculate its difference with the input vector z e The Euclidean distance between .

[0118] d i,m =‖z e -Ci ,m‖

[0119] 2. Select the nearest neighbor: From all the calculated distances d i,m In the above example, the feature code corresponding to the distance with the smallest value is selected as the nearest neighbor.

[0120]

[0121] Among them, m * =argmin m d i,m .

[0122] 3. Update input: Update the selected feature code z e+1 As input to the next RVQ encoder, or in the case of the last RVQ encoder, as the compressed representation.

[0123] In this way, each RVQ encoder can effectively extract key information from the input vector and represent it as a codeword in the feature codebook. This method not only reduces the dimension of the data, but also retains the important features in the speech signal, making subsequent speech processing tasks more efficient.

[0124] At the receiving end, the inverse process is used to recover the original speech features from the compressed representation. is the received feature code, and the inverse quantization process includes the following steps:

[0125] 1. Find the signature code: In the signature book C i Find and The corresponding feature code.

[0126]

[0127] Among them, m * is the index of the received feature code in the codebook.

[0128] 2. Restore the input vector: Restore the feature code As input to the next RVQ decoder, or in the case of the last RVQ decoder, as the recovered original speech features.

[0129] Through this inverse process, the receiver can recover the original speech features from the compressed representation, thereby supporting various speech processing tasks.

[0130] In some optional implementations of the embodiments of the present application, the semantic decoder includes a priori encoder, a diffusion model and a SoundStream module, and the receiving terminal converts the second input text tag vector The residual features of the second speech sample And the second voice sample feature code The step of inputting the data into a semantic decoder for speech synthesis operation to obtain target speech data corresponding to the input text data specifically includes the following steps:

[0131] The second input text token vector Input to the prior encoder to obtain the regulation information for controlling the diffusion model speech synthesis process;

[0132] The second speech sample residual features of the adjustment information And the second voice sample feature code Input to the diffusion model to obtain SoundStream features;

[0133] The SoundStream features are converted into target speech data according to the SoundStream module.

[0134] In the embodiment of the present application, the semantic decoder consists of three modules, including a priori encoder, a diffusion model and a SoundStream.

[0135] In an embodiment of the present application, the a priori encoder is intended to generate regulation information for controlling the diffusion model speech synthesis process.

[0136] In the embodiment of the present application, the diffusion model is intended to generate SoundStream features related to the target speech.

[0137] In the embodiment of the present application, the SoundStream module is responsible for converting the features generated by the diffusion model into the final audio signal. It adopts a stream-based generative model architecture that can efficiently process high-dimensional audio data and generate high-quality speech output.

[0138] In the embodiment of the present application, through the collaborative work of these three modules, the semantic decoder can convert text and speech features into high-quality speech signals, thereby achieving the task of speech synthesis.

[0139] In some optional implementations of the embodiments of the present application, the a priori encoder includes a text encoder, an expansion module, and a pitch predictor, which converts the second input text tag vector The step of inputting the information to the prior encoder to obtain the adjustment information μ for controlling the diffusion model speech synthesis process specifically includes the following steps:

[0140] The second input text token vector Input to the text encoder to get the phoneme-level hidden sequence h p and duration vector d1;

[0141] According to the expansion module and the duration vector d1, the phoneme-level hidden sequence h p Perform expansion operation to obtain the frame-level hidden sequence h f ;

[0142] The residual features of the second speech sample are analyzed by the pitch predictor The second voice sample feature code And the frame-level hidden sequence h f Perform a pitch prediction operation to obtain the pitch information p1 of the target speech;

[0143] The frame-level hidden sequence h f And the pitch information p1 are spliced ​​to obtain the adjustment information μ.

[0144] In the embodiment of the present application, the prior encoder is composed of a text encoder, an expansion module and a pitch predictor. The text encoder and the pitch predictor adopt a transformer encoder structure. Specifically, by reconstructing the WavLM feature Residual vector and marker vector As input, the text encoder outputs a phoneme-level hidden sequence h p and a duration vector d1 of a phoneme. Next, according to the duration vector d1, h p Expanded to the frame-level hidden sequence h f . The pitch predictor then uses and h f To predict the pitch information p1 of the target speech. Finally, p1 and h f They are added together to obtain the final adjustment information, denoted as μ.

[0145] In some optional implementations of the embodiments of the present application, the step of inputting the adjustment information μ into the diffusion model to obtain the SoundStream feature specifically includes the following steps:

[0146] According to the first stochastic differential equation, the adjustment information μ is forward diffused to obtain Gaussian noise S T ;

[0147] According to the second stochastic differential equation, the Gaussian noise S T Perform a backward diffusion operation to obtain the SoundStream feature.

[0148] In the embodiment of the present application, the working mechanism of the diffusion model includes a forward process and a backward process, which can be expressed as two stochastic differential equations. Specifically, the forward process is a real SoundStream feature S g0 Add noise and convert it to Gaussian noise S T, where T∈[0,1] is the normalization step. The backward process starts from S T Remove noise from and predict SoundStream features, denoted as S0. The forward process can be expressed as:

[0149]

[0150] Among them, β t is the noise level at time step t, and ∈t is the noise sampled from a standard normal distribution.

[0151] The backward process aims to recover the original SoundStream feature S0 from the noise and can be expressed as:

[0152]

[0153] where ∈θ is the prediction noise function parameterized by the neural network, σ t is the standard deviation associated with the time steps and z is random noise sampled from a standard normal distribution.

[0154] In this way, the diffusion model is able to gradually remove the noise and predict the SoundStream features corresponding to the target speech.

[0155] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0156] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0157] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through computer-readable instructions, and the computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0158] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0159] Further references Figure 5 , as a response to the above Figure 2 The present application provides an embodiment of a semantic communication system, and the device embodiment is similar to Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0160] like Figure 5 As shown, the semantic communication system 200 of the embodiment of the present application includes: a sending terminal 210 and a receiving terminal 220, wherein:

[0161] The sending terminal 210 is used to receive a semantic communication request sent by a user terminal, wherein the semantic communication request includes input text data and voice sample data;

[0162] The sending terminal 210 is further configured to perform a vector conversion operation on the input text data according to the text word segmenter to obtain a first input text tag vector Among them, d t represents the dimension of the label vector;

[0163] The sending terminal 210 is also used to perform a feature extraction operation on the voice sample data according to the sending semantic knowledge base to obtain a first voice sample feature code And the speech sample index code, where d ω Indicates the dimension of the feature code;

[0164] The sending terminal 210 is further configured to generate a first speech sample feature code according to the Transformer encoder. Perform residual feature extraction to obtain the residual features of the first speech sample Among them, d r represents the dimension of the residual vector;

[0165] The sending terminal 210 is further used to integrate the first input text tag vector t, the first voice sample residual feature r and the voice sample index code to obtain a communication transmission signal;

[0166] The sending terminal 210 is also used to send the communication transmission signal to the receiving terminal 220;

[0167] The receiving terminal 220 is used to perform a channel decoding operation on the communication transmission signal to obtain a decoded second input text tag vector The residual features of the second speech sample and a voice sample index code;

[0168] The receiving terminal 220 is further configured to perform an inverse feature extraction operation on the voice sample index code according to the received semantic knowledge base to obtain a second voice sample feature code.

[0169] The receiving terminal 220 is further configured to convert the second input text tag vector The residual features of the second speech sample And the second voice sample feature code The data is input to the semantic decoder for speech synthesis operation to obtain target speech data corresponding to the input text data.

[0170] In an embodiment of the present application, a semantic communication system 200 is provided, including: a sending terminal 210, configured to receive a semantic communication request sent by a user terminal, wherein the semantic communication request includes input text data and voice sample data; the sending terminal 210 is further configured to perform a vector conversion operation on the input text data according to a text segmenter to obtain a first input text tag vector Among them, d t The sending terminal 210 is also used to perform a feature extraction operation on the voice sample data according to the sending semantic knowledge base to obtain a first voice sample feature code And the speech sample index code, where d ω The transmitting terminal 210 is also used to perform the feature code of the first speech sample according to the Transformer encoder. Perform residual feature extraction to obtain the residual features of the first speech sample Among them, d r The transmitting terminal 210 is further used to integrate the first input text tag vector t, the first voice sample residual feature r and the voice sample index code to obtain a communication transmission signal; the transmitting terminal 210 is further used to send the communication transmission signal to the receiving terminal 220; the receiving terminal 220 is used to perform a channel decoding operation on the communication transmission signal to obtain a decoded second input text tag vector The residual features of the second speech sample The receiving terminal 220 is also used to perform an inverse feature extraction operation on the voice sample index code according to the received semantic knowledge base to obtain a second voice sample feature code. The receiving terminal 220 is further configured to convert the second input text tag vector The residual features of the second speech sample And the second voice sample feature code The data is input to the semantic decoder for speech synthesis operation to obtain the target speech data corresponding to the input text data. Compared with the prior art, the present application first uses a pre-trained large speech model and residual vector quantization technology to build a semantic knowledge base at the sending end and the receiving end respectively. The sending end is responsible for efficient semantic extraction, while the receiving end is used to achieve natural and fluent speech synthesis, thereby generating speech data with significant fidelity.

[0171] In some optional implementations of the embodiments of the present application, the sending semantic knowledge base includes a WavLM speech model, a converter encoder, a residual vector quantization encoder, and the calculation module includes:

[0172] The sending terminal is further used to perform a feature extraction operation on the voice sample data according to the WavLM voice model to obtain a first voice sample feature code ω;

[0173] The transmitting terminal is further configured to perform a vector conversion operation on the first speech sample feature code ω according to the converter encoder to obtain a source vector

[0174] The transmitting terminal is also used to perform sequential quantization operations on the source vector z1 according to the residual vector quantization encoder to obtain the speech sample index code z i+1 .

[0175] To solve the above technical problems, the present application also provides a computer device. Figure 6 , Figure 6 This is a basic structural block diagram of a computer device according to an embodiment of the present application.

[0176] The computer device 300 includes a memory 310, a processor 320, and a network interface 330 that are interconnected and communicated through a system bus. It should be noted that the figure only shows a computer device 300 having components 310-330, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (Application Specific Integrated Circuit, ASIC), programmable gate arrays (Field-Programmable Gate Array, FPGA), digital processors (Digital Signal Processor, DSP), embedded devices, etc.

[0177] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.

[0178] The memory 310 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 310 can be an internal storage unit of the computer device 300, such as a hard disk or memory of the computer device 300. In other embodiments, the memory 310 can also be an external storage device of the computer device 300, such as a plug-in hard disk equipped on the computer device 300, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card, etc. Of course, the memory 310 can also include both the internal storage unit of the computer device 300 and its external storage device. In the embodiment of the present application, the memory 310 is generally used to store the operating system and various application software installed on the computer device 300, such as computer-readable instructions of the semantic communication method, etc. In addition, the memory 310 can also be used to temporarily store various data that have been output or are to be output.

[0179] The processor 320 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip in some embodiments. The processor 320 is generally used to control the overall operation of the computer device 300. In the embodiment of the present application, the processor 320 is used to run the computer-readable instructions stored in the memory 310 or process data, such as computer-readable instructions for running the semantic communication method.

[0180] The network interface 330 may include a wireless network interface or a wired network interface. The network interface 330 is generally used to establish a communication connection between the computer device 300 and other electronic devices.

[0181] The computer device provided in this application first uses a pre-trained large-scale speech model and residual vector quantization technology to build a semantic knowledge base at the sending end and the receiving end respectively. The sending end is responsible for efficient semantic extraction, while the receiving end is used to achieve natural and fluent speech synthesis, thereby generating speech data with significant fidelity.

[0182] The present application also provides another embodiment, namely, providing a computer-readable storage medium, wherein the computer-readable storage medium stores computer-readable instructions, and the computer-readable instructions can be executed by at least one processor to enable the at least one processor to perform the steps of the semantic communication method as described above.

[0183] The computer-readable storage medium provided in the present application first utilizes a pre-trained large-scale speech model and residual vector quantization technology to construct a semantic knowledge base at the sending end and the receiving end respectively. The sending end is responsible for efficient semantic extraction, while the receiving end is used to achieve natural and fluent speech synthesis, thereby generating speech data with significant fidelity.

[0184] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0185] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.

Claims

1. A semantic communication method, characterized in that , including the following steps: The sending terminal receives a semantic communication request sent by a user terminal, wherein the semantic communication request includes input text data and voice sample data; The sending terminal performs a vector conversion operation on the input text data according to the text word segmenter to obtain a first input text tag vector Among them, d t represents the dimension of the label vector; The sending terminal performs a feature extraction operation on the voice sample data according to the sending semantic knowledge base to obtain a first voice sample feature code And the speech sample index code, where d ω Indicates the dimension of the feature code; The sending terminal generates the first speech sample feature code according to the Transformer encoder Perform residual feature extraction to obtain the residual features of the first speech sample Among them, d r represents the dimension of the residual vector; The transmitting terminal integrates the first input text tag vector t, the first voice sample residual feature r and the voice sample index code to obtain a communication transmission signal Among them, d x represents the dimension of the channel encoder output; The transmitting terminal transmits the communication transmission signal x to the receiving terminal; The receiving terminal performs a channel decoding operation on the communication transmission signal x to obtain a decoded second input text tag vector The residual features of the second speech sample and the voice sample index code; The receiving terminal performs an inverse feature extraction operation on the voice sample index code according to the received semantic knowledge base to obtain a second voice sample feature code. The receiving terminal converts the second input text label vector The second speech sample residual feature And the second voice sample feature code The text data is input to a semantic decoder for speech synthesis to obtain target speech data corresponding to the input text data.

2. The semantic communication method according to claim 1, characterized in that: The sending semantic knowledge base includes a WavLM speech model, a converter encoder, and a residual vector quantization encoder. The sending terminal performs a feature extraction operation on the speech sample data according to the sending semantic knowledge base to obtain a first speech sample feature code. And the step of obtaining the voice sample index code specifically includes the following steps: Perform a feature extraction operation on the speech sample data according to the WavLM speech model to obtain the first speech sample feature code ω; According to the converter encoder, the first speech sample feature code ω is subjected to a vector conversion operation to obtain a source vector The source vector z1 is sequentially quantized according to the residual vector quantization encoder to obtain the speech sample index code z i+1 .

3. The semantic communication method according to claim 2, characterized in that: The transmission semantic knowledge base also includes a feature codebook corresponding to the residual vector quantization encoder, the speech sample index code z i+1 It is expressed as: Here, ‖·‖ represents the Euclidean distance.

4. The semantic communication method according to claim 1, characterized in that: The semantic decoder includes a priori encoder, a diffusion model and a SoundStream module. The receiving terminal converts the second input text tag vector The second speech sample residual feature And the second voice sample feature code The step of inputting the data into a semantic decoder for speech synthesis operation to obtain target speech data corresponding to the input text data specifically comprises the following steps: The second input text token vector Input to the a priori encoder to obtain the adjustment information μ for controlling the diffusion model speech synthesis process; Inputting the adjustment information μ into the diffusion model to obtain SoundStream features; The SoundStream features are converted into the target voice data according to the SoundStream module.

5. The semantic communication method according to claim 4, characterized in that: The a priori encoder comprises a text encoder, an expansion module and a pitch predictor, wherein the second input text token vector The step of inputting the information into the a priori encoder to obtain the adjustment information μ for controlling the diffusion model speech synthesis process specifically includes the following steps: The second input text token vector Input to the text encoder to obtain the phoneme-level hidden sequence h p and duration vector d1; The phoneme-level hidden sequence h is transformed into a hidden sequence according to the expansion module and the duration vector d1. p Perform an expansion operation to obtain the frame-level hidden sequence h f ; The pitch predictor is used to predict the residual features of the second speech sample The second voice sample feature code And the frame level hidden sequence h f Perform a pitch prediction operation to obtain the pitch information p1 of the target speech; The frame-level hidden sequence h f And the pitch information p1 is spliced ​​to obtain the adjustment information μ.

6. The semantic communication method according to claim 4, characterized in that: The step of inputting the adjustment information μ into the diffusion model to obtain the SoundStream feature specifically includes the following steps: According to the first stochastic differential equation, the adjustment information μ is forward diffused to obtain Gaussian noise S T ; According to the second stochastic differential equation, the Gaussian noise S T A backward diffusion operation is performed to obtain the SoundStream feature.

7. A semantic communication system, characterized in that: The system comprises a sending terminal and a receiving terminal, wherein: The sending terminal is used to receive a semantic communication request sent by a user terminal, wherein the semantic communication request includes input text data and voice sample data; The sending terminal is further used to perform a vector conversion operation on the input text data according to the text word segmenter to obtain a first input text tag vector Among them, d t represents the dimension of the label vector; The sending terminal is further used to perform a feature extraction operation on the voice sample data according to the sending semantic knowledge base to obtain a first voice sample feature code And the speech sample index code, where d ω Indicates the dimension of the feature code; The sending terminal is further configured to generate a feature code of the first speech sample according to a Transformer encoder. Perform residual feature extraction to obtain the residual features of the first speech sample Among them, d r represents the dimension of the residual vector; The sending terminal is further used to integrate the first input text tag vector t, the first voice sample residual feature r and the voice sample index code to obtain a communication transmission signal; The sending terminal is further used to send the communication transmission signal to the receiving terminal; The receiving terminal is used to perform a channel decoding operation on the communication transmission signal to obtain a decoded second input text tag vector The residual features of the second speech sample and the voice sample index code; The receiving terminal is further used to perform an inverse feature extraction operation on the voice sample index code according to the received semantic knowledge base to obtain a second voice sample feature code The receiving terminal is further configured to convert the second input text mark vector The second speech sample residual feature And the second voice sample feature code The text data is input to a semantic decoder for speech synthesis to obtain target speech data corresponding to the input text data.

8. The semantic communication system according to claim 7, characterized in that: The sending semantic knowledge base includes a WavLM speech model, a converter encoder, and a residual vector quantization encoder. The sending terminal is further used to perform a feature extraction operation on the speech sample data according to the WavLM speech model to obtain the first speech sample feature code ω; The transmitting terminal is further used to perform a vector conversion operation on the first speech sample feature code ω according to the converter encoder to obtain a source vector The transmitting terminal is further used to perform a sequential quantization operation on the source vector z1 according to the residual vector quantization encoder to obtain the speech sample index code z i+1 .

9. A computer device comprising a memory and a processor, characterized in that: The memory stores computer-readable instructions, and when the processor executes the computer-readable instructions, the steps of the semantic communication method according to any one of claims 1 to 6 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the steps of the semantic communication method according to any one of claims 1 to 6 are implemented.

Images