Voice reconstruction method and device, electronic equipment and storage medium

Abstract

The invention discloses a voice reconstruction method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining a coding feature of a first voice unit corresponding to a first moment, and obtaining a text prediction result of a second voice unit related to the first voice unit; obtaining an explicit semantic feature of the first voice unit based on the text prediction result of the second voice unit, and obtaining an implicit semantic feature of the first voice unit based on the coding feature of the first voice unit; fusing the explicit semantic feature and the implicit semantic feature to obtain a fused semantic feature of the first voice unit; and based on the fused semantic features of the first voice unit, synthesizing a reconstructed audio corresponding to the first moment. Through the method, the accuracy of audio reconstruction can be improved.

Description

Technical Field

[0001] The present application relates to the field of natural language processing, and in particular to a speech reconstruction method, device, electronic device and storage medium. Background Art

[0002] In the field of natural language processing, speech reconstruction technology aims to convert the abnormal prosody in speech audio of individuals with articulation disorders into normal prosody without changing the semantics or timbre. However, the pronunciation of the same text by individuals with different levels of articulation disorders varies significantly, resulting in a larger mapping space between speech audio and text, and consequently, poor accuracy in the generated reconstructed audio. Summary of the Invention

[0003] The main technical problem solved by this application is to provide a speech reconstruction method, device, electronic device and storage medium, which can improve the accuracy of reconstructed audio.

[0004] In order to solve the above technical problems, a technical solution adopted in this application is to provide a speech reconstruction method, which includes: obtaining the encoding features of a first speech unit corresponding to a first moment, and obtaining the text prediction results of a second speech unit related to the first speech unit; obtaining the explicit semantic features of the first speech unit based on the text prediction results of the second speech unit, and obtaining the implicit semantic features of the first speech unit based on the encoding features of the first speech unit; fusing the explicit semantic features and the implicit semantic features to obtain the fused semantic features of the first speech unit; and synthesizing the reconstructed audio corresponding to the first moment based on the fused semantic features of the first speech unit.

[0005] In order to solve the above technical problems, another technical solution adopted in the present application is: to provide a speech reconstruction device, which includes: a coding acquisition and text prediction module, which is used to obtain the coding features of the first speech unit corresponding to the first moment, and obtain the text prediction results of the second speech unit related to the first speech unit; an explicit and implicit semantic feature acquisition module, which is used to obtain the explicit semantic features of the first speech unit based on the text prediction results of the second speech unit, and to obtain the implicit semantic features of the first speech unit based on the coding features of the first speech unit; a fusion semantic acquisition module, which is used to fuse the explicit semantic features and the implicit semantic features to obtain the fusion semantic features of the first speech unit; and a reconstructed audio acquisition module, which is used to synthesize the reconstructed audio corresponding to the first moment based on the fusion semantic features of the first speech unit.

[0006] In order to solve the above technical problems, another technical solution adopted by the present application is: to provide an electronic device, which includes a memory and a processor, the memory stores program instructions, and the processor is used to execute the program instructions to implement the above-mentioned speech reconstruction method.

[0007] In order to solve the above technical problems, another technical solution adopted by the present application is: providing a computer-readable storage medium, which is used to store program instructions, and the program instructions can be executed to implement the above-mentioned speech reconstruction method.

[0008] The above scheme obtains the encoding features of a first speech unit corresponding to a first moment and obtains the text prediction results of a second speech unit related to the first speech unit. Based on the text prediction results of the second speech unit, explicit semantic features of the first speech unit are obtained, and based on the encoding features of the first speech unit, implicit semantic features of the first speech unit are obtained. The explicit semantic features and implicit semantic features are fused to obtain fused semantic features of the first speech unit. Reconstructed audio corresponding to the first moment is synthesized based on the fused semantic features of the first speech unit. Therefore, even if the set of mapping spaces between speech audio and text is large, using the fused semantic features of the explicit and implicit semantic features as the semantic features of the speech unit can provide a larger semantic search space than using only the explicit semantic features as the semantic features of the speech unit. Furthermore, the synthesized reconstructed audio relies on the fused semantic features of the explicit and implicit semantic features. Therefore, even if the text prediction results of the speech unit are incorrect, resulting in incorrect explicit semantic features, the implicit semantic features are still present, making the synthesized reconstructed audio less likely to be incorrect. This improves the fault tolerance of the reconstructed audio synthesis to erroneous text prediction results for the corresponding speech unit. Therefore, the accuracy of reconstructed audio can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 This is a flow chart of an embodiment of a speech reconstruction method provided by the present application;

[0010] Figure 2 This is a structural diagram of a specific embodiment of the speech reconstruction model provided by this application;

[0011] Figure 3 This is a schematic diagram of the framework of an embodiment of the speech reconstruction device of the present application;

[0012] Figure 4 This is a schematic diagram of the framework of an embodiment of the electronic device of the present application;

[0013] Figure 5 It is a schematic diagram of a framework of an embodiment of a computer-readable storage medium of the present application. DETAILED DESCRIPTION

[0014] In order to make the purpose, technical solutions and effects of this application clearer and more specific, this application is further described in detail below with reference to the accompanying drawings and examples.

[0015] It should be noted that the term "several" in this article means at least one, and the terms "first", "second", etc. are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. The term "and / or" is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the previous and next associated objects are in an "or" relationship. In addition, the term "at least one" in this article means any combination of at least two of any one or more of a plurality of. For example, including at least one of A, B, and C, can mean including any one or more elements selected from the set consisting of A, B, and C.

[0016] See also Figure 1 , Figure 1 This is a flow chart of an embodiment of the speech reconstruction method provided by the present application. It should be noted that if there are substantially the same results, this embodiment is not based on Figure 1 The process sequence shown is limited. Figure 1 As shown, this embodiment includes:

[0017] Step S11: Obtain the encoding feature of the first speech unit corresponding to the first moment, and obtain the text prediction result of the second speech unit related to the first speech unit.

[0018] The first moment in this article can be the current moment, or it can refer to any specific time node. It can be limited according to actual needs and is not restricted here.

[0019] In one embodiment, an encoder may be used to encode the first speech unit corresponding to the first moment to obtain an encoding feature of the first speech unit.

[0020] Specifically, the speech audio can be obtained, and preprocessing operations (such as noise reduction, speech enhancement, resampling, etc.) can be performed on the speech audio to obtain the acoustic features of the first speech unit corresponding to the first moment, and the acoustic features of the first speech unit can be encoded using an encoder to obtain the encoded features of the first speech unit. The acoustic features corresponding to the first speech unit can be stored in the form of a set. For example, X = [x1, x2, ..., x t ], where x t Represents the acoustic features corresponding to the t-th speech unit. Acoustic features can represent speaker characteristics, such as the speaker's timbre.

[0021] In a specific embodiment, the specific formula corresponding to the encoding feature of the first speech unit obtained by using the first speech unit is as follows: Among them, x t is the acoustic feature corresponding to the first speech unit corresponding to the first moment, is the coding feature of the first speech unit. Can contain implicit semantic information and fewer acoustic features.

[0022] In another embodiment, the second speech unit is the first speech unit, or the second speech unit is a speech unit corresponding to a second moment, wherein the second moment is located after the first moment, for example, the second moment is the next moment after the first moment. Of course, the second moment can also be the next two moments after the first moment, and this can be set according to actual needs and is not limited here.

[0023] In another embodiment, the encoding features of the second speech unit and the reference text of the second speech unit can be used to generate a text prediction result of the second speech unit, wherein the reference text of the second speech unit is the text corresponding to the speech unit located before the second speech unit.

[0024] It can be understood that the reference text of the second speech unit is the text corresponding to the speech unit before the second moment, and the specific selection of the text corresponding to the speech unit corresponding to the moment can be selected according to actual needs. For example, the reference text of the second speech unit is the text corresponding to the third speech unit, and the third speech unit is the speech unit corresponding to the moment before the second speech unit. The text corresponding to the third speech unit is determined based on the text prediction result of the third speech unit. For another example, when the text corresponding to the third speech unit is valid text information, the reference text of the second speech unit is the text corresponding to the third speech unit. When the text corresponding to the third speech unit is invalid text information, the reference text of the second speech unit is the reference text of the third speech unit. The text prediction result of the third speech unit is generated using the encoding features of the third speech unit and the reference text of the third speech unit.

[0025] The third speech unit is the speech unit corresponding to the previous moment of the second speech unit, the second speech unit is the speech unit corresponding to the second moment, and the second moment is the moment after the first moment. Therefore, taking the third speech unit as the first speech unit as an example, when the text corresponding to the first speech unit is valid text information, the reference text of the second speech unit is the text corresponding to the first speech unit. When the text corresponding to the first speech unit is invalid text information, the reference text of the second speech unit is the reference text of the first speech unit. The text prediction result of the first speech unit is generated using the encoding features of the first speech unit and the reference text of the first speech unit. The method for obtaining the text prediction result of the first speech unit can refer to the method for obtaining the text prediction result of the second speech unit, and will not be repeated here.

[0026] In other words, if the text corresponding to the first speech unit is invalid, the text prediction result corresponding to the second moment is generated using the encoding features of the second speech unit and the reference text of the first speech unit. If the text corresponding to the first speech unit is valid, the text prediction result corresponding to the second moment is generated using the encoding features of the second speech unit and the text corresponding to the first speech unit, and the text corresponding to the first speech unit is determined based on the text prediction result of the first speech unit.

[0027] In another specific embodiment, the coding features of the second speech unit and the text features of the corresponding text are used for prediction, and the specific formula for obtaining the text prediction result of the second speech unit is as follows: Among them, x t is the acoustic feature corresponding to the first speech unit at time t, y u is the reference text of the first speech unit, for example, the text corresponding to time t-1, is the predicted text of the first speech unit, is the text prediction result corresponding to the first speech unit. It can express more explicit semantic information.

[0028] In one embodiment, the second time may be the next time after the first time. For example, the first time is time t, and the second time is time t+1. The explicit semantic feature corresponding to the first speech unit may be obtained by the following formula: in, is the explicit semantic feature corresponding to the first speech unit at time t, W exp is the trainable mapping matrix, is the predicted text corresponding to the first speech unit at time t, y u+1 The text corresponding to the first speech unit at time t. If it is valid text information, the text prediction result corresponding to the second speech unit at time t+1 is That is, using the second speech unit x corresponding to time t+1 t+1 The text y corresponding to the first speech unit at time t u+1 Generated. In the case where the text corresponding to the first speech unit is invalid text information (such as the text corresponding to the first speech unit is a blank character), the text prediction result corresponding to time t+1 is That is, using the speech unit x at time t+1 t+1 and the reference text at time t (e.g., the text y corresponding to the speech unit at time t-1) u) is generated, wherein the specific generation process can refer to step S11. Among them, valid text information can be understood as the existence of text, or it can be understood as the text is consistent with the real text, which can be set according to the actual situation. In addition, blank characters can be used Align speech units with text.

[0029] In another specific embodiment, the reference text of the second speech unit can be decoded by a decoder to obtain the text features of the reference text of the second speech unit. The coding features of the second speech unit and the text features of the reference text of the second speech unit are used to predict the text of the second speech unit to obtain the text prediction result of the second speech unit. The reference text can be stored in the form of a set. For example, Y = [y1, y2, ..., y u ], where, for example, y u The text representation corresponding to the reference text of the u-th label.

[0030] It can be understood that the decoder is used to decode the text corresponding to the speech unit before the second moment to obtain the text features of the corresponding text. The encoding features of the second speech unit and the text features of the corresponding text are used to perform prediction to obtain the text prediction result of the second speech unit corresponding to the second moment.

[0031] In a specific embodiment, a decoder is used to decode the reference text to obtain the specific formula corresponding to the text features of the reference text as follows: Among them, y u For reference text, is the text feature of the reference text.

[0032] In another embodiment, the speech unit is a speech frame.

[0033] In another embodiment, the text prediction result includes a plurality of predicted texts and probabilities corresponding to the plurality of predicted texts, and the text corresponding to the speech unit is the predicted text whose probability meets a preset condition in the text prediction result of the speech unit.

[0034] In a specific embodiment, the text prediction result can be represented in the form of a probability distribution, which includes a number of predicted texts and the probabilities corresponding to the predicted texts. The predicted text with the highest probability is used as the text corresponding to the speech unit.

[0035] In another specific embodiment, the encoding features and the reference text may be normalized, and the normalized data may be used as the text prediction result. For example, taking the first moment as an example, the text prediction result at the first moment may be obtained by the following formula: Among them, x tis the acoustic feature corresponding to the first speech unit at the first moment, y u For reference text, is the predicted text corresponding to the first speech unit, is the text prediction result, is the encoding feature of the first speech unit, is the text feature of the reference text.

[0036] In another embodiment, the steps of obtaining the encoding features of the first speech unit corresponding to the first moment and obtaining the text prediction result of the second speech unit related to the first speech unit are implemented by the speech recognition module of the speech reconstruction model.

[0037] Combine Figure 2 To illustrate, Figure 2 This is a structural diagram of a specific embodiment of the speech reconstruction model provided by the present application. The speech reconstruction model includes a speech recognition module 21, which includes an encoder 211, a decoder 212, and a joint network 213. The first speech unit x corresponding to the first moment is t Input to encoder 211 to obtain the encoding feature of the first speech unit corresponding to the first moment The reference text y u Input into decoder 212 to obtain the corresponding text features Encoding features and text features Input into the joint network to obtain the text prediction result of the second speech unit

[0038] Among them, the encoder 211 can use a Conformer network with strong generalization. The decoder 212 uses a multi-layer LSTM network. The joint network usually uses a full connection layer (FC) to fuse the output of the encoder and decoder 212. The encoder 211 can be used to decouple acoustic features from semantic features, and the decoder 212 can be used to recognize the context of text content. In addition, in order to reduce the delay of the speech recognition module in DSR applications, the Conformer network uses a mask with a left field of view of 16 and a right field of view of 0 to ensure real-time self-attention.

[0039] To ensure the real-time performance of the speech reconstruction method (DSR), the speech recognition module (ASR) in this embodiment can be a recurrent neural network transcriber model (RNNT) with natural real-time capabilities.

[0040] Step S12: Obtaining the explicit semantic features of the first speech unit based on the text prediction result of the second speech unit, and obtaining the implicit semantic features of the first speech unit based on the encoding features of the first speech unit.

[0041] In one embodiment, the product of the text prediction result corresponding to the second moment and the explicit parameter can be used as the explicit semantic feature of the first speech unit.

[0042] In yet another embodiment, the explicit parameters are trainable mapping matrices.

[0043] In another embodiment, the operation result between the coding feature and the implicit parameter can be obtained, and the operation result can be processed using a gated linear unit to obtain the implicit semantic feature of the first speech unit.

[0044] In one embodiment, the implicit parameters are trainable parameters.

[0045] In yet another specific embodiment, the gated linear unit is a selective gated linear unit.

[0046] In another specific embodiment, the implicit parameter includes a first implicit parameter and a second implicit parameter. The product of the encoding feature and the first implicit parameter can be obtained, and the sum of the product and the second implicit parameter is used as the operation result.

[0047] For example, the implicit semantic feature of the first speech unit can be obtained by the following formula: Among them, SwitchGLU(·) is the selection gated linear unit, M imp is the first implicit parameter, b imp is the second implicit parameter, is the coding feature corresponding to the first speech unit, is the implicit semantic feature corresponding to the first speech unit.

[0048] In another embodiment, the steps of obtaining the explicit semantic features of the first speech unit based on the text prediction results of the second speech unit and obtaining the implicit semantic features of the first speech unit based on the encoding features are implemented by an enhancement module of the speech reconstruction model.

[0049] Continue to combine Figure 2 To illustrate, Figure 2 : This is a structural diagram of a specific embodiment of the speech reconstruction model provided by the present application. The speech reconstruction model also includes an enhancement module 22, wherein the enhancement module 22 can be a frame-level adaptor module (Adaptor). Since the implicit semantic features lack text content context and the explicit semantic features do not contain acoustic features, the enhancement module simultaneously takes these two complementary semantic features as input. That is, the coded features output by the speech recognition module 21 are and text prediction results As the input of the enhancement module 22. Specifically, the encoding feature Get the implicit semantic features corresponding to the first phonetic unit Predicting results using text Get the text corresponding to the first phonetic unit And based on the text corresponding to the first speech unit Get the explicit semantic features corresponding to the first phonetic unit The specific calculation method can be referred to the relevant description in step S12 and will not be repeated here.

[0050] Considering the dimensionality difference between explicit semantic features and implicit semantic features in the fusion process, the forward operator can be used to align the text prediction results to the explicit semantic features at the frame level. The encoded features contain more information, but only a part of this information contains semantic information, and the rest mainly corresponds to whitespace. To remove this information redundancy, a gate control operator can be used to stimulate valid frames and suppress invalid frames.

[0051] Step S13: Fusing the explicit semantic features and the implicit semantic features to obtain fused semantic features of the first speech unit.

[0052] In one embodiment, the explicit semantic features and the implicit semantic features may be weighted to obtain the fused semantic features of the first speech unit.

[0053] In a specific embodiment, the product of the first weighting coefficient and the explicit semantic feature and the product of the second weighting coefficient and the implicit semantic feature can be added together, and the resulting sum can be used as the fused semantic feature of the first speech unit. The sum of the first weighting coefficient and the second weighting coefficient is fixed.

[0054] For example, the fused semantic feature of the first speech unit can be obtained by the following formula: in, is the fusion semantic feature of the first speech unit, λ is the first weighting coefficient, is the explicit semantic feature, 1-λ is the second weighting coefficient, It is an implicit semantic feature, and the sum of the first weighting coefficient λ and the second weighting coefficient 1-λ is 1. Specifically, the first weighting coefficient λ can be 0.5, and the specific value can be set according to actual needs and is not limited here.

[0055] In another embodiment, the step of fusing the explicit semantic features and the implicit semantic features to obtain the fused semantic features of the first speech unit is implemented by an enhancement module of the speech reconstruction model.

[0056] Continue to combine Figure 2 To illustrate, Figure 2 1 is a structural diagram of a specific embodiment of the speech reconstruction model provided by the present application. The speech reconstruction model also includes an enhancement module 22, wherein the enhancement module 22 can be a frame-level adaptor module (Adaptor). The coding features output by the speech recognition module 21 are converted into and text prediction results As the input of the enhancement module 22, we will not elaborate on it here. and implicit semantic features Perform weighted fusion to obtain the fusion semantic features corresponding to the first speech unit And integrate semantic features As the output of the enhancement module 22.

[0057] Considering that the mapping set between speech audio and text corresponding to different articulatory disorders is relatively large, the enhancement module uses a gate control operator to fuse explicit semantic features and implicit semantic features to obtain fused semantic features. Among them, the enhancement module is used to connect the speech recognition module with the incremental text-to-speech technology (TTS). The enhancement module follows the two principles of high information richness and strong adaptability. By introducing fused semantic features, it can amplify the semantic search space to improve the fault tolerance of the incremental text-to-speech technology (TTS) for erroneous recognition results, so as to further compensate for the damage to the intelligibility of the reconstructed audio caused by erroneous recognition results, that is, it can improve the accuracy of the reconstructed audio and the intelligibility of the content. In addition, the enhancement module can enable ASR and TTS to achieve gradient backpropagation and joint fine-tuning to further reduce the gap between ASR and TTS.

[0058] Step S14: synthesizing the reconstructed audio corresponding to the first moment based on the fused semantic features of the first speech unit.

[0059] In one embodiment, a speech quantization codebook for the first moment can be generated based on the fused semantic features of at least one target speech unit and a speech quantization codebook before the first moment, wherein the at least one target speech unit includes the first speech unit and the at least one target speech unit further includes at least one of the following: a speech unit before the first moment and a target number of speech units after the first moment, where the target number is less than a preset threshold. Reconstructed audio corresponding to the first moment is generated using the speech quantization codebook at the first moment.

[0060] In a specific embodiment, at least one target speech unit further includes the fused semantic features corresponding to the target number of speech units after the first moment. The step of generating the speech quantization codebook at the first moment based on the fused semantic features of at least one target speech unit and the speech quantization codebook before the first moment is implemented using a quantization model. The quantization model is trained by using a large receptive field to guide a small receptive field. Each receptive field corresponds to a field value, and the field value corresponding to the large receptive field is greater than the field value corresponding to the small receptive field. The target number is the minimum field value corresponding to different receptive fields.

[0061] For example, the first moment is the t+1 moment, and the speech quantization codebook at the first moment is where θ AR is a decoder-only GPT autoregressive model, which can be shared by different receptive fields in multi-task learning. k can be used to represent different receptive fields (which can be set according to actual needs), and c <t as the input of the θ AR autoregressive model, c t+1 as the output of θ AR , is the fused semantic features corresponding to the speech units before the first moment (specifically, the speech units at each moment before the first moment, or the speech units at several moments before the first moment), the speech unit at the first moment, and the target number of speech units after the first moment. c ＜t is the speech quantization codebook before the first moment, specifically, the speech quantization codebooks at each moment before the first moment, or the speech quantization codebooks at several moments before the first moment. c t+1 is the speech quantization codebook at the first moment output by θ AR . Of course, the speech quantization codebook at the first moment can also be a predicted speech quantization codebook whose probability meets the preset conditions, and there is no limitation here. Among them, for the input of the autoregressive model, the enhanced module frame-level semantics and the historical speech quantization codebook can be fused in a splicing manner. When t < k, c t can be set as a fixed constant.

[0062] Among them, to ensure the real-time performance of the entire DSR system link, the speech quantization codebook at the first moment can be input into the HiFi-GAN vocoder in a windowed manner to obtain the reconstructed speech waveform, so as to further obtain the corresponding reconstructed audio.

[0063] In another specific embodiment, considering that the prosody and quality of reconstructed audio in TTS are poor due to limited context, a multiple wait-k strategy can be introduced into the autoregressive model. This involves employing a multi-task learning approach to ensure that the autoregressive model can adapt to semantic frame sequences with different wait times of different lengths. As k increases, the autoregressive model's response time increases, and its performance improves. To balance lower response time with better audio synthesis performance, a knowledge distillation approach can be further introduced, allowing a large receptive field of view to guide the training of a small receptive field of view, ensuring that the synthesized audio quality of the small receptive field of view is as close as possible to that of the large field of view. Specifically, the quantization model training step includes: for each receptive field of view, using the fused semantic features of several speech units corresponding to the receptive field of view and the speech quantization codebook before the sample time, to generate a speech quantization codebook for the receptive field of view at the sample time. The several speech units corresponding to the receptive field of view include the sample speech unit corresponding to the sample time, the fused semantic features corresponding to the sample speech unit before the sample time, and the field value sample speech units corresponding to the receptive field of view after the sample time. For each receptive field of view, the first loss of the receptive field of view is obtained by using the difference between the speech quantization codebook for the receptive field of view at the sample time and the actual sample quantization codebook corresponding to the sample time. The perceptual fields other than the maximum perceptual field of view are used as target perceptual fields of view. The difference between the speech quantization codebook for the target perceptual field of view at the sample time and the speech quantization codebook for the reference perceptual field of view at the sample time is used to obtain the second loss for the target perceptual field of view. The field of view value corresponding to the reference perceptual field of view is the sum of the field of view value corresponding to the target perceptual field of view and a preset value. The total loss of the quantization model is calculated by combining the first and second losses. The total loss is used to adjust the parameters of the quantization model. The preset values ​​can be set according to actual needs and are not limited here.

[0064] For example, the first loss of the quantized model during training can be obtained by the following formula: Among them, k can be used to represent different perception horizons, is the first loss, CE(·) is the cross entropy function, c t+1 is the actual speech quantization codebook at the sample time, The second loss of the quantization model during training can be obtained by the following formula: k∈(1,10), where k can be used to represent different perception horizons. For the second loss, and The specific meaning of can refer to the relevant description in step S14. In addition, Corresponding to the target perception field of view, Corresponding to the reference perception field of view, the field of view value corresponding to the target perception field of view is k, and the field of view value corresponding to the reference perception field of view is k+10, that is, the preset value is 10, D KL (·) represents the KL divergence loss function (Kullback-Leibler, KL), which can be used to measure the difference between the two probability distributions mentioned above. The total loss of the quantization model during training can be obtained by the following formula: in, is the total loss, To feel the first loss when the field of view k is equal to 1, To feel the first loss when the field of view k is equal to 10, To feel the first loss when the field of view k is equal to 20, Feel the second loss when the field of view k is equal to 1, Feel the second loss when the field of view k is equal to 10.

[0065] In another embodiment, the step of synthesizing the reconstructed audio corresponding to the first moment based on the fused semantic features of the first speech unit is implemented by a speech synthesis module of the speech reconstruction model.

[0066] In a specific embodiment, the speech synthesis module includes a quantization model, which is used to generate a speech quantization codebook at the first moment based on the fused semantic features of the first speech unit and the speech quantization codebook before the first moment, and the speech quantization codebook at the first moment is used to generate reconstructed audio corresponding to the first moment.

[0067] In another embodiment, the quantization model and the enhancement module have the same frame shift. For example, the quantization model and the enhancement module have the same hop size. That is, the sequence lengths of the quantization model and the enhancement module can be kept consistent, and have a natural alignment relationship.

[0068] Continue to combine Figure 2 To illustrate, Figure 2 The structure diagram of a specific embodiment of the speech reconstruction model provided by the present application is shown in FIG. The speech reconstruction model further includes a speech synthesis module 23, which includes a quantization model 231. The fusion semantic features output by the enhancement module 22 are As the input of the quantization model 231, the soft prediction output of the quantization model 231 is obtained. During the training process, the soft prediction output of the quantization model 231 is The actual speech quantization codebook c corresponding to the sample time t+1 Get the first loss corresponding to each perception field And the soft prediction output according to the quantization model 231 Get the second losses corresponding to other perception fields except the maximum perception field And the sum of each first loss and the second loss is taken as the total loss. For the specific calculation method, please refer to the relevant description in step S13 and will not be elaborated here. When the total loss converges, the training of the quantization model 231 is completed. During the application process, the perception field k can be selected according to actual needs. For example, in the actual application process, the perception field k is set to 1, that is, the perception field k corresponding to the soft prediction output of the quantization model 231 is 1. Specifically, the quantization model 231 is adapted to wait for the semantic frame sequence of the length of the perception field k. The speech quantization codebook at the first moment is input into the HiFi-GAN based voice coder in a windowed manner to obtain the reconstructed speech waveform, so as to further obtain the corresponding reconstructed audio.

[0069] For existing TTS modules, word-level semantic information is usually used as input. At this time, the word-level semantics of ASR decoding need to wait for enough acoustic features before they can be stimulated, resulting in a long response time for the first frame after the DSR system reconstructs the audio, which is uncertain. This embodiment introduces a TTS with an autoregressive model. The autoregressive model adopts a decoder-only GPT autoregressive structure, and cleverly uses the frame-level semantic feature output of the above-mentioned enhancement module to predict the speech quantization codebook of the reconstructed audio.

[0070] See Figure 3 , Figure 3 It is a schematic diagram of the framework of an embodiment of the speech reconstruction device of the present application. The speech reconstruction device 300 includes a coding acquisition and text prediction module 310, an explicit and implicit semantic feature acquisition module 320, a fused semantic acquisition module 330, and a reconstructed audio acquisition module 340. Among them, the coding acquisition and text prediction module 310 is used to obtain the coding features of the first speech unit corresponding to the first moment, and obtain the text prediction results of the second speech unit related to the first speech unit. The explicit and implicit semantic feature acquisition module 320 is used to obtain the explicit semantic features of the first speech unit based on the text prediction results of the second speech unit, and to obtain the implicit semantic features of the first speech unit based on the coding features of the first speech unit. The fused semantic acquisition module 330 is used to fuse the explicit semantic features and the implicit semantic features to obtain the fused semantic features of the first speech unit. The reconstructed audio acquisition module 340 is used to synthesize the reconstructed audio corresponding to the first moment based on the fused semantic features of the first speech unit.

[0071] In some embodiments, the second speech unit is the first speech unit, or the second speech unit is the speech unit corresponding to the second moment, and the second moment is the next moment of the first moment.

[0072] In some embodiments, the encoding acquisition and text prediction module 310 specifically executes the acquisition of the text prediction result of the second speech unit related to the first speech unit, including: using the encoding features of the second speech unit and the reference text of the second speech unit to generate the text prediction result of the second speech unit, wherein the reference text of the second speech unit is the text corresponding to the speech unit located before the second speech unit.

[0073] In some embodiments, the encoding acquisition and text prediction module 310 specifically executes the use of the encoding features of the second speech unit and the reference text of the second speech unit to generate a text prediction result of the second speech unit, including: using a decoder to decode the reference text of the second speech unit to obtain the text features of the reference text of the second speech unit; using the encoding features and the text features of the second speech unit to perform prediction to obtain the text prediction result of the second speech unit.

[0074] In some embodiments, the reference text of the second speech unit is the text corresponding to the third speech unit, the third speech unit is the speech unit corresponding to the previous moment of the second speech unit, and the text corresponding to the third speech unit is determined based on the text prediction result of the third speech unit; or, when the text corresponding to the third speech unit is valid text information, the reference text of the second speech unit is the text corresponding to the third speech unit, and when the text corresponding to the third speech unit is invalid text information, the reference text of the second speech unit is the reference text of the third speech unit, and the text prediction result of the third speech unit is generated using the encoding features of the third speech unit and the reference text of the third speech unit.

[0075] In some embodiments, the explicit and implicit semantic feature acquisition module 320 specifically executes the text prediction result based on the first speech unit to obtain the explicit semantic feature of the first speech unit, including: taking the product of the text prediction result corresponding to the second moment and the explicit parameter as the explicit semantic feature of the first speech unit, wherein the second moment is located after the first moment, and the text prediction result corresponding to the second moment is the text prediction result of the second speech unit corresponding to the second moment.

[0076] In some embodiments, the explicit parameters are trainable mapping matrices.

[0077] In some embodiments, when the text corresponding to the first speech unit is invalid text information, the text prediction result corresponding to the second moment is generated using the encoding features of the second speech unit and the reference text. When the text corresponding to the first speech unit is valid text information, the text prediction result corresponding to the second moment is generated using the encoding features of the second speech unit and the text corresponding to the first speech unit. The text corresponding to the first speech unit is determined based on the text prediction result of the first speech unit.

[0078] In some embodiments, the explicit and implicit semantic feature acquisition module 320 obtains the implicit semantic features of the first speech unit based on the coding features, including: obtaining the operation results between the coding features and the implicit parameters, and processing the operation results using a gated linear unit to obtain the implicit semantic features of the first speech unit.

[0079] In some embodiments, the implicit parameters are trainable parameters.

[0080] In some embodiments, the gated linear unit is a selective gated linear unit.

[0081] In some embodiments, the explicit and implicit semantic feature acquisition module 320 specifically executes implicit parameters including a first implicit parameter and a second implicit parameter, and obtains the operation result between the encoding feature and the implicit parameter, including: obtaining the product between the encoding feature and the first implicit parameter, and taking the sum of the product and the second implicit parameter as the operation result.

[0082] In some embodiments, the encoding acquisition and text prediction module 310 specifically executes the acquisition of the encoding features of the first speech unit corresponding to the first moment, including: using an encoder to encode the first speech unit corresponding to the first moment to obtain the encoding features of the first speech unit.

[0083] In some embodiments, the encoding acquisition and text prediction module 310 generates a text prediction result for the first speech unit using the encoding features and the reference text, including: decoding the reference text using a decoder to obtain text features of the reference text; and performing prediction using the encoding features of the first speech unit and the text features of the reference text to obtain a text prediction result for the first speech unit.

[0084] In some embodiments, the fusion semantic acquisition module 330 specifically performs the fusion of explicit semantic features and implicit semantic features to obtain the fusion semantic features of the first speech unit, including: weighting the explicit semantic features and implicit semantic features to obtain the fusion semantic features of the first speech unit.

[0085] In some embodiments, the speech unit is a speech frame.

[0086] In some embodiments, the text prediction result includes a number of predicted texts and probabilities corresponding to the predicted texts. The text corresponding to the speech unit is the predicted text whose probability meets a preset condition in the text prediction result of the speech unit.

[0087] In some embodiments, the reconstructed audio acquisition module 340 specifically performs the synthesis of the reconstructed audio corresponding to the first moment based on the fused semantic features of the first speech unit, including: generating a speech quantization codebook for the first moment based on the fused semantic features of at least one target speech unit and a speech quantization codebook before the first moment, wherein the at least one target speech unit includes the first speech unit, and the at least one target speech unit further includes at least one of the following: a speech unit before the first moment, and a target number of speech units after the first moment, where the target number is less than a preset threshold. The reconstructed audio corresponding to the first moment is generated using the speech quantization codebook at the first moment.

[0088] In some embodiments, at least one target speech unit further includes fused semantic features corresponding to a target number of speech units after a first moment, and the step of generating a speech quantization codebook for the first moment based on the fused semantic features of the at least one target speech unit and the speech quantization codebook before the first moment is implemented using a quantization model. The quantization model is obtained by using a large receptive field of view to guide small receptive field of view training, where each receptive field of view corresponds to a field of view value, the field of view value corresponding to the large receptive field of view is greater than the field of view value corresponding to the small receptive field of view, and the target value is the minimum field of view value corresponding to different receptive fields of view.

[0089] In some embodiments, the quantization model training step includes: for each perceptual field of view, generating a speech quantization codebook for the perceptual field of view at the sample time using the fused semantic features of several speech units corresponding to the perceptual field of view and a speech quantization codebook before the sample time, wherein the several speech units corresponding to the perceptual field of view include the sample speech unit corresponding to the sample time, the fused semantic features corresponding to the sample speech units before the sample time, and the field value sample speech units corresponding to the perceptual field of view after the sample time. For each perceptual field of view, a first loss for the perceptual field of view is obtained by using the difference between the speech quantization codebook for the perceptual field of view at the sample time and the actual sample quantization codebook corresponding to the sample time. Perceptual fields other than the maximum perceptual field of view are used as target perceptual fields of view, and a second loss for the target perceptual field of view is obtained by using the difference between the speech quantization codebook for the target perceptual field of view at the sample time and the speech quantization codebook for the reference perceptual field of view at the sample time, wherein the field value corresponding to the reference perceptual field of view is the sum of the field value corresponding to the target perceptual field of view and a preset value. The first and second losses are combined to obtain a total loss for the quantization model. The total loss is used to adjust the parameters of the quantization model.

[0090] In some embodiments, the steps of obtaining the encoding features of the first speech unit corresponding to the first moment and obtaining the text prediction results of the second speech unit related to the first speech unit are implemented by the speech recognition module of the speech reconstruction model.

[0091] In some embodiments, the steps of obtaining the explicit semantic features of the first speech unit based on the text prediction results of the second speech unit, obtaining the implicit semantic features of the first speech unit based on the encoding features, and fusing the explicit semantic features and the implicit semantic features to obtain the fused semantic features of the first speech unit are implemented by an enhanced module of the speech reconstruction model.

[0092] In some embodiments, the step of synthesizing the reconstructed audio corresponding to the first moment based on the fused semantic features of the first speech unit is implemented by a speech synthesis module of the speech reconstruction model.

[0093] In some embodiments, the speech synthesis module includes a quantization model, the quantization model being configured to generate a speech quantization codebook at a first moment in time based on a fused semantic feature of the first speech unit and a speech quantization codebook before a first moment in time, the speech quantization codebook at the first moment in time being used to generate reconstructed audio corresponding to the first moment in time. The quantization model and the enhancement module have the same frame shift.

[0094] See also Figure 4 , Figure 4 1 is a schematic diagram of the framework of an embodiment of an electronic device of the present application. Electronic device 40 includes a memory 41 and a processor 42 coupled to each other. Processor 42 is configured to execute program instructions stored in memory 41 to implement the steps of any of the above-described speech reconstruction method embodiments. In a specific implementation scenario, electronic device 40 may include, but is not limited to, a microcomputer and a server. Furthermore, electronic device 40 may also include mobile devices such as laptops and tablet computers, without limitation herein.

[0095] Specifically, the processor 42 is used to control itself and the memory 41 to implement the steps in any of the above-mentioned speech reconstruction method embodiments. The processor 42 can also be called a CPU (Central Processing Unit). The processor 42 may be an integrated circuit chip with signal processing capabilities. The processor 42 can also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. In addition, the processor 42 can be implemented by an integrated circuit chip.

[0096] See also Figure 5 , Figure 5 The computer-readable storage medium 50 stores program instructions 51 that can be executed by a processor, and the program instructions 51 are used to implement the steps of any of the above-mentioned speech reconstruction method embodiments.

[0097] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments. For the sake of brevity, it will not be repeated here.

[0098] The above description of the various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced with each other and will not be repeated herein for the sake of brevity.

[0099] In the several embodiments provided in this application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation methods described above are only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation. For example, units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0100] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0101] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) or a processor to execute all or part of the steps of each embodiment method of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

Claims

1. A speech reconstruction method, characterized in that: The method comprises: Obtaining a coding feature of a first speech unit corresponding to a first moment, and obtaining a text prediction result of a second speech unit related to the first speech unit; Obtaining explicit semantic features of the first speech unit based on a text prediction result of the second speech unit, and obtaining implicit semantic features of the first speech unit based on an encoding feature of the first speech unit; fusing the explicit semantic feature and the implicit semantic feature to obtain a fused semantic feature of the first speech unit; Based on the fused semantic features of the first speech unit, reconstructed audio corresponding to the first moment is synthesized.

2. The method according to claim 1, characterized in that The second speech unit is the first speech unit, or the second speech unit is a speech unit corresponding to a second moment, and the second moment is the next moment of the first moment.

3. The method according to claim 2, characterized in that The obtaining of a text prediction result of a second speech unit related to the first speech unit includes: A text prediction result of the second speech unit is generated by using the encoding features of the second speech unit and the reference text of the second speech unit, wherein the reference text of the second speech unit is the text corresponding to the speech unit located before the second speech unit.

4. The method according to claim 3, characterized in that The generating a text prediction result of the second speech unit by using the encoding feature of the second speech unit and the reference text of the second speech unit includes: Decoding the reference text of the second speech unit using a decoder to obtain text features of the reference text of the second speech unit; Performing prediction using the encoding feature of the second speech unit and the text feature to obtain a text prediction result of the second speech unit; And / or, the reference text of the second speech unit is the text corresponding to the third speech unit, the third speech unit is the speech unit corresponding to the previous moment of the second speech unit, and the text corresponding to the third speech unit is determined based on the text prediction result of the third speech unit; or, when the text corresponding to the third speech unit is valid text information, the reference text of the second speech unit is the text corresponding to the third speech unit, and when the text corresponding to the third speech unit is invalid text information, the reference text of the second speech unit is the reference text of the third speech unit, and the text prediction result of the third speech unit is generated using the encoding features of the third speech unit and the reference text of the third speech unit.

5. The method according to claim 1, wherein The obtaining of the explicit semantic features of the first speech unit based on the text prediction result of the second speech unit includes: multiplying the text prediction result of the second speech unit by the explicit parameter as the explicit semantic feature of the first speech unit; And / or, obtaining the implicit semantic feature of the first speech unit based on the encoding feature of the first speech unit includes: Obtaining a result of an operation between the coding feature of the first speech unit and an implicit parameter, and processing the result of the operation using a gated linear unit to obtain an implicit semantic feature of the first speech unit.

6. The method according to claim 5, characterized in that The explicit parameter is a trainable mapping matrix; And / or, the implicit parameter is a trainable parameter; And / or, the gated linear unit is a selective gated linear unit; And / or, the implicit parameter includes a first implicit parameter and a second implicit parameter, and obtaining the operation result between the encoding feature and the implicit parameter includes: The product of the encoding feature and the first implicit parameter is obtained, and the sum of the product and the second implicit parameter is used as the operation result.

7. The method according to claim 1, characterized in that The obtaining of the coding feature of the first speech unit corresponding to the first moment includes: Encoding the first speech unit corresponding to the first moment using an encoder to obtain an encoding feature of the first speech unit; And / or, fusing the explicit semantic feature and the implicit semantic feature to obtain the fused semantic feature of the first speech unit includes: performing weighted processing on the explicit semantic features and the implicit semantic features to obtain a fused semantic feature of the first speech unit; And / or, the speech unit is a speech frame; And / or, the text prediction result includes several predicted texts and probabilities corresponding to the several predicted texts, and the text corresponding to the speech unit is the predicted text whose probability in the text prediction result of the speech unit meets a preset condition.

8. The method according to claim 1, characterized in that The synthesizing the reconstructed audio corresponding to the first moment based on the fused semantic features of the first speech unit includes: Generate a speech quantization codebook at the first moment based on the fused semantic features of at least one target speech unit and a speech quantization codebook before the first moment, wherein the at least one target speech unit includes the first speech unit and further includes at least one of the following: a speech unit before the first moment and a target number of speech units after the first moment, where the target number is less than a preset threshold; The reconstructed audio corresponding to the first moment is generated using the speech quantization codebook at the first moment.

9. The method according to claim 8, characterized in that The at least one target speech unit also includes the fused semantic features corresponding to the target number of speech units after the first moment, and the step of generating the speech quantization codebook at the first moment based on the fused semantic features of the at least one target speech unit and the speech quantization codebook before the first moment is implemented by using a quantization model; wherein, The quantitative model is obtained by using a large perceptual field of view to guide the training of a small perceptual field of view. Each perceptual field of view corresponds to a field of view value. The field of view value corresponding to the large perceptual field of view is greater than the field of view value corresponding to the small perceptual field of view. The target value is the minimum field of view value corresponding to different perceptual fields of view.

10. The method according to claim 9, characterized in that The training steps of the quantization model include: For each of the perception horizons, a speech quantization codebook of the perception horizon at the sample moment is generated by using the fused semantic features of the plurality of speech units corresponding to the perception horizon and the speech quantization codebook before the sample moment, wherein the plurality of speech units corresponding to the perception horizon include a sample speech unit corresponding to the sample moment, the fused semantic features corresponding to the sample speech unit before the sample moment, and a field of view value sample speech units corresponding to the perception horizon after the sample moment; For each of the perception horizons, obtaining a first loss of the perception horizon by using a difference between a speech quantization codebook of the perception horizon at the sample moment and an actual sample quantization codebook corresponding to the sample moment; Taking all perceptual fields other than the maximum perceptual field of view as target perceptual fields of view, and using the difference between the speech quantization codebook of the target perceptual field of view at the sample time and the speech quantization codebook of the reference perceptual field of view at the sample time to obtain a second loss of the target perceptual field of view, wherein the field of view value corresponding to the reference perceptual field of view is the sum of the field of view value corresponding to the target perceptual field of view and a preset value; Combining the first losses and the second losses to obtain a total loss of the quantization model; The total loss is used to adjust the parameters of the quantization model.

11. The method according to claim 1, wherein The steps of obtaining the encoding features of the first speech unit corresponding to the first moment and obtaining the text prediction result of the second speech unit related to the first speech unit are implemented by the speech recognition module of the speech reconstruction model; The steps of obtaining the explicit semantic features of the first speech unit based on the text prediction result of the second speech unit, obtaining the implicit semantic features of the first speech unit based on the coding features, and fusing the explicit semantic features and the implicit semantic features to obtain the fused semantic features of the first speech unit are It is realized by the enhancement module of the speech reconstruction model; The step of synthesizing the reconstructed audio corresponding to the first moment based on the fused semantic features of the first speech unit is implemented by the speech synthesis module of the speech reconstruction model; The speech synthesis module includes a quantization model, the quantization model is used to generate a speech quantization codebook at the first moment based on the fused semantic features of the first speech unit and the speech quantization codebook before the first moment, and the speech quantization codebook at the first moment is used to generate reconstructed audio corresponding to the first moment; The quantization model and the enhancement module have the same frame shift.

12. A speech reconstruction device, characterized in that: The device comprises: a code acquisition and text prediction module, configured to acquire a code feature of a first speech unit corresponding to a first moment, and acquire a text prediction result of a second speech unit related to the first speech unit; an explicit and implicit semantic feature acquisition module, configured to obtain the explicit semantic features of the first speech unit based on the text prediction result of the second speech unit, and to obtain the implicit semantic features of the first speech unit based on the encoding features of the first speech unit; a fusion semantic acquisition module, configured to fuse the explicit semantic features with the implicit semantic features to obtain a fusion semantic feature of the first speech unit; The reconstructed audio acquisition module is used to synthesize the reconstructed audio corresponding to the first moment based on the fused semantic features of the first speech unit.

13. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores program instructions, and the processor is used to execute the program instructions to implement the speech reconstruction method according to any one of claims 1 to 11.

14. A computer-readable storage medium, characterized in that The computer-readable storage medium is used to store program instructions, and the program instructions can be executed to implement the speech reconstruction method according to any one of claims 1 to 11.

Images