Speech smoothing model training method, speech smoothing method and related device

Abstract

The application discloses a speech smoothing model training method, a speech smoothing method and related equipment. The method performs speech recognition prediction on sample speech data through a coding network and a first decoding network in a speech smoothing model to obtain a predicted text recognition result. First hidden features output by the coding network and second hidden features output by the first decoding network are obtained. The first hidden features and the second hidden features are subjected to speech smoothing prediction through a second decoding network in the speech smoothing model to obtain a predicted smoothing result. The speech smoothing model is trained based on a total loss function constructed based on each prediction result to obtain a trained speech smoothing model. Thus, original speech information is utilized, and personal information such as pronunciation habits of a speaker is completely retained, speech recognition misjudgment is reduced, and the accuracy of speech smoothing is improved.

Description

Technical Field

[0001] This application relates to the field of industrial technology, and in particular to a speech smoothing model training method, speech smoothing method and related equipment. Background Technology

[0002] Speech smoothing refers to the post-processing adjustment techniques used to address issues such as spoken speech insertion, speech overlap, speech order reversal, and missing punctuation in speech recognition results.

[0003] Speech smoothing technology performs tasks such as punctuation annotation, removal of inserted words, and adjustment of language order in the text generated from speech recognition. This provides excellent preprocessing for subsequent tasks such as reading, organizing and archiving speech information, and summarizing speech segments. For example, in a meeting setting, speech smoothing technology can remove irrelevant spoken text, incorrect word order, and slips of the tongue, providing a semantically correct backup of the meeting summary that is easy to retrieve, thus reducing the difficulty of organizing the text after speech recognition.

[0004] Among the related technologies, the main ones are the annotation of overlapping inserted words based on CRF (conditional random field), end-to-end text-to-text speech smoothing methods, and speech smoothing methods from non-smooth spoken speech to smooth text. These speech smoothing methods have problems such as poor speech smoothing effect. Summary of the Invention

[0005] Therefore, the purpose of this application is to provide a speech smoothing model training method, a speech smoothing method, and related equipment to solve at least one of the above-mentioned technical problems. The technical solution is as follows:

[0006] On the one hand, this application provides a method for training a speech smoothing model, including:

[0007] Obtain training samples, which include sample speech data, as well as the corresponding real text sequence and the corresponding smooth text sequence;

[0008] A speech smoothing model is determined, wherein the speech smoothing model sequentially includes an encoding network, a first decoding network, and a second decoding network;

[0009] The sample speech data is used to perform speech recognition prediction through the encoding network and the first decoding network to obtain the predicted text recognition result;

[0010] The first hidden feature output by the encoding network and the second hidden feature output by the first decoding network are obtained. The first hidden feature and the second hidden feature are then used by the second decoding network to perform speech smoothness prediction to obtain a prediction smoothness result.

[0011] Based on the predicted recognition result, the real text sequence, the smooth text sequence, and the predicted smoothness result, the total training loss is determined;

[0012] The speech smoothing model is trained based on the total training loss until the training termination condition is met, thus obtaining the trained speech smoothing model.

[0013] In an optional implementation, the step of performing speech recognition prediction on the sample speech data through the encoding network and the first decoding network to obtain the predicted text recognition result includes:

[0014] The sample speech data is input into the encoding module of the encoding network for feature encoding processing to obtain the first hidden feature;

[0015] The first hidden feature and the first target text sequence in the real text sequence are input into the first decoding module in the first decoding network for decoding processing to obtain the second hidden feature; the first target text sequence is the text sequence in the real text sequence before the prediction of the current recognized text.

[0016] The second hidden feature is input into the recognition module in the first decoding network to perform speech recognition prediction, and a predicted text recognition result is obtained; the predicted text recognition result is used to indicate the predicted recognition text sequence corresponding to the sample speech data.

[0017] In an optional implementation, the step of inputting the first hidden feature and the first target text sequence in the real text sequence into the first decoding module in the first decoding network for decoding processing to obtain the second hidden feature includes:

[0018] Obtain the first text embedding feature corresponding to the first target text sequence in the real text sequence;

[0019] Based on the first decoding module in the first decoding network, attention is calculated on the first hidden feature and the first text embedding feature to obtain the second hidden feature.

[0020] In an optional implementation, the step of performing speech smoothness prediction on the first latent feature and the second latent feature through the second decoding network to obtain a prediction smoothness result includes:

[0021] The first hidden feature, the second hidden feature, and the second target text sequence in the smooth text sequence are input into the second decoding module in the second decoding network for decoding processing to obtain the third hidden feature; the second target text sequence is the text sequence in the smooth text sequence before the prediction of the current smooth text.

[0022] The third hidden feature is input into the smoothing module in the second decoding network to perform speech smoothing prediction, and the predicted smoothing result is obtained; the predicted smoothing result is used to indicate the predicted smoothing text sequence corresponding to the sample speech data.

[0023] In one optional embodiment, the second decoding module includes a first decoding submodule and a second decoding submodule;

[0024] The step of inputting the first hidden feature, the second hidden feature, and the second target text sequence in the smooth text sequence into the second decoding module in the second decoding network for decoding processing to obtain the third hidden feature includes:

[0025] Obtain the second text embedding feature corresponding to the second target text sequence in the smooth text sequence;

[0026] Based on the first decoding submodule, attention calculation is performed on the first latent feature and the second text embedding feature to obtain the intermediate latent feature;

[0027] Based on the second decoding submodule, attention calculation is performed on the second hidden feature and the intermediate hidden feature to obtain the third hidden feature.

[0028] In an optional implementation, determining the total training loss based on the predicted text recognition result, the real text sequence, the smooth text sequence, and the predicted smoothness result includes:

[0029] Based on the difference between the predicted text recognition result and the real text sequence, the text recognition loss is determined;

[0030] Based on the difference between the smooth text sequence and the predicted smoothness result, the smoothness loss is determined;

[0031] The total training loss is determined based on the recognition loss and the smoothing loss.

[0032] In an optional embodiment, when the encoding network includes a syllable recognition module, the step of performing speech recognition prediction on the sample speech data through the encoding network and the first decoding network to obtain a predicted text recognition result further includes:

[0033] The first hidden feature is input into the syllable recognition module for syllable recognition processing to obtain the predicted syllable recognition result;

[0034] The determination of the total training loss based on the recognition loss and the smoothing loss includes:

[0035] Based on the difference between the predicted syllable recognition result and the syllable sequence corresponding to the real text sequence, the syllable recognition loss is determined;

[0036] The total training loss is determined based on the syllable recognition loss, the recognition loss, and the smoothness loss.

[0037] In an optional implementation, determining the total training loss based on the syllable recognition loss, the recognition loss, and the smoothness loss includes:

[0038] Obtain the loss weights corresponding to the syllable recognition loss, the recognition loss, and the smoothness loss, respectively;

[0039] Based on their respective loss weights, the syllable recognition loss, the recognition loss, and the smoothness loss are weighted and summed to obtain the total training loss.

[0040] On the other hand, this application also provides a speech smoothing method, including:

[0041] Obtain the speech to be smoothed;

[0042] The trained speech smoothing model is invoked to perform speech smoothing processing on the speech to be smoothed, and the speech smoothing processing result is obtained;

[0043] The trained speech smoothing model is obtained by any of the methods described above.

[0044] On the other hand, this application also provides a training device for a speech smoothing model, comprising:

[0045] The sample acquisition module is used to acquire training samples, which include sample speech data, as well as the real text sequence and the corresponding smooth text sequence corresponding to the sample speech data;

[0046] A model determination module is used to determine a speech smoothing model, wherein the speech smoothing model includes an encoding network, a first decoding network, and a second decoding network in sequence;

[0047] The first prediction module is used to perform speech recognition prediction on the sample speech data through the encoding network and the first decoding network to obtain the predicted text recognition result.

[0048] The second prediction module is used to obtain the first hidden feature output by the encoding network and the second hidden feature output by the first decoding network, and to perform speech smoothness prediction on the first hidden feature and the second hidden feature through the second decoding network to obtain the prediction smoothness result.

[0049] The loss determination module is used to determine the total training loss based on the predicted recognition result, the real text sequence, the smooth text sequence, and the predicted smoothness result;

[0050] The training module is used to train the speech smoothing model based on the total training loss until the training termination condition is met, thereby obtaining the trained speech smoothing model.

[0051] In an optional implementation, the first prediction module includes:

[0052] The first processing submodule is used to input the sample speech data into the encoding module in the encoding network for feature encoding processing to obtain the first hidden feature;

[0053] The second processing submodule is used to input the first hidden feature and the first target text sequence in the real text sequence into the first decoding module in the first decoding network for decoding processing to obtain the second hidden feature; the first target text sequence is the text sequence in the real text sequence before the prediction of the current recognized text;

[0054] The first prediction submodule is used to input the second latent feature into the recognition module in the first decoding network to perform speech recognition prediction and obtain the predicted text recognition result; the predicted text recognition result is used to indicate the predicted recognition text sequence corresponding to the sample speech data.

[0055] In an optional implementation, the second processing submodule is specifically used for:

[0056] Obtain the first text embedding feature corresponding to the first target text sequence in the real text sequence;

[0057] Based on the first decoding module in the first decoding network, attention is calculated on the first hidden feature and the first text embedding feature to obtain the second hidden feature.

[0058] In an optional implementation, the second prediction module includes:

[0059] The third processing submodule is used to input the first hidden feature, the second hidden feature, and the second target text sequence in the smooth text sequence into the second decoding module in the second decoding network for decoding processing to obtain the third hidden feature; the second target text sequence is the text sequence in the smooth text sequence before the prediction of the current smooth text.

[0060] The second prediction submodule is used to input the third hidden feature into the smoothing module in the second decoding network to perform speech smoothing prediction and obtain the prediction smoothing result; the prediction smoothing result is used to indicate the prediction smoothing text sequence corresponding to the sample speech data.

[0061] In one optional embodiment, the second decoding module includes a first decoding submodule and a second decoding submodule; the third processing submodule is specifically used for:

[0062] Obtain the second text embedding feature corresponding to the second target text sequence in the smooth text sequence;

[0063] Based on the first decoding submodule, attention calculation is performed on the first latent feature and the second text embedding feature to obtain the intermediate latent feature;

[0064] Based on the second decoding submodule, attention calculation is performed on the second hidden feature and the intermediate hidden feature to obtain the third hidden feature.

[0065] In one optional implementation, the loss determination module includes:

[0066] The first loss determination submodule is used to determine the text recognition loss based on the difference between the predicted text recognition result and the real text sequence;

[0067] The second loss determination submodule is used to determine the smoothing loss based on the difference between the smooth text sequence and the predicted smoothing result;

[0068] The first total loss determination submodule is used to determine the total training loss based on the recognition loss and the smoothing loss.

[0069] In an alternative embodiment, where the encoding network includes a syllable recognition module, the first prediction module further includes:

[0070] The third prediction submodule is used to input the first hidden feature into the syllable recognition module for syllable recognition processing to obtain the predicted syllable recognition result.

[0071] The first total loss determination submodule includes:

[0072] The third loss determination submodule is used to determine the syllable recognition loss based on the difference between the predicted syllable recognition result and the syllable sequence corresponding to the real text sequence;

[0073] The total loss determination unit is used to determine the total training loss based on the syllable recognition loss, the recognition loss, and the smoothness loss.

[0074] In an optional implementation, the total loss determination unit is specifically used for:

[0075] Obtain the loss weights corresponding to the syllable recognition loss, the recognition loss, and the smoothness loss, respectively;

[0076] Based on their respective loss weights, the syllable recognition loss, the recognition loss, and the smoothness loss are weighted and summed to obtain the total training loss.

[0077] On the other hand, this application also provides a voice smoothing device, comprising:

[0078] The voice acquisition module is used to acquire the voice to be smoothed.

[0079] The speech smoothing module is used to call the trained speech smoothing model to perform speech smoothing processing on the speech to be smoothed, and obtain the speech smoothing processing result;

[0080] The trained speech smoothing model is obtained by any of the methods described above.

[0081] On the other hand, this application also provides a computer device, the device including a processor and a memory, the memory storing at least one instruction or at least one program, the at least one instruction or at least one program being loaded and executed by the processor to implement the training method of the speech smoothness model as described above, or the training method of the speech smoothness model.

[0082] On the other hand, this application also provides a computer-readable storage medium storing at least one instruction or at least one program, wherein the at least one instruction or at least one program is loaded and executed by a processor to implement the training method of the speech smoothness model as described above, or the training method of the speech smoothness model.

[0083] This application provides a speech smoothing model training method, speech smoothing method, and related equipment. The method uses an encoding network and a first decoding network in the speech smoothing model to perform speech recognition prediction on sample speech data, obtaining a predicted text recognition result. It then acquires a first latent feature output by the encoding network and a second latent feature output by the first decoding network. The second decoding network in the speech smoothing model then performs speech smoothing prediction on these first and second latent features, obtaining a predicted smoothing result. The speech smoothing model is trained based on a total loss function constructed from the prediction results, resulting in a trained speech smoothing model. This approach utilizes the original speech information while also fully preserving the speaker's pronunciation habits and other personal information, reducing misjudgments in speech recognition and improving the accuracy of speech smoothing.

[0084] It should be understood that the above general description and the following detailed description are merely exemplary and do not limit this application. Attached Figure Description

[0085] To more clearly illustrate the technical solutions and advantages in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0086] Figure 1 This is a flowchart illustrating a speech smoothing model training method provided in an embodiment of this application;

[0087] Figure 2 This is a partial flowchart illustrating another speech smoothness model training method provided in an embodiment of this application;

[0088] Figure 3 This is a partial flowchart illustrating another speech smoothness model training method provided in an embodiment of this application;

[0089] Figure 4 This is a schematic diagram of the model structure of a speech smoothing model provided in an embodiment of this application;

[0090] Figure 5 This is a flowchart illustrating a speech smoothing method provided in an embodiment of this application;

[0091] Figure 6 This is a block diagram of a training device for a speech smoothness model provided in an embodiment of this application.

[0092] Figure 7 This is a block diagram of a voice smoothing device provided in an embodiment of this application.

[0093] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application. Detailed Implementation

[0094] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are merely one embodiment of this application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0095] As used herein, “an embodiment” or “an embodiment” refers to a specific feature, structure or characteristic that may be included in at least one implementation of this application.

[0096] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or server that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0097] Speech smoothing technology performs tasks such as punctuation annotation, removal of inserted words, and adjustment of language order in the text generated from speech recognition. This provides excellent preprocessing for subsequent tasks such as reading, organizing and archiving speech information, and summarizing speech segments. For example, in a meeting setting, speech smoothing technology can remove irrelevant spoken text, incorrect word order, and slips of the tongue, providing a semantically correct backup of the meeting summary that is easy to retrieve, thus reducing the difficulty of organizing the text after speech recognition.

[0098] Currently, mainstream speech smoothing techniques include: 1) speech smoothing methods based on CRF (conditional random field) overlapping word annotation and end-to-end text-to-text smoothing; and 2) speech smoothing methods from non-smooth spoken speech to smooth text. However, the inventors have found that these methods have at least the following shortcomings: Regarding method 1), when speakers insert spoken words, make incorrect word order changes, or slips of the tongue, they provide a large amount of intonation and emotional information. End-to-end text-to-text methods often misinterpret this information, resulting in poor speech smoothing performance. Regarding method 2), because non-smooth speech and smooth text do not correspond, training the speech smoothing model can introduce some speech recognition errors, thus also leading to poor speech smoothing performance.

[0099] To address the deficiencies such as poor voice smoothing effect in existing voice smoothing technologies, this application provides a method for training a voice smoothing model, a voice smoothing method, and related devices. Specifically, by determining a voice smoothing model that sequentially includes an encoding network, a first decoding network, and a second decoding network, performing voice recognition prediction on sample voice data through the encoding network and the first decoding network to obtain a predicted text recognition result; obtaining a first hidden feature output by the encoding network and a second hidden feature output by the first decoding network, and performing voice smoothing prediction on the first hidden feature and the second hidden feature through the second decoding network to obtain a predicted smoothing result; determining a total training loss based on the predicted recognition result, the true text sequence, the smoothed text sequence, and the predicted smoothing result; training the voice smoothing model based on the total training loss until the training end condition is met, and obtaining a trained voice smoothing model, thereby reducing misjudgments in voice recognition and improving the accuracy of voice smoothing.

[0100] The following describes in detail a method for training a voice smoothing model, a voice smoothing method, and related devices involved in the embodiments of this application with reference to the accompanying drawings.

[0101] Figure 1 It is a flowchart of a method for training a voice smoothing model provided in the embodiments of this application. This method can be applied to a training device for a voice smoothing model. The training device for the voice smoothing model can include a server. The server can be an independent server, a server cluster, a cloud server, etc. This method can be implemented in software and / or hardware. Please refer to Figure 1 , and this method can include:

[0102] S101, obtain training samples, where the training samples include sample voice data, and the true text sequence and the corresponding smoothed text sequence corresponding to the sample voice data.

[0103] Among them, the sample voice data can be the audio data of the sample voice.

[0104] The true text sequence refers to the sequence corresponding to the true text of the sample voice data. For example, if the syllables corresponding to the sample voice data are {e woyaoququ e gongyuan}, the corresponding true text sequence is {呃, 我, 要, 去, 去, 呃, 公, 园}.

[0105] The smoothed text sequence refers to the text sequence after voice smoothing of the sample voice data. For example, for the syllables {e woyaoququ e gongyuan} corresponding to the above sample voice data, its corresponding smoothed text sequence is {我, 要, 去, 图, 书, 馆}.

[0106] S102, determine the speech smoothing model, which sequentially includes an encoding network, a first decoding network, and a second decoding network.

[0107] The speech smoothing model is a machine learning model used to smooth the speech of sample speech. This speech smoothing model consists of an encoding network, a first decoding network, and a second decoding network, namely…

[0108] The speech smoothness model consists of three parts, θ = [θ encoder ,θ decoder-raw ,θ decoder-flu For example, sample speech data is input into a speech smoothing model, and is processed sequentially by the encoding network, the first decoding network, and the second decoding network of the speech smoothing model to obtain the corresponding prediction results.

[0109] S103, the sample speech data is used for speech recognition prediction through the encoding network and the first decoding network to obtain the predicted text recognition result.

[0110] In an optional implementation, the step of performing speech recognition prediction on the sample speech data through the encoding network and the first decoding network to obtain the predicted text recognition result includes:

[0111] S1031, The sample speech data is input into the encoding module of the encoding network for feature encoding processing to obtain the first hidden feature.

[0112] Optionally, before inputting the sample speech data into the speech smoothing model, the acoustic features extracted from the sample speech data can be obtained through the embedding module, resulting in a speech feature sequence x = (x1, x2, ..., x...). s The acoustic features can be F-bank (Filter Bank) features, Mel-Frequency Cepstral Coefficients (MFCCs) features, PLP features, etc. Next, the speech feature sequence x is input into the encoding module of the speech smoothness model's encoding network for feature encoding processing, converting it into the first latent feature h. enc =(h e1 ,h e2 ,…,h et ), where t is the audio frame length. This first latent feature can be used to characterize the encoded information of each audio frame in the sample speech data and the hidden dependencies between each audio frame and other audio frame sequences. For example, the encoded information includes speech content, pauses, silences, and non-smooth spoken words (e.g., um, uh, etc.).

[0113] S1032, input the first hidden feature and the first target text sequence in the real text sequence into the first decoding module in the first decoding network for decoding processing to obtain the second hidden feature; the first target text sequence is each text sequence in the real text sequence before predicting the current recognized text.

[0114] In an optional implementation manner, the step of inputting the first hidden feature and the first target text sequence in the real text sequence into the first decoding module in the first decoding network for decoding processing to obtain the second hidden feature includes:

[0115] Obtain the first text embedding feature corresponding to the first target text sequence in the real text sequence;

[0116] Based on the first decoding module in the first decoding network, perform attention calculation on the first hidden feature and the first text embedding feature to obtain the second hidden feature.

[0117] Optionally, the first text embedding feature refers to the embedding feature corresponding to the first target text sequence, and the first target text sequence is each text sequence in the real text sequence before predicting the current recognized text.

[0118] Exemplarily, the real text sequence can be represented as y raw =(y r1 , y r2 ,…, y rs ), where y ri is the i-th real text, and s is the total length of the text sequence. For the real text sequence {text1, text2, text3, text4,...}, during the prediction recognition process, the prediction and recognition are performed character by character for each text in the real text sequence. For example, if the currently predicted recognized text is text1, the first target text sequence can be the start symbol. If the currently predicted recognized text is text2, the first target text sequence can be < / s + text1>. If the currently predicted recognized text is text3, the first target text sequence can be < / s + text1 + text2>, and so on. That is, each time the current text is predicted and recognized, it is necessary to cover the current recognized text and each text after the current recognized text in the real text sequence.

[0119] Optionally, the first text embedding feature corresponding to the first target text sequence in the real text sequence can be obtained through an embedding module. Then, input the first text embedding feature and the first hidden feature h enc into the first decoding module in the first decoding network in the voice smoothing model, and perform attention calculation on the first hidden feature h enc and the first text embedding feature to obtain the second hidden feature h raw=(h r1 ,h r2 ,…,h rs ), h rj Let j be the j-th hidden feature. This second hidden feature contains feature information of all text content corresponding to the speech data, as well as the hidden dependencies between each text and other texts. For example, all text content can include normal speech text, spoken information text (such as "um," "uh," etc.), and text with repeated words (such as "this," "this," said twice, so the text content also contains "this" twice). That is, all text content corresponds one-to-one with the speech content.

[0120] S1033, the second hidden feature is input into the recognition module in the first decoding network to perform speech recognition prediction, and a predicted text recognition result is obtained; the predicted text recognition result is used to indicate the predicted recognition text sequence corresponding to the sample speech data.

[0121] Optionally, the second hidden feature h raw As the output of the first decoding module, it is input to the recognition module in the first decoding network for speech recognition prediction, resulting in a predicted text recognition result. This predicted text recognition result is used to indicate the predicted recognized text sequence corresponding to the sample speech data. This predicted text recognition result can be represented as a predicted text label sequence. Let k be the k-th text label.

[0122] S104, obtain the first hidden feature output by the encoding network and the second hidden feature output by the first decoding network, and perform speech smoothness prediction on the first hidden feature and the second hidden feature through the second decoding network to obtain the prediction smoothness result.

[0123] The first hidden feature output by the encoding network is h. enc =(h e1 ,h e2 ,…,h et The second hidden feature output by the first decoding network is h. raw =(h r1 ,h r2 ,…,h rs ).

[0124] In an alternative implementation, such as Figure 2 As shown, the step of performing speech smoothness prediction on the first latent feature and the second latent feature through the second decoding network to obtain the predicted smoothness result includes:

[0125] S201. Input the first hidden feature, the second hidden feature, and the second target text sequence in the smooth text sequence into the second decoding module in the second decoding network for decoding processing to obtain a third hidden feature. The second target text sequence is each text sequence in the smooth text sequence before predicting the current smooth text.

[0126] In an optional embodiment, the second decoding module includes a first decoding sub-module and a second decoding sub-module. The step of inputting the first hidden feature, the second hidden feature, and the second target text sequence in the smooth text sequence into the second decoding module in the second decoding network for decoding processing to obtain a third hidden feature includes:

[0127] S2011. Obtain the second text embedding feature corresponding to the second target text sequence in the smooth text sequence.

[0128] S2012. Based on the first decoding sub-module, perform attention calculation on the first hidden feature and the second text embedding feature to obtain an intermediate hidden feature.

[0129] S2013. Based on the second decoding sub-module, perform attention calculation on the second hidden feature and the intermediate hidden feature to obtain the third hidden feature.

[0130] Optionally, the second text embedding feature refers to the embedding feature corresponding to the second target text sequence, and the second target text sequence is each text sequence in the smooth text sequence before predicting the current smooth text.

[0131] Exemplarily, the smooth text sequence can be represented as y flu =(y f1 , y f2 , …, y fv ), where y fi is the i-th smooth text. For the smooth text sequence {text a, text b, text c, text d,...}, during the prediction of smoothness, it is predicted smoothly character by character according to the text sequence of the smooth text sequence. For example, if the current smooth text to be predicted is text a, the first target text sequence can be the start symbol. If the current smooth text to be predicted is text b, the first target text sequence can be < / s+text a>. If the current smooth text to be predicted is text c, the first target text sequence can be < / s+text a+text b>, and so on. That is, each time when predicting the current smooth text, it is necessary to cover the current smooth text and each text after the current text in the smooth text sequence.

[0132] Optionally, the second text embedding feature corresponding to the second target text sequence in the smooth text sequence can be obtained through the embedding module. Then, the second text embedding feature and the first latent feature h are combined. enc The input is fed into the first decoding submodule of the second decoding network in the speech smoothness model, and the first latent feature h is processed. enc Attention is calculated on the second text embedding features to obtain intermediate latent features. Then, the second latent feature h is... raw The intermediate latent features are input into the second decoding submodule of the second decoding network to process the second latent feature h. raw Attention is calculated between the intermediate latent features to obtain the third latent feature h. flu =(h f1 ,h f2 ,…,h fV ), h fj Let j be the j-th sub-hidden feature. This third hidden feature contains feature information of all smooth text content corresponding to the speech data, as well as the dependency relationships between each smooth text and other smooth text. For example, for the input speech "This is a mobile phone", its corresponding smooth text is "This is a mobile phone".

[0133] S202, the third hidden feature is input into the smoothing module in the second decoding network to perform speech smoothing prediction and obtain the predicted smoothing result; the predicted smoothing result is used to indicate the predicted smoothing text sequence corresponding to the sample speech data.

[0134] Optionally, the third hidden feature h flu As the output of the second decoding module, it is input to the smoothing module in the second decoding network for speech smoothing prediction, yielding a predicted smoothing result. This predicted smoothing result is used to indicate the predicted smoothed text sequence corresponding to the sample speech data. This predicted smoothing result can be represented as a predicted smoothed text label sequence. This is the k-th smooth text tag.

[0135] S105, based on the predicted recognition result, the real text sequence, the smooth text sequence, and the predicted smoothness result, determine the total training loss.

[0136] In an optional implementation, determining the total training loss based on the predicted text recognition result, the real text sequence, the smooth text sequence, and the predicted smoothness result includes:

[0137] Based on the difference between the predicted text recognition result and the real text sequence, the text recognition loss is determined;

[0138] Based on the difference between the smooth text sequence and the predicted smoothness result, the smoothness loss is determined;

[0139] The total training loss is determined based on the recognition loss and the smoothing loss.

[0140] Optionally, the text recognition loss can be determined based on the difference between the predicted text recognition result and the actual text sequence. This text recognition loss is used to define the second latent feature h. raw Under the given conditions, the predicted text label sequence corresponding to the predicted text recognition result predicted by the first decoding module. With the real text sequence y raw =(y r1 ,y r2 ,…,y rs The cross-entropy loss between the two sequences, where s is the length of the text sequence. This text recognition loss is Loss. raw It can be represented as:

[0141]

[0142] Where s is the total length of the text sequence, and p is the probability of correctly identifying the text sequence.

[0143] Optionally, the smoothing loss can be determined based on the difference between the smoothed text sequence and the predicted smoothing result. This smoothing loss is defined on the third latent feature h. flu Under the given conditions, the predicted smooth text label sequence corresponding to the predicted smooth result predicted by the second decoding module. Compared to the real original smooth text sequence y flu =(y f1 ,y f2 ,…,y fv The cross-entropy loss between ) and the smooth loss. raw It can be represented as:

[0144]

[0145] Where v is the total length of the smooth text sequence, and p is the probability of correctly predicting the smooth text sequence.

[0146] In another optional embodiment, where the encoding network includes a syllable recognition module, the step of performing speech recognition prediction on the sample speech data through the encoding network and the first decoding network to obtain a predicted text recognition result further includes:

[0147] The first hidden feature is input into the syllable recognition module for syllable recognition processing to obtain the predicted syllable recognition result.

[0148] Optionally, the first latent feature output by the encoding module is input into the syllable recognition module in the encoding network for syllable recognition processing to obtain the predicted syllable recognition result. Then,

[0149] At this time, as Figure 3 As shown, determining the total training loss based on the recognition loss and the smoothing loss includes:

[0150] S301, Based on the difference between the predicted syllable recognition result and the syllable sequence corresponding to the real text sequence, determine the syllable recognition loss.

[0151] Optionally, the syllable recognition loss is determined based on the difference between the predicted syllable recognition result and the syllable sequence corresponding to the real text sequence. This syllable recognition module can be a CTC (Connectionist Temporal Classification) module, which recognizes each audio frame of the speech, thus ensuring that the recognition of each audio frame is conditionally independent. The CTC module maximizes the value of the real text sequence y corresponding to the speech. raw Based on the difference between the predicted syllable recognition result and the syllable sequence corresponding to the real text sequence, the CTC loss corresponding to the syllable recognition loss is determined, which can be expressed as:

[0152] Loss ctc =-ln p(y raw |h enc )=-lnΠ t p(y r |h et )

[0153] Where t is the total length of the audio frame, and p is the probability of predicting the syllable sequence corresponding to the real text sequence.

[0154] S302, based on the syllable recognition loss, the recognition loss, and the smoothness loss, determine the total training loss.

[0155] In an optional implementation, determining the total training loss based on the syllable recognition loss, the recognition loss, and the smoothness loss includes:

[0156] S3021, Obtain the loss weights corresponding to the syllable recognition loss, the recognition loss, and the smoothness loss respectively;

[0157] S3022, based on their respective loss weights, the syllable recognition loss, the recognition loss, and the smoothness loss are weighted and summed to obtain the total training loss.

[0158] Optionally, before determining the total training loss, it is necessary to obtain the loss weights corresponding to the syllable recognition loss, recognition loss, and smoothness loss. Then, based on their respective loss weights, the syllable recognition loss, recognition loss, and smoothness loss are weighted and summed to obtain the total training loss. That is, the total training loss can be expressed as:

[0159] Loss combined (x,y raw ,y flu )=λ1Loss ctc (x,y raw )+λ2Loss raw (x,y raw )+λ3Loss flu (x,y raw ,y fluen )

[0160] Wherein, λ1, λ2 and λ3 are the loss weights corresponding to syllable recognition loss, recognition loss and smoothness loss respectively, and the sum of these three loss weights can be 1.

[0161] S106, Train the speech smoothing model based on the total training loss until the training termination condition is met, and obtain the trained speech smoothing model.

[0162] Optionally, after determining the total training loss, the speech smoothing model can be trained using this total training loss until the training termination condition is met, thus obtaining the trained speech smoothing model. This training termination condition can be minimizing the total training loss, reaching a preset number of training iterations, or meeting preset training accuracy conditions, etc.

[0163] Figure 4 This is a schematic diagram of the model structure of a speech smoothing model provided in the embodiments of this application, such as... Figure 4 As shown, the speech smoothing module includes an encoding network, a first decoding network, and a second decoding network.

[0164] The encoding network includes an embedding module, an encoding module, and a CTC module. The encoding module includes a multi-head attention module 1 and a first feedforward module.

[0165] The first decoding network includes an embedding module, a first decoding module, and a recognition module. The first decoding module includes a multi-head attention module 2, a second feedforward module, and a third feedforward module.

[0166] The second decoding network includes an embedding module, a second decoding module, and a smoothing module. The second decoding module includes a multi-head attention module 3, a fourth feedforward module, a fifth feedforward module, and a sixth feedforward module connected sequentially. The first decoding submodule is the fourth feedforward module, and the second decoding submodule includes the fifth and sixth feedforward modules.

[0167] Each of the aforementioned feedforward modules includes a feedforward network structure. In addition, each multi-head attention module and each feedforward module can introduce a residual structure. This residual structure can include a residual module for performing residual connections and a normalization module for performing normalization operations. This residual structure can alleviate the problem of model degradation.

[0168] The total training loss is constructed by using the losses corresponding to the encoding network, the first decoding network, and the second decoding network in the speech smoothing model. The model is then trained using this total training loss to obtain the trained speech smoothing model. The trained speech smoothing module is then deployed to achieve the speech smoothing processing task.

[0169] In the above embodiment, the speech recognition prediction of sample speech data is performed through the encoding network and the first decoding network in the speech smoothing model to obtain the predicted text recognition result. The first latent feature output by the encoding network and the second latent feature output by the first decoding network are obtained. Speech smoothing prediction is then performed on these first and second latent features through the second decoding network in the speech smoothing model to obtain the predicted smoothing result. The speech smoothing model is trained based on the total loss function constructed from the various prediction results to obtain the trained speech smoothing model. Thus, while utilizing the original speech information, it also completely preserves the speaker's pronunciation habits and other personal information, reducing misjudgments in speech recognition and improving the accuracy of speech smoothing.

[0170] Figure 5 This is a flowchart illustrating a speech smoothing method provided in an embodiment of this application. The method can be applied to a speech smoothing device, which may include at least one of a terminal and a server. The terminal includes, but is not limited to, personal computers, laptops, and smartphones. The server may be a standalone server, a server cluster, or a cloud server. The method can be implemented in software and / or hardware. Please refer to [link to relevant documentation]. Figure 5 The method may include:

[0171] S501, obtain the voice to be smoothed;

[0172] S502, the trained speech smoothing model is invoked to perform speech smoothing processing on the speech to be smoothed, and the speech smoothing processing result is obtained;

[0173] The trained speech smoothing model is obtained by any of the methods described above.

[0174] Optionally, the speech to be smoothed can be the original speech requiring smoothing in various scenarios, such as meeting scenarios, recording scenarios, training scenarios, and other scenarios requiring speech recognition. Next, a trained speech smoothing model is invoked to process the acquired speech to be smoothed, obtaining the smoothed speech result. For example, taking a meeting scenario, by removing useless spoken text information, incorrect word order, and slips of the tongue from the meeting conversation, the original speech is smoothed, resulting in the processed smoothed text.

[0175] For details and beneficial effects not disclosed in the embodiments of the speech smoothing method of this application, please refer to the above embodiments of this application.

[0176] The following are embodiments of the apparatus described in this application, which can be used to execute the method embodiments described above. For details and beneficial effects not disclosed in the apparatus embodiments of this application, please refer to the method embodiments of this application.

[0177] Please refer to Figure 6 This diagram illustrates a block diagram of a speech smoothness model training apparatus according to an embodiment of this application. The speech smoothness model training apparatus has the functionality to implement the method example described above; this functionality can be implemented in hardware or by hardware executing corresponding software. The speech smoothness model training apparatus may include:

[0178] The sample acquisition module 610 is used to acquire training samples, the training samples including sample speech data, and the real text sequence and the corresponding smooth text sequence corresponding to the sample speech data;

[0179] The model determination module 620 is used to determine a speech smoothing model, which sequentially includes an encoding network, a first decoding network, and a second decoding network.

[0180] The first prediction module 630 is used to perform speech recognition prediction on the sample speech data through the encoding network and the first decoding network to obtain the predicted text recognition result.

[0181] The second prediction module 640 is used to obtain the first hidden feature output by the encoding network and the second hidden feature output by the first decoding network, and to perform speech smoothness prediction on the first hidden feature and the second hidden feature through the second decoding network to obtain the prediction smoothness result.

[0182] The loss determination module 650 is used to determine the total training loss based on the predicted recognition result, the real text sequence, the smooth text sequence, and the predicted smoothness result;

[0183] Training module 660 is used to train the speech smoothing model based on the total training loss until the training termination condition is met, thereby obtaining the trained speech smoothing model.

[0184] In an optional implementation, the first prediction module 630 includes:

[0185] The first processing submodule is used to input the sample speech data into the encoding module in the encoding network for feature encoding processing to obtain the first hidden feature;

[0186] The second processing submodule is used to input the first hidden feature and the first target text sequence in the real text sequence into the first decoding module in the first decoding network for decoding processing to obtain the second hidden feature; the first target text sequence is the text sequence in the real text sequence before the prediction of the current recognized text;

[0187] The first prediction submodule is used to input the second latent feature into the recognition module in the first decoding network to perform speech recognition prediction and obtain the predicted text recognition result; the predicted text recognition result is used to indicate the predicted recognition text sequence corresponding to the sample speech data.

[0188] In an optional implementation, the second processing submodule is specifically used for:

[0189] Obtain the first text embedding feature corresponding to the first target text sequence in the real text sequence;

[0190] Based on the first decoding module in the first decoding network, attention is calculated on the first hidden feature and the first text embedding feature to obtain the second hidden feature.

[0191] In an optional implementation, the second prediction module 640 includes:

[0192] The third processing submodule is used to input the first hidden feature, the second hidden feature, and the second target text sequence in the smooth text sequence into the second decoding module in the second decoding network for decoding processing to obtain the third hidden feature; the second target text sequence is the text sequence in the smooth text sequence before the prediction of the current smooth text.

[0193] The second prediction submodule is used to input the third hidden feature into the smoothing module in the second decoding network to perform speech smoothing prediction and obtain the prediction smoothing result; the prediction smoothing result is used to indicate the prediction smoothing text sequence corresponding to the sample speech data.

[0194] In one optional embodiment, the second decoding module includes a first decoding submodule and a second decoding submodule; the third processing submodule is specifically used for:

[0195] Obtain the second text embedding feature corresponding to the second target text sequence in the smooth text sequence;

[0196] Based on the first decoding submodule, attention calculation is performed on the first latent feature and the second text embedding feature to obtain the intermediate latent feature;

[0197] Based on the second decoding submodule, attention calculation is performed on the second hidden feature and the intermediate hidden feature to obtain the third hidden feature.

[0198] In an optional implementation, the loss determination module 650 includes:

[0199] The first loss determination submodule is used to determine the text recognition loss based on the difference between the predicted text recognition result and the real text sequence;

[0200] The second loss determination submodule is used to determine the smoothing loss based on the difference between the smooth text sequence and the predicted smoothing result;

[0201] The first total loss determination submodule is used to determine the total training loss based on the recognition loss and the smoothing loss.

[0202] In an alternative embodiment, where the encoding network includes a syllable recognition module, the first prediction module further includes:

[0203] The third prediction submodule is used to input the first latent feature into the syllable recognition module for syllable recognition processing to obtain the predicted syllable recognition result; the first total loss determination submodule includes:

[0204] The third loss determination submodule is used to determine the syllable recognition loss based on the difference between the predicted syllable recognition result and the syllable sequence corresponding to the real text sequence;

[0205] The total loss determination unit is used to determine the total training loss based on the syllable recognition loss, the recognition loss, and the smoothness loss.

[0206] In an optional implementation, the total loss determination unit is specifically used for:

[0207] Obtain the loss weights corresponding to the syllable recognition loss, the recognition loss, and the smoothness loss, respectively;

[0208] Based on their respective loss weights, the syllable recognition loss, the recognition loss, and the smoothness loss are weighted and summed to obtain the total training loss.

[0209] For details and beneficial effects not disclosed in the embodiments of the speech smoothness model training device of this application, please refer to the above embodiments of this application.

[0210] Please refer to Figure 7 The diagram illustrates a block diagram of a speech smoothing device according to an embodiment of this application. This speech smoothing device has the functionality to implement the method example described above; the functionality can be implemented in hardware or by hardware executing corresponding software. The speech smoothing device may include:

[0211] The voice acquisition module 710 is used to acquire the voice to be smoothed.

[0212] The speech smoothing module 720 is used to call the trained speech smoothing model to perform speech smoothing processing on the speech to be smoothed, and obtain the speech smoothing processing result;

[0213] The trained speech smoothing model is obtained by any of the methods described above.

[0214] For details and beneficial effects not disclosed in the embodiments of the voice smoothing device of this application, please refer to the above embodiments of this application.

[0215] This application also provides a computer device, the device including a processor and a memory, the memory storing at least one instruction or at least one program, the at least one instruction or at least one program being loaded and executed by the processor to implement the training method of the speech smoothness model as described above, or the training method of the speech smoothness model.

[0216] This application also provides a computer-readable storage medium storing at least one instruction or at least one program, which is loaded and executed by a processor to implement the training method for a speech smoothing model as described above, or the training method for a speech smoothing model.

[0217] In some embodiments, the computer device (not shown) may include a processor, memory, and a network interface connected via a system bus. The processor of the computer device provides computing and control capabilities. The memory of the computer device includes non-volatile storage media and internal memory. The non-volatile storage media stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface of the computer device is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements a method for training a speech smoothing model, or a method for training a speech smoothing model.

[0218] It should be noted that the order of the embodiments described above is merely for descriptive purposes and does not represent the superiority or inferiority of the embodiments. Furthermore, specific embodiments have been described above. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps described in the claims can be performed in a different order than that shown in the embodiments and still achieve the desired result. Additionally, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0219] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the device and server embodiments are basically similar to the method embodiments, so the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0220] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0221] The above description is the preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications are also considered to be within the scope of protection of this application.

Claims

1. A training method for a speech smoothness model, characterized in that, include: Obtain training samples, which include sample speech data, as well as the corresponding real text sequence and the corresponding smooth text sequence; A speech smoothing model is determined, wherein the speech smoothing model sequentially includes an encoding network, a first decoding network, and a second decoding network; The sample speech data is used to perform speech recognition prediction through the encoding network and the first decoding network to obtain the predicted text recognition result; The first hidden feature output by the encoding network and the second hidden feature output by the first decoding network are obtained. The first hidden feature and the second hidden feature are then used by the second decoding network to perform speech smoothness prediction to obtain the prediction smoothness result. Based on the predicted text recognition result, the real text sequence, the smooth text sequence, and the predicted smoothness result, the total training loss is determined; The speech smoothing model is trained based on the total training loss until the training termination condition is met, thus obtaining the trained speech smoothing model.

2. The training method for the speech smoothness model according to claim 1, characterized in that, The step of performing speech recognition prediction on the sample speech data through the encoding network and the first decoding network to obtain the predicted text recognition result includes: The sample speech data is input into the encoding module of the encoding network for feature encoding processing to obtain the first hidden feature; The first hidden feature and the first target text sequence in the real text sequence are input into the first decoding module in the first decoding network for decoding processing to obtain the second hidden feature; the first target text sequence is the text sequence in the real text sequence before the prediction of the current recognized text. The second hidden feature is input into the recognition module in the first decoding network to perform speech recognition prediction, and a predicted text recognition result is obtained; the predicted text recognition result is used to indicate the predicted recognition text sequence corresponding to the sample speech data.

3. The training method for the speech smoothness model according to claim 2, characterized in that, The step of inputting the first hidden feature and the first target text sequence in the real text sequence into the first decoding module in the first decoding network for decoding processing to obtain the second hidden feature includes: Obtain the first text embedding feature corresponding to the first target text sequence in the real text sequence; Based on the first decoding module in the first decoding network, attention is calculated on the first hidden feature and the first text embedding feature to obtain the second hidden feature.

4. The training method for the speech smoothness model according to claim 1, characterized in that, The step of performing speech smoothness prediction on the first latent feature and the second latent feature through the second decoding network to obtain the predicted smoothness result includes: The first hidden feature, the second hidden feature, and the second target text sequence in the smooth text sequence are input into the second decoding module in the second decoding network for decoding processing to obtain the third hidden feature; the second target text sequence is the text sequence in the smooth text sequence before the prediction of the current smooth text. The third hidden feature is input into the smoothing module in the second decoding network to perform speech smoothing prediction, and the predicted smoothing result is obtained; the predicted smoothing result is used to indicate the predicted smoothing text sequence corresponding to the sample speech data.

5. The training method for the speech smoothness model according to claim 4, characterized in that, The second decoding module includes a first decoding submodule and a second decoding submodule; The step of inputting the first hidden feature, the second hidden feature, and the second target text sequence in the smooth text sequence into the second decoding module in the second decoding network for decoding processing to obtain the third hidden feature includes: Obtain the second text embedding feature corresponding to the second target text sequence in the smooth text sequence; Based on the first decoding submodule, attention calculation is performed on the first latent feature and the second text embedding feature to obtain the intermediate latent feature; Based on the second decoding submodule, attention calculation is performed on the second hidden feature and the intermediate hidden feature to obtain the third hidden feature.

6. The training method for the speech smoothness model according to any one of claims 1-5, characterized in that, The determination of the total training loss based on the predicted text recognition result, the real text sequence, the smooth text sequence, and the predicted smoothness result includes: Based on the difference between the predicted text recognition result and the real text sequence, the text recognition loss is determined; Based on the difference between the smooth text sequence and the predicted smoothness result, the smoothness loss is determined; The total training loss is determined based on the recognition loss and the smoothing loss.

7. The training method for the speech smoothness model according to claim 6, characterized in that, When the encoding network includes a syllable recognition module, the step of performing speech recognition prediction on the sample speech data through the encoding network and the first decoding network to obtain the predicted text recognition result further includes: The first hidden feature is input into the syllable recognition module for syllable recognition processing to obtain the predicted syllable recognition result; The determination of the total training loss based on the recognition loss and the smoothing loss includes: Based on the difference between the predicted syllable recognition result and the syllable sequence corresponding to the real text sequence, the syllable recognition loss is determined; The total training loss is determined based on the syllable recognition loss, the recognition loss, and the smoothness loss.

8. The training method for the speech smoothness model according to claim 7, characterized in that, The determination of the total training loss based on the syllable recognition loss, the recognition loss, and the smoothness loss includes: Obtain the loss weights corresponding to the syllable recognition loss, the recognition loss, and the smoothness loss, respectively; Based on their respective loss weights, the syllable recognition loss, the recognition loss, and the smoothness loss are weighted and summed to obtain the total training loss.

9. A method for smoothing speech, characterized in that, include: Obtain the speech to be smoothed; The trained speech smoothing model is invoked to perform speech smoothing processing on the speech to be smoothed, and the speech smoothing processing result is obtained; The trained speech smoothing model is obtained by the method described in any one of claims 1-8.

10. A training device for a speech smoothness model, characterized in that, include: The sample acquisition module is used to acquire training samples, which include sample speech data, as well as the real text sequence and the corresponding smooth text sequence corresponding to the sample speech data; A model determination module is used to determine a speech smoothing model, wherein the speech smoothing model includes an encoding network, a first decoding network, and a second decoding network in sequence; The first prediction module is used to perform speech recognition prediction on the sample speech data through the encoding network and the first decoding network to obtain the predicted text recognition result. The second prediction module is used to obtain the first hidden feature output by the encoding network and the second hidden feature output by the first decoding network, and to perform speech smoothness prediction on the first hidden feature and the second hidden feature through the second decoding network to obtain the prediction smoothness result. The loss determination module is used to determine the total training loss based on the predicted text recognition result, the real text sequence, the smooth text sequence, and the predicted smoothness result; The training module is used to train the speech smoothing model based on the total training loss until the training termination condition is met, thereby obtaining the trained speech smoothing model.

11. A voice smoothing device, characterized in that, include: The voice acquisition module is used to acquire the voice to be smoothed. The speech smoothing module is used to call the trained speech smoothing model to perform speech smoothing processing on the speech to be smoothed, and obtain the speech smoothing processing result; The trained speech smoothing model is obtained by the method described in any one of claims 1-8.

12. A computer-readable storage medium, characterized in that, The storage medium stores at least one instruction or at least one program, which is loaded and executed by a processor to implement the training method of the speech smoothing model as described in any one of claims 1-8, or the speech smoothing method as described in claim 9.

Images