Voice data processing method and device, voice generation method and device

Abstract

The embodiments of this specification provide a speech data processing method and device, and a speech generation method and device, wherein the speech data processing method comprises: acquiring multiple speech data having a frame-level alignment relationship; constructing self-supervisory features corresponding to the multiple speech data respectively, and generating average features for eliminating timbre information based on the self-supervisory features corresponding to the multiple speech data respectively; training an average feature prediction network using the average features and the multiple speech data until a target average feature prediction network that meets the training stop condition is obtained; wherein the target average feature prediction network is used to construct a target average feature for eliminating timbre information for the target speech data in a speech conversion task.

Description

Technical Field

[0001] The embodiments of the present specification relate to the field of audio processing technology, and in particular to a method and device for processing voice data, and a method and device for generating voice. Background Art

[0002] With the development of computer technology, audio processing technology has been applied in more and more scenarios. Especially in the voice conversion task in the voice conversion scenario, by targeting a piece of speech, the timbre can be changed without changing its pronunciation content, so as to facilitate the use of downstream businesses, such as dubbing. In the prior art, in order to achieve the above effect, the timbre information and content information in the speech are usually decoupled, and then the timbre information that meets the use requirements is replaced to achieve the goal. At present, the more conventional operation is to implement it through the GAN network, feature decoupling based on the ASR system, or feature decoupling based on spectral perturbations. However, the above schemes all have the problem of incomplete decoupling. In this case, it is easy to cause the final converted speech to deviate from the actual expectation. Therefore, an effective solution is urgently needed to solve the above problem. Summary of the invention

[0003] In view of this, an embodiment of this specification provides a method for processing speech data. One or more embodiments of this specification also relate to a speech data processing device, a speech generation method, a speech generation device, a computing device, a computer-readable storage medium, and a computer program product to solve the technical defects existing in the prior art.

[0004] According to a first aspect of an embodiment of this specification, a method for processing speech data is provided, including:

[0005] Acquire multiple speech data with frame-level alignment relationship;

[0006] Constructing self-supervisory features corresponding to the plurality of speech data respectively, and generating an average feature for eliminating timbre information according to the self-supervisory features corresponding to the plurality of speech data respectively;

[0007] Training an average feature prediction network using the average feature and the plurality of speech data until a target average feature prediction network that meets a training stop condition is obtained;

[0008] The target average feature prediction network is used to construct a target average feature that eliminates timbre information for target speech data in a speech conversion task.

[0009] Optionally, the acquiring of a plurality of speech data having a frame-level alignment relationship includes:

[0010] Determining text information and multiple speaker identifiers corresponding to the network training task, and performing fixed processing on a duration prediction module in a text-to-speech model associated with the network training task;

[0011] Inputting the text information and the multiple speaker identifiers into a fixed processed text-to-speech model for speech synthesis to obtain speech data corresponding to the multiple speaker identifiers respectively;

[0012] The speech data corresponding to the multiple speaker identifiers have a frame-level alignment relationship.

[0013] Optionally, constructing the self-supervisory features corresponding to the plurality of speech data respectively includes:

[0014] Inputting the plurality of speech data into an audio processing model for processing to obtain self-supervisory features corresponding to the plurality of speech data respectively;

[0015] Among them, the self-supervisory features include the content information and timbre information of the speech data.

[0016] Optionally, generating an average feature for eliminating timbre information according to the self-supervisory features respectively corresponding to the plurality of speech data includes:

[0017] Perform feature average calculation on the self-supervisory features corresponding to the plurality of speech data, and obtain the average feature that eliminates the timbre information according to the calculation result; or,

[0018] Content features are extracted from the self-supervisory features respectively corresponding to the plurality of speech data, and feature average calculation is performed on the content features respectively corresponding to the plurality of speech data, and average features that eliminate timbre information are obtained according to the calculation results.

[0019] Optionally, the step of training an average feature prediction network using the average feature and the plurality of speech data until a target average feature prediction network that satisfies a training stop condition is obtained comprises:

[0020] Selecting sample speech data from the plurality of speech data, and determining sample self-supervisory features corresponding to the sample speech data;

[0021] Inputting the sample self-supervisory features into the average feature prediction network for processing to obtain predicted average features;

[0022] The average feature prediction network is adjusted based on the predicted average feature and the average feature until a target average feature prediction network that meets a training stop condition is obtained.

[0023] Optionally, the adjusting parameters of the average feature prediction network based on the predicted average feature and the average feature until a target average feature prediction network that meets a training stop condition is obtained includes:

[0024] Calculating a loss value based on the predicted average feature and the average feature, and adjusting parameters of the average feature prediction network based on the loss value;

[0025] Check whether the average feature prediction network after parameter adjustment meets the training stop condition;

[0026] If not, selecting associated speech data from the plurality of speech data, using the associated speech data as sample speech data, and executing the step of determining a sample self-supervisory feature corresponding to the sample speech data;

[0027] If so, the average feature prediction network after parameter adjustment is used as the target average feature prediction network.

[0028] According to a second aspect of the embodiments of this specification, there is provided a speech generation method, comprising:

[0029] Acquire text information corresponding to the speech conversion task, and convert the text information into target speech data;

[0030] Constructing a target self-supervisory feature corresponding to the target speech data;

[0031] Inputting the target self-supervisory feature into the target average feature prediction network in the above method for processing to obtain the target average feature;

[0032] The target average feature is input into a speech conversion network for processing to obtain speech data associated with a target speaker in the speech conversion task.

[0033] According to a third aspect of the embodiments of this specification, there is provided a speech data processing device, including:

[0034] An acquisition module is configured to acquire a plurality of speech data having a frame-level alignment relationship;

[0035] A construction module is configured to construct self-supervisory features corresponding to the plurality of speech data respectively, and generate an average feature for eliminating timbre information according to the self-supervisory features corresponding to the plurality of speech data respectively;

[0036] A training module, configured to train an average feature prediction network using the average feature and the plurality of speech data until a target average feature prediction network that meets a training stop condition is obtained;

[0037] The target average feature prediction network is used to construct a target average feature that eliminates timbre information for target speech data in a speech conversion task.

[0038] According to a fourth aspect of the embodiments of this specification, there is provided a speech generating device, including:

[0039] An information acquisition module is configured to acquire text information corresponding to the speech conversion task and convert the text information into target speech data;

[0040] A feature construction module is configured to construct a target self-supervisory feature corresponding to the target speech data;

[0041] A feature processing module is configured to input the target self-supervisory feature into the target average feature prediction network in the above method for processing to obtain the target average feature;

[0042] The speech generating module is configured to input the target average feature into the speech conversion network for processing, and obtain speech data associated with the target speaker in the speech conversion task.

[0043] According to a fifth aspect of an embodiment of this specification, a computing device is provided, including:

[0044] Memory and processor;

[0045] The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the above-mentioned voice data processing method or voice generation method are implemented.

[0046] According to a sixth aspect of the embodiments of this specification, a computer-readable storage medium is provided, which stores computer-executable instructions, which, when executed by a processor, implement the steps of the above-mentioned voice data processing method or voice generation method.

[0047] According to the seventh aspect of the embodiments of this specification, a computer program product is provided, including a computer program or instructions, which, when executed by a processor, implement the steps of the above-mentioned voice data processing method or voice generation method.

[0048] The speech data processing method provided in this embodiment can completely decouple timbre information and content information so that the downstream can use speech features that only contain content information and eliminate timbre information. A neural network that can eliminate timbre information can be trained; specifically, multiple speech data with frame-level alignment can be obtained first; the speech data can be aligned from dimensions such as content and duration. On this basis, self-supervisory features corresponding to multiple speech data can be constructed first, and on this basis, average features that eliminate timbre information can be generated according to the self-supervisory features corresponding to multiple speech data; thus, features that only contain content information but not timbre information can be obtained. Based on this, the average feature prediction network is trained in combination with the average feature and multiple speech data until a target average feature prediction network that meets the training stop condition is obtained; that is, the target average feature prediction network can be used to construct a target average feature that eliminates timbre information for the target speech data in the speech conversion task, so that the downstream business can use the target average feature for speech conversion processing, thereby achieving the purpose of completely decoupling timbre information and content information, reducing the impact of timbre information when converting speech, and more accurately generating speech data corresponding to the specified speaker. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 is a schematic diagram of a voice data processing method provided by an embodiment of this specification;

[0050] Figure 2 is a flow chart of a voice data processing method provided by an embodiment of this specification;

[0051] Figure 3 is a flow chart of a speech generation method provided by an embodiment of this specification;

[0052] Figure 4 It is a structural schematic diagram of a voice data processing device provided by an embodiment of this specification;

[0053] Figure 5 It is a structural schematic diagram of a speech generating device provided by an embodiment of this specification;

[0054] Figure 6 It is a structural block diagram of a computing device provided by an embodiment of this specification. DETAILED DESCRIPTION

[0055] Many specific details are described in the following description to facilitate a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of this specification, so this specification is not limited to the specific implementation disclosed below.

[0056] The terms used in one or more embodiments of this specification are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of this specification. The singular forms of "a", "said" and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0057] It should be understood that although the terms first, second, etc. may be used to describe various information in one or more embodiments of this specification, this information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0058] In addition, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0059] In this specification, a method for processing speech data is provided. One or more embodiments of this specification also relate to a speech data processing device, a speech generation method, a speech generation device, a computing device, a computer-readable storage medium, and a computer program product, which are described in detail one by one in the following embodiments.

[0060] In practical applications, GAN is short for Generative Adversarial Network. It is a deep learning architecture consisting of two competing neural networks - the generator tries to forge realistic data samples, while the discriminator tries to distinguish between real samples and samples forged by the generator. Through continuous adversarial training between the two, the generator eventually learns to generate data that is difficult to distinguish between true and false. In the voice conversion task, the decoupling network will read the complete audio and obtain the desired content information (it does not want to include timbre information), so a discriminator will be added to input the output of the generator to determine who the original speaker of this information is. In the adversarial process, the generator will generate features that are unrelated to the speaker to deceive the discriminator, thereby achieving the effect of feature decoupling. The decoupled content information is restored to the original audio through the restoration network, and the retention of content information is achieved by the audio restoration task.

[0061] ASR (Automatic Speech Recognition) is a process of gradually converting audio into text in the process of speech recognition. Therefore, in the speech recognition network, the information contained in the last hidden layer is considered to be a bottleneck feature of BNF (bottle neck feature) that contains content information but no timbre information. In the task of speech conversion, this feature can usually be directly used as a speaker-independent feature for subsequent speech conversion. In other words, the ASR system plays the role of feature decoupling.

[0062] Spectral perturbation technology refers to the use of resonance peak perturbation operations (a perturbation method that can change the resonance peaks of sound, and the timbre can be perturbed in this way) to obtain sounds that are aligned with the original sound and correspond to other timbres. By converting sounds of other timbres into the original sound, the model's ability to decouple the speaker is trained.

[0063] However, the GAN-based decoupling method usually fails to retain the complete content information while decoupling the timbre information, and often loses content information, resulting in stuttering and swallowing of words in the synthesized sound. The ASR-based decoupling method may cause errors in the original content information due to speech recognition errors, such as recognizing another word. Based on the spectrum perturbation method, since the resonance peak perturbation does not have a very large impact on the timbre, the timbre is still not much different from the original timbre, resulting in the synthesized sound not being very similar to the target timbre. Therefore, an effective solution is urgently needed to solve the above problems.

[0064] See also Figure 1As shown in the schematic diagram, the speech data processing method provided in this embodiment can train a neural network that can eliminate timbre information in order to completely decouple timbre information and content information so that the downstream can use speech features that only contain content information and eliminate timbre information; specifically, multiple speech data with frame-level alignment can be obtained first; and the speech data can be aligned from dimensions such as content and duration. On this basis, self-supervisory features corresponding to multiple speech data can be constructed first, and on this basis, average features that eliminate timbre information can be generated according to the self-supervisory features corresponding to multiple speech data; thus, features that only contain content information but not timbre information can be obtained. Based on this, the average feature prediction network is trained in combination with the average feature and multiple speech data until a target average feature prediction network that meets the training stop condition is obtained; that is, the target average feature prediction network can be used to construct a target average feature that eliminates timbre information for the target speech data in the speech conversion task, so that the downstream business can use the target average feature for speech conversion processing, thereby achieving the purpose of completely decoupling timbre information and content information, reducing the impact of timbre information when converting speech, and more accurately generating speech data corresponding to the specified speaker.

[0065] See also Figure 2 , Figure 2 A flow chart of a voice data processing method provided according to an embodiment of the present specification is shown, which specifically includes the following steps.

[0066] Step S202, obtaining a plurality of speech data having a frame-level alignment relationship.

[0067] The voice data processing method provided in this embodiment can be applied to voice data conversion scenarios in any scenario, such as dubbing scenarios, dubbing for game objects, animation objects, and film and television objects, and the voice data processing method provided in this embodiment can accurately and more realistically generate the voice data corresponding to the specified speaker for use. For example, in a voice call scenario, it can support converting the user's pronunciation into the voice of other speakers, such as robot voice simulation and film and television character voice simulation, thereby increasing the fun of the user's call. This embodiment uses the dubbing scenario as an example to illustrate the voice data processing method. The descriptions in other scenarios can refer to the same or corresponding descriptions in this embodiment, and this embodiment does not make any limitation here.

[0068] Specifically, the multiple voice data with frame-level alignment relationship refer to audio clips expressed by different speakers for the same text content, and the multiple voice data have consistent speech content and the duration of each word is completely consistent, which facilitates the subsequent construction of average features.

[0069] Based on this, in order to completely decouple timbre information and content information so that the downstream can use speech features that only contain content information and eliminate timbre information, a neural network that can eliminate timbre information can be trained; specifically, multiple speech data with frame-level alignment can be obtained first; the speech data can be aligned in terms of content and duration. On this basis, self-supervisory features corresponding to multiple speech data can be constructed first, and on this basis, average features that eliminate timbre information can be generated based on the self-supervisory features corresponding to multiple speech data; thus, features that only contain content information but not timbre information can be obtained. Based on this, the average feature prediction network is trained in combination with the average feature and multiple speech data until a target average feature prediction network that meets the training stop condition is obtained; and the target average feature prediction network can be used to construct a target average feature that eliminates timbre information for the target speech data in the speech conversion task, so that downstream services can use the target average feature for speech conversion processing.

[0070] Furthermore, when acquiring multiple voice data, in order to ensure that the multiple voice data have a frame alignment relationship, so as to facilitate subsequent use, a text-to-speech network can be used. In this embodiment, the specific implementation method is as follows:

[0071] Determine the text information and multiple speaker identifiers corresponding to the network training task, and perform fixed processing on the duration prediction module in the text-to-speech model associated with the network training task; input the text information and the multiple speaker identifiers into the fixedly processed text-to-speech model for speech synthesis to obtain speech data corresponding to the multiple speaker identifiers; wherein the speech data corresponding to the multiple speaker identifiers have a frame-level alignment relationship.

[0072] Specifically, the network training task specifically refers to the task of training the average feature prediction network; correspondingly, the text information specifically refers to the text required to construct multiple voice data; correspondingly, the speaker identification specifically refers to the identification information corresponding to different speakers and each speaker has a different timbre; when generating voice data, voice data corresponding to different speakers can be generated based on the text information. Correspondingly, the text-to-speech model refers to a model that has input text information and a speaker identification, and outputs voice data corresponding to the speaker identification. The model can use the vits model. Correspondingly, the duration prediction module specifically refers to a module in the text-to-speech model that is used to control the duration information of the output voice data. By fixing the parameters of the module, the model can output multiple voice data with frame alignment relationships.

[0073] The number of the multiple speaker identifiers can be set according to actual needs, such as 20 or 30, and this embodiment does not impose any limitation thereto.

[0074] Based on this, in order to generate multiple speech data with frame alignment, different timbre information and the same content information, the text information and multiple speaker identifiers corresponding to the network training task can be determined first. At this time, the duration prediction module in the text-to-speech model associated with the network training task can be fixedly processed; since the duration prediction modules corresponding to different speakers will obtain prediction results of different durations, in order to obtain frame-level aligned speech of different speakers (consistent content and the same duration of each word), the duration prediction module can be frozen, and the speaker identifier can be changed to obtain multiple speech data with frame alignment, different timbre information and the same content information. Therefore, the text information and multiple speaker identifiers can be input into the fixedly processed text-to-speech model for speech synthesis, so that the speech data corresponding to the multiple speaker identifiers can be obtained according to the synthesis results; and the speech data corresponding to the multiple speaker identifiers have a frame-level alignment relationship for subsequent use.

[0075] For example, based on the network training task, the text information is determined to be {Today the weather is fine...}, and it is determined to use 20 different speakers to train the average feature prediction network; on this basis, the duration prediction module in the text-to-speech model vits can be frozen, and by inputting the text information {Today the weather is fine...} and 20 different speaker identifiers into the text-to-speech model vits for processing, 20 voice data with different timbres but frame-level alignment relationships can be obtained, namely, voice data Y1 corresponding to speaker A1, voice data Y2 corresponding to speaker A2... voice data Y20 corresponding to speaker A20, so that the average feature prediction network with the ability to eliminate timbre information can be trained in combination with the above voice data in the future.

[0076] In summary, by using the text-to-speech model of the frozen duration prediction module to process text information, multiple speech data with frame alignment relationships can be generated, and the speech data have different timbres but the same content information, so that when the network is subsequently trained in combination with the speech data with frame alignment relationships, the prediction ability of the trained network can be guaranteed.

[0077] Step S204, constructing self-supervisory features corresponding to the plurality of speech data respectively, and generating an average feature for eliminating timbre information according to the self-supervisory features corresponding to the plurality of speech data respectively.

[0078] Specifically, after obtaining multiple speech data with frame alignment relationships as described above, further, in order to be able to train an average feature prediction network that can eliminate timbre information in features, and to achieve features that only retain content information for subsequent speech generation, it is necessary to construct sample pairs before training the average feature prediction network; and the samples can use multiple speech data with frame alignment relationships, and the sample labels need to be calculated in combination with multiple speech data. Therefore, self-supervisory features corresponding to multiple speech data can be constructed, and then average features that eliminate timbre information can be generated based on the self-supervisory features corresponding to multiple speech data, so that the average features can be used as sample labels for training the average feature prediction network.

[0079] Among them, the self-supervisory feature specifically refers to the high-dimensional vector expression obtained after feature conversion of the speech data, and the self-supervisory feature corresponding to each speech data is different; correspondingly, the average feature specifically refers to the vector expression obtained after integrating the self-supervisory features corresponding to multiple speech data. The average feature eliminates the timbre information and only retains the content information, which is convenient for subsequent use. It can be understood that since multiple speech data have a frame alignment relationship, the self-supervisory features constructed by each have the same dimensional representation except for the timbre information representation. When constructing the average feature, the representation of the timbre information related to the self-supervisory feature is eliminated, and the remaining representation is averaged, so that the average feature that eliminates the timbre information can be obtained for subsequent use.

[0080] Furthermore, when constructing the self-supervisory features corresponding to each voice data, an audio processing model may be used. In this embodiment, the specific implementation is as follows:

[0081] The multiple speech data are input into an audio processing model for processing to obtain self-supervisory features corresponding to the multiple speech data respectively; wherein the self-supervisory features include content information and timbre information of the speech data.

[0082] Specifically, the audio processing model refers to a model that can convert speech data into a high-dimensional vector expression, with the input being speech data and the output being the vector expression corresponding to the speech data. Correspondingly, the content information is the information corresponding to the speech content in the speech data, and the timbre information is the information corresponding to the speaker.

[0083] Based on this, when constructing self-supervised features, multiple speech data can be input into the audio processing model for processing, so that the audio processing model can be used to generate self-supervised features corresponding to each speech data; and, since the audio processing model does not perform special learning to remove timbre information during the training phase, the features extracted by the audio processing model will contain the timbre information and content information involved in the speech data, and the timbre information can be subsequently eliminated by calculation to obtain the average feature for use.

[0084] In summary, by adopting the audio processing model to construct self-supervisory features, the efficiency of constructing self-supervisory features can be effectively improved, so that the subsequent neural network training can be completed more quickly.

[0085] Furthermore, when calculating the average feature, the self-supervisory features of multiple speech data may be combined to calculate the average. In this embodiment, the specific implementation is as follows:

[0086] Perform feature averaging calculation on the self-supervisory features respectively corresponding to the plurality of speech data, and obtain the average feature for eliminating the timbre information according to the calculation result; or extract content features from the self-supervisory features respectively corresponding to the plurality of speech data, and perform feature averaging calculation on the content features respectively corresponding to the plurality of speech data, and obtain the average feature for eliminating the timbre information according to the calculation result.

[0087] Specifically, feature average calculation specifically refers to the calculation process of averaging the self-supervisory features corresponding to multiple speech data to obtain a vector expression. Correspondingly, content features specifically refer to the vector expression of the self-supervisory features that only associates content information and ignores timbre features.

[0088] Based on this, after obtaining the self-supervised features corresponding to multiple speech data, on the one hand, feature averaging calculation can be performed on the self-supervised features corresponding to the multiple speech data, and the average features for eliminating timbre information can be directly obtained based on the calculation results; on the other hand, content features can also be extracted from the self-supervised features corresponding to the multiple speech data, and on this basis, feature averaging calculation can be performed on the content features corresponding to the multiple speech data, so that the average features for eliminating timbre information can be obtained based on the calculation results for subsequent use.

[0089] That is to say, since multiple speech data have a frame-level alignment relationship, the content information of each speech data is consistent, while the timbre information is different. Therefore, after obtaining the self-supervisory features, an average calculation is performed to obtain the average features without affecting the content information, so as to facilitate subsequent use.

[0090] Using the above example, after obtaining the voice data Y1 to Y20 corresponding to speakers A1 to A20, the voice data Y1 to Y20 can be sequentially input into the self-supervised model for processing, so as to obtain the self-supervised feature V1 corresponding to the voice data Y1, the self-supervised feature V2 corresponding to the voice data Y2…the self-supervised feature V20 corresponding to the voice data Y20. Considering that the current self-supervised feature contains not only content information but also timbre information corresponding to different speakers. Therefore, the average feature V can be obtained by calculating the average of the self-supervised features V1 to V20, so as to eliminate the timbre information without affecting the content information, so as to facilitate the subsequent use of the average feature V to train the average feature prediction network.

[0091] In summary, by calculating the mean to construct the average feature, it is possible to eliminate the timbre information in the feature, thereby retaining the expression of the corresponding content information. The network trained on this basis has stronger prediction capabilities.

[0092] Step S206, using the average feature and the multiple speech data to train an average feature prediction network until a target average feature prediction network that meets the training stop condition is obtained; wherein the target average feature prediction network is used to construct a target average feature that eliminates timbre information for the target speech data in the speech conversion task.

[0093] Specifically, after obtaining the average features corresponding to the multiple speech data as described above, further, considering that in the speech conversion task, the reason why the conversion accuracy is affected is because the features carry more or less timbre information, the speech data processing method provided in this embodiment combines the speech data and the average features to train an average feature prediction network that can eliminate timbre information, and then supports the subsequent use of the features output by the network for speech conversion, thereby improving the conversion accuracy. Therefore, in the training stage, the calculated average features can be used as sample labels, and the multiple speech data can be used as samples, and the average feature prediction network can be trained by constructing sample pairs until a target average feature prediction network that meets the training stop condition is obtained; and the target average feature prediction network obtained after training can be used to construct a target average feature that eliminates timbre information for the target speech data in the speech conversion task, so that the downstream speech conversion task can generate speech data of a specified speaker based on the target average feature.

[0094] Among them, the average feature prediction network specifically refers to the self-supervisory features of the input speech data, and the self-supervisory features can be mapped to the neural network of the average features to achieve the purpose of eliminating the timbre information in the self-supervisory features. Correspondingly, the target average feature prediction network specifically refers to the neural network obtained after training the average feature prediction network. Correspondingly, the training stop condition is the condition for stopping the training of the average feature prediction network, which can be a loss value comparison condition, an iteration number condition or a verification set verification condition, and this embodiment does not make any limitation here.

[0095] Furthermore, when the model is trained, the speech data is actually converted into sample features, input into the average feature prediction network for processing, and parameter adjustment is performed in combination with the sample labels. In this embodiment, the specific implementation is as follows:

[0096] Select sample speech data from the multiple speech data, and determine sample self-supervised features corresponding to the sample speech data; input the sample self-supervised features into an average feature prediction network for processing to obtain predicted average features; adjust the parameters of the average feature prediction network based on the predicted average features and the average features until a target average feature prediction network that meets the training stop condition is obtained.

[0097] Furthermore, the parameter adjustment of the average feature prediction network based on the predicted average feature and the average feature until a target average feature prediction network that meets the training stop condition is obtained includes:

[0098] A loss value is calculated based on the predicted average feature and the average feature, and the average feature prediction network is adjusted based on the loss value; whether the average feature prediction network after the adjustment satisfies the training stop condition; if not, associated voice data is selected from the multiple voice data, the associated voice data is used as sample voice data, and the step of determining the sample self-supervisory feature corresponding to the sample voice data is executed; if so, the average feature prediction network after the adjustment is used as the target average feature prediction network.

[0099] Specifically, the sample speech data specifically refers to any one of the multiple speech data, and correspondingly, the sample self-supervised feature specifically refers to the self-supervised feature corresponding to the sample speech data; correspondingly, the predicted average feature specifically refers to the vector expression obtained after the average feature prediction network is used to perform average feature prediction on the self-supervised feature.

[0100] Based on this, in the training stage of the average feature prediction network, sample speech data can be selected from multiple speech data first, and the sample self-supervised features corresponding to the sample speech data can be determined; at this time, the sample self-supervised features can be input into the average feature prediction network for processing to obtain the predicted average features; then the loss value can be calculated based on the predicted average features and the average features, and the average feature prediction network can be adjusted based on the loss value; after the parameters are adjusted, it can be detected whether the average feature prediction network after the adjustment meets the training stop conditions; if not, it means that the network still needs to continue training, so associated speech data can be selected from multiple speech data, and the associated speech data can be used as sample speech data, and the step of determining the sample self-supervised features corresponding to the sample speech data is executed until the conditions are met, and the average feature prediction network after the adjustment can be used as the target average feature prediction network.

[0101] Continuing with the above example, after obtaining the average feature V, you can select speech data Y1 as the sample speech data. After determining the self-supervised feature V1 corresponding to the speech data Y1, input the self-supervised feature V1 into the average feature prediction network for processing. At this time, the predicted average feature V11 will be obtained. The average feature prediction network is adjusted based on the average features V and V11. After the adjustment is completed, new samples are selected to continue training until the target average feature prediction network that meets the prediction accuracy requirements is obtained.

[0102] In practical applications, after obtaining the target average feature prediction network, it is deployed to the speech conversion task. After obtaining the speech data, the self-supervised features corresponding to the speech data can be constructed first, and then the self-supervised features can be input into the target average feature prediction network. The self-supervised features are mapped from vector expressions containing content information and timbre information to average features without timbre information. Finally, the average features are input into the speech conversion network for processing. According to the processing results, the speech data corresponding to the specified speaker can be obtained for use by downstream businesses.

[0103] The speech data processing method provided in this embodiment can completely decouple timbre information and content information so that the downstream can use speech features that only contain content information and eliminate timbre information. A neural network that can eliminate timbre information can be trained; specifically, multiple speech data with frame-level alignment can be obtained first; the speech data can be aligned from dimensions such as content and duration. On this basis, self-supervisory features corresponding to multiple speech data can be constructed first, and on this basis, average features that eliminate timbre information can be generated according to the self-supervisory features corresponding to multiple speech data; thus, features that only contain content information but not timbre information can be obtained. Based on this, the average feature prediction network is trained in combination with the average feature and multiple speech data until a target average feature prediction network that meets the training stop condition is obtained; that is, the target average feature prediction network can be used to construct a target average feature that eliminates timbre information for the target speech data in the speech conversion task, so that the downstream business can use the target average feature for speech conversion processing, thereby achieving the purpose of completely decoupling timbre information and content information, reducing the impact of timbre information when converting speech, and more accurately generating speech data corresponding to the specified speaker.

[0104] Corresponding to the above method embodiment, this specification also provides an embodiment of a speech generation method, Figure 3 FIG. 1 is a flow chart of a speech generation method provided by an embodiment of the present specification. Figure 3 As shown, the method includes:

[0105] Step S302: Acquire text information corresponding to the speech conversion task, and convert the text information into target speech data.

[0106] Step S304: construct a target self-supervisory feature corresponding to the target speech data.

[0107] Step S306: input the target self-supervisory feature into the target average feature prediction network in the above method for processing to obtain the target average feature.

[0108] Step S308: input the target average feature into a speech conversion network for processing to obtain speech data associated with the target speaker in the speech conversion task.

[0109] Specifically, the speech conversion task specifically refers to the task of converting text information into timbre information corresponding to a specified speaker. Accordingly, the target speech data specifically refers to the task of converting text information into an audio clip corresponding to any speaker. Accordingly, the target self-supervisory feature specifically refers to the self-supervisory feature corresponding to the target speech data; accordingly, the target average feature specifically refers to the average feature obtained after eliminating the timbre information in the target self-supervisory feature. The speech data of the target speaker is the audio clip corresponding to the timbre information of the target speaker.

[0110] Based on this, when performing a voice conversion task, the text information corresponding to the voice conversion task can be obtained first, and the text information can be converted into target voice data. After that, the target self-supervisory feature corresponding to the target voice data can be constructed, and then the target self-supervisory feature can be input into the target average feature prediction network in the above method for processing, so as to obtain the target average feature without timbre information, and finally the target average feature can be input into the voice conversion network for processing, and the voice data of the target speaker in the associated voice conversion task can be obtained according to the processing result.

[0111] For example, in the game character dubbing scenario, the dubbing teacher for the target game character is unable to complete the dubbing task due to illness. At this time, another dubbing teacher can first dub the target game character, and then perform voice conversion to obtain the audio of the corresponding sick dubbing teacher. In this process, another dubbing teacher can first record the specified lines as target voice data. Then it can be input into the self-supervised model for processing to obtain self-supervised features containing content information and timbre information; further, the self-supervised features are input into the target average feature prediction network for processing to obtain the target average features that eliminate the timbre information, and finally the target average features are input into the voice conversion model for processing, and the voice data of the corresponding sick dubbing teacher can be obtained according to the processing results, thereby completing the dubbing task of the target game character.

[0112] In summary, by processing the self-supervisory features in combination with the average feature prediction network, it is possible to eliminate the timbre information in the self-supervisory features and reduce the impact of the timbre information when converting speech, so that the speech data corresponding to the specified speaker can be generated more accurately.

[0113] Corresponding to the above method embodiment, this specification also provides a voice data processing device embodiment, Figure 4 FIG. 1 shows a schematic diagram of the structure of a voice data processing device provided by an embodiment of the present specification. Figure 4 As shown, the device comprises:

[0114] An acquisition module 402 is configured to acquire a plurality of speech data having a frame-level alignment relationship;

[0115] A construction module 404 is configured to construct self-supervisory features corresponding to the plurality of speech data respectively, and generate an average feature for eliminating timbre information according to the self-supervisory features corresponding to the plurality of speech data respectively;

[0116] A training module 406 is configured to train an average feature prediction network using the average feature and the plurality of speech data until a target average feature prediction network that meets a training stop condition is obtained;

[0117] The target average feature prediction network is used to construct a target average feature that eliminates timbre information for target speech data in a speech conversion task.

[0118] In an optional embodiment, the acquisition module 402 is further configured to:

[0119] Determine the text information and multiple speaker identifiers corresponding to the network training task, and perform fixed processing on the duration prediction module in the text-to-speech model associated with the network training task; input the text information and the multiple speaker identifiers into the fixedly processed text-to-speech model for speech synthesis to obtain speech data corresponding to the multiple speaker identifiers; wherein the speech data corresponding to the multiple speaker identifiers have a frame-level alignment relationship.

[0120] In an optional embodiment, the construction module 404 is further configured to:

[0121] The multiple speech data are input into an audio processing model for processing to obtain self-supervisory features corresponding to the multiple speech data respectively; wherein the self-supervisory features include content information and timbre information of the speech data.

[0122] In an optional embodiment, the construction module 404 is further configured to:

[0123] Perform feature averaging calculation on the self-supervisory features respectively corresponding to the plurality of speech data, and obtain the average feature for eliminating the timbre information according to the calculation result; or extract content features from the self-supervisory features respectively corresponding to the plurality of speech data, and perform feature averaging calculation on the content features respectively corresponding to the plurality of speech data, and obtain the average feature for eliminating the timbre information according to the calculation result.

[0124] In an optional embodiment, the training module 406 is further configured to:

[0125] Select sample speech data from the multiple speech data, and determine sample self-supervised features corresponding to the sample speech data; input the sample self-supervised features into an average feature prediction network for processing to obtain predicted average features; adjust the parameters of the average feature prediction network based on the predicted average features and the average features until a target average feature prediction network that meets the training stop condition is obtained.

[0126] In an optional embodiment, the training module 406 is further configured to:

[0127] A loss value is calculated based on the predicted average feature and the average feature, and the average feature prediction network is adjusted based on the loss value; whether the average feature prediction network after the adjustment satisfies the training stop condition; if not, associated voice data is selected from the multiple voice data, the associated voice data is used as sample voice data, and the step of determining the sample self-supervisory feature corresponding to the sample voice data is executed; if so, the average feature prediction network after the adjustment is used as the target average feature prediction network.

[0128] The speech data processing device provided in this embodiment can train a neural network that can eliminate timbre information in order to completely decouple timbre information and content information so that the downstream can use speech features that only contain content information and eliminate timbre information. Specifically, multiple speech data with frame-level alignment can be obtained first; the speech data can be aligned from dimensions such as content and duration. On this basis, self-supervisory features corresponding to multiple speech data can be constructed first, and on this basis, average features that eliminate timbre information can be generated according to the self-supervisory features corresponding to multiple speech data; thus, features that only contain content information but not timbre information can be obtained. Based on this, the average feature prediction network is trained in combination with the average feature and multiple speech data until a target average feature prediction network that meets the training stop condition is obtained; that is, the target average feature prediction network can be used to construct a target average feature that eliminates timbre information for the target speech data in the speech conversion task, so that the downstream business can use the target average feature for speech conversion processing, thereby achieving the purpose of completely decoupling timbre information and content information, reducing the impact of timbre information when converting speech, and more accurately generating speech data corresponding to the designated speaker.

[0129] The above is a schematic scheme of a voice data processing device of this embodiment. It should be noted that the technical scheme of the voice data processing device and the technical scheme of the voice data processing method described above belong to the same concept, and the details not described in detail in the technical scheme of the voice data processing device can be referred to the description of the technical scheme of the voice data processing method described above.

[0130] Corresponding to the above method embodiment, this specification also provides a speech generation device embodiment, Figure 5 FIG. 1 shows a schematic diagram of the structure of a speech generating device provided by an embodiment of the present specification. Figure 5 As shown, the device comprises:

[0131] An information acquisition module 502 is configured to acquire text information corresponding to the speech conversion task and convert the text information into target speech data;

[0132] A feature construction module 504 is configured to construct a target self-supervisory feature corresponding to the target speech data;

[0133] The feature processing module 506 is configured to input the target self-supervisory feature into the target average feature prediction network in the above method for processing to obtain the target average feature;

[0134] The speech generation module 508 is configured to input the target average feature into the speech conversion network for processing, and obtain speech data associated with the target speaker in the speech conversion task.

[0135] The above is a schematic scheme of a speech generation device of this embodiment. It should be noted that the technical scheme of the speech generation device and the technical scheme of the above speech generation method belong to the same concept, and the details not described in detail in the technical scheme of the speech generation device can be referred to the description of the technical scheme of the above speech generation method.

[0136] Figure 6 The block diagram of a computing device 600 according to an embodiment of the present specification is shown. The components of the computing device 600 include but are not limited to a memory 610 and a processor 620. The processor 620 is connected to the memory 610 via a bus 630, and the database 650 is used to store data.

[0137] The computing device 600 also includes an access device 640 that enables the computing device 600 to communicate via one or more networks 660. Examples of these networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 640 may include one or more of any type of network interface (e.g., a network interface card (NIC)) of wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a world-wide interoperability for microwave access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, and a near field communication (NFC).

[0138] In one embodiment of the present specification, the above components of the computing device 600 and Figure 6 Other components not shown in the figure may also be connected to each other, for example, via a bus. It should be understood that Figure 6 The computing device structure block diagram shown is only for the purpose of illustration, and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0139] The computing device 600 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, etc.), a mobile phone (e.g., a smart phone), a wearable computing device (e.g., a smart watch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or a personal computer (PC). The computing device 600 may also be a mobile or stationary server.

[0140] The processor 620 is used to execute the following computer executable instructions, which, when executed by the processor, implement the steps of the above-mentioned voice data processing method or voice generation method.

[0141] The above is a schematic scheme of a computing device of this embodiment. It should be noted that the technical scheme of the computing device and the technical scheme of the above-mentioned voice data processing method or voice generation method belong to the same concept, and the details not described in detail in the technical scheme of the computing device can be referred to the description of the technical scheme of the above-mentioned voice data processing method or voice generation method.

[0142] An embodiment of the present specification further provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the steps of the above-mentioned voice data processing method or voice generation method.

[0143] The above is a schematic scheme of a computer-readable storage medium of this embodiment. It should be noted that the technical scheme of the storage medium and the technical scheme of the above-mentioned voice data processing method or voice generation method belong to the same concept, and the details not described in detail in the technical scheme of the storage medium can be referred to the description of the technical scheme of the above-mentioned voice data processing method or voice generation method.

[0144] An embodiment of the present specification also provides a computer program product, including a computer program or instructions, which implement the steps of the above-mentioned voice data processing method or voice generation method when executed by a processor.

[0145] The above is a schematic scheme of a computer program product of this embodiment. It should be noted that the technical scheme of the computer program product and the technical scheme of the above-mentioned voice data processing method or voice generation method belong to the same concept, and the details not described in detail in the technical scheme of the computer program product can be referred to the description of the technical scheme of the above-mentioned voice data processing method or voice generation method.

[0146] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0147] The computer instructions include computer program codes, which may be in source code form, object code form, executable files or some intermediate forms, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium may be appropriately increased or decreased according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0148] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of this specification are not limited by the order of the actions described, because according to the embodiments of this specification, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the embodiments of this specification.

[0149] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0150] The preferred embodiments of this specification disclosed above are only used to help explain this specification. The optional embodiments do not describe all the details in detail, nor do they limit the invention to the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of the embodiments of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the embodiments of this specification, so that technicians in the relevant technical field can understand and use this specification well.

Claims

1. A method for processing speech data, characterized in that: include: Acquire multiple speech data with frame-level alignment relationship; Inputting the plurality of speech data into an audio processing model for processing, obtaining self-supervisory features corresponding to the plurality of speech data respectively, wherein the self-supervisory features include content information and timbre information of the speech data, performing feature average calculation on the self-supervisory features corresponding to the plurality of speech data respectively, and obtaining an average feature eliminating timbre information according to the calculation result, wherein the plurality of speech data have the same content information and different timbre information; Training an average feature prediction network using the average feature and the plurality of speech data until a target average feature prediction network that meets a training stop condition is obtained; The target average feature prediction network is used to construct a target average feature that eliminates timbre information for target speech data in a speech conversion task.

2. The method for processing speech data according to claim 1, characterized in that: The acquiring of a plurality of speech data having a frame-level alignment relationship comprises: Determining text information and multiple speaker identifiers corresponding to the network training task, and performing fixed processing on a duration prediction module in a text-to-speech model associated with the network training task; Inputting the text information and the multiple speaker identifiers into a fixed processed text-to-speech model for speech synthesis to obtain speech data corresponding to the multiple speaker identifiers respectively; The speech data corresponding to the multiple speaker identifiers have a frame-level alignment relationship.

3. The method for processing speech data according to claim 1, characterized in that: The step of training an average feature prediction network using the average feature and the plurality of speech data until a target average feature prediction network that satisfies a training stop condition is obtained includes: Selecting sample speech data from the plurality of speech data, and determining sample self-supervisory features corresponding to the sample speech data; Inputting the sample self-supervisory features into the average feature prediction network for processing to obtain predicted average features; The average feature prediction network is adjusted based on the predicted average feature and the average feature until a target average feature prediction network that meets a training stop condition is obtained.

4. The method for processing speech data according to claim 3, characterized in that: The step of adjusting the parameters of the average feature prediction network based on the predicted average feature and the average feature until a target average feature prediction network that meets a training stop condition is obtained includes: Calculating a loss value based on the predicted average feature and the average feature, and adjusting parameters of the average feature prediction network based on the loss value; Check whether the average feature prediction network after parameter adjustment meets the training stop condition; If not, selecting associated speech data from the plurality of speech data, using the associated speech data as sample speech data, and executing the step of determining a sample self-supervisory feature corresponding to the sample speech data; If so, the average feature prediction network after parameter adjustment is used as the target average feature prediction network.

5. A speech generation method, characterized in that: include: Acquire text information corresponding to the speech conversion task, and convert the text information into target speech data; Constructing a target self-supervisory feature corresponding to the target speech data; Inputting the target self-supervisory feature into the target average feature prediction network in the method according to any one of claims 1 to 4 for processing to obtain the target average feature; The target average feature is input into a speech conversion network for processing to obtain speech data associated with a target speaker in the speech conversion task.

6. A speech data processing device, characterized in that: include: An acquisition module is configured to acquire a plurality of speech data having a frame-level alignment relationship; A construction module is configured to input the plurality of speech data into an audio processing model for processing, obtain self-supervisory features corresponding to the plurality of speech data respectively, wherein the self-supervisory features include content information and timbre information of the speech data, perform feature average calculation on the self-supervisory features corresponding to the plurality of speech data respectively, and obtain an average feature eliminating timbre information according to the calculation result, wherein the plurality of speech data have the same content information and different timbre information; A training module, configured to train an average feature prediction network using the average feature and the plurality of speech data until a target average feature prediction network that meets a training stop condition is obtained; The target average feature prediction network is used to construct a target average feature that eliminates timbre information for target speech data in a speech conversion task.

7. A speech generating device, characterized in that: include: An information acquisition module is configured to acquire text information corresponding to the speech conversion task and convert the text information into target speech data; A feature construction module is configured to construct a target self-supervisory feature corresponding to the target speech data; A feature processing module, configured to input the target self-supervisory feature into the target average feature prediction network in the method according to any one of claims 1 to 4 for processing to obtain the target average feature; The speech generating module is configured to input the target average feature into the speech conversion network for processing, and obtain speech data associated with the target speaker in the speech conversion task.

8. A computing device, characterized in that include: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the method described in any one of claims 1 to 5 are implemented.

9. A computer-readable storage medium, characterized in that: It stores computer executable instructions, which, when executed by a processor, implement the steps of the method described in any one of claims 1 to 5.

10. A computer program product, characterized in that The method comprises a computer program or an instruction, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 5.

Images