Audio processing model training method, audio conversion method and device

Abstract

The present application provides a training method, an audio conversion method, an apparatus, an electronic device and a computer-readable storage medium for an audio processing model, wherein the training method comprises: determining a sample pronunciation feature of sample audio data; inputting the sample pronunciation feature into a first model to be trained for numerical encoding to obtain a numerical first output feature, wherein the first model to be trained is a model for numerical encoding of data based on past information; inputting the sample pronunciation feature into a second model for numerical encoding to obtain a numerical second output feature, wherein the second model is a model for processing data based on past and future information; adjusting the parameters of the first model to be trained based on a first parameter adjustment principle that reduces the difference between the first output feature and the second output feature to obtain an adjusted first model; and determining an audio processing model based on the adjusted first model. In this way, the streaming singing conversion can realize streaming reasoning while ensuring the effect of streaming reasoning.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a training method for an audio processing model, an audio conversion method, a device, an electronic device, and a computer-readable storage medium. Background Art

[0002] Singing voice conversion is a subset of speech conversion, which refers to converting a speaker's singing audio into another person's voice. It can be applied to online live broadcast, film and television dubbing and other fields. Streaming singing voice conversion can return recognition results in real time during the audio processing process. Compared with the whole segment conversion method, the streaming conversion model can be applied to application scenarios with real-time requirements.

[0003] Since streaming singing is real-time, it is impossible to obtain the singing information of the currently converted streaming singing at a future moment during the conversion. Therefore, the existing streaming singing conversion method is usually based on a model that does not rely on future information input for singing processing, for example, through a unidirectional recurrent neural network, causal convolution model, etc. for singing processing during singing conversion.

[0004] However, since the singing processing model in the singing conversion network cannot perceive future information, the conversion accuracy of the singing conversion network is low, resulting in poor effect of the converted singing. Summary of the invention

[0005] In view of this, the present application provides an audio processing model training method, an audio conversion method, a device, an electronic device and a computer-readable storage medium, which can improve the accuracy of singing conversion and make the effect of the converted singing better.

[0006] A first aspect of an embodiment of the present application provides a method for training an audio processing model, the method comprising:

[0007] Determine sample pronunciation features corresponding to each audio segment of the sample audio data;

[0008] Inputting the sample pronunciation feature into a first model to be trained for numerical encoding to obtain a numerical first output feature corresponding to the sample pronunciation feature, wherein the first model to be trained is a model for numerically encoding pronunciation features based on past information, wherein the past information is a past pronunciation feature corresponding to a past audio segment generated before the audio segment;

[0009] Inputting the sample pronunciation feature into a second model for numerical encoding to obtain a numerical second output feature corresponding to the sample pronunciation feature, wherein the second model is a model for numerically encoding the pronunciation feature based on the past information and future information, and the future information is a future pronunciation feature corresponding to a future audio segment generated after the audio segment;

[0010] Based on a first parameter adjustment principle of reducing the difference between the first output feature and the second output feature, adjusting the parameters of the first model to be trained to obtain an adjusted first model;

[0011] An audio processing model is determined according to the adjusted first model.

[0012] A second aspect of an embodiment of the present application provides an audio conversion method, the method comprising:

[0013] Obtain source audio to be converted and timbre information of the target speaker;

[0014] Extracting pronunciation features of the source audio, wherein the pronunciation features are features of the source audio after at least the timbre information of the source speaker is filtered out;

[0015] Inputting the pronunciation feature into a pre-trained audio processing model to obtain a first audio extraction feature corresponding to the source audio, wherein the audio processing model is trained according to the training method for an audio processing model described in any one of the first aspects;

[0016] The first audio extraction feature is fused with the timbre information to obtain converted audio corresponding to the target speaker.

[0017] A third aspect of an embodiment of the present application provides a training device for an audio processing model, comprising:

[0018] A first determining unit, used to determine sample pronunciation features corresponding to each audio segment of the sample audio data;

[0019] A first extraction unit is used to input the sample pronunciation feature into a first model to be trained for numerical encoding to obtain a first numerical output feature corresponding to the sample pronunciation feature, wherein the first model to be trained is a model that numerically encodes the pronunciation feature based on past information, and the past information is a past pronunciation feature corresponding to a past audio segment generated before the audio segment; input the sample pronunciation feature into a second model for numerical encoding to obtain a second numerical output feature corresponding to the sample pronunciation feature, wherein the second model is a model that numerically encodes the pronunciation feature based on the past information and future information, and the future information is a future pronunciation feature corresponding to a future audio segment generated after the audio segment;

[0020] An adjustment unit, configured to adjust parameters of the first model to be trained based on a first parameter adjustment principle of reducing a difference between the first output feature and the second output feature, so as to obtain an adjusted first model;

[0021] The second determining unit is used to determine the audio processing model according to the adjusted first model.

[0022] A fourth aspect of the present application provides an audio conversion device, including:

[0023] An acquisition unit, used to acquire source audio to be converted and timbre information of a target speaker;

[0024] A second extraction unit is used to extract the pronunciation feature of the source audio, where the pronunciation feature is the feature of the source audio after at least the timbre information of the source speaker is filtered out;

[0025] a processing unit, configured to input the pronunciation feature into a pre-trained audio processing model to obtain a first audio extraction feature corresponding to the source audio, wherein the audio processing model is trained according to the training method for the audio processing model described in any one of the first aspects;

[0026] A fusion unit is used to fuse the first audio extraction feature with the timbre information to obtain a converted audio corresponding to a target speaker.

[0027] A fifth aspect of an embodiment of the present application provides an electronic device, including: a memory and a processor, wherein the memory and the processor are coupled.

[0028] The memory is used to store one or more computer instructions.

[0029] The processor is used to execute one or more computer instructions to implement the method described in the first aspect or the second aspect above.

[0030] The sixth aspect of the embodiments of the present application further provides a computer-readable storage medium on which one or more computer instructions are stored, characterized in that the instructions are executed by a processor to implement the method described in the first aspect or the second aspect above.

[0031] Compared with the prior art, the embodiments of the present application have the following advantages:

[0032] The training method of the audio processing model provided in the embodiment of the present application, after determining the sample pronunciation features corresponding to each audio segment of the sample audio data, inputs the sample pronunciation features into the first model to be trained for numerical encoding to obtain the first numerical output features, and inputs the sample pronunciation features into the second model for numerical encoding to obtain the second numerical output features. Since the first model to be trained is a model that numerically encodes pronunciation features based on past information, the past information is the past pronunciation features corresponding to the past audio segments generated before the audio segment, and the second model is a model that numerically encodes pronunciation features based on past and future information, and the future information is the future pronunciation features corresponding to the future audio segments generated after the audio segment, that is, the first model to be trained cannot perceive future information when performing numerical encoding, and the second model can perceive future information when performing numerical encoding. Therefore, the second output feature is a feature obtained on the basis of perceiving past and future information, and the first output feature is a feature without perceiving future information. The embodiment of the present application adjusts the parameters of the first model to be trained based on the first parameter adjustment principle of reducing the difference between the first output feature and the second output feature. In this way, the output of the adjusted first model can be more consistent with the output result of the model that can perceive future information. That is to say, the present application distills the first model to be trained that cannot perceive future information based on the output result of the second model that can perceive future information. In this way, although the adjusted first model obtained by distillation does not use future information when performing digital encoding, the result of digital encoding of the adjusted first model is also able to perceive future information well. Therefore, the audio processing model determined according to the adjusted first model can also perceive future information well.

[0033] When performing audio conversion, the vocal features of the source audio are extracted using an audio processing model that can perceive future information well. Therefore, the converted audio obtained by fusing the extracted first audio extraction features with the timbre of the target speaker is also converted audio based on the perception of future information. Therefore, the converted audio has higher accuracy, better intelligibility and sound quality, more stable fundamental frequency, and better conversion effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments of the present application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0035] Figure 1 A flowchart of an example of a method for training an audio processing model provided in an embodiment of the present application;

[0036] Figure 2 A flowchart of another example of a method for training an audio processing model provided in an embodiment of the present application;

[0037] Figure 3 A schematic diagram of the structure of the prediction network in the embodiment of the present application;

[0038] Figure 4 A flowchart of an audio conversion method provided in an embodiment of the present application;

[0039] Figure 5 A schematic diagram of the structure of a training device for an audio processing model provided in an embodiment of the present application;

[0040] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0041] In order to enable those skilled in the art to better understand the technical solution of the present application, the present application is described clearly and completely below in conjunction with the drawings in the embodiments of the present application. However, the present application can be implemented in many other ways different from the above description. Therefore, based on the embodiments provided by the present application, all other embodiments obtained by those of ordinary skill in the art without creative work should fall within the scope of protection of the present application.

[0042] It should be noted that the terms "first", "second", "third", etc. in the claims, description and drawings of the present application are used to distinguish similar objects and are not used to describe a specific order or sequence. The data used in this way are interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including", "having" and their variants are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0043] Singing voice conversion is a subset of speech conversion, which refers to converting the singing audio of one speaker into the voice of another person. It can be applied to online live broadcast, film and television dubbing and other fields. The current mainstream singing voice conversion method is the whole segment conversion, that is, inputting the whole audio segment into the non-streaming singing voice conversion model, waiting for the non-streaming singing voice conversion model to complete the processing of the whole audio segment, and then outputting the whole converted audio segment. However, since the non-streaming singing voice model conversion method needs to wait for the whole audio segment to be processed before returning the result, it cannot be applied to application scenarios with real-time requirements.

[0044] Streaming singing voice conversion can return recognition results in real time during the audio processing process. Compared with the whole segment conversion method, the streaming conversion model can be applied to application scenarios with real-time requirements.

[0045] A common approach to singing voice conversion is to first extract the pronunciation features in the audio that are independent of the speaker, and then perform numerical encoding processing on the pronunciation features. For example, the pronunciation features are vectorized, the vectors are transformed in dimension, etc., so as to process the pronunciation features into numerical information that is convenient for singing voice conversion, thereby enriching the information content of the pronunciation features, and then fusing the processed pronunciation features with the timbre of the target speaker to obtain the singing voice converted into the timbre of the target speaker.

[0046] For streaming singing conversion, since the recognition results need to be returned in real time during the audio processing process, it is different from the conversion of the entire segment. Instead, it is necessary to convert the singing transmitted in real time. Usually, it is necessary to train a feature processing model for streaming conversion to meet the application scenarios of the real-time requirements of streaming singing conversion.

[0047] Since streaming singing is real-time, it is impossible to obtain the singing information of the currently converted streaming singing at a future moment during the conversion. Therefore, the existing streaming singing conversion method is usually based on a feature processing model that does not rely on future information input to process the pronunciation features of the singing, for example, through a unidirectional recurrent neural network, causal convolution model, etc. to perform singing processing during singing conversion.

[0048] However, since the feature processing model in the singing voice conversion network cannot perceive future information, the conversion accuracy of the singing voice conversion network is low, resulting in poor effect of the converted singing.

[0049] In order to improve the accuracy of singing conversion and make the converted singing sound better, the first embodiment of the present application provides a training method for an audio processing model.

[0050] The executor of the training method of the audio processing model provided in the present application may be an electronic device, which may be a server, a desktop computer, a laptop computer, a tablet computer, a mobile phone, a smart watch, a smart TV, etc., or other electronic devices with data processing functions, which are not specifically limited in the present application.

[0051] The model trained by the training method of the audio processing model provided in the present application can be used to process singing audio during singing conversion, and can also be used to process audio during other pronunciation conversions, for example, processing recitation audio to convert the timbre of the speaker of the recitation audio. In addition to the audio conversion scenario, the model trained by the training method of the audio processing model provided in the present application can also be used for audio processing in other scenarios, for example, for processing speech during speech recognition.

[0052] Figure 1 A flowchart of a method for training an audio processing model provided in an embodiment of the present application. It should be noted that the steps shown in the flowchart can be executed in a computer system such as a set of computer executable instructions, and in some cases, the steps can be executed in a logical order different from that shown in the flowchart.

[0053] The training method of the audio processing model provided in the present application includes the following steps S110 to S150.

[0054] Step S110: Determine sample pronunciation features corresponding to each audio segment of the sample audio data.

[0055] The sample audio data can be sample singing audio data, sample recitation audio data, etc., or other audio data containing the speaker's voice information. The sample audio data can be pre-recorded for training an audio processing model, or it can be various audios with the speaker's voice searched on the Internet. This application does not limit the specific content of the sample audio data.

[0056] In an embodiment of the present application, the sample audio data can be divided into multiple audio segments in the order of generation time from front to back. Specifically, it can be divided into multiple audio segments based on the audio duration, or divided into multiple audio segments based on the number of pronunciations. For example, a 20-second audio can be divided into 4 audio segments with every 5 seconds as an audio segment; for another example, an audio with a total of 100 pronunciations can be divided into 5 audio segments with every 20 pronunciations as an audio segment. The sample audio data can also be segmented in the order from front to back in other ways. In step S110, the sample pronunciation features corresponding to each audio segment of the sample audio data are determined.

[0057] The sample pronunciation feature can be understood as the feature of the sample audio data after at least the speaker's timbre information is filtered out. That is to say, the sample pronunciation feature is information that is independent of the speaker's timbre. The sample pronunciation feature is used to represent the features of each pronunciation in the sample audio data. Based on the pronunciation feature, the specific pronunciation in the sample audio data that is independent of the speaker's timbre can be determined.

[0058] The pronunciation features may be expressed in the form of text or in the form of phonemes, etc. The pronunciation features include the pronunciation features of each moment or each frame in the sample audio data.

[0059] Since audio such as singing contains not only semantics but also vocal information such as tone and rhythm, when training the audio processing model, the present application trains the model based on pronunciation features, thereby improving the scope of use of the trained audio processing model. In order to make the trained audio processing model not only suitable for scenarios such as conventional recitation where the pronunciation tone is consistent with the tone of the text itself, but also suitable for scenarios where the pronunciation tone of singing may be inconsistent with the tone of the text itself.

[0060] In an embodiment of the present application, the sample pronunciation features of the sample audio data can be determined by a pre-trained speech recognition model. Specifically, the sample audio data can be input into the pre-trained speech recognition model to obtain the sample pronunciation features corresponding to the sample audio data. The speech recognition model can be trained based on a conventional supervised training method, for example, the speech recognition model is trained based on an audio sample that has been set with corresponding audio features. The specific training method is not described in detail in this application.

[0061] Optionally, in step S110, the pronunciation features may be determined according to the following steps S111 to S112.

[0062] Step S111: converting each audio segment of the sample audio data into sample Melp spectrum data.

[0063] Step S112: Input the Melp spectrum data into a pre-trained speech recognition model to obtain bottleneck features corresponding to each audio segment of the sample audio data, and determine the bottleneck features as sample pronunciation features corresponding to each audio segment of the sample audio data. The speech recognition model is used to identify the pronunciation features of the audio data.

[0064] Step S120: Input the sample pronunciation features into the first model to be trained for numerical encoding to obtain a first numerical output feature corresponding to the sample pronunciation features. The first model to be trained is a model that numerically encodes pronunciation features based on past information.

[0065] The above-mentioned past information is a past pronunciation feature corresponding to a past audio segment generated before the audio segment corresponding to the sample pronunciation feature.

[0066] In this step, each sample pronunciation feature corresponding to each audio segment of the sample audio data can be input into the first model to be trained for numerical encoding. When the first model to be trained numerically encodes the target sample pronunciation feature among each sample pronunciation feature, it will numerically encode the target sample pronunciation feature based on the sample pronunciation feature before the target sample pronunciation feature. The sample pronunciation feature before the target sample pronunciation feature is the past pronunciation feature corresponding to the past audio segment generated before the audio segment corresponding to the target sample pronunciation feature.

[0067] For example, the sample audio data includes four audio segments a, b, c, and d, and these four audio segments respectively correspond to their own sample pronunciation features. The four sample pronunciation features corresponding to the four audio segments a, b, c, and d can be input into the first model to be trained for numerical encoding. When the sample pronunciation features corresponding to the audio segment a are numerically encoded, since there is no audio segment before a, the past information corresponding to a can be set to empty by default. At this time, the pronunciation features corresponding to a can be directly numerically encoded; when the sample pronunciation features corresponding to the audio segment b are numerically encoded, since the audio segment before b is a, the pronunciation features corresponding to b can be numerically encoded based on the pronunciation features corresponding to the audio segment a; when the sample pronunciation features corresponding to the audio segment c are numerically encoded, since the audio segments before c are a and b, the pronunciation features corresponding to c can be numerically encoded based on the pronunciation features corresponding to the audio segments a and b.

[0068] Specifically, the first model to be trained can be a unidirectional recurrent neural network model, a causal convolutional neural network model, etc., or other models that digitally encode data based on past information. The first model to be trained digitally processes data based on past information, which can be understood as the first model to be trained digitally encodes data based on past information and not based on future information.

[0069] Step S130: Input the above sample pronunciation features into the second model for numerical encoding to obtain a numerical second output feature corresponding to the sample pronunciation features. The second model is a model that numerically encodes the pronunciation features based on the above past information and future information.

[0070] The above-mentioned future information is a future pronunciation feature corresponding to a future audio segment generated after the audio segment corresponding to the sample pronunciation feature.

[0071] In this step, each sample pronunciation feature corresponding to each audio segment of the sample audio data can be input into the second model for numerical encoding. When the second model numerically encodes the target sample pronunciation feature among the sample pronunciation features, it numerically encodes the target sample pronunciation feature based on the sample pronunciation feature after the target sample pronunciation feature. The sample pronunciation feature after the target sample pronunciation feature is the future pronunciation feature corresponding to the future audio segment generated after the audio segment corresponding to the target sample pronunciation feature.

[0072] Taking the above sample audio data including four audio segments a, b, c, and d as an example, the four sample pronunciation features corresponding to the four audio segments a, b, c, and d can be input into the second model for numerical encoding. When the second model numerically encodes the sample pronunciation features corresponding to the audio segment a, since there is no audio segment before a, and the audio segments after a are b, c, and d, the pronunciation features corresponding to a can be numerically encoded based on the three future pronunciation features corresponding to the three future audio segments b, c, and d; when the second model numerically encodes the sample pronunciation features corresponding to the audio segment b, since the audio segment before b is a, and the audio segments after b are c and d, the pronunciation features corresponding to b can be numerically encoded based on the past audio features corresponding to the past audio segment a, and the two future pronunciation features corresponding to the two future audio segments c and d.

[0073] The second model can be a trained model or an untrained model. If the second model is an untrained model, the second model can be trained synchronously during the audio processing model training process in the present application.

[0074] The second model can be a bidirectional recurrent neural network model, a non-causal convolutional neural network model, etc., or it can be other models that process data based on past information and future information. That is to say, the second model can perceive future information when processing data.

[0075] In the embodiment of the present application, the first model to be processed and the second model can be at least one of the numerical coding processes such as vector coding processing, matrix coding processing, dimensionality increase processing, etc. for the sample pronunciation features, or other numerical coding processing for the sample pronunciation features. The specific processing content can be determined according to actual needs, and this application does not specifically limit it.

[0076] It can be understood that the first model to be processed and the second model perform the same dimensional processing on the pronunciation features. For example, the first model to be processed is used to perform vectorized encoding on the pronunciation features, and then the second model also performs vectorized encoding on the pronunciation features.

[0077] Step S140: Based on the first parameter adjustment principle of reducing the difference between the first output feature and the second output feature, the parameters of the first model to be trained are adjusted to obtain an adjusted first model.

[0078] Specifically, the audio processing model can be trained through a large amount of sample audio data, and whether the first training model has completed training is determined based on whether the difference between the first output feature and the second output feature meets the convergence condition, thereby obtaining the adjusted first model. The convergence condition can be, for example, that the difference between the first output feature and the second output feature is less than a first preset threshold, or that, for a plurality of sample audio data, the ratio of the difference between the first output feature and the second output feature less than the first preset threshold reaches a first preset ratio threshold, and the first preset ratio threshold can be, for example, 80% to 95%, or other relatively large ratios.

[0079] Step S150: determining an audio processing model according to the adjusted first model.

[0080] Specifically, the adjusted first model can be directly determined as the audio processing model, or the adjusted first model can be further optimized to obtain the audio processing model. The specific processing method can be determined according to the specific application scenario of the audio processing model.

[0081] For example, when the audio processing model provided in the present application is used for an audio timbre conversion scenario, if the second model is a pre-trained model that meets the timbre conversion scenario, the adjusted first model can be directly determined as the audio processing model.

[0082] In order to make the converted audio obtained after the timbre conversion more accurate, the audio processing model can also be determined by the following specific implementation. In this specific implementation, before step S150, the following steps S140a to S140b can also be included, and step S150 can be implemented by the following step S151.

[0083] Step S140a: Determine the sample timbre characteristics of the sample audio data.

[0084] Specifically, the sample timbre features of the sample audio data can be identified by a pre-trained timbre recognition model; the sample timbre features of the sample audio data can also be determined by manual marking, that is, the timbre of the sample audio data is manually identified and then the timbre is identified; or, a timbre library can be pre-set, the timbre library includes various timbre features, and the timbre consistent with the timbre of the sample audio data can be determined from the timbre library by a pre-trained timbre determination model. The pre-trained timbre recognition model and timbre determination model can be obtained by supervised training or unsupervised training, and will not be described in detail in this application.

[0085] Alternatively, if Figure 2 As shown, the speaker identification corresponding to the sample audio data may also be obtained, and the timbre feature corresponding to the speaker identification may be determined from a pre-stored timbre library through the speaker embedding layer as the sample timbre feature.

[0086] Step S140b: Fusing the sample timbre feature with the first output feature to obtain fused audio data.

[0087] Specifically, the sample timbre feature and the first output feature can be fused through a fusion model; alternatively, the harmonic distribution of the sample timbre can be obtained, and the first output feature can be made to have a harmonic distribution consistent with the harmonic distribution through a filter to obtain fused audio data; fusion can also be performed in other ways, which are not specifically limited in this application.

[0088] Step S151: Based on the second parameter adjustment principle of reducing the difference between the fused audio data and the sample audio data, the adjusted first model is parameter adjusted to obtain an audio processing model.

[0089] Specifically, when the difference between the fused audio data and the sample audio data satisfies the second convergence condition, it can be determined that the adjustment of the parameters of the adjusted first model is completed, thereby obtaining the audio processing model. The second convergence condition can be, for example, that the difference between the fused audio data and the corresponding sample audio data is less than a second preset threshold, or, for multiple sample audio data, the ratio of the difference between the fused audio data and the sample audio data that is less than the second preset threshold reaches a second preset ratio threshold. The second preset ratio threshold can be the same as the first preset ratio threshold, or it can be different, and this application does not specifically limit it.

[0090] This embodiment adjusts the parameters of the adjusted first model by determining whether the fused audio data after the fusion of the first output feature and the timbre feature is accurate, so that the feature processing capability of the obtained audio processing model can better meet the scenario of timbre and pronunciation fusion. In this way, for the scenario of timbre and pronunciation fusion, the timbre fusion accuracy of the features processed by the audio processing model is higher.

[0091] The training method of the audio processing model provided in the embodiment of the present application, after determining the sample pronunciation features corresponding to each audio segment of the sample audio data, inputs the sample pronunciation features into the first model to be trained for numerical encoding to obtain the first numerical output features, and inputs the sample pronunciation features into the second model for numerical encoding to obtain the second numerical output features. Since the first model to be trained is a model that numerically encodes pronunciation features based on past information, the past information is the past pronunciation features corresponding to the past audio segments generated before the audio segment, and the second model is a model that numerically encodes pronunciation features based on past and future information, and the future information is the future pronunciation features corresponding to the future audio segments generated after the audio segment, that is, the first model to be trained cannot perceive future information when performing numerical encoding, and the second model can perceive future information when performing numerical encoding. Therefore, the second output feature is a feature obtained on the basis of perceiving past and future information, and the first output feature is a feature without perceiving future information. The embodiment of the present application adjusts the parameters of the first model to be trained based on the first parameter adjustment principle of reducing the difference between the first output feature and the second output feature. In this way, the output of the adjusted first model can be more consistent with the output result of the model that can perceive future information. That is to say, the present application distills the first model to be trained that cannot perceive future information based on the output result of the second model that can perceive future information. In this way, although the adjusted first model obtained by distillation does not use future information when performing digital encoding, the result of digital encoding of the adjusted first model is also able to perceive future information well. Therefore, the audio processing model determined according to the adjusted first model can also perceive future information well.

[0092] When performing audio conversion, the vocal features of the source audio are extracted using an audio processing model that can perceive future information well. Therefore, the converted audio obtained by fusing the extracted first audio extraction features with the timbre of the target speaker is also converted audio based on the perception of future information. Therefore, the converted audio has higher accuracy, better intelligibility and sound quality, more stable fundamental frequency, and better conversion effect.

[0093] In a specific embodiment, the method may further include the following step S160.

[0094] Step S160: adjusting the parameters of the second model based on the second parameter adjustment principle to obtain an adjusted second model.

[0095] Step S160 is to adjust the parameters of the second model based on the second parameter adjustment principle of reducing the difference between the fusion audio data and the sample audio data to obtain the adjusted second model. The specific parameter adjustment method of the second model can refer to the adjustment method of the adjusted first model above, that is, the conventional supervised model training method, which will not be described in detail here.

[0096] This embodiment also updates the parameters of the second model based on the fused audio data, so that the features extracted by the second model are more in line with the scenario where timbre and pronunciation are integrated, so that the second model can more accurately extract features that meet the fusion scenario when the audio processing model is subsequently trained, so that the audio processing model finally trained can more accurately extract features that meet the fusion scenario. In addition, this embodiment updates and trains the second model, and the second model can be used directly to perform feature processing on non-streaming audio for the fusion of timbre and pronunciation, so that the training process of this application can simultaneously train models for the processing of both streaming and non-streaming audio, thereby improving the efficiency of model training.

[0097] The first model to be trained and the second model are Figure 2 The causal convolutional encoder and the non-causal convolutional encoder in the first model to be trained and the second model form Figure 2 The dual-mode encoder in.

[0098] In a specific embodiment, in step S140b, the fused audio data can be obtained specifically according to the following step A, and the training method of the above audio processing model can also include the following step S170.

[0099] Step A: Input the sample timbre feature and the first output feature into a first fusion model to obtain first fused audio data.

[0100] Step S170: Based on the principle of reducing the difference between the first fused audio data and the sample audio data, the parameters of the first fusion model are adjusted to obtain an adjusted first fusion model.

[0101] The first fusion model may be a model for fusing data based on past first fusion information, wherein the past first fusion information includes the first output feature corresponding to the past pronunciation feature and the timbre feature corresponding to the past pronunciation feature.

[0102] Exemplarily, the sample audio data also includes four audio segments a, b, c, and d for illustration. When the first fusion model fuses the sample timbre feature and the first output feature corresponding to a, since there is no audio segment before a, the fusion can be performed directly; when the first fusion model fuses the sample timbre feature and the first output feature corresponding to b, the audio segment before b includes a, so the first fusion model will fuse the sample timbre feature and the first output feature corresponding to b based on the first output feature corresponding to a; when the first fusion model fuses the sample timbre feature and the first output feature corresponding to c, the audio segment before c includes a and b, so the first fusion model will fuse the sample timbre feature and the first output feature corresponding to c based on the first output features corresponding to a and b respectively.

[0103] The specific adjustment process in step S170 can refer to the process of adjusting parameters of the second model and the first fusion model mentioned above, which will not be described in detail here.

[0104] The first fusion model may be a neural network model, a decision tree model, a support vector machine model, etc., or other machine learning models.

[0105] Specifically, the first fusion model may be a model that processes data based on past information. For example, the first fusion model may be a causal convolution model, a unidirectional recurrent neural network model, etc. Thus, since the first output feature is a feature obtained based on past information, when the first fusion model is a model that processes data based on past information, the first fusion model is more suitable for fusing the first output feature, so that the audio fusion is more accurate. In addition, when the first fusion model is a model that processes data based on past information, the adjusted first fusion model obtained after parameter adjustment can subsequently perform more accurate fusion processing on streaming audio. This embodiment updates and trains the first fusion model, which can be directly used for audio fusion later, thereby improving the efficiency of model training.

[0106] The first fusion model can also be a model that processes data based on past and future information. For example, the first fusion model can be a non-causal convolution model, a bidirectional recurrent neural network model, etc., which can also achieve the fusion of timbre and pronunciation. This application does not specifically limit this.

[0107] The first fused audio data obtained in step A is the fused audio data obtained in step S140b, that is, the fused audio data obtained in step S140b includes the first fused audio data.

[0108] Optionally, in step S140b, the fused audio data can be obtained specifically according to the following step B, and the training method of the above audio processing model can also include the following step S180.

[0109] Step B: Input the sample timbre feature and the second output feature into the second fusion model to obtain second fused audio data.

[0110] The second fusion model is a model that processes data based on past and future information. The second fusion model can be a bidirectional neural network model, a non-causal convolution model, or other models that process data based on past and future information.

[0111] The second fusion model is a model for fusing data based on past second fusion information and future second fusion information, wherein the past second fusion information includes second output features corresponding to past pronunciation features, and the future second fusion information includes second output features corresponding to future pronunciation features.

[0112] Exemplarily, the sample audio data also includes four audio segments a, b, c, and d. When the second fusion model fuses the sample timbre feature and the second output feature corresponding to a, since there is no audio segment before a and the audio segment after a includes three audio segments b, c, and d, the second fusion model will fuse the sample timbre feature and the second output feature corresponding to a based on the second output features corresponding to the three future pronunciation features of the three future audio segments b, c, and d; when the second fusion model fuses the sample timbre feature and the second output feature corresponding to b, since the audio segment before b is a and the audio segment after b includes three audio segments c and d, the second fusion model will fuse the sample timbre feature and the second output feature corresponding to b based on the second output features corresponding to the past pronunciation features of the past audio segment a and the second output features corresponding to the two future pronunciation features of the two future audio segments c and d.

[0113] The second fused audio data obtained in step B is the fused audio data obtained in step S140b, that is, the fused audio data obtained in step S140b also includes the second fused audio data.

[0114] Step S180: Based on the principle of reducing the difference between the second fused audio data and the sample audio data, the parameters of the second fusion model are adjusted to obtain an adjusted second fusion model.

[0115] The parameter adjustment process of step S180 can be the same as step S170, which will not be described in detail here.

[0116] In this embodiment, the adjusted second fusion model can be fused with the timbre of the target speaker when subsequently applied to non-streaming audio conversion. Multiple models can be obtained through one training, thereby improving the efficiency of model training.

[0117] like Figure 2As shown, the first fusion model and the second fusion model are Figure 2 The first fused audio data and the second fused audio data are Figure 2 Streaming audio, non-streaming audio in .

[0118] The adjusted second fusion model can be fused with the timbre of the target speaker when it is subsequently applied to the conversion of non-streaming audio.

[0119] In one implementation, before step S150, the method may further include the following step a. Step S150 may be implemented according to the following step S152.

[0120] Step a: Input the first output feature into the prediction network to obtain sample prediction data corresponding to the sample audio data.

[0121] The prediction network is used to determine the corresponding audio data (and the sample prediction data in step a) according to the first output feature. The prediction network may include one prediction model or multiple prediction models. The prediction model included in the prediction network may be a pre-trained model or an untrained model. When the prediction model is an untrained model, the prediction model may be subsequently trained to update parameters.

[0122] Step S152: Based on the principle of reducing the difference between the sample prediction data and the sample audio data, the parameters of the adjusted first model are adjusted to obtain an audio processing model.

[0123] Specifically, when the difference between the sample prediction data and the sample audio data satisfies the third convergence condition, it can be determined that the adjustment of the parameters of the adjusted first model is completed, thereby obtaining the audio processing model. The third convergence condition can be, for example, that the difference between the sample prediction data and the corresponding sample audio data is less than a third preset threshold, or, for multiple sample audio data, the ratio of the difference between the sample prediction data and the sample audio data that is less than the third preset threshold reaches a third preset ratio threshold, and the third preset ratio threshold can be the same as the first preset ratio threshold, or it can be different, and this application does not specifically limit it.

[0124] This implementation predicts the first output feature through a prediction network, and further adjusts the parameters of the adjusted first model according to the accuracy of the prediction result, so as to improve the accuracy of the obtained audio processing model.

[0125] Optionally, the method may further include the following steps: inputting the second output feature into the prediction network to obtain second prediction data corresponding to the sample audio data; and adjusting the parameters of the second model based on the principle of reducing the difference between the second sample prediction data and the sample audio data to obtain an adjusted second model. The feature extraction of the adjusted second model is more accurate, and the accuracy of subsequent non-streaming audio conversion is higher.

[0126] Optionally, the prediction network may include a first prediction model and a second prediction model, wherein the first prediction model is used to predict data of a next frame based on data of a current frame, and the second prediction model is used to select data of a next frame from multiple frame data.

[0127] Specifically, Figure 2 As shown, the above-mentioned prediction network can be a hybrid predictive coding (Hybrid Predictive Coding, HPC), the above-mentioned first prediction model can be an autoregressive prediction coding (Autoregressive Predictive Coding, APC), and the above-mentioned second prediction model can be a contrastive prediction coding (Contrastive Predictive Coding, CPC), and the contrastive prediction coding and the autoregressive prediction coding respectively use a network with an autoregressive structure (g-net).

[0128] CPC learns representations by classifying real future frames and randomly selected negative samples. Contrastive predictive encoders use prior knowledge to select negative samples, which can better learn representations for specific tasks. Contrastive predictive encoders are a contrastive learning method. The input audio is passed through the encoder to obtain hidden representations, and then through the autoregressive network to obtain sequence features. We select hidden representations that are closer to the target point as positive samples, and those that are farther away as negative samples.

[0129] APC is a generative model that directly predicts future frames. Given an input sequence, it predicts the spectrum of the future period. APC's direct prediction approach is not affected by the negative sample selection method and has better generalization than CPC. Combining the two methods of contrast prediction encoder and autoregressive prediction encoder can make the output of the hybrid predictive encoder have better generalization.

[0130] Step a can be implemented by following steps a-1 and a-2. Step S152 can be implemented by following steps S152a.

[0131] Step a-1: Input the above-mentioned first output feature into the first prediction model to obtain first sample prediction data corresponding to the sample audio data.

[0132] Step a-2: Input the above-mentioned first output feature into the second prediction model to obtain second sample prediction data corresponding to the sample audio data.

[0133] Step S152a: Based on the principle of reducing the difference between the first sample prediction data, the second sample prediction data and the sample audio data, adjusting the parameters of the adjusted first model.

[0134] For example, taking the first output feature as an example, Figure 3 As shown, the Z sequence is the first output feature of the input, and the first output feature is input to the autoregressive prediction encoder of the hybrid predictive encoder to obtain the first prediction sample data, wherein the first sample prediction data is sequence data, that is, the first r sequence, that is, Figure 3 In Sequence, z t represents the value of Z sequence at time t, The first output feature is input to the comparative prediction encoder of the hybrid predictive encoder to obtain the second sample prediction data, that is, the second r sequence, that is, Figure 3 r sequence in .

[0135] This embodiment uses the first prediction model and the second prediction model for prediction respectively, so as to further update the parameters of the adjusted first model according to the two prediction results, which can not only learn representations for specific tasks but also make the output results more generalizable, so that the audio processing model has better generalization and feature processing is more accurate.

[0136] The second output feature can also be input into the first prediction model and the second prediction model respectively to obtain prediction data. The specific process refers to the first output feature and will not be described in detail here.

[0137] In one embodiment, the above step A can be implemented according to the following steps A-1 and A-2.

[0138] Step A-1: ​​Obtaining pitch features of sample audio data;

[0139] Step A-2: Input the sample timbre feature, the first output feature and the pitch feature into a first fusion model to obtain first fused audio data.

[0140] The pitch feature can be extracted by a pre-trained pitch extraction model, which can be obtained by training a related machine learning algorithm, which is not described in detail in this application. Pitch is used to indicate the pitch height of the speaker. The pitch of the speaker at different times is usually different. By fusing the pitch, the fused audio can be obtained more accurately, so that the final audio processing model has higher feature processing accuracy when applied to audio fusion scenarios.

[0141] The second embodiment of the present application also provides an audio conversion method. The executor of the audio conversion method provided by the present application can be an electronic device, which can be a server, a desktop computer, a laptop computer, a tablet computer, a mobile phone, a smart watch, a smart TV, etc. It can also be other electronic devices with data processing functions, which are not specifically limited in the present application.

[0142] The audio conversion method provided in this application can be used to convert the audio of singing, speech, etc. in live broadcasting scenes, dubbing scenes, etc., and the specific application scenes are not limited. Specifically, it can be used to convert the streaming audio transmitted in real time during singing conversion, and it can also be used to convert non-streaming audio.

[0143] like Figure 4 As shown, the audio conversion method provided by the present application includes the following steps S210 to S240.

[0144] Step S210: Acquire the source audio to be converted and the timbre characteristics of the target speaker.

[0145] The source audio may be singing, speech audio, dialogue audio of a film or TV series or other video, or other forms of audio.

[0146] In step S210, the timbre characteristics of the source audio to be converted and the target speaker can be determined based on the audio conversion request sent by the received client, wherein the audio conversion request can include the audio information of the source audio to be converted and the timbre information of the target speaker. The audio information of the source audio can be a specific audio content or an identifier corresponding to the audio. The electronic device can search the corresponding audio from the audio library as the source audio through the identifier corresponding to the audio. The timbre information can be a specific timbre feature or a timbre identifier corresponding to the timbre. The electronic device can also determine the corresponding timbre feature through the timbre identifier.

[0147] Step S220: extracting pronunciation features of the source audio.

[0148] The pronunciation feature is a feature of the source audio after at least the timbre information of the source speaker is filtered out.

[0149] The process of extracting pronunciation features in this step can refer to the method of extracting sample pronunciation features in step S110 of the first embodiment, and will not be described in detail here.

[0150] Step S230: input the pronunciation feature into a pre-trained audio processing model to obtain a first audio extraction feature corresponding to the source audio.

[0151] The audio processing model is trained according to the audio processing model training method described in any one of the first embodiments.

[0152] Step S240: Fusing the first audio extraction feature with the timbre information to obtain converted audio corresponding to the target speaker.

[0153] The way of performing fusion in step S240 can refer to the specific execution process of step S140b in the first embodiment, which will not be described in detail here.

[0154] It can be seen from the first embodiment that since the audio processing model that can perceive future information well is used to extract the occurrence characteristics of the source audio, the converted audio obtained by fusing the extracted first audio extraction features with the timbre of the target speaker is also a converted audio based on the perception of future information. Therefore, the converted audio has higher accuracy, better intelligibility and sound quality, more stable fundamental frequency, and better conversion effect.

[0155] Optionally, the step S240 can specifically obtain the converted audio corresponding to the target speaker according to the following steps: input the first audio extraction feature and the timbre information into a pre-trained first fusion model to obtain the converted audio corresponding to the target speaker, and the first fusion model is trained in the same manner as the adjusted first fusion model in the training method of the audio processing model described in the first embodiment. It can be seen from the first embodiment that since the first fusion model is a model that processes data based on past information, it is more suitable for fusing streaming audio data and has higher fusion accuracy.

[0156] Optionally, in step S230, the pronunciation features are input into a pre-trained audio processing model, which can be specifically implemented by following the steps S231.

[0157] Step S231: When the source audio is a real-time audio stream, the pronunciation features are input into a pre-trained audio processing model.

[0158] The audio conversion method may further include the following steps S250 to S260.

[0159] Step S250: When the source audio is a non-real-time audio stream, the pronunciation feature is input into the second model to obtain a second audio extraction feature corresponding to the source audio, and the second model is trained according to the training method of the audio processing model described in the first embodiment by training the adjusted second model.

[0160] Step S260: Fusing the second audio extraction feature with the timbre information to obtain converted audio corresponding to the target speaker.

[0161] As can be seen from the first embodiment, the audio processing model is obtained by training the causal convolution model and the unidirectional neural network model, which are models that process data based on past information. Therefore, the audio processing model is more suitable for processing streaming audio, and the processing result is more accurate, so that the fusion result is also more accurate. For non-real-time audio streams, that is, non-streaming audio, this embodiment performs feature processing through the second model. Since the second model is trained by the method of training and adjusting the second model in the first embodiment, it can be seen from the first embodiment that the second model is a model that processes data based on past and future information. Therefore, the second model processes non-streaming audio more accurately, and the fusion result is also more accurate.

[0162] Optionally, the step S260 may be implemented according to the following step S261.

[0163] Step S261: Input the second audio extraction feature and the timbre information into a pre-trained second fusion model to obtain the converted audio corresponding to the target speaker, and the second fusion model is trained according to the method of training the adjusted second fusion model in the training method of the audio processing model described in the first embodiment. It can be seen from the first embodiment that since the second fusion model is a model for data fusion based on past and future information, it is more suitable for fusing non-streaming data and has higher fusion accuracy.

[0164] The third embodiment of the present application also provides a training device for an audio processing model, such as Figure 5 As shown, the device comprises:

[0165] A first determining unit 310 is used to determine sample pronunciation features corresponding to each audio segment of the sample audio data;

[0166] The first extraction unit 320 is used to input the sample pronunciation feature into a first model to be trained for numerical encoding to obtain a first numerical output feature corresponding to the sample pronunciation feature, wherein the first model to be trained is a model that numerically encodes the pronunciation feature based on past information, and the past information is a past pronunciation feature corresponding to a past audio segment generated before the audio segment corresponding to the sample pronunciation feature; input the sample pronunciation feature into a second model for numerical encoding to obtain a second numerical output feature corresponding to the sample pronunciation feature, wherein the second model is a model that numerically encodes the pronunciation feature based on the past information and future information, and the future information is a future pronunciation feature corresponding to a future audio segment generated after the audio segment corresponding to the sample pronunciation feature;

[0167] An adjusting unit 330 is configured to adjust parameters of the first model to be trained based on a first parameter adjustment principle of reducing a difference between the first output feature and the second output feature to obtain an adjusted first model;

[0168] The second determining unit 340 is configured to determine an audio processing model according to the adjusted first model.

[0169] Optionally, the first model to be trained is a causal convolutional model, and the second model is a non-causal convolutional model.

[0170] Optionally, the first determination unit 310 is further configured to: determine a sample timbre feature of the sample audio data;

[0171] The device also includes:

[0172] A first fusion unit, used for fusing the sample timbre feature with the first output feature to obtain fused audio data;

[0173] The second determination unit 340 is specifically configured to: based on a second parameter adjustment principle of reducing the difference between the fused audio data and the sample audio data, perform parameter adjustment on the adjusted first model to obtain an audio processing model.

[0174] Optionally, the adjusting unit 330 is further used to: perform parameter adjustment on the second model based on the second parameter adjustment principle to obtain an adjusted second model.

[0175] Optionally, the first fusion unit is specifically used to: input the sample timbre feature and the first output feature into a first fusion model to obtain first fused audio data;

[0176] The adjustment unit is further used to: based on the principle of reducing the difference between the first fused audio data and the sample audio data, adjust the parameters of the first fusion model to obtain an adjusted first fusion model.

[0177] Optionally, the first fusion model is a model for fusing data based on past first fusion information, and the past first fusion information includes a first output feature corresponding to the past pronunciation feature.

[0178] Optionally, the first fusion unit is further used to: input the sample timbre feature and the second output feature into a second fusion model to obtain second fused audio data, wherein the second fusion model is a model for fusing data based on past second fusion information and future second fusion information, wherein the past second fusion information includes the second output feature corresponding to the past pronunciation feature, and the future second fusion information includes the second output feature corresponding to the future pronunciation feature;

[0179] The adjustment unit is further used to: based on the principle of reducing the difference between the second fused audio data and the sample audio data, adjust the parameters of the second fusion model to obtain an adjusted second fusion model.

[0180] Optionally, the device further comprises: a prediction unit, configured to input the first output feature into a prediction network to obtain sample prediction data corresponding to the sample audio data;

[0181] The second determination unit is specifically used to: based on the principle of reducing the difference between the sample prediction data and the sample audio data, adjust the parameters of the adjusted first model to obtain an audio processing model.

[0182] Optionally, the prediction network includes a first prediction model and a second prediction model, the first prediction model is used to predict the data of the next frame according to the data of the current frame, and the second prediction model is used to select the data of the next frame from the multiple frame data;

[0183] The prediction unit is specifically configured to: input the first output feature into the first prediction model to obtain first sample prediction data corresponding to the sample audio data; input the first output feature into the second prediction model to obtain second sample prediction data corresponding to the sample audio data;

[0184] The second determination unit is specifically configured to: adjust parameters of the adjusted first model based on the principle of reducing the difference between the first sample prediction data, the second sample prediction data and the sample audio data.

[0185] Optionally, the first fusion unit is specifically used to: obtain pitch features of sample audio data; input the sample timbre features, the first output features and the pitch features into a first fusion model to obtain first fused audio data.

[0186] Optionally, the first determining unit is specifically configured to: convert each audio segment of the sample audio data into sample Melp spectrum data;

[0187] The Melp spectrum data is input into a pre-trained speech recognition model to obtain bottleneck features corresponding to each audio segment of the sample audio data, and the bottleneck features are determined as sample pronunciation features corresponding to each audio segment of the sample audio data. The speech recognition model is used to identify the pronunciation features of the audio data.

[0188] A fourth embodiment of the present application further provides an audio conversion device, the device comprising:

[0189] An acquisition unit, used to acquire source audio to be converted and timbre information of a target speaker;

[0190] A second extraction unit is used to extract the pronunciation feature of the source audio, where the pronunciation feature is the feature of the source audio after at least the timbre information of the source speaker is filtered out;

[0191] A processing unit, configured to input the pronunciation feature into a pre-trained audio processing model to obtain a first audio extraction feature corresponding to the source audio, wherein the audio processing model is trained according to the training method for the audio processing model described in any one of the first embodiments;

[0192] A fusion unit is used to fuse the first audio extraction feature with the timbre information to obtain a converted audio corresponding to a target speaker.

[0193] Optionally, the fusion unit is specifically used for:

[0194] The first audio extraction feature and the timbre feature are input into a pre-trained first fusion model to obtain the converted audio corresponding to the target speaker. The first fusion model is trained by training the adjusted first fusion model in the training method of the audio processing model described in the first embodiment.

[0195] Optionally, the processing unit is specifically used to: when the source audio is a real-time audio stream, input the pronunciation feature into a pre-trained audio processing model;

[0196] The processing unit is also used for: when the source audio is a non-real-time audio stream, inputting the pronunciation features into a second model to obtain a second audio extraction feature corresponding to the source audio, the second model being trained according to the training method of the audio processing model described in the first implementation and the adjusted second model; fusing the second audio extraction feature with the timbre information to obtain the converted audio corresponding to the target speaker.

[0197] Optionally, the fusion unit is specifically used to: input the second audio extraction features and the timbre information into a pre-trained second fusion model to obtain the converted audio corresponding to the target speaker, and the second fusion model is trained by training the adjusted second fusion model in the training method of the audio processing model described in the first embodiment.

[0198] It should be noted that the information interaction, execution process, etc. between the modules / units in the device are the same as those in this application. Figures 1 to 3 The corresponding method embodiments are based on the same concept. For specific contents, please refer to the description in the method embodiments shown above in this application, which will not be repeated here.

[0199] Next, an electronic device provided by an embodiment of the present application is introduced. Figure 6, Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device 500 can be specifically a virtual reality VR device, a mobile phone, a tablet, a laptop computer, a smart wearable device, a monitoring data processing device, or a radar data processing device, etc., which is not limited here. Among them, the electronic device 500 can be deployed with Figure 1 or Figure 4 The training device or conversion device described in the corresponding embodiment is used to implement Figures 1 to 4 Specifically, the electronic device 500 includes: a receiver 501, a transmitter 502, a processor 503 and a memory 504 (wherein the number of the processor 503 in the execution device 500 can be one or more, Figure 6 In the example of FIG. 5 , a processor 503 is used, where the processor 503 may include an application processor 5031 and a communication processor 5032. In some embodiments of the present application, the receiver 501, the transmitter 502, the processor 503 and the memory 504 may be connected via a bus or other means.

[0200] The memory 504 may include a read-only memory and a random access memory, and provides instructions and data to the processor 503. A portion of the memory 504 may also include a non-volatile random access memory (NVRAM). The memory 504 stores processor and operation instructions, executable modules or data structures, or subsets thereof, or extended sets thereof, wherein the operation instructions may include various operation instructions for implementing various operations.

[0201] The processor 503 controls the operation of the execution device. In a specific application, the various components of the execution device are coupled together through a bus system, wherein the bus system includes not only a data bus but also a power bus, a control bus, and a status signal bus, etc. However, for the sake of clarity, various buses are referred to as bus systems in the figure.

[0202] The method disclosed in the above embodiment of the present application can be applied to the processor 503, or implemented by the processor 503. The processor 503 can be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 503. The above processor 503 can be a general processor, a digital signal processor (digital signal processing, DSP), a microprocessor or a microcontroller, and can further include an application specific integrated circuit (application specific integrated circuit, ASIC), a field programmable gate array (field-programmable gate array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The processor 503 can implement or execute the various methods, steps and logic block diagrams disclosed in the embodiment of the present application. The general processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiment of the present application can be directly embodied as a hardware decoding processor to execute, or the hardware and software modules in the decoding processor can be combined and executed. The software module may be located in a storage medium mature in the art, such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 504, and the processor 503 reads the information in the memory 504 and completes the steps of the above method in combination with its hardware.

[0203] The receiver 501 can be used to receive input digital or character information and generate signal input related to the relevant settings and function control of the execution device. The transmitter 502 can be used to output digital or character information through the first interface; the transmitter 502 can also be used to send instructions to the disk group through the first interface to modify the data in the disk group; the transmitter 502 can also include a display device such as a display screen.

[0204] In the embodiment of the present application, the application processor 5031 in the processor 503 is used to execute Figures 1 to 4 The training method or conversion method in the corresponding embodiment. It should be noted that the specific manner in which the application processor 5031 executes each step is different from the specific manner in which the application processor 5031 executes each step. Figures 1 to 4 The corresponding method embodiments are based on the same concept, and the technical effects they bring are the same as those in this application. Figures 1 to 4 The corresponding method embodiments are the same. For specific contents, please refer to the description in the method embodiments shown above in this application, which will not be repeated here.

[0205] An embodiment of the present application further provides a computer-readable storage medium, which includes computer instructions. When the computer instructions are executed by a processor, they are used to implement any one of the technical solutions in the first embodiment or the second embodiment of the present application.

[0206] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0207] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0208] Among them, computer-readable media include permanent and non-permanent, removable and non-removable media that can achieve information storage by any method or technology. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined in this article, computer-readable media does not include non-temporary computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0209] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0210] Although the present application is disclosed as above in the form of a preferred embodiment, it is not intended to limit the present application. Any technical personnel in this field may make possible changes and modifications without departing from the spirit and scope of the present application. Therefore, the scope of protection of the present application shall be based on the scope defined by the claims of the present application.

Claims

1. A training method for an audio processing model, characterized in that: The method comprises: Determine sample pronunciation features corresponding to each audio segment of the sample audio data; Inputting the sample pronunciation feature into a first model to be trained for numerical encoding to obtain a numerical first output feature corresponding to the sample pronunciation feature, wherein the first model to be trained is a model that numerically encodes the pronunciation feature based on past information, wherein the past information is a past pronunciation feature corresponding to a past audio segment generated before the audio segment corresponding to the sample pronunciation feature; Inputting the sample pronunciation feature into a second model for numerical encoding to obtain a numerical second output feature corresponding to the sample pronunciation feature, wherein the second model is a model for numerically encoding the pronunciation feature based on the past information and future information, and the future information is a future pronunciation feature corresponding to a future audio segment generated after the audio segment corresponding to the sample pronunciation feature; Based on a first parameter adjustment principle of reducing the difference between the first output feature and the second output feature, adjusting the parameters of the first model to be trained to obtain an adjusted first model, including: determining whether the first model to be trained has completed training according to whether the difference between the first output feature and the second output feature meets a convergence condition, thereby obtaining the adjusted first model; Determining the audio processing model according to the adjusted first model includes: determining the adjusted first model as the audio processing model, or further optimizing the adjusted first model to obtain the audio processing model.

2. The method according to claim 1, characterized in that The first model to be trained is a causal convolutional model, and the second model is a non-causal convolutional model.

3. The method according to claim 1, characterized in that Before determining the trained audio processing model according to the adjusted first model, the method further includes: Determining sample timbre characteristics of the sample audio data; Fusion the sample timbre feature with the first output feature to obtain fused audio data; The step of determining the audio processing model according to the adjusted first model comprises: Based on the second parameter adjustment principle of reducing the difference between the fused audio data and the sample audio data, the parameters of the adjusted first model are adjusted to obtain an audio processing model.

4. The method according to claim 3, characterized in that The method further comprises: The parameters of the second model are adjusted based on the second parameter adjustment principle to obtain an adjusted second model.

5. The method according to claim 3, characterized in that: The step of fusing the sample timbre feature with the first output feature to obtain fused audio data includes: Inputting the sample timbre feature and the first output feature into a first fusion model to obtain first fused audio data; The method further comprises: Based on the principle of reducing the difference between the first fused audio data and the sample audio data, the parameters of the first fusion model are adjusted to obtain an adjusted first fusion model.

6. The method according to claim 5, characterized in that The first fusion model is a model for fusing data based on past first fusion information, and the past first fusion information includes a first output feature corresponding to the past pronunciation feature.

7. The method according to claim 6, characterized in that The method further comprises: Inputting the sample timbre feature and the second output feature into a second fusion model to obtain second fused audio data, wherein the second fusion model is a model for fusing data based on past second fusion information and future second fusion information, wherein the past second fusion information includes the second output feature corresponding to the past pronunciation feature, and the future second fusion information includes the second output feature corresponding to the future pronunciation feature; Based on the principle of reducing the difference between the second fused audio data and the sample audio data, the parameters of the second fusion model are adjusted to obtain an adjusted second fusion model.

8. The method according to claim 1, characterized in that Before determining the audio processing model according to the adjusted first model, the method further includes: Inputting the first output feature and the second output feature into a prediction network to obtain sample prediction data corresponding to the sample audio data; The step of determining the audio processing model according to the adjusted first model comprises: Based on the principle of reducing the difference between the sample prediction data and the sample audio data, the parameters of the adjusted first model are adjusted to obtain an audio processing model.

9. The method according to claim 8, characterized in that The prediction network includes a first prediction model and a second prediction model, the first prediction model is used to predict the data of the next frame according to the data of the current frame, and the second prediction model is used to select the data of the next frame from the multiple frame data; The step of inputting the first output feature and the second output feature into a prediction network to obtain sample prediction data corresponding to the sample audio data includes: Inputting the first output feature and the second output feature into the first prediction model to obtain first sample prediction data corresponding to the sample audio data; Inputting the first output feature and the second output feature into the second prediction model to obtain second sample prediction data corresponding to the sample audio data; The step of adjusting parameters of the adjusted first model based on the principle of reducing the difference between the sample prediction data and the sample audio data comprises: Based on the principle of reducing the difference between the first sample prediction data, the second sample prediction data and the sample audio data, the parameters of the adjusted first model are adjusted.

10. The method according to claim 5, characterized in that The step of inputting the sample timbre feature and the first output feature into a first fusion model to obtain first fused audio data includes: Get the pitch characteristics of sample audio data; The sample timbre feature, the first output feature and the pitch feature are input into a first fusion model to obtain first fused audio data.

11. The method according to any one of claims 1 to 10, characterized in that: The step of determining the sample pronunciation features corresponding to each audio segment of the sample audio data includes: Convert each audio segment of the sample audio data into sample Melp spectrum data; The Melp spectrum data is input into a pre-trained speech recognition model to obtain bottleneck features corresponding to each audio segment of the sample audio data, and the bottleneck features are determined as sample pronunciation features corresponding to each audio segment of the sample audio data. The speech recognition model is used to identify the pronunciation features of the audio data.

12. An audio conversion method, characterized in that: The method comprises: Obtain source audio to be converted and timbre information of the target speaker; Extracting pronunciation features of the source audio, wherein the pronunciation features are features of the source audio after at least the timbre information of the source speaker is filtered out; Inputting the pronunciation feature into a pre-trained audio processing model to obtain a first audio extraction feature corresponding to the source audio, wherein the audio processing model is trained according to the training method of the audio processing model according to any one of claims 1 to 11; The first audio extraction feature is fused with the timbre information to obtain converted audio corresponding to the target speaker.

13. The audio conversion method according to claim 12, characterized in that: The step of fusing the first audio extraction feature with the timbre information to obtain converted audio corresponding to the target speaker includes: The first audio extraction feature and the timbre information are input into a pre-trained first fusion model to obtain the converted audio corresponding to the target speaker, and the first fusion model is trained by training the adjusted first fusion model in the training method of the audio processing model according to claim 5 or 6.

14. The audio conversion method according to claim 12, characterized in that: The step of inputting the pronunciation features into a pre-trained audio processing model comprises: When the source audio is a real-time audio stream, inputting the pronunciation features into a pre-trained audio processing model; The audio conversion method further includes: When the source audio is a non-real-time audio stream, the pronunciation feature is input into the second model to obtain a second audio extraction feature corresponding to the source audio, and the second model is trained by training the adjusted second model in the training method of the audio processing model according to claim 4; The second audio extraction feature is fused with the timbre information to obtain converted audio corresponding to the target speaker.

15. The audio conversion method according to claim 14, characterized in that: The step of fusing the second audio extraction feature with the timbre information to obtain converted audio corresponding to the target speaker includes: The second audio extraction features and the timbre information are input into a pre-trained second fusion model to obtain the converted audio corresponding to the target speaker, and the second fusion model is trained by training the adjusted second fusion model in the training method of the audio processing model according to claim 7.

16. A training device for an audio processing model, characterized in that: include: A first determining unit, used to determine sample pronunciation features corresponding to each audio segment of the sample audio data; A first extraction unit is used to input the sample pronunciation feature into a first model to be trained for numerical encoding to obtain a first numerical output feature corresponding to the sample pronunciation feature, wherein the first model to be trained is a model that numerically encodes the pronunciation feature based on past information, and the past information is a past pronunciation feature corresponding to a past audio segment generated before the audio segment corresponding to the sample pronunciation feature; input the sample pronunciation feature into a second model for numerical encoding to obtain a second numerical output feature corresponding to the sample pronunciation feature, wherein the second model is a model that numerically encodes the pronunciation feature based on the past information and future information, and the future information is a future pronunciation feature corresponding to a future audio segment generated after the audio segment corresponding to the sample pronunciation feature; an adjusting unit, configured to adjust parameters of the first model to be trained based on a first parameter adjustment principle of reducing the difference between the first output feature and the second output feature to obtain an adjusted first model, comprising: determining whether the first model to be trained has completed training according to whether the difference between the first output feature and the second output feature meets a convergence condition, thereby obtaining the adjusted first model; The second determination unit is used to determine the audio processing model according to the adjusted first model, including: determining the adjusted first model as the audio processing model, or further optimizing the adjusted first model to obtain the audio processing model.

17. An audio conversion device, characterized in that: include: An acquisition unit, used to acquire source audio to be converted and timbre information of a target speaker; A second extraction unit is used to extract the pronunciation feature of the source audio, where the pronunciation feature is the feature of the source audio after at least the timbre information of the source speaker is filtered out; A processing unit, configured to input the pronunciation feature into a pre-trained audio processing model to obtain a first audio extraction feature corresponding to the source audio, wherein the audio processing model is trained according to the training method for an audio processing model according to any one of claims 1 to 11; A fusion unit is used to fuse the first audio extraction feature with the timbre information to obtain a converted audio corresponding to the target speaker.

18. An electronic device, characterized in that: The electronic device comprises: a memory and a processor; the memory and the processor are coupled; The memory is used to store one or more computer instructions; The processor is configured to execute the one or more computer instructions to implement the method according to any one of claims 1-15.

19. A computer-readable storage medium having one or more computer instructions stored thereon, characterized in that: The instruction is executed by a processor to implement the method according to any one of claims 1 to 15.

Images