Training method for audio processing model, and method and apparatus for audio conversion

By training an audio processing model that can perceive future information, the problem of not being able to obtain future information in streaming singing voice conversion is solved, the accuracy and sound quality of singing voice conversion is improved, and it is suitable for real-time application scenarios.

WO2024239962A1PCT designated stage expired Publication Date: 2024-11-28NETEASE (HANGZHOU) NETWORK CO LTD

Patent Information

Application Number
PCT/CN2024/091745
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-05-19
Filing Date
2024-05-08
Publication Date
2024-11-28

AI Technical Summary

Technical Problem

The existing streaming singing voice conversion method cannot obtain information about the current singing voice in the future, resulting in low accuracy of singing voice conversion and poor conversion effect.

Method used

By training an audio processing model, the model inputs the sample pronunciation characteristics corresponding to each audio band of the sample audio data into two models for numerical encoding. The first model is based on past information encoding, and the second model is based on past and future The information encoding of the first model is adjusted to enable it to perceive future information, thereby improving conversion accuracy.

Benefits of technology

It improves the accuracy and sound quality of singing voice conversion, making the converted singing voice better, the basic frequency is more stable, and is suitable for real-time application scenarios.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN2024091745_28112024_PF_FP_ABST
    Figure CN2024091745_28112024_PF_FP_ABST
Patent Text Reader

Abstract

A training method for an audio processing model, a method and apparatus for audio conversion, an electronic device, and a computer-readable storage medium. The training method comprises: determining a sample pronunciation feature of sample audio data (S110); and inputting the sample pronunciation feature into a first model to be trained for numericalization coding to obtain a numericalized first output feature, the first model to be trained being a model for performing numericalization coding on data on the basis of past information (S120); inputting the sample pronunciation feature into a second model for numericalization coding to obtain a numericalized second output feature, the second model being a model for processing data on the basis of past and future information (S130); on the basis of a first parameter adjustment principle of reducing the difference between the first output feature and the second output feature, performing parameter adjustment on the first model to be trained to obtain an adjusted first model (S140); and determining an audio processing model according to the adjusted first model (S150). The method enables stream song conversion to achieve stream reasoning while ensuring the effect of stream reasoning.
Need to check novelty before this filing date? Find Prior Art

Description

Audio processing model training method, audio conversion method and device

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to Chinese patent application number 202310576603.5 filed on May 19, 2023, entitled “Training method, audio conversion method and device for audio processing model”, the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present disclosure relates to the field of computer technology, and in particular to a training method for an audio processing model, an audio conversion method, an apparatus, an electronic device, and a computer-readable storage medium. Background Art

[0004] Singing voice conversion, a subset of speech conversion, involves converting one person's singing audio into the voice of another. It can be applied to areas such as online live streaming and dubbing for film and television dramas. Streaming singing voice conversion returns recognition results in real time while processing audio. Compared to whole-segment conversion methods, streaming conversion models are suitable for applications with real-time requirements.

[0005] Since streaming singing is real-time, it is impossible to obtain the singing information of the currently converted streaming singing at a future moment during the conversion. Therefore, the existing streaming singing conversion method is usually based on a model that does not rely on future information input for singing processing, for example, through a unidirectional recurrent neural network, a causal convolution model, etc. for singing processing during singing conversion.

[0006] However, since the singing processing model in the singing conversion network cannot perceive future information, the conversion accuracy of the singing conversion network is low, resulting in poor effect of the converted singing.

[0007] Summary of the Invention

[0008] In view of this, the present disclosure provides a training method for an audio processing model, an audio conversion method, an apparatus, an electronic device and a computer-readable storage medium, which can improve the accuracy of singing conversion and make the effect of the converted singing better.

[0009] A first aspect of the present disclosure provides a method for training an audio processing model, the method comprising:

[0010] Determining sample pronunciation features corresponding to each audio segment of the sample audio data;

[0011] Inputting the sample pronunciation feature into a first to-be-trained model for numerical encoding to obtain a numerical first output feature corresponding to the sample pronunciation feature, wherein the first to-be-trained model is a model that numerically encodes the pronunciation feature based on past information, wherein the past information is past pronunciation features corresponding to past audio segments generated before the audio segment;

[0012] Inputting the sample pronunciation feature into a second model for numerical encoding to obtain a numerical second output feature corresponding to the sample pronunciation feature, wherein the second model is a model for numerically encoding the pronunciation feature based on the past information and future information, and the future information is a future pronunciation feature corresponding to a future audio segment generated after the audio segment;

[0013] Based on a first parameter adjustment principle of reducing the difference between the first output feature and the second output feature, adjusting the parameters of the first to-be-trained model to obtain an adjusted first model;

[0014] An audio processing model is determined according to the adjusted first model.

[0015] A second aspect of the present disclosure provides an audio conversion method, the method comprising:

[0016] Obtain the source audio to be converted and the timbre information of the target speaker;

[0017] Extracting pronunciation features of the source audio, wherein the pronunciation features are features of the source audio after at least timbre information of a source speaker is filtered out;

[0018] Inputting the pronunciation feature into a pre-trained audio processing model to obtain a first audio extraction feature corresponding to the source audio, wherein the audio processing model is trained according to the audio processing model training method described in any one of the first aspects;

[0019] The first audio extraction feature is fused with the timbre information to obtain converted audio corresponding to the target speaker.

[0020] A third aspect of the present disclosure provides a training device for an audio processing model, including:

[0021] A first determining unit is configured to determine a sample pronunciation feature corresponding to each audio segment of the sample audio data;

[0022] The first extraction unit is configured to input the sample pronunciation feature into a first model to be trained for numerical encoding to obtain a first numerical output feature corresponding to the sample pronunciation feature, wherein the first model to be trained is a model that numerically encodes the pronunciation feature based on past information, and the past information is a past pronunciation feature corresponding to a past audio segment generated before the audio segment; input the sample pronunciation feature into a second model for numerical encoding to obtain a second numerical output feature corresponding to the sample pronunciation feature, wherein the second model is a model that numerically encodes the pronunciation feature based on the past information and future information, and the future information is a future pronunciation feature corresponding to a future audio segment generated after the audio segment;

[0023] an adjusting unit configured to perform parameter adjustment on the first to-be-trained model based on a first parameter adjustment principle that reduces a difference between the first output feature and the second output feature, to obtain an adjusted first model;

[0024] The second determining unit is configured to determine an audio processing model according to the adjusted first model.

[0025] A fourth aspect of the present disclosure provides an audio conversion device, including:

[0026] An acquisition unit configured to acquire source audio to be converted and timbre information of a target speaker;

[0027] A second extraction unit is configured to extract pronunciation features of the source audio, where the pronunciation features are features of the source audio after at least timbre information of a source speaker is filtered out;

[0028] a processing unit configured to input the pronunciation feature into a pre-trained audio processing model to obtain a first audio extraction feature corresponding to the source audio, wherein the audio processing model is trained according to the audio processing model training method according to any one of the first aspects;

[0029] The fusion unit is configured to perform fusion of the first audio extraction feature and the timbre information to obtain converted audio corresponding to the target speaker.

[0030] A fifth aspect of an embodiment of the present disclosure provides an electronic device, including: a memory and a processor, wherein the memory and the processor are coupled.

[0031] The memory is used to store one or more computer instructions.

[0032] The processor is configured to execute one or more computer instructions to implement the method described in the first aspect or the second aspect above.

[0033] The sixth aspect of the embodiments of the present disclosure further provides a computer-readable storage medium on which one or more computer instructions are stored, characterized in that the instructions are executed by a processor to implement the method described in the first or second aspect above.

[0034] Compared with the related art, the embodiments of the present disclosure have the following advantages:

[0035] The training method of the audio processing model provided by the embodiment of the present disclosure, after determining the sample pronunciation features corresponding to each audio segment of the sample audio data, inputs the sample pronunciation features into the first model to be trained for numerical encoding to obtain the first numerical output features, and inputs the sample pronunciation features into the second model for numerical encoding to obtain the second numerical output features. Since the first model to be trained is a model that numerically encodes pronunciation features based on past information, the past information is the past pronunciation features corresponding to the past audio segments generated before the audio segment, and the second model is a model that numerically encodes pronunciation features based on past and future information, and the future information is the future pronunciation features corresponding to the future audio segments generated after the audio segment, that is, the first model to be trained cannot perceive future information when performing numerical encoding, and the second model can perceive future information when performing numerical encoding. Therefore, the second output feature is a feature obtained based on the perception of past and future information, and the first output feature is a feature that does not perceive future information. The embodiment of the present disclosure adjusts the parameters of the first model to be trained based on the first parameter adjustment principle of reducing the difference between the first output feature and the second output feature. In this way, the output of the adjusted first model can be more consistent with the output result of the model that can perceive future information. That is, the present disclosure distills the first model to be trained that cannot perceive future information based on the output result of the second model that can perceive future information. In this way, although the adjusted first model obtained by distillation does not use future information when performing digital encoding, the result of the digital encoding of the adjusted first model is also able to perceive future information well. Therefore, the audio processing model determined according to the adjusted first model can also perceive future information well.

[0036] During audio conversion, the vocal features of the source audio are extracted using an audio processing model that can perceive future information well. Therefore, the converted audio obtained by fusing the extracted first audio extraction features with the timbre of the target speaker is also a converted audio based on the perception of future information. Therefore, the converted audio has higher accuracy, better intelligibility and sound quality, more stable fundamental frequency, and better conversion effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the following briefly introduces the drawings required for use in the description of the embodiments of the present disclosure. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0038] FIG1 is a flow chart of an example of a method for training an audio processing model according to one embodiment of the present disclosure;

[0039] FIG2 is a flow chart of another example of a method for training an audio processing model according to one embodiment of the present disclosure;

[0040] FIG3 is a schematic diagram of the structure of a prediction network in one of the embodiments of the present disclosure;

[0041] FIG4 is a flow chart of an audio conversion method provided by one of the embodiments of the present disclosure;

[0042] FIG5 is a schematic diagram of the structure of a training device for an audio processing model provided by one of the embodiments of the present disclosure;

[0043] FIG6 is a schematic structural diagram of an electronic device provided in one of the embodiments of the present disclosure. DETAILED DESCRIPTION

[0044] To enable those skilled in the art to better understand the technical solutions of the present disclosure, the present disclosure is described clearly and completely below in conjunction with the accompanying drawings of the embodiments of the present disclosure. However, the present disclosure can be implemented in many other ways different from the above description. Therefore, based on the embodiments provided by the present disclosure, all other embodiments obtained by those skilled in the art without creative work should fall within the scope of protection of the present disclosure.

[0045] It should be noted that the terms "first", "second", "third", etc. in the claims, description and drawings of the present disclosure are used to distinguish similar objects and are not used to describe a specific order or sequence. The data used in this way are interchangeable where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including", "having" and their variations are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0046] Singing voice conversion, a subset of speech conversion, involves converting one person's singing audio into the voice of another. It can be applied to areas such as online live streaming and dubbing for film and television dramas. The current mainstream singing voice conversion method is full-segment conversion. This involves inputting the entire audio segment into a non-streaming singing voice conversion model, waiting for the model to complete processing, and then outputting the entire converted audio segment. However, because the non-streaming singing voice conversion method requires waiting for the entire audio segment to be processed before returning a result, it cannot be applied to application scenarios with real-time requirements.

[0047] Streaming singing voice conversion can return recognition results in real time during the audio processing process. Compared with the whole-segment conversion method, the streaming conversion model can be applied to application scenarios with real-time requirements.

[0048] A common practice for singing voice conversion is to first extract the pronunciation features in the audio that are not related to the speaker, and then perform numerical encoding processing on the pronunciation features. For example, the pronunciation features are vectorized, the vectors are transformed in dimension, etc., so as to process the pronunciation features into numerical information that is convenient for singing voice conversion, thereby enriching the information content of the pronunciation features, and then fusing the processed pronunciation features with the timbre of the target speaker to obtain the singing voice converted into the timbre of the target speaker.

[0049] For streaming singing conversion, since the recognition results need to be returned in real time during the audio processing process, it is different from the conversion of the entire segment. Instead, it is necessary to convert the singing transmitted in real time. Usually, it is necessary to train a feature processing model for streaming conversion to meet the application scenarios of the real-time requirements of streaming singing conversion.

[0050] Since streaming singing is real-time, it is impossible to obtain the singing information of the currently converted streaming singing at a future moment during the conversion. Therefore, the existing streaming singing conversion method is usually based on a feature processing model that does not rely on future information input to process the pronunciation features of the singing, for example, through a unidirectional recurrent neural network, a causal convolution model, etc. to perform singing processing during singing conversion.

[0051] However, since the feature processing model in the singing voice conversion network cannot perceive future information, the conversion accuracy of the singing voice conversion network is low, resulting in poor effect of the converted singing.

[0052] In order to improve the accuracy of singing voice conversion and make the converted singing voice more effective, the first embodiment of the present disclosure provides a training method for an audio processing model.

[0053] The executor of the training method of the audio processing model provided in the present disclosure may be an electronic device, which may be a server, a desktop computer, a laptop computer, a tablet computer, a mobile phone, a smart watch, a smart TV, etc., or other electronic devices with data processing functions, which are not specifically limited in the present disclosure.

[0054] The model trained by the audio processing model training method provided in the present disclosure can be used to process singing audio during singing conversion, and can also be used to process audio during other pronunciation conversions, for example, processing recitation audio to convert the speaker's timbre of the recitation audio. In addition to audio conversion scenarios, the model trained by the audio processing model training method provided in the present disclosure can also be used for audio processing in other scenarios, for example, for processing speech during speech recognition.

[0055] Figure 1 is a flowchart of a method for training an audio processing model according to an embodiment of the present disclosure. It should be noted that the steps shown in the flowchart can be executed in a computer system such as a set of computer-executable instructions, and in some cases, the steps can be executed in a different logical order than that shown in the flowchart.

[0056] The training method of the audio processing model provided in the present disclosure includes the following steps S110 to S150.

[0057] Step S110: Determine sample pronunciation features corresponding to each audio segment of the sample audio data.

[0058] The sample audio data can be sample singing audio data, sample recitation audio data, etc., or other audio data containing the speaker's voice information. The sample audio data can be pre-recorded for training the audio processing model, or it can be various audios with the speaker's voice searched on the Internet. The present disclosure does not limit the specific content of the sample audio data.

[0059] In an embodiment of the present disclosure, the sample audio data can be divided into multiple audio segments in the order of generation time from front to back. Specifically, it can be divided into multiple audio segments based on the audio duration, or based on the number of pronunciations. For example, a 20-second audio segment can be divided into 4 audio segments with every 5 seconds as an audio segment; for another example, an audio segment with a total of 100 pronunciations can be divided into 5 audio segments with every 20 pronunciations as an audio segment. The sample audio data can also be segmented in the order from front to back by other methods. In step S110, the sample pronunciation features corresponding to each audio segment of the sample audio data are determined.

[0060] The sample pronunciation feature can be understood as the feature of the sample audio data after at least the speaker's timbre information is filtered out. That is to say, the sample pronunciation feature is information that is unrelated to the speaker's timbre. The sample pronunciation feature is used to represent the features of each pronunciation in the sample audio data. Based on the pronunciation feature, the specific pronunciation in the sample audio data that is unrelated to the speaker's timbre can be determined.

[0061] The pronunciation features may be expressed in text form or in the form of phonemes, etc. The pronunciation features include the pronunciation features of each moment or each frame in the sample audio data.

[0062] Since audio such as singing contains not only semantics but also vocal information such as tone and rhythm, when training the audio processing model, the present disclosure trains the model based on pronunciation features to improve the scope of use of the trained audio processing model in order to make the trained audio processing model not only applicable to scenarios such as conventional recitation where the pronunciation tone is consistent with the tone of the text itself, but also applicable to scenarios where the pronunciation tone of singing may be inconsistent with the tone of the text itself.

[0063] In an embodiment of the present disclosure, sample pronunciation features of sample audio data can be determined using a pre-trained speech recognition model. Specifically, the sample audio data can be input into the pre-trained speech recognition model to obtain sample pronunciation features corresponding to the sample audio data. The speech recognition model can be trained based on conventional supervised training methods, for example, by training the speech recognition model based on audio samples that have been assigned corresponding audio features. The specific training method will not be described in detail in this disclosure.

[0064] Optionally, in step S110 , the pronunciation features may be determined according to the following steps S111 to S112 .

[0065] Step S111: converting each audio segment of the sample audio data into sample Melp spectrum data.

[0066] Step S112: Input the Melp spectrum data into a pre-trained speech recognition model to obtain bottleneck features corresponding to each audio segment of the sample audio data, and determine the bottleneck features as sample pronunciation features corresponding to each audio segment of the sample audio data. The speech recognition model is used to identify the pronunciation features of the audio data.

[0067] Step S120: Input the above-mentioned sample pronunciation features into the first model to be trained for numerical encoding to obtain the numerical first output features corresponding to the sample pronunciation features. The first model to be trained is a model that numerically encodes pronunciation features based on past information.

[0068] The above-mentioned past information is a past pronunciation feature corresponding to a past audio segment generated before the audio segment corresponding to the sample pronunciation feature.

[0069] In this step, each sample pronunciation feature corresponding to each audio segment of the sample audio data can be input into the first model to be trained for numerical encoding. When the first model to be trained numerically encodes the target sample pronunciation feature among the sample pronunciation features, it numerically encodes the target sample pronunciation feature based on the sample pronunciation feature before the target sample pronunciation feature. The sample pronunciation feature before the target sample pronunciation feature is the past pronunciation feature corresponding to the past audio segment generated before the audio segment corresponding to the target sample pronunciation feature.

[0070] For example, the sample audio data includes four audio segments a, b, c, and d, and these four audio segments correspond to their own sample pronunciation features. The four sample pronunciation features corresponding to the four audio segments a, b, c, and d can be input into the first model to be trained for numerical encoding. When the sample pronunciation features corresponding to the audio segment a are numerically encoded, since there is no audio segment before a, the past information corresponding to a can be set to empty by default. At this time, the pronunciation features corresponding to a can be directly numerically encoded; when the sample pronunciation features corresponding to the audio segment b are numerically encoded, since the audio segment before b is a, the pronunciation features corresponding to b can be numerically encoded based on the pronunciation features corresponding to the audio segment a; when the sample pronunciation features corresponding to the audio segment c are numerically encoded, since the audio segments before c are a and b, the pronunciation features corresponding to c can be numerically encoded based on the pronunciation features corresponding to the audio segments a and b.

[0071] Specifically, the first model to be trained can be a unidirectional recurrent neural network model, a causal convolutional neural network model, or other model that digitally encodes data based on past information. The first model to be trained digitally processes data based on past information, which can be understood as the first model to be trained digitally encodes data based on past information and not future information.

[0072] Step S130: Input the above-mentioned sample pronunciation features into the second model for numerical encoding to obtain the numerical second output features corresponding to the sample pronunciation features. The second model is a model that numerically encodes the pronunciation features based on the above-mentioned past information and future information.

[0073] The above-mentioned future information is a future pronunciation feature corresponding to a future audio segment generated after the audio segment corresponding to the sample pronunciation feature.

[0074] In this step, each sample pronunciation feature corresponding to each audio segment of the sample audio data can be input into the second model for numerical encoding. When the second model numerically encodes the target sample pronunciation feature among the sample pronunciation features, it numerically encodes the target sample pronunciation feature based on the sample pronunciation feature subsequent to the target sample pronunciation feature. The sample pronunciation feature subsequent to the target sample pronunciation feature is the future pronunciation feature corresponding to the future audio segment generated after the audio segment corresponding to the target sample pronunciation feature.

[0075] Taking the example of the sample audio data in the above text including four audio segments a, b, c, and d, the four sample pronunciation features corresponding to the four audio segments a, b, c, and d can be input into the second model for numerical encoding. When the second model numerically encodes the sample pronunciation features corresponding to audio segment a, since there is no audio segment before a, and the audio segments after a are b, c, and d, the pronunciation features corresponding to a can be numerically encoded based on the three future pronunciation features corresponding to the three future audio segments b, c, and d; when the second model numerically encodes the sample pronunciation features corresponding to audio segment b, since the audio segment before b is a, and the audio segments after b are c and d, the pronunciation features corresponding to b can be numerically encoded based on the past audio features corresponding to the past audio segment a, and the two future pronunciation features corresponding to the two future audio segments c and d.

[0076] The second model can be a trained model or a model that has not been trained. If the second model is a model that has not been trained, the second model can be trained synchronously during the audio processing model training process disclosed in the present invention.

[0077] The second model can be a bidirectional recurrent neural network model, a non-causal convolutional neural network model, etc., or other models that process data based on past information and future information. That is to say, the second model can perceive future information when processing data.

[0078] In the embodiment of the present disclosure, the first model to be processed and the second model can be at least one of the numerical coding processing processes such as vector coding processing, matrix coding processing, and dimension increase processing of the sample pronunciation features, or can be other numerical coding processing of the sample pronunciation features. The specific processing content can be determined according to actual needs and is not specifically limited in the present disclosure.

[0079] It can be understood that the first model to be processed and the second model perform the same dimensional processing on the pronunciation features. For example, the first model to be processed is used to vectorize the pronunciation features, and the second model also vectorizes the pronunciation features.

[0080] Step S140: Based on the first parameter adjustment principle of reducing the difference between the first output feature and the second output feature, the parameters of the first model to be trained are adjusted to obtain an adjusted first model.

[0081] Specifically, the audio processing model can be trained using a large amount of sample audio data, and whether the first trained model has completed training is determined based on whether the difference between the first output feature and the second output feature meets a convergence condition, thereby obtaining an adjusted first model. The convergence condition can be, for example, that the difference between the first output feature and the second output feature is less than a first preset threshold, or that, for a plurality of sample audio data, the ratio of the differences between the first output feature and the second output feature that are less than the first preset threshold reaches a first preset ratio threshold. The first preset ratio threshold can be, for example, 80% to 95%, or another relatively large ratio.

[0082] Step S150: Determine an audio processing model based on the adjusted first model.

[0083] Specifically, the adjusted first model can be directly determined as the audio processing model, or the adjusted first model can be further optimized to obtain the audio processing model. The specific processing method can be determined according to the specific application scenario of the audio processing model.

[0084] For example, when the audio processing model provided by the present disclosure is used for an audio timbre conversion scenario, if the second model is a pre-trained model that meets the timbre conversion scenario, the adjusted first model can be directly determined as the audio processing model.

[0085] In order to make the converted audio obtained after the timbre conversion more accurate, the audio processing model can also be determined by the following specific implementation method. In this specific implementation method, before step S150, the following steps S140a to S140b can be included, and step S150 can be implemented according to the following step S151.

[0086] Step S140a: Determine the sample timbre characteristics of the sample audio data.

[0087] Specifically, the sample timbre features of the sample audio data can be identified using a pre-trained timbre recognition model. Alternatively, the sample timbre features of the sample audio data can be determined by manual labeling, i.e., the timbre of the sample audio data is manually identified and then the timbre is identified. Alternatively, a timbre library can be pre-set, containing various timbre features, and a pre-trained timbre determination model can be used to determine the timbre consistent with the timbre of the sample audio data from the timbre library. The pre-trained timbre recognition model and timbre determination model can be trained using supervised or unsupervised training methods, which will not be described in detail in this disclosure.

[0088] Optionally, as shown in FIG2 , a speaker identifier corresponding to the sample audio data may be obtained, and the timbre feature corresponding to the speaker identifier may be determined from a pre-stored timbre library through the speaker embedding layer as the sample timbre feature.

[0089] Step S140b: Fusing the sample timbre feature with the first output feature to obtain fused audio data.

[0090] Specifically, the sample timbre feature and the first output feature can be fused through a fusion model; alternatively, the harmonic distribution of the sample timbre can be obtained, and the first output feature can be made to have a harmonic distribution consistent with the harmonic distribution through a filter to obtain fused audio data; fusion can also be performed in other ways, which are not specifically limited in this disclosure.

[0091] Step S151: Based on the second parameter adjustment principle of reducing the difference between the fused audio data and the sample audio data, the parameters of the adjusted first model are adjusted to obtain an audio processing model.

[0092] Specifically, when the difference between the fused audio data and the sample audio data satisfies a second convergence condition, it can be determined that the adjustment of the parameters of the adjusted first model is complete, thereby obtaining the audio processing model. The second convergence condition can be, for example, that the difference between the fused audio data and the corresponding sample audio data is less than a second preset threshold, or that, for multiple sample audio data, the ratio of the differences between the fused audio data and the sample audio data that are less than the second preset threshold reaches a second preset ratio threshold. The second preset ratio threshold can be the same as or different from the first preset ratio threshold, and is not specifically limited in this disclosure.

[0093] This embodiment adjusts the parameters of the adjusted first model by judging whether the fused audio data after the fusion of the first output feature and the timbre feature is accurate, so that the feature processing capability of the obtained audio processing model can better meet the scenario of timbre and pronunciation fusion. In this way, for the scenario of timbre and pronunciation fusion, the timbre fusion accuracy of the features processed by the audio processing model is higher.

[0094] The training method of the audio processing model provided by the embodiment of the present disclosure, after determining the sample pronunciation features corresponding to each audio segment of the sample audio data, inputs the sample pronunciation features into the first model to be trained for numerical encoding to obtain the first numerical output features, and inputs the sample pronunciation features into the second model for numerical encoding to obtain the second numerical output features. Since the first model to be trained is a model that numerically encodes pronunciation features based on past information, the past information is the past pronunciation features corresponding to the past audio segments generated before the audio segment, and the second model is a model that numerically encodes pronunciation features based on past and future information, and the future information is the future pronunciation features corresponding to the future audio segments generated after the audio segment, that is, the first model to be trained cannot perceive future information when performing numerical encoding, and the second model can perceive future information when performing numerical encoding. Therefore, the second output feature is a feature obtained based on the perception of past and future information, and the first output feature is a feature that does not perceive future information. The embodiment of the present disclosure adjusts the parameters of the first model to be trained based on the first parameter adjustment principle of reducing the difference between the first output feature and the second output feature. In this way, the output of the adjusted first model can be more consistent with the output result of the model that can perceive future information. That is, the present disclosure distills the first model to be trained that cannot perceive future information based on the output result of the second model that can perceive future information. In this way, although the adjusted first model obtained by distillation does not use future information when performing digital encoding, the result of the digital encoding of the adjusted first model is also able to perceive future information well. Therefore, the audio processing model determined according to the adjusted first model can also perceive future information well.

[0095] During audio conversion, the vocal features of the source audio are extracted using an audio processing model that can perceive future information well. Therefore, the converted audio obtained by fusing the extracted first audio extraction features with the timbre of the target speaker is also a converted audio based on the perception of future information. Therefore, the converted audio has higher accuracy, better intelligibility and sound quality, more stable fundamental frequency, and better conversion effect.

[0096] In a specific embodiment, the method may further include the following step S160.

[0097] Step S160: adjusting the parameters of the second model based on the second parameter adjustment principle to obtain an adjusted second model.

[0098] Step S160 adjusts the parameters of the second model based on the second parameter adjustment principle of reducing the difference between the fused audio data and the sample audio data, thereby obtaining an adjusted second model. The specific parameter adjustment method for the second model can be referenced above for adjusting the parameters of the adjusted first model, i.e., conventional supervised model training methods, and will not be further described here.

[0099] This embodiment also updates the parameters of the second model based on the fused audio data, so that the features extracted by the second model are more consistent with the scenario where timbre and pronunciation are integrated, so that the second model can more accurately extract features that meet the fusion scenario when the audio processing model is subsequently trained, so that the audio processing model finally trained can more accurately extract features that meet the fusion scenario. In addition, this embodiment updates and trains the second model, and the second model can be used directly to perform feature processing on non-streaming audio for the fusion of timbre and pronunciation. This allows the present disclosure to train models for processing both streaming and non-streaming audio in one training process, thereby improving the efficiency of model training.

[0100] The first model to be trained and the second model are the causal convolutional encoder and the non-causal convolutional encoder in Figure 2. The first model to be trained and the second model form the dual-mode encoder in Figure 2.

[0101] In a specific embodiment, in step S140b, the fused audio data can be obtained specifically according to the following step A, and the training method of the above audio processing model can also include the following step S170.

[0102] Step A: Input the sample timbre feature and the first output feature into a first fusion model to obtain first fused audio data.

[0103] Step S170: Based on the principle of reducing the difference between the first fused audio data and the sample audio data, the parameters of the first fusion model are adjusted to obtain an adjusted first fusion model.

[0104] The first fusion model may be a model for fusing data based on past first fusion information, wherein the past first fusion information includes the first output feature corresponding to the past pronunciation feature and the timbre feature corresponding to the past pronunciation feature.

[0105] For example, the sample audio data includes four audio segments a, b, c, and d. When the first fusion model fuses the sample timbre feature with the first output feature corresponding to a, since there is no audio segment before a, the fusion can be performed directly; when the first fusion model fuses the sample timbre feature with the first output feature corresponding to b, the audio segment before b includes a, so the first fusion model will fuse the sample timbre feature with the first output feature corresponding to b based on the first output feature corresponding to a; when the first fusion model fuses the sample timbre feature with the first output feature corresponding to c, the audio segment before c includes a and b, so the first fusion model will fuse the sample timbre feature with the first output feature corresponding to c based on the first output features corresponding to a and b respectively.

[0106] The specific adjustment process in step S170 can refer to the process of adjusting the parameters of the second model and the first fusion model above, which will not be described in detail here.

[0107] The first fusion model can be a neural network model, a decision tree model, a support vector machine model, etc., or other machine learning models.

[0108] Specifically, the first fusion model can be a model that processes data based on past information. For example, the first fusion model can be a causal convolution model, a unidirectional recurrent neural network model, etc. In this way, since the first output feature is a feature obtained based on past information, when the first fusion model is a model that processes data based on past information, the first fusion model is more suitable for fusing the first output feature, making the audio fusion more accurate. In addition, when the first fusion model is a model that processes data based on past information, the adjusted first fusion model obtained after parameter adjustment can subsequently perform more accurate fusion processing on streaming audio. This embodiment updates and trains the first fusion model, which can be directly used for audio fusion later, thereby improving the efficiency of model training.

[0109] The first fusion model can also be a model that processes data based on past and future information. For example, the first fusion model can be a non-causal convolution model, a bidirectional recurrent neural network model, etc., which can also achieve the fusion of timbre and pronunciation. This is not specifically limited in the present disclosure.

[0110] The first fused audio data obtained in step A is the fused audio data obtained in step S140b. In other words, the fused audio data obtained in step S140b includes the first fused audio data.

[0111] Optionally, in step S140b, the fused audio data may be obtained specifically according to the following step B, and the training method of the above audio processing model may further include the following step S180.

[0112] Step B: Input the sample timbre feature and the second output feature into the second fusion model to obtain second fused audio data.

[0113] The second fusion model is a model that processes data based on past and future information. The second fusion model can be a bidirectional neural network model, a non-causal convolutional model, or other models that process data based on past and future information.

[0114] The second fusion model is a model that fuses data based on past second fusion information and future second fusion information. The past second fusion information includes second output features corresponding to past pronunciation features, and the future second fusion information includes second output features corresponding to future pronunciation features.

[0115] For example, the sample audio data also includes four audio segments a, b, c, and d. When the second fusion model fuses the sample timbre feature and the second output feature corresponding to a, since there is no audio segment before a and the audio segment after a includes three audio segments b, c, and d, the second fusion model will fuse the sample timbre feature and the second output feature corresponding to a based on the second output features corresponding to the three future pronunciation features of the three future audio segments b, c, and d; when the second fusion model fuses the sample timbre feature and the second output feature corresponding to b, since the audio segment before b is a and the audio segment after b includes three audio segments c and d, the second fusion model will fuse the sample timbre feature and the second output feature corresponding to b based on the second output features corresponding to the past pronunciation features of the past audio segment a and the second output features corresponding to the two future pronunciation features of the two future audio segments c and d.

[0116] The second fused audio data obtained in step B is the fused audio data obtained in step S140b. In other words, the fused audio data obtained in step S140b also includes the second fused audio data.

[0117] Step S180: Based on the principle of reducing the difference between the second fused audio data and the sample audio data, the parameters of the second fusion model are adjusted to obtain an adjusted second fusion model.

[0118] The parameter adjustment process of step S180 can be the same as step S170 and will not be described in detail here.

[0119] In this embodiment, the adjusted second fusion model can be fused with the timbre of the target speaker when subsequently applied to non-streaming audio conversion. Multiple models can be obtained through one training, thereby improving the efficiency of model training.

[0120] As shown in Figure 2, the first fusion model and the second fusion model are the dual-mode decoder in Figure 2. The first fusion audio data and the second fusion audio data are the streaming audio and the non-streaming audio in Figure 2.

[0121] The adjusted second fusion model can be fused with the timbre of the target speaker when subsequently applied to the conversion of non-streaming audio.

[0122] In one embodiment, before step S150, the method may further include the following step a. Step S150 may be implemented according to the following step S152.

[0123] Step a: Input the first output feature into the prediction network to obtain sample prediction data corresponding to the sample audio data.

[0124] The prediction network is used to determine the corresponding audio data (and the sample prediction data in step a) based on the first output feature. The prediction network may include one prediction model or multiple prediction models. The prediction model included in the prediction network may be a pre-trained model or an untrained model. When the prediction model is an untrained model, the prediction model may be subsequently subjected to parameter update training.

[0125] Step S152: Based on the principle of reducing the difference between the sample prediction data and the sample audio data, the parameters of the adjusted first model are adjusted to obtain an audio processing model.

[0126] Specifically, when the difference between the sample prediction data and the sample audio data satisfies a third convergence condition, it can be determined that the adjustment of the parameters of the adjusted first model is complete, thereby obtaining the audio processing model. The third convergence condition can be, for example, that the difference between the sample prediction data and the corresponding sample audio data is less than a third preset threshold, or that, for multiple sample audio data, the ratio of the differences between the sample prediction data and the sample audio data that are less than the third preset threshold reaches a third preset ratio threshold. The third preset ratio threshold can be the same as or different from the first preset ratio threshold, and is not specifically limited in this disclosure.

[0127] This embodiment predicts the first output feature through a prediction network, and further adjusts the parameters of the adjusted first model according to the accuracy of the prediction result, which can improve the accuracy of the obtained audio processing model.

[0128] Optionally, the method may further include the following steps: inputting the second output feature into the prediction network to obtain second prediction data corresponding to the sample audio data; and adjusting parameters of the second model based on the principle of reducing the difference between the second sample prediction data and the sample audio data to obtain an adjusted second model. The adjusted second model has more accurate feature extraction and higher accuracy in subsequent non-streaming audio conversion.

[0129] Optionally, the prediction network may include a first prediction model and a second prediction model, wherein the first prediction model is used to predict data of a next frame based on data of a current frame, and the second prediction model is used to select data of a next frame from multiple frame data.

[0130] Specifically, as shown in Figure 2, the above-mentioned prediction network can be a hybrid predictive coding (Hybrid Predictive Coding, HPC), the above-mentioned first prediction model can be an autoregressive predictive coding (Autoregressive Predictive Coding, APC), and the above-mentioned second prediction model can be a contrastive predictive coding (CPC). The contrastive predictive coding and the autoregressive predictive coding respectively use a network with an autoregressive structure (g-net).

[0131] CPC learns representations by classifying real future frames and randomly selected negative examples. Contrastive Predictive Encoders use prior knowledge to select negative examples, enabling more targeted representation learning for specific tasks. Contrastive Predictive Encoders are a contrastive learning method. Input audio passes through the encoder to generate a hidden representation, which is then passed through an autoregressive network to generate sequence features. We select hidden representations that are closer to the target point as positive examples, while those farther away are negative examples.

[0132] APC is a generative model that directly predicts future frames. Given an input sequence, it predicts the spectrum of the future time period. APC's direct prediction approach is unaffected by the negative sample selection method and has better generalization than CPC. Combining the contrastive predictive encoder and the autoregressive predictive encoder allows for even better generalization of the output of the hybrid predictive encoder.

[0133] Step a can be implemented by following steps a-1 and a-2. Step S152 can be implemented by following steps S152a.

[0134] Step a-1: Input the above-mentioned first output feature into the first prediction model to obtain first sample prediction data corresponding to the sample audio data.

[0135] Step a-2: Input the above-mentioned first output feature into the second prediction model to obtain second sample prediction data corresponding to the sample audio data.

[0136] Step S152a: Based on the principle of reducing the difference between the first sample prediction data, the second sample prediction data and the sample audio data, adjusting the parameters of the adjusted first model.

[0137] For example, taking the first output feature as an example, as shown in FIG3 , the Z sequence is the first output feature of the input, and the first output feature is input to the autoregressive prediction encoder of the hybrid predictive encoder to obtain the first prediction sample data, wherein the first sample prediction data is sequence data, that is, the first r sequence, that is, Sequence, zt represents the value of Z sequence at time t, Denotes the predicted value of the Z sequence at time t. The first output feature is input to the contrast prediction encoder of the hybrid predictive encoder to obtain the second sample prediction data, that is, the second r sequence, that is, the r sequence in FIG3 .

[0138] This embodiment uses the first prediction model and the second prediction model for prediction respectively, so as to further update the parameters of the adjusted first model according to the two prediction results. It can not only learn representations for specific tasks but also make the output results more generalizable, so that the audio processing model has better generalization and feature processing is more accurate.

[0139] The second output feature can also be input into the first prediction model and the second prediction model respectively to obtain prediction data. The specific process refers to the first output feature and will not be described in detail here.

[0140] In one embodiment, the above step A can be implemented according to the following steps A-1 and A-2.

[0141] Step A-1: ​​Obtaining pitch features of sample audio data;

[0142] Step A-2: Input the sample timbre feature, the first output feature, and the pitch feature into a first fusion model to obtain first fused audio data.

[0143] Pitch features can be extracted using a pre-trained pitch extraction model, which can be trained using a relevant machine learning algorithm and will not be described in detail in this disclosure. Pitch represents the height of the speaker's voice, and the pitch of a speaker at different times is typically different. By fusing pitches, a more accurate fused audio can be obtained, resulting in a more accurate feature processing model when applied to audio fusion scenarios.

[0144] The second embodiment of the present disclosure also provides an audio conversion method. The executor of the audio conversion method provided by the present disclosure can be an electronic device, which can be a server, a desktop computer, a laptop computer, a tablet computer, a mobile phone, a smart watch, a smart TV, etc., or other electronic devices with data processing functions, which are not specifically limited by the present disclosure.

[0145] The audio conversion method provided by this disclosure can be used to convert audio such as singing and speaking in live broadcasts and dubbing scenarios, and the specific application scenarios are not limited. Specifically, it can be used to convert real-time streaming audio when converting singing, and it can also be used to convert non-streaming audio.

[0146] As shown in FIG4 , the audio conversion method provided by the present disclosure includes the following steps S210 to S240 .

[0147] Step S210: Acquire the source audio to be converted and the timbre characteristics of the target speaker.

[0148] The source audio may be singing, speech audio, audio lines from a film or TV series or other videos, or other forms of audio.

[0149] In step S210, the timbre characteristics of the source audio to be converted and the target speaker can be determined based on the audio conversion request received from the client, wherein the audio conversion request can include audio information of the source audio to be converted and timbre information of the target speaker. The audio information of the source audio can be specific audio content or an identifier corresponding to the audio. The electronic device can search the audio library for the corresponding audio as the source audio based on the identifier corresponding to the audio. The timbre information can be specific timbre characteristics or a timbre identifier corresponding to the timbre. The electronic device can also determine the corresponding timbre characteristics based on the timbre identifier.

[0150] Step S220: extracting pronunciation features of the source audio.

[0151] The pronunciation feature is a feature of the source audio after at least the timbre information of the source speaker is filtered out.

[0152] The process of extracting pronunciation features in this step can refer to the method of extracting sample pronunciation features in step S110 in the first embodiment, and will not be described in detail here.

[0153] Step S230: Input the pronunciation feature into a pre-trained audio processing model to obtain a first audio extraction feature corresponding to the source audio.

[0154] The audio processing model is trained according to the audio processing model training method described in any one of the first embodiments.

[0155] Step S240: Fusing the first audio extraction feature with the timbre information to obtain converted audio corresponding to the target speaker.

[0156] The fusion method of step S240 can refer to the specific execution process of step S140b in the first embodiment, which will not be described in detail here.

[0157] It can be seen from the first embodiment that since the audio processing model that can perceive future information well is used to extract the occurrence characteristics of the source audio, the converted audio obtained by fusing the extracted first audio extraction features with the timbre of the target speaker is also a converted audio based on the perception of future information. Therefore, the converted audio has higher accuracy, better intelligibility and sound quality, more stable fundamental frequency, and better conversion effect.

[0158] Optionally, step S240 may specifically obtain the converted audio corresponding to the target speaker by the following steps: inputting the first audio extraction feature and the timbre information into a pre-trained first fusion model to obtain the converted audio corresponding to the target speaker, wherein the first fusion model is trained by training the adjusted first fusion model in the audio processing model training method described in the first embodiment. As can be seen from the first embodiment, since the first fusion model processes data based on past information, it is more suitable for fusing streaming audio data and has higher fusion accuracy.

[0159] Optionally, in step S230 , the pronunciation features are input into a pre-trained audio processing model, which can be specifically implemented by following the steps S231 .

[0160] Step S231: When the source audio is a real-time audio stream, the pronunciation features are input into a pre-trained audio processing model.

[0161] The audio conversion method may further include the following steps S250 to S260.

[0162] Step S250: When the source audio is a non-real-time audio stream, the pronunciation feature is input into the second model to obtain a second audio extraction feature corresponding to the source audio. The second model is trained according to the method of training the adjusted second model in the training method of the audio processing model described in the first embodiment.

[0163] Step S260: Fusing the second audio extraction feature with the timbre information to obtain converted audio corresponding to the target speaker.

[0164] As can be seen from the first embodiment, the audio processing model is obtained by training models such as causal convolution models and unidirectional neural network models that process data based on past information. Therefore, the audio processing model is more suitable for processing streaming audio, and the processing results are more accurate, thereby making the fusion results more accurate. For non-real-time audio streams, that is, non-streaming audio, this embodiment uses a second model for feature processing. Since the second model is trained using the method of training and adjusting the second model in the first embodiment, as can be seen from the first embodiment, the second model processes data based on past and future information. Therefore, the second model processes non-streaming audio more accurately, and the fusion results are also more accurate.

[0165] Optionally, the step S260 may be implemented according to the following step S261.

[0166] Step S261: Input the second audio extraction features and the timbre information into a pre-trained second fusion model to obtain converted audio corresponding to the target speaker. The second fusion model is trained using the method for training the adjusted second fusion model in the audio processing model training method described in the first embodiment. As can be seen from the first embodiment, because the second fusion model fuses data based on past and future information, it is more suitable for fusing non-streaming data and achieves higher fusion accuracy.

[0167] The third embodiment of the present disclosure further provides a training device for an audio processing model, as shown in FIG5 , the device comprising:

[0168] The first determining unit 310 is configured to determine the sample pronunciation features corresponding to each audio segment of the sample audio data;

[0169] The first extraction unit 320 is configured to input the sample pronunciation feature into a first model to be trained for numerical encoding to obtain a first numerical output feature corresponding to the sample pronunciation feature, wherein the first model to be trained is a model that numerically encodes the pronunciation feature based on past information, and the past information is a past pronunciation feature corresponding to a past audio segment generated before the audio segment corresponding to the sample pronunciation feature; input the sample pronunciation feature into a second model for numerical encoding to obtain a second numerical output feature corresponding to the sample pronunciation feature, wherein the second model is a model that numerically encodes the pronunciation feature based on the past information and future information, and the future information is a future pronunciation feature corresponding to a future audio segment generated after the audio segment corresponding to the sample pronunciation feature;

[0170] An adjusting unit 330 is configured to perform parameter adjustment on the first to-be-trained model based on a first parameter adjustment principle that reduces a difference between the first output feature and the second output feature, thereby obtaining an adjusted first model;

[0171] The second determining unit 340 is configured to determine an audio processing model according to the adjusted first model.

[0172] Optionally, the first model to be trained is a causal convolutional model, and the second model is a non-causal convolutional model.

[0173] Optionally, the first determining unit 310 is further configured to perform: determining a sample timbre feature of the sample audio data;

[0174] The device further comprises:

[0175] a first fusion unit configured to fuse the sample timbre feature with the first output feature to obtain fused audio data;

[0176] The second determining unit 340 is specifically configured to perform: based on a second parameter adjustment principle of reducing the difference between the fused audio data and the sample audio data, adjusting the parameters of the adjusted first model to obtain an audio processing model.

[0177] Optionally, the adjusting unit 330 is further configured to perform: adjusting parameters of the second model based on the second parameter adjustment principle to obtain an adjusted second model.

[0178] Optionally, the first fusion unit is specifically configured to perform: inputting the sample timbre feature and the first output feature into a first fusion model to obtain first fused audio data;

[0179] The adjustment unit is further configured to perform: based on the principle of reducing the difference between the first fused audio data and the sample audio data, adjusting parameters of the first fusion model to obtain an adjusted first fusion model.

[0180] Optionally, the first fusion model is a model that fuses data based on past first fusion information, and the past first fusion information includes a first output feature corresponding to the past pronunciation feature.

[0181] Optionally, the first fusion unit is further configured to execute: inputting the sample timbre feature and the second output feature into a second fusion model to obtain second fused audio data, wherein the second fusion model is a model that fuses data based on past second fusion information and future second fusion information, the past second fusion information includes the second output feature corresponding to the past pronunciation feature, and the future second fusion information includes the second output feature corresponding to the future pronunciation feature;

[0182] The adjustment unit is further configured to perform: based on the principle of reducing the difference between the second fused audio data and the sample audio data, adjusting parameters of the second fusion model to obtain an adjusted second fusion model.

[0183] Optionally, the apparatus further comprises: a prediction unit configured to input the first output feature into a prediction network to obtain sample prediction data corresponding to the sample audio data;

[0184] The second determining unit is specifically configured to perform: based on the principle of reducing the difference between the sample prediction data and the sample audio data, adjusting the parameters of the adjusted first model to obtain an audio processing model.

[0185] Optionally, the prediction network includes a first prediction model and a second prediction model, the first prediction model is used to predict data of the next frame based on data of the current frame, and the second prediction model is used to select data of the next frame from multiple frames of data;

[0186] The prediction unit is specifically configured to execute: inputting the first output feature into the first prediction model to obtain first sample prediction data corresponding to the sample audio data; inputting the first output feature into the second prediction model to obtain second sample prediction data corresponding to the sample audio data;

[0187] The second determining unit is specifically configured to perform: adjusting parameters of the adjusted first model based on the principle of reducing the difference between the first sample prediction data, the second sample prediction data and the sample audio data.

[0188] Optionally, the first fusion unit is specifically configured to perform: obtaining pitch features of sample audio data; inputting the sample timbre features, the first output features and the pitch features into a first fusion model to obtain first fused audio data.

[0189] Optionally, the first determining unit is specifically configured to perform: converting each audio segment of the sample audio data into sample Melp spectrum data;

[0190] The melep spectrum data is input into a pre-trained speech recognition model to obtain bottleneck features corresponding to each audio segment of the sample audio data, and the bottleneck features are determined as sample pronunciation features corresponding to each audio segment of the sample audio data. The speech recognition model is used to identify the pronunciation features of the audio data.

[0191] A fourth embodiment of the present disclosure further provides an audio conversion device, the device comprising:

[0192] An acquisition unit configured to acquire source audio to be converted and timbre information of a target speaker;

[0193] A second extraction unit is configured to extract pronunciation features of the source audio, where the pronunciation features are features of the source audio after at least timbre information of a source speaker is filtered out;

[0194] a processing unit configured to input the pronunciation feature into a pre-trained audio processing model to obtain a first audio extraction feature corresponding to the source audio, wherein the audio processing model is trained according to the audio processing model training method described in any one of the first embodiments;

[0195] The fusion unit is configured to perform fusion of the first audio extraction feature and the timbre information to obtain converted audio corresponding to the target speaker.

[0196] Optionally, the fusion unit is specifically configured to perform:

[0197] The first audio extraction feature and the timbre feature are input into a pre-trained first fusion model to obtain the converted audio corresponding to the target speaker. The first fusion model is trained by training the adjusted first fusion model in the training method of the audio processing model described in the first embodiment.

[0198] Optionally, the processing unit is specifically configured to perform: when the source audio is a real-time audio stream, inputting the pronunciation features into a pre-trained audio processing model;

[0199] The processing unit is also configured to perform: when the source audio is a non-real-time audio stream, inputting the pronunciation feature into a second model to obtain a second audio extraction feature corresponding to the source audio, the second model being trained according to the training method of the audio processing model described in the first embodiment after training the adjusted second model; fusing the second audio extraction feature with the timbre information to obtain the converted audio corresponding to the target speaker.

[0200] Optionally, the fusion unit is specifically configured to perform: inputting the second audio extraction feature and the timbre information into a pre-trained second fusion model to obtain the converted audio corresponding to the target speaker, and the second fusion model is trained according to the method of training the adjusted second fusion model in the training method of the audio processing model described in the first embodiment.

[0201] It should be noted that the information interaction, execution process, etc. between the modules / units in the device are based on the same concept as the various method embodiments corresponding to Figures 1 to 3 in this disclosure. For specific contents, please refer to the description in the method embodiments shown above in this disclosure, and will not be repeated here.

[0202] Next, an electronic device provided by an embodiment of the present disclosure is introduced. Please refer to Figure 6. Figure 6 is a structural schematic diagram of an electronic device provided by an embodiment of the present disclosure. The electronic device 500 can be specifically manifested as a virtual reality VR device, a mobile phone, a tablet, a laptop computer, a smart wearable device, a monitoring data processing device or a radar data processing device, etc., which is not limited here. Among them, the electronic device 500 can be deployed with the training device or conversion device described in the corresponding embodiment of Figure 1 or Figure 4 to implement the functions in the corresponding embodiments of Figures 1 to 4. Specifically, the electronic device 500 includes: a receiver 501, a transmitter 502, a processor 503 and a memory 504 (wherein the number of processors 503 in the execution device 500 can be one or more, and Figure 6 takes one processor as an example), wherein the processor 503 may include an application processor 5031 and a communication processor 5032. In some embodiments of the present disclosure, the receiver 501, the transmitter 502, the processor 503 and the memory 504 can be connected via a bus or other means.

[0203] The memory 504 may include a read-only memory and a random access memory, and provides instructions and data to the processor 503. A portion of the memory 504 may also include non-volatile random access memory (NVRAM). The memory 504 stores processor and operation instructions, executable modules, or data structures, or subsets or extended sets thereof. The operation instructions may include various operation instructions for implementing various operations.

[0204] Processor 503 controls the operation of the execution device. In specific applications, the various components of the execution device are coupled together via a bus system. In addition to a data bus, the bus system may also include a power bus, a control bus, and a status signal bus. However, for clarity, all of these buses are referred to as a bus system in the figure.

[0205] The methods disclosed in the above embodiments of the present disclosure can be applied to or implemented by the processor 503. The processor 503 can be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by hardware integrated logic circuits in the processor 503 or by software instructions. The above processor 503 can be a general-purpose processor, a digital signal processor (DSP), a microprocessor, or a microcontroller, and can further include an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The processor 503 can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of the present disclosure. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in conjunction with the embodiments of the present disclosure can be directly implemented and executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium well-known in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. The storage medium is located in memory 504, and processor 503 reads the information in memory 504 and performs the steps of the above method in conjunction with its hardware.

[0206] Receiver 501 can be used to receive input digital or character information and generate signal input related to executing device-related settings and function control. Transmitter 502 can be used to output digital or character information through the first interface. Transmitter 502 can also be used to send instructions to the disk pack through the first interface to modify data in the disk pack. Transmitter 502 can also include a display device such as a display screen.

[0207] In the embodiment of the present disclosure, the application processor 5031 in the processor 503 is used to execute the training method or conversion method in the embodiments corresponding to Figures 1 to 4. It should be noted that the specific manner in which the application processor 5031 executes each step is based on the same concept as the various method embodiments corresponding to Figures 1 to 4 in the present disclosure, and the technical effects brought about are the same as the various method embodiments corresponding to Figures 1 to 4 in the present disclosure. For specific details, please refer to the description of the method embodiments shown above in the present disclosure, and no further details will be given here.

[0208] An embodiment of the present disclosure further provides a computer-readable storage medium, which includes computer instructions. When the computer instructions are executed by a processor, they are used to implement any one of the technical solutions in the first embodiment or the second embodiment of the present disclosure.

[0209] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0210] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0211] Computer-readable media include permanent and non-permanent, removable and non-removable media that can be used to store information using any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include non-transitory media such as modulated data signals and carrier waves.

[0212] Those skilled in the art will appreciate that embodiments of the present disclosure may be provided as methods, systems, or computer program products. Thus, the present disclosure may take the form of a fully hardware embodiment, a fully software embodiment, or an embodiment combining software and hardware. Furthermore, the present disclosure may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0213] Although the present disclosure is disclosed as above in terms of preferred embodiments, it is not intended to limit the present disclosure. Any person skilled in the art may make possible changes and modifications without departing from the spirit and scope of the present disclosure. Therefore, the scope of protection of the present disclosure shall be based on the scope defined by the claims of the present disclosure.

Claims

1. A training method for an audio processing model, the method comprising: Determine sample pronunciation features corresponding to each audio segment of the sample audio data; Inputting the sample pronunciation feature into a first model to be trained for numerical encoding to obtain a numerical first output feature corresponding to the sample pronunciation feature, wherein the first model to be trained is a model that numerically encodes the pronunciation feature based on past information, wherein the past information is a past pronunciation feature corresponding to a past audio segment generated before the audio segment corresponding to the sample pronunciation feature; Inputting the sample pronunciation feature into a second model for numerical encoding to obtain a numerical second output feature corresponding to the sample pronunciation feature, wherein the second model is a model for numerically encoding the pronunciation feature based on the past information and future information, and the future information is a future pronunciation feature corresponding to a future audio segment generated after the audio segment corresponding to the sample pronunciation feature; Based on a first parameter adjustment principle of reducing the difference between the first output feature and the second output feature, adjusting the parameters of the first model to be trained to obtain an adjusted first model; An audio processing model is determined according to the adjusted first model.

2. The method according to claim 1, wherein: The first model to be trained is a causal convolutional model, and the second model is a non-causal convolutional model.

3. The method according to claim 1, wherein: Before determining the trained audio processing model according to the adjusted first model, the method further includes: Determining sample timbre characteristics of the sample audio data; Fusion the sample timbre feature with the first output feature to obtain fused audio data; The step of determining the audio processing model according to the adjusted first model comprises: Based on the second parameter adjustment principle of reducing the difference between the fused audio data and the sample audio data, the parameters of the adjusted first model are adjusted to obtain an audio processing model.

4. The method according to claim 3, wherein: The method further comprises: The parameters of the second model are adjusted based on the second parameter adjustment principle to obtain an adjusted second model.

5. The method according to claim 3, wherein: The step of fusing the sample timbre feature with the first output feature to obtain fused audio data includes: Inputting the sample timbre feature and the first output feature into a first fusion model to obtain first fused audio data; The method further comprises: Based on the principle of reducing the difference between the first fused audio data and the sample audio data, the parameters of the first fusion model are adjusted to obtain an adjusted first fusion model.

6. The method according to claim 5, wherein: The first fusion model is a model for fusing data based on past first fusion information, and the past first fusion information includes a first output feature corresponding to the past pronunciation feature.

7. The method according to claim 6, wherein: The method further comprises: Inputting the sample timbre feature and the second output feature into a second fusion model to obtain second fused audio data, wherein the second fusion model is a model for fusing data based on past second fusion information and future second fusion information, wherein the past second fusion information includes the second output feature corresponding to the past pronunciation feature, and the future second fusion information includes the second output feature corresponding to the future pronunciation feature; Based on the principle of reducing the difference between the second fused audio data and the sample audio data, the parameters of the second fusion model are adjusted to obtain an adjusted second fusion model.

8. The method according to claim 1, wherein: Before determining the audio processing model according to the adjusted first model, the method further includes: Inputting the first output feature and the second output feature into a prediction network to obtain sample prediction data corresponding to the sample audio data; The step of determining the audio processing model according to the adjusted first model comprises: Based on the principle of reducing the difference between the sample prediction data and the sample audio data, the parameters of the adjusted first model are adjusted to obtain an audio processing model.

9. The method according to claim 8, wherein: The prediction network includes a first prediction model and a second prediction model, the first prediction model is used to predict the data of the next frame according to the data of the current frame, and the second prediction model is used to select the data of the next frame from the multiple frame data; The step of inputting the first output feature and the second output feature into a prediction network to obtain sample prediction data corresponding to the sample audio data includes: Inputting the first output feature and the second output feature into the first prediction model to obtain first sample prediction data corresponding to the sample audio data; Inputting the first output feature and the second output feature into the second prediction model to obtain second sample prediction data corresponding to the sample audio data; The step of adjusting parameters of the adjusted first model based on the principle of reducing the difference between the sample prediction data and the sample audio data comprises: Based on the principle of reducing the difference between the first sample prediction data, the second sample prediction data and the sample audio data, the parameters of the adjusted first model are adjusted.

10. The method according to claim 5, wherein: The step of inputting the sample timbre feature and the first output feature into a first fusion model to obtain first fused audio data includes: Get the pitch characteristics of sample audio data; The sample timbre feature, the first output feature and the pitch feature are input into a first fusion model to obtain first fused audio data.

11. The method according to any one of claims 1 to 10, wherein: The step of determining the sample pronunciation features corresponding to each audio segment of the sample audio data includes: Convert each audio segment of the sample audio data into sample Melp spectrum data; The Melp spectrum data is input into a pre-trained speech recognition model to obtain bottleneck features corresponding to each audio segment of the sample audio data, and the bottleneck features are determined as sample pronunciation features corresponding to each audio segment of the sample audio data. The speech recognition model is used to identify the pronunciation features of the audio data.

12. An audio conversion method, the method comprising: Obtain source audio to be converted and timbre information of the target speaker; Extracting pronunciation features of the source audio, wherein the pronunciation features are features of the source audio after at least the timbre information of the source speaker is filtered out; Inputting the pronunciation feature into a pre-trained audio processing model to obtain a first audio extraction feature corresponding to the source audio, wherein the audio processing model is trained according to the training method of the audio processing model according to any one of claims 1 to 11; The first audio extraction feature is fused with the timbre information to obtain converted audio corresponding to the target speaker.

13. The audio conversion method according to claim 12, wherein: The step of fusing the first audio extraction feature with the timbre information to obtain converted audio corresponding to the target speaker includes: The first audio extraction feature and the timbre information are input into a pre-trained first fusion model to obtain the converted audio corresponding to the target speaker, and the first fusion model is trained by training the adjusted first fusion model in the training method of the audio processing model according to claim 5 or 6.

14. The audio conversion method according to claim 12, wherein: The step of inputting the pronunciation features into a pre-trained audio processing model comprises: When the source audio is a real-time audio stream, inputting the pronunciation features into a pre-trained audio processing model; The audio conversion method further includes: When the source audio is a non-real-time audio stream, the pronunciation feature is input into the second model to obtain a second audio extraction feature corresponding to the source audio, and the second model is trained by training the adjusted second model in the training method of the audio processing model according to claim 4; The second audio extraction feature is fused with the timbre information to obtain converted audio corresponding to the target speaker.

15. The audio conversion method according to claim 14, wherein: The step of fusing the second audio extraction feature with the timbre information to obtain converted audio corresponding to the target speaker includes: The second audio extraction features and the timbre information are input into a pre-trained second fusion model to obtain the converted audio corresponding to the target speaker, and the second fusion model is trained by training the adjusted second fusion model in the training method of the audio processing model according to claim 7.

16. A training device for an audio processing model, comprising: A first determining unit is configured to determine a sample pronunciation feature corresponding to each audio segment of the sample audio data; The first extraction unit is configured to perform numerical encoding of the sample pronunciation feature into a first model to be trained, and obtain a numerical first output feature corresponding to the sample pronunciation feature, wherein the first model to be trained is based on A model for numerically encoding pronunciation features based on past information, wherein the past information is a past pronunciation feature corresponding to a past audio segment generated before the audio segment corresponding to the sample pronunciation feature; the sample pronunciation feature is input into a second model for numerical encoding to obtain a numerical second output feature corresponding to the sample pronunciation feature, wherein the second model is a model for numerically encoding pronunciation features based on the past information and future information, wherein the future information is a future pronunciation feature corresponding to a future audio segment generated after the audio segment corresponding to the sample pronunciation feature; an adjusting unit, configured to perform parameter adjustment on the first model to be trained based on a first parameter adjustment principle to reduce a difference between the first output feature and the second output feature, so as to obtain an adjusted first model; The second determining unit is used to determine the audio processing model according to the adjusted first model.

17. An audio conversion device, comprising: An acquisition unit, configured to acquire source audio to be converted and timbre information of a target speaker; A second extraction unit is configured to extract a pronunciation feature of the source audio, wherein the pronunciation feature is a feature of the source audio after at least the timbre information of the source speaker is filtered out; A processing unit, configured to input the pronunciation feature into a pre-trained audio processing model to obtain a first audio extraction feature corresponding to the source audio, wherein the audio processing model is trained according to the training method of the audio processing model according to any one of claims 1 to 11; The fusion unit is configured to perform fusion of the first audio extraction feature and the timbre information to obtain a converted audio corresponding to the target speaker.

18. An electronic device, comprising: Memory and processor; The memory is coupled to the processor; The memory is used to store one or more computer instructions; The processor is configured to execute the one or more computer instructions to implement the method according to any one of claims 1-15.

19. A computer-readable storage medium having one or more computer instructions stored thereon, the instructions being executed by a processor to implement the method according to any one of claims 1 to 15.

Citation Information

Patent Citations

  • Speech recognition method

    CN113327590A

  • Neural network training method and device, speech recognition method and device, equipment and medium

    CN115331660A

  • Model training method, voice conversion method and device, equipment and storage medium

    CN115641860A

  • Audio processing model training method, audio conversion method and device

    CN116631421A

  • Custom tone and vocal synthesis method and apparatus, electronic device, and storage medium

    WO2022156479A1

Cited By

  • Audio processing method and device

    CN120510853A