Training method for audio processing model, and method and apparatus for audio conversion

Abstract

A training method for an audio processing model, a method and apparatus for audio conversion, an electronic device, and a computer-readable storage medium. The training method comprises: determining a sample pronunciation feature of sample audio data (S110); and inputting the sample pronunciation feature into a first model to be trained for numericalization coding to obtain a numericalized first output feature, the first model to be trained being a model for performing numericalization coding on data on the basis of past information (S120); inputting the sample pronunciation feature into a second model for numericalization coding to obtain a numericalized second output feature, the second model being a model for processing data on the basis of past and future information (S130); on the basis of a first parameter adjustment principle of reducing the difference between the first output feature and the second output feature, performing parameter adjustment on the first model to be trained to obtain an adjusted first model (S140); and determining an audio processing model according to the adjusted first model (S150). The method enables stream song conversion to achieve stream reasoning while ensuring the effect of stream reasoning.

Description

Method for training audio processing model, audio conversion method and device

[0001] Cross-reference to related applications

[0002] This application claims priority to Chinese Patent Application No. 202310576603.5, filed May 19, 2023, entitled "Method for Training Audio Processing Model, Audio Conversion Method and Device", the entire contents of which are incorporated herein by reference in its entirety. TECHNICAL FIELD

[0003] The present disclosure relates to the technical field of computer, and particularly relates to a method for training an audio processing model, an audio conversion method, a device, an electronic device and a computer readable storage medium. BACKGROUND

[0004] Song conversion is a subset of speech conversion, which refers to converting a singer's singing audio into another person's voice, and can be applied to online live broadcast, film and television dubbing and other fields. Streaming song conversion can return recognition results in real time during audio processing. Compared with the whole segment conversion mode, the streaming conversion model can be applied to application scenarios with real-time requirements.

[0005] Due to the real-time nature of streaming songs, the song information of the streaming song being converted at the future time cannot be obtained during conversion. Therefore, the existing streaming song conversion mode usually performs song processing based on a model that does not rely on future information input, for example, song processing during song conversion by a unidirectional recurrent neural network, a causal convolution model, etc.

[0006] However, since the song processing model in the song conversion network cannot perceive future information, the conversion accuracy of the song conversion network is low, which can result in poor effects of the converted song.

[0007] SUMMARY

[0008] Therefore, the present disclosure provides a method for training an audio processing model, an audio conversion method, a device, an electronic device and a computer readable storage medium, which can improve the accuracy of song conversion and make the converted song better.

[0009] A first aspect of an embodiment of the present disclosure provides a method for training an audio processing model, the method comprising:

[0010] determining sample pronunciation features corresponding to each audio segment of sample audio data;

[0011] input the sample pronunciation feature into a first to-be-trained model for numerical coding to obtain a first output feature corresponding to the sample pronunciation feature after numerical coding, the first to-be-trained model being a model for numerical coding of pronunciation features based on past information, the past information being past pronunciation features corresponding to past audio segments generated before the audio segment;

[0012] input the sample pronunciation feature into a second model for numerical coding to obtain a second output feature corresponding to the sample pronunciation feature after numerical coding, the second model being a model for numerical coding of pronunciation features based on the past information and future information, the future information being future pronunciation features corresponding to future audio segments generated after the audio segment;

[0013] perform parameter adjustment on the first to-be-trained model based on a first parameter adjustment principle for reducing the difference between the first output feature and the second output feature to obtain an adjusted first model;

[0014] determine an audio processing model according to the adjusted first model.

[0015] A second aspect of the embodiments of the present disclosure provides an audio conversion method, and the method comprises:

[0016] obtain source audio to be converted and timbre information of a target sound producer;

[0017] extract a pronunciation feature of the source audio, the pronunciation feature being a feature of the source audio after filtering out at least the timbre information of a source sound producer;

[0018] input the pronunciation feature into a pre-trained audio processing model to obtain a first audio extraction feature corresponding to the source audio, the audio processing model being trained according to the training method of the audio processing model in any one of the first aspect;

[0019] fuse the first audio extraction feature and the timbre information to obtain converted audio corresponding to the target sound producer.

[0020] A third aspect of the embodiments of the present disclosure provides a training device of an audio processing model, and the training device comprises:

[0021] a first determination unit configured to determine sample pronunciation features corresponding to each audio segment of sample audio data;

[0022] The first extraction unit is configured to perform numerical coding of the sample pronunciation feature by inputting the sample pronunciation feature into a first to-be-trained model, to obtain a first output feature corresponding to the sample pronunciation feature after numerical coding, the first to-be-trained model being a model for numerical coding of a pronunciation feature based on past information, the past information being past pronunciation features corresponding to past audio segments generated before the audio segment; and perform numerical coding of the sample pronunciation feature by inputting the sample pronunciation feature into a second model, to obtain a second output feature corresponding to the sample pronunciation feature after numerical coding, the second model being a model for numerical coding of a pronunciation feature based on the past information and future information, the future information being future pronunciation features corresponding to future audio segments generated after the audio segment.

[0023] The adjusting unit is configured to perform parameter adjustment on the first to-be-trained model based on a first parameter adjustment principle of reducing the difference between the first output feature and the second output feature, to obtain an adjusted first model.

[0024] The second determining unit is configured to determine an audio processing model according to the adjusted first model.

[0025] The fourth aspect of the embodiments of the present disclosure provides an audio conversion device, comprising:

[0026] The obtaining unit is configured to obtain source audio to be converted and timbre information of a target sound producer.

[0027] The second extraction unit is configured to extract a pronunciation feature of the source audio, the pronunciation feature being a feature of the source audio after filtering out at least the timbre information of the source sound producer.

[0028] The processing unit is configured to input the pronunciation feature into a pre-trained audio processing model, to obtain a first audio extraction feature corresponding to the source audio, the audio processing model being trained according to the training method of the audio processing model of any one of the first aspect.

[0029] The fusion unit is configured to fuse the first audio extraction feature and the timbre information, to obtain converted audio corresponding to the target sound producer.

[0030] The fifth aspect of the embodiments of the present disclosure provides an electronic device, comprising a memory and a processor, the memory and the processor being coupled.

[0031] The memory is configured to store one or more computer instructions.

[0032] The processor is configured to execute the one or more computer instructions to implement the method of the first aspect or the second aspect.

[0033] The sixth aspect of the embodiments of the present disclosure further provides a computer readable storage medium, which stores one or more computer instructions, and the instructions are executed by a processor to implement the method in the first aspect or the second aspect.

[0034] Compared with the related art, the embodiments of the present disclosure have the following advantages:

[0035] The training method of the audio processing model provided by the embodiments of the present disclosure, after determining the sample pronunciation features corresponding to each audio segment of the sample audio data, inputs the sample pronunciation features into the first to-be-trained model to obtain the first output features after numerical coding, and inputs the sample pronunciation features into the second model to obtain the second output features after numerical coding. Since the first to-be-trained model is a model for numerical coding of pronunciation features based on past information, the past information is past pronunciation features corresponding to past audio segments generated before the audio segment, and the second model is a model for numerical coding of pronunciation features based on past and future information, the future information is future pronunciation features corresponding to future audio segments generated after the audio segment, that is, the first to-be-trained model cannot perceive future information when performing numerical coding, and the second model can perceive future information when performing numerical coding. Therefore, the second output features are features obtained on the basis of perceiving past and future information, and the first output features are features without perceiving future information. The embodiments of the present disclosure adjust the parameters of the first to-be-trained model based on the first parameter adjustment principle of reducing the difference between the first output features and the second output features. In this way, the output result of the adjusted first model can be more consistent with the output result of the model that can perceive future information. That is, the present disclosure distills the first to-be-trained model that cannot perceive future information based on the output result of the second model that can perceive future information. In this way, the adjusted first model obtained by distillation does not use future information when performing numerical coding, but the result of the adjusted first model when performing numerical coding is also able to perceive future information well. Therefore, the audio processing model determined by the adjusted first model can also perceive future information well.

[0036] When performing audio conversion, since the vocal feature of the source audio is extracted by using the audio processing model that can perceive future information well, the converted audio obtained by fusing the first audio extraction feature and the timbre of the target performer is also a converted audio based on perceiving future information. Therefore, the converted audio is more accurate, has better intelligibility and sound quality, has a more stable fundamental frequency, and has better conversion effect. BRIEF DESCRIPTION OF DRAWINGS

[0037] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the following will briefly introduce the drawings needed to be used in the description of the embodiments of the present disclosure. Obviously, the drawings in the following description are only some embodiments of the present disclosure, and for those skilled in the art, other drawings can also be obtained from these drawings without creative labor.

[0038] FIG. 1 is a flow diagram of an example of a method for training an audio processing model according to an embodiment of the present disclosure;

[0039] FIG. 2 is a flow diagram of another example of a method for training an audio processing model according to an embodiment of the present disclosure;

[0040] FIG. 3 is a structural diagram of a prediction network according to an embodiment of the present disclosure;

[0041] FIG. 4 is a flow diagram of an audio conversion method according to an embodiment of the present disclosure;

[0042] FIG. 5 is a structural diagram of a training device for an audio processing model according to an embodiment of the present disclosure;

[0043] FIG. 6 is a structural diagram of an electronic device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0044] In order to enable those skilled in the art to better understand the technical solutions of the present disclosure, the present disclosure will be described in detail below with reference to the drawings in the embodiments of the present disclosure. However, the present disclosure can be implemented in many different ways other than the above description, and therefore, all other embodiments obtained by those skilled in the art based on the embodiments provided in the present disclosure without creative labor should be within the scope of protection of the present disclosure.

[0045] It should be noted that the terms "first", "second", "third", etc. in the claims, description and drawings of the present disclosure are used to distinguish similar objects and do not necessarily describe a specific order or sequence. The data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present disclosure described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include", "have" and their variants are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device that includes a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but can include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0046] Song conversion is a subset of voice conversion, which refers to converting the singing audio of one speaker into the voice of another person, and can be applied to online live broadcast, film and television dubbing and other fields. The current mainstream song conversion method is whole segment conversion, that is, the whole segment audio is input into a non-streaming song conversion model, and the non-streaming song conversion model processes the whole segment audio and then outputs the converted audio. However, since the non-streaming song conversion model needs to wait for the whole segment audio processing to be completed before returning the result, it cannot be applied to application scenarios with real-time requirements.

[0047] Streaming song conversion can return the recognition result in real time during the processing of the audio. Compared with the whole segment conversion method, the streaming conversion model can be applied to application scenarios with real-time requirements.

[0048] A common practice for song conversion is to first extract the pronunciation features in the audio that are independent of the speaker, and then perform numerical coding processing on the pronunciation features, such as vectorization processing of the pronunciation features, dimension transformation of the vectors, etc., to process the pronunciation features into numerical information that is convenient for song conversion, thereby enriching the information content of the pronunciation features, and then fusing the processed pronunciation features with the timbre of the target sound producer to obtain a song converted into the timbre of the target sound producer.

[0049] For streaming song conversion, since the recognition result needs to be returned in real time during the processing of the audio, it is different from whole segment conversion, but needs to convert the streaming song transmitted in real time. In general, a feature processing model for streaming conversion needs to be trained to meet the real-time requirement of the application scenario of streaming song conversion.

[0050] Since streaming song has real-time performance, the song information of the streaming song being converted at a future time cannot be obtained during conversion. Therefore, the existing streaming song conversion method usually performs pronunciation feature processing of the song based on a feature processing model that does not depend on future information input, such as a unidirectional recurrent neural network, a causal convolution model, etc.

[0051] However, since the feature processing model in the song conversion network cannot perceive future information, the conversion accuracy of the song conversion network is low, which can result in poor effect of the converted song.

[0052] To improve the accuracy of song conversion and make the converted song better, the first embodiment of the present disclosure provides a method for training an audio processing model.

[0053] The execution subject of the training method of the audio processing model provided in the present disclosure can be an electronic device, which can be a server, a desktop computer, a notebook computer, a tablet computer, a mobile phone, a smart watch, a smart television, or the like, or other electronic devices with data processing functions, which are not specifically limited in the present disclosure.

[0054] The model trained by the training method of the audio processing model provided in the present disclosure can be used to process singing audio when singing conversion is performed, or can be used to process audio when other pronunciation conversion is performed, for example, to process recitation audio to convert the vocal timbre of the recitation audio. In addition to the audio conversion scene, the model trained by the training method of the audio processing model provided in the present disclosure can also be used for audio processing in other scenes, for example, processing of speech when speech recognition is performed.

[0055] FIG. 1 is a flowchart of the training method of the audio processing model provided in the embodiments of the present disclosure. It should be noted that the steps shown in the flowchart can be executed in a computer system such as a set of computer executable instructions, while in some cases, the steps disclosed can be executed in a different logical order than that shown in the flowchart.

[0056] The training method of the audio processing model provided in the present disclosure includes the following steps S110-S150.

[0057] Step S110: Determine the sample pronunciation features corresponding to each audio segment of the sample audio data.

[0058] The sample audio data can be sample singing audio data, sample recitation audio data, or other audio data containing speaker voice information. The sample audio data can be pre-recorded for training the audio processing model, or can be various audio with speaker pronunciation searched on the network, and the specific content of the sample audio data is not limited in the present disclosure.

[0059] In the embodiments of the present disclosure, the sample audio data can be divided into multiple audio segments in the order from the beginning to the end according to the production time. Specifically, the audio segments can be divided based on the audio duration, or the audio segments can be divided based on the number of pronunciations. For example, a 20-second audio can be divided into four audio segments with each 5-second audio segment. For another example, an audio including a total of 100 pronunciations can be divided into five audio segments with each 20-pronunciation audio segment, or the sample audio data can be segmented in other manners in the order from the beginning to the end. The sample pronunciation features corresponding to each audio segment of the sample audio data are determined in step S110.

[0060] The sample pronunciation feature can be understood as a feature of the sample audio data after filtering out at least the speaker timbre information, that is, the sample pronunciation feature is information irrelevant to the speaker timbre, and the sample pronunciation feature is used to represent the features of each pronunciation in the sample audio data. According to the pronunciation feature, the specific pronunciation irrelevant to the speaker timbre in the sample audio data can be determined.

[0061] The pronunciation feature can be represented in the form of text or in the form of phonemes, etc. The pronunciation feature includes the pronunciation feature of each time point or each frame in the sample audio data.

[0062] Since the audio such as singing contains not only semantics but also tone, prosody and other sound information, when training the audio processing model, in order to make the trained audio processing model not only suitable for the scene where the pronunciation tone is consistent with the tone of the text itself such as recitation, but also suitable for the scene where the pronunciation tone may not be consistent with the tone of the text itself such as singing, the model is trained based on the pronunciation feature in the embodiment of the disclosure, and the use range of the trained audio processing model is improved.

[0063] In the embodiment of the disclosure, the sample pronunciation feature of the sample audio data can be determined by a pre-trained speech recognition model. Specifically, the sample audio data can be input into the pre-trained speech recognition model to obtain the sample pronunciation feature corresponding to the sample audio data. The speech recognition model can be trained based on a conventional supervised training method, for example, the speech recognition model is trained based on an audio sample that has been set with corresponding audio features, and the specific training method is not described in detail in the disclosure.

[0064] Optionally, in step S110, the pronunciation feature can be determined in the following steps S111-S112.

[0065] Step S111: converting the sample audio data into sample mel-frequency spectrum data corresponding to each audio segment.

[0066] Step S112: inputting the mel-frequency spectrum data into a pre-trained speech recognition model to obtain the bottleneck feature corresponding to each audio segment of the sample audio data, and determining the bottleneck feature as the sample pronunciation feature corresponding to each audio segment of the sample audio data, wherein the speech recognition model is used to identify the pronunciation feature of the audio data.

[0067] Step S120: inputting the above sample pronunciation feature into a first to-be-trained model to perform numerical coding to obtain a numerical first output feature corresponding to the sample pronunciation feature, and the first to-be-trained model is a model for numerically coding the pronunciation feature based on past information.

[0068] The past information is past pronunciation features corresponding to past audio segments generated before the audio segment corresponding to the sample pronunciation feature.

[0069] In this step, each sample pronunciation feature corresponding to each audio segment of the sample audio data can be input into the first to-be-trained model for numerical encoding. When the first to-be-trained model performs numerical encoding on a target sample pronunciation feature in the sample pronunciation features, the target sample pronunciation feature is numerically encoded based on sample pronunciation features before the target sample pronunciation feature. The sample pronunciation features before the target sample pronunciation feature are past pronunciation features corresponding to past audio segments generated before the audio segment corresponding to the target sample pronunciation feature.

[0070] For example, the sample audio data includes four audio segments a, b, c, and d, and the four audio segments correspond to respective sample pronunciation features. The four sample pronunciation features corresponding to the four audio segments a, b, c, and d can be input into the first to-be-trained model for numerical encoding. When the sample pronunciation feature corresponding to the audio segment a is numerically encoded, since there is no audio segment before a, the past information corresponding to a can be set as empty by default, and the pronunciation feature corresponding to a is directly numerically encoded. When the sample pronunciation feature corresponding to the audio segment b is numerically encoded, since the audio segment before b is a, the pronunciation feature corresponding to b is numerically encoded based on the pronunciation feature corresponding to the audio segment a. When the sample pronunciation feature corresponding to the audio segment c is numerically encoded, since the audio segments before c are a and b, the pronunciation feature corresponding to c is numerically encoded based on the pronunciation features corresponding to the audio segments a and b.

[0071] Specifically, the first to-be-trained model can be a unidirectional recurrent neural network model, a causal convolutional neural network model, or other models that numerically encode data based on past information. Numerical processing of data based on past information by the first to-be-trained model can be understood as numerical encoding of data by the first to-be-trained model based on past information and without future information.

[0072] Step S130: inputting the sample pronunciation feature into a second model for numerical encoding to obtain a numerically encoded second output feature corresponding to the sample pronunciation feature, the second model being a model that numerically encodes pronunciation features based on the past information and the future information.

[0073] The future information is future pronunciation features corresponding to future audio segments generated after the audio segment corresponding to the sample pronunciation feature.

[0074] In this step, each sample pronunciation feature corresponding to each audio segment of the sample audio data can be input into the second model for numerical coding. When the second model performs numerical coding on a target sample pronunciation feature in the sample pronunciation features, the second model can perform numerical coding on the target sample pronunciation feature based on sample pronunciation features after the target sample pronunciation feature. The sample pronunciation features after the target sample pronunciation feature are future pronunciation features corresponding to future audio segments generated after the audio segment corresponding to the target sample pronunciation feature.

[0075] Taking the sample audio data including four audio segments a, b, c, and d as an example, four sample pronunciation features corresponding to the four audio segments a, b, c, and d can be input into the second model for numerical coding. When the second model performs numerical coding on the sample pronunciation feature corresponding to the audio segment a, since there is no audio segment before a and the audio segments after a are b, c, and d, the sample pronunciation feature corresponding to a can be numerically coded based on three future pronunciation features corresponding to the three future audio segments b, c, and d. When the second model performs numerical coding on the sample pronunciation feature corresponding to the audio segment b, since the audio segment before b is a and the audio segments after b are c and d, the sample pronunciation feature corresponding to b can be numerically coded based on the past audio feature corresponding to the past audio segment a and the two future pronunciation features corresponding to the two future audio segments c and d.

[0076] The second model can be a trained model or an untrained model. If the second model is an untrained model, the second model can be trained synchronously in the process of training the audio processing model in the present disclosure.

[0077] The second model can be a bidirectional recurrent neural network model, a non-causal convolutional neural network model, or other models that process data based on past information and future information. That is, the second model can perceive future information when processing data.

[0078] In the embodiments of the present disclosure, the first to-be-processed model and the second model can perform at least one numerical coding process on the sample pronunciation feature, such as vector coding processing, matrix coding processing, and dimension increasing processing. Alternatively, the first to-be-processed model and the second model can perform other numerical coding processing on the sample pronunciation feature. The specific processing content can be determined according to actual requirements, and the present disclosure does not make specific limitations.

[0079] It can be understood that the first to-be-processed model and the second model perform the same dimension processing on the pronunciation feature. For example, if the first to-be-processed model is used to perform vector coding on the pronunciation feature, the second model also performs vector coding on the pronunciation feature.

[0080] Step S140: adjusting the first to-be-trained model based on a first parameter adjustment principle of reducing the difference between the first output feature and the second output feature, to obtain an adjusted first model.

[0081] Specifically, the audio processing model can be trained through a large amount of sample audio data, and whether the first to-be-trained model is trained is determined according to whether the difference between the first output feature and the second output feature meets a convergence condition, so as to obtain the adjusted first model. The convergence condition may, for example, be that the difference between the first output feature and the second output feature is less than a first preset threshold, or that, for a plurality of sample audio data, the proportion of the difference between the first output feature and the second output feature being less than the first preset threshold reaches a first preset proportion threshold, which may, for example, be 80% to 95%, or may be another relatively large proportion.

[0082] Step S150: determining the audio processing model according to the adjusted first model.

[0083] Specifically, the adjusted first model can be directly determined as the audio processing model, or the adjusted first model can be further optimized to obtain the audio processing model. The specific processing manner can be determined according to the specific application scenario of the audio processing model.

[0084] For example, when the audio processing model provided by the present disclosure is used in the timbre conversion scene of audio, if the second model is a model that has been pre-trained and meets the timbre conversion scene, the adjusted first model can be directly determined as the audio processing model.

[0085] In order to make the converted audio after timbre conversion more accurate, the audio processing model can also be determined through the following specific implementation. In the specific implementation, the following steps S140a to S140b can be included before step S150, and step S150 can be implemented according to the following step S151.

[0086] Step S140a: determining a sample timbre feature of the sample audio data.

[0087] Specifically, the sample timbre feature of the sample audio data can be identified through a pre-trained timbre identification model; the sample timbre feature of the sample audio data can also be determined through artificial marking, that is, the timbre of the sample audio data is identified by artificial identification, and then the timbre is marked; or, a timbre library can be pre-set, the timbre library includes various timbre features, and a pre-trained timbre determination model is used to determine the timbre consistent with the timbre of the sample audio data from the timbre library. The pre-trained timbre identification model and the timbre determination model can be trained through a supervised training manner or an unsupervised training manner, and the present disclosure will not be described in detail.

[0088] Optionally, as shown in FIG. 2, the speaker identifier corresponding to the sample audio data can also be obtained, and the timbre feature corresponding to the speaker identifier is determined from the pre-stored timbre library through the speaker embedding layer as the sample timbre feature.

[0089] Step S140b: fuse the sample timbre feature and the first output feature to obtain fused audio data.

[0090] Specifically, the sample timbre feature and the first output feature can be fused through a fusion model; or, the harmonic distribution of the sample timbre can be obtained, and the first output feature is made to have a harmonic distribution consistent with the harmonic distribution through a filter to obtain the fused audio data; or, fusion can be performed through other manners, which is not specifically limited by the present disclosure.

[0091] Step S151: based on a second parameter adjustment principle for reducing the difference between the fused audio data and the sample audio data, the parameters of the adjusted first model are adjusted to obtain an audio processing model.

[0092] Specifically, when the difference between the fused audio data and the sample audio data meets a second convergence condition, it can be determined that the parameters of the adjusted first model are adjusted, thereby obtaining the audio processing model. The second convergence condition may, for example, be that the difference between the fused audio data and the corresponding sample audio data is less than a second preset threshold, or, for multiple sample audio data, the proportion of the difference between the fused audio data and the sample audio data being less than the second preset threshold reaches a second preset proportion threshold. The second preset proportion threshold can be the same as or different from the first preset proportion threshold, which is not specifically limited by the present disclosure.

[0093] The present embodiment adjusts the parameters of the adjusted first model by judging whether the fused audio data after fusing the first output feature and the timbre feature is accurate, so that the feature processing capability of the obtained audio processing model is more capable of meeting the scene of fusing timbre and pronunciation. In this way, for the scene of fusing timbre and pronunciation, the fusion accuracy of the features processed by the audio processing model for timbre fusion is higher.

[0094] The training method of the audio processing model provided in the embodiments of the present disclosure includes the following steps. After determining the sample pronunciation features corresponding to each audio segment of the sample audio data, the sample pronunciation features are input into a first to-be-trained model for numerical encoding to obtain numerical first output features, and the sample pronunciation features are input into a second model for numerical encoding to obtain numerical second output features. Since the first to-be-trained model is a model for numerically encoding pronunciation features based on past information, the past information is past pronunciation features corresponding to past audio segments generated before the audio segment, and the second model is a model for numerically encoding pronunciation features based on past and future information, the future information is future pronunciation features corresponding to future audio segments generated after the audio segment, that is, the first to-be-trained model cannot perceive future information when numerically encoding, and the second model can perceive future information when numerically encoding. Therefore, the second output features are features obtained based on the perception of past and future information, and the first output features are features without the perception of future information. The embodiments of the present disclosure adjust the parameters of the first to-be-trained model based on a first parameter adjustment principle for reducing the difference between the first output features and the second output features. In this way, the output result of the adjusted first model can be more consistent with the output result of the model that can perceive future information. In other words, the present disclosure distills the first to-be-trained model that cannot perceive future information based on the output result of the second model that can perceive future information. In this way, the adjusted first model obtained by distillation does not use future information when numerically encoding, but the result of the adjusted first model when numerically encoding can also perceive future information well. Therefore, the audio processing model determined by the adjusted first model can also perceive future information well.

[0095] When performing audio conversion, the vocal feature of the source audio is extracted by using the audio processing model that can perceive future information well. Therefore, the converted audio obtained by fusing the first audio extraction feature and the timbre of the target speaker is also a converted audio based on the perception of future information. Therefore, the converted audio has higher accuracy, better intelligibility and sound quality, more stable fundamental frequency, and better conversion effect.

[0096] In one specific embodiment, the method can further include the following step S160.

[0097] Step S160: adjusting the parameters of the second model based on the second parameter adjustment principle to obtain an adjusted second model.

[0098] The step S160 is to adjust the parameters of the second model based on a second parameter adjustment principle for reducing the difference between the fused audio data and the sample audio data, to obtain an adjusted second model. The specific parameter adjustment manner of the second model can refer to the adjustment manner of the adjusted first model in the foregoing, that is, a conventional supervised model training manner, which will not be described here in detail.

[0099] The embodiment updates the parameters of the second model based on the fused audio data, so that the features extracted by the second model are more consistent with the scene of the fusion of timbre and pronunciation, so that the second model can more accurately extract features meeting the fusion scene when training the audio processing model subsequently, thereby enabling the audio processing model finally trained to more accurately extract features meeting the fusion scene. In addition, the embodiment updates and trains the second model, and the second model can be directly used for feature processing of non-streaming audio for the fusion of timbre and pronunciation subsequently, so that the training process of the present disclosure can train the models for processing of both streaming and non-streaming audio at the same time, thereby improving the model training efficiency.

[0100] The first to-be-trained model and the second model, that is, the causal convolutional encoder and the acausal convolutional encoder in FIG. 2, form a dual-mode encoder in FIG. 2.

[0101] In a specific embodiment, the step S140b can obtain the fused audio data in the following step A, and the training method of the audio processing model can further include the following step S170.

[0102] Step A: input the sample timbre features and the first output features into a first fusion model to obtain first fused audio data.

[0103] Step S170: adjust the parameters of the first fusion model based on a principle for reducing the difference between the first fused audio data and the sample audio data, to obtain an adjusted first fusion model.

[0104] The first fusion model can be a model for fusing data based on past first fusion information. The past first fusion information includes the first output features corresponding to the past pronunciation features and the timbre features corresponding to the past pronunciation features.

[0105] For example, the sample audio data also includes four audio segments a, b, c, and d. When the first fusion model fuses the sample timbre feature and the first output feature corresponding to a, since there is no audio segment before a, the fusion can be directly performed. When the first fusion model fuses the sample timbre feature and the first output feature corresponding to b, the audio segments before b include a, and therefore, the first fusion model fuses the sample timbre feature and the first output feature corresponding to b based on the first output feature corresponding to a. When the first fusion model fuses the sample timbre feature and the first output feature corresponding to c, the audio segments before c include a and b, and therefore, the first fusion model fuses the sample timbre feature and the first output feature corresponding to c based on the first output features corresponding to a and b, respectively.

[0106] The specific adjustment process in step S170 can refer to the process of adjusting the parameters of the second model and the first fusion model described above, and will not be described in detail here.

[0107] The first fusion model can be a neural network model, a decision tree model, a support vector machine model, or other machine learning models.

[0108] Specifically, the first fusion model can be a model that processes data based on past information, for example, a causal convolution model, a unidirectional recurrent neural network model, etc. In this way, since the first output feature is a feature obtained based on past information, when the first fusion model is a model that processes data based on past information, the first fusion model is more suitable for fusing the first output feature, making the audio fusion more accurate. In addition, when the first fusion model is a model that processes data based on past information, the adjusted first fusion model obtained after parameter adjustment can more accurately fuse and process the streaming audio in the subsequent process. This embodiment updates and trains the first fusion model, which can be directly used for audio fusion in the subsequent process, improving the model training efficiency.

[0109] The first fusion model can also be a model that processes data based on past and future information, for example, a non-causal convolution model, a bidirectional recurrent neural network model, etc. In this way, the fusion of timbre and pronunciation can also be achieved, and the present disclosure is not specifically limited.

[0110] The first fusion audio data obtained in step A, i.e., the fusion audio data obtained in step S140b, that is, the fusion audio data obtained in step S140b includes the first fusion audio data.

[0111] Optionally, the fusion audio data can also be obtained according to the following step B in step S140b, and the training method of the audio processing model can further include the following step S180.

[0112] Step B: inputting the sample timbre feature and the second output feature into a second fusion model to obtain second fusion audio data.

[0113] The second fusion model is a model for processing data based on past and future information. The second fusion model can be a bidirectional neural network model, a non-causal convolution model, or other models for processing data based on past and future information.

[0114] The second fusion model is a model for fusing data based on past second fusion information and future second fusion information. The past second fusion information includes second output features corresponding to past pronunciation features, and the future second fusion information includes second output features corresponding to future pronunciation features.

[0115] For example, the sample audio data also includes four audio segments a, b, c, and d. When the second fusion model fuses the sample timbre feature and the second output feature corresponding to a, there is no audio segment before a, and the audio segments after a include three audio segments b, c, and d. Therefore, the second fusion model fuses the sample timbre feature and the second output feature corresponding to a based on the second output features corresponding to the three future pronunciation features of the three future audio segments b, c, and d. When the second fusion model fuses the sample timbre feature and the second output feature corresponding to b, the audio segment before b is a, and the audio segments after b include three audio segments c and d. Therefore, the second fusion model fuses the sample timbre feature and the second output feature corresponding to b based on the second output feature corresponding to the past pronunciation feature of the past audio segment a and the second output features corresponding to the two future pronunciation features of the two future audio segments c and d.

[0116] The second fusion audio data obtained in step B, i.e., the fusion audio data obtained in step S140b, also includes the second fusion audio data, that is, the fusion audio data obtained in step S140b also includes the second fusion audio data.

[0117] Step S180: adjusting the parameters of the second fusion model based on the principle of reducing the difference between the second fusion audio data and the sample audio data to obtain an adjusted second fusion model.

[0118] The parameter adjustment process of step S180 can be parameter step S170, which will not be described in detail here.

[0119] The adjusted second fusion model in this embodiment can be applied to fusion with the target sound color in subsequent non-streaming audio conversion. One training can obtain multiple models, improving the efficiency of model training.

[0120] The first fusion model and the second fusion model are a double-mode decoder as shown in FIG. 2. The first fusion audio data and the second fusion audio data are streaming audio and non-streaming audio as shown in FIG. 2.

[0121] The adjusted second fusion model can be applied to the fusion with the target speaker timbre in subsequent conversion of the non-streaming audio.

[0122] In an implementation, before step S150, the method can further include the following step a. Step S150 can be implemented in the following step S152.

[0123] Step a: input the first output feature into a prediction network to obtain sample prediction data corresponding to the sample audio data.

[0124] The prediction network is used to determine the corresponding audio data (and the sample prediction data in step a) according to the first output feature. The prediction network can include one prediction model or multiple prediction models. The prediction model included in the prediction network can be a pre-trained model or an untrained model. When the prediction model is an untrained model, the prediction model can be subsequently trained for parameter update.

[0125] Step S152: based on the principle of reducing the difference between the sample prediction data and the sample audio data, the parameters of the adjusted first model are adjusted to obtain an audio processing model.

[0126] Specifically, when the difference between the sample prediction data and the sample audio data meets a third convergence condition, it is determined that the parameters of the adjusted first model are adjusted, thereby obtaining the audio processing model. The third convergence condition can be, for example, that the difference between the sample prediction data and the corresponding sample audio data is less than a third preset threshold, or that, for multiple sample audio data, the proportion of the difference between the sample prediction data and the sample audio data that is less than the third preset threshold reaches a third preset proportion threshold. The third preset proportion threshold can be the same as or different from the first preset proportion threshold, which is not specifically limited in the present disclosure.

[0127] The present implementation can improve the accuracy of the obtained audio processing model by predicting the first output feature through the prediction network and further adjusting the parameters of the adjusted first model according to the accuracy of the prediction result.

[0128] Optionally, the method can further include the following steps: inputting the second output feature into the prediction network to obtain second prediction data corresponding to the sample audio data; and adjusting the parameters of the second model based on the principle of reducing the difference between the second sample prediction data and the sample audio data to obtain an adjusted second model. The feature extraction of the adjusted second model is more accurate, and the accuracy of subsequent conversion of the non-streaming audio is higher.

[0129] Optionally, the prediction network can include a first prediction model and a second prediction model, the first prediction model being configured to predict data of a next frame according to data of a current frame, and the second prediction model being configured to select data of the next frame from a plurality of frame data.

[0130] Specifically, as shown in FIG. 2, the prediction network can be a hybrid predictive coding (HPC), the first prediction model can be an autoregressive predictive coding (APC), and the second prediction model can be a contrastive predictive coding (CPC). The contrastive predictive coding and the autoregressive predictive coding each use a network with an autoregressive structure (g-net).

[0131] The CPC learns representations by classifying real future frames and randomly selected negative examples. The contrastive predictive coding selects negative examples using prior knowledge, which can better learn representations for specific tasks. The contrastive predictive coding is a contrastive learning method. The input audio obtains hidden representations through an encoder, and then obtains sequence features through an autoregressive network. We select hidden representations close to the target point as positive examples, and select hidden representations far from the target point as negative examples.

[0132] The APC is a generative model that directly predicts future frames. Given an input sequence, the APC predicts the spectrum of the future period. The direct prediction of the APC is not affected by the selection method of the negative examples, and has better generalization than the CPC. Combining the two methods of the contrastive predictive coding and the autoregressive predictive coding can make the output of the hybrid predictive coding have better generalization.

[0133] Step a can be implemented according to the following step a-1 and step a-2. Step S152 can be implemented according to the following step S152a.

[0134] Step a-1: input the first output feature into the first prediction model to obtain first sample prediction data corresponding to the sample audio data.

[0135] Step a-2: input the first output feature into the second prediction model to obtain second sample prediction data corresponding to the sample audio data.

[0136] Step S152a: based on the principle of reducing the difference between the first sample prediction data, the second sample prediction data and the sample audio data, the first model after adjustment is adjusted in parameters.

[0137] Exemplarily, taking the first output feature as an example, as shown in FIG. 3, the Z sequence is the input first output feature, the first output feature is input into the autoregressive prediction encoder of the hybrid predictive encoder to obtain first prediction sample data, wherein the first sample prediction data is sequence data, that is, the first r sequence, that is, the r sequence in FIG. 3. The Z sequence is the input first output feature, the first output feature is input into the autoregressive prediction encoder of the hybrid predictive encoder to obtain first prediction sample data, wherein the first sample prediction data is sequence data, that is, the first r sequence, that is, the r sequence in FIG. 3. The Z sequence is the input first output feature, the first output feature is input into the autoregressive prediction encoder of the hybrid predictive encoder to obtain first prediction sample data, wherein the first sample prediction data is sequence data, that is, the first r sequence, that is, the r sequence in FIG. 3.

[0138] The first prediction model and the second prediction model are used for prediction respectively in the embodiment, so that the first model after adjustment is further updated according to two prediction results, the representation can be learned for a specific task, and the output result has better generalization, so that the generalization of the audio processing model is better, and the feature processing is more accurate.

[0139] The second output feature can also be input into the first prediction model and the second prediction model respectively to obtain prediction data, and the specific process is referred to the first output feature, which is not described in detail here.

[0140] In an implementation, the above step A can be implemented as step A-1 and step A-2.

[0141] Step A-1: obtaining a pitch feature of the sample audio data;

[0142] Step A-2: inputting the sample timbre feature, the first output feature and the pitch feature into a first fusion model to obtain first fusion audio data.

[0143] The pitch feature can be extracted by a pre-trained pitch extraction model, and the pitch extraction model can be trained by a related machine learning algorithm, which is not specifically introduced in the disclosure. The pitch is used to represent the pitch height of the performer, and the pitch of the performer at different time points is usually different. By fusing the pitch, the fusion audio can be more accurately obtained, so that the feature processing accuracy of the finally obtained audio processing model is higher when applied to the audio fusion scene.

[0144] The second embodiment of the disclosure also provides an audio conversion method. The execution subject of the audio conversion method provided by the disclosure can be an electronic device, which can be a server, a desktop computer, a notebook computer, a tablet computer, a mobile phone, a smart watch, a smart television, etc., or other electronic devices with data processing functions, which are not specifically limited by the disclosure.

[0145] The audio conversion method provided by the present disclosure can be used in live scenes, dubbing scenes, etc. to convert audio of singing, speaking, etc. The specific application scenarios are not limited. Specifically, the method can be used to convert streaming audio transmitted in real time when converting singing, or can be used to convert non-streaming audio.

[0146] As shown in FIG. 4, the audio conversion method provided by the present disclosure includes the following steps S210-S240.

[0147] Step S210: Obtain source audio to be converted and timbre characteristics of a target sounder.

[0148] The source audio can be singing, speaking audio, dialogue audio of a film or other video, etc. or other forms of audio.

[0149] In step S210, the source audio to be converted and the timbre characteristics of the target sounder can be determined based on an audio conversion request sent by a received client, wherein the audio conversion request can include audio information of the source audio to be converted and timbre information of the target sounder. The audio information of the source audio can be specific audio content or an identifier corresponding to the audio, and the electronic device can find the corresponding audio as the source audio from an audio library through the identifier corresponding to the audio. The timbre information can be specific timbre characteristics or a timbre identifier corresponding to the timbre, and the electronic device can also determine the corresponding timbre characteristics through the timbre identifier.

[0150] Step S220: Extract pronunciation characteristics of the source audio.

[0151] The pronunciation characteristics are characteristics of the source audio after at least filtering out the timbre information of the source sounder.

[0152] The process of extracting the pronunciation characteristics in this step can refer to the manner of extracting the sample pronunciation characteristics in step S110 of the first embodiment, which will not be described in detail here.

[0153] Step S230: Input the pronunciation characteristics into a pre-trained audio processing model to obtain first audio extraction characteristics corresponding to the source audio.

[0154] The audio processing model is trained according to the training method of the audio processing model in any one of the first embodiments.

[0155] Step S240: Fuse the first audio extraction characteristics with the timbre information to obtain converted audio corresponding to the target sounder.

[0156] The fusion manner in step S240 can refer to the specific execution process of step S140b in the first embodiment, which will not be described in detail here.

[0157] As can be known from the first embodiment, since the feature extraction is performed on the occurrence characteristics of the source audio by using the audio processing model capable of well perceiving future information, the converted audio obtained by fusing the first audio extraction feature and the timbre of the target sound producer is also the converted audio based on the perception of future information, so that the converted audio has higher accuracy, better intelligibility and sound quality, more stable fundamental frequency and better conversion effect.

[0158] Optionally, the step S240 can obtain the converted audio corresponding to the target sound producer by the following steps: inputting the first audio extraction feature and the timbre information into a pre-trained first fusion model to obtain the converted audio corresponding to the target sound producer, wherein the first fusion model is trained according to the manner of training the adjusted first fusion model in the training method of the audio processing model according to the first embodiment. As can be known from the first embodiment, since the first fusion model is a model for processing data based on past information, it is more suitable for fusing streaming audio data and has higher fusion accuracy.

[0159] Optionally, the step S230 of inputting the pronunciation feature into the pre-trained audio processing model can be implemented by the following step S231.

[0160] The step S231: when the source audio is a real-time audio stream, inputting the pronunciation feature into the pre-trained audio processing model.

[0161] The audio conversion method can further include the following steps S250-S260.

[0162] The step S250: when the source audio is a non-real-time audio stream, inputting the pronunciation feature into a second model to obtain a second audio extraction feature corresponding to the source audio, wherein the second model is trained according to the manner of training the adjusted second model in the training method of the audio processing model according to the first embodiment.

[0163] The step S260: fusing the second audio extraction feature and the timbre information to obtain the converted audio corresponding to the target sound producer.

[0164] It can be known from the first embodiment that the audio processing model is obtained by training the model for processing data based on past information, such as the causal convolution model and the one-way neural network model, and therefore, the audio processing model is more suitable for processing streaming audio, and the processing result is more accurate, so that the fusion result is also more accurate. For non-real-time audio stream, that is, non-streaming audio, the second model is used for feature processing in this embodiment. As can be known from the first embodiment, the second model is a model for processing data based on past and future information, and therefore, the second model is more accurate for processing non-streaming audio, and the fusion result is also more accurate.

[0165] Optionally, the step S260 can be implemented in the following step S261.

[0166] Step S261: inputting the second audio extraction feature and the timbre information into a pre-trained second fusion model to obtain converted audio corresponding to the target speaker, the second fusion model being trained according to the manner of training the adjusted second fusion model in the training method of the audio processing model according to the first embodiment. As can be known from the first embodiment, the second fusion model is a model for data fusion based on past and future information, and therefore, the second fusion model is more suitable for fusing non-streaming data, and the fusion accuracy is higher.

[0167] The third embodiment of the present disclosure further provides a training device of an audio processing model, as shown in FIG. 5, the device comprises:

[0168] The first determination unit 310 is configured to determine the sample pronunciation feature corresponding to each audio segment of the sample audio data.

[0169] The first extraction unit 320 is configured to input the sample pronunciation feature into a first to-be-trained model for numerical encoding to obtain a numerical first output feature corresponding to the sample pronunciation feature, the first to-be-trained model being a model for numerical encoding of the pronunciation feature based on past information, the past information being past pronunciation features corresponding to past audio segments generated before the audio segment corresponding to the sample pronunciation feature; and input the sample pronunciation feature into a second model for numerical encoding to obtain a numerical second output feature corresponding to the sample pronunciation feature, the second model being a model for numerical encoding of the pronunciation feature based on the past information and future information, the future information being future pronunciation features corresponding to future audio segments generated after the audio segment corresponding to the sample pronunciation feature.

[0170] The adjusting unit 330 is configured to perform parameter adjustment on the first to-be-trained model based on a first parameter adjustment principle of reducing the difference between the first output feature and the second output feature, to obtain an adjusted first model.

[0171] The second determining unit 340 is configured to determine the audio processing model according to the adjusted first model.

[0172] Optionally, the first to-be-trained model is a causal convolution model, and the second model is an acausal convolution model.

[0173] Optionally, the first determining unit 310 is further configured to determine a sample timbre feature of the sample audio data.

[0174] The apparatus further includes:

[0175] The first fusion unit is configured to fuse the sample timbre feature and the first output feature to obtain fused audio data.

[0176] The second determining unit 340 is specifically configured to perform parameter adjustment on the adjusted first model based on a second parameter adjustment principle of reducing the difference between the fused audio data and the sample audio data, to obtain an audio processing model.

[0177] Optionally, the adjusting unit 330 is further configured to perform parameter adjustment on the second model based on the second parameter adjustment principle, to obtain an adjusted second model.

[0178] Optionally, the first fusion unit is specifically configured to input the sample timbre feature and the first output feature into a first fusion model to obtain first fused audio data.

[0179] The adjusting unit is further configured to perform parameter adjustment on the first fusion model based on a principle of reducing the difference between the first fused audio data and the sample audio data, to obtain an adjusted first fusion model.

[0180] Optionally, the first fusion model is a model of fusing data based on past first fusion information, and the past first fusion information includes a first output feature corresponding to the past pronunciation feature.

[0181] Optionally, the first fusion unit is further configured to perform: inputting the sample timbre feature and the second output feature into a second fusion model to obtain second fusion audio data, the second fusion model being a model for fusing data based on past second fusion information and future second fusion information, the past second fusion information including the second output feature corresponding to the past pronunciation feature, and the future second fusion information including the second output feature corresponding to the future pronunciation feature.

[0182] The adjustment unit is further configured to perform: performing parameter adjustment on the second fusion model based on a principle of reducing difference between the second fusion audio data and the sample audio data to obtain an adjusted second fusion model.

[0183] Optionally, the apparatus further comprises: a prediction unit configured to perform: inputting the first output feature into a prediction network to obtain sample prediction data corresponding to the sample audio data.

[0184] The second determination unit is specifically configured to perform: performing parameter adjustment on the adjusted first model based on a principle of reducing difference between the sample prediction data and the sample audio data to obtain the audio processing model.

[0185] Optionally, the prediction network comprises a first prediction model and a second prediction model, the first prediction model being used to predict data of a next frame according to data of a current frame, and the second prediction model being used to select data of the next frame from a plurality of frame data.

[0186] The prediction unit is specifically configured to perform: inputting the first output feature into the first prediction model to obtain first sample prediction data corresponding to the sample audio data; and inputting the first output feature into the second prediction model to obtain second sample prediction data corresponding to the sample audio data.

[0187] The second determination unit is specifically configured to perform: performing parameter adjustment on the adjusted first model based on a principle of reducing difference between the first sample prediction data, the second sample prediction data and the sample audio data.

[0188] Optionally, the first fusion unit is specifically configured to perform: obtaining a pitch feature of the sample audio data; and inputting the sample timbre feature, the first output feature and the pitch feature into a first fusion model to obtain first fusion audio data.

[0189] Optionally, the first determination unit is specifically configured to perform: converting each audio segment of the sample audio data into sample mel-frequency spectrum data.

[0190] input the mel-frequency spectrum data into a pre-trained speech recognition model to obtain a bottleneck feature corresponding to each audio segment of the sample audio data, and determine the bottleneck feature as a sample pronunciation feature corresponding to each audio segment of the sample audio data, the speech recognition model being used to identify a pronunciation feature of audio data.

[0191] The fourth embodiment of the present disclosure also provides an audio conversion device, the device comprising:

[0192] an acquisition unit configured to acquire source audio to be converted and timbre information of a target sound producer;

[0193] a second extraction unit configured to extract a pronunciation feature of the source audio, the pronunciation feature being a feature of the source audio after filtering out at least source sound producer timbre information;

[0194] a processing unit configured to input the pronunciation feature into a pre-trained audio processing model to obtain a first audio extraction feature corresponding to the source audio, the audio processing model being trained according to the training method of the audio processing model of any one of the first embodiment;

[0195] a fusion unit configured to fuse the first audio extraction feature and the timbre information to obtain converted audio corresponding to the target sound producer.

[0196] Optionally, the fusion unit is specifically configured to perform:

[0197] input the first audio extraction feature and the timbre feature into a pre-trained first fusion model to obtain converted audio corresponding to the target sound producer, the first fusion model being trained according to the manner of training the adjusted first fusion model in the training method of the audio processing model of the first embodiment.

[0198] Optionally, the processing unit is specifically configured to perform: when the source audio is a real-time audio stream, input the pronunciation feature into a pre-trained audio processing model.

[0199] the processing unit is further configured to perform: when the source audio is a non-real-time audio stream, input the pronunciation feature into a second model to obtain a second audio extraction feature corresponding to the source audio, the second model being trained according to the manner of training the adjusted second model in the training method of the audio processing model of the first embodiment; and fuse the second audio extraction feature and the timbre information to obtain converted audio corresponding to the target sound producer.

[0200] Optionally, the fusion unit is specifically configured to perform: inputting the second audio extraction feature and the timbre information into a pre-trained second fusion model to obtain converted audio corresponding to the target speaker, the second fusion model being trained according to a manner of training the adjusted second fusion model in the training method of the audio processing model according to the first embodiment.

[0201] It should be noted that the information interaction and execution process between the modules / units in the device are based on the same concept as the various method embodiments of the present disclosure corresponding to FIG. 1 to FIG. 3, and the specific content can be referred to the description in the foregoing method embodiments of the present disclosure, which will not be repeated here.

[0202] Next, an electronic device provided by an embodiment of the present disclosure is introduced. Referring to FIG. 6, FIG. 6 is a structural schematic diagram of an electronic device provided by an embodiment of the present disclosure. The electronic device 500 can specifically be a virtual reality (VR) device, a mobile phone, a tablet, a notebook computer, a smart wearable device, a monitoring data processing device, or a radar data processing device, etc., which is not limited here. The electronic device 500 can be deployed with the training device or the conversion device described in the embodiments corresponding to FIG. 1 or FIG. 4, to implement the functions in the embodiments corresponding to FIG. 1 to FIG. 4. Specifically, the electronic device 500 includes a receiver 501, a transmitter 502, a processor 503, and a memory 504 (wherein the number of processors 503 in the electronic device 500 can be one or more, and one processor is taken as an example in FIG. 6), wherein the processor 503 can include an application processor 5031 and a communication processor 5032. In some embodiments of the present disclosure, the receiver 501, the transmitter 502, the processor 503, and the memory 504 can be connected through a bus or other means.

[0203] The memory 504 can include a read-only memory and a random access memory, and provide the processor 503 with instructions and data. A part of the memory 504 can also include a non-volatile random access memory (NVRAM). The memory 504 stores processor and operation instructions, executable modules or data structures, or subsets thereof, or expanded sets thereof, wherein the operation instructions can include various operation instructions for implementing various operations.

[0204] The processor 503 controls the operation of the execution device. In specific applications, various components of the execution device are coupled together through a bus system, which can include a data bus in addition to a power bus, a control bus, and a state signal bus, etc. However, for the sake of clarity, all kinds of buses are referred to as a bus system in the figure.

[0205] The method disclosed in the embodiments of the present disclosure can be applied to the processor 503 or implemented by the processor 503. The processor 503 can be an integrated circuit chip having a signal processing capability. In the implementation process, the steps of the method can be completed by an integrated logic circuit or an instruction in the form of software in the processor 503. The processor 503 described above can be a general processor, a digital signal processor (DSP), a microprocessor or a microcontroller, and can further include an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component. The processor 503 can implement or execute the methods, steps and logic block diagrams disclosed in the embodiments of the present disclosure. The general processor can be a microprocessor or the processor can also be any conventional processor or the like. The steps of the method disclosed in combination with the embodiments of the present disclosure can be directly embodied as a hardware code processor for execution, or a combination of hardware and software modules in the code processor for execution. The software module can be located in a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register or other mature storage medium in the art. The storage medium is located in the storage 504, and the processor 503 reads the information in the storage 504 and combines the hardware to complete the steps of the method.

[0206] The receiver 501 can be used to receive input digital or character information, and generate signal input related to the relevant settings and function control of the execution device. The transmitter 502 can be used to output digital or character information through the first interface; the transmitter 502 can also be used to send instructions to the disk group through the first interface to modify the data in the disk group; the transmitter 502 can also include a display device such as a display screen.

[0207] In the embodiments of the present disclosure, the application processor 5031 in the processor 503 is used to execute the training method or the conversion method in the corresponding embodiments of FIGS. 1 to 4. It should be noted that the specific manner of the application processor 5031 executing each step is based on the same concept as the method embodiments corresponding to FIGS. 1 to 4 in the present disclosure, and the technical effects brought by the same are the same as the method embodiments corresponding to FIGS. 1 to 4 in the present disclosure. For specific content, refer to the description of the method embodiments in the foregoing disclosure, which will not be repeated here.

[0208] The embodiment of the present disclosure further provides a computer readable storage medium, which comprises computer instructions, and the computer instructions are used for implementing any one of the technical solutions in the first embodiment or the second embodiment of the present disclosure when executed by a processor.

[0209] In one typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0210] The memory can include non-persistent memory in the computer readable medium, random access memory (RAM), and / or non-volatile memory such as read-only memory (ROM) or flash memory. The memory is an example of computer readable media.

[0211] The computer readable media includes both volatile and nonvolatile media, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile discs (DVDs) or other optical storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible to a computing device. According to the definition herein, the computer readable medium does not include transitory media, such as modulated data signals and carrier waves.

[0212] Those skilled in the art should understand that the embodiments of the present disclosure can be provided as a method, a system or a computer program product. Therefore, the present disclosure can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Moreover, the present disclosure can take the form of a computer program product implemented on one or more computer usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer usable program code.

[0213] The present disclosure is disclosed with reference to the preferred embodiments, but it is not intended to limit the present disclosure. Any person skilled in the art can make possible changes and modifications without departing from the spirit and scope of the present disclosure, so the protection scope of the present disclosure should be subject to the scope defined by the claims of the present disclosure.

Claims

1. A method for training an audio processing model, the method comprising: determining a sample pronunciation feature corresponding to each audio segment of sample audio data; inputting the sample pronunciation feature into a first to-be-trained model to perform numerical coding, to obtain a numerically coded first output feature corresponding to the sample pronunciation feature, the first to-be-trained model being a model for numerically coding a pronunciation feature based on past information, the past information being a past pronunciation feature corresponding to a past audio segment generated before the audio segment corresponding to the sample pronunciation feature; inputting the sample pronunciation feature into a second model to perform numerical coding, to obtain a numerically coded second output feature corresponding to the sample pronunciation feature, the second model being a model for numerically coding a pronunciation feature based on the past information and future information, the future information being a future pronunciation feature corresponding to a future audio segment generated after the audio segment corresponding to the sample pronunciation feature; performing parameter adjustment on the first to-be-trained model based on a first parameter adjustment principle for reducing a difference between the first output feature and the second output feature, to obtain an adjusted first model; determining an audio processing model according to the adjusted first model. The first to-be-trained model is a causal convolution model, and the second model is an acausal convolution model.

2. The method of claim 1, wherein, Before the determining of the trained audio processing model according to the adjusted first model, the method further comprises:

3. The method of claim 1, wherein, determining a sample timbre feature of the sample audio data; fusing the sample timbre feature and the first output feature to obtain fused audio data; The determining of the audio processing model according to the adjusted first model comprises: performing parameter adjustment on the adjusted first model based on a second parameter adjustment principle for reducing a difference between the fused audio data and the sample audio data, to obtain an audio processing model. The method further comprises:

4. The method of claim 3, wherein, performing parameter adjustment on the second model based on the second parameter adjustment principle, to obtain an adjusted second model. The fusing of the sample timbre feature and the first output feature to obtain fused audio data comprises:

5. The method of claim 3, wherein, inputting the sample timbre feature and the first output feature into a first fusion model to obtain first fused audio data; The method further comprises: performing parameter adjustment on the first fusion model based on a principle for reducing a difference between the first fused audio data and the sample audio data, to obtain an adjusted first fusion model. The first fusion model is a model for fusing data based on past first fusion information, the past first fusion information including the first output feature corresponding to the past pronunciation feature.

6. The method of claim 5, wherein, The method further comprises:

7. The method of claim 6, wherein, inputting the sample timbre feature and the second output feature into a second fusion model to obtain second fused audio data, the second fusion model being a model for fusing data based on past second fusion information and future second fusion information, the past second fusion information including the second output feature corresponding to the past pronunciation feature, and the future second fusion information including the second output feature corresponding to the future pronunciation feature; ​ The second fusion model is adjusted in parameters based on a principle of reducing a difference between the second fusion audio data and the sample audio data, to obtain an adjusted second fusion model.

8. The method of claim 1, wherein, Before the audio processing model is determined according to the adjusted first model, the method further comprises: inputting the first output feature and the second output feature into a prediction network to obtain sample prediction data corresponding to the sample audio data; The audio processing model is determined according to the adjusted first model, comprising: The adjusted first model is adjusted in parameters based on a principle of reducing a difference between the sample prediction data and the sample audio data, to obtain the audio processing model.

9. The method of claim 8, wherein, The prediction network comprises a first prediction model and a second prediction model, the first prediction model is used to predict data of a next frame according to data of a current frame, and the second prediction model is used to select data of the next frame from a plurality of frame data; The first output feature and the second output feature are inputted into the first prediction model to obtain first sample prediction data corresponding to the sample audio data; The first output feature and the second output feature are inputted into the second prediction model to obtain second sample prediction data corresponding to the sample audio data; The adjusted first model is adjusted in parameters based on a principle of reducing a difference between the sample prediction data and the sample audio data, comprising: The adjusted first model is adjusted in parameters based on a principle of reducing a difference between the first sample prediction data, the second sample prediction data and the sample audio data. The sample timbre feature and the first output feature are inputted into a first fusion model to obtain first fusion audio data, comprising:

10. The method of claim 5, wherein, obtaining a pitch feature of the sample audio data; The sample timbre feature, the first output feature and the pitch feature are inputted into the first fusion model to obtain the first fusion audio data. The sample pronunciation feature corresponding to each audio segment of the sample audio data is determined, comprising:

11. The method according to any one of claims 1 to 10, wherein, the sample audio data is converted into sample mel-frequency spectrum data; The mel-frequency spectrum data is inputted into a pre-trained speech recognition model to obtain a bottleneck feature corresponding to each audio segment of the sample audio data, and the bottleneck feature is determined as the sample pronunciation feature corresponding to each audio segment of the sample audio data, the speech recognition model is used to identify pronunciation features of audio data.

12. An audio conversion method, the method comprising: obtaining source audio to be converted and timbre information of a target sounder; extracting pronunciation features of the source audio, the pronunciation features being features of the source audio after filtering out at least timbre information of a source sounder; inputting the pronunciation features into a pre-trained audio processing model to obtain first audio extraction features corresponding to the source audio, the audio processing model being trained according to the training method of the audio processing model in any one of claims 1 to 11. ​ The first audio extraction feature is fused with the timbre information to obtain converted audio corresponding to the target sounder.

13. The audio conversion method of claim 12, wherein, The first audio extraction feature is fused with the timbre information to obtain converted audio corresponding to the target sounder. The first audio extraction feature is fused with the timbre information to obtain converted audio corresponding to the target sounder.

14. The audio conversion method of claim 12, wherein, The first audio extraction feature is fused with the timbre information to obtain converted audio corresponding to the target sounder. The first audio extraction feature is fused with the timbre information to obtain converted audio corresponding to the target sounder. The first audio extraction feature is fused with the timbre information to obtain converted audio corresponding to the target sounder. The first audio extraction feature is fused with the timbre information to obtain converted audio corresponding to the target sounder. The first audio extraction feature is fused with the timbre information to obtain converted audio corresponding to the target sounder.

15. The audio conversion method of claim 14, wherein, The first audio extraction feature is fused with the timbre information to obtain converted audio corresponding to the target sounder. The first audio extraction feature is fused with the timbre information to obtain converted audio corresponding to the target sounder. The first audio extraction feature is fused with the timbre information to obtain converted audio corresponding to the target sounder. The first audio extraction feature is fused with the timbre information to obtain converted audio corresponding to the target sounder. The first audio extraction feature is fused with the timbre information to obtain converted audio corresponding to the target sounder.

16. An audio processing model training device, comprising: a first determination unit configured to determine sample pronunciation features corresponding to each audio segment of sample audio data; a first extraction unit configured to input the sample pronunciation features into a first to-be-trained model for numerical encoding to obtain numerical first output features corresponding to the sample pronunciation features, the first to-be-trained model being a model for numerical encoding of pronunciation features based on past information, the past information being past pronunciation features corresponding to past audio segments generated before the audio segment corresponding to the sample pronunciation features; and input the sample pronunciation features into a second model for numerical encoding to obtain numerical second output features corresponding to the sample pronunciation features, the second model being a model for numerical encoding of pronunciation features based on the past information and future information, the future information being future pronunciation features corresponding to future audio segments generated after the audio segment corresponding to the sample pronunciation features; an adjustment unit configured to adjust the parameters of the first to-be-trained model based on a first parameter adjustment principle of reducing the difference between the first output features and the second output features to obtain an adjusted first model; a second determination unit configured to determine an audio processing model based on the adjusted first model.

17. An audio conversion device, comprising: An acquisition unit configured to acquire a source audio to be converted and timbre information of a target vocal performer; A second extraction unit configured to extract a pronunciation feature of the source audio, the pronunciation feature being a feature of the source audio after filtering out at least source vocal performer timbre information; A processing unit configured to input the pronunciation feature into a pre-trained audio processing model to obtain a first audio extraction feature corresponding to the source audio, the audio processing model being trained according to the training method of the audio processing model in any one of claims 1 to 11; A fusion unit configured to fuse the first audio extraction feature with the timbre information to obtain converted audio corresponding to the target vocal performer.

18. An electronic device, comprising: A memory and a processor; The memory and the processor are coupled; The memory is configured to store one or more computer instructions; The processor is configured to execute the one or more computer instructions to implement the method in any one of claims 1-15.

19. A computer-readable storage medium having one or more computer instructions stored thereon, the instructions being executed by a processor to implement the method in any one of claims 1-15.

Images