Audio synthesis method, apparatus, device, computer-readable medium, and program product

Abstract

Embodiments of the present disclosure disclose an audio synthesis method, device, equipment, computer readable medium and program product. A specific implementation of the method comprises: obtaining target audio corresponding to a first language and accent correction information corresponding to an audio synthesis scene, wherein the audio synthesis scene is a scene for synthesizing audio corresponding to a second language based on the target audio; extracting audio feature information corresponding to the target audio; and generating the synthesized audio corresponding to the second language by using a pre-trained speech synthesis large model according to the audio feature information and the accent correction information. The implementation is related to artificial intelligence, and by using the accent correction information, in the process of synthesizing the audio corresponding to the second language based on the target audio corresponding to the first language, the disturbance of the accent noise can be avoided, so that the synthesized audio without the accent noise can be accurately and efficiently generated.

Description

Technical Field

[0001] Embodiments of this disclosure relate to the field of computer technology, and more specifically to audio synthesis methods, apparatus, devices, computer-readable media, and program products. Background Technology

[0002] Currently, with the continuous development of artificial intelligence, various intelligent processing techniques for audio have become one of the main research directions. For audio synthesis under the premise of language conversion, the common approach is to convert and generate synthesized audio in the corresponding second language based on the input audio in the first language using text-to-speech (TTS) technology based on large-scale language models (LLM).

[0003] However, when using the above method, the following technical problems often arise: While existing voiceprint extraction models in speech synthesis technology extract identity features (Timbre), during large-scale data training, voiceprint vectors inevitably encode the speaker's prosodic habits, accent, and language-specific pronunciation. When using the voiceprint of a speaker with a strong native accent (e.g., Chinese) to synthesize a target language (e.g., English), the model imposes the native prosodic prior from the voiceprint onto the target language, resulting in synthesized speech with a distinct source language accent (e.g., "Chinglish"). Summary of the Invention

[0004] The summary portion of this disclosure is intended to provide a brief overview of the concepts, which will be described in detail in the detailed description portion. This summary portion is not intended to identify key or essential features of the claimed technical solutions, nor is it intended to limit the scope of the claimed technical solutions.

[0005] Some embodiments of this disclosure provide audio synthesis methods, apparatuses, electronic devices, computer-readable media, and program products to address the technical problems mentioned in the background section above.

[0006] In a first aspect, some embodiments of this disclosure provide an audio synthesis method, including: obtaining target audio corresponding to a first language and accent correction information corresponding to an audio synthesis scenario, wherein the audio synthesis scenario is a scenario of synthesizing audio corresponding to a second language based on the target audio; extracting audio feature information corresponding to the target audio; and generating synthesized audio corresponding to the second language using a pre-trained speech synthesis large model based on the audio feature information and the accent correction information.

[0007] Optionally, the extraction of audio feature information corresponding to the target audio includes: inputting the target audio into a pre-trained audio feature extraction model to obtain audio feature information, wherein the loss function corresponding to the audio feature extraction model is a deep learning loss function used for face recognition and voiceprint recognition; and the generation of synthesized audio corresponding to the second language using a pre-trained speech synthesis model based on the audio feature information and the accent correction information includes: performing accent semantic correction on the audio feature information based on the accent correction information to obtain first corrected feature information; and inputting the first corrected feature information into the speech synthesis model to obtain synthesized audio.

[0008] Optionally, the above-mentioned accent semantic correction of the audio feature information based on the accent correction information to obtain the first corrected feature information includes: multiplying the accent correction information with the target adjustment coefficient to obtain multiplied feature information; adding the multiplied feature information to the audio feature information to obtain initial corrected feature information; and projecting the initial corrected feature information onto a unit hypersphere to obtain the first corrected feature information.

[0009] Optionally, the extraction of audio feature information corresponding to the target audio includes: inputting the target audio into the target intermediate feature processing layer in a pre-trained audio feature extraction model to obtain first intermediate feature information as audio feature information; and generating synthesized audio corresponding to the second language using a pre-trained speech synthesis model based on the audio feature information and the accent correction information includes: performing accent semantic correction on the audio feature information based on the accent correction information to obtain second corrected feature information; inputting the second corrected feature information into the remaining network layer in the audio feature extraction model to obtain audio processing feature information; and inputting the audio processing feature information into the speech synthesis model to obtain synthesized audio.

[0010] Optionally, the above-mentioned method of generating synthesized audio corresponding to the second language using a pre-trained speech synthesis model based on the audio feature information and the accent correction information includes: inputting the audio feature information into a network module before the target network layer in the speech synthesis model to obtain second intermediate feature information; performing accent semantic correction on the second intermediate feature information based on the accent correction information to obtain third corrected feature information; and inputting the third corrected feature information into the target network layer in the speech synthesis model to obtain synthesized audio.

[0011] Optionally, the above-mentioned accent semantic correction of the audio feature information based on the accent correction information to obtain the first corrected feature information includes: obtaining the pre-trained linear transformation matrix corresponding to the accent correction information; and multiplying the linear transformation matrix with the audio feature information to obtain the first corrected feature information.

[0012] Optionally, the accent correction information is generated through the following steps: obtaining a first sample dataset corresponding to the first language and a second sample dataset corresponding to the second language; determining the first spherical centroid corresponding to the first sample dataset and the second spherical centroid corresponding to the second sample dataset; selecting feature dimensions related to accent semantics from each feature dimension corresponding to the audio feature information to obtain a feature dimension set; generating mask information based on the feature dimension set; and generating accent correction information based on the mask information, the first spherical centroid, and the second spherical centroid.

[0013] Optionally, the above-mentioned selection of feature dimensions related to accent semantics from each feature dimension corresponding to the audio feature information to obtain a feature dimension set includes: using at least one of the following methods: Fisher discriminant ratio method, mutual information-based dimension selection method, principal component analysis-based selection method, and gradient-based sensitivity analysis method to select feature dimensions related to accent semantics from each feature dimension corresponding to the audio feature information to obtain a feature dimension set.

[0014] Secondly, some embodiments of this disclosure provide an audio synthesis apparatus, including: an acquisition unit configured to acquire target audio corresponding to a first language and accent correction information corresponding to an audio synthesis scenario, wherein the audio synthesis scenario is a scenario of synthesizing audio corresponding to a second language based on the target audio; an extraction unit configured to extract audio feature information corresponding to the target audio; and a generation unit configured to generate synthesized audio corresponding to the second language using a pre-trained speech synthesis model based on the audio feature information and the accent correction information.

[0015] Optionally, the extraction unit can be configured to: input the target audio into a pre-trained audio feature extraction model to obtain audio feature information, wherein the loss function corresponding to the audio feature extraction model is a deep learning loss function used for face recognition and voiceprint recognition. The generation unit can be configured to: perform accent semantic correction on the audio feature information based on the accent correction information to obtain first corrected feature information; input the first corrected feature information into the speech synthesis model to obtain synthesized audio.

[0016] Optionally, the generation unit can be configured to: multiply the above-mentioned accent correction information with the target adjustment coefficient to obtain multiplied feature information; add the above-mentioned multiplied feature information with the above-mentioned audio feature information to obtain initial correction feature information; and project the above-mentioned initial correction feature information onto a unit hypersphere to obtain first correction feature information.

[0017] Optionally, the extraction unit can be configured to: input the target audio into the target intermediate feature processing layer of a pre-trained audio feature extraction model to obtain first intermediate feature information, which serves as audio feature information. The generation unit can be configured to: perform accent semantic correction on the audio feature information based on the accent correction information to obtain second corrected feature information; input the second corrected feature information into the remaining network layers of the audio feature extraction model to obtain audio processing feature information; and input the audio processing feature information into the speech synthesis model to obtain synthesized audio.

[0018] Optionally, the generation unit can be configured to: input the above-mentioned audio feature information into the network module before the target network layer in the above-mentioned large speech synthesis model to obtain the second intermediate feature information; perform accent semantic correction on the above-mentioned second intermediate feature information according to the above-mentioned accent correction information to obtain the third corrected feature information; and input the above-mentioned third corrected feature information into the target network layer in the above-mentioned large speech synthesis model to obtain the synthesized audio.

[0019] Optionally, the generation unit can be configured to: obtain a pre-trained linear transformation matrix corresponding to the above accent correction information; multiply the above linear transformation matrix with the above audio feature information to obtain the first correction feature information.

[0020] Thirdly, some embodiments of this disclosure provide an electronic device, including: one or more processors; and a storage device having one or more programs stored thereon, such that when the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any implementation of the first aspect.

[0021] Fourthly, some embodiments of this disclosure provide a computer-readable medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the method as described in any implementation of the first aspect.

[0022] Fifthly, some embodiments of this disclosure provide a computer program product, including a computer program that, when executed by a processor, implements the method described in any of the implementations of the first aspect above.

[0023] The above-described embodiments of this disclosure have the following beneficial effects: By utilizing accent correction information in the audio synthesis methods of some embodiments of this disclosure, the disturbance of accent noise can be avoided during the synthesis of audio corresponding to a second language based on the target audio corresponding to a first language, thereby accurately and efficiently generating synthesized audio without accent noise. Specifically, the reason for the presence of accent noise in synthesized audio is that, although the voiceprint extraction model in existing speech synthesis technology extracts identity features (Timbre), in large-scale data training, the voiceprint vector inevitably encodes the speaker's prosodic habits, accent, and language-specific pronunciation. When using the voiceprint of a speaker with a strong native accent (e.g., Chinese) to synthesize a target language (e.g., English), the model will impose the native prosodic prior in the voiceprint onto the target language, resulting in synthesized speech with a noticeable source language accent (e.g., "Chinglish"). Based on this, the audio synthesis methods of some embodiments of this disclosure first obtain the target audio corresponding to the first language and the accent correction information corresponding to the audio synthesis scenario, wherein the audio synthesis scenario is a scenario in which audio corresponding to a second language is synthesized based on the target audio. Here, accent correction information is used to correct accents during the subsequent audio feature information conversion from the first language to the second language, thus avoiding accent problems in the synthesized audio. By pre-setting accent correction information for two types of speech for audio conversion, no modification to the network structure or full training based on a complex training sample set is required. This achieves effective and accurate accent control with high versatility. Then, audio feature information corresponding to the target audio is extracted to facilitate subsequent accent noise removal. Finally, based on the audio feature information and accent correction information, a pre-trained speech synthesis model can accurately and efficiently remove and correct accent noise, generating the synthesized audio corresponding to the second language. In summary, using highly versatile accent correction information can avoid accent noise perturbations, accurately and efficiently generating synthesized audio without accent noise. Attached Figure Description

[0024] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and elements are not necessarily drawn to scale.

[0025] Figure 1 This is a schematic diagram illustrating an application scenario of an audio synthesis method according to some embodiments of the present disclosure; Figure 2 This is a flowchart of some embodiments of the audio synthesis method according to the present disclosure; Figure 3 These are flowcharts of other embodiments of the audio synthesis method according to this disclosure; Figure 4 This is a schematic diagram of the structure of some embodiments of the audio synthesis apparatus according to the present disclosure; Figure 5 This is a schematic diagram of the structure of an electronic device suitable for implementing some embodiments of the present disclosure. Detailed Implementation

[0026] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0027] It should also be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings. Unless otherwise specified, the embodiments and features described in this disclosure can be combined with each other.

[0028] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0029] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0030] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0031] Before performing any of the operations involving the collection, storage, or use of user personal information (e.g., target audio) disclosed in this disclosure, the relevant organizations or individuals shall fulfill their obligations, including conducting personal information security impact assessments, informing personal information subjects, and obtaining prior authorization and consent from personal information subjects.

[0032] This disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0033] Figure 1 This is a schematic diagram illustrating an application scenario of an audio synthesis method according to some embodiments of the present disclosure.

[0034] exist Figure 1 In the application scenario, firstly, the electronic device 101 can acquire the target audio 102 corresponding to the first language 103 and the accent correction information 104 corresponding to the audio synthesis scenario. The audio synthesis scenario is a scenario where the audio corresponding to the second language 108 is synthesized based on the target audio 102. In this application scenario, the first language 103 can be "Chinese". The second language 108 can be "English". The accent correction information 104 can be a "Chinese" to "English" accent correction plugin. Then, the electronic device 101 can extract the audio feature information 105 corresponding to the target audio 102. In this application scenario, the audio feature information 105 can be an audio feature vector. Finally, the electronic device 101 can generate the synthesized audio 107 corresponding to the second language 108 based on the audio feature information 105 and the accent correction information 104, using a pre-trained speech synthesis large model 106.

[0035] It should be noted that the aforementioned electronic device 101 can be either hardware or software. When the electronic device is hardware, it can be implemented as a distributed cluster consisting of multiple servers or terminal devices, or as a single server or a single terminal device. When the electronic device is software, it can be installed in the hardware devices listed above. It can be implemented as, for example, multiple software programs or software modules used to provide distributed services, or as a single software program or software module. No specific limitations are made here.

[0036] It should be understood that Figure 1 The number of electronic devices shown is merely illustrative. Any number of electronic devices can be used depending on the implementation requirements.

[0037] Continue to refer to Figure 2 The diagram illustrates a flow 200 of some embodiments of an audio synthesis method according to the present disclosure. This audio synthesis method includes the following steps: Step 201: Obtain the target audio corresponding to the first language and the accent correction information corresponding to the audio synthesis scenario.

[0038] In some embodiments, the entity executing the above-described audio synthesis method (e.g.) Figure 1The electronic device 101 shown can acquire the target audio corresponding to the first language and the accent correction information corresponding to the audio synthesis scenario via a wired or wireless connection. The first language can be the language corresponding to the audio content in the target audio. For example, the first language can be the language to be converted. The target audio can be the audio to be processed for subsequent generation of synthesized audio. The audio synthesis scenario can be a scenario where audio corresponding to the second language is synthesized based on the target audio. For example, the audio synthesis scenario can be a translation scenario, where the target audio in the first language is converted into audio content in the second language. The audio synthesis scenario can also be a dialogue scenario between different languages, where a response is given to a question corresponding to the target audio in the first language to generate a response audio in the second language. The accent correction information can be the content used to correct accents in the language conversion scenario. Specifically, during the language conversion process, the synthesized audio may also have significant accent noise due to the accent problems of the original audio. Therefore, based on the accent correction information, it is possible to achieve synthesized audio without accent problems. For example, accent correction information can be a correction vector between the first and second languages ​​(a segmentation vector representing the deviation generated during the conversion from the first language to the second language, i.e., a vector that corrects feature drift in the synthesized audio caused by accent features during the generation of synthesized audio). Alternatively, accent feature information can also be a script embedded in the correction vector. Here, the accent feature information is specifically designed for the first and second languages. That is, the accent feature information is specifically used to synthesize the second language audio from the first language audio. For audio synthesis or conversion between other languages, the corresponding accent feature information can be obtained for accent correction. In practice, for converting audio from "English" to "Chinese," the first accent correction information can be used. For converting audio from "Korean" to "Chinese," the second accent correction information can be used. The content of the first and second accent correction information differs.

[0039] As an example, firstly, a training sample set for converting the first language into the second language is obtained. Then, the aforementioned execution entity can utilize a multimodal large model to extract accent deviation feature information corresponding to each training sample in the training sample set, obtaining an accent deviation feature information set. This accent deviation feature information can be the feature difference between the converted audio of the first language corresponding to the training samples and the audio without accent noise. Next, the above accent deviation feature information set is averaged to obtain average accent deviation feature information. Finally, using the aforementioned multimodal large model, feature removal information (i.e., accent correction information) is generated for the above average accent deviation feature information. This feature removal information is then multiplied with the audio feature information converted based on the first language corresponding to the audio to obtain audio with accent noise removed.

[0040] In some optional implementations of certain embodiments, the above-mentioned accent correction information is generated through the following steps: The first step is to obtain the first sample dataset corresponding to the first language and the second sample dataset corresponding to the second language. The first sample data can be speech data corresponding to the first language, and the second sample data can be speech data corresponding to the second language. For example, the first sample data could be speech content in Chinese, and the second sample data could be speech content in English. For example, the number of samples in both the first and second sample datasets can be 1000. Furthermore, the first and second sample datasets should maintain a statistically balanced distribution in terms of gender (50% male / 50% female), age group (old / middle-aged / young), recording equipment, and acoustic environment.

[0041] It should be noted that the first and second sample datasets correspond to two completely independent groups of people with no overlap, meaning there is no need to search for rare bilingual individuals who can speak both their first and second languages.

[0042] The second step involves determining the first spherical centroid corresponding to the first sample dataset and the second spherical centroid corresponding to the second sample dataset. The first spherical centroid is the normalized centroid corresponding to the first sample dataset, and the second spherical centroid is the normalized centroid corresponding to the second sample dataset. The first spherical centroid represents the sum of voiceprint features corresponding to each of the first sample data points. The second spherical centroid represents the sum of voiceprint features corresponding to each of the second sample data points. In practice, the first and second spherical centroids can be in vector form.

[0043] As an example, firstly, each first sample data is input into the voiceprint extraction model (i.e., the audio feature extraction model) to generate first audio feature information, resulting in a first audio feature information set. Then, each second sample data is input into the voiceprint extraction model (i.e., the audio feature extraction model) to generate second audio feature information, resulting in a second audio feature information set. It should be noted that the first and second audio feature information here can be voiceprint feature vectors with a feature dimension of 192. Next, the first average audio feature information corresponding to the aforementioned first audio feature information set is determined. Then, the second average audio feature information corresponding to the aforementioned second audio feature information set is determined. Finally, the first and second average audio feature information are normalized respectively to obtain the first spherical centroid and the solar centroid.

[0044] As another example, the median or the trimmed mean can be used to calculate the language center to reduce the interference of outliers on the statistical results, thus obtaining the first and second spherical centroids.

[0045] As another example, Gaussian mixture models (GMMs) or clustering algorithms (such as K-Means) can be used to perform multimodal modeling on monolingual data, selecting principal component centers as representative vectors instead of single centroids to obtain the first and second spherical centroids.

[0046] The third step is to filter out the feature dimensions related to accent semantics from the various feature dimensions corresponding to the audio feature information, thus obtaining a feature dimension set. For feature vectors where the audio feature information is the target dimension, the feature dimension set can be dimensional information related to accent semantic content. That is, the set of feature elements corresponding to the feature dimension set determines the accent semantic content. For example, each feature dimension can have 192 feature dimensions. Alternatively, the feature dimension set might have 20 feature dimensions.

[0047] As an example, firstly, accent-related features are extracted from multiple audio feature sets, resulting in multiple accent feature sets. Then, for each audio feature set, the feature dimension group with the highest similarity to the corresponding accent feature is extracted. Finally, the feature dimensions with the highest frequency of occurrence are selected from the resulting multiple feature dimension groups to form the feature dimension set.

[0048] The fourth step is to generate mask information based on the aforementioned feature dimension set. This mask information is used to select and acquire accent-related features for subsequent accent correction processing. In other words, the mask information can be semantic content related to accent features.

[0049] As an example, the aforementioned execution entity can generate a mask vector corresponding to the feature dimension set, which serves as mask information.

[0050] Fifth step: Generate accent correction information based on the above mask information, the above first spherical centroid, and the above second spherical centroid.

[0051] As an example, first, the vector difference between the centroids of the first and second spheres is determined. Then, the vector difference is multiplied by the mask information to obtain the accent correction information.

[0052] Optionally, the above process of filtering out feature dimensions related to accent semantics from the various feature dimensions corresponding to the audio feature information yields a feature dimension set, including: The aforementioned execution entity can utilize at least one of the following methods: Fisher's Discriminant Ratio (FSR), mutual information-based dimensional selection, principal component analysis (PCA)-based selection, or gradient-based sensitivity analysis, to select feature dimensions relevant to accent semantics from the various feature dimensions corresponding to audio feature information, thus obtaining a feature dimension set. The mutual information-based dimensional selection method can calculate the mutual information between each feature dimension and the language label (Chinese / English), selecting the feature dimension with the largest information gain. The PCA-based selection method can perform Linear Discriminant Analysis (LDA) or PCA on the difference vector, projecting the original feature space onto the orthogonal subspace that best distinguishes languages, rather than directly selecting the original dimensions. The gradient-based sensitivity analysis method, if a small amount of gradient backpropagation is allowed, can calculate the gradient of the language classification loss with respect to the feature input, selecting the dimension with the largest absolute gradient value as the sensitive feature channel (i.e., feature dimension).

[0053] The Fisher discriminant ratio method involves assigning sensitivity scores to each feature dimension and then selecting the highest-scoring feature dimensions from these scores to form a feature dimension set. The sensitivity score reflects the degree to which a feature dimension conveys accent-related information. The sensitivity score can be a value between 0 and 1; a higher value indicates a stronger correlation with accent-related features.

[0054] In practice, sensitive scores can be generated through the following steps: First, for each feature dimension, firstly, based on the first sample dataset, generate the first standard deviation and the first eigenvector corresponding to the feature dimension. Secondly, based on the second sample dataset, generate the second standard deviation and the second eigenvector corresponding to the feature dimension. Thirdly, determine the vector difference information corresponding to the first and second eigenvectors. For example, determine the vector difference between the first and second eigenvectors. Determine the vector magnitude corresponding to the vector difference. Fourthly, add the first and second standard deviations to obtain the sum. Fifthly, divide the vector difference information by the sum to obtain the sensitivity score.

[0055] Step 202: Extract the audio feature information corresponding to the target audio.

[0056] In some embodiments, the executing entity may extract audio feature information corresponding to the target audio. The audio feature information may characterize the semantic content of the audio features corresponding to the target audio.

[0057] As an example, the aforementioned execution entity can utilize a pre-trained audio feature extraction model to extract audio feature information corresponding to the target audio. In practice, the audio feature extraction model can be a voiceprint feature extraction model.

[0058] In some optional implementations of certain embodiments, the extraction of audio feature information corresponding to the target audio includes the following steps: The aforementioned execution entity can input the target audio into the target intermediate feature processing layer of a pre-trained audio feature extraction model to obtain the first intermediate feature information, which serves as the audio feature information. Specifically, the audio feature extraction model can be a speaker encoder, and the target intermediate feature processing layer can be one of the first few network layers in the speaker encoder that performs feature processing. The audio feature extraction model can be a module in a speech synthesis system, responsible for encoding a segment of input reference audio into a fixed-dimensional vector (Embedding), which contains identity information such as the speaker's timbre, gender, and age. The audio feature extraction model can be a network model used for speaker feature extraction. Speaker feature information can be feature content related to speaker features. The target intermediate feature processing layer can be the penultimate layer or an intermediate layer of the speaker extractor (i.e., the audio feature extraction model). As long as the features of this layer have clear semantic separability and subsequent operations include a normalization step, accent feature correction can be performed.

[0059] Step 203: Based on the above audio feature information and the above accent correction information, use the pre-trained speech synthesis model to generate the synthesized audio corresponding to the above second language.

[0060] In some embodiments, the aforementioned execution entity can generate synthesized audio corresponding to the second language using a pre-trained large-scale speech synthesis model, based on the aforementioned audio feature information and accent correction information. The large-scale speech synthesis model can be a large model that performs automated and intelligent speech synthesis. In practice, the large-scale speech synthesis model can be a large model that supports speech synthesis. In practice, the large-scale speech synthesis model can support the following functions: zero-sample voice cloning, fine-grained emotion control, and multilingual and dialect mixing. The large-scale speech synthesis model can be a commercially available large-scale model. Further details will not be elaborated. The language of the audio content corresponding to the synthesized audio is the second language.

[0061] As an example, firstly, the aforementioned audio feature information and accent correction information are concatenated to obtain audio concatenation feature information. Then, prompting information is generated to produce synthesized audio under the second speech based on the aforementioned audio concatenation feature information and the audio synthesis task. Finally, the prompting information is input into the aforementioned large-scale speech synthesis model to obtain the synthesized audio.

[0062] In some optional implementations of certain embodiments, the execution entity can generate synthesized audio corresponding to the second language using a pre-trained speech synthesis model based on the audio feature information and the accent correction information, including the following steps: The first step is to perform accent semantic correction on the audio feature information based on the accent correction information mentioned above, thereby obtaining the second corrected feature information. The second corrected feature information can be the feature content after accent correction.

[0063] As an example, the aforementioned executing entity can multiply or add the accent correction information and the accent feature information to obtain the second correction feature information.

[0064] The second step involves inputting the aforementioned second corrected feature information into the remaining network layers of the audio feature extraction model to obtain audio processing feature information. The remaining network layers can be any network layers in the audio feature extraction model other than the target intermediate feature processing layer.

[0065] The third step is to input the audio processing feature information into the large speech synthesis model to obtain the synthesized audio.

[0066] In some optional implementations of certain embodiments, the execution entity can generate synthesized audio corresponding to the second language using a pre-trained speech synthesis model based on the audio feature information and the accent correction information, including the following steps: The first step involves inputting the aforementioned audio feature information into the network module preceding the target network layer in the aforementioned large-scale speech synthesis model to obtain the second intermediate feature information. The target network layer can be a linear layer or a fully connected layer in the large-scale speech synthesis model.

[0067] The second step involves semantically correcting the second intermediate feature information based on the aforementioned accent correction information to obtain the third corrected feature information. For details on the implementation, please refer to the generation of the second corrected feature information.

[0068] The third step involves inputting the aforementioned third modified feature information into the target network layer of the aforementioned speech synthesis model to obtain the synthesized audio.

[0069] The above-described embodiments of this disclosure have the following beneficial effects: By utilizing accent correction information in the audio synthesis methods of some embodiments of this disclosure, the disturbance of accent noise can be avoided during the synthesis of audio corresponding to a second language based on the target audio corresponding to a first language, thereby accurately and efficiently generating synthesized audio without accent noise. Specifically, the reason for the presence of accent noise in synthesized audio is that, although the voiceprint extraction model in existing speech synthesis technology extracts identity features (Timbre), in large-scale data training, the voiceprint vector inevitably encodes the speaker's prosodic habits, accent, and language-specific pronunciation. When using the voiceprint of a speaker with a strong native accent (e.g., Chinese) to synthesize a target language (e.g., English), the model will impose the native prosodic prior in the voiceprint onto the target language, resulting in synthesized speech with a noticeable source language accent (e.g., "Chinglish"). Based on this, the audio synthesis methods of some embodiments of this disclosure first obtain the target audio corresponding to the first language and the accent correction information corresponding to the audio synthesis scenario, wherein the audio synthesis scenario is a scenario in which audio corresponding to a second language is synthesized based on the target audio. Here, accent correction information is used to correct accents during the subsequent audio feature information conversion from the first language to the second language, thus avoiding accent problems in the synthesized audio. By pre-setting accent correction information for two types of speech for audio conversion, no modification to the network structure or full training based on a complex training sample set is required. This achieves effective and accurate accent control with high versatility. Then, audio feature information corresponding to the target audio is extracted to facilitate subsequent accent noise removal. Finally, based on the audio feature information and accent correction information, a pre-trained speech synthesis model can accurately and efficiently remove and correct accent noise, generating the synthesized audio corresponding to the second language. In summary, using highly versatile accent correction information can avoid accent noise perturbations, accurately and efficiently generating synthesized audio without accent noise.

[0070] Further reference Figure 3 The diagram illustrates a flow 300 of another embodiment of the audio synthesis method according to the present disclosure. This audio synthesis method includes the following steps: Step 301: Obtain the target audio corresponding to the first language and the accent correction information corresponding to the audio synthesis scenario.

[0071] In some embodiments, the specific implementation of step 301 and its resulting technical effects can be found in [reference needed]. Figure 2 Step 201 in the corresponding embodiment will not be repeated here.

[0072] Step 302: Input the target audio into the pre-trained audio feature extraction model to obtain audio feature information.

[0073] In some embodiments, the executing entity (e.g. Figure 1 The electronic device 101 shown can input the target audio to a pre-trained audio feature extraction model to obtain audio feature information. The loss function corresponding to the audio feature extraction model is a deep learning loss function used for face recognition and voiceprint recognition. In practice, the deep learning loss function used for face recognition and voiceprint recognition can be the ArcFace (Additive Angular Margin Loss) loss function or similar variants (e.g., CosFace, Sphereace) loss function.

[0074] In practice, the key characteristic of this loss function lies in forcing L2 normalization of the feature vector and weight vector. This indicates that all voiceprint features are not actually distributed in Euclidean space, but rather on a D-1 dimensional unit hypersphere. Under this geometric structure, the similarity between two voiceprints is determined by the cosine similarity of their corresponding angle, rather than by Euclidean distance. A geometric interpretation of accent: Systematic Angular Bias is based on the above geometric constraints. Although the ArcFace function strives to cluster samples from the same speaker on the hypersphere (Intra-class Compactness), the "language / accent" attribute, as a strong domain feature, introduces a systematic angular drift on the hypersphere. Assume an ideal "accent-free / pure tone" feature is located at point [point missing]. When the speaker speaks Chinese, the feature vector corresponding to Chinese will have a slight angular shift towards the "Chinese statistical center." When the speaker speaks English, the feature vector corresponding to English will shift towards the "English statistical center." The objective of this disclosure is to find and correct the tangential component (i.e., accent correction information) that causes the deflection. Since features must remain normalized, simple linear addition and subtraction would result in a magnitude non-1 (i.e., deviating from the hypersphere), which is theoretically impractical. This disclosure introduces a tangent perturbation and re-projection mechanism (i.e., correcting accent feature information based on spherical correction and feature dimension filtering).

[0075] Step 303: Based on the above accent correction information, perform accent semantic correction on the above audio feature information to obtain the first corrected feature information.

[0076] In some embodiments, the execution entity may perform accent semantic correction on the audio feature information based on the accent correction information to obtain first corrected feature information.

[0077] As an example, the aforementioned executing entity can add the accent correction information and the aforementioned audio feature information to obtain the first correction feature information.

[0078] In some optional implementations of certain embodiments, the execution entity may perform accent semantic correction on the audio feature information based on the accent correction information to obtain first corrected feature information, including the following steps: The first step is to multiply the above accent correction information with the target adjustment coefficient to obtain the multiplied feature information. The target adjustment coefficient can be an adjustable intensity coefficient, i.e., an intensity coefficient for adjusting the accent. For example, the target adjustment coefficient can be 0.4-1.

[0079] Here, by setting a target adjustment coefficient, the previously "black box" issue of accent can be transformed into a quantifiable and interpretable mathematical control. Users can flexibly adjust the corresponding value of the target adjustment coefficient according to business needs (e.g., foreign language teaching requires standard pronunciation, while film dubbing requires retaining exotic accents), achieving personalized customization.

[0080] The second step is to add the multiplied feature information to the audio feature information to obtain the initial corrected feature information.

[0081] The third step is to project the above-mentioned initial corrected feature information onto the unit hypersphere to obtain the first corrected feature information.

[0082] Here, by projecting onto the unit hypersphere, the initial correction feature information that deviates from the sphere can be pulled back onto the corresponding unit hypersphere, which geometrically rotates the vector by an angle along the direction of accent differences, rather than simply translating it, thus ensuring the stability of the synthesized audio.

[0083] In some optional implementations of certain embodiments, the execution entity may perform accent semantic correction on the audio feature information based on the accent correction information to obtain first corrected feature information, including the following steps: The first step is to obtain the pre-trained linear transformation matrix corresponding to the aforementioned accent correction information. This linear transformation matrix can be trained using a lightweight mapping network. It can be a linear transformation matrix with fewer parameters, used to implement feature correction. For the scheme of linear additive perturbation + spherical reprojection, this disclosure achieves this by training a linear transformation matrix.

[0084] The second step is to multiply the above linear transformation matrix with the above audio feature information to obtain the first corrected feature information.

[0085] Step 304: Input the first corrected feature information into the above-mentioned speech synthesis model to obtain synthesized audio.

[0086] In some embodiments, the execution entity may input the first modified feature information into the large speech synthesis model to obtain synthesized audio.

[0087] from Figure 3 It can be seen from this that, with Figure 2 Compared to the description of some corresponding embodiments, Figure 3In some corresponding embodiments, the audio synthesis method process 300, based on an audio feature extraction model with a loss function that is a deep learning loss function used for face recognition and voiceprint recognition, can accurately correct the semantics of the corresponding output of the audio feature extraction model to obtain more accurate synthesized audio.

[0088] Further reference Figure 4 As an implementation of the methods shown in the above figures, this disclosure provides some embodiments of an audio synthesis apparatus, which are similar to... Figure 2 Corresponding to the method embodiments shown, this audio synthesis device can be specifically applied to various electronic devices.

[0089] like Figure 4 As shown, an audio synthesis apparatus 400 includes: an acquisition unit 401, an extraction unit 402, and a generation unit 403. The acquisition unit 401 is configured to acquire target audio corresponding to a first language and accent correction information corresponding to an audio synthesis scenario, wherein the audio synthesis scenario is a scenario where audio corresponding to a second language is synthesized based on the target audio; the extraction unit 402 is configured to extract audio feature information corresponding to the target audio; and the generation unit 403 is configured to generate synthesized audio corresponding to the second language using a pre-trained speech synthesis model based on the audio feature information and the accent correction information.

[0090] In some optional implementations of certain embodiments, the extraction unit 402 may be further configured to: input the target audio into a pre-trained audio feature extraction model to obtain audio feature information, wherein the loss function corresponding to the audio feature extraction model is a deep learning loss function used for face recognition and voiceprint recognition. The generation unit 403 may be further configured to: perform accent semantic correction on the audio feature information based on the accent correction information to obtain first corrected feature information; input the first corrected feature information into the speech synthesis model to obtain synthesized audio.

[0091] In some optional implementations of some embodiments, the generation unit 403 may be further configured to: multiply the above-mentioned accent correction information with the target adjustment coefficient to obtain multiplied feature information; add the above-mentioned multiplied feature information with the above-mentioned audio feature information to obtain initial correction feature information; and project the above-mentioned initial correction feature information onto a unit hypersphere to obtain first correction feature information.

[0092] In some optional implementations of some embodiments, the extraction unit 402 may be further configured to: input the target audio to the target intermediate feature processing layer in the pre-trained audio feature extraction model to obtain first intermediate feature information as audio feature information.

[0093] In some optional implementations of some embodiments, the generation unit 403 may be further configured to: perform accent semantic correction on the audio feature information according to the accent correction information to obtain second corrected feature information; input the second corrected feature information into the remaining network layer in the audio feature extraction model to obtain audio processing feature information; and input the audio processing feature information into the speech synthesis model to obtain synthesized audio.

[0094] In some optional implementations of some embodiments, the generation unit 403 may be further configured to: input the above-mentioned audio feature information into a network module before the target network layer in the above-mentioned speech synthesis large model to obtain second intermediate feature information; perform accent semantic correction on the above-mentioned second intermediate feature information according to the above-mentioned accent correction information to obtain third corrected feature information; and input the above-mentioned third corrected feature information into the target network layer in the above-mentioned speech synthesis large model to obtain synthesized audio.

[0095] In some optional implementations of some embodiments, the generation unit 403 may be further configured to: obtain a pre-trained linear transformation matrix corresponding to the above-mentioned accent correction information; multiply the above-mentioned linear transformation matrix with the above-mentioned audio feature information to obtain the first correction feature information.

[0096] It is understandable that the units described in the audio synthesis device 400 are related to the reference. Figure 2 The steps in the described method correspond to each other. Therefore, the operations, features, and beneficial effects described above for the method also apply to the audio synthesis device 400 and the units contained therein, and will not be repeated here.

[0097] The following is for reference. Figure 5 It illustrates electronic devices suitable for implementing some embodiments of this disclosure (e.g., Figure 1 A schematic diagram of the structure of electronic device 101)500. Figure 5 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments of this disclosure.

[0098] like Figure 5 As shown, the electronic device 500 may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory 502 or a program loaded from a storage device 508 into a random access memory 503. The random access memory 503 also stores various programs and data required for the operation of the electronic device 500. The processing unit 501, the read-only memory 502, and the random access memory 503 are interconnected via a bus 504. An input / output interface 505 is also connected to the bus 504.

[0099] Typically, the following devices can be connected to the input / output interface 505: input devices 506 including, for example, a touchscreen, touchpad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.; output devices 507 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 508 including, for example, magnetic tape, hard disk, etc.; and communication devices 509. Communication device 509 allows electronic device 500 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 5 An electronic device 500 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively. Figure 5 Each box shown can represent a device or multiple devices as needed.

[0100] In particular, according to some embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, some embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 509, or installed from a storage device 508, or installed from a read-only memory 502. When the computer program is executed by the processing device 501, it performs the functions defined above in the methods of some embodiments of this disclosure.

[0101] It should be noted that, in some embodiments of this disclosure, the computer-readable medium described above may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In some embodiments of this disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In some embodiments of this disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0102] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0103] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device. The aforementioned computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to: acquire target audio corresponding to a first language and accent correction information corresponding to an audio synthesis scenario, wherein the audio synthesis scenario is a scenario of synthesizing audio corresponding to a second language based on the target audio; extract audio feature information corresponding to the target audio; and, based on the audio feature information and the accent correction information, generate synthesized audio corresponding to the second language using a pre-trained speech synthesis model.

[0104] Computer program code for performing operations of some embodiments of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0105] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0106] The units described in some embodiments of this disclosure can be implemented in software or hardware. The described units can also be housed in a processor; for example, a processor may be described as including an acquisition unit, an extraction unit, and a generation unit. The names of these units do not necessarily limit the unit itself; for example, an acquisition unit may also be described as "a unit that acquires target audio corresponding to a first language and accent correction information corresponding to an audio synthesis scenario."

[0107] The functions described above in this document can be performed at least in part by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), and so on.

[0108] Some embodiments of this disclosure also provide a computer program product, including a computer program that, when executed by a processor, implements any of the above-described audio synthesis methods.

[0109] The above description is merely a selection of preferred embodiments of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in the embodiments of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described inventive concept. For example, technical solutions formed by substituting the above-described features with (but not limited to) technical features with similar functions disclosed in the embodiments of this disclosure.

Claims

1. An audio synthesis method, comprising: Obtain the target audio corresponding to the first language and the accent correction information corresponding to the audio synthesis scenario, wherein the audio synthesis scenario is a scenario in which the audio corresponding to the second language is synthesized based on the target audio; Extract the audio feature information corresponding to the target audio; Based on the audio feature information and the accent correction information, a pre-trained speech synthesis model is used to generate synthesized audio corresponding to the second language.

2. The method according to claim 1, wherein, The step of extracting the audio feature information corresponding to the target audio includes: The target audio is input into a pre-trained audio feature extraction model to obtain audio feature information, wherein the loss function corresponding to the audio feature extraction model is a deep learning loss function used for face recognition and voiceprint recognition; and The step of generating synthesized audio corresponding to the second language using a pre-trained speech synthesis model based on the audio feature information and the accent correction information includes: Based on the accent correction information, the audio feature information is semantically corrected to obtain the first corrected feature information; The first corrected feature information is input into the speech synthesis model to obtain synthesized audio.

3. The method according to claim 2, wherein, The step of performing accent semantic correction on the audio feature information based on the accent correction information to obtain first corrected feature information includes: The accent correction information is multiplied by the target adjustment coefficient to obtain the multiplied feature information; The multiplied feature information is added to the audio feature information to obtain the initial corrected feature information; The initial corrected feature information is projected onto the unit hypersphere to obtain the first corrected feature information.

4. The method according to claim 1, wherein, The step of extracting the audio feature information corresponding to the target audio includes: The target audio is input into the target intermediate feature processing layer of a pre-trained audio feature extraction model to obtain first intermediate feature information, which serves as the audio feature information; and The step of generating synthesized audio corresponding to the second language using a pre-trained speech synthesis model based on the audio feature information and the accent correction information includes: Based on the accent correction information, the audio feature information is semantically corrected to obtain the second corrected feature information; The second corrected feature information is input into the remaining network layers of the audio feature extraction model to obtain audio processing feature information; The audio processing feature information is input into the speech synthesis model to obtain synthesized audio.

5. The method according to claim 1, wherein, The step of generating synthesized audio corresponding to the second language using a pre-trained speech synthesis model based on the audio feature information and the accent correction information includes: The audio feature information is input into the network module before the target network layer in the speech synthesis model to obtain the second intermediate feature information. Based on the accent correction information, the second intermediate feature information is semantically corrected to obtain the third corrected feature information; The third corrected feature information is input into the target network layer of the large speech synthesis model to obtain synthesized audio.

6. The method according to claim 2, wherein, The step of performing accent semantic correction on the audio feature information based on the accent correction information to obtain first corrected feature information includes: Obtain the pre-trained linear transformation matrix corresponding to the accent correction information; The first corrected feature information is obtained by multiplying the linear transformation matrix with the audio feature information.

7. The method according to claim 1, wherein, The accent correction information is generated through the following steps: Obtain the first sample dataset corresponding to the first language and the second sample dataset corresponding to the second language; Determine the first spherical centroid corresponding to the first sample dataset and the second spherical centroid corresponding to the second sample dataset; From the audio feature information, feature dimensions related to accent semantics are selected to obtain the feature dimension set; Generate mask information based on the feature dimension set; Accent correction information is generated based on the mask information, the first spherical centroid, and the second spherical centroid.

8. The method according to claim 7, wherein, The step of filtering out the feature dimensions related to accent semantics from the various feature dimensions corresponding to the audio feature information to obtain a feature dimension set includes: By using at least one of the following methods—Fisher discriminant ratio-based method, mutual information-based dimensional selection method, principal component analysis-based selection method, and gradient-based sensitivity analysis method—feature dimensions related to accent semantics are selected from each feature dimension corresponding to audio feature information, thus obtaining a feature dimension set.

9. An audio synthesis apparatus, comprising: The acquisition unit is configured to acquire the target audio corresponding to the first language and the accent correction information corresponding to the audio synthesis scenario, wherein the audio synthesis scenario is a scenario in which the audio corresponding to the second language is synthesized based on the target audio; The extraction unit is configured to extract audio feature information corresponding to the target audio. The generation unit is configured to generate synthesized audio corresponding to the second language using a pre-trained speech synthesis model based on the audio feature information and the accent correction information.

10. An electronic device, comprising: One or more processors; Storage device, on which one or more programs are stored, When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-8.

11. A computer-readable medium having a computer program stored thereon, wherein, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-8.

12. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-8.

Images