Audio data generation method, method for training deep learning model and agent

Abstract

The present disclosure provides an audio data generation method, a method for training a deep learning model and an agent, relating to the technical field of artificial intelligence, and particularly to the technical fields of voice technology, large model, intelligent customer service and the like. The audio data generation method comprises: receiving target language information; performing feature extraction on the target language information by a target encoder corresponding to a target style attribute among a plurality of preset style attributes, to obtain language features; and fusing the language features and pronunciation identity features to generate target audio data, the pronunciation identity features representing pronunciation identity attributes of a pronunciation object, the identity features being obtained by performing feature extraction on pronunciation data of the pronunciation object by an identity feature encoder, and the target audio data representing pronunciation of the pronunciation object on the target language information according to the target style attribute.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, particularly to the fields of voice technology, large-scale models, and intelligent customer service. Background Technology

[0002] With the rapid development of technology, speech synthesis technology can be used to generate audio data for dialogue with users in scenarios such as smart customer service and tourist guidance. For example, speech audio corresponding to guide text can be synthesized and played through portable voice broadcasting devices held by users to help them quickly understand the cultural relics in museums. Summary of the Invention

[0003] This disclosure provides a method for generating audio data, a method for training a deep learning model, and an intelligent agent.

[0004] According to one aspect of this disclosure, an audio data generation method is provided, comprising: receiving target language information; extracting features from the target language information based on a target encoder corresponding to a target style attribute from among multiple encoders corresponding to their respective preset style attributes, to obtain language features; and fusing the language features and pronunciation identity features to generate target audio data, wherein the pronunciation identity features represent the pronunciation identity attribute of a pronunciation object, and the pronunciation data of the pronunciation object is extracted using an identity feature encoder to obtain pronunciation identity features, and the target audio data represents the pronunciation of the pronunciation object in accordance with the target style attribute to the target language information.

[0005] According to another aspect of this disclosure, a method for training a deep learning model is provided. The deep learning model includes encoders corresponding to multiple preset style attributes. The method for training the deep learning model includes: receiving training samples, the training samples including sample target language information, sample pronunciation identity features, and labeled audio data. The sample pronunciation identity features represent the pronunciation identity attributes of the sample pronunciation object. The deep learning model's identity feature encoder extracts features from the pronunciation data of the sample pronunciation object to obtain sample pronunciation identity features. The labeled audio data represents the sample pronunciation object pronouncing the sample target language information according to the target style attributes. The deep learning model's target encoder, which corresponds to the target style attributes among the multiple encoders, extracts features from the sample target language information to obtain sample language features. The deep learning model's decoder fuses the sample language features and sample pronunciation identity features to generate sample target audio data. The deep learning model is then trained based on the sample target audio data and labeled audio data to obtain the trained deep learning model.

[0006] According to another aspect of this disclosure, an audio data generation apparatus is provided, comprising: a first receiving module for receiving target language information; a language feature acquisition module for extracting features from the target language information based on a target encoder corresponding to a target style attribute from among multiple encoders corresponding to preset style attributes, thereby obtaining language features; and a fusion module for fusing language features and pronunciation identity features to generate target audio data, wherein the pronunciation identity features represent the pronunciation identity attribute of a pronunciation object, and the pronunciation identity features are obtained by extracting features from the pronunciation data of the pronunciation object using an identity feature encoder, and the target audio data represents the pronunciation of the target language information by the pronunciation object according to the target style attribute.

[0007] According to another aspect of this disclosure, an apparatus for training a deep learning model is provided. The deep learning model includes encoders corresponding to multiple preset style attributes. The apparatus for training the deep learning model includes: a second receiving module for receiving training samples, the training samples including sample target language information, sample pronunciation identity features, and labeled audio data. The sample pronunciation identity features represent the pronunciation identity attributes of the sample pronunciation object. The pronunciation identity features are obtained by extracting features from the pronunciation data of the sample pronunciation object using the identity feature encoder of the deep learning model. The labeled audio data represents the pronunciation of the sample target language information by the sample pronunciation object according to the target style attribute; a sample language feature acquisition module for extracting features from the sample target language information using the target encoder corresponding to the target style attribute among the multiple encoders to obtain sample language features; a generation module for fusing the sample language features and sample pronunciation identity features using the decoder of the deep learning model to generate sample target audio data; and a training module for training the deep learning model based on the sample target audio data and labeled audio data to obtain the trained deep learning model.

[0008] According to another aspect of this disclosure, an artificial intelligence agent is provided, comprising: an input module for receiving input information; a processing module for determining a target task based on the input information received by the input module, determining a large model based on the target task, and obtaining output information by calling the large model to execute the method provided in the embodiments of this disclosure; and an output module for outputting the output information obtained by the processing module.

[0009] According to another aspect of this disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform a method provided according to an embodiment of this disclosure.

[0010] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are used to cause the computer to perform a method provided according to an embodiment of this disclosure.

[0011] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the method provided according to embodiments of this disclosure.

[0012] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0013] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0014] Figure 1 The illustration schematically shows an exemplary system architecture for which audio data generation methods and apparatus can be applied according to embodiments of the present disclosure.

[0015] Figure 2 A flowchart illustrating an audio data generation method according to an embodiment of the present disclosure is shown schematically.

[0016] Figure 3 A schematic diagram of a deep learning model according to an embodiment of the present disclosure is shown.

[0017] Figure 4 The schematic diagram illustrates the principle of target language information according to an embodiment of the present disclosure.

[0018] Figure 5 A flowchart illustrating a method for training a deep learning model according to an embodiment of the present disclosure is shown schematically.

[0019] Figure 6 The schematic diagram illustrates a method for training a deep learning model according to an embodiment of the present disclosure.

[0020] Figure 7 A schematic diagram illustrating the principle of determining tag audio data according to an embodiment of the present disclosure is shown.

[0021] Figure 8 A block diagram of an audio data generation apparatus according to an embodiment of the present disclosure is shown schematically.

[0022] Figure 9 A block diagram of an apparatus for training a deep learning model according to an embodiment of the present disclosure is shown schematically.

[0023] Figure 10A schematic block diagram of an artificial intelligence agent according to an embodiment of the present disclosure is shown.

[0024] Figure 11 A schematic block diagram of an example electronic device is shown that can be used to implement the audio data generation method and the method for training a deep learning model in the embodiments of this disclosure. Detailed Implementation

[0025] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0026] In the technical solutions disclosed herein, the acquisition, storage, and application of any type of information, such as user personal information, comply with the provisions of relevant laws and regulations, necessary confidentiality measures have been taken, and they do not violate public order and good morals.

[0027] The inventors discovered that during the speech synthesis process, the synthesized audio data is difficult to vividly represent the speaker's diverse pronunciation styles, and it is difficult to accurately represent the speaker's timbre and pronunciation style, resulting in problems such as a floating timbre and a weakened sense of character.

[0028] This disclosure provides an audio data generation method, a method for training a deep learning model, and an intelligent agent. The audio data generation method includes: receiving target language information; extracting features from the target language information using a target encoder corresponding to a target style attribute from among multiple encoders corresponding to their respective preset style attributes, to obtain language features; and fusing the language features and pronunciation identity features to generate target audio data, wherein the pronunciation identity features represent the pronunciation identity attribute of the pronunciation object, and the pronunciation identity features are obtained by extracting features from the pronunciation data of the pronunciation object using an identity feature encoder, and the target audio data represents the pronunciation object pronouncing the target language information according to the target style attribute.

[0029] According to embodiments of this disclosure, feature extraction of target language information is performed using a target encoder corresponding to the target style attribute from among multiple preset style attributes. This allows the extracted language features to accurately represent the audio semantics of pronunciation of the target language information according to the target style attribute. By fusing the pronunciation identity features extracted from the pronunciation data by the identity feature encoder with the language features to generate target audio data, confusion between the timbre attribute and pronunciation style attribute of the target object in the generated target audio data can be avoided, thus preventing a weakening of the audio data's character representation. This improves the accuracy of the target audio data's representation of pronunciation style and pronunciation identity attributes, thereby enhancing the quality of the audio data.

[0030] Figure 1 The illustration schematically shows an exemplary system architecture for which audio data generation methods and apparatus can be applied according to embodiments of the present disclosure.

[0031] It is important to note that Figure 1 The examples shown are merely examples of system architectures applicable to embodiments of this disclosure, intended to help those skilled in the art understand the technical content of this disclosure. They do not imply that embodiments of this disclosure cannot be used in other devices, systems, environments, or scenarios. For instance, in another embodiment, an exemplary system architecture for applying the audio data generation method and apparatus may include a terminal device. However, the terminal device can implement the audio data generation method and apparatus provided in the embodiments of this disclosure without interacting with a server.

[0032] like Figure 1 As shown, the system architecture 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 serves as a medium for providing communication links between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired and / or wireless communication links, etc.

[0033] Users can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 via the network 104 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as knowledge reading applications, web browser applications, search applications, instant messaging tools, email clients, and / or social platform software, etc. (for example only).

[0034] The first terminal device 101, the second terminal device 102, and the third terminal device 103 can be various electronic devices with displays and support web browsing, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0035] Server 105 can be a server that provides various services, such as a backend management server that supports the content browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103 (this is just an example). The backend management server can analyze and process data such as received user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal devices.

[0036] It should be noted that the audio data generation method provided in this embodiment can generally be executed by the first terminal device 101, the second terminal device 102, or the third terminal device 103. Correspondingly, the audio data generation device provided in this embodiment can also be disposed in the first terminal device 101, the second terminal device 102, or the third terminal device 103.

[0037] Alternatively, the audio data generation method provided in this embodiment can generally be executed by server 105. Correspondingly, the audio data generation apparatus provided in this embodiment can generally be located in server 105. The audio data generation method provided in this embodiment can also be executed by a server or server cluster that is different from server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105. Correspondingly, the audio data generation apparatus provided in this embodiment can also be located in a server or server cluster that is different from server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105.

[0038] For example, when a user is reading an ebook online, the first terminal device 101, the second terminal device 102, and the third terminal device 103 can acquire the target content in the ebook that the user is looking at, and then send the acquired target content to the server 105. The server 105 analyzes the target content to determine its feature information; predicts the content that the user is interested in based on the feature information; and extracts the content that the user is interested in. Alternatively, a server or server cluster that can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or the server 105 can analyze the target content and ultimately extract the content that the user is interested in.

[0039] It should be understood that Figure 1The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0040] Figure 2 A flowchart illustrating an audio data generation method according to an embodiment of the present disclosure is shown schematically.

[0041] like Figure 2 As shown, the audio data generation method includes operations S210~S23.

[0042] During operation S210, target language information is received.

[0043] In operation S220, the target encoder corresponding to the target style attribute is used to extract features from the target language information to obtain language features, based on the encoders corresponding to the multiple preset style attributes.

[0044] In operation S230, language features and pronunciation identity features are fused to generate target audio data.

[0045] According to embodiments of this disclosure, target language information may include any information such as text or phonemes that represents the pronunciation content used to identify target audio data. In some examples, target language information may include target text and target pronunciation markers used to identify sub-language audio segments.

[0046] Paralinguistic audio segments can be audio data that represents nonverbal vocalizations (or paralinguistic vocalizations). Paralinguistic audio segments can represent the pronunciation content that does not represent written language during a person's speech. For example, paralinguistic audio segments can represent sounds that a person would actually make during actual speech, such as laughter, sighs, panting, throat clearing, and crying.

[0047] Multiple target pronunciation markers can be used to identify corresponding sub-language audio segments using any type of marker, such as characters or symbols. For example, target pronunciation markers can be embedded among multiple text words in the target text to determine target language information. However, this is not limited to this; target language information can also be determined based on other methods. For instance, text words in the target text and multiple target sub-language markers can be arranged based on timestamp positions in a preset timeline. This allows the target language information to represent the pronunciation times of the speech segments associated with the text words and the sub-language audio segments corresponding to the sub-language markers.

[0048] Preset style attributes represent the pronunciation style of the target language information, such as emotion and rhythm. For example, preset style attributes can represent whispering style, high-pitched style, broadcasting style, etc. By using encoders corresponding to multiple different preset style attributes to extract features from the target language information separately, the extracted language features can each represent the pronunciation mode of the target language information according to their respective predicted style attributes.

[0049] Therefore, the target encoder extracts language features from the target language information, which can represent the pronunciation of the target language information according to the target style attribute corresponding to the target encoder. For example, if the target style attribute represents the whispering style, the language features can represent the pronunciation of the target language information "speak softly" according to the whispering style.

[0050] According to embodiments of this disclosure, the pronunciation identity feature represents the pronunciation identity attribute of the pronunciation object. The pronunciation identity feature may, for example, represent the timbre attribute of the pronunciation object, or it may also represent the voiceprint feature attribute of the pronunciation object.

[0051] Target audio data represents the pronunciation of target language information by a speech subject according to target style attributes. For example, target audio data could be audio data representing an authorized person pronouncing the target language information "Please tell me loudly" according to the high-pitched pronunciation style attribute.

[0052] By using an identity feature encoder to extract features from the pronunciation data of the pronunciation object, pronunciation identity features are obtained. This allows the pronunciation identity features and language features to be extracted by different encoders, thereby fully decoupling the process of pronunciation semantic understanding of pronunciation identity attributes and pronunciation style attributes. This avoids confusion between pronunciation identity attributes and pronunciation style attributes caused by the same encoder performing feature extraction or feature embedding, which would otherwise lead to confusion between the features used to generate the target audio data. This improves the accuracy of the target audio data in representing the timbre and pronunciation style of the pronunciation object.

[0053] Figure 3 A schematic diagram of a deep learning model according to an embodiment of the present disclosure is shown.

[0054] like Figure 3 As shown, the deep learning model includes encoders corresponding to multiple preset style attributes, and decoders shared by multiple preset style attributes. The encoders corresponding to the multiple preset style attributes are, respectively, the whisper style attribute encoder corresponding to the whisper style attribute, the baseline style attribute encoder corresponding to the baseline style attribute, ... and the high-pitched style attribute encoder corresponding to the high-pitched style attribute.

[0055] Voice identity features can be obtained by extracting voiceprint features from the voice data of a voice object using an identity feature encoder. Multiple preset voice identity features are cached in a designated storage area so that the matching voice identity features can be retrieved from the designated storage area in a timely manner according to the needs of the target audio data.

[0056] A whispering style attribute encoder is used to extract features from the target language information, obtaining language features corresponding to the pronunciation patterns of whispering style attributes. By fusing pronunciation identity features and language features using a decoder, the target audio data can represent the pronunciation of the target language information by the speaker according to the whispering style attribute.

[0057] In some embodiments, the target language information includes target text and target pronunciation tokens for identifying sub-language audio segments. The target pronunciation tokens, for example, are used to map encoded feature vectors corresponding to predefined sub-language pronunciation tokens (sub-language pronunciation tokens) in a set of sub-language pronunciation tokens, where the encoded feature vectors represent sub-language audio segments.

[0058] In some examples, feature extraction of target language information based on the target encoder corresponding to the target style attribute among the encoders corresponding to multiple preset style attributes may include: encoding the target text based on a set of text tags to obtain text features, wherein the text features represent the mapping relationship between the target text and preset text tags in the set of text tags; encoding the target pronunciation tags based on a set of paralinguistic pronunciation tags to obtain paralinguistic features; and fusing the text features and paralinguistic features to determine the language features.

[0059] Among them, the paralinguistic features represent the mapping relationship between the target pronunciation marker and the paralinguistic pronunciation markers in the paralinguistic pronunciation marker set. The paralinguistic pronunciation markers are used to identify preset paralinguistic audio segments.

[0060] Figure 4 The schematic diagram illustrates the principle of target language information according to an embodiment of the present disclosure.

[0061] like Figure 4As shown, the target language information is "I [a2b3] am too tired...". The target text can be a word in the target language information, and the target pronunciation markers a2 and b3 are the target pronunciation markers. The word "I" in the target language information is mapped to text marker 12 in the text marker table, and the target pronunciation marker a2 is mapped to the corresponding preset secondary language marker a2 in the secondary language pronunciation marker table, and the target pronunciation marker b3 is mapped to the corresponding preset secondary language marker b3 in the secondary language pronunciation marker table. Each preset secondary language marker in the secondary language pronunciation marker table has a mapping relationship with a preset encoding feature vector, which is used to represent the corresponding secondary language audio segment. This achieves the determination of the encoding feature vector representing the secondary language audio segment after encoding the target pronunciation marker. It should be understood that in this example, the text marker set is the text marker table, and the secondary language pronunciation marker set is the secondary language pronunciation marker table.

[0062] Therefore, a text encoder can be used to encode the target text in the target language information to obtain text features, which represent the pronunciation content of the text words. A paralinguistic encoder is then used to encode the target pronunciation markers a2 and b3 to obtain paralinguistic features. For example, the target pronunciation markers can be embedded using the encoded feature vectors corresponding to each preset paralinguistic marker in the paralinguistic marker table, resulting in paralinguistic features comprising multiple encoded feature vectors. By fusing text features and paralinguistic features, the resulting language features represent the pronunciation content of the text words and paralinguistic audio segments according to the order of the text and target pronunciation markers in the target language information.

[0063] Therefore, by using different encoders to encode the target text and target pronunciation markers separately, the text features and the encoded features representing the paralinguistic audio segments are decoupled, avoiding pronunciation confusion between the pronunciation content of the target text and the pronunciation content of the paralinguistic audio information in the target audio data. Simultaneously, a decoder fuses the language features and pronunciation identity features to complete audio decoding, resulting in target audio data that pronounces the target language information according to the pronunciation identity attributes and target pronunciation style attributes of the pronunciation object.

[0064] The target audio data includes target speech information and paralinguistic audio information. The target speech information represents the pronunciation content of the speaker in relation to the target text, while the paralinguistic audio information represents the pronunciation content of the speaker's timbre according to the target style attribute "whispering style," specifically the paralinguistic audio segment "crying" corresponding to the target pronunciation marker a2. This avoids confusion between the pronunciation of the text and the paralinguistic pronunciation, ensures consistency between the pronunciation methods of the text and the paralinguistic audio segments and the target style attribute, and reflects the speaker's timbre attribute. This improves the data quality of the target audio data, prevents the weakening of the timbre attribute, and enhances the vividness and naturalness of the pronunciation.

[0065] In some embodiments, the audio data generation method further includes: processing input information of the target object using a large model, determining target style attributes, and target language information for responding to the input information.

[0066] The input information for the target audience can be text or images used to represent any need, such as product information consultation or navigation route setting. By utilizing a large model to process the input information, the large model can understand the intended need of the input information and output target language information to meet that need. Simultaneously, the target style attributes output by the large model are used to respond to the emotional intent expressed by the input information. Target audio data is used for dialogue with the target audience.

[0067] For example, given the input message "Where is the barcode?", the large model understands the input speech and outputs the target language information "It's under the can." Simultaneously, by analyzing the emotional attributes of the input speech, the model determines the target audience's emotional attribute to be "anxious," thus prompting the model to output the target style attribute "joyful" to respond to the target audience's emotional needs. In this way, deep learning models can be used to process target style attributes, target language information, and pronunciation identity features, outputting target audio data to engage in dialogue with the target audience, promptly meeting their actual needs and soothing their current emotions.

[0068] In some embodiments, the target style attributes are determined by analyzing the target language information using a large model.

[0069] For example, in video and animation production, target language information can be the script content within the production script. A large model analyzes the text, phoneme sequences, and other script content to determine the pronunciation style of the video or animation character, thus obtaining the target style attribute. A target encoder corresponding to this style attribute can then process the script content, outputting language features representing pronunciation according to the target style attribute. By fusing these language features with the pronunciation identity features of the video or animation character, target audio data is generated. This enables automated pronunciation of the script content according to its emotional tone, rhythm, and other pronunciation styles. The large model controls the pronunciation style of the target audio data, maintaining the character's timbre while ensuring that the target audio data played synchronously with the video or animation can vividly represent diverse pronunciation styles. This reduces the negative impact of pronunciation style and timbre confusion in synchronously played audio data on the production content, improving video and animation production efficiency.

[0070] In some embodiments, the audio data generation method further includes: driving a virtual avatar displayed in an interactive interface to perform a pronunciation action based on the target audio data.

[0071] For example, target audio data can be used to drive a virtual cartoon character to perform lip movements, enabling the target to converse with the virtual cartoon character in the interactive interface. The virtual cartoon character can flexibly and vividly switch between various style attributes of the target audio data, such as whispering style and happy style, according to the target's input information, to meet the target's interactive needs and enhance the target's interactive experience.

[0072] For example, in specific scenarios such as nighttime scenes, the target user can interact with a virtual avatar through an interactive interface. The large model determines the target style attribute as a "whisper style attribute" by detecting the input information and the target user's surrounding environment. This allows the pronunciation style of the target audio data to adapt to the target user's current environment. The target user can then receive the target audio data pronounced by the virtual avatar using the whisper style attribute and engage in dialogue with it, thereby enhancing the target user's interaction needs in specific scenarios and preventing the target audio data from being too loud and disturbing other objects in the environment.

[0073] Figure 5 A flowchart illustrating a method for training a deep learning model according to an embodiment of the present disclosure is shown schematically.

[0074] like Figure 5 As shown, the method for training a deep learning model includes operations S510~S540.

[0075] The S510 is used to receive training samples.

[0076] In operation S520, the target encoder corresponding to the target style attribute among multiple encoders is used to extract features from the target language information of the sample, thereby obtaining the sample language features.

[0077] When operating the S530, the decoder of the deep learning model is used to fuse sample language features and sample pronunciation identity features to generate sample target audio data.

[0078] In the S540, a deep learning model is trained based on the sample target audio data and the label audio data to obtain the trained deep learning model.

[0079] According to embodiments of this disclosure, the training samples include target language information, pronunciation identity features, and tagged audio data. The pronunciation identity features represent the pronunciation identity attributes of the sample pronunciation object, and are obtained by extracting features from the pronunciation data of the sample pronunciation object using a deep learning model's identity feature encoder.

[0080] Tag audio data represents the pronunciation of a sample speech object according to the target style attribute in relation to the target language information. Tag audio data can be determined by collecting audio data of the pronunciation content of the sample speech object according to the target style attribute. Alternatively, tag audio data can also be determined by speech synthesis based on any type of model, such as a multimodal large model, according to the target style attribute and the speech identity characteristics of the sample speech object. The embodiments of this disclosure do not limit the specific method of determining tag audio data.

[0081] In some examples, a loss function can be used to process the target audio data and the labeled audio data to obtain a loss value. This loss value is then used to adjust the model parameters of the deep learning model until the loss converges, resulting in a trained deep learning model.

[0082] For example, adjusting the model parameters of a deep learning model using loss values ​​may include adjusting the model parameters of the target encoder and decoder in the deep learning model using loss values, and fixing the model parameters of other encoders that are different from the target encoder, so that multiple encoders can learn their respective pronunciation style attributes, avoiding confusion between multiple encoders corresponding to different pronunciation style attributes.

[0083] It should be noted that the trained deep learning model determined by the method for training the deep learning model provided in this disclosure can be used in the aforementioned audio data generation method. The technical terms involved in the method for training the deep learning model provided in this disclosure, including but not limited to sample target audio data, sample target language information, sample language features, and sample pronunciation identity features, have the same or similar meanings as the technical terms involved in the audio data generation method provided in this disclosure, including but not limited to target audio data, target language information, language features, and pronunciation identity features. Therefore, the embodiments of this disclosure will not be repeated here.

[0084] In some embodiments, the training samples further include reference language features, which are determined by one of a plurality of encoders that processes the target language information of the samples according to a reference style attribute. The reference language features can represent the audio semantic information of pronouncing the target language information of the samples according to the reference style attribute.

[0085] Training a deep learning model based on sample target audio data and labeled audio data may include: determining first loss information based on the difference between sample target audio data and labeled audio data; determining second loss information based on sample language features and reference language features; and training a deep learning model based on the first loss information and the second loss information.

[0086] The first loss information can be a first loss value, obtained by processing the sample target audio data and the label audio data using a loss function. The second loss value, representing the second loss information, can be determined by processing the sample language features and the reference language features using a loss function. The second loss information represents the difference in pronunciation style between the sample language features and the reference language features.

[0087] In one example, the optimization objective of the loss function is to maximize the difference between sample language features and reference language features. For instance, a negative cross-entropy loss function can be used to process the sample language features and reference language features, yielding a second loss information. This allows the reference language features to be used as negative samples for the sample language features. The second loss information then constrains the adjustment direction of the target encoder's model parameters during training, creating a difference in the adjustment direction of model parameters compared to encoders corresponding to other reference style attributes. This improves the target encoder's ability to extract features from target language information according to the target style attribute, avoiding confusion of pronunciation style attributes among multiple encoders that makes it difficult for language features to accurately represent the target style attribute. Therefore, training a deep learning model based on the first and second loss information can improve the ability of the target audio data output by the trained deep learning model to represent the target style attribute.

[0088] In some embodiments, determining the second loss information based on the sample language features and the reference language features may include: performing classification attribute recognition on the sample language features and the reference language features respectively to obtain classification recognition results corresponding to the sample language features and the reference language features respectively; and determining the second loss information based on multiple classification recognition results.

[0089] For example, a trained classification model is used to process sample language features and reference language features, outputting the classification results for each. The classification results indicate that the probability of the sample language feature being classified as class 1 is 0.9, and the probability of it being classified as class 2 is 0.1. The classification results for the reference language features indicate that the probability of it being classified as class 1 is 0.3, and the probability of it being classified as class 2 is 0.7. A loss function is then used to process these classification results to obtain second loss information. It should be understood that the greater the difference between the classification results of the sample language features and the reference language features, the smaller the loss value corresponding to the second loss information.

[0090] By classifying and identifying the attributes of both sample and reference language features, and then using a loss function to process the classification results to obtain a second loss information, the second loss information can more accurately represent the degree of style attribute difference between the sample and reference language features. This avoids the situation where the loss value obtained by processing the differences between hidden features is insufficient to accurately represent the degree of style attribute difference. Therefore, by using the second and first loss information to adjust the model parameters of the target encoder and decoder, the multiple encoders of the deep learning model can each extract pronunciation semantic features of language information according to their corresponding preset style attributes. This reduces the defect of multiple different pronunciation styles causing confusion in the generated audio data, and improves the accuracy and vividness of the pronunciation style in the target audio data.

[0091] In some embodiments, determining the second loss information based on multiple classification results may include: determining classification loss information based on the differences between the multiple classification results; and determining the second loss information based on the classification loss information and style difference loss information.

[0092] Style difference loss information is determined by processing the sample language features and reference language features using a loss function. Classification loss information is determined by processing the classification and recognition results corresponding to the sample language features and reference language features using a loss function.

[0093] In one example, the second loss information is determined based on the sum of style difference loss information and classification loss information. Thus, style difference loss information can be determined by detecting the degree of difference between the hidden features representing the semantics of pronunciation content output by different encoders in the deep learning model, and classification loss information can be determined using the classification results corresponding to the sample language features and the reference language features. This allows the second loss information to represent the audio semantic differences between the sample language features and the reference language features for pronunciation style attributes from multiple dimensions. This enables the second loss information to more accurately represent the target encoder's and other encoders' understanding of audio styles for different pronunciation style attributes. The second loss information can then be used to adjust the model parameters of the target encoder, improving the accuracy of the target encoder's output language features in representing the pronunciation content of the target style attributes, enhancing the accuracy of the target audio data in expressing pronunciation style attributes and the timbre of the pronunciation object, and ultimately improving the target audio quality.

[0094] In some embodiments, determining first loss information based on the difference between sample target audio data and labeled audio data may include: determining speech loss information based on the difference between sample target audio data and labeled audio data; determining pronunciation identity loss information based on the difference between sample target audio data and reference audio data; and determining first loss information based on speech loss information and pronunciation identity loss information.

[0095] Speech loss information represents the degree of difference between the target audio data and the labeled audio data. Training a deep learning model based on speech loss information allows the model to more accurately learn how the speaker pronounces the target language information according to the target style attributes, thereby improving the naturalness and credibility of the target audio data.

[0096] The reference audio data is determined by processing the reference language features and sample pronunciation identity features using a decoder. The reference audio data and the sample target audio data are determined based on the same sample pronunciation identity features, indicating that the timbre of the same sample pronunciation object is pronounced according to different pronunciation style attributes to the sample target language information.

[0097] In one example, a loss function is used to process the target audio data and reference audio data to obtain pronunciation identity loss information. Pronunciation identity loss information represents the degree of difference in timbre between the target audio data and the reference audio data. The smaller the difference in timbre, the smaller the pronunciation identity loss information; the pronunciation identity loss information is directly proportional to the degree of difference in timbre. Therefore, the pronunciation identity loss information can be used to constrain the training process of a deep learning model, enabling the deep learning model to fully learn the audio semantics of pronunciation of target language information based on the same timbre and different pronunciation style attributes. This allows the target audio data output by the trained deep learning model to accurately represent the pronunciation expression of the specified pronunciation object in different pronunciation styles. The trained deep learning model can flexibly adjust the pronunciation style attributes of the target audio data according to actual scenario requirements or user needs, maintaining a match between the timbre of the target audio data and the pronunciation timbre of the pronunciation object, thus improving the data quality of the target audio data.

[0098] Figure 6 The schematic diagram illustrates a method for training a deep learning model according to an embodiment of the present disclosure.

[0099] like Figure 6 As shown, the deep learning model includes an identity feature encoder, encoders corresponding to multiple preset style attributes, and a decoder shared by the multiple preset style attributes. The encoders corresponding to the multiple preset style attributes are: a whisper style attribute encoder corresponding to the whisper style attribute, a baseline style attribute encoder corresponding to the baseline style attribute, ..., and a high-pitched style attribute encoder corresponding to the high-pitched style attribute. The sample pronunciation identity feature can be obtained by extracting voiceprint features from the pronunciation data of the sample pronunciation object using the identity feature encoder.

[0100] For example, the target encoder corresponding to the target style attribute is a whisper style attribute encoder, and the encoders corresponding to the reference style attributes are a baseline style attribute encoder and a high-pitched style attribute encoder, respectively. The sample target language information is input into the whisper style attribute encoder, the baseline style attribute encoder, and the high-pitched style attribute encoder, respectively, and the outputs are whisper style language features, baseline style language features, and high-pitched style language features. The whisper style language features are the sample language features, while the baseline style language features and the high-pitched style language features are both reference language features.

[0101] The decoder processes whisper style language features and sample pronunciation identity features, and outputs sample target audio data. The sample target audio data represents the pronunciation of sample target language information based on the timbre of the sample pronunciation object according to the whisper style attribute.

[0102] The decoder processes baseline style language features and sample pronunciation identity features, outputting baseline style audio data as the first reference audio data. This baseline style audio data represents the pronunciation of the sample target language information based on the timbre of the sample pronunciation object and according to baseline style attributes. For example, the baseline style audio data represents the pronunciation of the sample target language information based on a smooth rhythm, tone, and sound intensity.

[0103] A decoder is used to process high-pitched language features and sample pronunciation identity features, outputting high-pitched audio data as a second reference audio data. This high-pitched audio data represents the pronunciation of the sample target language information based on the timbre of the sample pronunciation object according to the high-pitched style attribute. For example, high-pitched audio data indicates that the sample pronunciation object pronounces the sample target language information based on a high sound intensity.

[0104] A loss function is used to process the target audio data, baseline style audio data, and high-pitched style audio data to obtain pronunciation identity loss information. A loss function is then used to process the target audio data and labeled audio data to obtain speech loss information. Finally, the pronunciation identity loss information and speech loss information are fused to obtain the first loss information.

[0105] A loss function is used to process whispering style language features, baseline style language features, and high-pitched style language features to obtain style difference loss information. The loss function is then used to process the classification results of the classification model for whispering style language features, baseline style language features, and high-pitched style language features to determine classification loss information. Finally, by fusing the style difference loss information and the classification loss information, a second loss information is determined.

[0106] The model parameters of the whisper style encoder and decoder are adjusted based on the first and second loss information to obtain a trained deep learning model.

[0107] It should be understood that, Figure 6 In the example shown, the baseline style attribute encoder and the high-pitched style attribute encoder can also be used as target encoders to perform the training process, and the embodiments of this disclosure will not be described in detail here.

[0108] In some embodiments, the labeled audio data is determined based on the following operations: processing the audio data to be processed using multiple speech recognition models to obtain multiple texts to be processed; performing phonemic processing on the multiple texts to be processed to obtain phoneme sequences corresponding to each of the multiple texts to be processed; performing similarity evaluation on the multiple phoneme sequences to obtain similarity evaluation results corresponding to the audio data to be processed; and determining the labeled audio data from the multiple audio data to be processed based on the similarity evaluation results corresponding to each of the multiple audio data to be processed.

[0109] Speech recognition models can include deep learning models built based on attention network algorithms or convolutional neural network algorithms. Speech recognition models obtain the text to be processed by processing the audio data to be processed.

[0110] Multiple different speech recognition models can have different model parameters, different model structures, or different numbers of model parameters. There can be text differences between the multiple texts to be processed output by multiple speech recognition models.

[0111] In some examples, phonemicization of multiple texts to be processed can include performing grapheme-to-phoneme (G2P) conversion on the texts to convert the text characters into corresponding phoneme sequences. Each text to be processed corresponds to a different phoneme sequence, which represents the pronunciation content of the corresponding text. The phoneme data in the phoneme sequences represents semantic information related to the pronunciation content of the texts to be processed, such as stress and prosody.

[0112] For example, phonemic conversion algorithms or tools based on the Transformer architecture, encoders and decoders built on recurrent neural networks, etc., can be used to process multiple texts into phonemes.

[0113] Similarity evaluation of multiple phoneme sequences can include calculating the similarity between these sequences to determine the differences between the output phoneme sequences representing the pronunciation content after performing diverse speech recognition on the same audio data to be processed. The similarity evaluation results can then be used to represent the audio quality of the audio data to be processed in relation to the text content. By processing the similarity evaluation results of multiple audio data to be processed based on a similarity threshold, labeled audio data that meets the audio quality requirements can be identified from the multiple audio data to be processed. The recognized text obtained from speech recognition of the labeled audio data is then used as the sample target language information, facilitating the alignment of the sample target language information with the labeled audio data to achieve reliable labeling of the labeled audio data.

[0114] Figure 7 A schematic diagram illustrating the principle of determining tag audio data according to an embodiment of the present disclosure is shown.

[0115] like Figure 7As shown, the audio data to be processed is output to speech recognition model 1, speech recognition model 2, ... to speech recognition model n, resulting in multiple different texts to be processed, designated as Text to be processed 1, Text to be processed 2, ... to Text to be processed n. A phoneme conversion model is used to process Text to be processed 1, Text to be processed 2, ... to Text to be processed n, outputting phoneme sequence 1, Phoneme sequence 2, ... to Phoneme sequence n. A filtering module performs similarity assessment on phoneme sequence 1, Phoneme sequence 2, ... to Phoneme sequence n to obtain similarity evaluation results. Based on a similarity threshold, audio data meeting the requirements is filtered out as labeled audio data. A labeling module labels the qualified audio data as labeled audio data, achieving the output of temporally aligned labeled audio data and sample target language information.

[0116] In some embodiments, the multiple audio data to be processed may include pronunciation data corresponding to multiple style attributes. For example, the multiple audio data to be processed may correspond to whisper style attribute, baseline style attribute, two-dimensional pronunciation style attribute, high-pitched style attribute, and sad style attribute, respectively. To avoid the drawbacks of low stability and reliability of the recognized text due to the speech recognition of the audio data to be processed, and to avoid the drawbacks of the pronunciation content of the audio data to be processed being too complex to be used for model training, multiple different speech recognition models are used in parallel to perform multiple speech-to-text transcription processes on each audio data to be processed, so as to output multiple texts to be processed.

[0117] A phoneme transcription model is used to process the audio data to obtain a baseline phoneme sequence. Another phoneme transcription model with a larger number of parameters is then used to process the audio data and multiple texts to obtain phoneme sequences for each text. Similarity is then calculated between the baseline phoneme sequence and the phoneme sequences of the texts to obtain a similarity evaluation result for the audio data. Based on a similarity threshold, the similarity evaluation results for the audio data are filtered to obtain highly reliable labeled audio data. The phoneme sequences corresponding to the labeled audio data are aligned with the recognized text, and the style attributes of the labeled audio data are labeled to obtain training data for training the deep learning model. Thus, through multi-speech recognition, phoneme transcription, and phoneme sequence similarity filtering, the quality of the audio data to be processed is evaluated, resulting in the selection of high-quality labeled audio data.

[0118] In some embodiments, the audio data to be processed includes whispered audio data representing the pronunciation of sample target language information according to whispered style attributes. The whispered audio data is determined based on the following operations: denoising the initial audio data to obtain intermediate audio data; detecting audio data frames in the intermediate audio data to obtain a sequence of speech data frames characterizing the pronunciation of the sample pronunciation object; and denoising the speech data frame sequence to obtain the whispered audio data.

[0119] The initial audio data represents the pronunciation of the target language information by the sample pronunciation object according to the whispering style attribute. For example, the initial audio data represents the audio data of the sample pronunciation object softly reciting a bedtime story.

[0120] For example, the initial audio data with whisper style attributes is first subjected to full-segment denoising to obtain intermediate audio data. Then, Voice Activity Detection (VAD) is performed on the denoised intermediate audio data, and the intermediate audio data is segmented to segment the intermediate audio data that can clearly represent the whispered speech. This avoids deleting audio segments representing the pronunciation of the sample pronunciation object from the initial audio data, improving the segmentation stability of the speech data frames, and ensuring that the resulting speech data frame sequence can clearly and completely represent the speech content of the sample pronunciation object. The resulting speech data frame sequence is then further denoised to obtain a denoised speech data frame sequence with clear pronunciation and low ambient noise, which serves as the audio data to be processed.

[0121] Figure 8 A block diagram of an audio data generation apparatus according to an embodiment of the present disclosure is shown schematically.

[0122] like Figure 8 As shown, the audio data generation device 800 includes: a first receiving module 810, a language feature acquisition module 820, and a fusion module 830.

[0123] The first receiving module 810 is used to receive target language information.

[0124] The language feature acquisition module 820 is used to extract language features from the target language information based on the target encoder corresponding to the target style attribute among the encoders corresponding to multiple preset style attributes, and obtain language features.

[0125] The fusion module 830 is used to fuse language features and pronunciation identity features to generate target audio data. The pronunciation identity features represent the pronunciation identity attributes of the pronunciation object. The pronunciation identity features are obtained by using the identity feature encoder to extract features from the pronunciation data of the pronunciation object. The target audio data represents the pronunciation of the pronunciation object according to the target style attributes.

[0126] According to embodiments of this disclosure, the target language information includes target text and target pronunciation markers for identifying sub-language audio segments, wherein the language feature acquisition module includes: a text feature acquisition unit, a sub-language feature acquisition unit, and a language feature acquisition unit.

[0127] The text feature acquisition unit is used to encode the target text based on the text tag set to obtain text features. The text features represent the mapping relationship between the target text and the preset text tags in the text tag set.

[0128] The sub-language feature acquisition unit is used to encode the target pronunciation mark based on the sub-language pronunciation mark set to obtain sub-language features. The sub-language features represent the mapping relationship between the target pronunciation mark and the sub-language pronunciation mark in the sub-language pronunciation mark set. The sub-language pronunciation mark is used to identify the preset sub-language audio segment.

[0129] The language feature acquisition unit is used to fuse text features and paralinguistic features to determine language features.

[0130] According to embodiments of this disclosure, the audio data generation apparatus further includes a target style attribute determination module.

[0131] The target style attribute determination module is used to process the input information of the target object using a large model, determine the target style attributes, and the target language information used to respond to the input information. Among them, the target audio data is used to communicate with the target object.

[0132] According to embodiments of this disclosure, the target style attributes are determined by analyzing target language information using a large model.

[0133] According to embodiments of this disclosure, the audio data generation apparatus further includes a driver module.

[0134] The driver module is used to drive the virtual image displayed in the interactive interface to perform pronunciation actions based on the target audio data.

[0135] Figure 9 A block diagram of an apparatus for training a deep learning model according to an embodiment of the present disclosure is shown schematically.

[0136] like Figure 9 As shown, the apparatus 900 for training a deep learning model includes: a second receiving module 910, a sample language feature acquisition module 920, a generation module 930, and a training module 940. The deep learning model includes encoders corresponding to multiple preset style attributes.

[0137] The second receiving module 910 is used to receive training samples. The training samples include sample target language information, sample pronunciation identity features and label audio data. The sample pronunciation identity features represent the pronunciation identity attributes of the sample pronunciation object. The pronunciation identity features are obtained by using the identity feature encoder of the deep learning model to extract features from the pronunciation data of the sample pronunciation object. The label audio data represents the pronunciation of the sample target language information by the sample pronunciation object according to the target style attributes.

[0138] The sample language feature acquisition module 920 is used to extract features from the sample target language information by utilizing the target encoder corresponding to the target style attribute among multiple encoders, and obtain the sample language features.

[0139] The generation module 930 is used to generate sample target audio data by fusing sample language features and sample pronunciation identity features using the decoder of a deep learning model.

[0140] Training module 940 is used to train a deep learning model based on sample target audio data and labeled audio data to obtain a trained deep learning model.

[0141] According to embodiments of this disclosure, the training samples further include reference language features, which are determined by the encoder corresponding to the reference style attribute among multiple encoders processing the target language information of the sample; wherein, the training module includes: a first loss information determination unit, a second loss information determination unit, and a training unit.

[0142] The first loss information determination unit is used to determine the first loss information based on the difference between the sample target audio data and the label audio data.

[0143] The second loss information determination unit is used to determine the second loss information based on the sample language features and the reference language features, wherein the second loss information represents the pronunciation style difference between the sample language features and the reference language features.

[0144] The training unit is used to train a deep learning model based on the first loss information and the second loss information.

[0145] According to embodiments of this disclosure, the second loss information determination unit includes: a first obtaining subunit and a first determining subunit.

[0146] The first obtaining subunit is used to perform classification attribute recognition on the sample language features and the reference language features respectively, and obtain the classification recognition results corresponding to the sample language features and the reference language features respectively.

[0147] The first determining subunit is used to determine the second loss information based on multiple classification and recognition results.

[0148] According to an embodiment of this disclosure, the first determining subunit is configured to: determine classification loss information based on the differences between multiple classification recognition results; and determine second loss information based on the classification loss information and style difference loss information, wherein the style difference loss information is determined by processing sample language features and reference language features using a loss function.

[0149] According to embodiments of this disclosure, the first loss information determination unit includes: a second determination subunit, a third determination subunit, and a fourth determination subunit.

[0150] The second determining subunit is used to determine speech loss information based on the difference between the sample target audio data and the labeled audio data.

[0151] The third determining subunit is used to determine the pronunciation identity loss information based on the difference between the sample target audio data and the reference audio data. The reference audio data is determined by using a decoder to process reference language features and sample pronunciation identity features.

[0152] The fourth determining subunit is used to determine the first loss information based on the speech loss information and the pronunciation identity loss information.

[0153] According to embodiments of this disclosure, the labeled audio data is determined based on the following operations: processing the audio data to be processed using multiple speech recognition models to obtain multiple texts to be processed; performing phonemic processing on the multiple texts to be processed to obtain phoneme sequences corresponding to each of the multiple texts to be processed, wherein the phoneme sequences represent the pronunciation content of the texts to be processed; performing similarity evaluation on the multiple phoneme sequences to obtain similarity evaluation results corresponding to the audio data to be processed; and determining the labeled audio data from the multiple audio data to be processed based on the similarity evaluation results corresponding to each of the multiple audio data to be processed.

[0154] According to embodiments of this disclosure, the audio data to be processed includes whispered audio data representing the pronunciation of sample target language information according to whispered style attributes. The whispered audio data is determined based on the following operations: denoising the initial audio data to obtain intermediate audio data; detecting the audio data frames in the intermediate audio data to obtain a speech data frame sequence representing the pronunciation of the sample pronunciation object; and denoising the speech data frame sequence to obtain the whispered audio data.

[0155] Figure 10 A schematic block diagram of an artificial intelligence agent according to an embodiment of the present disclosure is shown.

[0156] In embodiments of this disclosure, such as Figure 10 As shown, the AI ​​agent 1000 may include an input module 1010, a processing module 1020, and an output module 1030.

[0157] Input module 1010 is used to receive input information;

[0158] The processing module 1020 is used to determine the target task based on the input information received by the input module, determine the large model based on the target task, and obtain output information by calling the large language model to execute the audio data generation method provided in the embodiments of this disclosure, or by calling the large model to execute the method for training a deep learning model provided in the embodiments of this disclosure.

[0159] Output module 1030 is used to output the output information obtained by the processing module.

[0160] According to embodiments of this disclosure, the input module 1010 is responsible for receiving or sensing information such as queries, requests, instructions, signals, or data from the outside world (e.g., users or the external environment), and converting it into a format that the AI ​​agent 1000 can understand and process. The input module 1010 is the primary link for the AI ​​agent 1000 to interact with the outside world, enabling the AI ​​agent 1000 to efficiently and accurately obtain necessary "sensory" information from the outside world and respond to this information.

[0161] In the example, input module 1010 can input target language information, as described above.

[0162] In the example, processing module 1020 is the core support for the AI ​​agent 1000's ability to handle complex tasks. Processing module 1020 can execute the audio data generation method and the deep learning model training method described above.

[0163] In the example, the performance of the processing module 1020 is closely related to the large model on which the AI ​​agent 1000 is based. To fully leverage the capabilities of the large model, the internal structure of the processing module 1020 can be designed to be highly configurable and scalable to handle various types of tasks and requirements in real-world scenarios.

[0164] In the example, after the AI ​​agent 1000 acquires the required voice, the processing module 1020 can use a large model to process the target language information and pronunciation identity features to obtain the target audio data, and then pass the target audio data to the output module 1030.

[0165] Understandably, while large language models possess excellent language understanding and generation capabilities, like humans, their ability to solve tasks is limited without the aid of any tools. Once the AI ​​agent 1000 is given the ability to invoke tools, it can perform tasks such as using a calculator to complete mathematical calculations, using Python to perform data analysis, and using a search engine to create weather forecasts.

[0166] In the example, output module 1030 can output the target audio data described above.

[0167] The AI ​​agent 1000 according to the embodiments of this disclosure can simply and effectively improve the level of intelligence, as well as enhance flexibility and versatility.

[0168] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0169] According to an embodiment of the present disclosure, an electronic device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method described above.

[0170] According to embodiments of the present disclosure, a non-transitory computer-readable storage medium stores computer instructions, wherein the computer instructions are used to cause a computer to perform the method described above.

[0171] According to an embodiment of this disclosure, a computer program product includes a computer program that, when executed by a processor, implements the method described above.

[0172] Figure 11 A schematic block diagram of an example electronic device is shown that can be used to implement the audio data generation method and the method for training a deep learning model according to embodiments of the present disclosure. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0173] like Figure 11 As shown, the electronic device 1100 includes a computing unit 1101, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1102 or a computer program loaded into a random access memory (RAM) 1103 from a storage unit 1108. The RAM 1103 may also store various programs and data required for the operation of the electronic device 1100. The computing unit 1101, ROM 1102, and RAM 1103 are interconnected via a bus 1104. An input / output (I / O) interface 1105 is also connected to the bus 1104.

[0174] Multiple components in electronic device 1100 are connected to I / O interface 1105, including: input unit 1106, such as keyboard, mouse, etc.; output unit 1107, such as various types of displays, speakers, etc.; storage unit 1108, such as disk, optical disk, etc.; and communication unit 1109, such as network card, modem, wireless transceiver, etc. Communication unit 1109 allows electronic device 1100 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0175] The computing unit 1101 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 1101 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 1101 performs the various methods and processes described above, such as audio data generation methods and methods for training deep learning models. For example, in some embodiments, the audio data generation methods and methods for training deep learning models can be implemented as computer software programs tangibly contained in a machine-readable medium, such as storage unit 1108. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 1100 via ROM 1102 and / or communication unit 1109. When the computer program is loaded into RAM 1103 and executed by the computing unit 1101, one or more steps of the audio data generation methods and methods for training deep learning models described above can be performed. Alternatively, in other embodiments, the computing unit 1101 may be configured in any other suitable manner (e.g., by means of firmware) to perform an audio data generation method or a method for training a deep learning model.

[0176] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0177] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0178] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0179] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0180] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0181] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, distributed system servers, or servers incorporating blockchain technology.

[0182] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0183] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A method for generating audio data, comprising: Receive target language information; Based on the target encoder corresponding to the target style attribute from among the encoders corresponding to multiple preset style attributes, the target language information is feature extracted to obtain language features; as well as The language features and pronunciation identity features are fused to generate target audio data. The pronunciation identity features represent the pronunciation identity attributes of the pronunciation object. The pronunciation identity features are obtained by using an identity feature encoder to extract features from the pronunciation data of the pronunciation object. The target audio data represents the pronunciation of the pronunciation object according to the target style attributes in the target language information.

2. The method according to claim 1, wherein, The target language information includes target text and target pronunciation markers used to identify sub-language audio segments. The step of extracting features from the target language information based on the target encoder corresponding to the target style attribute from among the encoders corresponding to multiple preset style attributes includes: The target text is encoded based on a set of text tags to obtain text features, which characterize the mapping relationship between the target text and preset text tags in the set of text tags. Encoding the target pronunciation markers based on a set of secondary language pronunciation markers yields secondary language features, wherein the secondary language features characterize the mapping relationship between the target pronunciation markers and the secondary language pronunciation markers in the set of secondary language pronunciation markers, and the secondary language pronunciation markers are used to identify preset secondary language audio segments; and The language features are determined by fusing the text features with the sub-language features.

3. The method according to claim 1, further comprising: The input information of the target object is processed using a large model to determine the target style attribute and the target language information used to respond to the input information, wherein the target audio data is used to engage in dialogue with the target object.

4. The method according to claim 1, wherein, The target style attribute is determined by analyzing the target language information using the large model.

5. The method according to claim 1 or 3, further comprising: The virtual avatar displayed on the interactive interface is driven to perform pronunciation actions based on the target audio data.

6. A method for training a deep learning model, the deep learning model comprising encoders corresponding to a plurality of preset style attributes, the method comprising: The training samples are received, including sample target language information, sample pronunciation identity features, and labeled audio data. The sample pronunciation identity features represent the pronunciation identity attributes of the sample pronunciation object. The deep learning model's identity feature encoder is used to extract features from the pronunciation data of the sample pronunciation object to obtain the sample pronunciation identity features. The labeled audio data represents the pronunciation of the sample pronunciation object according to the target style attributes of the sample target language information. The target encoder, which corresponds to the target style attribute among the multiple encoders, is used to extract features from the sample target language information to obtain sample language features; The sample language features and sample pronunciation identity features are fused using the decoder of the deep learning model to generate sample target audio data; as well as The deep learning model is trained based on the target audio data and the labeled audio data to obtain the trained deep learning model.

7. The method according to claim 6, wherein, The training samples also include reference language features, which are determined by the encoder corresponding to the reference style attribute among the multiple encoders processing the target language information of the samples; The step of training the deep learning model based on the sample target audio data and the labeled audio data includes: The first loss information is determined based on the difference between the sample target audio data and the labeled audio data; A second loss information is determined based on the sample language features and the reference language features, wherein the second loss information represents the pronunciation style difference between the sample language features and the reference language features; and The deep learning model is trained based on the first loss information and the second loss information.

8. The method according to claim 7, wherein, The second loss information is determined based on the sample language features and the reference language features, including: Classification attribute recognition is performed on the sample language features and the reference language features respectively to obtain classification recognition results corresponding to the sample language features and the reference language features respectively; and The second loss information is determined based on multiple classification and recognition results.

9. The method according to claim 8, wherein, Determining the second loss information based on multiple classification and recognition results includes: Classification loss information is determined based on the differences between multiple classification and recognition results; and The second loss information is determined based on the classification loss information and the style difference loss information, wherein the style difference loss information is determined by processing the sample language features and the reference language features using a loss function.

10. The method according to claim 7, wherein, Based on the difference between the sample target audio data and the labeled audio data, the first loss information is determined, including: Based on the difference between the sample target audio data and the labeled audio data, speech loss information is determined; Based on the difference between the sample target audio data and the reference audio data, pronunciation identity loss information is determined, wherein the reference audio data is determined by processing the reference language features and the sample pronunciation identity features using the decoder; and The first loss information is determined based on the speech loss information and the pronunciation identity loss information.

11. The method according to claim 6, wherein, The tagged audio data is determined based on the following operations: Multiple speech recognition models are used to process the audio data to be processed, resulting in multiple texts to be processed. Phonetic processing is performed on multiple texts to be processed to obtain phoneme sequences corresponding to each of the multiple texts to be processed, wherein the phoneme sequences represent the pronunciation content of the texts to be processed; A similarity assessment is performed on multiple phoneme sequences to obtain a similarity assessment result corresponding to the audio data to be processed; as well as The tagged audio data is determined from the multiple audio data to be processed based on the similarity evaluation results corresponding to each of the multiple audio data to be processed.

12. The method according to claim 11, wherein, The audio data to be processed includes whispered audio data representing the pronunciation of the sample target language information according to whispered style attributes, and the whispered audio data is determined based on the following operations: The initial audio data is denoised to obtain intermediate audio data; The audio data frames in the intermediate audio data are detected to obtain a sequence of speech data frames representing the pronunciation of the sample pronunciation object; as well as The speech data frame sequence is denoised to obtain the whispered audio data.

13. An audio data generation apparatus, comprising: The first receiving module is used to receive target language information; The language feature acquisition module is used to extract language features from the target language information based on the target encoder corresponding to the target style attribute among the encoders corresponding to multiple preset style attributes, and obtain language features. The fusion module is used to fuse the language features and pronunciation identity features to generate target audio data. The pronunciation identity features represent the pronunciation identity attributes of the pronunciation object. The pronunciation identity features are obtained by using an identity feature encoder to extract features from the pronunciation data of the pronunciation object. The target audio data represents the pronunciation object pronouncing the target language information according to the target style attribute.

14. An apparatus for training a deep learning model, the deep learning model including encoders corresponding to a plurality of preset style attributes, the apparatus comprising: The second receiving module is used to receive training samples, which include sample target language information, sample pronunciation identity features, and labeled audio data. The sample pronunciation identity features represent the pronunciation identity attributes of the sample pronunciation object. The deep learning model's identity feature encoder extracts features from the pronunciation data of the sample pronunciation object to obtain the sample pronunciation identity features. The labeled audio data represents the pronunciation of the sample pronunciation object according to the target style attributes of the sample target language information. The sample language feature acquisition module is used to extract features from the sample target language information using the target encoder that corresponds to the target style attribute among the multiple encoders, and obtain sample language features; as well as The generation module is used to fuse the sample language features and sample pronunciation identity features using the decoder of the deep learning model to generate sample target audio data; as well as The training module is used to train the deep learning model based on the sample target audio data and the label audio data to obtain the trained deep learning model.

15. An intelligent agent of artificial intelligence, comprising: The input module is used to receive input information; The processing module is configured to determine a target task based on the input information received by the input module, determine a large model based on the target task, and obtain output information by calling the large model to execute the method of any one of claims 1 to 12. An output module is used to output the output information obtained by the processing module.

16. An electronic device comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1 to 12.

17. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1 to 12.

18. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1 to 12.

Images