Method, apparatus, device and storage medium for audio generation

Abstract

Embodiments of the present disclosure relate to a method, apparatus, device and storage medium for audio generation. The method proposed herein comprises: determining a text segment to be processed from text content; constructing context information associated with the text segment based on associated content of the text segment in the text content, the associated content comprising at least one of: a preceding segment of the text segment; a subsequent segment of the text segment; and providing the text segment and the context information to a model to generate audio content corresponding to the text segment, wherein at least one expressive attribute of the audio content is determined based on the context information. In this way, embodiments of the present disclosure can improve the accuracy of at least one expressive attribute of the generated audio content by means of the context information associated with the text segment, and thus can improve the quality of the audio content.

Description

Technical Field

[0001] The exemplary embodiments disclosed herein relate generally to the field of computers, and more particularly to methods, apparatus, devices, and computer-readable storage media for audio generation. Background Technology

[0002] With the development of computer technology, some platforms and applications provide users with various audio content, bringing them numerous conveniences. For example, some platforms or applications can provide users with audio content corresponding to certain text content, allowing users to not only view this text content but also listen to its corresponding audio content. How to improve the quality of audio generation is a key concern. Summary of the Invention

[0003] In a first aspect of this disclosure, a method for audio generation is provided. The method includes: determining a text segment to be processed from text content; constructing contextual information associated with the text segment based on associated content within the text content, the associated content including at least one of the following: a preceding segment of the text segment; a subsequent segment of the text segment; and providing the text segment and the contextual information to a model to generate audio content corresponding to the text segment, wherein at least one expressive attribute of the audio content is determined based on the contextual information.

[0004] In a second aspect of this disclosure, an apparatus for audio generation is provided. The apparatus includes: a first determining module configured to determine a text segment to be processed from text content; a constructing module configured to construct contextual information associated with the text segment based on associated content within the text content, the associated content including at least one of the following: a preceding segment of the text segment; a subsequent segment of the text segment; and a providing module configured to provide the text segment and the contextual information to a model to generate audio content corresponding to the text segment, wherein at least one expressive attribute of the audio content is determined based on the contextual information.

[0005] In a third aspect of this disclosure, an electronic device is provided. The device includes at least one processing unit; and at least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit. When executed by the at least one processing unit, the instructions cause the device to perform the method of the first aspect.

[0006] In a fourth aspect of this disclosure, a computer-readable storage medium is provided. The computer-readable storage medium stores a computer program that can be executed by a processor to implement the method of the first aspect.

[0007] It should be understood that the content described in this content section is not intended to limit the key or essential features of the embodiments of this disclosure, nor is it intended to restrict the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0008] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein:

[0009] Figure 1 A schematic diagram is shown of an example environment in which embodiments of the present disclosure may be implemented;

[0010] Figure 2 A schematic block diagram illustrating an example process for generating audio content according to some embodiments of the present disclosure is shown;

[0011] Figure 3 A flowchart illustrating an example process for audio generation according to some embodiments of this disclosure is shown;

[0012] Figure 4 A schematic structural block diagram of an example apparatus for audio generation according to some embodiments of the present disclosure is shown; and

[0013] Figure 5 A block diagram of an electronic device capable of implementing several embodiments of the present disclosure is shown. Detailed Implementation

[0014] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0015] It should be noted that the headings of any section / subsection provided herein are not limiting. Various embodiments are described throughout this document, and embodiments of any type may be included under any section / subsection. Furthermore, embodiments described in any section / subsection may be combined in any way with any other embodiments described in the same section / subsection and / or different sections / subsections.

[0016] In the description of embodiments of this disclosure, the term "comprising" and similar terms should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". Other explicit and implicit definitions may also be included below. The terms "first", "second", etc., may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0017] The embodiments of this disclosure may involve user data, data acquisition, and / or use. All of these aspects comply with applicable laws, regulations, and relevant provisions. In the embodiments of this disclosure, all data collection, acquisition, processing, manipulation, forwarding, and use are conducted with the user's knowledge and confirmation. Accordingly, in implementing the embodiments of this disclosure, the type, scope of use, and usage scenarios of any data or information that may be involved should be communicated to the user and their authorization obtained in accordance with relevant laws and regulations through appropriate means. The specific methods of notification and / or authorization may vary depending on the actual situation and application scenario, and the scope of this disclosure is not limited in this respect.

[0018] In this specification and the embodiments, any processing of personal information will be carried out only under the premise of legality (such as obtaining the consent of the personal information subject, or being necessary for the performance of a contract), and will only be carried out within the scope stipulated or agreed upon. A user's refusal to process personal information other than that necessary for basic functions will not affect the user's use of basic functions.

[0019] As mentioned above, with the development of computer technology, some platforms or applications provide users with various audio content, bringing them many conveniences. For example, some platforms or applications can provide users with audio content corresponding to certain text content, allowing users to not only view this text content but also listen to its corresponding audio content.

[0020] Embodiments of this disclosure propose an audio generation scheme. According to this scheme, a text segment to be processed can be determined from text content. Further, contextual information associated with the text segment can be constructed based on the associated content within the text content, the associated content including at least one of the following: a preceding segment of the text segment; a subsequent segment of the text segment. Additionally, the text segment and contextual information can be provided to the model to generate audio content corresponding to the text segment, wherein at least one expressive attribute of the audio content is determined based on the contextual information.

[0021] Based on this approach, at least one expressive attribute of the audio content in the embodiments of this disclosure can be determined based on contextual information associated with the text fragment, without needing to analyze the text fragment through limited tagging or classification tools. This improves the accuracy of at least one expressive attribute, thereby enhancing the quality of the audio content. Furthermore, based on such contextual information, the embodiments of this disclosure can also support a more accurate understanding of semantic information and other details of the text fragment, thus further improving the accuracy of at least one expressive attribute and consequently further enhancing the quality of the generated audio content.

[0022] Therefore, embodiments of this disclosure can improve the accuracy of at least one expressive attribute of the generated audio content by using contextual information associated with text fragments, thereby improving the quality of the audio content.

[0023] The following section provides a detailed description of various example implementations of this scheme, with reference to the accompanying drawings.

[0024] Example Environment

[0025] Figure 1 A schematic diagram of an example environment 100 in which embodiments of the present disclosure can be implemented is shown. For example... Figure 1 As shown, example environment 100 may include electronic device 110.

[0026] In this example environment 100, the electronic device 110 can support calling model 120 to generate audio content. Model 120 can be any suitable generative model application for audio generation, which can generate corresponding audio content (e.g., audio 140) based on input text (e.g., text 130).

[0027] In some embodiments, electronic device 110 may include a terminal device and / or a server to generate audio. Model 120 may be deployed in such a terminal device or server to support the generation of audio content. Model 120 may also be deployed in other electronic devices and supported by being invoked by electronic device 110 via, for example, an interface, to implement the process of generating audio content. The terminal device may be any type of mobile terminal, fixed terminal, or portable terminal, including mobile phones, desktop computers, laptop computers, notebook computers, netbook computers, tablet computers, media computers, multimedia tablets, handheld computers, portable gaming terminals, VR / AR devices, personal communication system (PCS) devices, personal navigation devices, personal digital assistants (PDAs), audio / video players, digital cameras / camcorders, positioning devices, television receivers, radio receivers, e-book devices, gaming devices, or any combination of the foregoing, including accessories and peripherals of these devices or any combination thereof.

[0028] A server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks, and big data and artificial intelligence platforms. Servers can also include computing systems / servers, such as mainframes, edge computing nodes, and computing devices in cloud environments, etc.

[0029] A communication connection can be established between the terminal device and the server. This communication connection can be established via wired or wireless means. The communication connection may include, but is not limited to, Bluetooth, mobile network, Universal Serial Bus (USB), and Wireless Fidelity (WiFi) connections, and the embodiments of this disclosure are not limited in this respect. In the embodiments of this disclosure, the terminal device and the server can achieve signaling interaction through the communication connection between them.

[0030] It should be understood that the structure and function of the various elements in environment 100 are described for illustrative purposes only and do not imply any limitation on the scope of this disclosure.

[0031] The following description will continue with reference to the accompanying drawings, which will provide some exemplary embodiments of this disclosure.

[0032] Example audio generation process

[0033] The following will be referenced Figure 2 This document describes some example audio generation processes according to embodiments of the present disclosure. Figure 2 A schematic block diagram of an example process 200 for generating audio content according to some embodiments of the present disclosure is shown. Process 200 may be implemented, for example, in... Figure 1 The electronic device shown is located at 110.

[0034] In some embodiments, such a process 200 can be used in text-to-speech scenarios. As an example, such text can include content from any suitable literary work, including but not limited to novels, scripts, etc. As an example, such text can include dialogue (e.g., dialogue between multiple characters), narration, and other text content in various language expressions.

[0035] To improve reading efficiency, some traditional solutions use classification models to predict labels such as sentiment, paralinguistics, and speaker for each sentence of text. Based on these label predictions, corresponding audio segments are generated sentence by sentence and then spliced ​​together to form the audio corresponding to the text. However, such labels usually need to be designed manually in advance, limiting them to a finite set of labels and human experience, resulting in low-quality audio generated based on these labels. In view of this, embodiments of this disclosure provide an audio generation scheme that can improve audio quality, which will be described below in conjunction with the appendix. Figure 2 The audio generation scheme is illustrated by taking the audio corresponding to the generated text content 211 as an example.

[0036] In some embodiments, according to Figure 2 In the process 200 shown, electronic device 110 can acquire a text fragment to be processed from text content 211. As an example, such text content 211 could be a complete chapter from a literary work (e.g., the entire content of a chapter in a novel). Such chapter content could include more than a specified number of sentences. Thus, such chapter content could have a continuous storyline, a more complete plot, a more complete scene, etc., making such text fragments more relevant to their preceding and / or subsequent text content.

[0037] To improve the integrity of the audio generated based on text content 211, electronic device 110 can set a sliding window to segment the text content 211 for processing the segmented text fragments separately. As an example, such a sliding window can be configured to slide in units of sentences; for instance, such a sliding window can be configured to slide at a fixed preset number of sentences. As an example, the text fragments to be processed obtained from text content 211 can be... Figure 2 The text fragment 221 shown below is based on Figure 2Taking the text fragment 221 shown as an example to generate the corresponding audio content 271, the specific implementation of generating the audio content corresponding to the text fragment is described in an exemplary manner.

[0038] As an example, embodiments of this disclosure can generate multiple audio contents corresponding to multiple text segments based on a similar implementation to generating audio content 271, and then concatenate the multiple audio contents corresponding to multiple text segments to generate target audio content corresponding to text content 211. This disclosure is not intended to limit the specific implementation of such concatenation, and will not elaborate on it.

[0039] In some embodiments, the electronic device 110 determines associated content of the text segment 221 from the text content 211. Such associated content can be a preceding segment of the text segment 221 (e.g., the previous text segment 222 of the text segment 221) and / or a subsequent segment of the text segment 221 (e.g., the next text segment 223 of the text segment 221). As an example, such preceding and / or subsequent segments of the text segment 221 can be associated with the text segment 221. For example, such associations can include having a continuous storyline, having continuous dialogue content, etc. As an example, such preceding and / or subsequent segments can be determined from the text content 211 via a sliding window.

[0040] To enable the model to more accurately understand text fragment 221, electronic device 110 can determine the associated content of text fragment 221 from text content 211 based on a first window size. This first window size can be the sliding window length described above. As an example, electronic device 110 can first determine the length of text fragment 221 and subtract the length of text fragment 221 from the first window size to obtain the remaining length. Furthermore, electronic device 110 can allocate this remaining length to preceding and subsequent fragments of text fragment 221. Thus, embodiments of this disclosure can construct contextual information associated with text fragment 221 based on its associated content.

[0041] As an example, the length of such a sliding window and the size of the first window can be in units of word count. For example, if the sliding window length includes N words and the text segment 221 includes M words, then the electronic device 110 can allocate (NM) / 2 words to the preceding and subsequent segments of the text segment 221 respectively, to obtain a preceding segment containing at most (NM) / 2 words and / or a subsequent segment containing at most (NM) / 2 words. As an example, the electronic device 110 can also allocate the length of the preceding and subsequent segments of the text segment 221 based on a preset ratio, so that the lengths of the preceding and subsequent segments meet such a preset ratio.

[0042] In some embodiments, in order to ensure the accuracy of the model processing of text segments as described below, such a first window size may be smaller than the second window size used to construct context information during the training of the model. For example, such a second window size may indicate the maximum length of the preceding and subsequent segments of the training sample segments during the training of the model.

[0043] In some embodiments, such a text fragment 221 may be the beginning or end of text content 211. For example, text fragment 221 may include the first or last paragraph of text content 211. Thus, electronic device 110 may construct context information associated with text fragment 221 based solely on the earlier fragments of text fragment 221 or solely on the later fragments of text fragment 221. As an example, the embodiments disclosed below are exemplified by constructing context information associated with text fragment 221 based on the later fragments and the earlier fragments of text fragment 221; other cases can be referred to the embodiments of this disclosure.

[0044] Based on this, the electronic device 110 can provide the model 120 with text fragment 221 and contextual information to generate audio content corresponding to the text fragment 221. As an example, such a model 120 may include at least... Figure 2 The encoding unit 240, autoregressive unit 250, diffusion unit 260, and decoding unit 270 are shown. Furthermore, when the electronic device 110 provides the model 120 with text fragment 221 and contextual information, the model 120 can process such text fragment 221 and contextual information through the included multiple units, thereby generating audio content corresponding to the text fragment 221.

[0045] Furthermore, the electronic device 110 can provide the text fragment 221 and context information to the encoding unit 240, and obtain the input text embedding 241 output by the encoding unit 240. As an example, such an encoding unit 240 can be implemented by any suitable encoder, which can, for example, generate a corresponding text embedding based on the input text. As an example, such an input text embedding 241 may include a first part corresponding to the prior fragment 222 (e.g., text embedding 232), a second part corresponding to the text fragment 221 (e.g., text embedding 231), and a third part corresponding to the subsequent content (e.g., text embedding 233).

[0046] To improve the accuracy of the model in recognizing the text fragment 221 to be processed, the second part corresponding to the text fragment 221 can be associated with a preset identifier. As an example, such an identifier can identify the start point (e.g., such an identifier could include a start point identifier) ​​and the end point (e.g., such an identifier could include an end point identifier) ​​of the second part. To improve the accuracy of model processing, if the associated content of the text fragment 221 does not include a preceding fragment or a subsequent fragment, the electronic device 110 can also associate the second part corresponding to the text fragment 221 with such a start point identifier and an end point identifier.

[0047] Furthermore, the electronic device 110 can provide the input text embedding 241 to the autoregressive unit 250 of the model 120 to generate a first audio feature 251. As an example, such a first audio feature 251 can be a discrete feature. As an example, such an autoregressive unit 250 can be implemented by any suitable tool or model layer, which can, for example, generate corresponding audio features based on the input text embedding. As an example, the first audio feature 251 generated by such an autoregressive unit 250 can be a discrete feature. For example, such an autoregressive unit 250 can generate corresponding feature elements item by item based on the input text embedding.

[0048] In some embodiments, the decoding strategy of such an autoregressive unit 250 may employ a sampling decoding strategy based on a predetermined number of samples. To improve the diversity of decoding, the decoding strategy of such an autoregressive unit 250 may employ a decoding strategy based on a predetermined number of samples and a preset temperature parameter. The following description uses a decoding strategy based on a predetermined number of samples and a temperature parameter as an example. Embodiments of such a decoding strategy based on a predetermined number of samples can be found in the embodiments described below, except that the step of updating the probability distribution using the preset temperature parameter is not performed.

[0049] As an example, when electronic device 110 provides input text embedding 241 to the autoregressive unit 250 of model 120, it can determine the probability distribution of the target output token based on such input text embedding 241. Taking the target output token as the i-th element, the probability distribution p... i Let p represent the probability of the i-th element, and use a preset temperature parameter T to update the probability distribution of the target input token (for example, such a temperature parameter T can adjust the shape of such a probability distribution). i It can be expressed as formula (1).

[0050]

[0051] Therefore, the electronic device 110 can determine multiple candidate elements for the i-th element through such an autoregressive unit 250. Furthermore, the electronic device 110 can sort these multiple candidate elements using such an autoregressive unit 250. For example, the probability distribution of the sorted multiple candidate elements can be expressed as formula (2).

[0052] p′=p′1,p′2,…,p′ n (2)

[0053] Furthermore, the electronic device 110 can determine a predetermined number (e.g., k) of candidate elements from a sorted pool of candidate elements. To improve sampling accuracy, the probability distribution of such a set of candidate elements can be updated. As an example, such an update may include normalization or other suitable probability distribution adjustments. Taking the normalization of the probability distribution of such a set of candidate elements so that the sum of the sampling probabilities of such a set of candidate elements is 1 as an example, the probability distribution of such a set of candidate elements can be adjusted to satisfy the following formula (3).

[0054]

[0055] Based on the updated sampling probability, the electronic device 110 can sample the target element corresponding to the i-th element from such a set of candidate elements through the autoregressive unit 250.

[0056] Based on this, the embodiments of this disclosure can increase the diversity of sampling through such a predetermined number, thereby improving the quality of the audio content generated by the subsequent model 120. Furthermore, the embodiments of this disclosure can balance the diversity and accuracy of sampling through such a preset temperature parameter, thereby further improving the quality of the audio content generated by the subsequent model 120.

[0057] In some embodiments, since the first audio feature 251 is a discrete feature, the audio content directly generated based on the first audio feature 251 has low quality. Therefore, when the electronic device 110 generates the first audio feature 251 through the autoregressive unit 250, it can provide the first audio feature 251 to the diffusion unit 260 to supplement the first audio feature 251 with semantic information and audio expression information, thereby generating a second audio feature 261. As an example, the second audio feature 261 can be a continuous feature. As an example, the diffusion unit 260 can be implemented by any suitable tool or model layer, which can, for example, generate corresponding continuous audio features for the input discrete audio feature diffusion information.

[0058] Furthermore, the electronic device 110 can provide such a second audio feature 261 to the decoding unit 270 to generate audio content 271 corresponding to the text segment 221. As an example, such a decoding unit 270 can be implemented by any suitable decoder, which can, for example, decode and generate the corresponding audio content based on the input audio features. Thus, embodiments of this disclosure can determine at least one expressive attribute of the generated audio content 271 based on such contextual information.

[0059] As an example, at least one expressive attribute can be associated with the character, performance attribute, speech expression type, etc., of such text fragment 221. For instance, based on such contextual information, embodiments of this disclosure can determine that such text fragment 221 belongs to dialogue-type text or narration-type text, and thus can generate audio content for the performed character or audio content for the performed narration. As another example, based on such contextual information, embodiments of this disclosure can determine the character corresponding to such text fragment 221; for example, such text fragment 221 can be dialogue content of a target character, and thus can generate audio content for the sound effects of the target character.

[0060] It should be understood that Figure 2 The audio diagram of the audio content 271 is merely an illustrative description of an embodiment of this disclosure and is not intended to limit the specific audio content of such audio content 271. Furthermore, the specific content items mentioned in the embodiments of this disclosure (e.g., length, number of sentences, number of words, probability distribution identifier, element identifier, temperature parameter identifier, predetermined number identifier, etc.) are merely exemplary and are not intended to limit the scope of this disclosure.

[0061] Based on this approach, embodiments of this disclosure can improve the accuracy of at least one expressive attribute of the generated audio content by using contextual information associated with text fragments, thereby improving the quality of the audio content.

[0062] Example model training process

[0063] To improve the quality of the audio content generated by the model, such a model 120 can be pre-trained. The following description further illustrates an example training process for such a model 120, assuming it is trained at electronic device 110. It should be understood that such a model 120 can also be trained on other devices and can support the implementation of the audio generation process of this disclosure by being invoked or deployed. This disclosure only uses the example of training model 120 at electronic device 110 for illustrative purposes.

[0064] In some embodiments, the electronic device 110 can segment sample text content to obtain sample text fragments. As an example, such sample text content may include any suitable sample literary work content, which may include, for example, text fragments in multiple language expressions, dialogue content related to one or more characters, narration content, etc.

[0065] To improve model training performance, the electronic device 110 can segment the sample text content into multiple candidate segments based on reference information, and then determine the final sample text segment based on these candidate segments. As an example, such reference information can improve the semantic integrity of the segmented sample text segments.

[0066] To improve the quality of segmentation, such reference information can indicate the grammatical structure of the sample text content, which in turn indicates multiple sentences within the sample text content. For example, such sample text content can be chapter text. Thus, electronic device 110 can segment such chapter text based on its grammatical structure to obtain sample text fragments containing a predetermined number of sentences. Alternatively or additionally, such reference information can also indicate semantic information of the sample text content, which can indicate the speech expression type corresponding to different content within the sample text content. For example, such speech expression types include dialogue types, narration types, etc. For instance, electronic device 110 can segment a sample text fragment according to the dialogue content corresponding to a character. Or, for example, electronic device 110 can segment dialogue content and narration content into different sample text fragments, etc. In this way, model 120 can more accurately understand the textual information of such sample text fragments.

[0067] In some embodiments, if a candidate segment is too long, it can be reasonably segmented to determine a sample text segment. Specifically, in response to a target candidate segment among multiple candidate segments having a length greater than a threshold, the electronic device 110 can segment such a target candidate segment based on a preset length to determine such a sample text segment. As an example, such a preset length can have a preset constraint relationship with such a threshold. Such a preset length, such a threshold, and such a constraint relationship can all be set according to actual project needs. The embodiments of this disclosure are not intended to limit the specific values ​​of such a preset length and such a threshold, or the constraint relationship between them.

[0068] Furthermore, the electronic device 110 can obtain training association content of sample text segments from the sample text content based on a preset window size (e.g., a second window size). Furthermore, the electronic device 110 can construct training context information based on the training association content. Additionally, the electronic device 110 can train a model 120 based on sample text segments, corresponding sample audio segments, and training context information. As an example, the specific implementation of such a training process can be found in a similar implementation method during the application of model 120 in the audio generation process described above; the embodiments of this disclosure will not be repeated here.

[0069] Based on this approach, embodiments of this disclosure can provide more information for model training based on sample context information associated with sample text fragments, without relying on limited labeling or classification tools to support model analysis of such sample text fragments, thereby improving the quality of the trained model. Furthermore, embodiments of this disclosure, based on such context information, can also enable the model to more accurately understand the semantics and other information of text fragments, thereby further improving the quality of the trained model.

[0070] Therefore, embodiments of this disclosure can assess the quality of the trained model by using sample context information associated with sample text fragments.

[0071] Example process

[0072] Figure 3 A flowchart of an example audio generation process 300 according to some embodiments of the present disclosure is shown. Process 300 can be implemented at electronic device 110. Reference is made below. Figure 1 Describe the process 300.

[0073] like Figure 3 As shown, in box 310, electronic device 110 determines the text segment to be processed from the text content.

[0074] At box 320, electronic device 110 constructs contextual information associated with the text fragment based on the associated content of the text fragment in the text content. The associated content includes at least one of the following: the preceding fragment of the text fragment; the subsequent fragment of the text fragment.

[0075] At box 330, electronic device 110 provides the model with text fragments and contextual information to generate audio content corresponding to the text fragments, wherein at least one expressive attribute of the audio content is determined based on the contextual information.

[0076] In some embodiments, providing a text fragment and contextual information to the model to generate audio content corresponding to the text fragment includes: generating an input text embedding based on the text fragment and contextual information; providing the input text embedding to the model's autoregressive unit to generate a first audio feature; providing the first audio feature to the model's diffusion unit to generate a second audio feature; and providing the second audio feature to the model's decoding unit to generate audio content corresponding to the text fragment.

[0077] In this way, embodiments of the present disclosure can generate input text embeddings based on text fragments and contextual information, thereby extracting and decoding features from such input text embeddings, thus obtaining more accurate and comprehensive text information corresponding to the text fragments, and improving the quality of the generated audio content.

[0078] In some embodiments, the input text embedding includes: a first portion corresponding to a prior fragment; a second portion corresponding to a text fragment, wherein the second portion is associated with a preset identifier; and a third portion corresponding to subsequent content.

[0079] In this way, embodiments of the present disclosure can more accurately distinguish the feature information of text fragments from the feature information of the context used for reference, thereby improving the accuracy of subsequent feature extraction.

[0080] In some embodiments, the first audio feature is a discrete feature, and the second audio feature is a continuous feature.

[0081] In this way, embodiments of this disclosure can obtain more comprehensive feature information, thereby further improving the quality of the generated audio content.

[0082] In some embodiments, providing an input text embedding to the autoregressive unit of the model to generate a first audio feature includes: determining a probability distribution of the model with respect to a target output token based on the input text embedding; sorting a plurality of candidate elements based on the probability distribution; determining a predetermined number of candidate elements from the sorted plurality of candidate elements; updating the sampling probability of the candidate elements; and sampling a target element corresponding to the target output token from the candidate elements based on the updated sampling probability.

[0083] In this way, embodiments of the present disclosure can improve the diversity of sampling by using such a predetermined number of samples, thereby enabling audio generation diversity and reducing the probability of errors in audio generation, thus improving the quality of audio content generated by subsequent models.

[0084] In some embodiments, before sorting multiple candidate elements based on the probability distribution, process 300 further includes updating the probability distribution based on a preset temperature parameter.

[0085] In this way, embodiments of the present disclosure can balance the diversity and accuracy of model sampling through such preset temperature parameters, thereby further improving the quality of audio content generated by the subsequent model.

[0086] In some embodiments, process 300 further includes: determining associated content of the text fragment from the text content based on the first window size.

[0087] In this way, embodiments of the present disclosure can improve the accuracy of the scope of associated content, thereby enabling the constructed contextual information to have higher quality.

[0088] In some embodiments, the size of the first window is smaller than the size of the second window used to build context information during model training.

[0089] In this way, the embodiments of this disclosure can ensure that the model obtains information related to the text fragment from the context information, avoid redundant information from interfering with the audio content generated by the model, and thus improve the audio generation efficiency.

[0090] In some embodiments, the model is trained based on the following process: segmenting the sample text content to obtain sample text fragments; obtaining training-related content of the sample text fragments from the sample text content based on a preset window size; constructing training context information based on the training-related content; and training the model based on the sample text fragments, the sample audio fragments corresponding to the sample text fragments, and the training context information.

[0091] In this way, embodiments of the present disclosure can improve the quality of the model by training such a model, thereby improving the quality of the audio content generated based on the model.

[0092] In some embodiments, segmenting sample text content to obtain sample text fragments includes: segmenting the sample text content into multiple candidate fragments based on reference information of the sample text content; and determining the sample text fragments based on the multiple candidate fragments.

[0093] In this way, the embodiments of this disclosure can ensure the integrity of the semantic and other information contained in the sample text fragments, improve the accuracy of the model in recognizing the sample text fragments, and thus improve the quality of the trained model.

[0094] In some embodiments, determining a sample text segment based on multiple candidate segments includes: in response to a target candidate segment among the multiple candidate segments having a length greater than a threshold, segmenting the target candidate segment based on a preset length to determine the sample text segment.

[0095] In this way, the embodiments of this disclosure can avoid using excessively long sample text fragments to train the model, thereby improving the stability of the model training process and thus improving the efficiency of training the model.

[0096] In some embodiments, the reference information indicates at least one of the following: the grammatical structure of the sample text content, indicating multiple sentences in the sample text content; and / or the semantic information of the sample text content, indicating the speech expression type corresponding to different content in the sample text content.

[0097] In this way, the embodiments of this disclosure can more reasonably segment the sample text content, so that the segmented sample text fragments can contain more accurate grammatical, semantic and other information, thereby further improving the quality of the trained model.

[0098] In some embodiments, the text content is the entire content of a chapter of the novel.

[0099] In this way, embodiments of the present disclosure can process more relevant text content between text fragments, thereby improving the accuracy of at least one expressive attribute of audio content determined based on contextual information.

[0100] Based on this approach, at least one expressive attribute of the audio content in the embodiments of this disclosure can be determined based on contextual information associated with the text fragment, without needing to analyze the text fragment through limited tagging or classification tools. This improves the accuracy of at least one expressive attribute, thereby enhancing the quality of the audio content. Furthermore, based on such contextual information, the embodiments of this disclosure can also support a more accurate understanding of semantic information and other details of the text fragment, thus further improving the accuracy of at least one expressive attribute and consequently further enhancing the quality of the generated audio content.

[0101] Therefore, embodiments of this disclosure can improve the accuracy of at least one expressive attribute of the generated audio content by using contextual information associated with text fragments, thereby improving the quality of the audio content.

[0102] Example devices and equipment

[0103] Embodiments of this disclosure also provide corresponding apparatus for implementing the above methods or processes. Figure 4 A schematic structural block diagram of an example device 400 for audio generation according to certain embodiments of the present disclosure is shown. Device 400 may be implemented as or included in electronic device 110. Various modules / components in device 400 may be implemented by hardware, software, firmware, or any combination thereof.

[0104] like Figure 4As shown, the apparatus 400 includes a first determining module 410 configured to determine a text segment to be processed from text content; a constructing module 420 configured to construct context information associated with the text segment based on the associated content of the text segment in the text content, wherein the associated content includes at least one of the following: a preceding segment of the text segment; a subsequent segment of the text segment; and a providing module 430 configured to provide the text segment and the context information to the model to generate audio content corresponding to the text segment, wherein at least one expressive attribute of the audio content is determined based on the context information.

[0105] In some embodiments, the providing module 430 is further configured to: generate an input text embedding based on the text fragment and context information; provide the input text embedding to the autoregressive unit of the model to generate a first audio feature; provide the first audio feature to the diffusion unit of the model to generate a second audio feature; and provide the second audio feature to the decoding unit of the model to generate audio content corresponding to the text fragment.

[0106] In some embodiments, the providing module 430 is further configured to: a first portion corresponding to a prior fragment; a second portion corresponding to a text fragment, wherein the second portion is associated with a preset identifier; and a third portion corresponding to subsequent content.

[0107] In some embodiments, the first audio feature is a discrete feature, and the second audio feature is a continuous feature.

[0108] In some embodiments, the providing module 430 is further configured to: determine the probability distribution of the model with respect to the target output token based on the input text embedding; sort a plurality of candidate elements based on the probability distribution; determine a predetermined number of candidate elements from the sorted plurality of candidate elements; update the sampling probability of the candidate elements; and sample the target element corresponding to the target output token from the candidate elements based on the updated sampling probability.

[0109] In some embodiments, the device 400 further includes an update module configured to update the probability distribution based on preset temperature parameters.

[0110] In some embodiments, the apparatus 400 further includes a second determining module configured to: determine associated content of a text fragment from the text content based on a first window size.

[0111] In some embodiments, the size of the first window is smaller than the size of the second window used to build context information during model training.

[0112] In some embodiments, the apparatus 400 further includes a model training module configured to: segment sample text content to obtain sample text fragments; obtain training-related content of the sample text fragments from the sample text content based on a preset window size; construct training context information based on the training-related content; and train a model based on the sample text fragments, the sample audio fragments corresponding to the sample text fragments, and the training context information.

[0113] In some embodiments, the model training module is further configured to: segment the sample text content into multiple candidate segments based on reference information of the sample text content; and determine the sample text segments based on the multiple candidate segments.

[0114] In some embodiments, the model training module is further configured to: in response to the length of the target candidate segment among multiple candidate segments being greater than a threshold, segment the target candidate segment based on a preset length to determine the sample text segment.

[0115] In some embodiments, the reference information indicates at least one of the following: the grammatical structure of the sample text content, indicating multiple sentences in the sample text content; and / or the semantic information of the sample text content, indicating the speech expression type corresponding to different content in the sample text content.

[0116] In some embodiments, the text content is the entire content of a chapter of the novel.

[0117] The modules included in device 400 can be implemented in various ways, including software, hardware, firmware, or any combination thereof. In some embodiments, one or more units can be implemented using software and / or firmware, such as machine-executable instructions stored on a storage medium. In addition to or as an alternative to machine-executable instructions, some or all of the modules in device 400 can be implemented at least partially by one or more hardware logic components. By way of example, and not limitation, exemplary types of hardware logic components that can be used include field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASHCs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), and so on.

[0118] Figure 5 A block diagram of an electronic device 500 in which one or more embodiments of the present disclosure may be implemented is shown. It should be understood that... Figure 5 The electronic device 500 shown is merely exemplary and should not be construed as limiting the functionality and scope of the embodiments described herein. Figure 5 The electronic device 500 shown can be used to achieve Figure 1 Electronic devices 110 or Figure 4 Device 400.

[0119] like Figure 5 As shown, electronic device 500 is in the form of a general-purpose electronic device. Components of electronic device 500 may include, but are not limited to, one or more processors or processing units 510, memory 520, storage device 530, one or more communication units 540, one or more input devices 550, and one or more output devices 560. Processing unit 510 may be a physical or virtual processor and is capable of performing various processes according to programs stored in memory 520. In a multiprocessor system, multiple processing units execute computer-executable instructions in parallel to improve the parallel processing capability of electronic device 500.

[0120] Electronic device 500 typically includes multiple computer storage media. Such media can be any accessible media that is accessible to electronic device 500, including but not limited to volatile and non-volatile media, removable and non-removable media. Memory 520 can be volatile memory (e.g., registers, cache, random access memory (RAM)), non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. Storage device 530 can be removable or non-removable media and can include machine-readable media, such as flash drives, disks, or any other media that can be used to store information and / or data and can be accessed within electronic device 500.

[0121] Electronic device 500 may further include additional removable / non-removable, volatile / non-volatile storage media. Although not explicitly stated... Figure 5 As shown, disk drives for reading from or writing to removable, non-volatile disks (e.g., "floppy disks") and optical disk drives for reading from or writing to removable, non-volatile optical disks can be provided. In these cases, each drive can be connected to a bus (not shown) via one or more data media interfaces. Memory 520 may include computer program product 525 having one or more program modules configured to perform various methods or actions of various embodiments of this disclosure.

[0122] Communication unit 540 enables communication with other electronic devices via a communication medium. Additionally, the functionality of components of electronic device 500 can be implemented using a single computing cluster or multiple computing machines capable of communicating via communication connections. Therefore, electronic device 500 can operate in a networked environment using logical connections to one or more other servers, network personal computers (PCs), or another network node.

[0123] Input device 550 can be one or more input devices, such as a mouse, keyboard, trackball, etc. Output device 560 can be one or more output devices, such as a monitor, speaker, printer, etc. Electronic device 500 can also communicate with one or more external devices (not shown) via communication unit 540 as needed. These external devices include storage devices, display devices, etc., and can communicate with one or more devices that enable user interaction with electronic device 500, or with any device that enables electronic device 500 to communicate with one or more other electronic devices (e.g., network card, modem, etc.). Such communication can be performed via input / output (H / O) interfaces (not shown).

[0124] According to an exemplary implementation of this disclosure, a computer-readable storage medium is provided that stores computer-executable instructions thereon, wherein the computer-executable instructions are executed by a processor to implement the methods described above. According to an exemplary implementation of this disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, which are executed by a processor to implement the methods described above.

[0125] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatuses, devices, and computer program products implemented according to this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0126] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processing unit of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0127] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions that execute on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0128] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0129] Various implementations of this disclosure have been described above. The foregoing description is exemplary and not exhaustive, nor is it limited to the disclosed implementations. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described implementations. The terminology used herein is chosen to best explain the principles, practical applications, or improvements to technology in the market, or to enable others skilled in the art to understand the various implementations disclosed herein.

Claims

1. A method for audio generation, comprising: Identify the text segment to be processed from the text content; Based on the associated content of the text fragment within the text content, contextual information associated with the text fragment is constructed. The associated content includes at least one of the following: a preceding fragment of the text fragment; a subsequent fragment of the text fragment; and... The model is provided with the text fragment and the context information to generate audio content corresponding to the text fragment, wherein at least one expressive attribute of the audio content is determined based on the context information.

2. The method according to claim 1, wherein providing the text fragment and the context information to the model to generate the audio content corresponding to the text fragment comprises: Based on the text fragment and the context information, an input text embedding is generated; The input text embedding is provided to the autoregressive unit of the model to generate a first audio feature; The first audio feature is provided to the diffusion unit of the model to generate the second audio feature; as well as The second audio feature is provided to the decoding unit of the model to generate the audio content corresponding to the text segment.

3. The method of claim 2, wherein the input text embedding comprises: The first part corresponding to the prior segment; A second part corresponding to the text fragment, wherein the second part is associated with a preset identifier; as well as The third part corresponds to the following content.

4. The method according to claim 2, wherein the first audio feature is a discrete feature and the second audio feature is a continuous feature.

5. The method of claim 2, wherein providing the input text embedding to the autoregressive unit of the model to generate the first audio feature comprises: Based on the input text embedding, determine the probability distribution of the model with respect to the target output token; Based on the probability distribution, sort multiple candidate elements; From the sorted plurality of candidate elements, a predetermined number of candidate elements are determined; Update the sampling probability of the aforementioned set of candidate elements; as well as Based on the updated sampling probability, a target element corresponding to the target output token is sampled from the set of candidate elements.

6. The method of claim 5, further comprising, before sorting the plurality of candidate elements based on the probability distribution: The probability distribution is updated based on preset temperature parameters.

7. The method according to claim 1, further comprising: Based on the first window size, the associated content of the text fragment is determined from the text content.

8. The method of claim 7, wherein the first window size is smaller than the second window size used to construct context information during the training of the model.

9. The method of claim 1, wherein the model is trained based on the following process: The sample text content is segmented to obtain sample text fragments; Based on a preset window size, the training association content of the sample text fragment is obtained from the sample text content; Construct training context information based on the training-related content; as well as The model is trained based on the sample text fragment, the corresponding sample audio fragment, and the training context information.

10. The method of claim 9, wherein segmenting the sample text content to obtain the sample text fragments comprises: Based on the reference information of the sample text content, the sample text content is segmented into multiple candidate segments; as well as Based on the multiple candidate fragments, the sample text fragment is determined.

11. The method of claim 10, wherein determining the sample text segment based on the plurality of candidate segments comprises: In response to the fact that the length of the target candidate segment among the plurality of candidate segments is greater than a threshold, the target candidate segment is segmented based on a preset length to determine the sample text segment.

12. The method of claim 10, wherein the reference information indicates at least one of the following: The syntactic structure of the sample text content indicates multiple statements within the sample text content; and / or The semantic information of the sample text content indicates the speech expression type corresponding to different contents in the sample text content.

13. The method according to claim 1, wherein the text content is the entire content of a chapter of a novel.

14. An apparatus for audio generation, comprising: The first determining module is configured to determine the text segment to be processed from the text content; A construction module is configured to construct contextual information associated with the text fragment based on the associated content within the text content. The associated content includes at least one of the following: a preceding fragment of the text fragment; a subsequent fragment of the text fragment; and... A module is configured to provide the model with the text fragment and the context information to generate audio content corresponding to the text fragment, wherein at least one expressive attribute of the audio content is determined based on the context information.

15. An electronic device comprising: At least one processing unit; as well as At least one memory, coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, which, when executed by the at least one processing unit, cause the electronic device to perform the method according to any one of claims 1 to 13.

16. A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method according to any one of claims 1 to 13.

Images