Audio generation methods and audio generation methods based on timbre materials

By constructing script prompts and using a task processing model to generate audio generation scripts, combined with timbre materials, the problem of poor speech quality in existing audio generation technologies has been solved, achieving high-quality interactive audio generation and improving production efficiency and user experience.

CN123072369APending Publication Date: 2026-11-10ALIBABA (CHINA) CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202510592110.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2025-05-08
Publication Date
2026-11-10

AI Technical Summary

Technical Problem

Existing audio generation technologies rely on text-to-speech conversion, resulting in poor audio quality, a lack of authenticity and naturalness, and difficulty in meeting the needs of emerging content formats such as podcasts and live-streaming e-commerce.

Method used

By acquiring audio reference documents and generation parameters, script prompts are constructed, an audio generation script is generated using a task processing model, and high-quality interactive audio is generated by combining timbre materials.

Benefits of technology

It improves the realism and quality of audio generation scripts, reduces labor costs, and increases production efficiency and user information absorption efficiency, making it suitable for innovative applications in the media, education, and entertainment industries.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123072369A_ABST
    Figure CN123072369A_ABST
Patent Text Reader

Abstract

This specification provides an audio generation method and an audio generation method based on timbre materials. The audio generation method includes: obtaining an audio reference document and audio generation parameters; constructing script prompts based on the audio reference document and audio generation parameters, wherein the script prompts guide a task processing model to perform role-playing dialogue to obtain an audio generation script; generating the audio generation script using the task processing model based on the script prompts; and generating the target audio corresponding to the audio reference document based on the audio generation script. The audio reference document and audio generation parameters ensure that the audio generation script meets the generation expectations. Guiding the task processing model to perform role-playing dialogue through script prompts makes the generated audio generation script not only highly realistic but also accurately reflects the role-playing interactions, improving the efficiency and quality of audio generation script generation, and further improving the efficiency and quality of target audio generation.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The embodiments in this specification relate to the field of audio generation technology, and in particular to audio generation methods and audio generation methods based on timbre materials. Background Technology

[0002] Compared to text or images, audio offers a stronger sense of immersion and emotional delivery, better capturing user attention and enhancing the interactive experience. Therefore, efficiently generating natural, fluent, and expressive audio content has become a key focus for content creators and businesses.

[0003] Currently, audio generation primarily relies on text-to-speech (TTS) technology, directly converting input documents into speech signals. However, the resulting speech lacks realism, often exhibiting mechanical pronunciation and unnatural intonation, leading to poor audio quality. This makes it ill-suited for emerging content formats such as podcasts and live-streaming e-commerce. Therefore, a high-quality audio generation solution is urgently needed. Summary of the Invention

[0004] In view of the above, embodiments of this specification provide an audio generation method. One or more embodiments of this specification also relate to an audio generation method based on timbre materials, an information processing method based on a task processing model, a task platform, an audio generation device, an audio generation device based on timbre materials, an information processing device based on a task processing model, a computing device, an electronic device, a computer-readable storage medium, and a computer program product, to address the technical deficiencies existing in the prior art.

[0005] According to a first aspect of the embodiments of this specification, an audio generation method is provided, comprising: obtaining an audio reference document and audio generation parameters; constructing script prompt information based on the audio reference document and audio generation parameters, wherein the script prompt information is used to guide a task processing model to perform role dialogue to obtain an audio generation script; generating the audio generation script using the task processing model based on the script prompt information; and generating a target audio corresponding to the audio reference document based on the audio generation script.

[0006] According to a second aspect of the embodiments of this specification, an audio generation method based on timbre material is provided, comprising: acquiring task data for an audio generation task, wherein the task data includes an audio reference document, audio generation parameters, and target timbre material; constructing script prompt information based on the audio reference document and the audio generation parameters, wherein the script prompt information is used to guide a task processing model to perform role dialogue to obtain an audio generation script; generating an audio generation script using the task processing model based on the script prompt information; and generating target audio corresponding to the audio reference document based on the target timbre material and the audio generation script.

[0007] According to a third aspect of the embodiments of this specification, an information processing method based on a task processing model is provided, applied to a task platform, comprising: receiving a model request sent by a terminal device; and determining a target task processing model from multiple task processing models based on the model request, wherein the target task processing model is used in the processing of an audio generation method.

[0008] According to a fourth aspect of the embodiments of this specification, a task platform is provided, including a request interface and a response unit; the request interface is used to receive a model request sent by a terminal device, wherein the model request includes at least one of a scene identifier of a target scene, scene input data of the target scene, and model specification parameters; the response unit is used to determine a target task processing model from multiple task processing models based on the model request, wherein the target task processing model is used in the processing of an audio generation method.

[0009] According to a fifth aspect of the embodiments of this specification, an audio generation apparatus is provided, comprising: a first acquisition module configured to acquire an audio reference document and audio generation parameters; a first construction module configured to construct script prompt information based on the audio reference document and the audio generation parameters, wherein the script prompt information is used to guide a task processing model to perform role dialogue to obtain an audio generation script; a first generation module configured to generate an audio generation script using the task processing model based on the script prompt information; and a second generation module configured to generate target audio corresponding to the audio reference document based on the audio generation script.

[0010] According to a sixth aspect of the embodiments of this specification, an audio generation apparatus based on timbre material is provided, comprising: a second acquisition module configured to acquire task data for an audio generation task, wherein the task data includes an audio reference document, audio generation parameters, and target timbre material; a second construction module configured to construct script prompt information based on the audio reference document and the audio generation parameters, wherein the script prompt information is used to guide a task processing model to perform role dialogue to obtain an audio generation script; a third generation module configured to generate an audio generation script using the task processing model based on the script prompt information; and a fourth generation module configured to generate target audio corresponding to the audio reference document based on the target timbre material and the audio generation script.

[0011] According to a seventh aspect of the embodiments of this specification, an information processing apparatus based on a task processing model is provided, applied to a task platform, comprising: a receiving module configured to receive a model request sent by a terminal device; and a determining module configured to determine a target task processing model from a plurality of task processing models based on the model request, wherein the target task processing model is used in the processing of an audio generation method.

[0012] According to an eighth aspect of the embodiments of this specification, a computing device is provided, comprising: a memory and a processor; the memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, wherein the computer programs / instructions, when executed by the processor, implement the steps of the methods provided in the first, second, or third aspects described above.

[0013] According to a ninth aspect of the embodiments of this specification, an electronic device is provided, including: a memory and a processor, the memory and the processor being connected via a bus; the memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, wherein the computer programs / instructions, when executed by the processor, implement the steps of the methods provided in the first, second, or third aspects described above.

[0014] According to a tenth aspect of the embodiments of this specification, a computer-readable storage medium is provided that stores a computer program / instructions that, when executed by a processor, implement the steps of the methods provided in the first, second, or third aspects described above.

[0015] According to an eleventh aspect of the embodiments of this specification, a computer program product is provided, including a computer program / instructions that, when executed by a processor, implement the steps of the methods provided in the first, second, or third aspects described above.

[0016] An embodiment of this specification provides an audio generation method, comprising: obtaining an audio reference document and audio generation parameters; constructing script prompt information based on the audio reference document and audio generation parameters, wherein the script prompt information is used to guide a task processing model to perform role dialogue to obtain an audio generation script; generating the audio generation script using the task processing model based on the script prompt information; and generating the target audio corresponding to the audio reference document based on the audio generation script. The audio reference document provides the content framework of the audio generation script, and the audio generation parameters further refine the details of the audio generation script, ensuring that the audio generation script highly conforms to the generation expectations. Guiding the task processing model to perform role dialogue through script prompt information makes the generated audio generation script not only highly realistic but also accurately reflects the role interaction relationship, improving the efficiency and quality of audio generation script generation, and further improving the efficiency and quality of target audio generation. Attached Figure Description

[0017] Figure 1 This is a flowchart of an audio generation method provided in one embodiment of this specification;

[0018] Figure 2This is an architecture diagram of an audio generation system provided in one embodiment of this specification;

[0019] Figure 3 This is a flowchart illustrating an audio generation method based on timbre materials, provided in one embodiment of this specification.

[0020] Figure 4 This is a flowchart illustrating the processing steps of an audio generation method provided in one embodiment of this specification.

[0021] Figure 5 This is a flowchart illustrating the processing procedure of an information processing method based on a task processing model, provided in one embodiment of this specification.

[0022] Figure 6 This is a schematic diagram of the structure of a task platform provided in one embodiment of this specification;

[0023] Figure 7 This is a schematic diagram of the structure of an audio generation device provided in one embodiment of this specification;

[0024] Figure 8 This is a schematic diagram of an audio generation device based on timbre materials provided in one embodiment of this specification;

[0025] Figure 9 This is a schematic diagram of the structure of an information processing device based on a task processing model provided in one embodiment of this specification;

[0026] Figure 10 This is a structural block diagram of a computing device provided in one embodiment of this specification;

[0027] Figure 11 This is a structural block diagram of an electronic device provided in one embodiment of this specification. Detailed Implementation

[0028] Many specific details are set forth in the following description to provide a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this specification. Therefore, this specification is not limited to the specific implementations disclosed below.

[0029] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to limit the scope of the one or more embodiments of this specification. The singular forms “a,” “described,” and “the” used in one or more embodiments of this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items. The term “at least one” in one or more embodiments of this application means “one or more,” and “a plurality of” means “two or more.” The term “comprising” is an open-ended description and should be understood as “including but not limiting,” and may include other content in addition to what has been described.

[0030] It should be understood that although the terms first, second, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first may also be referred to as second without departing from the scope of one or more embodiments of this specification, and similarly, second may also be referred to as first. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0031] Furthermore, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties. Moreover, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0032] In one or more embodiments of this specification, a large model refers to a deep learning model with a large number of model parameters, typically containing hundreds of millions, tens of billions, hundreds of billions, trillions, or even tens of trillions of model parameters. A large model can also be called a foundation model. It is pre-trained on a large-scale unlabeled corpus to produce a pre-trained model with hundreds of millions of parameters. Such models can adapt to a wide range of downstream tasks and have good generalization ability; examples include large-scale language models and multi-modal pre-training models.

[0033] In practical applications, large models only require a small number of samples to fine-tune the pre-trained model before they can be applied to different tasks. Large models can be widely used in fields such as Natural Language Processing (NLP) and Computer Vision. Specifically, they can be applied to computer vision tasks such as Visual Question Answering (VQA), Image Captioning (IC), and Image Generation, as well as natural language processing tasks such as text-based sentiment classification, text summarization, and machine translation. The main application scenarios for large models include digital assistants, intelligent robots, search, online education, office software, e-commerce, and intelligent design.

[0034] First, the terms and concepts used in one or more embodiments of this specification will be explained.

[0035] A language model (LM) is a statistical model used to understand and generate natural language. Its goal is to analyze a text and predict the probability distribution of the next word or character. Language models can be used for a variety of text generation tasks.

[0036] Large-scale language models (LLMs) are neural network models with greater scale and capabilities. By increasing the number of parameters, layers, and attention mechanisms, LLMs can better capture the semantics, structure, and relationships of text. LLMs have achieved significant results in natural language processing tasks.

[0037] A prompt is an input text or instruction that triggers an action or generates a response. For example, an input sentence provided by a user to an artificial intelligence language model is a prompt.

[0038] Document / web parsing refers to the technology of analyzing the text and image layout of documents or web pages and converting the content into plain text output. Simple document / web parsing tools only extract the text as is, ignoring information such as layout and charts. Excellent document / web parsing can obtain more information as output, such as font and font size, image descriptions, and table data conversion.

[0039] Text-to-speech (TTS) is a technology that converts written text into natural language speech output. The input is text, and the output is audio; that is, a TTS system can read a text string and read it aloud using a synthesized voice. The development of TTS technology has progressed from simply piecing together pre-recorded words and phrases to using sophisticated algorithms to generate smooth and natural speech. Modern TTS systems typically rely on deep learning models that learn pronunciation patterns from large amounts of human speech data, thereby synthesizing speech that sounds very natural, including correct intonation, stress, and emotional expression.

[0040] Voice cloning, also known as voice replication or personalized voice synthesis, refers to using artificial intelligence and machine learning techniques to mimic the voice characteristics of a specific individual. The input is text to be read aloud and the voice to be cloned; the output is audio of the text read aloud by the cloned voice. This technology analyzes a person's speech samples, capturing their unique voice features such as timbre, intonation, and rhythm, and then simulates that voice to generate the final audio from the text to be read aloud. Voice cloning not only requires accurately replicating the target person's voice in pronunciation but also maintaining consistency across different contexts, including accurately expressing emotion and tone.

[0041] A Uniform Resource Locator (URL) is a string used to identify the location of every information resource on the Internet and how to access it. Simply put, a URL is a standard address format used to indicate where a resource (such as a webpage, image, or video) is located on the network and how to access it.

[0042] HyperText Markup Language (HTML) is the standard markup language used to create web pages. It consists of a series of elements, defined by tags, which tell the browser how to display or interact with the content. HTML documents typically contain text, images, videos, and other multimedia content, and can link to other documents or pages, forming a "hypertext" network.

[0043] Root Mean Square (RMS) normalization is an audio processing technique used to adjust the volume level of an audio signal to give it a uniform loudness. This method calculates the RMS value of an audio signal and adjusts the overall volume of the audio accordingly, so that different audio segments sound at a consistent volume when played.

[0044] Double-blind evaluation: This refers to a evaluation process in which neither the evaluator (the person conducting the evaluation) nor the evaluated object (the product, service, or experimental group being evaluated) knows who received which treatment or which version of the test. This method is mainly used to avoid the influence of subjective bias caused by the evaluator's or participant's own expectations on the evaluation results.

[0045] With the rise and popularization of interactive content formats such as podcasts and live-streaming e-commerce, businesses are showing an increasingly strong demand for efficient and personalized content production. This trend places higher demands on content creation. Against this backdrop, high-quality scriptwriting and realistic, conversational speech synthesis have become key elements driving industry progress. On the one hand, content creators need to ensure script quality from the source, meaning they must not only conceive creative and logically rigorous initial drafts but also ensure their information richness, thereby reducing the time costs of subsequent proofreading, review, refinement, and optimization. On the other hand, the productivity of content creators reading scripts aloud is limited, so advanced speech technology is needed to replicate natural, fluent, and even more expressive voices, thereby improving production efficiency and enhancing the listener experience. Regarding reading efficiency, traditional information acquisition media mainly rely on text reading, which is often inefficient when dealing with complex or professional content. In contrast, receiving information through hearing (such as listening to content shared by book bloggers or news anchors) allows users to quickly grasp key information in a short time, greatly improving the speed and quality of information absorption. This shift not only caters to the needs of modern fast-paced life, but also provides a brand-new option for users who seek efficient learning, emotional companionship, and liberation of visual attention.

[0046] Based on this, embodiments of this specification propose an automatic synthesis scheme for generating interactive audio content. First, a high-quality interactive audio generation script is synthesized; then, using the audio generation script, personalized, high-quality interactive audio is produced. This transforms a given audio reference document into multi-person interactive audio or single-person spoken audio, such as podcast, radio, or live-streaming sales-style audio. The scheme proposed in these embodiments can, on the one hand, serve as an auxiliary tool for content creators, facilitating the production of high-quality audio content without requiring skilled hosting abilities or relying on live hosts or actors. This reduces the labor costs and time investment in traditional broadcasting and video production, improving production efficiency. It can inspire more innovative applications and service models in media, education, and entertainment industries. On the other hand, this easily understood dialogue format can significantly improve users' information absorption and transmission efficiency. The conversational expression helps users quickly grasp the key points, reducing reading and comprehension time.

[0047] This specification provides an audio generation method, and also relates to an audio generation method based on timbre materials, an information processing method based on a task processing model, a task platform, an audio generation device, an audio generation device based on timbre materials, an information processing device based on a task processing model, a computing device, an electronic device, a computer-readable storage medium, and a computer program product, which will be described in detail in the following embodiments.

[0048] See Figure 1 , Figure 1 This specification shows a flowchart of an audio generation method according to an embodiment, which specifically includes the following steps:

[0049] Step 102: Obtain the audio reference document and audio generation parameters.

[0050] It's important to note that audio reference documents are the foundational materials used to guide the generation of audio content. They typically include information related to the target audio, character settings, dialogue scenarios, or background explanations. Audio reference documents can be script drafts, plot descriptions, product introductions, or any other form of material, designed to provide context and reference for subsequent audio generation. Audio reference documents can be documents from different scenarios, such as legal documents in a legal context, code documentation in a coding context, or financial documents in a financial context.

[0051] Audio generation parameters refer to a series of settings or conditions used to control and adjust the audio generation process. Audio generation parameters can be pre-set parameters or user-defined configurable parameters. These parameters allow users to customize the presentation of audio content according to specific needs, thereby achieving personalized generation and meeting application requirements in different scenarios. Audio generation parameters include, but are not limited to, audio requirements, number of characters, character identities, number of dialogue turns, audio style, audio language, and audio speaking speed. The specific selection depends on the actual situation, and this specification does not impose any limitations on these parameters in the embodiments. For example, audio generation parameters could be: Number of characters - two, Style - lighthearted and informative, Requirements - none, Language - Chinese.

[0052] In practical applications, there are various ways to obtain audio reference documents and audio generation parameters, and the specific method should be selected according to the actual situation. This specification does not impose any limitations on these methods in its embodiments. In one possible implementation, the audio reference document and audio generation parameters can be received from the user via a client. In another possible implementation, the audio generation parameters can be randomly set, and a document download link or webpage URL link sent by the user via a client can be received, allowing the audio reference document to be downloaded based on the link.

[0053] Step 104: Based on the audio reference document and audio generation parameters, construct script prompts, which are used to guide the task processing model to conduct role dialogue to obtain the audio generation script.

[0054] It should be noted that script prompts refer to prompts constructed based on the audio reference document and audio generation parameters. The purpose of constructing script prompts is to provide clear instructions and contextual cues to the task processing model, guiding it to generate audio generation scripts that meet expectations. Script prompts can act as a bridge between the audio reference document and the task processing model, helping the model understand the specific requirements when generating the script and ensuring that the generated script accurately conveys the expected content.

[0055] In practical applications, there are various ways to construct script prompts based on audio reference documents and audio generation parameters. The specific method chosen depends on the actual situation, and this specification does not impose any limitations on this approach. In one possible implementation, preset prompts, audio reference documents, and audio generation parameters can be concatenated to obtain script prompts. The preset prompts are general prompts, such as "Please generate script prompts based on the audio reference document and audio generation parameters." In another optional embodiment, script prompts can be constructed based on a script prompt template, according to the audio reference document and audio generation parameters. The script prompts include at least one of the following: model role information, model task instructions, model processing rules, and script examples.

[0056] In one optional embodiment of this specification, constructing script prompt information based on the audio reference document and audio generation parameters may include the following steps:

[0057] Obtain a script prompt template, wherein the script prompt template includes at least one of the following: model role information, model task instructions, model processing rules, and script examples;

[0058] Add the audio reference document and audio generation parameters to the script prompt template to obtain script prompt information.

[0059] It should be noted that script prompt templates are predefined, structured frameworks that guide the construction of script prompt messages. These templates provide a standardized input framework to the task processing model, ensuring that it can efficiently organize audio reference documents and audio generation parameters to generate expected script prompt messages.

[0060] Model role information refers to the role information that the model needs to simulate, as defined in the script prompt template. Model role information includes, but is not limited to, the role's identity (e.g., profession), personality traits (e.g., humorous, serious, friendly), language style (e.g., formal, colloquial), and emotional expression (e.g., happy, sad, angry). Model role information helps the task processing model understand and simulate the language and behavioral characteristics of a specific role, thereby generating audio generation scripts that are closer to real-world scenarios. For example, model role information could be: You are a professional host, and your task is to transform the provided input text into a relaxed, fun, imaginative, and informational podcast with approximately {number of dialogue rounds} {number of participants} for equal discussion and exchange, with a style similar to {style parameter}, rather than requiring you to summarize or interpret the input text.

[0061] Model task instructions refer to the specific tasks that the task processing model is required to complete, as outlined in the script prompt template. These instructions provide clear task objectives or operational requirements for the model. Model task instructions include, but are not limited to, generating a multi-role dialogue, expressing emotion in a specific tone, or outputting script content in a specific format. For example, a model task instruction could be: "Carefully read the input text, identify the main topics, key points, and any interesting facts or anecdotes. Avoid rigid question-and-answer formats; interact directly by expressing opinions and sharing. The goal is a natural, authentic, and equal two-person dialogue flow between two hosts within a broader theme."

[0062] Model processing rules refer to a series of constraints or logical rules defined in the script prompt template. These rules standardize the behavior of the task processing model, ensuring that the generated audio scripts meet expected standards in terms of language expression, logical structure, and emotional delivery, avoiding content that does not meet actual needs. Model processing rules include, but are not limited to, grammatical requirements (such as avoiding repetitive words), dialogue logic (such as the interaction between characters), output format (such as outputting dialogue in JSON format), and contextual consistency (such as maintaining coherent tone). For example, a model processing rule could be: "Note: When a host speaks, the other party should respond appropriately with short replies such as 'um' or 'oh.' Please learn and imitate this. Do not mention the host's introduction or name; following the example, directly generate a final podcast dialogue without outputting anything else."

[0063] Script examples refer to one or more exemplary script contents provided in the script prompt template. Script examples demonstrate how to generate compliant script prompts based on audio reference documents and audio generation parameters. Script examples typically include specific character dialogue, tone of voice, and scene descriptions. Script examples provide an intuitive reference for task processing models, helping them to better construct script prompts.

[0064] In practical applications, there are multiple ways to obtain script prompt templates, and the specific method should be selected according to the actual situation. This specification does not impose any limitations on this method. In one possible implementation, the script prompt template can be received from a user via a client. In another possible implementation, the script prompt template can be read from other data acquisition devices or a database.

[0065] Furthermore, there are multiple ways to add the audio reference document and audio generation parameters to the script prompt template to obtain script prompt information. The specific method chosen depends on the actual situation, and this specification does not impose any limitations on this approach. In one possible implementation, the audio reference document and audio generation parameters can be directly added to the script prompt template to obtain script prompt information. In another possible implementation, since the audio reference document may contain some non-text content, the text content from the audio reference document and the audio generation parameters can be added to the script prompt template to obtain script prompt information.

[0066] The solution implemented in this specification provides a content framework for the audio generation script through an audio reference document, and further refines the details of the audio generation script using audio generation parameters. This ensures that the audio generation script highly meets the generation expectations. By combining different elements in the script prompt template, the flexibility of the script prompt information is guaranteed, further ensuring the personalization and quality of the audio generation script.

[0067] In one optional embodiment of this specification, before constructing the script prompt information based on the audio reference document and audio generation parameters, the following steps may also be included:

[0068] Perform text parsing on the audio reference document to obtain the audio reference text;

[0069] The audio reference text is subjected to text normalization processing to obtain a standard reference text;

[0070] Based on the audio reference document and audio generation parameters, constructing script prompts may include the following steps:

[0071] Construct script prompts based on audio generation parameters and standard reference text.

[0072] It should be noted that audio reference text refers to the specific text content extracted after parsing the audio reference document. Text standardization processing refers to the process of formatting, standardizing, and optimizing the audio reference text. The purpose of text standardization processing is to unify the text format (such as removing redundant information, correcting grammatical errors, and standardizing punctuation marks) to improve the accuracy and consistency of subsequent script prompts. Standard reference text refers to the audio reference text after text standardization processing, which has a unified format, clear language expression, and standardized structure.

[0073] In practical applications, when parsing audio reference documents to obtain the audio reference text, the parsing method can be tailored to the document type. For example, for text documents, the audio reference text can be opened and read directly; for binary documents, specific libraries can be used to read the audio reference text; for HTML files, the entire HTML code can be read as the audio reference text. Methods for parsing audio reference documents include, but are not limited to, intelligent document parsing tools and open-source code libraries.

[0074] Furthermore, there are various ways to perform text standardization processing on the audio reference text to obtain the standard reference text. The specific method chosen depends on the actual situation, and this specification does not limit the specific approach. In one possible implementation, scripts or software tools can be used to perform text standardization processing on the audio reference text to obtain the standard reference text. For example, software tools can be used to batch replace characters, automatically check spelling and grammar, etc. In another possible implementation, NLP techniques, such as entity recognition, can be used for more advanced text analysis and processing to obtain the standard reference text.

[0075] For example, the system can receive a document download link or a webpage URL input by the user; download and read the audio reference document based on the document download link or webpage URL (the webpage also downloads its HTML code as a document); perform text parsing on the audio reference document to obtain the audio reference text; and perform text standardization processing on the audio reference text to obtain the standard reference text.

[0076] The scheme described in this specification involves parsing the audio reference document to extract the audio reference text, followed by text standardization to eliminate ambiguity and inconsistencies, thereby obtaining standard reference text. Subsequently, by combining the audio generation parameters and the standard reference text, precise script prompts are constructed. This step-by-step processing method effectively improves the quality and adaptability of the script prompts, ensuring that the audio generation script better meets requirements, while reducing error propagation and improving the overall efficiency and output quality of the audio generation task.

[0077] In one optional embodiment of this specification, the task processing model may brainstorm based on the audio reference document to enhance the script prompt information. That is, before constructing the script prompt information based on the audio reference document and audio generation parameters, the following steps may also be included:

[0078] Input the thinking prompts and audio reference documents into the task processing model to obtain dialogue assistance information. The thinking prompts are used to guide the task processing model to think about the audio reference documents from at least one dialogue dimension.

[0079] Based on the audio reference document and audio generation parameters, constructing script prompts may include the following steps:

[0080] Based on the audio generation parameters, audio reference documents, and dialogue assistance information, construct script prompts.

[0081] It should be noted that the thinking prompts typically contain explicit instructions or questions to help the task processing model conduct in-depth analysis of the audio reference document from at least one dialogue dimension. The task processing model refers to an artificial intelligence model with natural language understanding capabilities. The task processing model can be a large model or a deep learning model trained on sample data corresponding to the task being performed. For example, if the task processing model here is performing a dialogue assistance information generation task, then the sample data can be sample assistance information, sample thinking prompts, and sample audio reference documents. The thinking prompts include at least one of the following: model role information, task description information, model task instructions, model processing rules, and thinking examples. The thinking prompts can be information pre-set by the user or information read from other data acquisition devices or databases.

[0082] For example, consider the method of generating dialogue assistance information by invoking a task processing model using separate thinking prompts. The model role information could be: "You are a Python programmer. You can use the provided application programming interface (API) to build a document analysis graph, generate code, and execute it." The task description information could be: "API introduction: Below is the API documentation for the document analysis graph class, which includes detailed attribute descriptions; Class: Document Analysis Graph; This class is used to manage the nodes and edges of the document analysis graph and perform analysis through a document agent." The model task instructions could be: "Build the analysis graph step by step by breaking down the user's question into multi-faceted sub-questions that can be answered by reading the document analysis. Each analyzed question should be a single question, i.e., a question about a specific person, event, thing, specific time point, specific location, or knowledge point, not a composite question (such as a certain time period). Pay special attention to the analysis of the style and format specified by the user." The model processing rules could be: "Note that the content of each analysis node must be a single question, not multiple questions. Always wait for the code to return results before analysis; do not fabricate analysis results. Do not repeat the same question; you can continue to ask questions based on existing questions. Only generate code to build the graph; do not generate any other responses."

[0083] Dialogue aid information refers to information generated by the task processing model to supplement and improve script prompts. Dialogue aid information can be understood as the brainstorming and arrangement results of the task processing model. Dialogue aid information includes, but is not limited to, conceptual inspiration, program planning, program highlights, script ideas, and thought-provoking questions; the specific selection depends on the actual situation, and this specification does not impose any limitations on this. Dialogue aid information includes aid information for at least one dialogue dimension, such as intent dimension, time dimension, character interaction dimension, emotion dimension, etc. The format of dialogue aid information includes code format, JSON format, Markdown format, etc. When constructing script prompts based on audio generation parameters, audio reference documents, and dialogue aid information, the dialogue aid information can be standardized. The script prompts are then constructed based on the audio generation parameters, audio reference documents, and the standardized dialogue aid information.

[0084] It should be noted that the implementation method of "constructing script prompt information based on audio generation parameters, audio reference documents and dialogue assistance information" can refer to the above implementation method of "constructing script prompt information based on audio reference documents and audio generation parameters", and will not be repeated in the embodiments of this specification.

[0085] In practical applications, before inputting the thinking prompts and audio reference documents into the task processing model, key information can be extracted from the audio reference document to obtain key document information. Then, the thinking prompts and key document information are input into the task processing model to obtain dialogue assistance information, thereby reducing the data processing load of the task processing model and improving the efficiency of generating dialogue assistance information. Here, key document information refers to key information in the audio reference document, such as the document title and body text.

[0086] Furthermore, if the thinking prompts include multiple dialogue dimensions, one possible implementation of this specification is to directly input the thinking prompts including multiple dialogue dimensions and the audio reference document into the task processing model to obtain dialogue assistance information including multiple dialogue dimensions. Another possible implementation of this specification is to parse the thinking prompts to obtain the thinking prompt sub-information corresponding to each dialogue dimension. For any given dialogue dimension, the thinking prompt sub-information corresponding to that dialogue dimension and the audio reference document are input into the task processing model to obtain the dialogue assistance sub-information corresponding to that dialogue dimension. The dialogue assistance sub-information corresponding to each dialogue dimension is then concatenated to obtain the dialogue assistance information.

[0087] By applying the solutions in the embodiments of this specification, and by introducing thought prompts, the task processing model is guided to organize and conceive the audio reference document from at least one dialogue dimension, generating richer and more comprehensive dialogue assistance information. In this way, by combining the audio generation parameters, the audio reference document, and the dialogue assistance information, more refined script prompts are constructed. This not only improves the richness and accuracy of the script prompts, but also ensures that the generated audio script is closer to actual needs in terms of logic, naturalness, and emotional expression, thereby significantly improving the quality and realism of the target audio.

[0088] In one optional embodiment of this specification, the dialogue assistance information generated by the task processing model may be relatively concise, that is, the dialogue assistance information includes at least one dialogue information point; then, it can be supplemented or expanded based on at least one dialogue information point to obtain associated information. Before constructing the script prompt information based on the audio generation parameters, audio reference document, and dialogue assistance information, the following steps may also be included:

[0089] From the dialogue association data, at least one dialogue information point is parsed out as associated information, wherein the dialogue association data and at least one dialogue information point are associated.

[0090] The dialogue auxiliary information and related information are fused to obtain fused auxiliary information;

[0091] Based on the audio generation parameters, audio reference documents, and dialogue assistance information, constructing script prompts may include the following steps:

[0092] Based on the audio generation parameters, audio reference documents, and fusion assistance information, construct script prompts.

[0093] It should be noted that dialogue-related data refers to datasets that are associated with dialogue information points. This data can originate from multiple sources, such as text, audio transcriptions, and background information. Associations can include background-related, role-related, and time-related relationships. Dialogue-related data can be extracted from databases or obtained by searching dialogue information points using online search tools. Dialogue information points are information units marked as important or requiring special attention in dialogue support information, such as specific topics, roles, or interaction methods. Related information refers to additional information parsed from dialogue-related data that is directly related to at least one dialogue information point. Related information includes, but is not limited to, contextual information, role background, and event development. Integrated support information refers to the set of information obtained by integrating dialogue support information and related information, aiming to provide more comprehensive support for constructing script prompts.

[0094] In practical applications, there are multiple ways to parse the associated information corresponding to at least one dialogue information point from dialogue association data. The specific method chosen depends on the actual situation, and this specification does not impose any limitations on this method. One possible implementation of this specification involves calling a text parsing tool to integrate, analyze, and extract the associated information corresponding to at least one dialogue information point from the dialogue association data. Another possible implementation involves inputting information extraction prompts and dialogue association data into a task processing model to obtain the associated information corresponding to at least one dialogue information point. The information extraction prompts are obtained by adding the dialogue information point to an information extraction template. The information extraction template includes at least one of the following: model role information, model task instructions, model processing rules, and extraction examples.

[0095] For example, the model's role information could be "You are an intelligent assistant skilled at extracting important information from document content and providing clear, logical, and structured answers." The model's task instruction could be "Please carefully collect information related to the dialogue information points from the document content." The model's processing rules could be "Ensure that the response is comprehensive, substantive, and helpful; the response can be presented concisely and clearly using a structured approach and specific examples."

[0096] The scheme implemented in this specification, by parsing the association information corresponding to at least one dialogue information point from the dialogue association data and fusing this association information with dialogue auxiliary information, yields richer and more comprehensive fused auxiliary information. This helps deepen the task processing model's understanding of the audio reference document content, especially regarding specific dialogue contexts, character backgrounds, or emotional expressions. Based on the fused auxiliary information, script prompts are constructed by combining audio generation parameters and the audio reference document. This more accurately guides the task processing model to generate high-quality audio generation scripts, improving not only the accuracy of the audio script in details but also enhancing the emotional realism and contextual adaptability of the target audio, achieving a more personalized and natural audio output effect.

[0097] Step 106: Using the task processing model, generate an audio generation script based on the script prompts.

[0098] It should be noted that the task processing model used to generate audio generation scripts refers to a model with strong natural language understanding and generation capabilities, capable of simulating human dialogue logic, emotional expression, and contextual awareness. The audio generation script refers to the specific text content used to guide the audio generation process. It includes the elements required for audio generation, such as text content, emotional expression, tone, speaking rate, role allocation, and timeline, and describes these elements through clear formats and rules to ensure that the final generated audio meets the expected goals.

[0099] For example, the audio generation script could be: {"Dialogue": [{"Text": "Hello listeners", "Role": 1 "Role 1"}, {"Text": "Hello everyone, today we'll talk about a particularly interesting topic—the pet market. In recent years, the pet market has experienced explosive growth; in 2024, the number of pets exceeded 124 million, and the market size exceeded 300 billion yuan", "Role": "Role 2"}, {"Text": "Yes, that number is indeed quite astonishing. And I've found that the main force of pet owners now is young people, those born in the 90s and 00s."} "The post-90s generation accounts for the majority," "Role": "Role 1"}, {"Text": "Yes, especially the post-90s generation, which accounted for 41.2%, and the post-00s generation also quickly climbed to 25.6%. This shows that the younger generation really regards pets as a part of their lives," "Role": "Role 2"}, {"Text": "Yes, I agree. The pet market has huge potential, and more innovative products and services will emerge in the future. That's all for today's discussion. Thank you for listening, see you next time," "Role": "Role 1"}, {"Text": "See you next time," "Role": "Role 2"}]}.

[0100] In practical applications, there are various ways to generate audio generation scripts based on script prompts using task processing models. The specific method chosen depends on the actual situation, and this specification does not impose any limitations on this approach. One possible implementation of this specification involves using a single task processing model to generate the audio generation script based on script prompts. In this method, the audio reference document and audio generation parameters can be directly assembled into a script prompt template to obtain script prompt information. This script prompt information is then input into a task processing model to obtain the audio generation script. Another possible implementation of this specification involves using multiple task processing models to generate the audio generation script based on script prompts. In this method, the process of generating the audio generation script using task processing models based on script prompts can be viewed as synthesizing the audio generation script through a multi-agent group chat. Each task processing model can be considered a humanoid intelligent agent with fixed personas and background information. Specifically, firstly, agents matching the number of participants and style can be selected based on audio generation parameters (such as the number of participants and style). Then, the audio reference document and audio generation parameters are assembled into an opening statement task instruction, which is input to any agent. The agent then begins its opening statement in its own unique style. Subsequently, the agent's speech, background information (audio reference document and audio generation parameters), and podcast synthesis task instructions are input to downstream agents, automatically responding and engaging in conversation. This process repeats until the agent ends the dialogue on its own. The speech from all agents is then combined and structured to obtain the audio generation script.

[0101] In one optional embodiment of this specification, before generating the audio generation script based on script prompts using the task processing model, the following steps may be included:

[0102] Based on the audio generation parameters, at least one task processing model is selected from multiple candidate processing models, wherein the multiple candidate processing models correspond to different dialogue roles.

[0103] Using a task processing model, an audio generation script can be generated based on script prompts, which may include the following steps:

[0104] Using at least one task processing model, an audio generation script is generated based on script prompts.

[0105] It's important to note that candidate processing models refer to a set of pre-trained models capable of performing specific tasks (such as text analysis, sentiment recognition, and dialogue generation). Each candidate processing model may be optimized for different dialogue roles or scenarios. For example, one candidate processing model might be designed for a lighthearted and cheerful style, while another might be designed for a more emotionally stable style. The task processing model refers to the model among the multiple candidate processing models that meets the audio generation parameters. These audio generation parameters not only guide the generation of the target audio but also serve as an important basis for selecting task processing models. For example, if the audio generation parameters require a "gentle female voice," then a candidate processing model that excels at handling female character dialogues can be selected.

[0106] By applying the solutions in the embodiments of this specification, a suitable task processing model is selected from multiple candidate processing models based on the audio generation parameters. This fully utilizes the expertise of different models, which can significantly improve the quality and efficiency of the audio generation script and ensure that the final output target audio can accurately meet the requirements.

[0107] Step 108: Generate the target audio corresponding to the audio reference document based on the audio generation script.

[0108] It should be noted that target audio refers to the audio file ultimately generated based on the audio generation script. Target audio can take the form of voice narration in podcast programs or live-streaming e-commerce, or voice responses from virtual assistants. Through target audio, information can be directly conveyed to listeners or viewers, evoking emotional resonance or promoting interaction.

[0109] In practical applications, there are multiple ways to generate the target audio corresponding to the audio reference document based on the audio generation script. The specific method chosen depends on the actual situation, and this specification does not impose any limitations on this approach. In one possible implementation, the target audio corresponding to the audio reference document can be generated directly from the audio generation script. In another possible implementation, the audio generation script can be interactively enhanced, and the target audio corresponding to the audio reference document can be generated using the interactively enhanced target generation script.

[0110] The solution implemented in this specification provides a content framework for the audio generation script through an audio reference document, and further refines the details of the audio generation script using audio generation parameters, ensuring that the audio generation script highly matches the generation expectations. By guiding the task processing model through character dialogue using script prompts, the generated audio generation script not only possesses a high degree of realism but also accurately reflects the character interaction relationships, improving the efficiency and quality of audio generation script generation, and further enhancing the efficiency and quality of target audio generation.

[0111] In one possible implementation of this specification, since the audio generation script may only contain linear dialogue (i.e., characters taking turns speaking), lacking the complexity and interactivity of real dialogue, interactive enhancements can be used to make the script more closely resemble real dialogue scenarios. That is, generating the target audio corresponding to the audio reference document based on the audio generation script may include the following steps:

[0112] Interactive enhancements are applied to the audio generation script to obtain the target generation script, which includes overlapping dialogue content between the first dialogue character and the second dialogue character, and the first dialogue character and the second dialogue character are different dialogue characters.

[0113] Based on the target generation script, generate the target audio corresponding to the audio reference document.

[0114] It should be noted that interactive enhancement refers to a script optimization method that, based on the audio generation script, introduces logically related and temporally overlapping dialogue interactions between multiple different dialogue roles to make the audio generation script more immersive, realistic, and interactive. Interactive enhancement methods include, but are not limited to, adjusting the dialogue rhythm and inserting overlapping dialogue content between roles; the specific choice depends on the actual situation, and this specification does not impose any limitations on these methods. Overlapping dialogue content refers to content where multiple roles speak simultaneously within the same time period (e.g., responding, speaking in unison, or answering in rapid succession). This design can enhance the realism and dynamic effect of the dialogue, simulating natural conversation in reality. The target generation script refers to the audio generation script after interactive enhancement, which can more accurately reflect complex dialogue scenarios, especially overlapping dialogue content involving multiple roles. The first and second dialogue roles refer to roles in the audio generation script with different identities, responsibilities, or functions. For example, customer service representatives and customers, teachers and students, hosts and guests, etc. The first and second dialogue roles are usually roles specified by audio generation parameters or roles created based on audio reference documents.

[0115] For example, referencing the above audio generation script, the target generated script obtained by interactively enhancing the audio generation script can be: {"Dialogue": [{"Text": "Hello listeners", [Response: Hello~], "Role": 1 "Role 1"}, {"Text": "Hello everyone, today we're going to talk about a particularly interesting topic—the pet market. You know, in recent years, the pet market has experienced explosive growth. In 2024, the number of pets exceeded 124 million, and the market size exceeded 300 billion yuan. [Response: So much!]", "Role": "Role 2"}, {"Text": "Wow, that number is indeed quite astonishing. And I've recently discovered that the main force of pet owners now is young people..." The post-90s and post-00s generations make up the majority. [Echo: That's right], "Character": "Character 1"}, {"Text": "Yes, especially the post-90s generation, which accounts for 41.2%, and the post-00s generation has also risen to 25.6%. This shows that [pause] the younger generation really treats pets as family members," "Character": "Character 2"}, {"Text": "Haha, I think so too! In short, the pet market has unlimited potential, let's wait and see~ We had a great chat today, thanks for listening, see you next time! [Wave]", "Character": "Character 1"}, {"Text": "See you next time! [Suddenly remembers] By the way, listeners who own pets, remember to share photos of your furry friends~ See you next time~", "Character": "Character 2"}]}.

[0116] In practical applications, there are various ways to interactively enhance the audio generation script to obtain the target generated script. The specific method chosen depends on the actual situation, and this specification does not limit this approach. One possible implementation of this specification involves automatically enhancing the audio generation script using predefined rules. For example, if a customer expresses "anger," overlapping dialogue with a customer service representative responding simultaneously is inserted. If the dialogue contains the keyword "dispute," sections where both characters speak simultaneously are automatically added. Another possible implementation of this specification utilizes a task processing model to interactively enhance the audio generation script and obtain the target generated script.

[0117] By applying the solutions in the embodiments of this specification and interactively enhancing the audio generation script, a target generation script containing overlapping dialogue content can be obtained, thereby making the generated target audio more realistic and vivid, and more suitable for applications in complex dialogue scenarios, such as film and television dubbing, customer service training, and virtual assistant development.

[0118] In one optional embodiment of this specification, the above-described interactive enhancement of the audio generation script to obtain the target generation script may include the following steps:

[0119] The interactive prompts and audio generation script are input into the task processing model to obtain the target generation script. The interactive prompts are used to guide the task processing model to interactively enhance the audio generation script.

[0120] It should be noted that interactive prompts include at least one of the following: model role information, model task instructions, model processing rules, and enhancement examples. An interactive prompt could be something like, "Optimize the following two-person dialogue script to make it suitable for two live broadcasters to use for program broadcasting, with the focus on enhancing the sense of realism and naturalness in the interaction."

[0121] The model task instructions could be: "Learn optimization methods from examples in the colloquial expression knowledge base and / or interactive optimization knowledge base, and apply them in other ways. This requires adding more colloquial expressions you can think of to the dialogue, or changing sentence structure. Make your speech more paused, rhythmic, and emphatic, thus more colloquial and natural, and sound like you're thinking as you speak. It's not simply about adding softer interjections; the important thing is to introduce pauses to show thinking and give your partner time to think. Depending on the context and atmosphere of the dialogue, appropriately inject brief responses from your partner, such as [response: Indeed] or [response: Hmm], to show that your partner is listening. Appropriately introduce quick responses, repetitions, or laughter into the dialogue. Enhance the interactivity of the dialogue, strengthen memorability, and demonstrate the speakers' chemistry and relationship depth. The overall goal of optimization is to reflect genuine interaction and the natural flow of communication between the two people."

[0122] The model processing rules can be: "Injected placeholders [a:b] represent statements from the other speaker. Injected placeholders [a:b] should avoid appearing at the beginning or end of any speaker's complete turn of speech. Placeholders should be injected in appropriate places, but not excessively. All statements should be optimized to resemble conversational expressions of 'thinking while speaking.' No other content or reasoning explanations should be output." An enhanced example is shown below:

[0123] Conversational Expressions Knowledge Base: Original Sentence: "Hello everyone! Hey, the AI ​​world is really buzzing lately. I heard the 'scaling law' is about to fail." Optimized Sentence: "Hello everyone! Hey, the AI ​​world is really buzzing lately. I just heard the 'scaling law' is about to fail." Tips: Adding "ah" after "buzzing" and "just I" before "heard" makes the expression more conversational and natural. Original Sentence: "Traditional research methods often take months or even years, while new technologies can provide reliable predictions in hours." Optimized Sentence: "Traditional research methods often take months, ah, even years, while new technologies can provide reliable predictions in hours." Tips: Adding "ah" to interrupt "months, ah, even years" softens the statement, introduces a pause, gives the audience time to understand and digest the information, and also strengthens the emphasis on the length of time "years." Inserting "then" after "in hours" further emphasizes the time.

[0124] Interactive engagement optimization knowledge base: Original sentence: "As someone who has been working in the gallery industry for quite some time, and in a city like City A, a city of extravagance and decadence, having come from an even more extravagant city like City B to City A, a city of extravagance and decadence." Optimized version: "As someone who has been working in the gallery industry for quite some time, and in a city like City A, a city of extravagance and decadence [repeated: haha, extravagant and decadent], having come from an even more extravagant city like City B to City A, a city of extravagance and decadence." Tip: When participants in a conversation find certain wording particularly interesting or amusing, they will repeat the wording. Original sentence: "Yes, yes, yes, every time I cook, I feel like I'm fighting a battle in the kitchen." Optimized version: "Yes, yes, yes, every time I cook, I feel like I'm fighting a battle in the kitchen. [together: laughter]." Tips: [laughter] indicates laughter, accompanied by laughing while reading the text. When one person says something humorous or funny, both people will laugh together.

[0125] By applying the solutions described in the embodiments of this specification, and inputting interactive prompts and audio generation scripts into the task processing model, an interactively enhanced target generation script can be obtained. This method fully utilizes the capabilities of the task processing model, significantly improving the realism and expressiveness of the script, and providing important support for generating high-quality target audio.

[0126] In one optional embodiment of this specification, when generating the target audio, target timbre material can also be obtained, and audio generation based on the target timbre material can be achieved through sound replication. That is, the above-mentioned generation of the target audio corresponding to the audio reference document according to the audio generation script may include the following steps:

[0127] Obtain the target timbre material corresponding to the audio reference document;

[0128] Based on the target timbre material and the audio generation script, generate the target audio corresponding to the audio reference document.

[0129] It's important to note that target voice materials refer to sound samples or voice templates used to generate target audio. Target voice materials include, but are not limited to, the vocal characteristics of a specific character (such as accent) and emotional expression style (such as gentle, serious, excited, etc.). Target voice materials provide the vocal foundation for generating target audio, ensuring that the generated target audio matches expectations in timbre, tone, and emotion. For example, using young and gentle voice materials to generate customer service dialogue audio.

[0130] In practical applications, there are various ways to obtain the target timbre material corresponding to the audio reference document, and the specific method should be selected according to the actual situation. This specification does not impose any limitations on this method. In one possible implementation, the target timbre material corresponding to the audio reference document can be received from the user via a client. In another possible implementation, the audio reference document can be read from other data acquisition devices or a database.

[0131] Furthermore, there are multiple ways to generate the target audio corresponding to the audio reference document based on the target timbre material and the audio generation script. The specific method chosen depends on the actual situation, and this specification does not impose any limitations on this approach. In one possible implementation, TTS technology can be used to convert the text content in the audio generation script into speech and apply the sound features of the target timbre material. If a closer fit to the target timbre material is required, deep learning technology can be used to train a TTS model capable of mimicking the target timbre material, and this TTS model can be used to generate the target audio corresponding to the audio reference document. In another possible implementation, all the speech in the audio generation script can be aligned with the target timbre material, and then audio can be synthesized line by line using sound cloning technology. Finally, audio tracks can be superimposed and / or spliced ​​according to the splicing rules corresponding to actual needs to obtain the target audio corresponding to the audio reference document.

[0132] By applying the solutions in the embodiments of this specification, high-quality target audio can be generated by acquiring target timbre materials and combining them with an audio generation script. This enables the sound to be replicated according to the target timbre materials, ensuring that the generated target audio meets the expected requirements in terms of timbre, emotion, and content. This makes it suitable for fields such as virtual assistant development, film and television dubbing, and advertising production.

[0133] In an optional embodiment of this specification, before generating the target audio corresponding to the audio reference document according to the target timbre material and the audio generation script, the audio generation script may be subjected to script standardization processing to obtain a standard generation script, and the target audio corresponding to the audio reference document is generated according to the target timbre material and the standard generation script. When performing script standardization processing on the audio generation script, all Arabic numerals, unit symbols, and proper nouns contained in the audio generation script may be replaced with Chinese. For example, "2004年5月5日" should be standardized to "二零零四年五月五日", "135km / h" should be standardized to "一百三十五千米每时", etc. The script standardization processing for the audio generation script may invoke a text standardization tool, such as the open-source python project chn_text_norm.

[0134] In an optional embodiment of this specification, the target timbre material may contain interference information such as background music and background noise. Therefore, the target timbre material may be subjected to material standardization processing, that is, before generating the target audio corresponding to the audio reference document according to the target timbre material and the audio generation script described above, the following steps may be further included:

[0135] Perform material standardization processing on the target timbre material to obtain a standard timbre material;

[0136] Generating the target audio corresponding to the audio reference document according to the target timbre material and the audio generation script may include the following steps:

[0137] Generate the target audio corresponding to the audio reference document according to the standard timbre material and the audio generation script.

[0138] It should be noted that the material standardization processing refers to a process of performing systematic and standardized processing on the target timbre material, which aims to optimize the quality of the target timbre material to make it meet specific format requirements or technical specifications. The manner of performing material standardization processing on the target timbre material includes but is not limited to denoising, equalization adjustment, plosive repair, and speech speed and loudness normalization, which is specifically selected according to actual situations, and the embodiments of this specification do not make any limitation on this. Tools for separating background music and background noise are, for example, audio processing tools or application programming interfaces, and speech speed and loudness normalization can be simply implemented according to audio processing algorithms, such as frame extraction and sampling algorithms and RMS normalization calculation.

[0139] In practical applications, for the implementation of "generating the target audio corresponding to the audio reference document according to the standard timbre material and the audio generation script", reference may be made to the above-mentioned implementation of "generating the target audio corresponding to the audio reference document according to the target timbre material and the audio generation script", and the embodiments of this specification will not be described in detail again.

[0140] By applying the solutions in the embodiments of this specification to standardize the target timbre material, it can be ensured that the target timbre material becomes a high-quality standard timbre material after standardization, thereby providing a solid foundation for generating the target audio.

[0141] Considering the large number of model parameters in the task processing model and the limited computing resources of the client, the audio generation method proposed in the embodiments of this specification can be applied to, for example, Figure 2 The audio generation system shown is not limited to this. See also... Figure 2 , Figure 2 This specification illustrates an architecture diagram of an audio generation system according to one embodiment of the present specification. The audio generation system may include a client 202 and a server 204.

[0142] Client 202 is used to send the audio reference document and audio generation parameters to server 204;

[0143] Server 204 is used to construct script prompts based on the audio reference document and audio generation parameters. The script prompts guide the task processing model to conduct role dialogue to obtain the audio generation script. Using the task processing model, the audio generation script is generated based on the script prompts. Based on the audio generation script, the target audio corresponding to the audio reference document is generated. The target audio is then sent to client 202.

[0144] Client 202 is also used to receive target audio sent by server 204.

[0145] like Figure 2 As shown, the task processing model is deployed in server 204. Server 204 can connect to one or more clients 202 via a local area network (LAN), wide area network (WAN), internet connection, or other types of data network. Client 202 can include, but is not limited to, smartphones, tablets, laptops, PDAs, personal computers (PCs), smart home devices, and in-vehicle devices. Client 202 can also interact with the user through a graphical user interface to invoke the task processing model, thereby implementing the audio generation method provided in the embodiments of this specification.

[0146] It is worth noting that the audio generation method provided in the embodiments of this specification is generally executed by the server. However, in other embodiments of this specification, if the client's runtime resources can meet the deployment and runtime conditions of the task processing model, the client can also have similar functions to the server, thereby executing the audio generation method provided in the embodiments of this specification. In other embodiments, the audio generation method provided in the embodiments of this specification can also be executed jointly by the client and the server.

[0147] The following is in conjunction with the appendix Figure 3 Taking the application of the audio generation method provided in this specification in a sound replication scenario as an example, the audio generation method will be further explained. Among other things, Figure 3 The flowchart illustrates an embodiment of an audio generation method based on timbre materials provided in this specification, specifically including the following steps:

[0148] Step 302: Obtain the task data for the audio generation task, which includes the audio reference document, audio generation parameters, and target timbre materials.

[0149] Step 304: Based on the audio reference document and audio generation parameters, construct script prompts, which are used to guide the task processing model to conduct role dialogue to obtain the audio generation script.

[0150] Step 306: Using the task processing model, generate an audio generation script based on the script prompts.

[0151] Step 308: Generate the target audio corresponding to the audio reference document based on the target timbre material and audio generation script.

[0152] It should be noted that the implementation methods of steps 302 to 308 are the same as those of steps 102 to 108 above, and will not be described again in the embodiments of this specification.

[0153] The solution implemented in this specification guides the task processing model to generate a high-quality audio generation script using script prompts, ensuring natural and smooth dialogue between characters. At the same time, it achieves high fidelity and customization of timbre through target timbre materials, enabling sound replication and significantly improving the naturalness, realism, fluency, personalization, and generation efficiency of the target audio.

[0154] See Figure 4 , Figure 4 This specification illustrates a flowchart of an audio generation method according to one embodiment. The audio generation workflow consists of seven units, each supported by an algorithm with different functions. The seven units will now be described in detail.

[0155] Parsing Unit: Performs text parsing on the audio reference document to obtain the audio reference text; performs text normalization on the audio reference text to obtain the standard reference text;

[0156] Ideation Unit: Input the thinking prompts and standard reference text into the task processing model to obtain dialogue assistance information, wherein the dialogue assistance information includes at least one dialogue information point;

[0157] The completion unit: extracts the associated information corresponding to at least one dialogue information point from the dialogue association data; and fuses the dialogue auxiliary information and the associated information to obtain fused auxiliary information.

[0158] Synthesis Unit: Constructs script prompts based on audio generation parameters, standard reference text, and fusion assistance information; selects at least one task processing model from multiple candidate processing models based on the audio generation parameters, wherein the multiple candidate processing models correspond to different dialogue roles; and generates an audio generation script using at least one task processing model based on the script prompts.

[0159] Interaction Enhancement Unit: Inputs the interaction prompts and audio generation script into the task processing model to obtain the target generation script. The interaction prompts are used to guide the task processing model to interactively enhance the audio generation script. The target generation script includes the overlapping dialogue content of the first dialogue character and the second dialogue character. The first dialogue character and the second dialogue character are different dialogue characters.

[0160] Standardization Unit: Standardizes the target timbre material to obtain standard timbre material; standardizes the audio generation script to obtain standard generation script;

[0161] Sound replication unit: Generates the target audio corresponding to the audio reference document based on standard timbre materials and standard generation scripts.

[0162] It's worth noting that the parsing, conceptualization, and completion units enhance the richness of background information required for synthesizing interactive audio generation scripts. The conceptualization unit even incorporates dialogue processes and key points, making the synthesis unit more professional when creating interactive audio generation scripts. During the audio generation script synthesis process in the conceptualization unit, meticulously designed model task instructions and carefully annotated script examples significantly improve the quality of the audio generation scripts. While lacking more advanced and engaging interactions (such as echoing, simultaneous speaking, or quick-response), its quality is already very high and can be directly used for audio generation. The interaction enhancement unit addresses the shortcomings of the synthesis unit from the text side and overcomes the limitations of sound synthesis or replication processes (the inability to directly synthesize multi-person dialogue, let alone interactive multi-person dialogue). The standardization process of the standardization unit further avoids common problems in sound synthesis or replication processes, such as digital pronunciation errors and poor timbre quality leading to poor replication effects. In the sound replication unit, mature sound replication models can be selected to directly replicate audio with a realistic feel and natural rhythm.

[0163] In practical applications, the conception unit, completion unit, interaction enhancement unit, and standardization unit all serve to enhance the audio's effect. If the desired effect is not a primary concern, the parsing unit, synthesis unit, and sound replication unit can constitute a streamlined audio synthesis workflow. Based on the parsing unit, synthesis unit, and sound replication unit, arbitrarily placing the conception unit, completion unit, interaction enhancement unit, or standardization unit back into the workflow can create new secondary workflows to achieve the goal of dialogue audio synthesis.

[0164] Double-blind evaluation showed that the audio generation scheme proposed in the embodiments of this specification significantly outperformed other schemes in terms of the quality of the audio generation script and the listening experience of the target audio. The evaluation data is shown in Table 1 below.

[0165] Table 1. Human Evaluation Scoring Table

[0166]

[0167] See Figure 5 , Figure 5 This specification illustrates a flowchart of an information processing method based on a task processing model, provided in one embodiment. This method is applied to a task platform and specifically includes the following steps:

[0168] Step 502: Receive the model request sent by the terminal device.

[0169] Step 504: Based on the model request, determine the target task processing model from multiple task processing models, wherein the target task processing model is used in the processing of the audio generation method.

[0170] It should be noted that multiple task processing models can be task processing models with different model specifications and parameters, adapted to different scenarios. The target task processing model is the task processing model suitable for the target scenario. The target scenario can be different scenarios, such as code scenarios, financial scenarios, chat scenarios, etc. The model request includes the scenario identifier of the target scenario, the scenario input data of the target scenario, and at least one of the model specification parameters. Based on the model request, there are multiple ways to determine the target task processing model from multiple task processing models, and the specific method is selected according to the actual situation. This specification does not limit this in any way. In the first possible implementation of this specification, the corresponding target task processing model can be searched from the task processing models included in the first model library based on the scenario identifier included in the model request; in the second possible implementation of this specification, the target task processing model can be trained and obtained based on the scenario input data included in the model request; in the third possible implementation of this specification, the target task processing model can be selected based on the model request.

[0171] For example, based on the scene identifier of the target scene, at least one pre-trained task processing model can be searched from the first model library. Then, based on the model specification parameters, an intermediate task processing model can be selected from the at least one task processing model. Finally, based on the scene input data of the target scene, the selected intermediate task processing model can be trained to obtain a target task processing model suitable for user needs.

[0172] The solutions implemented in the embodiments of this specification are adapted to user needs to obtain target task processing models, realize personalized model services, provide users with an efficient, flexible and easy-to-use model service solution, and improve user experience.

[0173] In one optional embodiment of this specification, determining the target task processing model from multiple task processing models based on a model request may include the following steps:

[0174] If the model request includes a scene identifier of the target scene, the target task processing model suitable for the target scene is searched from the first model library based on the scene identifier. The first model library stores multiple task processing models suitable for different scenes.

[0175] When the model request includes scene input data of the target scene, the task processing model to be trained that is suitable for the target scene is determined from multiple task processing models, and the task processing model to be trained is trained based on the scene input data to obtain the target task processing model.

[0176] If the model request includes model specification parameters, the target task processing model corresponding to the model specification parameters is searched from the second model library. The second model library stores multiple task processing models with different model specification parameters.

[0177] It should be noted that scene identifiers are unique or specific labels used to distinguish different scenes. The First Model Library is a database that stores and manages various pre-trained deep learning models. Multiple task processing models adapted to different scenes cover various application scenarios and needs. The First Model Library allows users to select the appropriate task processing model according to their needs, or directly call the appropriate task processing model to perform audio generation tasks through the application programming interface. The multiple task processing models adapted to different scenes are various models stored in the First Model Library suitable for different scenarios, each optimized for a specific application scenario. For example, based on the scene identifier "finance" of the target dialogue scenario, one can search the First Model Library for a target task processing model suitable for the financial scenario.

[0178] Different task processing models are adapted to different scenarios. For example, task processing model 1 is suitable for scenarios 1 and 2, while task processing model 2 is suitable for scenarios 2 and 3. The task processing model to be trained refers to the model among multiple task processing models that is suitable for the target scenario, but whose performance can be further optimized. If the target scenario is scenario 1, then the task processing model to be trained is task processing model 1 adapted to scenario 1. The task processing model to be trained may not only be applicable to the target scenario but also to other scenarios, making it a general task processing model applicable to different scenarios. The task processing model to be trained can be used to perform the audio generation task of the target scenario, but the results may not be very good. In this case, the task processing model to be trained can be optimized based on the scenario input data of the target scenario. For example, optimizing the task processing model to be trained based on the scenario input data of a financial scenario can yield a target task processing model suitable for the financial scenario. The scenario input data of the target scenario can be understood as the sample data used to train the task processing model. The method for training the task processing model to be trained based on the scenario input data of the target scenario to obtain the target task processing model can be supervised fine-tuning.

[0179] Model specifications refer to the various parameters that define the structure and behavior of a model. These parameters can be broadly categorized into two types: model parameters (learnable parameters) and hyperparameters. Model parameters are those automatically adjusted during model training using the backpropagation algorithm, including but not limited to weight matrices and bias terms. For example, in a simple fully connected layer, the weight matrix is ​​a two-dimensional tensor connecting neurons in the input and output layers; the bias term is a one-dimensional vector providing additional offset values ​​for each output neuron. Hyperparameters are parameters set before model training begins, used to control the model's learning process and architecture. Hyperparameters include, but are not limited to, the learning rate and the number of neurons per layer, selected based on specific requirements.

[0180] The solutions implemented in this specification, based on scenario requirements, accurately identify the target task processing model suitable for the corresponding scenario through scenario identification, making the audio generation task processing more accurate and more aligned with the target scenario; based on scenario requirements, further train a general task processing model to be trained using scenario input data to obtain a target task processing model adapted to the target scenario, making the target task processing model more aligned with the target scenario, thereby improving user experience and task processing quality; based on model specification parameters, the corresponding target task processing model can be accurately found, ensuring the efficient and stable operation of the target task processing model and improving user experience.

[0181] In one optional embodiment of this specification, after determining the target task processing model from multiple task processing models based on the model request, the following steps may be further included:

[0182] Deploy the target task processing model, and build an audio generation interface based on the target task processing model, so that the terminal device can schedule the target task processing model to execute audio generation tasks through the audio generation interface.

[0183] It should be noted that the audio generation interface is an interactive programming interface for the terminal device to schedule the target task processing model to perform audio generation tasks, and it is usually provided in the form of an application programming interface (API). Through the audio generation interface, users can input task data for the audio generation task to perform audio generation task processing.

[0184] In practical applications, there are various ways to deploy the target task processing model, and the specific method should be chosen based on the actual situation. This specification does not impose any limitations on this approach. One possible implementation of this specification is to deploy the target task processing model on cloud-side devices using infrastructure provided by a cloud service provider. Another possible implementation of this specification is to deploy the target task processing model on edge devices using a lightweight framework. For example, the target task processing model can be deployed on a distributed system, and an audio generation interface can be built based on the target task processing model and provided to terminal devices, enabling the terminal devices to schedule the target task processing model to execute audio generation tasks.

[0185] By applying the solutions provided in the embodiments of this specification, deploying the target task processing model, and building an audio generation interface based on the target task processing model, terminal devices can efficiently call the target task processing model, thereby improving the processing quality and response speed of audio generation tasks.

[0186] See Figure 6 , Figure 6 This specification shows a schematic diagram of the structure of a task platform 600 provided in one embodiment of the specification. The task platform 600 includes a request interface 602 and a response unit 604.

[0187] Request interface 602 is used to receive a model request sent by a terminal device, wherein the model request includes at least one of the following: scene identifier of the target scene, scene input data of the target scene, and model specification parameters.

[0188] The response unit 604 is used to determine the target task processing model from multiple task processing models based on the model request, wherein the target task processing model is used in the processing of the audio generation method.

[0189] In one optional embodiment of this specification, the task platform further includes an audio generation interface, which is constructed based on the target task processing model;

[0190] The audio generation interface is used by terminal devices to schedule and execute audio generation tasks.

[0191] The above is an illustrative scheme of a task platform according to this embodiment. It should be noted that the technical solution of this task platform and the technical solution of the information processing method based on the task processing model described above belong to the same concept. For details not described in detail in the technical solution of the task platform, please refer to the description of the technical solution of the information processing method based on the task processing model described above.

[0192] Corresponding to the above-described audio generation method embodiments, this specification also provides embodiments of audio generation apparatus. Figure 7 A schematic diagram of an audio generation apparatus according to one embodiment of this specification is shown. Figure 7 As shown, the device includes:

[0193] The first acquisition module 702 is configured to acquire an audio reference document and audio generation parameters;

[0194] The first construction module 704 is configured to construct script prompt information based on the audio reference document and audio generation parameters. The script prompt information is used to guide the task processing model to conduct role dialogue to obtain the audio generation script.

[0195] The first generation module 706 is configured to use a task processing model to generate an audio generation script based on script prompts.

[0196] The second generation module 708 is configured to generate the target audio corresponding to the audio reference document based on the audio generation script.

[0197] Optionally, the second generation module 708 is further configured to interactively enhance the audio generation script to obtain a target generation script, wherein the target generation script includes overlapping dialogue content of a first dialogue character and a second dialogue character, and the first dialogue character and the second dialogue character are different dialogue characters; and to generate target audio corresponding to the audio reference document based on the target generation script.

[0198] Optionally, the second generation module 708 is further configured to input the interactive prompt information and the audio generation script into the task processing model to obtain the target generation script, wherein the interactive prompt information is used to guide the task processing model to interactively enhance the audio generation script.

[0199] Optionally, the second generation module 708 is further configured to obtain the target timbre material corresponding to the audio reference document; and generate the target audio corresponding to the audio reference document based on the target timbre material and the audio generation script.

[0200] Optionally, the device further includes: a first processing module configured to perform material standardization processing on the target timbre material to obtain standard timbre material; and a second generation module 708 further configured to generate target audio corresponding to the audio reference document based on the standard timbre material and the audio generation script.

[0201] Optionally, the first construction module 704 is further configured to obtain a script prompt template, wherein the script prompt template includes at least one of model role information, model task instructions, model processing rules, and script examples; and to add an audio reference document and audio generation parameters to the script prompt template to obtain script prompt information.

[0202] Optionally, the device further includes: an input module configured to input thinking prompts and an audio reference document into a task processing model to obtain dialogue assistance information, wherein the thinking prompts are used to guide the task processing model to think about the audio reference document from at least one dialogue dimension; and a first construction module 704 further configured to construct script prompts based on audio generation parameters, the audio reference document, and the dialogue assistance information.

[0203] Optionally, the dialogue assistance information includes at least one dialogue information point; the device further includes: a parsing module configured to parse the association information corresponding to at least one dialogue information point from the dialogue association data, wherein the dialogue association data and at least one dialogue information point are associated; to fuse the dialogue assistance information and the association information to obtain fused assistance information; and a first construction module 704 further configured to construct script prompt information based on audio generation parameters, an audio reference document, and the fused assistance information.

[0204] Optionally, the device further includes: a filtering module configured to filter at least one task processing model from multiple candidate processing models based on audio generation parameters, wherein the multiple candidate processing models correspond to different dialogue roles; and a first generation module 706 further configured to generate an audio generation script using at least one task processing model based on script prompts.

[0205] Optionally, the device further includes: a second processing module configured to perform text parsing on the audio reference document to obtain audio reference text; and to perform text normalization processing on the audio reference text to obtain standard reference text; and a first construction module 704 further configured to construct script prompt information based on audio generation parameters and standard reference text.

[0206] The solution implemented in this specification provides a content framework for the audio generation script through an audio reference document, and further refines the details of the audio generation script using audio generation parameters, ensuring that the audio generation script highly matches the generation expectations. By guiding the task processing model through character dialogue using script prompts, the generated audio generation script not only possesses a high degree of realism but also accurately reflects the character interaction relationships, improving the efficiency and quality of audio generation script generation, and further enhancing the efficiency and quality of target audio generation.

[0207] The above is an illustrative scheme of an audio generation device according to this embodiment. It should be noted that the technical solution of this audio generation device and the technical solution of the audio generation method described above belong to the same concept. For details not described in detail in the technical solution of the audio generation device, please refer to the description of the technical solution of the audio generation method described above.

[0208] Corresponding to the above embodiments of the audio generation method based on timbre materials, this specification also provides embodiments of the audio generation apparatus based on timbre materials. Figure 8 A schematic diagram of an audio generation device based on timbre materials according to one embodiment of this specification is shown. Figure 8 As shown, the device includes:

[0209] The second acquisition module 802 is configured to acquire task data for the audio generation task, wherein the task data includes an audio reference document, audio generation parameters, and target timbre material;

[0210] The second construction module 804 is configured to construct script prompt information based on the audio reference document and audio generation parameters. The script prompt information is used to guide the task processing model to conduct role dialogue to obtain the audio generation script.

[0211] The third generation module 806 is configured to use a task processing model to generate an audio generation script based on script prompts.

[0212] The fourth generation module 808 is configured to generate the target audio corresponding to the audio reference document based on the target timbre material and the audio generation script.

[0213] The solution implemented in the embodiments of this specification uses script prompts to guide the task processing model to generate high-quality audio generation scripts, ensuring natural and smooth dialogue between characters. At the same time, it achieves high fidelity and customization of timbre through target timbre materials, realizing sound replication and significantly improving the naturalness, realism, fluency, personalization, and generation efficiency of the target audio.

[0214] The above is a schematic scheme of an audio generation device based on timbre material according to this embodiment. It should be noted that the technical solution of this audio generation device based on timbre material belongs to the same concept as the technical solution of the audio generation method based on timbre material described above. For details not described in detail in the technical solution of the audio generation device based on timbre material, please refer to the description of the technical solution of the audio generation method based on timbre material described above.

[0215] Corresponding to the above-described embodiments of the information processing method based on the task processing model, this specification also provides embodiments of the information processing apparatus based on the task processing model. Figure 9 A schematic diagram of the structure of an information processing device based on a task processing model provided in one embodiment of this specification is shown. Figure 9 As shown, the device is applied to a mission platform and includes:

[0216] The receiving module 902 is configured to receive model requests sent by the terminal device;

[0217] The determination module 904 is configured to determine the target task processing model from multiple task processing models based on the model request, wherein the target task processing model is used in the processing of the audio generation method.

[0218] Optionally, the determining module 904 is further configured to: when the model request includes a scene identifier of the target scene, search for a target task processing model suitable for the target scene from a first model library based on the scene identifier, wherein the first model library stores multiple task processing models suitable for different scenes; when the model request includes scene input data of the target scene, determine a task processing model to be trained suitable for the target scene from multiple task processing models, and train the task processing model to be trained based on the scene input data to obtain the target task processing model; when the model request includes model specification parameters, search for a target task processing model corresponding to the model specification parameters from a second model library, wherein the second model library stores multiple task processing models with different model specification parameters.

[0219] Optionally, the device further includes: a deployment module configured to deploy a target task processing model and, based on the target task processing model, construct an audio generation interface so that the terminal device can schedule the target task processing model to execute an audio generation task through the audio generation interface.

[0220] The solutions implemented in the embodiments of this specification are adapted to user needs to obtain target task processing models, realize personalized model services, provide users with an efficient, flexible and easy-to-use model service solution, and improve user experience.

[0221] The above is an illustrative scheme of an information processing device based on a task processing model according to this embodiment. It should be noted that the technical solution of this information processing device based on a task processing model belongs to the same concept as the technical solution of the information processing method based on a task processing model described above. For details not described in detail in the technical solution of the information processing device based on a task processing model, please refer to the description of the technical solution of the information processing method based on a task processing model described above.

[0222] Figure 10 A structural block diagram of a computing device 1000 provided in one embodiment of this specification is shown.

[0223] The computing device 1000 includes:

[0224] Memory 1010 and processor 1020;

[0225] The memory 1010 is used to store computer programs / instructions, and the processor 1020 is used to execute the computer programs / instructions. When the computer programs / instructions are executed by the processor 1020, they implement the steps of the above-mentioned audio generation method, or the audio generation method based on timbre materials, or the information processing method based on the task processing model.

[0226] In one or more embodiments of this specification, the computing device can be understood as an integrated smart terminal, including but not limited to a server, desktop computer, personal computer, all-in-one model machine, mobile phone, tablet computer or other portable smart terminal, etc., and the computing device may have the model described in the above embodiments of this application pre-installed.

[0227] Specifically, this computing device can pre-install various types of models, including but not limited to models in natural language processing, visual processing, speech processing, code processing, and multimodal task processing, thus providing diverse model selection. In different product forms, this computing device can support one or more model usage methods, including but not limited to model training, model invocation, model fine-tuning, model deployment, model inference, and application. In some product forms, this computing device also supports model management, including but not limited to multi-type model management (supporting the management of discriminative, generative, and other types of models), model version control (supporting the control of different model versions), and model evaluation (evaluating model performance and effectiveness based on model evaluation tools). In other product forms, this computing device can also create applications based on models, providing application programming interface (API) invocation capabilities. Models can be invoked into created applications through the API interface, and application management tools are provided for application management and monitoring.

[0228] Furthermore, the computing device may also include data management (supporting the creation and management of model tuning datasets), a training center (providing abundant training resources to help users learn and master artificial intelligence technology), and basic control capabilities (providing enterprise-level basic control capabilities to ensure the security and efficient operation of the system). Through the above functions, it provides a comprehensive and integrated device for artificial intelligence development, training, deployment, and application.

[0229] Figure 11 A structural block diagram of an electronic device 1100 provided according to one embodiment of this specification is shown.

[0230] The memory 1110 and the processor 1120 are connected via a bus 1130;

[0231] The memory 1110 is used to store computer programs / instructions, and the processor 1120 is used to execute the computer programs / instructions. When the computer programs / instructions are executed by the processor 1120, they implement the steps of the above-mentioned audio generation method, or the audio generation method based on timbre materials, or the information processing method based on the task processing model.

[0232] Specifically, the components of the electronic device 1100 include, but are not limited to, a memory 1110 and a processor 1120. The processor 1120 and the memory 1110 can be connected via a bus 1130.

[0233] Electronic device 1100 may also include access device 1140, which enables electronic device 1100 to communicate with database 1150 storing data via one or more networks 1160. Examples of such networks include Public Switched Telephone Network (PSTN), Local Area Network (LAN), Wide Area Network (WAN), Personal Area Network (PAN), or combinations of communication networks such as the Internet. Access device 1140 may include one or more of any type of wired or wireless network interface (e.g., Network Interface Card (NIC)), such as IEEE 802.11 Wireless Local Area Networks (WLAN) wireless interface, Wi-MAX (World Interoperability for Microwave Access) interface, Ethernet interface, Universal Serial Bus (USB) interface, cellular network interface, Bluetooth interface, Near Field Communication (NFC) interface, and so on.

[0234] In one embodiment of this specification, the above-described components of the electronic device 1100 and Figure 11 Other components, not shown, can also be connected to each other, for example, via a bus. It should be understood that... Figure 11 The block diagram of the electronic device shown is for illustrative purposes only and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0235] Electronic device 1100 can be any type of stationary or mobile electronic device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable electronic devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary electronic devices such as desktop computers or PCs. Electronic device 1100 can also be a mobile or stationary electronic device.

[0236] The above is an illustrative scheme of an electronic device according to this embodiment. It should be noted that the technical solution of this electronic device belongs to the same concept as the above-described audio generation method, audio generation method based on timbre materials, and information processing method based on task processing model. For details not described in detail in the technical solution of the electronic device, please refer to the description of the above-described audio generation method, audio generation method based on timbre materials, or information processing method based on task processing model.

[0237] An embodiment of this specification also provides a computer-readable storage medium storing a computer program / instructions that, when executed by a processor, implement the steps of the above-described audio generation method, or the audio generation method based on timbre materials, or the information processing method based on a task processing model.

[0238] The above is an illustrative scheme of a computer-readable storage medium according to this embodiment. It should be noted that the technical solution of this storage medium belongs to the same concept as the above-described audio generation method, audio generation method based on timbre materials, and information processing method based on task processing model. For details not described in detail in the technical solution of the storage medium, please refer to the description of the above-described audio generation method, audio generation method based on timbre materials, or information processing method based on task processing model.

[0239] An embodiment of this specification also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the above-described audio generation method, or the audio generation method based on timbre materials, or the information processing method based on a task processing model.

[0240] The above is an illustrative scheme of a computer program product according to this embodiment. It should be noted that the technical solution of this computer program product belongs to the same concept as the above-mentioned audio generation method, audio generation method based on timbre material, and information processing method based on task processing model. For details not described in detail in the technical solution of the computer program product, please refer to the description of the above-mentioned audio generation method, audio generation method based on timbre material, or information processing method based on task processing model.

[0241] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0242] Computer instructions include computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. Computer-readable media can include: any entity or device capable of carrying computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in computer-readable media can be appropriately added or removed according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media do not include electrical carrier signals and telecommunication signals.

[0243] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments in this specification are not limited to the described order of actions, because according to the embodiments in this specification, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the embodiments in this specification.

[0244] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0245] The preferred embodiments disclosed above are merely illustrative of this specification. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the embodiments described herein. These embodiments are selected and specifically described in this specification to better explain the principles and practical applications of the embodiments, thereby enabling those skilled in the art to better understand and utilize this specification. This specification is limited only by the claims and their full scope and equivalents.

Claims

1. An audio generation method, comprising: Obtain the audio reference document and audio generation parameters; Based on the audio reference document and the audio generation parameters, script prompt information is constructed, wherein the script prompt information is used to guide the task processing model to perform role dialogue to obtain the audio generation script; Using the task processing model, the audio generation script is generated based on the script prompts. Based on the audio generation script, generate the target audio corresponding to the audio reference document.

2. The method according to claim 1, wherein generating the target audio corresponding to the audio reference document based on the audio generation script comprises: Interactive enhancement is performed on the audio generation script to obtain a target generation script, wherein the target generation script includes overlapping dialogue content of a first dialogue character and a second dialogue character, and the first dialogue character and the second dialogue character are different dialogue characters. Based on the target generation script, generate the target audio corresponding to the audio reference document.

3. The method according to claim 2, wherein the interactive enhancement of the audio generation script to obtain the target generation script includes: The interactive prompts and the audio generation script are input into the task processing model to obtain the target generation script, wherein the interactive prompts are used to guide the task processing model to interactively enhance the audio generation script.

4. The method according to claim 1, wherein generating the target audio corresponding to the audio reference document based on the audio generation script comprises: Obtain the target timbre material corresponding to the audio reference document; Based on the target timbre material and the audio generation script, generate the target audio corresponding to the audio reference document.

5. The method according to claim 4, further comprising, before generating the target audio corresponding to the audio reference document based on the target timbre material and the audio generation script: The target timbre material is subjected to material standardization processing to obtain standard timbre material; The step of generating the target audio corresponding to the audio reference document based on the target timbre material and the audio generation script includes: Based on the standard timbre material and the audio generation script, generate the target audio corresponding to the audio reference document.

6. The method according to any one of claims 1 to 5, wherein constructing script prompt information based on the audio reference document and the audio generation parameters includes: Obtain a script prompt template, wherein the script prompt template includes at least one of model role information, model task instructions, model processing rules, and script examples; Add the audio reference document and the audio generation parameters to the script prompt template to obtain the script prompt information.

7. The method according to any one of claims 1 to 5, wherein before constructing the script prompt information based on the audio reference document and the audio generation parameters, the method further comprises: The thinking prompts and the audio reference document are input into the task processing model to obtain dialogue assistance information, wherein the thinking prompts are used to guide the task processing model to think about the audio reference document from at least one dialogue dimension; The step of constructing script prompt information based on the audio reference document and the audio generation parameters includes: The script prompt information is constructed based on the audio generation parameters, the audio reference document, and the dialogue assistance information.

8. The method according to claim 7, wherein the dialogue assistance information includes at least one dialogue information point; Before constructing the script prompt information based on the audio generation parameters, the audio reference document, and the dialogue assistance information, the method further includes: From the dialogue association data, the association information corresponding to the at least one dialogue information point is parsed out, wherein the dialogue association data and the at least one dialogue information point are associated. The dialogue assistance information and the associated information are fused to obtain fused assistance information; The step of constructing the script prompt information based on the audio generation parameters, the audio reference document, and the dialogue assistance information includes: The script prompt information is constructed based on the audio generation parameters, the audio reference document, and the fusion assistance information.

9. The method according to any one of claims 1 to 5, wherein before generating the audio generation script using the task processing model based on the script prompt information, the method further includes: Based on the audio generation parameters, at least one task processing model is selected from multiple candidate processing models, wherein the multiple candidate processing models correspond to different dialogue roles; The step of generating the audio generation script using the task processing model and based on the script prompts includes: The audio generation script is generated using the at least one task processing model and based on the script prompts.

10. The method according to any one of claims 1 to 5, wherein before constructing the script prompt information based on the audio reference document and the audio generation parameters, the method further comprises: The audio reference document is parsed to obtain the audio reference text; The audio reference text is subjected to text normalization processing to obtain a standard reference text; The step of constructing script prompt information based on the audio reference document and the audio generation parameters includes: The script prompt information is constructed based on the audio generation parameters and the standard reference text.

11. An audio generation method based on timbre materials, comprising: Obtain task data for the audio generation task, wherein the task data includes an audio reference document, audio generation parameters, and target timbre materials; Based on the audio reference document and the audio generation parameters, script prompt information is constructed, wherein the script prompt information is used to guide the task processing model to perform role dialogue to obtain the audio generation script; Using the task processing model, the audio generation script is generated based on the script prompts. Based on the target timbre material and the audio generation script, generate the target audio corresponding to the audio reference document.

12. An information processing method based on a task processing model, applied to a task platform, comprising: Receive model requests sent by terminal devices; Based on the model request, a target task processing model is determined from multiple task processing models, wherein the target task processing model is used in the processing of the method as described in any one of claims 1 to 10.

13. The method according to claim 12, wherein determining the target task processing model from multiple task processing models based on the model request includes: If the model request includes a scene identifier of the target scene, a target task processing model suitable for the target scene is searched from the first model library based on the scene identifier. The first model library stores multiple task processing models suitable for different scenes. When the model request includes scene input data of the target scene, a task processing model to be trained that is suitable for the target scene is determined from the plurality of task processing models, and the task processing model to be trained is trained based on the scene input data to obtain the target task processing model. If the model request includes model specification parameters, the target task processing model corresponding to the model specification parameters is searched from the second model library, wherein the second model library stores multiple task processing models with different model specification parameters.

14. The method according to claim 12, further comprising, after determining the target task processing model from multiple task processing models based on the model request: Deploy the target task processing model, and based on the target task processing model, construct an audio generation interface so that the terminal device can schedule the target task processing model to execute audio generation tasks through the audio generation interface.

15. A task platform, comprising a request interface and a response unit; The request interface is used to receive model requests sent by the terminal device, wherein... The model request includes at least one of the following: the scene identifier of the target scene, the scene input data of the target scene, and the model specification parameters. The response unit is configured to determine a target task processing model from multiple task processing models based on the model request, wherein the target task processing model is used in the processing of the method as described in any one of claims 1 to 10.

16. The task platform according to claim 15, further comprising an audio generation interface, wherein the audio generation interface is constructed based on the target task processing model; The audio generation interface is used for the terminal device to schedule and execute audio generation tasks.

17. A computing device, comprising: Memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, which, when executed by the processor, implement the steps of the method according to any one of claims 1 to 14.

18. An electronic device comprising: A memory and a processor, the memory and the processor being connected via a bus; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, which, when executed by the processor, implement the steps of the method according to any one of claims 1 to 14.

19. A computer-readable storage medium storing a computer program / instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1 to 14.

20. A computer program product comprising a computer program / instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1 to 14.