Tone vector generation model training method, audio data generation method and related equipment

Abstract

The invention discloses a tone vector generation model training method, an audio data generation method and related equipment. The timbre vector generation model training method comprises the steps of obtaining a training data set comprising a role timbre vector and a role description text; generating a role description vector based on the role description text; according to a preset time step, adding a random noise vector to the character tone vector to obtain an intermediate state vector of the corresponding time step; and taking the intermediate state vector and the role description vector as input, taking prediction of a flow field direction pointing from the intermediate state vector to a role timbre vector as a learning target, training an initial timbre vector model, and obtaining a pre-trained timbre vector generation model. The audio data generation method comprises the following steps: generating a role description vector based on a role description text; inputting the role description vector into a pre-trained timbre vector generation model to obtain a target timbre vector; and inputting the target timbre vector into a speech synthesis model, and generating audio data matched with the role description text.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and in particular to training methods for timbre vector generation models, audio data generation methods, and related devices. Background Technology

[0002] With advancements in AI-powered speech synthesis technology, text-to-speech (TTS) technology has been widely applied in fields such as intelligent customer service, virtual assistants, and audiobook production. Especially in audiobook scenarios, character-based and multi-voice generation technologies can significantly enhance listener immersion and emotional expression. Assigning different voices to different characters makes the characters in the work more vivid and the story atmosphere more layered, which has become an important development direction in audiobook production.

[0003] In existing technologies, the generation of character voices typically relies on manual recording or speech cloning methods based on a small number of samples. Manual recording requires professional voice actors, resulting in high production costs and long production cycles. While speech cloning methods can generate voices from a small number of samples, the generated results still depend on real speech data and are limited by the quantity and quality of training samples. Furthermore, some studies have attempted to use conditional speech synthesis models to guide voice generation using text features; however, these models often struggle to accurately understand the correlation between the character's voice features at different times, leading to a low degree of matching between the text description and the generated voice. Summary of the Invention

[0004] Based on the above problems, this application provides a training method for a timbre vector generation model, an audio data generation method, and related equipment, aiming to solve the problem of low matching degree between character timbre and text description in timbre matching scenarios.

[0005] In a first aspect, embodiments of this application provide a training method for a timbre vector generation model, including:

[0006] Obtain a training dataset; wherein the training dataset includes character timbre vectors and character description text corresponding to the character timbre vectors;

[0007] Generate a character description vector based on the character description text;

[0008] According to the preset time step, add the corresponding random noise vector to the character timbre vector to obtain the intermediate state vector of the corresponding time step;

[0009] Using the intermediate state vector and the character description vector as input, and predicting the flow direction from the intermediate state vector to the character timbre vector as the learning objective, an initial timbre vector model is trained to obtain a pre-trained timbre vector generation model; the flow direction is, under the corresponding time step condition, a direction vector representing the direction of the intermediate state vector along the time step evolution direction to the character timbre vector;

[0010] The pre-trained timbre vector generation model is used to generate corresponding character timbre vectors based on the input character description text, and the character timbre vectors are used to generate audio data that matches the character description text.

[0011] Secondly, embodiments of this application provide an audio data generation method, including:

[0012] Generate a text vector based on the input character text description;

[0013] The text vector is input into a pre-trained timbre vector generation model to obtain a target timbre vector corresponding to the character text description; wherein, the pre-trained timbre vector generation model is obtained by the training method of the timbre vector generation model described in the first aspect of the embodiments of this application or any specific implementation of the first aspect;

[0014] The target timbre vector is input into the speech synthesis model to generate audio data that matches the character's text description.

[0015] Thirdly, embodiments of this application also provide a training apparatus for a timbre vector generation model, comprising:

[0016] A data acquisition unit is used to acquire a training dataset; wherein, the training dataset includes character timbre vectors and character description text corresponding to the character timbre vectors;

[0017] A text vector generation unit is used to generate a character description vector based on the character description text;

[0018] The noise-adding unit is used to add a corresponding random noise vector to the character timbre vector according to a preset time step to obtain the intermediate state vector of the corresponding time step.

[0019] The model training unit is used to train an initial timbre vector model by taking the intermediate state vector and the character description vector as inputs and predicting the flow direction from the intermediate state vector to the character timbre vector as the learning objective, thereby obtaining a pre-trained timbre vector generation model; the flow direction is, under the corresponding time step condition, a direction vector representing the direction of the intermediate state vector along the time step evolution direction to the character timbre vector.

[0020] The pre-trained timbre vector generation model is used to generate corresponding character timbre vectors based on the input character description text, and the character timbre vectors are used to generate audio data that matches the character description text.

[0021] Fourthly, embodiments of this application also provide an audio data generation apparatus, including:

[0022] The text vector generation unit is used to generate character description vectors based on the input character description text.

[0023] A timbre vector generation unit is used to input the character description vector into a pre-trained timbre vector generation model to obtain a target timbre vector corresponding to the character description text; wherein, the pre-trained timbre vector generation model is obtained by the training method of the timbre vector generation model described in the first aspect of the embodiments of this application or any specific implementation of the first aspect;

[0024] The speech synthesis unit is used to input the target timbre vector into the speech synthesis model to generate audio data that matches the character description text.

[0025] Fifthly, embodiments of this application also provide a computer device, including:

[0026] Processor, memory, input / output interfaces;

[0027] The memory is either a short-term storage memory or a persistent storage memory;

[0028] The processor is configured to communicate with the memory and execute instructions in the memory to perform the training method for the timbre vector generation model described in the first aspect of the embodiments of this application or any specific implementation thereof, or the audio data generation method described in the second aspect of the embodiments of this application or any specific implementation thereof.

[0029] Sixthly, embodiments of this application also provide a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, is used to implement the training method for the timbre vector generation model described in the first aspect or any specific implementation of the first aspect of the embodiments of this application, or the audio data generation method described in the second aspect or any specific implementation of the second aspect of the embodiments of this application.

[0030] In a seventh aspect, embodiments of this application also provide a computer program product storing a computer program / instruction, which, when executed by a processor, is used to implement the training method for the timbre vector generation model described in the first aspect or any specific implementation of the first aspect of embodiments of this application, or the audio data generation method described in the second aspect or any specific implementation of the second aspect of embodiments of this application.

[0031] As can be seen from the above technical solutions, the embodiments of this application have the following advantages:

[0032] This application's embodiments construct a training dataset containing character timbre vectors and their corresponding text descriptions, enabling the model to learn the correspondence between timbre features and text semantics. This allows the model to acquire the ability to express the association between character descriptions and timbre features during the training phase. Using random noise vectors as initial data and character timbre vectors as target data, and employing noise addition operations to construct intermediate state vectors, the model can establish a continuous mapping relationship between noise and real timbre during training. This allows it to learn the evolutionary process of timbre generation, improving training stability and generation diversity. By using intermediate state vectors and text vectors as input, the model predicts the flow direction from the intermediate state vectors to the target timbre vectors, enabling it to directly learn the changing trends of timbre distribution and thus more efficiently capture the mapping patterns between text semantics and timbre features. The pre-trained timbre vector generation model obtained by the training method provided in this application embodiment can generate a matching timbre vector based solely on the input character text description during the inference stage. Then, the timbre vector is input into the speech synthesis model to generate audio data that matches the text description, thereby significantly improving the matching degree between timbre and text description and the naturalness of the generated speech, achieving low-cost and high-efficiency personalized timbre generation. Attached Figure Description

[0033] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0034] Figure 1 A schematic diagram of a system architecture provided for an embodiment of this application;

[0035] Figure 2 A schematic diagram illustrating the training method of a timbre vector generation model provided in this application embodiment;

[0036] Figure 3 This is a schematic flowchart of an audio data generation method provided in an embodiment of this application;

[0037] Figure 4 This application provides a schematic diagram of a character audio generation system architecture.

[0038] Figure 5 A schematic diagram of a training device for a timbre vector generation model provided in an embodiment of this application;

[0039] Figure 6 This is a schematic diagram of an audio data generation device provided in an embodiment of this application;

[0040] Figure 7 This is a schematic diagram of a computer device structure provided in an embodiment of this application. Detailed Implementation

[0041] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0042] The method provided in this application embodiment can be applied to, for example, Figure 1 In the system architecture shown, terminal 102 communicates with server 101 via a network, and data storage system 100 stores data that server 101 needs to process or requires. Data storage system 100 can be integrated onto server 101 or located in the cloud or on other network servers. Terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices, such as smartwatches, smart bracelets, and head-mounted devices. Server 101 can be implemented using a standalone server or a server cluster composed of multiple servers. In this embodiment, data storage system 100 can be a standalone physical storage server, a distributed storage system composed of multiple storage nodes, or a cloud storage system providing services such as object storage, relational databases, cache databases, cloud databases, and cloud storage. For example, in applications dealing with massive amounts of audiobook content, distributed object storage can be used to improve scalability and access throughput; in personalized audio generation scenarios with high real-time response requirements, a hybrid architecture combining a cache database can be used to improve data reading speed, but this is not limited to the above examples.

[0043] Terminal 102 can serve as the initiator of audio generation requests, receiving user-inputted character description text and transmitting it to server 101 via the network; alternatively, it can serve as the display terminal for generated audio data, receiving audio data returned from the server and providing it to the user for playback, editing, or export. Terminal 102 can also possess character description text preprocessing capabilities, such as initially extracting feature vectors like descriptive keywords, gender tags, and age tags based on a built-in lightweight model to reduce the computational burden on the server side. Furthermore, terminal 102 can provide a graphical user interface, allowing users to set the guiding scale coefficient for timbre generation via sliders or dropdown options, controlling the range of timbre diversity to obtain audio data output that better matches the desired style.

[0044] Server 101 is equipped with a timbre vector generation module and an audio data generation module, which are used to execute the training method of the timbre vector generation model and the audio data generation method provided in this application, respectively. The timbre vector generation module in server 101 first performs vectorization operations on the character description text based on the training dataset stored in data storage system 100, and combines a training strategy of noise injection and flow direction prediction to train a timbre vector generation model capable of expressing character semantics with high quality. During training, multiple iterations are used to optimize the learnable parameters within the model, ultimately improving the model's responsiveness to input conditions. After the model training is completed, the audio data generation module can respond to the audio generation request initiated by terminal 102, inputting the input character description vector into the pre-trained timbre vector generation model to generate a target timbre vector, and then inputting this target timbre vector into the speech synthesis model to generate audio data matching the description semantics, and finally returning the audio data to terminal 102.

[0045] It should be noted that the method provided in this application can be executed independently on the server side, independently on the terminal device side, or collaboratively by the terminal device and the server. For example, for high-performance terminal devices with strong computing power, such as tablets or head-mounted devices powered by artificial intelligence (AI) chips, the trained timbre vector generation model can be deployed locally on the terminal, allowing the terminal to complete the parsing of character description vectors, the generation of timbre vectors, and the local synthesis of audio. For terminal devices with limited computing resources, preprocessing tasks (such as text encoding and vector extraction) can be completed locally on the terminal, while the generation tasks (such as timbre vector generation and audio synthesis) can be delegated to the server. Furthermore, in application scenarios with higher requirements for real-time performance and response speed (such as voice interactive games or virtual anchor systems), a distributed collaborative computing architecture can also be adopted, allowing the terminal and server to collaborate efficiently through asynchronous interfaces to achieve low-latency, high-quality timbre synthesis output.

[0046] Figure 1 The system architecture of the illustrated embodiments is provided to more clearly illustrate the technical solutions of the embodiments of this application and does not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will understand, with the evolution of system architecture and the emergence of new business scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems. For example, the method of this application can also be applied to emerging scenarios such as AI dubbing platforms, personalized voice assistants, and virtual character voice customization systems, and the model deployment location, computing distribution method, and user interaction interface can be flexibly configured according to business needs, thereby achieving a more adaptable and scalable audio generation service.

[0047] The method of this application will be further described in detail below, and some specific possible implementation examples will be provided. Embodiments of this application provide a training method for a timbre vector generation model, such as... Figure 2 As shown, the method includes steps S201-S204.

[0048] S201: Obtain the training dataset;

[0049] Here, the training dataset includes character timbre vectors and corresponding character description texts. Specifically, audiobook audio data can be collected as the raw training corpus. Target characters appearing in each chapter of each audiobook are identified, ensuring that each character corresponds to at least one audio data segment and at least one character text description. To obtain the character timbre vectors, a trained speaker recognition model CAM++ is used to extract features from each character's audio segments, resulting in a multi-dimensional vector representing the character's timbre characteristics. This timbre vector serves as the target data for the initial timbre vector model in subsequent training. To improve the accuracy and standardization of character semantic information, this application embodiment selects to perform structured processing on character description text. Specifically, a text generation model (such as DeepSeek, Qwen, etc.) can be used to summarize the personality of each character in the chapter, and a consistent character description text template is used to generate character description text. For example, the character description text template can be composed of gender, age, identity, and personality. For example, the character description text can be "This character is a young woman, the eldest daughter of a noble family, with a frank and romantic personality, a charming and lively temperament, and a clear and bright voice" or "This character is a young man, a page / apprentice, with a curious and cautious personality, a thirst for knowledge, a youthful and shy temperament, and a clear and bright voice." For descriptive terms that are vague or ambiguous, such as "young man" or "mature man," standardization processing can also be performed, that is, converting them into clear age categories such as "youth" or "middle-aged" to ensure that the character description text is clear, consistent, structurally standardized, and trainable. Similarly, gender can be limited to male, female, non-human, etc.; age category can be limited to child, youth, middle-aged, elderly; identity tags can be selected from a preset identity vocabulary, such as commoner, martial artist, noble family, etc.; personality traits can be mapped to preset personality categories based on semantic analysis results, such as calm, lively, cautious, etc. All the words limited above can be expanded or adjusted according to the actual application scenario, and are not limited to the specific examples mentioned above.

[0050] To further enhance the scale and diversity of training data, a paired dataset of "character description text - character voice vector" can be constructed. Specifically, considering the potential temporal differences in personality descriptions of the same character across different chapters, the voice vectors of independent character voice segments from each chapter are obtained and combined with the character's personality description text in each chapter to create multiple data pairs with different character description texts and voice expressions. For example, for a character A that appears in N chapters, the voice vectors of character A's voice segments from different chapters are extracted, and N different character description texts are generated based on the chapter context. These N character description texts are then associated with the voice vectors extracted from the corresponding chapters to form N sets of "character description text - character voice vector" training samples, thereby expanding the scale of the training dataset.

[0051] By unifying and standardizing the character description text and extracting character timbre vectors, the training dataset becomes more comprehensive in terms of timbre representation and semantic expression. This enables the model to learn more accurately the mapping relationship between the semantics of the character description text and the character timbre vector. At the same time, by constructing diverse paired data samples based on the text and timbre changes of the same character in different chapter contexts, the model's ability to generalize to changes in character features with semantic changes is significantly improved. This makes the subsequently generated template timbre vectors more consistent with the semantic expectations of the character description text, thereby improving the audio generation effect and character matching degree.

[0052] S202: Generate character description vectors based on character description text;

[0053] To obtain text vectors that accurately represent the semantic features of character description texts, comparative experiments can be conducted using various mainstream text encoding models, including T5-base, T5-Large, Roberta-base, and Qwen3-0.6B. The text vectors extracted by each encoder are used to drive the timbre vector generation model for training. Based on the trained audio results, performance is evaluated based on metrics such as gender recognition accuracy, age expression accuracy, and semantic-timbre matching degree.

[0054] For example, suppose the experimental results show that Qwen3-0.6B achieves optimal performance across all the aforementioned metrics, and is able to more accurately understand key information such as gender, age, identity characteristics, and personality traits contained in the character text. Therefore, Qwen3-0.6B can be ultimately selected as the text encoder to improve the semantic consistency and expressive accuracy of timbre synthesis.

[0055] In the specific processing of the text encoder, the character description text obtained in step S201 can be input into the text encoder to obtain a high-dimensional character description vector. Since the text vectors output by text encoders (such as Qwen3-0.6B, T5-Large, etc.) are typically 1024, 2048, or 4096 dimensions, while the character timbre vectors output by timbre embedding models such as CAM++ are generally 256 or 512 dimensions, during the training phase, the model usually requires that each input vector have the same or compatible dimensions to ensure that the mapping relationship between the semantics of the character description text and the character timbre vector can be effectively learned. Directly using the original high-dimensional text vectors may lead to fusion difficulties and affect the training effect.

[0056] Therefore, to achieve alignment and adaptation of the feature space, a multilayer perceptron (MLP) module can be introduced into the high-dimensional character description vector to reduce or transform its dimensionality, ensuring that its output matches the character timbre vector in dimensionality, thus obtaining a dimensionally matched character description vector. Through this unified vector representation, the model can simultaneously receive textual semantics and timbre feature information, enabling subsequent training phases to more accurately learn the correspondence between text content and timbre changes.

[0057] S203: Add noise to the character's timbre vector using a random noise vector to construct an intermediate state vector;

[0058] For each character timbre vector obtained in step S201, during training, a corresponding random noise vector is added to the character timbre vector at preset time steps to construct an intermediate state vector for the corresponding time step. The random noise vector can be generated according to a preset random distribution, such as a symmetrical distribution with a mean of zero, to ensure that the direction of noise perturbation in each dimension is random. Based on this, according to preset noise intensity parameters, the random noise vector is mixed with the character timbre vector so that the resulting intermediate state vector is numerically located between the pure noise vector and the target character timbre vector. The noise intensity parameter can be set to a real number between zero and one. When the noise intensity is small, the intermediate state vector is closer to the character timbre vector; when the noise intensity is large, the intermediate state vector is closer to the random noise vector. In practical applications, multiple different noise intensity parameters can be used for the same character timbre vector to construct a series of intermediate state vectors from weak noise to strong noise at different time steps, which can be used to cover different stages of the transition from pure noise to the target timbre during training.

[0059] S204: Using the intermediate state vector and the character description vector as input, and predicting the flow direction from the intermediate state vector to the character timbre vector as the learning objective, train the initial timbre vector model to obtain the pre-trained timbre vector generation model.

[0060] During the training phase, for each intermediate state vector constructed in step S203, the corresponding character description vector generated in step S202 is used as a joint input to the initial timbre vector model. Here, the timbre vector model can be a generative model containing a multi-layer neural network structure, used to output a direction vector representing the trend of change from the current intermediate state vector toward the target character timbre vector. In order to construct a clear learning objective, this embodiment determines the theoretical target flow field direction based on the difference between the intermediate state vector and the character timbre vector in the vector space under the same sample. Here, the flow field direction can be understood as the direction vector representing the intermediate state vector along the time step evolution direction pointing toward the character timbre vector under the corresponding time step conditions, that is, the ideal change direction from the intermediate state position toward the target timbre position. The direction vector predicted by the model is compared with the target direction vector, and a loss function is constructed based on the difference between the two, so that the loss value can reflect the degree of deviation between the model output result and the real flow field direction. In the actual training process, the above process is repeated with multiple samples and intermediate states corresponding to different noise intensities. The gradient backpropagation algorithm is used to continuously adjust the internal learnable parameters of the timbre vector model, such as the weight parameters and bias parameters of each fully connected layer in the multi-layer neural network, and the nonlinear activation weights in the feature mapping module. This allows the model to stably output prediction results close to the real flow field direction after repeated optimization, thus obtaining a converged pre-trained timbre vector generation model. In this embodiment, the pre-trained timbre vector generation model is used to generate corresponding character timbre vectors based on the input character description text. Therefore, it can be understood that the character timbre vector is used to generate audio data that matches the character description text. Specifically, in the inference stage, the character description text is first encoded to obtain a character description vector in the same format as in the training stage. This vector is then combined with the initial noise vector or the corresponding initial state vector as the model input. The pre-trained timbre vector generation model is called multiple times. Each time, the current state vector is updated according to the flow field direction output by the model, so that the state vector gradually approaches the target timbre distribution in the vector space along the direction learned by the model, ultimately obtaining a stable character timbre vector. By inputting the character's timbre vector into the speech synthesis model and jointly modeling it with the text content to be synthesized, the timbre characteristics of the speech can be controlled to the style represented by the character's timbre vector while maintaining the correct output of the text content. This generates target audio data that matches the character description text in terms of character temperament, personality, and timbre style.

[0061] Through the aforementioned model training and application steps, the pre-trained timbre vector generation model does not aim to directly predict the final target timbre vector. Instead, it learns how to gradually approximate the target character timbre vector from different intermediate state vectors, thereby mastering the variation law of timbre smoothly evolving from the noise space to the real timbre space. Therefore, it can be understood that the training method using the flow field direction as the learning target is actually a training method used in this application to improve the model's fitting ability and training stability for complex timbre distributions. This enables the model to generate character timbre vectors that highly match the descriptions, relying solely on the character description text during the inference stage. Furthermore, when used in conjunction with a speech synthesis model, it achieves consistency between text semantics and timbre expression, significantly improving the overall effect of generated speech in terms of character sense, emotional expression, and naturalness of sound.

[0062] It should be noted that existing timbre generation technologies mainly include two types of schemes. One type is the timbre generation method based on speech cloning. This method primarily relies on a small number of real speech samples of the target speaker to clone its timbre features. Therefore, if there are no available speech samples for the target character in the early stages of creation, the above method cannot construct the character's timbre from scratch. This makes it difficult to meet the needs of setting a large number of zero-sample character voices in scenarios such as audiobook creation, resulting in limited audio generation effects and restricting the application scope of the above method.

[0063] Another type is text-driven timbre generation methods, which typically employ a technical architecture combining a text encoder and a timbre generator, controlling timbre generation through character description text. However, this method uses only the character description text as a single input condition, and the text content of different characters can be ambiguous and inconsistent, resulting in insufficient personalized timbre differences learned by the model and weak voice differentiation between different characters. Secondly, the above-mentioned mapping process from character description text to character timbre vectors suffers from poor consistency, leading to insufficient stability in timbre generation and a tendency for timbre performances to deviate from the character's established characteristics. For example, a sickly young lady might be generated with a loud and powerful voice, affecting the character's credibility. Furthermore, since most existing technologies directly allow the model to predict the target timbre vector itself, the generated timbre either becomes too close to the training data or deviates too much from the semantics of the character description text, resulting in distorted audio sound style. Based on the above problems, in the timbre modeling process, this application uses random noise vectors and target timbre vectors as training inputs and outputs, respectively, and uses the continuously changing intermediate state between them as a supervision signal. This enables the timbre generation model to learn the mapping relationship between the starting point and the ending point, as well as the law of evolution from noise space to timbre space, thereby learning the correspondence between character description text and character timbre vectors more stably and accurately. Specifically, in one embodiment, according to a preset time step, a corresponding random noise vector is added to the character timbre vector to obtain the intermediate state vector of the corresponding time step, including: randomly sampling time steps within a preset time interval; according to the time step, using the random noise vector as initial data and the character timbre vector as target data, performing linear interpolation between the random noise vector and the character timbre vector to obtain the intermediate state vector of the corresponding time step.

[0064] To enable the model to learn the continuous evolution from a state of random noise to a state of realistic character timbre during training, a set of time step parameters (i.e., the noise intensity parameters mentioned in step S203) can be pre-defined to control the degree of noise addition. The preset time step mentioned in step S203 can be understood as a time step value selected from a preset time interval according to a predetermined time step selection rule, used to characterize the stage of the current intermediate state vector in the entire noise addition to denoising evolution process. The time step selection rule includes at least a random sampling method.

[0065] Specifically, taking the selection method of random sampling as an example, the time step can be denoted as... The time interval is pre-set as a closed interval from 0 to 1, and time step values ​​are randomly selected from this pre-set time interval according to a uniform distribution. This ensures that the probability of each time step being selected within the time interval is basically the same, allowing the model to cover a series of different transition stages from a pure noise state to a state close to the target timbre during training, rather than learning only under a few fixed noise intensities. In another embodiment, the time step selection rule can also be a segmented selection method, that is, dividing the pre-set time interval into multiple continuous sub-intervals, and selecting at least one time step value in each sub-interval; wherein, the time step value in each sub-interval can be selected by random sampling, uniform selection, or other pre-set methods. Through the above segmented selection method, it can be ensured that during the training process, each noise-adding stage corresponds to at least one intermediate state vector participating in model training, thereby avoiding the concentration of time step values ​​in a local interval and improving the overall learning effect of the model on the evolution law of different noise intensity stages.

[0066] Furthermore, to improve numerical stability, when designing the time-step related noise figure, a positive parameter much smaller than 1 can be set as the minimum noise ratio to avoid the noise being completely zero when the time step is 0, thereby ensuring that a very small but non-zero noise component is retained at the beginning of the entire noise addition process.

[0067] Therefore, in practical applications, the time step can be set within the interval [0, 1]. Random sampling is performed, and the time step can be used as a noise intensity parameter to control the interpolation ratio between the noise vector and the character's timbre vector. This is then combined with a preset linear interpolation function to... With character timbre vector Interpolation is performed between them to obtain the intermediate state vector at the current time step. Here, the linear interpolation function can be expressed as:

[0068] (1)

[0069] In equation (1), This represents a preset minimum noise ratio parameter, with a value close to 0 (e.g., 1×10). -4 Or 1×10 -3 ), to prevent when The time is completely pure noise; random noise vector Typically derived from a Gaussian distribution with a mean of 0 and a variance of 1, i.e., satisfying a standard normal distribution; character timbre vector This originates from the labeled real timbre features in the training data, specifically the character timbre vector extracted using a timbre embedding model such as CAM++. Through interpolation formulas, at different time steps... Under these conditions, a state close to pure noise can be obtained ( When approaching 0, it gradually transitions to a state close to the true timbre. A series of intermediate state vectors (approaching 1) .

[0070] It is necessary to understand that Used to control the proportion of noise. With time steps As the noise increases, the noise weight will continuously decrease, meaning that the noise component in the intermediate state vector will become less and less, making the character's timbre vector gradually more stable and clear. Meanwhile, This process inherently increases the proportion of semantic features in the character description text, causing the true semantics of the character description text to gradually become the dominant factor in the character's vocal timbre vector in the later stages. Therefore, during the interpolation process, the intermediate state vector will exhibit a gradual transition from initially consisting mainly of random noise to increasingly resembling the character's vocal timbre vector as the semantic information of the character text vector gradually increases.

[0071] Through the above-mentioned noise addition process based on linear interpolation, a continuous transition path from the standard Gaussian distribution to the real timbre distribution can be constructed. By sampling intermediate state vectors at different time steps for subsequent training, the subsequent flow field prediction model can learn how to gradually move any intermediate state to the real character timbre vector under different noise intensities and different character description text semantics, thereby improving the model's accuracy and generalization ability in timbre generation.

[0072] Based on the multi-time-step intermediate state vectors constructed above, this embodiment further utilizes these intermediate state vectors as training supervision signals to guide the learning process of the timbre generation model. Specifically, in one embodiment, using the intermediate state vectors and character description vectors as inputs, and predicting the flow field direction from the intermediate state vectors to the character timbre vectors as the learning objective, an initial timbre vector model is trained to obtain a pre-trained timbre vector generation model. This includes: using the intermediate state vectors and character description vectors as inputs, predicting the target flow field direction at the corresponding time step through the initial timbre vector model; constructing a loss function based on the mean square error between the predicted target flow field direction and the actual flow field direction determined by the intermediate state vectors and the character timbre vectors; and performing backpropagation optimization on the parameters of the initial timbre vector model based on the loss function to obtain the pre-trained timbre vector generation model.

[0073] In this embodiment, a diffusion model architecture can be used to train the timbre vector generation model. The training objective is to predict the flow direction from the intermediate state vector to the character timbre vector, thereby optimizing the model's ability to model the character timbre vector. Specifically, during training, the intermediate state vector and the character description vector are used as inputs. The initial timbre vector model is constructed to predict the flow direction at the current time step. It is based on the noise vector With character timbre vector The weighted interpolation is used to obtain the result, and its calculation formula can be found in the above formula (1).

[0074] Based on the aforementioned intermediate state vector The settings can be configured to set the training objective as the predicted vector output by the model. Able to accurately fit from intermediate states The target character's timbre vector The true direction of the flow field, where the true vector field of the flow field is... It can be represented as:

[0075] (2)

[0076] in, Represents a random noise vector; This represents a preset minimum noise ratio parameter, with a value close to 0 (positive number). Represents the character's timbre vector. The real timbre features already labeled in the training data, that is, the character timbre vectors extracted by timbre embedding models such as CAM++; This represents the theoretical real flow field, while this represents the real flow field that the model needs to learn.

[0077] In this embodiment, the mean squared error (MSE) loss function can be used to measure the predicted value. Compared with the true value The difference between them, their loss function The definition is as follows:

[0078] (3)

[0079] in, Indicates at time step ,noise and character timbre vector All three factors together constitute the expected value under the given conditions. Represents a character description vector; Represents the intermediate state vector in the noisy environment Time step and character description vectors All three factors together constitute the flow field predicted by the model under the given conditions. Represents the actual flow field.

[0080] More specifically, in terms of model structure design, the initial timbre vector model adopts a network architecture based on DIT (Diffusion Transformer), containing four DIT blocks. Each layer contains a four-head self-attention mechanism and a feedforward network module, and nonlinear feature transformation is achieved through GELU (Gaussian Error Linear Unit) or SiLU (Sigmoid Linear Unit) activation. The character description vector is input into the pre-trained timbre vector generation model to obtain the target timbre vector corresponding to the character description text. This includes: mapping the time step encoding to the first scaling factor and first offset factor of each layer's neural network features; using the first scaling factor and first offset factor to perform channel-by-channel modulation on the current layer's features to control the denoising intensity at different time steps; mapping the character description vector to the second scaling factor and second offset factor of each layer's normalization operation; and using the second scaling factor and second offset factor to perform an affine transformation on the layer's normalized features to inject the semantic information of the character description vector into the layer's normalized features.

[0081] Specifically, regarding conditional information injection, the time step information t is applied to each layer of the network through FiLM (Feature-wise Linear Modulation), that is, the time step encoding is input into a nonlinear mapping network to generate a first scaling factor that varies with time. With the first offset coefficient And perform channel-by-channel modulation on the feature map of this layer. Specifically, for a feature map with a number of channels... The feature map, its first The output of each channel is represented as: Output feature map = Input feature map × + ,in Number of channels The size of the Transformer network feature map in the channel dimension is determined by the same dimension as the hidden layer of the model, and is used to accommodate multiple independent acoustic feature subspaces. In one feasible embodiment, the number of channels... It can be set to positive integers such as 256 and 512 to meet the timbre representation requirements of different model sizes, but is not limited to the above values. Those skilled in the art can adjust the values ​​according to computing power and the target timbre vector expression accuracy. Adjustments and optimizations were made. Second scaling factor. Indicates the action on the feature map. The scaling factor for each channel controls the amplification or suppression of the acoustic features of that channel, thereby altering the intensity of timbre expression at different diffusion stages to enhance or weaken specific timbre attributes (such as timbre brightness, vocal tension, and timbre smoothness). Second offset coefficient. Indicates the action on the feature map. The offset of each channel is used to adjust the baseline bias of that channel after feature normalization, making the generated result more consistent with the semantics of the character description in terms of timbre (such as expressing differences in style like calmness, liveliness, dignity, and gentleness). This channel-by-channel affine adjustment allows the model to adaptively adjust the intensity of timbre feature expression at different stages of diffusion sampling. For example, in the initial high-noise stage, it reconstructs the general outline of the timbre, while in the later low-noise stage, it focuses on adjusting timbre details such as breathiness, brightness, turbidity, and resonance changes to generate a more accurate target timbre vector.

[0082] Meanwhile, character description vectors, as another type of conditional input, are injected into the Transformer layer through Style-Adaptive Layer Normalization (SALN), which maps the character description vectors to the scaling parameters corresponding to the normalization layer. With offset parameter And perform channel-wise affine adjustment on the feature standardization results, the calculation form of which is: ,in Presentation layer normalization operation, This indicates a Hadamard multiplication by channel; Indicates the first Scaling factors on each channel used to enhance or suppress the amplitude of the acoustic features of that channel. Indicates the first Each channel has a translation factor used to control the degree of acoustic style shift. Since SALN operates on the feature statistics of each layer of the network, semantic information such as gender (e.g., male, female, non-human categories), age (e.g., youth, middle-aged, elderly), identity labels (e.g., commoner, warrior, noble), and personality traits (e.g., calm, lively, cautious) can be injected into different acoustic feature channels to establish a correspondence between semantics and timbre components at the feature level, thereby achieving timbre differentiation for different roles.

[0083] Through the above process, based on the constructed loss function, the model parameters are trained using an end-to-end backpropagation mechanism. The goal is to minimize the mean square error between the predicted flow field direction and the actual flow field direction, thereby continuously improving the model's ability to model text in the direction of timbre mapping. Finally, a pre-trained timbre vector generation model can be obtained, which can generate timbre vectors that match semantic information based on any character description text, and input them into the speech synthesis model to generate the corresponding character audio.

[0084] Furthermore, this application specifically addresses the problems of limited training data and lack of character style in existing technologies. Most publicly available audio data originates from general reading voices, such as news broadcasts, voice assistants, or narration. These voices typically have a neutral timbre and consistent style, lacking the ability to differentiate between multiple characters' tones and personalities, and also lacking the expressive differences corresponding to dramatic performance, such as age distinctions, character tension, authority, and temperament. Models trained using this type of audio data often only learn averaged timbre expressions. In multi-character audiobook scenarios, this easily leads to problems such as difficulty in distinguishing multiple characters' voices and inconsistencies in the voice style of the same character, resulting in decreased character expressiveness and listener immersion. This application addresses these issues by pairing character description text with character timbre vectors for training, enabling the model to establish a precise mapping from text to timbre distribution. This significantly improves the ability to express differences between characters and the consistency of the same character's voice across scenes, making it more suitable for applications such as multi-character audiobook production.

[0085] To enhance the generalization ability and robustness of the initial timbre vector model under conditions with and without text, in one embodiment, before constructing the loss function based on the mean square error between the predicted target flow field direction and the real flow field direction determined by the intermediate state vector and the character timbre vector, the training method of this application embodiment further includes: masking the character description vector with a preset probability, so that the initial timbre vector model predicts the target flow field direction of the intermediate state vector without the indication of the character description vector;

[0086] The loss function is constructed based on the mean square error between the predicted target flow field direction and the actual flow field direction determined by the intermediate state vector and the character timbre vector. This includes constructing the loss function based on the mean square error between the predicted target flow field direction and the actual flow field direction under two conditions: with and without character description vector indication. This allows the initial timbre vector generation model to learn the flow field mapping relationship under the conditions with and without character description vector indication based on the loss function.

[0087] In this embodiment of the application, the input role description vector can be masked with a preset probability during the training phase to form samples without role description indication. The probability value can be set to 0.2, that is, 20% of the training samples do not contain role description vectors during the training process. This is equivalent to constructing an unconditionally generated scenario, so that the model can learn the flow field prediction ability of intermediate state vectors even without semantic guidance.

[0088] After completing the aforementioned masking process, the training continues, specifically determining the true flow field direction based on the constructed intermediate state vector and character timbre vector, and constructing a loss function. Specifically, during loss function construction, the mean square error between the predicted target flow field direction and the true flow field direction is calculated under two conditions—conditional (with character description vectors) and unconditional (without character description vectors). The true flow field direction is defined as the gradient direction of the interpolation path between the character timbre vector and the noise vector, which can be derived from the vector difference. This indicates that the predicted value is ,in The character description vector is used, and in the unconditional case, it can be either the zero vector or a specific masking identifier. The final loss function is the weighted sum of the mean squared errors under the two conditions, which ensures that the model learns two types of flow field mappings simultaneously during training. This enhances the controllability of the model's dependence on the character description vector and helps to improve the model's generalization ability.

[0089] Accordingly, this application also provides an audio data generation method, which includes the following steps S301 to S303.

[0090] S301: Generate a character description vector based on the input character description text.

[0091] Character description text is used to summarize the personality of different characters in each chapter. The textual descriptions of different character personalities can include gender, age, status, and personality traits, such as "This character is a young male from a noble family, with a rebellious yet righteous personality, a heroic and unrestrained temperament, and a deep and mellow voice." This character description text is input into a text encoder, such as a semantic encoder built based on a pre-trained language model (e.g., BERT or Qwen), which performs word segmentation, encoding, and embedding mapping on the text, ultimately generating a set of continuous vectors as the semantic representation corresponding to the character description text, i.e., the character description vector.

[0092] S302: Input the character description vector into the pre-trained timbre vector generation model to obtain the target timbre vector corresponding to the character description text.

[0093] Here, the pre-trained timbre vector generation model is obtained according to the training method of the timbre vector generation model described in the first aspect or any specific implementation of the first aspect of the embodiments of this application; the timbre vector generation model is a diffusion model structure, and its training process refers to the training method of the timbre generation model of this application. In the embodiments of this application, the model input includes the character description vector, the intermediate state vector and time step encoding in the sampling generation process, and injects the character semantics into each layer feature map in the multi-layer DIT structure through SALN, thereby controlling the generation direction and finally outputting the target timbre vector.

[0094] S303: Input the target timbre vector into the speech synthesis model to generate audio data that matches the character description text.

[0095] The speech synthesis model can be an existing end-to-end speech synthesis architecture, such as a model built on the Tacotron series or VITS structure. After receiving the target timbre vector, it is used as a conditional input to modulate acoustic features (such as Mel spectrograms) for generation. Subsequently, the Mel spectrogram is converted into a final playable audio signal by a neural vocoder module (such as HiFi-GAN or WaveGlow).

[0096] To effectively map textual semantic information to the timbre vector space and gradually restore a target timbre vector matching the character description text from a random noise vector, in one embodiment, the character description vector is input into a pre-trained timbre vector generation model to obtain the target timbre vector corresponding to the character description text. This includes: using a randomly generated noise vector as the initial state vector; and in the iteration process from the start time step to the end time step, performing the following steps for each time step: inputting the current time step, the current state vector, and the character description vector into the timbre vector generation model to predict the flow direction of the current time step; updating the state vector according to the flow direction of the current time step, and using the updated state vector as the input of the next time step; and using the state vector output at the end time step as the target timbre vector.

[0097] When inputting the character description vector into the pre-trained timbre vector generation model, the first step is to input the randomly generated noise vector. This noise vector is input to the model as the initial state vector. Referring to the aforementioned example involving intermediate state vectors, this noise vector can be randomly sampled from a standard normal distribution, for example, following a Gaussian distribution with a mean of 0 and a variance of 1, denoted as . The noise vector, being prior data lacking any speaker timbre information, serves as the starting point for the reverse generation path. This allows the model to progressively infer the target timbre distribution from a purely noisy state, thus achieving controllability and randomness in the timbre generation process. Simultaneously, inputting a role description vector containing gender and age as a condition into the model enables the generated timbre to possess more clearly defined and distinctive role identity features.

[0098] During the iteration process from the start time step to the end time step, the pre-trained timbre vector generation model performs parameter update operations step by step to gradually eliminate noise and approximate the real timbre vector space. Specifically, the time step index t is set to monotonically change within a preset range, for example, t increases from 1 to T (e.g., T=1000). At each time step, the current time step code, the current state vector, and the character description vector are input into the pre-trained timbre vector generation model. Internally, the model uses a conditional diffusion structure combined with flow field prediction to predict the target flow field direction of the current time step based on the input content, that is, the gradient estimate from the current state vector to the target timbre vector direction, thereby generating the change path from the intermediate state vector to the target timbre vector, enabling the model to accurately converge to the target timbre space.

[0099] Based on the flow direction predicted at the current time step, the current state vector is updated to gradually approach the target timbre vector during continuous time inference. The state vector update can be performed using a progressive accumulation mode, where the flow direction vector is linearly superimposed with the time step interval according to a proportional coefficient to obtain the state vector for the next time step. This ensures that each time step completes a directional translation of the timbre feature towards the target region. Simultaneously, the character description vector, as a semantic injection signal, continuously participates in the state update throughout all iterations, ensuring that the flow direction prediction is always constrained by the semantic information of the character description vector. Finally, after completing all iterations at the preset termination time step, the output state vector is the target timbre vector.

[0100] Through the stepwise noise reduction timbre generation based on flow field prediction described above, this application embodiment realizes the ability to generate timbre vectors driven by character description text. Compared with the existing timbre cloning method whose training is limited by real speech samples, this application can automatically create timbre features that conform to the semantic style of character description text without inputting real audio.

[0101] To ensure semantic matching between the target timbre and the character text while achieving timbre diversity adjustment and generation flexibility control, in one embodiment, the character description vector is input into a pre-trained timbre vector generation model to obtain a target timbre vector corresponding to the character description text. This includes: receiving a guiding scale coefficient set by the adjustment operation in response to an adjustment operation input to an interactive interface element; generating a target timbre vector based on the character description vector and the guiding scale coefficient; wherein the guiding scale coefficient is used to control the degree of deviation between the target timbre vector and the timbre in the training dataset.

[0102] In this embodiment, a graphical user interface including interactive elements can be provided. These interactive elements may specifically be slider controls, numerical input boxes, or dial selectors, allowing users to adjust the guidance scale coefficient within a preset range (e.g., from 0 to 5) to control the degree of guidance during the target timbre vector generation process. Users can perform adjustment operations through the interactive components in this interface. The audio data generation system responds to the adjustment operation by receiving the guidance scale coefficient set by the user, enabling users to autonomously adjust between conservatively inheriting existing timbres and exploring entirely new timbres, thereby achieving more precise control over the character's timbre characteristics.

[0103] After receiving the guiding scale coefficient, it is combined with the character description vector to control the inference process of the timbre vector generation model. In specific implementation, the guiding scale coefficient guides the degree of fusion between conditional and unconditional generation results; its setting directly determines the deviation between the target timbre vector and the timbre in the training data. Specifically, it sets... A lower guiding scale factor (e.g., 0 to 1) will result in generated timbres that are closer to the timbral characteristics of similar characters in the training set, exhibiting a conservative and stable speech performance, suitable for scenarios with clearly defined characters or fixed styles. A higher guiding scale factor (e.g., 3 to 5) will encourage the generated timbres to expand and innovate beyond the original training data while retaining the basic semantics of the character, presenting a more personalized and creative timbral style, suitable for character settings that require distinct personality, dramatic changes, or stylistic leaps. Furthermore, embodiments of this application can provide a real-time audio preview function, allowing users to preview and select the optimal timbral output result at different scale factors.

[0104] To achieve controllable adjustment of character semantics and timbre innovation during timbre generation, in one embodiment, a target timbre vector is generated based on the character description vector and the guiding scale coefficient. This includes: inputting an empty vector into a timbre vector generation model to obtain a first vector result; inputting the character description vector and a randomly generated noise vector into the timbre vector generation model to obtain a second vector result; and weighting and fusing the first vector result and the second vector result based on the guiding scale coefficient to obtain the target timbre vector.

[0105] In one specific implementation, this application embodiment inputs an empty vector into a pre-trained timbre vector generation model to simulate the natural timbre generation trend of the model without any conditional input. The empty vector can be specifically set as an all-zero vector, a random vector following a normal distribution with a mean of zero, or other placeholder vectors without semantic information. By inputting this empty vector into the model, a first vector result is obtained. This result can be understood as the expression of the timbre vector in an unconditional state, representing the timbre representation generated by the model based on its inherent priors when no character semantic information is provided. Subsequently, a character description vector containing semantic information and a randomly generated noise vector are input together into the timbre vector generation model to generate a second vector result. In this way, by combining semantically conditional input (i.e., the character description vector) with random perturbation (i.e., the noise vector), the model is guided to generate timbre representations that highly match the character semantics but possess a certain degree of diversity.

[0106] Next, based on the set guiding scale coefficient, the aforementioned first vector result and second vector result are weighted and fused to obtain the final target timbre vector. In specific implementation, this fusion operation can be achieved using a linear interpolation formula: the target timbre vector equals the first vector result plus the guiding scale coefficient multiplied by the difference between the second vector result and the first vector result. When the scale coefficient is zero, the target timbre vector is an unconditionally generated result. As the scale coefficient increases, the target timbre vector tends more towards the generated result guided by conditional information composed of the character description vector and randomly generated noise vector. In this way, the model retains its basic generation capabilities while achieving responsive timbre generation based on the semantics of the character description text, ensuring both flexibility and accuracy in timbre generation.

[0107] like Figure 4 As shown, the overall architecture of the character audio generation system provided in this application embodiment may include three parts: a data layer, a model layer, and an application layer. These layers form a complete character audio generation chain through data flow and model inference processes. For example, in the data layer, character audio data can be collected, and character timbre vectors can be extracted using a timbre embedding model, such as a deep network-based timbre embedding extraction model (e.g., the Cam++ model). Simultaneously, the corresponding character description text is obtained, and the original text is cleaned and standardized, including removing ambiguous expressions and standardizing the format of gender / age / identity / personality descriptions, to form structured character description text. Finally, a paired training dataset of "character description text - character timbre vector" is constructed and used as the training basis for the timbre vector generation model.

[0108] At the model layer, a text encoder is first used to encode the character description text, such as using a pre-trained language model like Qwen-0.6B, to generate character description vectors. Then, an MLP is used to map the text embeddings to match the dimension of the character timbre vector. Next, the character description vector and random noise are used as input, and the result is inferred through a DiT-based diffusion generation network to output the target character timbre vector.

[0109] At the application layer, the generated character voice vector can be input into the speech synthesis engine, combined with the text to be dubbed, to generate high-quality audio with the target character's voice. Meanwhile, to enhance user controllability and interactive experience, this application embodiment provides an adjustable guidance scale coefficient. The user interface includes features such as slider adjustment, preview functionality, and audio library management. This character audio generation system can also be integrated into mini-programs, audio production software, or live streaming plugins, such as the OBS (Open Broadcaster Software) plugin, facilitating quick generation and application of character audio in real-world audio creation or content production scenarios.

[0110] To implement the training method of the timbre vector generation model in the embodiments of this application, the embodiments of this application also provide a training device for the timbre vector generation model, such as... Figure 5 As shown, the device includes:

[0111] The data acquisition unit 501 is used to acquire a training dataset; wherein, the training dataset includes character timbre vectors and character description text corresponding to the character timbre vectors;

[0112] The text vector generation unit 502 is used to generate a character description vector based on the character description text;

[0113] The noise-adding unit 503 is used to add a corresponding random noise vector to the character timbre vector according to a preset time step to obtain the intermediate state vector of the corresponding time step.

[0114] The model training unit 504 is used to train an initial timbre vector model by taking the intermediate state vector and the character description vector as inputs and predicting the flow direction from the intermediate state vector to the character timbre vector as the learning objective, so as to obtain a pre-trained timbre vector generation model; the flow direction is a direction vector that represents the intermediate state vector along the time step evolution direction to the character timbre vector under the corresponding time step condition.

[0115] The pre-trained timbre vector generation model is used to generate corresponding character timbre vectors based on the input character description text, and the character timbre vectors are used to generate audio data that matches the character description text.

[0116] In one embodiment, the noise-adding unit 503 is specifically used to: randomly sample time steps within a preset time interval; and according to the time steps, use a random noise vector as initial data and the character timbre vector as target data to perform linear interpolation between the random noise vector and the character timbre vector to obtain an intermediate state vector for the corresponding time step.

[0117] In one embodiment, the model training unit 504 is specifically used to: use the intermediate state vector and the character description vector as input to predict the target flow field direction at the corresponding time step through an initial timbre vector model; construct a loss function based on the mean square error between the predicted target flow field direction and the real flow field direction determined by the intermediate state vector and the character timbre vector; and perform backpropagation optimization on the parameters of the initial timbre vector model based on the loss function to obtain a pre-trained timbre vector generation model.

[0118] In one embodiment, the model training unit 504 is specifically used to: mask the character description vector with a preset probability, so that the initial timbre vector model can predict the target flow field direction of the intermediate state vector without the indication of the character description vector;

[0119] The model training unit 504 is specifically used to: construct a loss function based on the mean square error between the predicted target flow field direction and the actual flow field direction under two conditions: with and without character description vector indication, so that the initial timbre vector generation model learns the flow field mapping relationship under the condition with and without character description vector indication based on the loss function.

[0120] It should be noted that the training device for the timbre vector generation model provided in the above embodiments is only illustrated by the division of the above-described program modules. In practical applications, the above processing can be assigned to different program modules as needed, that is, the internal structure of the device can be divided into different program modules to complete all or part of the processing described above. Furthermore, the training device for the timbre vector generation model provided in the above embodiments and the training method embodiments for the timbre vector generation model belong to the same concept, and their specific implementation process is detailed in the method embodiments, which will not be repeated here.

[0121] To implement the audio data generation method of this application embodiment, this application embodiment also provides an audio data generation apparatus, such as... Figure 6As shown, the device includes:

[0122] The text vector generation unit 601 is used to generate a character description vector based on the input character description text;

[0123] The timbre vector generation unit 602 is used to input the character description vector into a pre-trained timbre vector generation model to obtain a target timbre vector corresponding to the character description text; wherein, the pre-trained timbre vector generation model is obtained by the training method of the timbre vector generation model described in the first aspect of the embodiments of this application or any specific implementation of the first aspect;

[0124] The speech synthesis unit 603 is used to input the target timbre vector into the speech synthesis model to generate audio data that matches the character description text.

[0125] In one embodiment, the timbre vector generation unit 602 is specifically used to: use a randomly generated noise vector as an initial state vector; and in the iteration process from the start time step to the end time step, perform the following steps for each time step: input the current time step, the current state vector, and the character description vector into the timbre vector generation model to predict the flow field direction of the current time step; update the state vector according to the flow field direction of the current time step, and use the updated state vector as the input of the next time step; and use the state vector output by the end time step as the target timbre vector.

[0126] In one embodiment, the timbre vector generation unit 602 is specifically used to: map the time step encoding to a first scaling factor and a first offset factor for the features of each layer of the neural network; use the first scaling factor and the first offset factor to perform channel-by-channel modulation on the features of the current layer to control the denoising intensity at different time steps; map the character description vector to a second scaling factor and a second offset factor for the normalization operation of each layer; use the second scaling factor and the second offset factor to perform an affine transformation on the normalized features of the layer to inject the semantic information of the character description vector into the normalized features of the layer.

[0127] In one embodiment, the timbre vector generation unit 602 is specifically configured to: receive a guiding scale coefficient set by the adjustment operation in response to an input to an interactive interface element; and generate the target timbre vector based on the character description vector and the guiding scale coefficient; wherein the guiding scale coefficient is used to control the degree of deviation between the target timbre vector and the timbre in the training dataset.

[0128] In one embodiment, the timbre vector generation unit 602 is specifically used for: inputting an empty vector into the timbre vector generation model to obtain a first vector result; inputting the character description vector and the randomly generated noise vector into the timbre vector generation model to obtain a second vector result; and weighting and fusing the first vector result and the second vector result based on the guiding scale coefficient to obtain the target timbre vector.

[0129] It should be noted that the audio data generation device provided in the above embodiments is only illustrated by the division of the above-described program modules. In practical applications, the above processing can be assigned to different program modules as needed, that is, the internal structure of the device can be divided into different program modules to complete all or part of the processing described above. In addition, the audio data generation device and the audio data generation method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process can be found in the method embodiments, which will not be repeated here.

[0130] Based on the hardware implementation of the above-mentioned program modules, and in order to implement the training method for the timbre vector generation model described in the first aspect or any specific implementation of the first aspect of the embodiments of this application, or the audio data generation method described in the second aspect or any specific implementation of the second aspect of the embodiments of this application, the embodiments of this application also provide a computer device, such as... Figure 7 As shown, the computer device 700 includes:

[0131] Processor 701, memory 702, and input / output interface 703;

[0132] The memory 702 is a short-term storage memory or a persistent storage memory;

[0133] The processor 701 is configured to communicate with the memory 702 and execute instructions in the memory 702 to perform the training method for the timbre vector generation model described in the first aspect or any specific implementation of the first aspect of the embodiments of this application, or the audio data generation method described in the second aspect or any specific implementation of the second aspect of the embodiments of this application.

[0134] Of course, in practical applications, the various components in the computer device 700 are coupled together through a bus system 704. It is understood that the bus system 704 is used to realize communication between these components. In addition to a data bus, the bus system 704 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in... Figure 7 The general designated all buses as Bus System 704.

[0135] The memory 702 in this embodiment is used to store various types of data to support the operation of the computer device 700. Examples of such data include any computer program used to operate on the computer device 700.

[0136] It is understood that when the processor in the computer device described above executes the computer program, it can also realize the functions of each unit in the corresponding device embodiments described above, which will not be repeated here. Exemplarily, the computer program can be divided into one or more modules / units, one or more modules / units are stored in memory and executed by the processor to complete the various embodiments of this application. One or more modules / units can be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program in the computer device. For example, the computer program can be divided into units in the aforementioned computer device, and each unit can implement the specific functions described in the corresponding computer device above.

[0137] Computer equipment can be desktop computers, laptops, handheld computers, and cloud servers, among other computing devices. Computer equipment may include, but is not limited to, processors and memory. Those skilled in the art will understand that processors and memory are merely examples of computer equipment and do not constitute a limitation on the computer equipment. It may include more or fewer components, or combinations of certain components, or different components. For example, computer equipment may also include input / output devices, network access devices, buses, etc.

[0138] A processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor. The processor is the control center of a computer device, connecting all parts of the computer device through various interfaces and lines.

[0139] Memory can be used to store computer programs and / or modules. The processor performs various functions of the computer device by running or executing the computer programs and / or modules stored in the memory, and by accessing data stored in the memory. Memory can primarily include a program storage area and a data storage area. The program storage area can store the operating system, at least one application program required for a function, etc.; the data storage area can store data created based on terminal usage, etc. Furthermore, memory can include high-speed random access memory, and can also include non-volatile memory, such as hard disks, RAM, plug-in hard disks, smart media cards (SMC), secure digital cards (SD), flash cards, at least one disk storage device, flash memory device, or other volatile solid-state storage devices.

[0140] This application also provides a computer-readable storage medium storing a computer program thereon. When the computer program is executed by a processor, it performs the training method for the timbre vector generation model described in the first aspect or any specific implementation of the first aspect of the embodiments of this application, or the audio data generation method described in the second aspect or any specific implementation of the second aspect of the embodiments of this application.

[0141] This application also provides a computer program product storing a computer program / instruction, which, when executed by a processor, is used to implement the training method of the timbre vector generation model described in the first aspect or any specific implementation of the first aspect of this application, or the audio data generation method described in the second aspect or any specific implementation of the second aspect of this application.

[0142] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0143] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between apparatuses or units through some interfaces, and may be electrical, mechanical, or other forms.

[0144] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0145] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0146] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A training method for a timbre vector generation model, characterized in that, include: Obtain a training dataset; wherein the training dataset includes character timbre vectors and character description text corresponding to the character timbre vectors; Generate a character description vector based on the character description text; According to the preset time step, add the corresponding random noise vector to the character timbre vector to obtain the intermediate state vector of the corresponding time step; Using the intermediate state vector and the character description vector as input, and predicting the flow direction from the intermediate state vector to the character timbre vector as the learning objective, an initial timbre vector model is trained to obtain a pre-trained timbre vector generation model; the flow direction is, under the corresponding time step condition, a direction vector representing the direction of the intermediate state vector along the time step evolution direction to the character timbre vector; The pre-trained timbre vector generation model is used to generate corresponding character timbre vectors based on the input character description text, and the character timbre vectors are used to generate audio data that matches the character description text.

2. The training method for the timbre vector generation model according to claim 1, characterized in that, The step of adding a corresponding random noise vector to the character's timbre vector according to a preset time step to obtain an intermediate state vector for the corresponding time step includes: Randomly sample time steps within a preset time interval; According to the time step, using the random noise vector as the initial data and the character timbre vector as the target data, linear interpolation is performed between the random noise vector and the character timbre vector to obtain the intermediate state vector of the corresponding time step.

3. The training method for the timbre vector generation model according to claim 1, characterized in that, The process of using the intermediate state vector and the character description vector as input, and predicting the flow direction from the intermediate state vector to the character timbre vector as the learning objective, trains an initial timbre vector model to obtain a pre-trained timbre vector generation model, including: Using the intermediate state vector and the character description vector as input, the target flow field direction at the corresponding time step is predicted through the initial timbre vector model; A loss function is constructed based on the mean square error between the predicted target flow field direction and the actual flow field direction determined by the intermediate state vector and the character timbre vector. Based on the loss function, the parameters of the initial timbre vector model are optimized by backpropagation to obtain a pre-trained timbre vector generation model.

4. The training method for the timbre vector generation model according to claim 3, characterized in that, Before constructing the loss function based on the mean square error between the predicted target flow field direction and the actual flow field direction determined by the intermediate state vector and the character timbre vector, the method further includes: The character description vector is masked with a preset probability, so that the initial timbre vector model can predict the target flow direction of the intermediate state vector without the indication of the character description vector. The loss function is constructed based on the mean square error between the predicted target flow field direction and the actual flow field direction determined by the intermediate state vector and the character timbre vector, including: A loss function is constructed based on the mean square error between the predicted target flow field direction and the actual flow field direction under two conditions: with and without character description vector indication. This allows the initial timbre vector generation model to learn the flow field mapping relationship under both conditions with and without character description vector indication.

5. An audio data generation method, characterized in that, include: Generate a character description vector based on the input character description text; The character description vector is input into a pre-trained timbre vector generation model to obtain a target timbre vector corresponding to the character description text; wherein the pre-trained timbre vector generation model is obtained by the method according to any one of claims 1-4; The target timbre vector is input into the speech synthesis model to generate audio data that matches the character description text.

6. The audio data generation method according to claim 5, characterized in that, The step of inputting the character description vector into a pre-trained timbre vector generation model to obtain a target timbre vector corresponding to the character description text includes: Use the randomly generated noise vector as the initial state vector; During the iteration from the start time step to the end time step, the following steps are performed for each time step: The current time step, the current state vector, and the character description vector are input into the timbre vector generation model to predict the flow field direction at the current time step. The state vector is updated according to the flow field direction at the current time step, and the updated state vector is used as the input for the next time step. The state vector output at the termination time step is used as the target timbre vector.

7. The audio data generation method according to claim 5, characterized in that, The step of inputting the character description vector into a pre-trained timbre vector generation model to obtain a target timbre vector corresponding to the character description text includes: The time step encoding is mapped to the first scaling factor and the first offset factor of each layer of neural network features. The first scaling factor and the first offset factor are used to modulate the current layer features channel by channel to control the denoising intensity at different time steps. The character description vector is mapped to the second scaling factor and the second offset factor of each layer normalization operation. The second scaling factor and the second offset factor are used to perform an affine transformation on the layer normalized features to inject the semantic information of the character description vector into the layer normalized features.

8. The audio data generation method according to claim 5, characterized in that, The step of inputting the character description vector into a pre-trained timbre vector generation model to obtain a target timbre vector corresponding to the character description text includes: In response to an adjustment operation input to an interactive interface element, the system receives a guide scale coefficient set by the adjustment operation. The target timbre vector is generated based on the character description vector and the guiding scale coefficient; The guiding scaling coefficient is used to control the degree of deviation between the target timbre vector and the timbre in the training dataset.

9. The audio data generation method according to claim 8, characterized in that, The step of generating the target timbre vector based on the character description vector and the guiding scale coefficient includes: The empty vector is input into the timbre vector generation model to obtain the first vector result; The character description vector and the randomly generated noise vector are input into the timbre vector generation model to obtain the second vector result; Based on the guiding scale coefficient, the first vector result and the second vector result are weighted and fused to obtain the target timbre vector.

10. A computer device, characterized in that, include: Processor, memory, and input / output interfaces; The memory is either a short-term storage memory or a persistent storage memory; The processor is configured to communicate with the memory and execute instructions in the memory to perform the training method of the timbre vector generation model as described in any one of claims 1 to 4 or the audio data generation method as described in any one of claims 5 to 9.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it is used to implement the training method of the timbre vector generation model as described in any one of claims 1 to 4 or the audio data generation method as described in any one of claims 5 to 9.

12. A computer program product having a computer program / instructions stored thereon, characterized in that, When executed by a processor, the computer program / instruction is used to implement the training method of the timbre vector generation model as described in any one of claims 1 to 4 or the audio data generation method as described in any one of claims 5 to 9.

Images