A method, system and apparatus for dubbing a novel content

Abstract

The application is suitable for the technical field of language synthesis, and provides a novel content dubbing method, which comprises the following steps: obtaining a novel text to be dubbed, preprocessing the novel text, identifying role dialogue sentences, environment description sentences and mixed sentences; extracting dialogue content and performing language emotion analysis to obtain language emotion labels; assigning corresponding general timbre embedding vectors to different roles, tracking state changes of the roles in plot development, and generating timbre gradient control parameters; based on the general timbre embedding vectors, the timbre gradient control parameters and the language emotion labels, performing voice synthesis on each dialogue content to generate dialogue voice waveforms; extracting environment semantic labels and intensity parameters from the environment description sentences to generate environment sound effect signals; mixing the dialogue voice waveforms and the environment sound effect signals to generate a dubbing audio file, so that the emotional expression of the text context and the smooth evolution of the same role in different age stages can be taken into account.

Description

Technical Field

[0001] This application belongs to the field of language synthesis technology, and in particular relates to a method, system and device for dubbing novel content. Background Technology

[0002] With the rapid growth of the audiobook market, more and more literary platforms and content creators are looking to efficiently convert massive amounts of novel texts into audio narration to meet users' listening needs in fragmented scenarios. Automated dubbing solutions based on text-to-speech (TTS) technology have become the mainstream technological path to replace traditional manual recording due to their significant advantages of high efficiency and low cost.

[0003] Existing dubbing systems typically employ a basic architecture of front-end text analysis and back-end speech synthesis. Natural language processing (NLP) is used to segment the novel into sentences, identify characters, and classify emotions, before a neural network acoustic model and vocoder generate the final speech waveform. However, when applied to the complex scenario of dubbing long novels, their emotion recognition often remains at a literal level, lacking a deep understanding of the context and pragmatic reasoning ability. They fail to grasp the deeper pragmatic meanings common in novels, such as irony, metaphor, and subtext, resulting in monotonous and mechanical emotional expression in the generated speech. Furthermore, the voice of the same character cannot smoothly evolve with age and emotional changes, leading to a disconnect between different listening experiences.

[0004] Therefore, there is an urgent need for a method of dubbing novel content that can take into account the emotional expression of the text context and the smooth evolution of the same character at different ages. Summary of the Invention

[0005] This application provides a method, system, and device for dubbing novel content, which can solve one of the problems of the prior art mentioned above.

[0006] In a first aspect, embodiments of this application provide a method for dubbing novel content, including: The novel text to be dubbed is obtained, and the novel text is preprocessed to identify character dialogue, environmental description, and mixed sentences. The dialogue content is extracted from the character dialogue statements and the mixed statements, and the language sentiment analysis is performed on the character dialogue statements and the mixed statements to obtain language sentiment tags; Assign corresponding universal timbre embedding vectors to different characters, track the changes in the characters' states as the plot develops, and generate timbre gradation control parameters; Based on a general timbre embedding vector, timbre gradation control parameters, and language emotion tags, speech synthesis is performed on the dialogue content to generate dialogue speech waveforms. Environmental semantic tags and intensity parameters are extracted from the environmental description statements, and environmental sound effect signals are generated based on the environmental semantic tags and intensity parameters; The dialogue voice waveform and the ambient sound effect signal are mixed to generate a dubbing audio file.

[0007] Furthermore, the process of acquiring the novel text to be dubbed, preprocessing the novel text, and identifying character dialogue, environmental descriptions, and mixed dialogue includes: The novel text is segmented into sentences, paragraphs, and sorted, and each sentence is assigned a unique sentence identifier. Based on the joint judgment of guide words, quotation mark features and speaker named entities, the probability of each sentence belonging to a role dialogue statement is determined. Based on context classifier recognition, determine the probability that each sentence belongs to the context description statement; Determine whether the probability of the character dialogue exceeds a preset first threshold and whether there is at least one auxiliary feature in the character dialogue statement recognition process. If so, determine that the corresponding sentence is a character dialogue statement. Determine whether the probability of the environmental description exceeds a preset second threshold and meets the condition of no dialogue. If so, determine that the corresponding sentence is an environmental description statement. If the probability of the environmental description exceeds the second threshold, but does not meet the condition of no dialogue, then the corresponding sentence is determined to be a mixed sentence.

[0008] Furthermore, the step of extracting dialogue content from the character dialogue statements and the mixed statements, and performing language sentiment analysis on the character dialogue statements and the mixed statements to obtain language sentiment tags, includes: Based on the dialogue identifier, the dialogue content is extracted from the character dialogue statements and the mixed statements, and the name of the speaking character is determined; Based on the dialogue content extracted from the character dialogue statements and the mixed statements, and combined with the context, pragmatic phenomena are detected using a pre-trained pragmatic inference model to generate language sentiment tags. The dialogue content, the language sentiment tags, and the character tags are encapsulated into character dialogue attributes.

[0009] Furthermore, the pragmatic reasoning model includes a literal sentiment analysis branch, a semantic intent reasoning branch, and a sentiment encoding fusion branch; wherein, The literal sentiment analysis branch takes the current dialogue content as input and is used to predict the basic sentiment category expressed by the current dialogue content at the literal word level, and obtain the literal sentiment probability distribution. The semantic intent reasoning branch uses the current dialogue content and its context window as joint input to identify pragmatic phenomena and output the intent category probability distribution. The emotion encoding fusion branch is used for literal emotion and pragmatic meaning. Figure 1 The system performs consistency judgments, and when the judgments are inconsistent, it generates composite sentiment codes and uses these composite sentiment codes as language sentiment labels. When the judgments are consistent, the language sentiment labels are determined based on the output of the literal sentiment branch.

[0010] Furthermore, the step of tracking the character's state changes during plot development and generating timbre gradation control parameters includes: Detect descriptive fragments that characterize changes in a role's state, including but not limited to aging, emotional accumulation, and physical condition; Based on the described fragment, timbre gradation control parameters are generated, including timbre feature offset and interpolation rate; The timbre gradation control parameters are used to progressively adjust the general timbre embedding vector.

[0011] Furthermore, the step of synthesizing speech based on the universal timbre embedding vector, timbre gradation control parameters, and language emotion tags to generate dialogue speech waveforms includes: Convert the dialogue content into a sequence of dialogue phonemes; For different roles, obtain their corresponding general timbre embedding vector and timbre gradation control parameters, combine them with language emotion tags as dialogue condition features, input them into the target acoustic decoder, and generate dialogue acoustic features. The acoustic features of the dialogue are input into a general neural vocoder to generate a dialogue speech waveform.

[0012] Furthermore, the target acoustic decoder includes a timbre conditional injection module and an emotion conditional layer; wherein, The timbre conditional injection module is equipped with a conditional mapping network. The network parameters of the conditional mapping network are adjusted by the timbre gradation control parameters. The conditional mapping network is used to map the general timbre embedding vector into a timbre feature map. The emotional condition layer is used to perform emotional condition modulation on the timbre feature map based on the language emotional tags.

[0013] Furthermore, the step of extracting environmental semantic tags and intensity parameters from the environmental description statements, and generating environmental sound effect signals based on the environmental semantic tags and intensity parameters, includes: Identify the core environmental words in the environmental description statements, map the core environmental words to environmental semantic tags, parse the modifiers of the core environmental words, and convert them into numerical intensity parameters; The environmental semantic tags are converted into content condition vectors, and the intensity parameters are converted into style condition vectors. Based on the content condition vector and the style condition vector, and combined with a pre-trained conditional neural sound effect generation model, an environmental sound effect signal is generated.

[0014] Secondly, embodiments of this application provide a novel content dubbing system, including: The first processing module is used to acquire the novel text to be dubbed, preprocess the novel text, and identify character dialogue, environmental description, and mixed sentences. The second processing module is used to extract dialogue content from the character dialogue statements and the mixed statements, and to perform language sentiment analysis on the character dialogue statements and the mixed statements to obtain language sentiment tags. The third processing module is used to assign corresponding general timbre embedding vectors to different characters, track the changes in the characters' states during the plot development, and generate timbre gradation control parameters. The fourth processing module is used to synthesize speech based on the general timbre embedding vector, timbre gradation control parameters, and language emotion tags, and generate dialogue speech waveforms. The fifth processing module is used to extract environmental semantic tags and intensity parameters from the environmental description statements, and generate environmental sound effect signals based on the environmental semantic tags and intensity parameters. The sixth processing module is used to mix the dialogue voice waveform and the environmental sound effect signal to generate a dubbing audio file.

[0015] Thirdly, embodiments of this application provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the above-described novel dubbing method.

[0016] The beneficial effects of the embodiments in this application compared with the prior art are: This application discloses a method for dubbing novel content. It extracts character dialogues, mixed sentences, and environmental descriptions from the novel text. It extracts dialogue content from the dialogues and mixed sentences to construct character dialogue attributes. Different timbres are used for dubbing different characters, enhancing the richness of the dubbing. Simultaneously, it introduces a multi-level sentiment analysis method based on contextual pragmatic reasoning to output linguistic sentiment tags for the dialogue content. This allows for the understanding and synthesis of complex and realistic human emotions such as irony and wry smiles, greatly enhancing the emotional impact of the voice. Furthermore, through full lifecycle tracking and gradual adjustment, the changes in the character's voice are made consistent with the plot development, appearing natural and seamless. This allows for the consideration of the emotional expression within the text context and the smooth evolution of the same character at different ages. Additionally, through environmental descriptions, the generated sound effects continuously change with the text descriptions, significantly enhancing the realism of the environmental sounds. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart illustrating a method for dubbing novel content according to an embodiment of the present invention; Figure 2 This is a schematic diagram of the structure of a novel dubbing system provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the structure of a computer device provided in an embodiment of the present invention. Detailed Implementation

[0019] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0020] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0021] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0022] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0023] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0024] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0025] Please see Figure 1 As shown, this invention is a method for dubbing novel content, comprising the following steps: S100. Obtain the novel text to be dubbed, preprocess the novel text, and identify character dialogue statements, environmental description statements, and mixed statements; In some embodiments, step S100 above includes: The novel text is segmented into sentences, paragraphs, and sorted, and each sentence is assigned a unique sentence identifier. Based on the joint judgment of guide words, quotation mark features and speaker named entities, the probability of each sentence belonging to a role dialogue statement is determined. Based on context classifier recognition, determine the probability that each sentence belongs to the context description statement; Determine whether the probability of the character dialogue exceeds a preset first threshold and whether there is at least one auxiliary feature in the character dialogue statement recognition process. If so, determine that the corresponding sentence is a character dialogue statement. Determine whether the probability of the environmental description exceeds a preset second threshold and meets the condition of no dialogue. If so, determine that the corresponding sentence is an environmental description statement. If the probability of the environmental description exceeds the second threshold, but does not meet the condition of no dialogue, then the corresponding sentence is determined to be a mixed sentence.

[0026] In this embodiment, unstructured novel text is parsed into structured attribute information. Specifically, the novel text is preprocessed, including removing irrelevant format characters, dividing it into sentences and paragraphs based on punctuation and line breaks, and assigning a globally unique sentence identifier to each sentence.

[0027] Furthermore, during sentence segmentation, when multiple statements exist within the same quotation mark, the quotation mark marker is retained before sentence segmentation. This ensures that dialogue spans multiple sentences, and multiple dialogue statements belonging to the same quotation mark paragraph can be merged into a complete dialogue content, ensuring the accuracy of sentiment analysis and speech synthesis. For example, when Yunmiao looks up and says, "Don't you remember me? I'm Miaomiao... How could you...", the quotation mark marker is retained during segmentation, and the sentence is divided into one sentence. In some embodiments, a quotation mark tracker is used to track the start and end positions of the quotation marks and segment the sentence accordingly.

[0028] After segmenting the novel text into sentences and paragraphs, a sequential sorting operation is further performed on the resulting sentences to establish a globally unified timeline benchmark. This ensures that the output order of each audio segment in the subsequent dubbing process strictly matches the original narrative order. Therefore, sentence identifiers are assigned based on the position of each sentence in the original novel text. These identifiers can be in the form of incremental integer numbers, starting from the beginning of the text and increasing sequentially. For text structure boundaries caused by chapters and sections, structural hierarchy information is preserved in the sentence identifiers, but the global order still increases linearly according to the original reading order. Furthermore, while assigning sentence identifiers, an ordered sentence sequence list is constructed. This list uses the sentence identifier as an index key and stores the original text content, paragraph number, and chapter number corresponding to each sentence. This ordered sentence sequence list serves as the timeline for the entire novel dubbing method, running through all subsequent steps and ensuring temporal consistency throughout the entire process from text parsing to dubbing audio output.

[0029] In some embodiments, a sequence classifier based on a pre-trained language model performs functional classification on each sentence, determining whether it belongs to a character dialogue statement or an environment description statement. For example, a finely tuned deep bidirectional pre-trained language model is used as the backbone network of the sequence classifier. This backbone network takes each sentence in the novel text as an input unit, extracts the contextual semantic representation of the sentence through a multi-layer Transformer encoder, and encapsulates the character dialogue probability determination process and the environment description probability determination process of each sentence into two linear classification heads, which are attached to the top layer of the backbone network to output the probability distribution of whether the sentence belongs to a character dialogue statement or an environment description statement, that is, output the character dialogue probability and the environment description probability.

[0030] Specifically, the sequence classifier includes an input representation layer, a context encoding layer, a multi-task feature fusion layer, and a classification label output layer. The input representation layer transforms the original text sentence into a numerical representation that the model can process. This involves segmenting the text sentence into a sequence of sub-word units and adding special start and end markers. Each sub-word unit is mapped to three embedding vectors: word embedding, position embedding, and paragraph embedding, representing the semantic information of the sub-word, its position in the sequence, and its source that distinguishes the current sentence from the surrounding sentences. The context encoding layer is composed of multiple stacked Transformer encoders, each layer containing a multi-head self-attention mechanism and a feedforward neural network. Through pre-training on large-scale corpora for tasks such as masked language models and next-sentence prediction, it achieves powerful contextual semantic modeling capabilities. The embedding sequence generated by the input representation layer is encoded layer by layer by the pre-trained language model. In each layer, the representation of each position is fused with information from all other positions in the sequence through an attention mechanism. Finally, the top layer outputs a deep feature vector sequence containing rich contextual semantics. The feature vector corresponding to the start marker in the model output is taken as the aggregated semantic representation of the entire input sentence, denoted as the sentence embedding vector. The multi-task feature fusion layer is used to extract differentiated discriminative features for different classification targets. Specifically, it includes two parallel feature transformation branches: a role dialogue feature branch and an environment description feature branch. The role dialogue feature branch consists of a fully connected layer and a non-linear activation function, which maps the sentence embedding vector to the role dialogue discriminative feature vector. During training, this branch receives additional input from auxiliary feature channels, including quotation mark presence markers, guide word matching types, and speaker entity detection results. These auxiliary features are encoded and concatenated with the sentence embedding vector, and then input into this branch to enable it to learn a representation sensitive to dialogue-related patterns. The environment description feature branch consists of another fully connected layer with the same structure but independent parameters and a non-linear activation function, which maps the sentence embedding vector to the environment description discriminative feature vector. During training, this branch introduces auxiliary learning objectives targeting the characteristics of environment description, such as descriptive text features, no-dialogue constraints, and contextual coherence, making it more sensitive to the atmosphere and descriptive language patterns unique to environment description.Finally, the classification label output layer performs independent binary classification decisions on the two sets of discriminative feature vectors output by the multi-task feature fusion layer. This classification label output layer receives the character dialogue discriminative feature vector through the character dialogue classification head, reduces its dimensionality through a fully connected layer, and outputs a scalar value between 0 and 1 through a sigmoid activation function, representing the probability that the current sentence belongs to the "character dialogue statement". Similarly, the environment description classification head receives the environment description discriminative feature vector, reduces its dimensionality through another fully connected layer, and outputs a scalar value between 0 and 1 through a sigmoid activation function, representing the probability that the current sentence belongs to the "environment description statement". Understandably, the two classification heads are independent of each other, each using an independent loss function for optimization, without mutual constraints; therefore, the probability values ​​output by the two are independent of each other.

[0031] Furthermore, a large-scale novel corpus was used as training data for the sequence classifier. Each training sample contained a complete sentence and its corresponding functional label, which included two core categories: "character dialogue" and "environmental description." To further improve classification accuracy, the training data also included location markers for auxiliary information related to character dialogue, such as guide words, text within quotation marks, and speaker entities, as well as refined annotations for descriptive texts describing the environment, such as pure scene descriptions, atmosphere rendering, descriptions of natural phenomena, and spatial layout descriptions. This allowed the model to implicitly learn the association between these structured cues and functional categories during training.

[0032] Furthermore, a multi-path feature analysis combining introductory words, quotation mark features, and speaker named entity features is performed to generate auxiliary feature mappings that are mapped into the character dialogue discrimination feature vector to determine the probability of character dialogue. Specifically, for quotation mark features, the system scans for paired quotation marks in the sentence, such as Chinese double quotation marks “”, single quotation marks '’, and English quotation marks "”. It's understandable that the presence of quotation marks is one of the strong features of character dialogue statements, but it's not a sufficient condition, as quotation marks may also appear in novel texts to emphasize or quote non-dialogue content. Therefore, the quotation mark detection result is used as the first dimension of the joint judgment feature, and its existence is encoded as a binary feature vector input to the auxiliary feature channel of the sequence classifier. For introductory words, a pattern matching method is used to detect whether the current sentence or its immediately preceding sentence contains typical verb patterns that guide the character's speech. Common typical verb patterns include “saying”, etc. Direct quotation markers such as “asked,” “shouted,” “whispered,” and “laughed,” as well as inner monologue markers such as “thought to oneself” and “muttered to oneself,” are used to maintain an expandable marker pattern library that covers common expressions from different languages ​​and literary styles. This allows for the acquisition of marker matching results for each sentence, which serve as the second feature. This second feature includes not only a Boolean value indicating whether a match has occurred but also the type of the matched marker, classifying it as either a direct quotation marker or an inner monologue marker for more precise classification. For speaker named entity recognition, the named entity recognition module detects the presence of person names in sentences. If the current sentence is text within quotation marks, the system backtracks to the preceding narrative paragraph to check for named entities that could represent speakers. Notably, the presence of speaker entities and their relative position to the text within quotation marks constitute the third feature for joint judgment. For example, when a structure is detected where “Zhang San said:” is followed by text within quotation marks, the credibility of the text within quotation marks as dialogue is significantly increased. Therefore, the above three-dimensional features and sentence embedding vectors are encoded and input into the character dialogue classification head. Finally, the probability that the sentence belongs to the "character dialogue statement" category is output. This probability is then combined with the probability that the sentence belongs to the environmental description statement to determine whether the sentence is a character dialogue statement.

[0033] Furthermore, while the classification system for identifying environmental descriptions complements that for character dialogues, the boundaries between environmental descriptions, action narrations, and psychological descriptions in novels are often unclear, making it difficult for rule-based classification methods to accurately distinguish them. Therefore, a context classifier is used to identify the input sentences. It is worth noting that this context classifier, along with the aforementioned sequence classifier, extracts descriptive text features, eliminates dialogue constraints, and considers contextual coherence as auxiliary learning objectives, providing contextual awareness beyond single-sentence boundaries for the identification of environmental descriptions. Specifically, for descriptive text features, during the training data annotation stage, text fragments such as pure scene descriptions, atmosphere renderings, descriptions of natural phenomena, and spatial layout descriptions are finely annotated. During training, the context classifier not only learns binary classification labels but also learns and predicts the descriptive degree score of sentences simultaneously through a multi-task learning framework. This descriptive degree score reflects the proportion of purely descriptive components in the sentence. There are no character dialogue constraints, meaning that the context classifier introduces negative sample constraints during training. Specifically, it learns the rule that environmental description sentences should not contain character dialogue content. Specifically, when a sentence contains both quotation mark features or speaker entities, this loss term penalizes the parameter updates of the sentence representation in the direction of environmental description, thus achieving mutual exclusion judgment with the classification of character dialogue sentences. Contextual coherence is crucial, as environmental descriptions in novels are typically presented consecutively at the paragraph level. Therefore, when determining the category of the current sentence, the context classifier considers not only the sentence's own features but also the classification results of adjacent sentences as contextual condition features. This enhances the classification confidence of the current sentence when several adjacent sentences are initially classified as environmental descriptions, thereby reducing classification jumps caused by a few ambiguous sentences. The introduction of the context classifier allows the sequence classifier to gain paragraph-level context awareness while maintaining sentence-level classification granularity, effectively solving the problem of blurred boundaries between environmental descriptions and action narrations, and significantly improving the accuracy and coherence of classification.

[0034] More specifically, the sequence classifier obtained through the above training, during the inference phase, outputs the probability of each sentence belonging to both "character dialogue statements" and "environmental description statements." Then, a strategy combining threshold determination and mutual exclusion rules is used to finally determine the sentence category. Specifically, it checks whether the character dialogue probability of the sentence exceeds a preset first threshold. This first threshold is fine-tuned on the validation set and is typically set between 0.6 and 0.75 to achieve a balance between precision and recall. When the character dialogue probability exceeds this first threshold, it further checks whether at least one auxiliary feature exists in the character dialogue statement recognition process, namely, quotation mark presence markers, guide word matching types, and speaker entity detection results. If these exist, the sentence is determined to be a character dialogue statement, preventing classification errors caused by misjudging certain special sentence structures, such as emphasized content with quotation marks or direct quotations that are not dialogues. If the character dialogue probability exceeds the first threshold but the aforementioned auxiliary features are not present, the sentence is marked as "pending determination." Furthermore, for sentences not identified as dialogue statements, the process proceeds to environmental description statement determination, specifically checking whether they exceed a preset second threshold. This second threshold can be the same as the first threshold or set independently based on category characteristics. When the environmental description probability exceeds the second threshold, it is also necessary to check and verify whether the current sentence meets the no-dialogue conditions, i.e., the sentence does not contain paired quotation marks, no matching of lead word patterns is detected, no named entities that can act as speakers are detected, and there is no obvious dialogue turn structure within the sentence's context window. If all of the above no-dialogue conditions are met, the current sentence is determined to be an environmental description statement. If any condition is not met, it indicates that the sentence may be an environmental description interspersed with brief dialogue or a dialogue scene containing environmental elements. The sentence is then marked as a "mixed statement," and its dialogue components and environmental components need to be extracted in subsequent processes to generate corresponding attributes. In addition, sentences with both the probability of character dialogue and the probability of environmental description being lower than their respective thresholds, as well as sentences marked as "pending" but which still cannot be clearly classified in subsequent processing, are uniformly marked as "other narrative statements." This category covers a large amount of narrative content in the novel text other than explicit dialogue and explicit environmental description, such as descriptions of character actions, descriptions of character psychological activities, pure narrative sentences that advance the plot, and descriptions of character appearance or state. For such texts, the default narration method is used for processing in subsequent processing.

[0035] S200. Extract dialogue content from the character dialogue statements and the mixed statements, and perform language sentiment analysis on the character dialogue statements and the mixed statements to obtain language sentiment tags. Specifically, a sentence marked as a "character dialogue statement" or "mixed statement" may contain descriptive content such as introductory phrases, action insertions, and scene descriptions in its original text, in addition to the dialogue content. Therefore, it is necessary to extract the pure dialogue content from it, determine the speaking character of the dialogue content, generate character dialogue attributes for it, and attach it to the corresponding sentence in the ordered sentence sequence list for subsequent dialogue dubbing.

[0036] In some embodiments, step S200 above includes: Based on the dialogue identifier, the dialogue content is extracted from the character dialogue statements and the mixed statements, and the name of the speaking character is determined; Based on the dialogue content extracted from the character dialogue statements and the mixed statements, and combined with the context, pragmatic phenomena are detected using a pre-trained pragmatic inference model to generate language sentiment tags. The dialogue content, the language sentiment tags, and the character tags are encapsulated into character dialogue attributes.

[0037] Specifically, for dialogue content identification, text within quotation marks or other dialogue markers is extracted as dialogue content. For example, some novels use dialogue without quotation marks, or dialogue distinguished by colons or specific formatting. Therefore, a dialogue boundary detection model based on syntactic analysis can identify the start and end points of the dialogue content. For speaking characters, a named entity recognition model is used to extract the name of the currently speaking character from the character's dialogue statements and / or mixed statements themselves, or from the paragraph or preceding narrative text. If pronouns are used in the text, a referencing resolution technique is used to link them to the most recently appearing explicit character name.

[0038] Furthermore, for the dialogue statements and mixed statements of the characters, it is necessary to further analyze their emotions in order to improve the emotional richness of the character voice acting. In this process, a multi-level sentiment analysis method based on contextual pragmatic reasoning is introduced to output language sentiment labels for the dialogue content.

[0039] In some embodiments, the pragmatic reasoning model includes a literal sentiment analysis branch, a semantic intent reasoning branch, and a sentiment encoding fusion branch; wherein, The literal sentiment analysis branch takes the current dialogue content as input and is used to predict the basic sentiment category expressed by the current dialogue content at the literal word level, and obtain the literal sentiment probability distribution. The semantic intent reasoning branch uses the current dialogue content and its context window as joint input to identify pragmatic phenomena and output the intent category probability distribution. The emotion encoding fusion branch is used for literal emotion and pragmatic meaning. Figure 1The system performs consistency judgments, and when the judgments are inconsistent, it generates composite sentiment codes and uses these composite sentiment codes as language sentiment labels. When the judgments are consistent, the language sentiment labels are determined based on the output of the literal sentiment branch.

[0040] Specifically, a pre-trained pragmatic inference model is used, taking the current dialogue content and the statements in the preceding and following context as input. Through deep semantic interaction modeling, it detects the presence of pragmatic phenomena such as irony, metaphor, pun, and forced smiles that are inconsistent between the literal meaning and the intended meaning. For example, when a character says "That's great!" after a business failure, the literal emotion is positive. However, by combining this with the negative context, a composite emotion code, such as "disappointment + self-deprecation," is generated as a linguistic emotion label, enabling the character to express the appropriate underlying tone.

[0041] In some embodiments, the pragmatic reasoning model adopts a multi-task learning architecture based on a pre-trained language model. Its backbone network is a deep bidirectional Transformer encoder pre-trained on a large-scale corpus, which can extract deep contextual semantic representations of the input dialogue content. On top of this backbone network, there are three functional branches in parallel: a literal sentiment analysis branch, a semantic intent reasoning branch, and a sentiment encoding fusion branch. The literal sentiment analysis branch adds a linear classification head to the sentence vector representation output by the backbone network to predict the basic sentiment category expressed by the current dialogue content at the literal lexical level. Specifically, this branch is pre-trained on a traditional sentiment classification dataset and can output literal sentiment category labels, such as "positive", "negative", and "neutral" and their corresponding confidence scores, providing a benchmark for subsequent pragmatic comparison, such as [positive: 0.92, negative: 0.04, neutral: 0.04]. The pragmatic intention inference branch adds a cross-attention layer and a conditional random field inference module to the backbone network output. This branch uses the current dialogue content and its context window as joint input. The size of the context window can be dynamically adjusted according to the paragraph structure of the novel text. In a preferred embodiment, the context window defaults to three sentences before and after the statement corresponding to the current dialogue content. Through deep semantic interaction modeling, it infers the speaker's true communicative intention. This branch is trained on a specially labeled pragmatic inference dataset and can identify pragmatic phenomena including but not limited to those in Table 1 below. It ultimately outputs intention category labels and their confidence scores, such as [irony: 0.87, self-deprecation: 0.65, forced smile: 0.72]. The sentiment encoding fusion branch fuses the literal sentiment analysis results with the pragmatic intention inference results. When two literal sentiments and pragmatic intentions are combined, the literal sentiment analysis results are encoded together. Figure 1When the two are consistent, this branch directly outputs the literal sentiment category label with the highest confidence score in the literal sentiment probability distribution as the language sentiment label, and determines the sentiment intensity value based on its corresponding confidence score. When the two are inconsistent, this branch performs composite sentiment encoding to generate a composite sentiment embedding vector that represents deep real emotion. Specifically, it obtains the literal sentiment probability distribution output by the literal sentiment analysis branch and the intention category probability distribution output by the pragmatic intention reasoning branch. By calculating the cosine similarity or KL divergence of the two probability distributions, it measures the consistency score between the two. When the consistency score is lower than a preset threshold, it is determined that there is a pragmatic phenomenon, and composite sentiment encoding is generated.

[0042] The composite emotion coding includes dominant emotion coding, emotion modulation parameters, and literal emotion residual coding. Dominant emotion coding represents the core emotional tone that the dialogue content ultimately needs to express. It is selected from the intention category labels output by the pragmatic intention inference branch, choosing the one with the highest confidence score as the dominant emotion. Emotion modulation parameters are a set of continuous-valued parameters used to finely control the acoustic features of the synthesized speech, including four dimensions: activation, tension, warmth, and stability. These measure the intensity of emotional expression, the degree of tension or relaxation, the warmth or coldness of emotion, and the stability or fluctuation of emotion, respectively. Specifically, they are calculated from the confidence scores of each intention category label output by the pragmatic intention inference branch through a learnable mapping matrix, resulting in a rich and realistic sense of layering in the final generated speech's emotional expression. Literal emotion residual coding retains some feature information of the original literal emotion and is superimposed on the dominant emotion coding with low weight. This preserves a subtle auditory cue of "insincere" in the synthesized speech, allowing listeners to simultaneously perceive the tension between the character's surface words and their true inner feelings.

[0043] Table 1 Examples of Pragmatic Phenomena

[0044] In addition, a standard sentiment table is maintained for the literal sentiment category labels output by the literal sentiment analysis branch and the intention category labels output by the semantic intention reasoning branch. For each category label, there is a corresponding standard sentiment embedding vector and a corresponding acoustic modulation tendency. For example, the standard sentiment embedding vector of the label "joy" has an acoustic modulation tendency that indicates an increase in the fundamental frequency mean, speech rate and spectral centroid.

[0045] Therefore, the above dialogue content, along with the character names and language emotion tags, are encapsulated as character dialogue attributes. It is worth noting that each character name is mapped to a globally unique character identifier, which is used to associate a general timbre embedding vector later. During subsequent dubbing, the dialogue content is synthesized based on the character and language emotion tags. For introductory phrases and action insertions in character dialogue sentences and / or mixed sentences, the default narration method is used for processing.

[0046] S300: Assign corresponding universal timbre embedding vectors to different characters, track the changes in the characters' states during the plot development, and generate timbre gradation control parameters; In this embodiment, a unique character record is created for each character in the global character database. Each character record includes a globally unique character identifier and character metadata. Specifically, the character identifier is an identifier generated based on a combination of a random number, the first appearance position, and an instance number, ensuring that there are no identifier conflicts in dubbing tasks across different works and batches. The first appearance position is the chapter number and global sentence number when the character is first identified in the novel, and the instance number is the sequential number of the character in the global characters when the character is first identified in the novel. The character metadata consists of basic metadata extracted and recorded from the novel for the character, specifically including the character name, the first appearance position, the first appearance context, and a list of character aliases. The character name is the name used when the character first appears in the novel; the first appearance position is the chapter number and global sentence number when the character is first identified in the novel; the first appearance context is a summary of the contextual text when the character first appears, used for character attribute inference; the list of character aliases is initially empty and is gradually expanded during subsequent referential resolution.

[0047] Furthermore, a timbre prototype library covering a wide range of population characteristics is pre-constructed. This timbre prototype library is trained on a large-scale multi-speaker speech dataset to obtain universal timbre embedding vectors for different speakers. Specifically, multilingual speech samples containing thousands of different speakers are collected. Each speaker's speech sample covers several standard sentences in their native language. All samples are forward-inferred through a trained speaker encoder to extract representative voiceprint vectors for each speaker. The embedding vectors extracted from multiple samples of the same speaker are then average-pooled to obtain the speaker's representative voiceprint vector. This representative voiceprint vector is used as a universal timbre embedding vector and associated with the timbre prototype library as a prototype entry. In addition, each timbre prototype entry is associated with a set of standardized semantic tags for matching and retrieval during timbre allocation. These semantic tags include gender, age group, and timbre. Gender is labeled based on the speaker's identity information, including male, female, and neutral voices. Age group is labeled based on the speaker's age range, including teenager, young adult, middle-aged, and elderly. Timbre is based on the spectral centroid clustering of its corresponding general timbre embedding vector, including bright, moderate, and deep. Through these multi-dimensional semantic tags, flexible matching and retrieval based on role attributes can be performed during timbre allocation.

[0048] Furthermore, regarding character attributes, clues that can be used to infer character attributes are extracted from the contextual text summary when the character first appears in the novel. Specifically, gender is determined by gender-related terms such as "he," "she," "sir," and "madam"; age-related descriptions such as "teenager," "young adult," "middle-aged man," and "old woman" are extracted to determine the character's age group; and direct descriptions related to the character's voice, such as "deep voice," "clear voice," and "hoarse voice," are extracted to determine the voice tone tendency. These textual clues are then mapped to character attribute tags, for example: {Gender: Male, Age Group: Middle-aged, Voice Tone Tendency: Deep}. Then, based on the inferred character attribute tags, a matching search is performed in a voice tone prototype database. For example, prototype entries matching the gender tag are first selected; among the matching entries, the similarity score for the age group tag is calculated; if a voice tone tendency tag exists, the similarity matching score with the voice tone tag in the prototype entry is calculated; finally, the scores of each dimension are weighted and summed, and the prototype entry with the highest total score is selected as the matching entry for the character and associated with the character's character identifier.

[0049] In some embodiments, a universal timbre embedding vector can be assigned to a character before or after their appearance via an interactive interface. For example, a prototype entry can be selected directly from a timbre prototype library, or a reference speech sample can be uploaded. Then, preprocessing operations such as removing silence and noise segments, retaining valid speech, and normalizing volume are performed on the reference speech sample. Frame-level feature extraction and temporal pooling are then performed to finally output a fixed-dimensional universal timbre embedding vector, which represents the speaker's voiceprint features of the reference speech sample.

[0050] In some embodiments, the step of tracking character state changes during plot development and generating timbre gradation control parameters includes: Detect descriptive fragments that characterize changes in role status, including aging, cumulative emotional shifts, and changes in physical and mental state; Based on the described fragment, timbre gradation control parameters are generated, including timbre feature offset and interpolation rate; The timbre gradation control parameters are used to progressively adjust the general timbre embedding vector.

[0051] A novel typically spans a long period of time and space, and characters may undergo significant life changes, such as from childhood to old age, from health to decline, and from naivety to maturity. If a character's voice remains unchanged throughout the story, it will reduce the listener's understanding of the character's growth and weaken the immersion in the narrative. To address this, this application employs full-lifecycle voice management, continuously analyzing the plot text to detect changes in the character's state and generate voice gradation control parameters, driving the smooth evolution of the character's voice in subsequent episodes.

[0052] In this embodiment, textual markers representing the progression of time are detected from the novel text. Specifically, an extensible library of time span expression patterns is maintained, which covers common time jump expressions in various languages ​​and is organized according to semantic categories. These categories include at least explicit year periods, ambiguous years, life stages, seasonal changes, event anchors, and chapter divisions. Explicit year periods are patterns where numerical values ​​can be directly extracted as time spans, such as "three years later," "five years have passed," and "ten years later." Ambiguous years are time span patterns with no specific numerical value, expressed semantically, such as "many years have passed in a flash," "several years later," and "time flies." During the process, a default medium span is used, such as 5 years for years and 6 months for months; life stages are time span patterns for characters at different ages, such as "him in his twilight years," "after entering middle age," and "his childhood," which can be estimated using a stage-age mapping table; seasonal changes, such as "another winter has come" and "spring has gone and autumn has come," are counted as 0.5 to 1 year for each seasonal change; event anchors are time span patterns based on plot nodes, such as "after the child was born" and "the year the war ended," which can be estimated by combining the event timeline in the context; chapter separations are the time spans recorded in the chapter titles, and the time span value is determined by parsing the time information in the titles. Therefore, by scanning the novel text sentence by sentence, when any pattern in the time span expression pattern library is matched, the global sentence number of that time marker and the quantified time span value are recorded.

[0053] Furthermore, a detection window is formed by expanding outwards from the sentence position where each time marker is located. The size of the detection window is dynamically adjusted according to the magnitude of the time span. For example, for a clear large time jump, such as "ten years later", the window is expanded to 5 to 8 sentences before and after to capture sufficient character state descriptions; for a short time jump, such as "the next day", the window is shrunk to 2 to 3 sentences before and after to reduce interference from irrelevant text.

[0054] More specifically, within the detection window, the named entity recognition model identifies the names of the characters that appear, and dependency parsing technology is used to extract state modifiers related to these characters. State modifiers include categories of character state changes such as age increase, voice change, physical condition, mental state, and emotional accumulation.

[0055] Understandably, when a time stamp exists within the detection window and a status modifier related to the current character is present, it is determined that the character has undergone a status change. Therefore, based on the status modifier, timbre gradation control parameters are generated.

[0056] Specifically, state change labels are determined based on the extracted state modifiers. For example, state modifiers that represent aging, such as "old," "white-haired," "elderly," and "aged," can be identified as "old" through similarity calculation. Similarly, "sick" that represents physical condition can be identified as "weak" through similarity calculation. For different state change labels, corresponding timbre feature offset parameters are set. These timbre feature offset parameters include acoustic feature offset parameters that represent timbre changes, such as fundamental frequency mean offset (Hz), fundamental frequency variance offset, speech rate offset (%), breathy voice ratio offset (%), spectral centroid offset (Hz), and volume offset (dB).

[0057] Specifically, upon detecting a change in a character's state, a timbre feature offset parameter is determined based on the state change label of the character's state modifier. This timbre feature offset parameter is then used to determine the timbre gradation control parameters, which consist of three parts: timbre feature offset, interpolation rate, and timbre change curve. The timbre feature offset parameter is defined as follows: if a character's state change may correspond to one or more categories of state modifiers such as age increase, voice change, physical condition, mental state, and emotional accumulation, one or more state change labels can be determined. For multiple state change labels, multiple timbre feature offset parameters need to be superimposed and fused to obtain the timbre feature offset.

[0058] The interpolation rate is used to determine the step size required for the timbre to transition from the current state to the target state. Specifically, the interpolation rate = 1.0 / max(N, 1), where N is the number of chapters between two adjacent time markers. This interpolation rate means that the timbre gradation should be evenly distributed across all chapters between two adjacent time markers, and in each chapter, each timbre parameter moves towards the target value by 1 / N of the total change. Taking the phrase "ten years later" appearing in Chapter 50 as an example, the interpolation rate = 1 / 50 = 0.02, meaning that each timbre parameter moves towards the target value by a step of 2% in each chapter. After accumulating through 50 chapters, by the time the time marker appears in the chapter, the timbre has just completed a full transition to the target state. If the same character triggers state change detection again in a subsequent chapter, for example, if "five years have passed, and he has become even older" is detected in Chapter 80, then the current partially transitioned timbre state is used as the new starting point, that is, the chapter interval between Chapter 50 and Chapter 80 is used as the new gradient interval. The interpolation rate of this interval is recalculated, thereby ensuring that the character's voice evolution trajectory is formed by connecting several smooth gradient segments end to end, ultimately forming a continuous, non-abrupt timbre change curve. In subsequent processes, the timbre gradient control parameters are used to perform timbre gradient adjustments on the corresponding character.

[0059] S400: performing speech synthesis on each of the dialogue content based on the general timbre embedding vector, the timbre gradient control parameter and the linguistic emotion label, to generate a dialogue speech waveform; In some embodiments, the above step S400 includes: converting each of the dialogue content into a dialogue phoneme sequence, acquiring the corresponding general timbre embedding vector and the timbre gradient control parameter for different roles, combining the linguistic emotion label as dialogue condition features, inputting the features into a target acoustic decoder, and generating dialogue acoustic features; inputting the dialogue acoustic features into a general neural vocoder to generate a dialogue speech waveform.

[0060] In this embodiment, the dialogue attribute of each role generated in step S200 is acquired, and the corresponding dialogue content is extracted therefrom. The dialogue content needs to be subjected to text processing to convert a natural language text into a standard phoneme sequence that can be directly consumed by the acoustic decoder, wherein the text processing includes text normalization, word segmentation, grapheme-to-phoneme conversion and prosodic annotation injection.

[0061] Specifically, text normalization is used to convert non-standard text symbols into a standard reading form. Specifically, novel texts usually include various types of text symbols, such as Arabic numerals, Roman numerals, punctuation marks, special symbols, and kaomoji appearing in network texts, which need to be unified into a standard reading form. For example, for Arabic numerals, the reading method is determined according to the context. For example, "2024年" is normalized to "二零二四年", and "第3章" is normalized to "第三章". For punctuation marks commonly found in novels, such as ellipsis, dash, tilde, etc., they are converted into corresponding prosodic tags according to their semantic functions, instead of being directly deleted. For example, the Chinese ellipsis "......" is converted into the prosodic tag [pause=long], and the dash "--" is converted into [pause=medium]. These prosodic tags will be parsed by the length regulator of the acoustic decoder to control the pause duration in the synthesized speech. For kaomoji or emoticons commonly found in online novels, such as , "QAQ", "(笑)", etc., they are converted into corresponding emotion expression phrases by maintaining an emoticon reading mapping table. For example, "QAQ" can be mapped to "呜呜呜", and the emoticon reading mapping table can be customized according to the novel type and user preferences.

[0062] Further, word segmentation segments the normalized continuous text into a sequence of lexical units with complete semantics. For example, a sequence labeling model is used for segmentation. The sequence labeling model is fine-tuned on a large amount of novel corpus, and is particularly optimized for common proper nouns in novels, such as personal names and place names, and can accurately identify the overall boundary of compound proper nouns such as "青云剑诀" and "九天玄女", so as to avoid pronunciation errors caused by incorrect segmentation.

[0063] Further, each word unit after word segmentation is mapped to its corresponding phoneme sequence through a G2P model. Specifically, the G2P model adopts a neural network-based polyphonic character disambiguation model. The model takes the entire sentence as input, captures the contextual context where the polyphonic character is located through a self-attention mechanism, and predicts its correct pronunciation. For example, the character "发" in "头发 (hair)" and "发" in "出发 (set out)" can be correctly distinguished by the model according to the context. For rare characters or variant characters in novels, the model uses a backoff mechanism to call a unified code table to ensure pronunciation output. While generating the phoneme sequence, corresponding prosodic annotation tags are embedded into the phoneme sequence according to language emotion tags. For example, for a dialogue with the "irony" tag, a [tone=sarcastic] tag is added after the trailing phoneme of the sentence. Thus, a structured dialogue phoneme sequence is output, including a phoneme ID sequence, a prosodic tag sequence, and an estimated sentence duration, wherein the phoneme ID sequence is an integer index of each phoneme in the phoneme set; the prosodic tag sequence is a prosodic control tag corresponding to each phoneme position; the estimated sentence duration is determined based on text length, specifically, estimated sentence duration = number of phonemes × basic phoneme duration × basic speech rate coefficient. This dialogue phoneme sequence is used as the text input of the target acoustic decoder, and together with the received general timbre embedding vector and emotion tag, it jointly drives the generation of dialogue acoustic features.

[0064] Further, the above dialogue phoneme sequence, the general timbre embedding vector of the corresponding character obtained through step S300, and the language emotion tag of the corresponding sentence in step S200 are input together into the target acoustic decoder to generate a Mel spectrum frame sequence containing the target timbre and rich emotions. The acoustic decoder of the present application, based on the traditional FastSpeech 2 architecture, introduces a standardized timbre condition interface and a multi-level emotion condition layer, realizing independent and joint control of speaker identity and emotion expression.

[0065] In some embodiments, the target acoustic decoder includes a timbre condition injection module and an emotion condition layer; wherein, The timbre condition injection module is provided with a condition mapping network, and the network parameters of the condition mapping network are adjusted through the timbre gradient control parameter, and the condition mapping network is configured to map the general timbre embedding vector to a timbre feature map; The emotion condition layer is configured to perform emotion condition modulation on the timbre feature map based on the language emotion tag.

[0066] Specifically, the target acoustic decoder adopts a non-autoregressive encoder-decoder structure. The encoder consists of multiple feedforward Transformer blocks, which receive the dialogue phoneme sequence and output the phoneme-level hidden state sequence. The length adjuster expands the phoneme-level hidden state into a frame-level hidden state sequence according to the prediction duration of each phoneme. The decoder also consists of multiple feedforward Transformer blocks, which decode the frame-level hidden state sequence and finally predict the Mel spectrum features of each frame through a linear projection layer.

[0067] More specifically, timbre condition injection modules and emotion condition layers are embedded between the various levels of the encoder and decoder, so that external conditions can participate in the generation process of acoustic features in a controllable manner.

[0068] For the timbre condition injection module, cross-attention guidance is implemented layer by layer in the decoder. Specifically, a cross-attention sub-layer is added to each feedforward Transformer block of the decoder. This sub-layer uses the general timbre embedding vector as a key-value pair and the hidden state of the current layer as a query. The cross-attention mechanism is used to calculate the modulation signal of the timbre condition on the current feature. The modulation signal is added back to the main feature path as a residual, thereby realizing the step-by-step injection of timbre information in each layer of the decoder. This allows the timbre condition to participate deeply in every stage of acoustic feature generation, resulting in more precise and stable control over the final timbre.

[0069] More specifically, a conditional mapping network is set up inside the timbre conditional injection module. This conditional mapping network is a shallow fully connected network located before the cross-attention sub-layer. It is used to map the general timbre embedding vector into modulation parameters. When a change in character state is detected and timbre gradation control parameters are generated, the weights and biases of the conditional mapping network are gradually updated through interpolation coefficients. This allows for a gradual adjustment of the timbre's control effect on acoustic features without changing the input timbre embedding vector. Specifically, the conditional mapping network includes a shared coding layer and multiple layer branch mapping networks. The shared coding layer converts the general timbre embedding vector into an intermediate timbre representation vector through two fully connected layers, which are then fed into multiple parallel layer branch mapping networks. Each layer branch mapping network corresponds to a feedforward Transformer block in the decoder. Specifically, one layer branch mapping network corresponds to one feedforward Transformer block in the decoder and generates feature modulation parameters {γ_i, β_i} for it. These feature modulation parameters modulate the timbre conditions through the cross-attention sub-layer, ultimately generating a timbre feature map. This timbre feature map is then further modulated emotionally through the emotional conditional layer.

[0070] In the process of generating timbre feature maps, for the timbre gradation control parameters generated by detecting changes in character state, before mapping the general timbre embedding vector to modulation parameters, the timbre feature offset must first be converted into specific adjustment amounts for the network parameters in the conditional mapping network. The adjusted conditional mapping network then maps the general timbre embedding vector. Specifically, an acoustic-parameter sensitivity matrix is ​​established, defining the mapping relationship between each neuron parameter in the conditional mapping network and each acoustic feature. This is obtained by applying a small perturbation Δw to each weight parameter of the conditional mapping network under standard synthesis conditions and measuring the change in output acoustic features caused by the perturbation. This is achieved through ΔW = argmin||J·Δw-Δtarget|| 2 +λ·||Δw|| 2 The adjustment amount ΔW of each parameter in the conditional mapping network is obtained by solving the equation, where Δtarget is a vector composed of the timbre feature offsets of each target acoustic feature, λ is a regularization coefficient used to constrain the magnitude of parameter adjustment to prevent excessive parameter changes from causing a decrease in synthesis quality, J is the acoustic-parameter sensitivity matrix, and Δw is the small perturbation applied to each weight parameter of the conditional mapping network during the establishment of the acoustic-parameter sensitivity matrix.

[0071] More specifically, before each new chapter begins synthesis, the relationship between the current chapter number k and the starting chapter number and the expected ending chapter number in the timbre gradation control parameters is checked. If k is within the gradation interval, the interpolation coefficient α_k corresponding to the current chapter is calculated, and the parameters of the conditional mapping network are updated accordingly. Specifically, α_k = min(k / N_current, 1.0), where N_current is the chapter number where the time stamp is located, and k is the current synthesis chapter number. Then, the parameter W_k used by the conditional mapping network in the synthesis of chapter k is the original parameter W_0 plus the parameter offset after scaling by the interpolation coefficient, W_k = W_0 + α_k·ΔW, where W_0 is the specific adjustment amount ΔW of each parameter in the conditional mapping network. When α_k = At time 0 (the character's first appearance), W_k = W_0, the conditional mapping network uses the original parameter W_0, and the timbre is in its initial state. When α_k = 1.0 (the chapter where the time marker is located), W_k = W_0 + ΔW, the parameter offset has been fully effective, and the timbre has fully reached the target state. In the middle chapters, the parameters transition linearly between W_0 and W_target. Since W_k only changes slightly between each chapter, the voice generated in each chapter shows a continuous and gradual evolution in timbre. Listeners cannot perceive the changes in a single chapter when listening continuously, but can clearly perceive the growth of the character's voice after crossing a large span of chapters.

[0072] Therefore, by applying the timbre gradation control parameters to the conditional mapping network of the target acoustic decoder, the modulation parameters of its output are adjusted by adjusting its parameter offset, thereby indirectly changing the control effect of the general timbre embedding vector on the acoustic feature generation process. During this process, the core voiceprint identifier of the character remains stable, only its "decoding method" is changed, which is closer to the natural evolution mechanism of human voice due to physiological changes.

[0073] For the sentiment conditional layer, it is embedded after each feedforward Transformer block and before the next Transformer block in the decoder. It receives language sentiment tags and performs sentiment conditional modulation on the timbre feature map X_spk of the current layer after timbre modulation by the conditional mapping network. The language sentiment tags are passed through a learnable sentiment mapping network to generate a sentiment scaling factor γ_emo and a sentiment offset β_emo corresponding to the number of channels. The sentiment-modulated feature map Y_emo is calculated by the following formula: Y_emo=γ_emo·X_spk+β_emo, where γ_emo and β_emo are predicted by the language sentiment tags through two independent fully connected layers. This allows the sentiment information to systematically change the overall prosodic tone of the synthesized speech by adjusting the global statistics of the features.

[0074] Furthermore, when literal emotion and pragmatic meaning... Figure 1 To achieve this, the emotion mapping network reads the acoustic modulation tendency from the standard emotion embedding vector corresponding to the language emotion label, and combines it with the emotion intensity value to generate the emotion scaling factor γ_emo and emotion offset β_emo for each decoder layer. Since the emotion structure of regular emotions is relatively simple, γ_emo and β_emo maintain a consistent activation mode across layers, only their numerical amplitude is modulated by the intensity parameter. For example, for the high-intensity "joy," based on its acoustic modulation tendency, the emotion conditional layer increases the scaling factor and offset of the fundamental frequency-related channels in all decoder layers, resulting in an overall upward distribution of the generated Mel spectrum in the fundamental frequency dimension; simultaneously, it moderately reduces the duration prediction value of the speech rate-related channels, making the synthesized speech exhibit a brisk rhythm. For "sadness," based on the acoustic modulation tendency, it globally reduces the mean fundamental frequency, increases fundamental frequency micro-vibrations between frames, and slows down the speech rate.

[0075] Furthermore, when the literal sentiment and pragmatic intent are inconsistent, the language sentiment label becomes a composite sentiment code. In this case, the sentiment mapping network of the sentiment conditional layer applies the three layers of information from the composite sentiment code to different levels of acoustic feature generation. Specifically, the dominant sentiment code is used to select the basic prosodic template. By maintaining a learnable prosodic template codebook in the sentiment conditional layer, each template corresponds to a statistical pattern of the fundamental frequency profile, energy envelope, and spectral tilt of a basic sentiment. When the dominant sentiment is "loss," a basic template with a low fundamental frequency mean, a falling sentence-end fundamental frequency profile, and low energy is selected; when the dominant sentiment is "self-deprecation," a composite fundamental frequency profile with a slightly rising ending is selected. The basic template serves as a prior bias for the decoder to predict acoustic features, ensuring the correct overall emotional tone of the synthesized speech. In composite emotion coding, the four dimensions of activation, tension, warmth, and stability in the emotion modulation parameters are decoupled into independent control signals, each acting on different dimensions of the acoustic characteristics. Specifically, activation regulates the fundamental frequency variance and energy variance. When activation is high, the inter-frame variation of fundamental frequency and energy increases, making the sound more dynamic and expressive; when activation is low, the sound tends to be flat. Tension regulates speech rate and pause frequency. When tension is high, speech rate increases and pauses decrease, conveying tension or urgency; when tension is low, speech rate slows and pauses increase. Warmth regulates the spectral centroid and high-frequency attenuation. When warmth is high, spectral energy concentrates in the mid-low frequencies, making the sound softer and more intimate; when warmth is low, high-frequency components increase, making the sound cold and distant. Stability regulates the amplitude of fundamental frequency jitter and volume fluctuations. When stability is low, slight fundamental frequency jitter and volume fluctuations appear in the sound, conveying emotional instability; when stability is high, the sound is smooth and stable. For literal emotion residual encoding, it is superimposed on the dominant emotion modulation with low weight. Its mechanism is as follows: when generating γ and β parameters in the emotion condition layer, in addition to considering the dominant emotion and modulation parameters, a small offset driven by literal residual encoding is added. For example, for the emotion of "irony", the dominant emotion sets the fundamental frequency profile to a low and descending type, but the literal residual "positive" introduces a slightly rising tail tone with low weight, so that the listener perceives a subtle rise in the overall sarcastic tone of the speech, thus producing the feeling of "saying the opposite of what is meant".

[0076] Furthermore, after conditional modulation by each decoder layer, the final linear projection layer maps the frame-level hidden state output by the decoder into a Mel spectrum frame sequence. It is worth noting that the Mel spectrum is a logarithmic amplitude spectrum uniformly distributed on the Mel scale, typically using an 80-dimensional Mel band. Each frame corresponds to approximately 12.5 milliseconds of speech duration, with frame shifts and overlaps between frames. The output Mel spectrum sequence fully characterizes all acoustic features of the dialogue statement in the time-frequency domain, including fundamental harmonic structure, formant distribution, spectral envelope, and aperiodic components.

[0077] Furthermore, the Mel spectrum frame sequence will be used as a standardized output of the dialogue acoustic features, and through a universal neural vocoder, waveform reconstruction will generate a high-fidelity dialogue speech waveform.

[0078] It is worth noting that for environmental descriptions, other narrative statements, and descriptive content other than dialogue in character dialogues and mixed statements in the novel text, a default narration method is used. In one embodiment, the aforementioned environmental descriptions, other narrative statements, and descriptive content other than dialogue in character dialogues and mixed statements are processed using the same method as the dialogue content, converted into a narration phoneme sequence, and then input into a narration acoustic decoder to decode it into the corresponding narration acoustic features. A high-fidelity narration speech waveform is then generated through waveform reconstruction using a general neural vocoder.

[0079] S500: Extract environmental semantic tags and intensity parameters from the environmental description statements, and generate environmental sound effect signals based on the environmental semantic tags and intensity parameters; In some embodiments, step S500 above includes: Identify the core environmental words in the environmental description statements, map the core environmental words to environmental semantic tags, parse the modifiers of the core environmental words, and convert them into numerical intensity parameters; The environmental semantic tags are converted into content condition vectors, and the intensity parameters are converted into style condition vectors. Based on the content condition vector and the style condition vector, and combined with a pre-trained conditional neural sound effect generation model, an environmental sound effect signal is generated.

[0080] In this embodiment, a hierarchical environment ontology tree is predefined, which covers major categories such as weather, location, time, and atmosphere, and their subcategories. Through a sequence labeling model, core environmental words in environmental description sentences, such as "wind," "rain," and "bell," are identified and mapped to standardized environmental semantic tags, such as "weather.wind," "weather.rain," and "ambience.bell." Furthermore, adjectives, adverbs, or complements describing the core environmental words in the environmental description sentences are obtained, such as "howling," "pouring rain," and "vaguely." Through a preset intensity mapping table, each modifier is converted into a numerical intensity parameter, specifically a floating-point number between 0.0 and 1.0, where 0.0 represents weak and 1.0 represents very strong. For example, for the tag "weather.wind," "light breeze" is mapped to 0.2, "strong wind" to 0.7, and "gale" to 1.0.

[0081] Furthermore, the above information is encapsulated as environmental description attributes to generate environmental sound effects that dynamically match the text content, significantly enhancing the realism of environmental sounds.

[0082] Specifically, a conditional neural sound effect generation model is used to achieve dynamic, real-time, and controllable synthesis of environmental sound effects. In one embodiment, the conditional neural sound effect generation model adopts a diffusion model audio generation architecture, which consists of a forward diffusion process and a backward denoising process. During the inference phase, only the backward denoising process is executed to generate the target sound effect. Specifically, the forward diffusion process is executed only during the training phase. Given a real environmental sound effect waveform x_0, the forward process gradually adds Gaussian noise to it over T time steps, generating a series of noisy signals x_1, x_2, ..., x_T. The reverse denoising process is executed during the inference phase. Starting from a randomly sampled Gaussian noise x_T, it gradually removes noise to finally generate the target environmental sound effect waveform. The denoising network predicts the noise component to be removed in each step. The denoising network adopts an improved WaveNet or a U-Net-based temporal convolutional architecture. Its main body consists of multiple downsampling blocks and upsampling blocks with skip connections in between to retain fine-grained temporal information. Each convolutional block contains a dilated convolutional layer, a gated activation unit, and a conditional injection layer. The conditional injection layer is responsible for receiving environmental semantic labels and intensity parameters and integrating them into the feature map. The diffusion time step t is embedded into each convolutional block through sinusoidal position encoding, enabling the network to perceive the current denoising progress.

[0083] Furthermore, environmental semantic tags are mapped to content condition vectors through a tag embedding layer. Specifically, each category tag in the environmental ontology tree is mapped to a learnable embedding vector. During training, these embedding vectors are jointly optimized with the denoising network parameters. During inference, the corresponding embedding vector is retrieved based on the environmental semantic tags. For multiple environmental semantic tags extracted from the same environmental description sentence, the embedding vectors of multiple environmental semantic tags are weighted and fused. The weights are initially determined by the strength parameters corresponding to each tag. Specifically, the higher the strength of the tag, the greater the weight of its embedding vector in the fusion. The fused content condition vector comprehensively represents the composite features of multiple environmental elements in the current scene. Then, the content condition vector is injected into each convolutional block of the denoising network through a cross-attention mechanism. Inside the convolutional block, the content condition vector is converted into keys and values ​​matching the dimension of the feature map through a linear projection layer. The feature map of the current layer is used as a query, and the conditional modulation signal is calculated through cross-attention and superimposed onto the main feature path in a residual manner. This injection method enables the environmental semantic tags to continuously guide the category features of the sound effect at each stage of the generation process, ensuring that the synthesized sound effect conforms to the target category in terms of overall timbre and texture.

[0084] Furthermore, the intensity parameter is transformed into a style condition vector to regulate the fine-grained acoustic performance of the generated sound effects. This includes volume envelope control, spectral distribution adjustment, and particle density control. Specifically, dynamic changes in environmental descriptions, such as increasing intensity or decreasing intensity, are detected. When a dynamic change is detected, the intensity parameter is a single scalar value ranging from [0.0, 1.0]. This scalar value is mapped to a style condition vector through a learnable intensity embedding layer and injected into the denoising network to regulate the overall intensity level of the generated sound effects. When a dynamic change is detected, the intensity parameter is transformed into a time-varying control curve. Based on the sentence number of the environmental description in the text and the estimated duration of each sentence, the paragraph position of the sentence in the text is mapped to the start and end times on the timeline. For example, if the environmental description "the wind is getting stronger" is estimated to affect an environmental description paragraph for 15 seconds, then the duration of the control curve is 15 seconds. Secondly, the starting and ending intensity cues in the dynamic description are analyzed and mapped to numerical values. For example, the starting value for "a gentle breeze" is 0.15, and the ending value for "a howling wind" is 0.95. If only one end is cued, the other end uses a default boundary value. Finally, an interpolation function is selected based on the type of dynamic descriptive word. For example, "getting bigger" and "gradually stronger" use linear or exponentially increasing curves; "gradually subsiding" and "slowly calming down" use linear or logarithmically decreasing curves; "fluctuating" uses a sine wave modulation curve; and "suddenly rising" uses a step-like surge curve followed by a flat section. The volume intensity of the control curve is defined at each point on the time axis. Thus, a continuous control curve that perfectly matches the duration and dynamic changes of the text description is generated frame by frame on the time axis. This control curve is mapped to a style conditional vector and injected into each convolutional block of the denoising network through a cross-attention mechanism.

[0085] In each step, the denoising network receives the current noisy signal x_t, the diffusion time step t, the content condition vector, and the style condition vector that changes frame by frame, and predicts the noise to be removed in that step. After T iterations, the final continuous ambient sound effect waveform is obtained, with a total duration of T_total. The volume change along the time axis corresponds precisely to the control curve. This waveform serves as the ambient sound effect signal and is mixed with the dialogue speech waveform and its narration speech waveform to generate a dubbing audio file.

[0086] S600: Mix the dialogue voice waveform and the ambient sound effect signal to generate a dubbing audio file.

[0087] In this embodiment, the generated dialogue voice waveform, narration voice waveform, and environmental sound effect signal, among other independent audio tracks, are merged into a coherent, balanced, and professionally sounding final dubbing audio file according to auditory aesthetic principles and technical specifications.

[0088] Specifically, the dialogue and narration waveforms are used as the main audio tracks, and the ambient sound effect signal is used as the background audio track for mixing and generation. The dialogue and narration waveforms are strictly aligned on the time axis according to the global sentence identifiers of the corresponding sentences to ensure the synchronization accuracy between different audio tracks. As for the ambient sound effect signal, it contains a control curve that changes over time in the corresponding environment description statement to describe the intensity fluctuations. It is aligned according to the start and end time points on the time axis to generate the dubbing audio file.

[0089] In some embodiments, the overlapping area of ​​the dialogue voice waveform, the narration voice waveform and the ambient sound effect signal on the time axis is detected. If a dialogue voice waveform exists in the overlapping area, the gain of the background audio track is smoothly reduced by 2 to 3 dB in a logarithmic curve within 0.1 seconds before the overlap begins, and restored to the original gain level within 0.1 seconds after the overlap ends. This makes the ambient sound effect automatically give way when the dialogue occurs, and the atmosphere naturally recovers when the dialogue ends, with a smooth and uninterrupted experience throughout.

[0090] Please see Figure 2 As shown, the present invention also provides a novel content dubbing system, the system comprising: First processing module 201: used to acquire the novel text to be dubbed, preprocess the novel text, and identify character dialogue statements, environmental description statements, and mixed statements; The second processing module 202 is used to extract dialogue content from the character dialogue statements and the mixed statements, and to perform language sentiment analysis on the character dialogue statements and the mixed statements to obtain language sentiment tags. The third processing module 203 is used to assign corresponding general timbre embedding vectors to different characters, track the changes in the characters' states during the plot development, and generate timbre gradation control parameters. The fourth processing module 204 is used to perform speech synthesis on the dialogue content based on the general timbre embedding vector, timbre gradation control parameters and language emotion tags, and generate dialogue speech waveforms. The fifth processing module 205 is used to extract environmental semantic tags and intensity parameters from the environmental description statements, and generate environmental sound effect signals based on the environmental semantic tags and intensity parameters; The sixth processing module 206 is used to mix the dialogue voice waveform and the environmental sound effect signal to generate a dubbing audio file.

[0091] It is understandable that, such as Figure 1 The content shown in the novel content dubbing method embodiments is applicable to the novel content dubbing system embodiments. The specific functions implemented by the novel content dubbing system embodiments are the same as those shown in the examples. Figure 1 The illustrated method for dubbing novel content is the same as the one used in the example, and the beneficial effects achieved are the same as those described above. Figure 1 The beneficial effects achieved by the illustrated method for dubbing novel content are the same.

[0092] It should be noted that the information interaction and execution process between the above systems are based on the same concept as the method embodiments of the present invention. For details on their specific functions and technical effects, please refer to the method embodiments section, which will not be repeated here.

[0093] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the system can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0094] Please see Figure 3 As shown, this embodiment of the invention also provides a computer device 3, including: a memory 302 and a processor 301, and a computer program 303 stored on the memory 302. When the computer program 303 is executed on the processor 301, it implements the novel content dubbing method as described in any of the above methods.

[0095] The computer device 3 may be a desktop computer, laptop, handheld computer, or cloud server, etc. The computer device 3 may include, but is not limited to, a processor 301 and a memory 302. Those skilled in the art will understand that... Figure 3 The computer device 3 is merely an example and does not constitute a limitation on the computer device 3. It may include more or fewer components than shown in the figure, or combine certain components, or different components, such as input / output devices, network access devices, etc.

[0096] The processor 301 may be a Central Processing Unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0097] In some embodiments, the memory 302 may be an internal storage unit of the computer device 3, such as a hard disk or memory of the computer device 3. In other embodiments, the memory 302 may be an external storage device of the computer device 3, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the computer device 3. Furthermore, the memory 302 may include both internal and external storage units of the computer device 3. The memory 302 is used to store the operating system, applications, bootloader, data, and other programs, such as the program code of the computer program. The memory 302 can also be used to temporarily store data that has been output or will be output.

[0098] This invention also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the novel content dubbing method as described in any of the above methods.

[0099] In this embodiment, if the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying computer program code to a photographic device / computer device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0100] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A method for dubbing novel content, characterized in that, include: The novel text to be dubbed is obtained, and the novel text is preprocessed to identify character dialogue, environmental description, and mixed sentences. The dialogue content is extracted from the character dialogue statements and the mixed statements, and the language sentiment analysis is performed on the character dialogue statements and the mixed statements to obtain language sentiment tags; Assign corresponding universal timbre embedding vectors to different characters, track the changes in the characters' states as the plot develops, and generate timbre gradation control parameters; Based on a general timbre embedding vector, timbre gradation control parameters, and language emotion tags, speech synthesis is performed on the dialogue content to generate dialogue speech waveforms. Environmental semantic tags and intensity parameters are extracted from the environmental description statements, and environmental sound effect signals are generated based on the environmental semantic tags and intensity parameters; The dialogue voice waveform and the ambient sound effect signal are mixed to generate a dubbing audio file.

2. The method as described in claim 1, characterized in that, The process of obtaining the novel text to be dubbed, preprocessing the novel text, and identifying character dialogue, environmental descriptions, and mixed dialogue includes: The novel text is segmented into sentences, paragraphs, and sorted, and each sentence is assigned a unique sentence identifier. Based on the joint judgment of guide words, quotation mark features and speaker named entities, the probability of each sentence belonging to a role dialogue statement is determined. Based on context classifier recognition, determine the probability that each sentence belongs to the context description statement; Determine whether the probability of the character dialogue exceeds a preset first threshold and whether there is at least one auxiliary feature in the character dialogue statement recognition process. If so, determine that the corresponding sentence is a character dialogue statement. Determine whether the probability of the environmental description exceeds a preset second threshold and meets the condition of no dialogue. If so, determine that the corresponding sentence is an environmental description statement. If the probability of the environmental description exceeds the second threshold, but does not meet the condition of no dialogue, then the corresponding sentence is determined to be a mixed sentence.

3. The method as described in claim 1, characterized in that, The process of extracting dialogue content from the character dialogue statements and the mixed statements, and performing language sentiment analysis on the character dialogue statements and the mixed statements to obtain language sentiment tags, includes: Based on the dialogue identifier, the dialogue content is extracted from the character dialogue statements and the mixed statements, and the name of the speaking character is determined; Based on the dialogue content extracted from the character dialogue statements and the mixed statements, and combined with the context, pragmatic phenomena are detected using a pre-trained pragmatic inference model to generate language sentiment tags. The dialogue content, the language sentiment tags, and the character tags are encapsulated into character dialogue attributes.

4. The method as described in claim 3, characterized in that, The pragmatic reasoning model includes a literal sentiment analysis branch, a semantic intent reasoning branch, and a sentiment encoding fusion branch; among which... The literal sentiment analysis branch takes the current dialogue content as input and is used to predict the basic sentiment category expressed by the current dialogue content at the literal word level, and obtain the literal sentiment probability distribution. The semantic intent reasoning branch uses the current dialogue content and its context window as joint input to identify pragmatic phenomena and output the intent category probability distribution. The sentiment encoding fusion branch is used to determine the consistency between literal sentiment and pragmatic intent. When the determination is inconsistent, a composite sentiment code is generated and used as a language sentiment label. When the determination is consistent, the language sentiment label is determined based on the output of the literal sentiment branch.

5. The method as described in claim 1, characterized in that, The method of tracking the character's state changes during plot development and generating timbre gradation control parameters includes: Detect descriptive fragments that characterize changes in a role's state, including but not limited to aging, emotional accumulation, and physical condition; Based on the described fragment, timbre gradation control parameters are generated, including timbre feature offset and interpolation rate; The timbre gradation control parameters are used to progressively adjust the general timbre embedding vector.

6. The method as described in claim 1, characterized in that, The process involves synthesizing speech from the dialogue content based on a universal timbre embedding vector, timbre gradation control parameters, and language emotion tags to generate dialogue speech waveforms, including: Convert the dialogue content into a sequence of dialogue phonemes; For different roles, obtain their corresponding general timbre embedding vector and timbre gradation control parameters, combine them with language emotion tags as dialogue condition features, input them into the target acoustic decoder, and generate dialogue acoustic features. The acoustic features of the dialogue are input into a general neural vocoder to generate a dialogue speech waveform.

7. The method as described in claim 6, characterized in that, The target acoustic decoder includes a timbre conditional injection module and an emotion conditional layer; wherein... The timbre conditional injection module is equipped with a conditional mapping network. The network parameters of the conditional mapping network are adjusted by the timbre gradation control parameters. The conditional mapping network is used to map the general timbre embedding vector into a timbre feature map. The emotional condition layer is used to perform emotional condition modulation on the timbre feature map based on the language emotional tags.

8. The method as described in claim 1, characterized in that, The step of extracting environmental semantic tags and intensity parameters from the environmental description statements, and generating environmental sound effect signals based on the environmental semantic tags and intensity parameters, includes: Identify the core environmental words in the environmental description statements, map the core environmental words to environmental semantic tags, parse the modifiers of the core environmental words, and convert them into numerical intensity parameters; The environmental semantic tags are converted into content condition vectors, and the intensity parameters are converted into style condition vectors. Based on the content condition vector and the style condition vector, and combined with a pre-trained conditional neural sound effect generation model, an environmental sound effect signal is generated.

9. A novel content dubbing system, characterized in that, include: The first processing module is used to acquire the novel text to be dubbed, preprocess the novel text, and identify character dialogue, environmental description, and mixed sentences. The second processing module is used to extract dialogue content from the character dialogue statements and the mixed statements, and to perform language sentiment analysis on the character dialogue statements and the mixed statements to obtain language sentiment tags. The third processing module is used to assign corresponding general timbre embedding vectors to different characters, track the changes in the characters' states during the plot development, and generate timbre gradation control parameters. The fourth processing module is used to synthesize speech based on the general timbre embedding vector, timbre gradation control parameters, and language emotion tags, and generate dialogue speech waveforms. The fifth processing module is used to extract environmental semantic tags and intensity parameters from the environmental description statements, and generate environmental sound effect signals based on the environmental semantic tags and intensity parameters. The sixth processing module is used to mix the dialogue voice waveform and the environmental sound effect signal to generate a dubbing audio file.

10. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method as described in any one of claims 1 to 8.

Images