Method and apparatus for automatically generating ssml text fusing split camera level information

Abstract

The application discloses a method and device for automatically generating SSML text by fusing shot level information, and the method comprises the following steps: S01, target short video original information is acquired and parsed, and information is structured in three levels of scripts, scenes and shots in sequence; S02, multi-source input features are acquired, including voice-over text, associated character features, associated scene features and associated shot features; S03, the multi-source input features are input into a pre-trained SSML generation model, the multi-source input features are encoded and fused to obtain fused features, and an SSML label sequence is obtained through a Transformer decoder, wherein each element in the sequence comprises an SSML label type and corresponding parameters, and the sequence is spliced in a hierarchical relationship to form a complete SSML text output. The application can improve the matching degree between SSML parameters and shots and characters, and improve the quality of SSML text generation.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of AIGC (Artificial Intelligence Generated Content) speech synthesis, in particular to a method and device for automatically generating SSML (Speech Synthesis Markup Language) text by fusing shot level information. BACKGROUND

[0002] SSML (Speech Synthesis Markup Language) is a markup language used to control speech synthesis, which defines parameters such as intonation, speed, pause, volume, etc. of speech through tags. For the automatic generation of SSML text for story-telling type short videos, the existing technology is usually based on single type of text information, ignoring the relevance of various information such as picture elements, character features, narrative rhythm between shots, and the character voice style is usually fixed, which may have the following problems:

[0003] 1. Lack of association between shot level information: In the process of generating SSML text based on single type of text information, each shot is processed independently, but in fact, scenes and shots are associated with each other, and there is a correlation between picture elements, character features, narrative rhythm, etc. of different shots. For example, the same character may appear in multiple scenes and multiple shots, and the basic features (personality, age, appearance features, basic intonation, etc.) of the same character are usually fixed, while some features of the same character may change dynamically with the development of the plot, such as intonation in different scenes. The traditional method of directly generating SSML text based on single type of text information may result in low matching degree between voice parameters (such as pause time, intonation) and shot picture rhythm, character state, etc., which may further result in incoherent story, inaccurate information expression, loose association of elements such as characters, scenes and shots in the script, and low quality of generated text.

[0004] 2. Lack of dynamic adaptability: Since the character voice style is fixed and cannot be dynamically adjusted with the advancement of the narrative of the scene (such as intonation migration caused by changes in character emotions), there may be a lack of explicit rules for the association between shot length and voice speed, pause, and the generated result may rely on manual correction. SUMMARY

[0005] The technical problem to be solved by the present application is that: in view of the above problems existing in the prior art, the present application provides a SSML text automatic generation method and device fusing shot level information, which can establish a dynamic association relationship between scenes, shots and characters, improve the matching degree between SSML parameters and shots and characters, and thus improve the SSML text generation quality.

[0006] To solve the above technical problems, the technical scheme provided by the present application is:

[0007] A SSML text automatic generation method fusing shot level information, comprising the following steps:

[0008] Step S01. Information structuring: obtaining target short video original information for analysis, and sequentially performing information structuring according to script level, scene level and shot level, wherein the script level is used to store global script information, the scene level is formed on the basis of the script level and is used to record scene narrative rhythm and narrative node information, and the shot level is formed on the basis of the scene level and is used to associate individual shot picture elements and voice-over text.

[0009] Step S02. Multi-source input feature acquisition: acquiring multi-source input features according to the information structuring result of step S01, wherein the multi-source input features include voice-over text, associated character features, associated scene features and associated shot features.

[0010] Step S03. SSML generation: inputting the multi-source input features into a pre-trained SSML generation model, encoding the multi-source input features by the SSML generation model, obtaining fusion features after fusion, obtaining an SSML label sequence by a Transformer decoder, wherein each element in the SSML label sequence includes an SSML label type and corresponding parameters, and sequentially concatenating the SSML label sequence according to a hierarchical relationship to form a complete SSML text output.

[0011] Further, in step S01, when information structuring is performed according to the script level, the scene level and the shot level, the script level includes global information, a character set and a scene set, scene features, narrative node information and an associated shot set are defined in the scene set to form the scene level on the basis of the script level, the scene features include narrative rhythm, shot features and associated information are defined in the shot set to form the shot level on the basis of the scene level, and the associated information includes voice-over text and picture elements.

[0012] Further, the global information includes any one or more of a video title, a narrative type, a voice style, the character set includes character features and character voice evolution features, the character features include any one or more of a character identifier, a character type, a character age, a character personality, a character feature description, a character comprehensive description, a character background story, the character voice evolution features include voice style evolution features and voice parameter adjustment coefficients for different scene ID, the voice parameter adjustment coefficients include a pitch offset feature pitchOffset, a speech rate coefficient rateCoeff, the scene features include a scene ID and a narrative rhythm, the narrative node information includes a node type and a parameter adjustment rule, and the shot features include a shot ID, a shot duration, and a semantic weight.

[0013] Further, in step S01, when structuring information according to the script level, the scene level, and the shot level, the script level provides a global story background, a character library, and a scene library, and defines a character basic voice parameter, the scene level serves as an intermediate layer to integrate the style reference of the corresponding shot, the scene level realizes character style migration with the narrative advancement through the voice parameter adjustment coefficients, and the semantic weight is constrained by the narrative rhythm; the shot level serves as the final carrier layer to associate parameters by referencing the information of the upper layer, the character field of the shot references the character object in the character set to pass the character attributes to the shot level, and the attributes in the scene level provide a unified style reference for the corresponding shot.

[0014] Further, in step S03, the SSML generation model includes an input layer, a sub-encoder layer, a fusion layer, and a decoder layer connected in sequence, wherein the fusion layer encodes the voice-over text into a text vector through a text sub-encoder, encodes the character voice evolution features into a voice parameter vector through a character sub-encoder, encodes the scene narrative features into a narrative vector through a narrative node sub-encoder, and encodes the shot features into a shot semantic vector through a shot semantic sub-encoder, and the text vector, the voice parameter vector, the narrative vector, and the shot semantic vector are weighted and fused to obtain a fusion vector .

[0015] Further, in the process of weighting and fusing the text vector, the voice parameter vector, the narrative vector, and the shot semantic vector to obtain a fusion vector , when the advancement speed of the scene narrative rhythm exceeds a preset threshold, the weight of the narrative vector is increased and the weight of the text vector is decreased, and when the node type is a conflict point, the weight of the narrative vector is increased and the weight of the shot semantic vector is decreased.

[0016] Further, in step S03, the Transformer decoder generates the fusion vector according to the obtained fusion vector generate an SSML tag sequence, wherein the SSML tag sequence comprises a prosody control tag and a pause time tag, the prosody control tag comprises one or more of a prosody control parameter of a speech rate, a tone, and a volume, and the pause time tag is used to control a pause time corresponding to a split screen.

[0017] Further, the SSML generation model uses the following loss function in the training process:

[0018]

[0019] wherein, denotes a text semantic loss, denotes a scene-level style loss, denotes a shot-level parameter loss, denotes a shot-level parameter loss a weight, denotes a scene-level style loss a weight, and the shot-level parameter loss uses an L1 loss to constrain the shot-level parameter prediction bias to constrain the shot-level parameter accuracy, and the scene-level style loss uses an L2 loss to constrain the consistency of the parameters within the scene;

[0020] The calculation expression of the text semantic loss is as follows:

[0021]

[0022] wherein, is an original text semantic sentiment score of the i-th scene and the j-th shot, is a sentiment score corresponding to the i-th scene and the j-th shot converted into an SSML sequence, denotes a scene index, denotes a total number of scenes in a script of a target short video, denotes a shot index, denotes a number of shots contained in the i-th scene; M The calculation expression of the shot-level parameter loss is as follows:

[0023] wherein

[0024]

[0025] ​​​​​For the first Act 1 The annotation parameters for each scene. For the first Act 1 Predictive parameters for each scene;

[0026] Actual style loss The calculation expression is:

[0027]

[0028] in, For the first The mean of the prediction parameters for each segment, For the first The average value of the annotation parameters for each segment.

[0029] An SSML text automatic generation device that integrates storyboard hierarchy information includes a processor and a memory, wherein the memory is used to store a computer program and the processor is used to execute the computer program to perform the method described above.

[0030] A computer-readable storage medium storing a computer program that, when executed, implements the method described above.

[0031] Compared with existing technologies, the advantages of this application are as follows: This application constructs a three-level dynamic semantic association structure of "script-act-storyboard", and realizes hierarchical management and cross-level association of information through structured layering and element binding. It can strengthen the deep binding of storyboard with character voice evolution and act narrative rhythm, and efficiently integrate storyboard hierarchical information. It solves the problem of loose association between character, act and storyboard elements in traditional generation methods. Based on this three-level dynamic semantic association structure, combined with the SSML generation model, it can realize the automatic generation of SSML text by dynamic semantic association at the storyboard level, improve the efficiency and intelligence of text generation, and improve the matching degree between voice parameters and storyboard rhythm and character state. It realizes dynamic adaptation of "character-narrative-storyboard" full-dimensional collaborative adaptation, making the generated result more in line with the needs of short video voice synthesis and improving the quality of generated text. Attached Figure Description

[0032] Figure 1 This is a schematic diagram illustrating the implementation process of the SSML text automatic generation method that integrates storyboard hierarchy information in this embodiment.

[0033] Figure 2 This is a schematic diagram illustrating the principle of constructing a three-level data structure of script, scene, and shot in this embodiment.

[0034] Figure 3 This is a schematic diagram of the structural principle of the SSML model in this embodiment. Detailed Implementation

[0035] The present application will be further described below with reference to the accompanying drawings and specific preferred embodiments, but this does not limit the scope of protection of the present application.

[0036] To facilitate understanding, the technical background involved in this application will be introduced first.

[0037] In traditional text-based SSML text generation multimodal content creation schemes, the information association at the scene level is often not considered, leading to disjointed storylines and pacing, and poor generation quality. For example, consider the production of a children's animated short film. This short film contains three consecutive acts under the theme of "Forest Adventure," each with multiple scenes. The protagonist is a lively, clear-voiced rabbit named "Jumpy." In a traditional single-text-driven SSML generation scheme, in scene 1 of act one, "Discovering the Mysterious Cave," the text description is "Jumpy excitedly shouts: 'Wow, there's a cave here!'" The system will only generate a fast-paced, high-pitched voice based on this text. In scene 3 of act two, "Exploring the Cave's Interior," the text is "Jumpy says somewhat fearfully: 'It's so dark here.'" Due to the lack of cross-scene character feature association, the system does not use Jumpy's clear, basic voice; instead, it generates a low, hoarse voice, making it difficult for viewers to associate the voices in the two scenes with the same character, severely disrupting the consistency of the character's image. Worse still, there is a scene transition between scene 4 of Act 2 and scene 1 of Act 3, "Finding the Treasure"—from "the dark corner of the cave" to "the glowing treasure scene." Traditional generation methods, because they do not consider the narrative rhythm connection between acts and scenes, will cause the ending voice of scene 4 to end abruptly, while the voice of scene 1 suddenly becomes loud and clear, without any pause to match the scene transition, resulting in a break in the story rhythm and a very poor viewing experience for the audience.

[0038] Take a virtual anchor live script as an example. The script includes two scenes, "product introduction" and "user interaction". In the "product introduction" scene, the picture element is "the anchor holds the new mobile phone and shows the screen details", and the text description is "the screen resolution of this mobile phone is very high". In the "user interaction" scene, the picture is "the anchor looks at the scroll and smiles in response", and the text is "the audience asks about the battery life, and the battery life of this mobile phone is very powerful". The traditional generation scheme only relies on single text to generate SSML, so it does not combine the picture element "show screen details" in "product introduction", and the voice lacks emphasis on "very high". Moreover, the voice does not reflect the friendly feeling of interaction with the audience in the "user interaction" scene, resulting in a serious disconnection between voice parameters and picture elements and scene states, inaccurate information expression, and difficulty for the audience to immerse themselves.

[0039] Although some of the existing solutions can partially alleviate the dynamic adaptability problem, they all have limitations: the parameter binding scheme based on SRT timestamp can achieve audio-video synchronization by relying on the subtitle timestamp to map the voice parameters, but it needs to rely on existing subtitles and ignores dynamic information such as characters and narration; the text segmentation scheme based on NLP semantic analysis can associate text semantics by splitting semantic pause units and generating parameters combined with sentiment analysis, but it lacks multi-modal linkage such as split screen pictures and duration; the dynamic voice template scheme of cloud service can provide pre-defined emotion templates, which has high integration but fixed templates, and cannot be adjusted with split screen details; the black box model based on attention mechanism can automatically capture multi-feature association, which is suitable for complex scenarios, but has poor interpretability, high training cost and long generation time. The above schemes only consider part of the associated information and can only alleviate the dynamic adaptability to a certain extent, and cannot achieve coordinated adaptation in multiple dimensions, making it difficult to meet the current short video scenario requirements.

[0040] The application can effectively solve the above problems by constructing a "script-act-scene" three-level dynamic semantic association structure and adopting nested association design from the script level to the act level and then to the scene level. In the "script-act-scene" three-level dynamic semantic association structure, the first script level is a global resource layer that stores shared information of the whole script and defines basic voice parameters of characters, such as the basic tone parameters of the character "Tiaotiao" and the basic style of the virtual anchor, to ensure that the basic characteristics of the same character are consistent in different acts and scenes, avoiding "mutation" of the character voice; the second act association layer is used to record the narrative rhythm and narrative node information between acts, for example, for the picture transition of "cave to treasure", the corresponding voice pause duration can be preset in this layer to ensure the coherence of the story rhythm; the third scene atom layer is used to associate picture elements and voice-over text of a single scene, such as "showing the mobile phone screen", which sets the emphasis voice parameters in this layer, and "looking at the bullet screen interaction", which configures the affectionate tone parameters, so that the voice is highly matched with the scene details. Through the above three-layer structure, the application can build a close association between scene-level information. After integrating the multi-source input features based on the three-layer structure information, the complete SSML text is generated by combining the SSML generation model, which can fully consider the information association between scene levels, realize dynamic adaptation of "character-narrative-scene" full-dimensional collaborative adaptation, solve the problem of insufficient dynamic adaptation of traditional SSML generation scheme, improve the quality of SSML text generation, and ensure the coherence and accuracy of multi-modal content.

[0041] The application will be further described below taking the story telling type short video scene as an example. As shown in Figure 1 The steps of the SSML text automatic generation method of the embodiment fusing scene-level information include:

[0042] Step S01. Information structuring: obtaining and analyzing the original information of the target short video, and sequentially structuring the information according to the script level, act level and scene level, wherein the script level is used to store the shared information of the whole script, the act level is formed on the basis of the script level and is used to record the narrative rhythm and narrative node information between acts, and the scene level is formed on the basis of the act level and is used to associate picture elements and voice-over text of a single scene;

[0043] Step S02. Multi-source input feature acquisition: acquiring multi-source input features according to the information structuring result of step S01, wherein the multi-source input features include voice-over text, associated character features, associated act features and associated scene features;

[0044] Step S03. SSML generation: input the multi-source input features into the pre-trained SSML generation model, encode and fuse the multi-source input features to obtain the fusion features, and obtain the SSML label sequence through the Transformer decoder, wherein each element in the SSML label sequence includes an SSML label type and corresponding parameters, and the complete SSML text output is formed by sequentially concatenating the SSML label sequence according to the hierarchical relationship.

[0045] By constructing a three-level dynamic semantic association structure of "script-episode-scene", the structure adopts a nested association design from the script level to the episode level and then to the scene level, and can realize hierarchical management and cross-level association of information through structured hierarchical binding and element binding, can strengthen the deep binding of scene and character voice evolution and episode narrative rhythm, efficiently fuse scene level information, solve the problem of loose association of character, episode and scene elements in traditional generation methods, and based on the three-level dynamic semantic association structure, further combined with the SSML generation model, the scene level dynamic semantic association can automatically generate a voice synthesis markup language (SSML) text, which can improve the text generation efficiency and intelligent degree, and can improve the matching degree between voice parameters and scene rhythm and character state, so that the generated result is more in line with the short video voice synthesis demand, and the generated text quality is improved. Using the SSML text can accurately control the tone, speed, pause and other parameters of voice synthesis, and can be applied to various voice synthesis engines.

[0046] In this embodiment, when the information is structured according to the three levels of script, episode and scene in step S01, the script level includes global information, a character set and an episode set, the episode set defines episode features, narrative node information and associated scene sets of each episode to form an episode level based on the script level, the episode features include narrative rhythm, etc., and the scene set defines scene features and associated information to form a scene level based on the episode level, the associated information includes voice-over text and picture elements, etc.

[0047] Specifically, in the three-level dynamic semantic association structure of "script-act-storyboard", the script layer provides the global story background, character library and act library. The act layer, as the intermediate layer, integrates the style benchmarks of its respective storyboards to ensure coherence. The storyboard layer, as the final carrier layer, focuses on the details of specific scenes. The storyboard, as the core node of information integration, can extract the age, personality and other characteristics of related characters in the character layer, as well as the background sound, style constraints and other content related to the act layer, through ID indexing and nested references. This makes each storyboard an "information node" containing multi-dimensional elements, which can realize structured binding and cross-level mapping mechanisms. This allows for deep association between storyboard level information and related characters and acts, which can then be provided to the subsequent SSML generation model to accurately generate SSML text that matches the scene and character state.

[0048] Specifically, such as Figure 2 As shown, in this embodiment, the global information at the script level specifically includes basic information such as video title, narrative type St, and basic voice style Bv. The script level also includes a character set and an act set. The character set includes character characteristics and character voice evolution characteristics. Character characteristics include character ID, character age Ag, character personality P, and may also include character type, character characteristic description, character comprehensive description, character background story, etc. Character voice evolution characteristics include voice style evolution characteristics VSE and voice parameter adjustment coefficients for different act IDs. Voice style evolution characteristics VSE include acoustic features (pitch, duration, etc.), emotional features (sadness, happiness, etc.), and style features, etc. Voice parameter adjustment... The coefficients include pitch offset features such as pitchOffset and rateCoeff; the scene set includes scene features, key narrative node information, and associated scene set. The scene features include scene ID and narrative rhythm Nr, the key narrative node information includes node type Nn and parameter adjustment rule paramAdjust, and the scene features include scene features and associated information. The scene features include scene ID (actId), scene duration Du_lq, and semantic weight Nw_lq, and the associated information includes visual elements Ve_lq, voice-over text Te_lq, associated characters Ch_lq, associated scene features, and associated scene features.

[0049] In specific application embodiments, semantic weight ( The value (0-1) can be determined by comprehensively considering the character's voice evolution features (such as intonation shifts and speech rate coefficients in different acts), the narrative node information of each act (such as parameter adjustment rules for conflict points), and the scene features (such as scene duration). Specifically, the semantic weight can be obtained by weighting the above three types of parameters. The weight of each parameter can be determined according to the feature type and the size of the key parameter. For example, in the character voice evolution feature, a low speech rate coefficient (such as "calm old man" rateCoeff=0.8) increases the semantic weight, and a high pitch offset (such as "excited child" pitchOffset=+10%) reduces the weight; in the scene narrative node, the "conflict point" increases the weight, and the "transition point" reduces the weight; in the shot feature, long duration and complex picture elements increase the weight, and short voice-over text reduces the weight.

[0050] Further, the embodiment sets the character voice reference through the voice parameter adjustment coefficient at the scene level, and the semantic weight inherits the parameter, which can realize the evolution of the character voice with the scene; the narrative node adjusts the semantic weight and the model feature weight to match the voice parameter, realizes the accurate synchronization of the narrative node and the voice rhythm, and can realize the dynamic evolution of the character voice style with the scene, the accurate synchronization of the narrative node and the voice rhythm, and meets the special needs of the story telling scene.

[0051] In a specific application embodiment, in order to realize the nested association of the three-level data structure, the following configuration can be performed:

[0052] The "appearanceCharacters" (character) field of the shot directly references the character object in the character set (Characters), and the core attributes such as "age" (age) and "personality" (personality) of the character are transmitted to the shot level through the reference, forming the attribute binding of "character - shot". For example, the character of "15 years old + lively" is associated with the voice parameters of high speech rate and high pitch by default in the shot;

[0053] The scene level (Acts) as an intermediate layer provides uniform style reference for the background sounds and voice-overs of the belonging shot, to ensure that the shots in the same scene keep consistent in the character voice style and scene adaptation logic.

[0054] Further, a key-value pair index structure can also be used, and the shot can directly obtain the character personality and scene description information through the character ID and scene ID, realize the information penetration type reference mechanism, make the shot directly call the underlying attributes of the character and scene, solve the problem of loose association of the character, scene and shot elements in the traditional scheme, and realize the efficient transmission and reuse of information.

[0055] The following takes the construction of the three-layer data structure based on the script-scene-shot in a specific application embodiment as an example:

[0056] 1. Script level: used to provide a global narrative framework and define the story type, and the JSON structure is as follows:

[0057] {

[0058] "title": "Video Title",

[0059] "storyType": "Narrative type (e.g., children's story / educational narration)",

[0060] "baseVoiceStyle": "Basic voice style (e.g., cheerful / composed)",

[0061] "characters": [ ], / / Collection of characters

[0062] "acts": [ ] / / Acts collection

[0063] }

[0064] The above script hierarchy can be represented as ,in The narrative type is used to determine the speech style benchmark (e.g., a high-pitched default for children's stories). The video title Based on basic speech style, For character collection, This is a collection of scenes.

[0065] 2. Character Set: Contains character characteristics and character voice evolution features. The specific JSON structure is as follows: [

[0067] {

[0068] "characterId": "Unique identifier for the character",

[0069] "age": "age",

[0070] "personality": "character"

[0071] "voiceStyleEvolution": [ / / Act-by-act speech style evolution]

[0072] {"actId": "1", "pitchOffset": "+5%", "rateCoeff": 1.1}, / / Act 1 intonation offset +5%

[0073] {"actId": "2", "pitchOffset": "+2%", "rateCoeff": 1.0} / / Pitch offset for Act 2 +2% ]

[0075] } ]

[0077] The above role set can be represented as: , wherein , is the unique identification of the role, is the age, is the personality, Records the voice parameter adjustment coefficient (pitchOffset is the pitch offset, rateCoeff is the speech rate coefficient) of the role in different acts, to realize dynamic style transfer.

[0078] 3, Act set: Define narrative rhythm and key narrative nodes, JSON structure is as follows: [

[0080] {

[0081] "actId": "Act unique identifier",

[0082] "narrativeRhythm": "Narrative rhythm (fast forward / slow down)",

[0083] "narrativeNodes": [ / / narrative nodes and voice parameter adjustment rules

[0084] {"nodeType": "Conflict point", "paramAdjust": {"rate": "+15%", "pitch": "+8%"}},

[0085] {"nodeType": "Turning point", "paramAdjust": {"breakTime": "+50ms"}}

[0086] ],

[0087] "shots": [ ] / / shot set

[0088] } ]

[0090] The above act set can be represented as: , wherein , is the unique identification of the act, is the narrative rhythm (used to determine the overall speech rate benchmark of the act, such as "fast forward" corresponding to 1.2 times the default speech rate), is the narrative node and parameter adjustment rule (wherein nodeType is the node type, and paramAdjust is the parameter adjustment value), Shot collection.

[0091] 4. Shot collection: associate specific picture elements with semantic weight, JSON structure as follows: [

[0093] {

[0094] "shotId": "shot unique identifier",

[0095] "duration": "duration (seconds)",

[0096] "narrativeWeight": "semantic weight (0-1)", / / The higher the longer the pause

[0097] "appearanceCharacters": ["character ID1", "character ID2"], / / Associate characters

[0098] "visualElements": "picture element description (such as "wave action" "lower head meditation")",

[0099] "textContent": "voiceover text"

[0100] } ]

[0102] The above shot collection can be represented as: , wherein , is the unique identifier of the shot, is the duration, is the semantic weight, is the associated character, is the picture element description, is the voiceover text.

[0103] In this embodiment, the multi-source input features include voiceover text, associated character features, associated scene features, and associated shot features, wherein the associated character features include the character personality, age, and evolution coefficient of the associated character, the associated scene features include the rhythm and node adjustment features of the associated scene, and the associated shot features include the shot duration, semantic weight, and picture elements of the associated shot. After information structuring according to step S01, the above multi-source input features can be obtained from the final information structured result, and then input into the SSML generation model to automatically generate the SSML text.

[0104] In step S03 of this embodiment, the SSML generation model specifically includes an input layer, a sub-encoder layer, a fusion layer, and a decoder layer, as shown in Figure 3 The input layer is configured to receive multi-source input features The sub-encoder layer is configured to encode the multi-source input features respectively, the fusion layer is configured to fuse the encoding results output by the encoders in the sub-encoder layer to obtain fusion features, and the decoder layer is configured to decode the fusion features output by the fusion layer to output SSML text. The sub-encoder layer includes a text sub-encoder, a character sub-encoder, a narrative node sub-encoder, and a shot semantic sub-encoder. The text sub-encoder is configured to encode the voice-over text into a text vector, the character sub-encoder is configured to encode the character voice evolution features into a voice parameter vector, the narrative node sub-encoder is configured to encode the scene rhythm and node adjustment rules into a narrative vector, and the shot semantic sub-encoder is configured to encode the shot features into a shot semantic vector including time length, semantic weight, and picture elements. The fusion layer is configured to weight and fuse the text vector, the voice parameter vector, the narrative vector, and the shot semantic vector to obtain a fusion vector .

[0105] Specifically, each encoder in the sub-encoder layer is configured as follows:

[0106] Text sub-encoder: configured to encode the voice-over text into a text vector, capturing the semantic content and emotional tendency of the text;

[0107] Character sub-encoder: configured to focus on the core features (personality + age) of the character to encode them into a vector. For example, a character with “70 years old + calm” is encoded into a feature vector with low speech rate and low pitch;

[0108] Narrative node sub-encoder: configured to encode the scene rhythm and node adjustment rules into a vector to obtain a narrative vector;

[0109] Shot semantic sub-encoder: configured to encode the picture elements and sound elements of the shot to generate a vector, realizing the feature mapping of converting visual description (such as “hand waving action”) into voice pause (such as 300ms pause).

[0110] In this embodiment, the fusion layer further adopts a rule-driven feature fusion method to integrate information, and weights and fuses the text vector, the voice parameter vector, the narrative vector, and the shot semantic vector to obtain a fusion vector When the advancing speed of the episodic narrative rhythm exceeds a preset threshold, the weight of the narrative vector is increased and the weight of the text vector is decreased, and when the node type is a conflict point, the weight of the narrative vector is increased and the weight of the shot semantic vector is decreased. By using a rule-driven feature fusion method to replace the traditional black box mechanism, accurate fusion of shot semantics, character features, and narrative nodes can be achieved, further improving the interpretability of parameter generation.

[0111] For example, an initial weight can be assigned to each vector in advance to generate an initial fusion vector For example, the rules are as follows:

[0112] Calculate the initial fusion weight:

[0113] (1)

[0114] According to the advancing speed of the episodic narrative rhythm and the node type, the weight of each vector is dynamically adjusted, for example, the dynamic adjustment rule can be configured as follows: when the episodic narrative rhythm is “fast advancing”, the weight is increased to 0.3, the weight is decreased to 0.25; when the node type is “conflict point”, the weight is increased to 0.35, the weight is decreased to 0.15.

[0115] In this embodiment, the Transformer decoder in step S03 generates an SSML label sequence according to the obtained fusion vector The SSML label sequence includes a tone control label and a pause time label. The tone control label includes tone control parameters such as speech rate, tone, and volume. The pause time label is used to control and guide the voice synthesis engine to output the pause time corresponding to the shot. Specifically, the decoder layer uses a label sequence generation and parameter prediction parallel method according to the characteristics of the SSML text. On the one hand, the basic SSML label (such as <prosody> 、 <break>), and on the other hand, according to the fused feature vector, specific parameters such as intonation, speech rate, pause time, etc. are predicted. For example, according to the shot length and picture elements, the decoder can accurately generate a pause time that matches the picture rhythm. For example, in a 3-second shot, the pause time corresponding to the "hand waving action" is set to 300ms.

[0116] The embodiment can realize the understanding and fusion of information at each level by adopting the above-mentioned "multi-feature sub-encoder - cross-attention fusion - hierarchical decoding" architecture.

[0117] In a specific application embodiment, the structure of the SSML model can be configured as follows:

[0118] 1. Input layer:

[0119] This layer receives multi-source input features , wherein: is the voice-over text ( ), is the character voice evolution feature (age + personality + evolution coefficient ), is the scene narrative feature (rhythm + node adjustment ), is the shot feature (length + semantic weight + picture elements )

[0120] 2. Sub-encoder layer:

[0121] This layer uses each encoder to convert the corresponding content into a string and then uses the BERT model to encode it into a semantic vector. The text sub-encoder encodes the text semantics using BERT, and outputs a vector ( with a feature dimension); the character sub-encoder encodes into a speech parameter vector (pitch, speech rate coefficient, etc.), and outputs ; the narrative node sub-encoder encodes the scene rhythm and node adjustment rules into a vector, and outputs ;

[0122] The shot semantic sub-encoder encodes the length, semantic weight, and picture elements (such as "hand waving" mapped to "interactive emphasis" tag features), and outputs .

[0123] 3. Fusion Layer: The text vector, speech parameter vector, narrative vector, and scene semantic vector are weighted and fused to obtain the fusion vector. The weights of each vector are dynamically adjusted based on the pace of the narrative and the type of nodes.

[0124] 4. Decoder layer:

[0125] The decoder layer includes a generation module and a parameter prediction module, wherein the generation module is used to generate the fusion vector output by the fusion layer. Generate SSML tag sequences (e.g.) <prosody> <break> <emphasis>) output, wherein <prosody>is a prosody control tag, including parameters such as speech rate, pitch, volume, etc. <break>is a pause time tag, <emphasis>is an emphasized tag; the parameter prediction module is configured to predict parameters of the tags according to the fusion vectors outputted by the fusion layer The specific SSML parameters are calculated, such as the pitch rate="115%" and the break time break time="300ms".

[0126] Specifically, the generation module adopts a Transformer decoder to output a sequence of SSML tags , which contains: , wherein is the decoder. The output tag sequence contains specific SSML tag types and corresponding parameters, such as <prosody rate="110%" pitch="+5%" volume="medium"> (a speech control tag containing parameters of speech rate, pitch, and volume), <break time="300ms"> (a break tag containing a break time), and embedded natural language text (such as "Good morning! The sunshine today is so good~"). <break time="300ms" / >

[0127] The decoder layer also includes a syntax verification module configured to forcibly verify the nesting relationship of the tags (such as <break>must be located <prosody>(inside) and parameter ranges (e.g., rate values 50%-150%) to ensure compliance with SSML specifications. When generating SSML text, the label sequences generated by the Transformer decoder are mapped to a hierarchical relationship (e.g., <speak>Root tag wrapping <prosody>Sub-labels, text content embedding labels between tags) are spliced in turn, while ensuring that the opening and closing of the tags correspond (such as each <prosody>Match< / prosody> ), and finally forming a complete SSML text that meets the grammar specifications.

[0128] The above SSML model needs to be trained before use. A story telling type short video data set is used for training. The data set contains multiple shot samples, covering various scene types such as children's stories and educational commentary. Each sample is labeled with voice-over text, shot level information (duration, semantic weight, etc.), and artificially calibrated SSML parameters. The training data set can be represented as sample , wherein , is the SSML label. The data labeling can be completed by multiple experts in the field of speech synthesis, and cross-validation is used to ensure consistency (Kappa coefficient ≥ 0.85).

[0129] In this embodiment, the SSML generation model uses the following loss function in the training process:

[0130] (2)

[0131] wherein represents the text semantic loss, represents the scene level style loss, represents the shot level parameter loss, represents the weight of the shot level parameter loss , and represents the weight of the scene level style loss , and the shot level parameter loss uses L1 loss to constrain the shot parameter prediction deviation to constrain the accuracy of the shot level parameter. The scene level style loss uses L2 loss to constrain the consistency of the scene parameters.

[0132] In the model training process, the multi-level loss function is used to calculate the matching degree between the original voice-over text semantics and the generated SSML sequence using the text semantic loss , which can ensure the matching of the text semantics and the SSML sequence. The shot level parameter loss can enhance the sensitivity to the deviation of the single shot SSML label sequence and the labeled result, and can prioritize the matching degree of the voice parameters of a single shot and the picture and the character. Through the scene level L2 loss, the consistency of the SSML label sequence of all shots in the same scene can be constrained in terms of style and logic, ensuring the coherence and unity of the voice style of multiple shots in the scene, ensuring the consistency of the scene level style, and through the integration of the shot, scene and character three levels of measurement results, the coherence and unity of the overall generated content can be effectively ensured, and the generated result can be accurately constrained.

[0133] Specifically, text semantic loss The calculation expression is:

[0134]

[0135] in, For the first Act 1 The original text semantic sentiment score of each scene can be obtained through a pre-trained sentiment classification model, with a range of 0-10. For the first Act 1 Each storyboard is converted into an SSML sequence, and the corresponding sentiment score can be determined based on the parameters in the generated SSML sequence. For example, pitch="+5%" corresponds to a score of 8, and pitch="-5%" corresponds to a score of 3. Indicates the episode index. , M This indicates the total number of scenes in the script of the target short video. Indicates the scene index. , Indicates the first The number of shots contained in a single scene.

[0136] Specifically, storyboard-level parameter loss The calculation expression is:

[0137] (4)

[0138] in For the first Act 1 The annotation parameters for each scene. For the first Act 1 Prediction parameters for each scene.

[0139] This embodiment uses the loss function of equation (4) above to constrain the deviation between the SSML tag sequence corresponding to a single scene and the annotation result, ensuring the accuracy of the content generated within a single scene. The SSML tag sequence corresponding to a single scene is a set of tags deeply bound to information such as scene elements, character status, and duration, mainly including: <prosody>For the core voice parameter tags, mark the speed, tone, volume, etc. parameters that adapt to the age (e.g. "15 years old" corresponds to rate="110%"), personality (e.g. "calm" corresponds to pitch="-3%"), and scene atmosphere (e.g. "midnight" corresponds to volume="soft") of the characters in the split shots. <break>For representative pause tags, mark the matching split-screen duration (e.g., 3 seconds, 5 seconds) and the pause duration (e.g., time="300ms" "500ms") of the picture action (e.g., waving, turning a book); meanwhile, include <speak>The root tag, as the overall container, can ensure the integrity and hierarchy of the speech synthesis control tags within a single scene, thereby accurately adapting to the various features of the scene.

[0140] Specifically, the loss of style at the scene level The calculation expression is:

[0141] (5)

[0142] in, For the first The mean of the prediction parameters for each segment, For the first The average value of the annotation parameters for each segment.

[0143] This embodiment uses the loss function of equation (5) above to constrain the consistency deviation in style and logic of the SSML tag sequences of all shots within the same segment, ensuring that the generated content of multiple shots within the segment is coherent and unified. The SSML tag sequence corresponding to a single segment is a set of style-unified tags generated based on the common features of all shots within that segment, mainly including: tags that reflect the overall tone of the segment. <prosody>Tag parameter range (e.g. a single scene character speech rate deviation not exceeding 20%, pitch shift amplitude kept consistent), adaptation of scene background sound (e.g. "schoolyard scene" scene uniformly associated with a medium volume reference) <volume>Related sub-labels, and ensuring logical coherence between the shots <break>Tag duration gradient rules (e.g., pausing duration gradually increases when transitioning from an action scene to a narrative scene within an episode), while also <speak>The root tag wraps the entire sequence of tags of a sub-scene. In this embodiment, the unified style parameter constraint can ensure that the voice synthesis of different shots in the sub-scene is consistent in intonation, rhythm, and atmosphere, thereby matching the overall style defined by the sub-scene overview.

[0144] Further, the total loss function can balance the requirements of single-shot accuracy and sub-scene coherence through weight distribution, so that the model can better understand and integrate information at different levels. For example, the weight of the sub-scene level style loss may be configured to be less than the weight of the shot level parameter loss to prioritize shot parameter accuracy.

[0145] The above fine-tuned SSML model can be used to generate high-matching SSML text in real time. The specific process includes:

[0146] Information extraction: Obtain multi-source shot level features from the target voiceover ;

[0147] Feature encoding and fusion stage: convert input features into semantic vectors through sub-encoders, generate feature vectors through sub-encoder layers, and then generate a fusion feature vector through a fusion layer. The vector contains multi-dimensional information such as text semantics, character features, sub-scene information, and shot parameters, according to the rule weight ;

[0148] Decoding generation stage: the Transformer decoder generates SSML text containing parameters such as intonation, speech rate, and pause based on the fusion feature vector. For example, for the shot "Good morning campus, 15-year-old lively student greets", the generated SSML text will contain parameters such as "rate='110%' pitch='+5%'", and a 300ms pause tag to match the "wave action", achieving a high degree of fit with the shot picture and character state.

[0149] In a specific application embodiment, in combination with the three-level data structure, the SSML generation sample obtained for a specific shot is as follows:

[0150] <speak>

[0151] <!-- Scene 1: Morning schoolyard, student greeting (3 seconds long, character age 15, lively personality) -->

[0152] <prosody rate="110%" pitch="+5%" volume="medium">

[0153] Good morning! Isn't the sunshine nice today?

[0154] <break time="300ms" / > <!-- Pauses to match the hand waving action in the picture -->

[0155] < / prosody>

[0156] <!-- Scene 2: Late-night study room, old man telling stories (5 seconds long, character age 70, calm personality) -->

[0157] <prosody rate="80%" pitch="-3%" volume="soft">

[0158] That winter, the snow fell especially heavily...

[0159] <break time="500ms" / > <!-- Pauses to match the action of turning the page in the picture -->

[0160] < / prosody>

[0161] < / speak>

[0162] The application can form a character voice evolution mechanism through the above-mentioned automatic generation mode of the SSML text: the script level character set defines the character voice evolution characteristics (including voice style evolution characteristics such as acoustic characteristics (pitch, length, etc.), emotional characteristics (sadness, happiness, etc.), and style characteristics, and different scene pitch offset characteristics pitchOffset, speech rate coefficient rateCoeff, and other voice parameter adjustment coefficients), the shot level references the character object through the "associated character" field to pass the evolution characteristics, and the evolution coefficient of the associated character is included in the multi-source input characteristics, and then the character sub-encoder of the SSML generation model encodes the voice parameter vector into the fusion vector Finally, the character voice SSML parameters of the adapted scene are generated, and a narrative node adjustment mechanism is formed: the scene level defines the narrative node information (including node types such as conflict points / turning points and parameter adjustment rules), and the narrative node information is taken as a core factor (such as increasing the semantic weight by 0.2-0.3) when calculating the semantic weight of the shot, and the pause time is extended by calculating the pause time according to the semantic weight, and the feature weight is dynamically adjusted according to the node type (such as increasing the narrative vector weight of the conflict point) by the fusion layer of the SSM generation model, so as to ensure that the voice rhythm is synchronized with the narrative node. Through the above-mentioned mechanism, the dynamic binding of the shot level information and the SSML parameters can be realized, so that the dynamic association between shots is more closely related, the voice rhythm is synchronized with the narrative node through the semantic weight and rhythm rule, the story telling scene can be focused on, the narrative adaptability is stronger, the parameter matching accuracy of the conflict point / turning point can be as high as 91%, and the generation efficiency can be greatly improved. In actual application, the automatic generation process can take ≤0.3 seconds per shot, which is more than 5 times more efficient than manual writing (1.5 seconds per shot on average).

[0163] To verify the effectiveness of the present application, the present application is compared with two methods in the prior art in a specific application embodiment:

[0164] Method 1: The existing mainstream black box model based on cross attention;

[0165] Method 2: The SSML automatic generation function of the open source tool Amazon Polly;

[0166] The evaluation indexes include:

[0167] Matching degree: the matching degree of the shot level information (duration, character characteristics, etc.) and the SSML parameters, scored by 3 experts (0-100), and the average value is taken;

[0168] Dynamic migration accuracy of character voice style: the proportion of shots that correctly migrate the character voice style (such as from "lively" to "calm"), determined by a pre-trained voice style classification model;

[0169] Narrative node synchronization rate with voice rhythm: the SSML parameters (speech rate, pause) corresponding to the narrative nodes (conflict points, turning points) meet the expected shot ratio, and the expected value is annotated by experts.

[0170] Generation time: the SSML generation time of a single shot (unit: seconds).

[0171] The experimental results obtained are shown in Table 1:

[0172] Table 1: Comparison of test results

[0173]

[0174] As shown in Table 1, in terms of matching degree, the present application is improved by (92.6-68.5) / 68.5≈35% compared with method 1, and by (92.6-72.3) / 72.3≈28% compared with method 2, and the difference is statistically significant by t-test (P<0.01); in terms of dynamic migration accuracy of character voice style, the present application is significantly higher than the comparative methods, verifying the effectiveness of the character voice evolution mechanism of the present application; in terms of narrative node synchronization rate, the present application is more accurate in parameter control of conflict points and turning points, verifying the effectiveness of the narrative node adjustment rule, and in terms of generation time, the present application is the most efficient, which is due to the rule-driven fusion mechanism reducing complex model calculation.

[0175] The embodiment also provides an SSML text automatic generation device fusing shot level information, comprising a processor and a memory, the memory is used to store a computer program, and the processor is used to execute the computer program to perform the above method.

[0176] It can be understood that the above method of the embodiment can be executed by a single device, such as a computer or a server, etc., and can also be applied to a distributed scenario to be completed by multiple devices cooperating with each other, in which case, one of the multiple devices can only execute one or more steps in the above method of the embodiment, and the multiple devices interact with each other to complete the above method. The processor can be implemented in the form of a general-purpose CPU, a microprocessor, an application-specific integrated circuit, or one or more integrated circuits, etc., for executing related programs to implement the above method of the embodiment. The memory can be implemented in the form of a read-only memory (ROM), a random access memory (RAM), a static storage device, and a dynamic storage device, etc. The memory can store an operating system and other application programs, and when the above method of the embodiment is implemented by software or firmware, the related program codes are saved in the memory and executed by the processor.

[0177] The embodiment further provides a computer readable storage medium storing a computer program, which is executed by a processor to implement the above method.

[0178] Those skilled in the art will appreciate that the above-described embodiments of the present application can be provided as methods, systems, or computer program products. Accordingly, the present application can be embodied in the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application can be embodied in the form of a computer program product on one or more computer readable storage media (including, but not limited to, disk memory, CD-ROMs, optical storage media, etc.) having computer usable program code embodied thereon. The present application is described in reference to the flow diagrams and / or block diagrams of the methods, apparatus (systems), and computer program products according to embodiments of the present application. It will be understood that each block of the flow diagrams and / or block diagrams, and combinations of blocks in the flow diagrams and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processing machine, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flow diagrams and / or block diagrams block or blocks. These computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture including instructions which implement the flow diagrams and / or block diagrams flow or flows and / or block or blocks specified in the flow diagrams and / or block diagrams. The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flow diagrams and / or block diagrams flow or flows and / or block or blocks. Figure 1 one or more flow or flows and / or block or blocks Figure 1 one or more flow or flows and / or block or blocks Figure 1 one or more flow or flows and / or block or blocks Figure 1 one or more flow or flows and / or block or blocks Figure 1 one or more flow or flows and / or block or blocks Figure 1 one or more flow or flows and / or block or blocks

[0179] The above description is only the preferred embodiment of the present application, and is not intended to limit the present application in any form. Although the present application has been disclosed as above with the preferred embodiment, it is not intended to limit the present application. Therefore, any simple modification, equivalent change and modification made to the above embodiment without departing from the technical solution of the present application, according to the technical essence of the present application, should fall within the scope of protection of the technical solution of the present application.< / speak> < / break> < / volume> < / prosody> < / speak> < / break> < / prosody> < / prosody> < / speak> < / prosody> < / break> ​< / emphasis> < / break> < / prosody> < / emphasis> < / break> < / prosody> < / break> < / prosody>

Claims

1. A method for automatically generating SSML text that integrates storyboard hierarchy information, characterized by the following steps: include: Step S01. Information Structuring: Obtain the original information of the target short video and parse it. Then, structuring the information according to the script level, the scene level, and the shot level in sequence. The script level is used to store the shared information of the entire script. The scene level is built on the script level. The scene level is used to record the narrative rhythm and narrative node information between scenes. The shot level is built on the scene level. The shot level is used to associate individual shot elements and voice-over text. Step S02. Multi-source input feature acquisition: Based on the information structuring results of step S01, multi-source input features are acquired. The multi-source input features include voice-over text, associated character features, associated scene features, and associated shot features. Step S03. SSML Generation: The multi-source input features are input into a pre-trained SSML generation model. The SSML generation model encodes the multi-source input features and then fuses them to obtain fused features. The fused features are then obtained through a Transformer decoder to obtain an SSML label sequence. Each element in the SSML label sequence includes the SSML label type and corresponding parameters. The SSML label sequence is then concatenated according to the hierarchical relationship to form a complete SSML text output. In step S03, the SSML generation model includes an input layer, a sub-encoder layer, a fusion layer, and a decoder layer connected in sequence. The input layer receives multi-source input features, the sub-encoder layer encodes the multi-source input features separately, the fusion layer fuses the encoding results output by each encoder in the sub-encoder layer to obtain fused features, and the decoder layer decodes the fused features output by the fusion layer to output SSML text. The sub-encoder layer includes a text sub-encoder, a character sub-encoder, a narrative node sub-encoder, and a scene semantic sub-encoder. The text sub-encoder encodes the voice-over text into a text vector, the character speech evolution features into a speech parameter vector, the scene narrative features into a narrative vector by encoding the scene rhythm and node adjustment rules into a narrative vector, and the scene semantic vector into a scene semantic vector by encoding duration, semantic weight, and scene elements into a scene semantic vector. The fusion layer weights and fuses the text vector, speech parameter vector, narrative vector, and scene semantic vector to obtain a fused vector. The text vector, speech parameter vector, narrative vector, and storyboard semantic vector are weighted and fused to obtain the fused vector. During the process, when the pace of the scene-by-scene narrative exceeds a preset threshold, the weight of the narrative vector is increased and the weight of the text vector is decreased. When the node type is a conflict point, the weight of the narrative vector is increased and the weight of the scene semantic vector is decreased.

2. The method for automatically generating SSML text by fusing storyboard hierarchy information according to claim 1, characterized in that, In step S01, when structuring information according to three levels—script, act, and storyboard—the script level includes global information, a set of characters, and a set of acts. In the set of acts, the act features, narrative node information, and associated storyboard sets of each act are defined to construct the act level based on the script level. The act features include narrative rhythm. In the set of storyboards, the storyboard features and associated information are defined to construct the storyboard level based on the act level. The associated information includes voice-over text, visual elements, and associated character features, associated act features, and associated storyboard features.

3. The method for automatically generating SSML text by fusing storyboard hierarchy information according to claim 2, characterized in that, The global information includes any one or more of the following: video title, narrative type, and voice style. The character set includes character features and character voice evolution features. The character features include any multiple of the following: character identifier, character type, character age, character personality, character feature description, comprehensive character description, and character background story. The character voice evolution features include voice style evolution features and voice parameter adjustment coefficients for different scenes. The voice parameter adjustment coefficients include pitch offset feature pitchOffset and speech rate coefficient rateCoeff. The scene features also include scene ID. The narrative node information includes node type and parameter adjustment rules. The scene features include scene ID, scene duration, and semantic weight.

4. The method for automatically generating SSML text by fusing storyboard hierarchy information according to claim 3, characterized in that, In step S01, when structuring information according to the script level, act level, and storyboard level, the script level provides the global story background, character library, and act library, and defines the basic voice parameters of the characters. The act level serves as an intermediate layer to integrate the style benchmarks of its respective storyboards. The act level adjusts the coefficients of the voice parameters to realize the migration of character style as the narrative progresses and constrains semantic weights through the narrative rhythm. The storyboard level serves as the final carrier layer, referencing information from the upper layer to make the parameters related. The character field of the storyboard references the character object in the character set to pass the character's attributes to the storyboard level. The attributes in the act level provide a unified style benchmark for the respective storyboards.

5. The method for automatically generating SSML text by fusing storyboard hierarchy information according to claim 1, characterized in that, In step S03, the Transformer decoder uses the obtained fusion vector... Generate an SSML tag sequence, which includes intonation control tags and pause time tags. The intonation control tags include any one or more intonation control parameters such as speech rate, pitch, and volume. The pause time tags are used to control and guide the pause time corresponding to the output scene of the speech synthesis engine.

6. The method for automatically generating SSML text with integrated storyboard hierarchy information according to any one of claims 1 to 5, characterized in that, The SSML generative model uses the following loss function during training: in, Indicates the loss of text semantics. This indicates a loss of style at the scene level. This indicates the loss of storyboard-level parameters. Indicates the loss of storyboard-level parameters The weight, Indicates stage-by-stage style loss Weights, storyboard-level parameter loss L1 loss is used to constrain the prediction bias of storyboard parameters in order to constrain the accuracy of storyboard-level parameters, and scene-level style loss is applied. L2 loss constraint is used to ensure consistency of intra-scene parameters; Text semantic loss The calculation expression is: in, For the first Act 1 The original text semantic sentiment score of each scene. For the first Act 1 Each storyboard is converted into an SSML sequence corresponding to a sentiment score. Indicates the episode index. M This indicates the total number of scenes in the script of the target short video. Indicates the scene index. Indicates the first The number of shots contained in each segment; Storyboard-level parameter loss The calculation expression is: in For the first Act 1 The annotation parameters for each scene. For the first Act 1 Predictive parameters for each scene; Actual style loss The calculation expression is: in, For the first The mean of the prediction parameters for each segment, For the first The average value of the annotation parameters for each segment.

7. An SSML text automatic generation device that integrates storyboard hierarchy information, comprising a processor and a memory, wherein the memory is used to store computer programs, characterized in that, The processor is used to execute the computer program to perform the method as described in any one of claims 1 to 6.

8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed, it implements the method as described in any one of claims 1 to 6.

Images