An IP knowledge graph-based cross-modal content generation method and system

Abstract

The application discloses a cross-modal content generation method and system based on an IP knowledge graph, and relates to the technical field of artificial intelligence. The method comprises the following steps: analyzing an original book text, extracting characters, world views and plot elements, and constructing an IP knowledge graph; in response to a cross-modal content generation request, performing multi-task semantic analysis based on the IP knowledge graph, determining a target medium form and a corresponding knowledge subgraph; inputting the knowledge subgraph into a multi-task large model, and synchronously or on demand generating multi-modal content adapted to different medium forms through a shared encoding layer and a task-specific decoding branch. The application solves the technical problems of difficult consistency maintenance and low generation efficiency in cross-medium content production by constructing a unified knowledge representation cornerstone and combining a multi-task generation architecture, and realizes high-quality, high-efficiency and multi-form derivation and generation of IP content.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, particularly to the fields of artificial intelligence, knowledge graphs, and natural language processing, and specifically to a method and system for cross-modal content generation based on IP knowledge graphs. Background Technology

[0002] Currently, in the field of IP (intellectual property) content development and derivatives, with the rapid development of artificial intelligence technology, cross-modal content generation methods and systems combining knowledge graphs and large language models have emerged. These technical solutions typically begin by analyzing the original IP text (such as novels or scripts) to extract key entity information; then, they utilize the extracted information to construct a preliminary knowledge graph structure, serving as the knowledge foundation for subsequent content generation; finally, by calling different content generation models, they generate derivative content adapted to different media formats, such as text scripts, image descriptions, or audio annotations, aiming to achieve multimodal development and automated production of IP content.

[0003] However, existing knowledge graph-based cross-modal content generation methods and systems still have significant technical shortcomings in practical applications. First, the knowledge graphs constructed by existing solutions are often fragmented or superficial, storing only isolated entity names and lacking structured representation and deep association of core IP elements (such as complex emotional entanglements between characters, the evolutionary rules of the worldview, and the causal and logical chains between key plots). This leads to deviations in the deeper narrative logic and worldview consistency of the generated content, even if it conforms to the IP setting on the surface. Second, most existing solutions adopt a "serial call" or "multi-model patchwork" architecture, that is, calling independently trained and unrelated generative models for different media formats (such as short videos and audiobooks). In this architecture, each model independently understands and generates content from the same IP source. Due to the lack of a unified knowledge representation base and collaborative generation mechanism, a "semantic gap" easily arises between the generated cross-modal content. For example, the same character may exhibit contradictory personality traits in derivative content across different media, or the same plot may present different development paths in short video scripts and comic storyboards, making it difficult to achieve true multimodal content collaboration and logical consistency. Furthermore, due to the lack of fine-grained knowledge trimming and task adaptation mechanisms tailored to media characteristics, existing solutions often require extensive manual screening and prompting of the knowledge graph when generating content for specific media, and the degree of automation and generation efficiency still need improvement.

[0004] Therefore, it is necessary to make technical improvements to existing cross-modal content generation methods and systems based on IP knowledge graphs in order to solve the above problems. Summary of the Invention

[0005] This invention aims to solve the above-mentioned problems and provides a cross-modal content generation method based on IP knowledge graphs to improve the efficiency and consistency of multimodal derivation of IP content. The technical solution is as follows:

[0006] A cross-modal content generation method based on IP knowledge graphs is characterized by the following steps:

[0007] Step S1: IP Knowledge Graph Construction Steps: Parse the original book text, extract character entities, worldview entities and key plot elements from the text, and construct a knowledge graph that uniformly represents the core ontology of the IP by identifying the semantic relationships between the entities; wherein, the semantic relationships include the participation relationship between characters and events, the location relationship between characters and scenes, and the temporal and causal relationships between events;

[0008] Step S2: Cross-modal generation request parsing step: In response to the received cross-modal content generation request, perform semantic parsing on the request to determine the target media form, and retrieve and extract a knowledge subgraph adapted to the target media form from the IP knowledge graph;

[0009] Step S3: Multi-task collaborative generation step: Input the knowledge subgraph into the trained multi-task large model. The multi-task large model includes a shared semantic encoding layer and multiple task-specific decoding branches. The shared semantic encoding layer is used to map the knowledge subgraph into a unified semantic vector. The multiple decoding branches correspond to content generation tasks of different media forms.

[0010] Step S4: Multimodal content output step: Activate one or more of the decoding branches corresponding to the target media form, and generate multimodal content adapted to different media forms synchronously or on demand based on the unified semantic vector; wherein, the multimodal content includes at least one of the following: short video scripts with visual prompts, audiobook storyboard text with emotion annotations, comic panel descriptions with image and text alignment, and peripheral product copywriting related to entities.

[0011] This invention also provides a corresponding cross-modal content generation system, including a graph construction module, an intent parsing and retrieval module, a multi-task generation engine, and a content output interface, wherein:

[0012] Knowledge graph construction module: used to parse the original book text, extract character entities, worldview entities and key plot elements from the text, and construct a knowledge graph that uniformly represents the core ontology of the IP by identifying the semantic relationships between the entities;

[0013] Intent parsing and retrieval module: used to receive cross-modal content generation requests, parse the target media form, and retrieve and extract knowledge subgraphs that are compatible with the target media form from the IP knowledge graph;

[0014] A multi-task generation engine includes a graph encoding unit, a shared semantic encoding layer, and multiple task-specific decoding branches. The graph encoding unit encodes the knowledge subgraph into graph-level vectors. The shared semantic encoding layer maps the graph-level vectors into unified semantic vectors. The multiple decoding branches generate multimodal content in corresponding media formats synchronously or on demand based on the unified semantic vectors.

[0015] Content output interface: used to convert the multimodal content output by the multi-task generation engine into a format that can be used by downstream tools, and to ensure the accurate correspondence of each modal output at the presentation level.

[0016] Beneficial effects:

[0017] High Consistency: By using a unified IP knowledge graph as the sole knowledge source, the system ensures the inherent consistency and logical self-consistency of generated content across different modalities (video, audio, image, text) in terms of character settings, plot logic, and worldview rules. This fundamentally avoids information conflicts and semantic contradictions caused by independent generation of multiple models in traditional methods. Highly Efficient Generation: Employing a multi-task, large-model architecture of "shared encoding + multi-path decoding," the system can generate content in multiple media formats in parallel with a single inference, or quickly switch and activate corresponding decoding branches as needed. This significantly improves the development efficiency of multimodal IP content derivatives and reduces redundant consumption of computing resources. Fine-Grained Controllability: Through a knowledge subgraph retrieval mechanism based on intent parsing and reward propagation pruning, the system can adaptively prune the most relevant knowledge subgraphs as generation input based on the specific needs of the user request (such as emphasizing specific characters or highlighting core plots). This makes the generation process highly controllable and interpretable, allowing users to generate the required content without complex prompting processes. Attached Figure Description

[0018] Figure 1 This is a flowchart illustrating the cross-modal content generation method based on IP knowledge graph in an embodiment of the present invention. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. However, it should be understood that the specific embodiments of this invention are only for explaining the invention and are not intended to limit the scope of protection of this invention.

[0020] Example 1:

[0021] This embodiment provides a cross-modal content generation method based on IP knowledge graph, which can be executed by a computer device configured with corresponding hardware and software.

[0022] Step S1: Detailed description of the steps involved in constructing the IP knowledge graph.

[0023] In the specific implementation of step S1, the original book text first needs to undergo deep semantic parsing. The core of this parsing process lies in using a pre-trained language model to transform unstructured natural language text into a machine-understandable vectorized representation. Currently, in the field of natural language processing, pre-trained language models mainly achieve this through self-supervised learning on massive unlabeled corpora, enabling the model to capture rich knowledge at the lexical, syntactic, and even semantic levels, thereby providing high-quality text representations for downstream tasks. Specifically, in the implementation scenario of this invention, the original book text often contains a large number of character titles, place name changes, and complex pronoun relationships. For example, in different chapters of a novel, the same protagonist may be referred to by their name, nickname, or pronouns such as "he" or "she." If entity recognition is based solely on literal matching, it will result in a large number of redundant nodes in the knowledge graph. For example, identifying "Link," "he," and "protagonist" as three separate entities would undoubtedly destroy the structured semantic expression of the knowledge graph, making it difficult to establish accurate inter-entity associations in subsequent relation extraction. To solve this problem, dependency parsing and pronoun resolution techniques must be introduced while using a pre-trained language model for encoding. Dependency parsing aims to reveal the grammatical modification relationships between words in a sentence, such as identifying which noun phrase a pronoun refers to grammatically. Reference resolution goes further, aiming to identify different expressions referring to the same real-world entity at the discourse level. Through the synergy of these two techniques, the system can accurately determine that "he" and the previously mentioned "Link" refer to the same character entity, thus merging these different expressions into a unified node during the knowledge graph construction phase, avoiding relation extraction errors caused by entity redundancy in subsequent steps.

[0024] To objectively evaluate the performance of this parsing process, precision, recall, and their harmonic mean F1 score are commonly used as evaluation metrics in this field. The F1 score is calculated as F1 = 2 × (Precision × Recall) / (Precision + Recall), where precision measures the proportion of truly positive examples among those predicted as positive, and recall measures the proportion of correctly predicted positive examples. This metric is widely used in information extraction to comprehensively evaluate the model's recognition performance; a higher F1 score indicates a better balance between accuracy and comprehensiveness.

[0025] After completing the dereference resolution and dependency parsing, it is necessary to identify and extract various entities that constitute the core ontology of the IP from the text. These include character entities, scene and prop entities in the worldview entity, and event entities in key plot elements. This entity extraction process relies on named entity recognition technology, which is essentially a sequence labeling task, that is, assigning a predefined entity category label to each token in the input text sequence. Regarding the choice of pre-trained language models, BERT and its variant ERNIE are both bidirectional encoders based on the Transformer architecture, capable of learning context-aware dynamic word vectors from the input text, effectively solving the deficiency of traditional word embedding models such as Word2Vec in handling polysemy. In particular, the ERNIE model, compared to the character or word-slice-based masking strategy used by BERT, introduces a multi-granularity knowledge modeling mechanism. Its pre-training process not only includes basic word masking but also adds phrase-level and entity-level masking strategies. From a mathematical perspective, ERNIE's explicit n-gram masking language model can be formalized as a learning objective loss function L = -logp(y|x), where x is the input text sequence, y is the sequence of masked n-gram language unit labels, and p(y|x) represents the probability of predicting y given x. Compared to continuous n-gram masking, explicit n-gram masking expands the semantic learning space from V... n (V is the vocabulary size, n is the gram length) The vocabulary size is significantly compressed to V×n, significantly improving the convergence speed and semantic learning efficiency of the pre-trained model. This means that ERNIE is trained to predict masked complete phrases or entities during the pre-training stage, thus forcing the model to learn higher-level semantic units and knowledge concepts. For example, when dealing with compound entity nouns unique to fantasy novels, such as "Star Sword" or "Dragon Clan Elders' Council," ERNIE, having seen a large number of similar phrase-level mask prediction tasks during pre-training, can better identify these continuous words as an indivisible semantic whole, rather than fragmented individual characters. This knowledge-enhanced characteristic has a significant advantage in recognizing the numerous proper nouns, complex settings, and nested relationships between entities present in IP texts. Experimental studies also show that the ERNIE-based pre-trained model can achieve an excellent F1 score of over 94% on the Chinese named entity recognition task. Therefore, this embodiment of the invention preferentially uses ERNIE as the core semantic encoder to improve the accuracy and robustness of entity recognition. Of course, those skilled in the art should understand that with the development of natural language processing technology, other pre-trained language models with similar capabilities, such as RoBERTa, can also be applied to this step, and can also achieve good entity recognition results through appropriate fine-tuning.

[0026] In the specific model fine-tuning process, a fully connected layer and a conditional random field layer are typically added above the output layer of the ERNIE encoder. The fully connected layer maps the high-dimensional semantic vector output by ERNIE to the dimensional space of entity labels, mathematically expressed as O = W·h + b, where O is the label score vector output by the fully connected layer, h is the hidden layer vector output by ERNIE, W is the weight matrix, and b is the bias term. The conditional random field layer is used to learn the transition constraint relationship between output labels, with the goal of maximizing the likelihood probability of the correct label sequence.

[0027] In a Conditional Random Field (CRF), given an input text sequence... and the corresponding label sequence The conditional probability of the model Calculated using the following formula:

[0028]

[0029] in Let be the score function for the label sequence y. Let exp represent all possible label sequences, and let exp denote an exponential function with base e; the summation of the denominator is over all possible label sequences. Normalize the probability to ensure the sum of probabilities equals 1. Score function. It consists of two parts:

[0030]

[0031] in Let N be the sequence position index (N-1, assuming there are N positions), and T be the label transition matrix. Indicates from the first Labels at each location Transfer to the +1 position label The transition score, where E is the emission score matrix. Indicates the first Each location is predicted as a label. The emission score (i.e., the corresponding value output by the fully connected layer). In the common BIO annotation system, the "I-PER" (inside the character) label must immediately follow the "B-PER" (beginning of the character) label, and cannot appear directly at the beginning of the sentence. The Conditional Random Field layer can ensure that the final predicted entity label sequence is syntactically and logically valid and optimal by modeling this transition dependency relationship between labels. After the model labels each token in the text with the corresponding entity type, the system can extract a series of discrete entity nodes from the text stream, such as "Link (character)", "Azeria (scene)", "Star Sword (item)", and "Dragon Invasion (event)".

[0032] After entity nodes are successfully identified and extracted, it is necessary to establish semantic relationships between these nodes. Otherwise, the knowledge graph will degenerate into an isolated list of entities, unable to represent the dynamic evolution logic and complex interactions of the IP world. This process of establishing relationships is called relation extraction, and its goal is to determine the specific type of relationship between two identified entities from the text. Taking the sentence "Link obtained the Star Sword at the summit of Star Peak in Azeroth" as an example, entity recognition has extracted "Link," "Azeroth," "Summit of Star Peak," and "Star Sword," but the system still needs to further determine what kind of relationship exists between these entities. Relation extraction models are usually based on attention mechanisms and bidirectional long short-term memory networks. Building upon the context-aware word vectors provided by ERNIE, they further enhance the understanding of sentence structure and the ability to focus on key information. Specifically, a sentence containing two entities can be input into the relation extraction model. The model captures long-distance dependencies between entities through a bidirectional long short-term memory network and uses an attention mechanism to dynamically calculate the contribution weight of different words in the sentence to judging the current entity relationship, thereby focusing on the words that best reflect the semantics of the relationship, such as the word "obtained" in this example. The mathematical expression of the attention mechanism can be formalized as follows:

[0033]

[0034] Where: Q is the query matrix formed by stacking the representations of all entities to be queried, K is the key matrix, and V is the value matrix, which is composed of the vector representations of all words in the sentence containing the entity, with each row corresponding to the semantic encoding of a word; by calculating the dot product of Q and K and normalizing it, the model can dynamically focus on the words in the sentence that are most relevant to the relationship (such as 'obtain'), and thus accurately predict the relationship type between the subject and the target entity (such as 'Star Sword'); The dimension of the key is used as a scaling factor to avoid excessively large inner products that could cause softmax gradient saturation. Softmax normalizes the matrix on each row to obtain the attention weight matrix. The query matrix Q is multiplied by the transpose of the key matrix K to obtain the attention score matrix, where the superscript T indicates the matrix transpose operation. The trained relation classifier then maps this semantic representation to predefined relation categories, for example, determining that there is a "possess" or "acquire" relationship between "Link" and "Star Sword," and a "located" relationship between "Link" and "Peak of Stars." For Chinese relation extraction tasks, using pre-trained models such as ERNIE to transform text information into dynamic word vectors and combining them with attention mechanisms to enhance the semantic representation of the sentences containing entities has been proven to effectively improve relation prediction performance. Furthermore, end-to-end relation extraction methods have gradually emerged in recent years. For example, the REBEL model models the relation extraction task as a sequence-to-sequence generation task, directly outputting a sequence of triples containing head entities, tail entities, and their relations through generative pre-trained models such as BART, which is formalized as:

[0035]

[0036] Where t is the time step index of the generation step. This represents the output token generated in step t. The output token sequence generated in step t-1, Given a sequence of text as input, Π represents the multiplication symbol, and the entire formula represents the expression for the given input... and historical output Under the given conditions, the probability product of the target sequence y is generated step by step. This end-to-end method avoids the error propagation problem in traditional pipeline models and achieves excellent results on various relation extraction benchmark datasets.

[0037] For complex logical relationships between event entities, including temporal relationships (such as "event A occurs and then event B follows") and causal relationships (such as "event C leads to event D"), it is often necessary to process them in conjunction with text-level contextual information. The system can analyze time adverbs, conjunctions, and the chronological order of events in multiple sentences, and use methods based on graph neural networks or dependency parsing to construct temporal and causal edges between event nodes. For example, after identifying the logical chain "Because of the dragon invasion, Link must go to find the Star Sword" in the text, the system establishes a "cause" relationship between the "dragon invasion" event node and the "find the Star Sword" event node, and a temporal relationship between "find the Star Sword" and "obtain the Star Sword". The relation propagation mechanism based on graph neural networks can be formalized as a node update equation:

[0038]

[0039] in Represents a node In the Hidden layer representation at the next iteration, superscript This is the index for the number of iteration steps (not an exponential power). For nodes The set of neighboring nodes (excluding the node itself). Indexing neighboring nodes, For connecting nodes and The eigenvectors of the edges, This equation updates the representation of the current node by aggregating information from neighboring nodes, effectively capturing long-distance dependencies across sentences and improving the performance of relation extraction.

[0040] Ultimately, through the two core steps of entity recognition and relation extraction, the fragmented information in the original book text is systematically transformed into a structured semantic network composed of nodes (roles, scenes, props, events) and edges (participation relationships, location relationships, temporal relationships, causal relationships). This knowledge graph not only stores "what is in the IP world," but more importantly, it records "what happened" and "why it happened," thus providing a logically consistent and deeply narrative knowledge foundation for subsequent multimodal content generation. For example, when a subsequent step requires generating a short video script, the system can accurately extract the most dramatic subgraph based on the causal chains and character relationships stored in this knowledge graph. This capability depends entirely on the depth and structure of the knowledge graph constructed in step S1.

[0041] Step S2: Expanded description of the cross-modal request parsing step

[0042] In the specific implementation of step S2, the user's cross-modal content generation request, expressed in natural language, needs to be transformed into a query instruction that the system can execute. Furthermore, the system needs to accurately retrieve the knowledge subgraph related to the target media form and core plot from the constructed IP knowledge graph. This process is essentially a semantic parsing and retrieval task from unstructured intent to structured query; its accuracy and efficiency directly determine the upper limit of the quality of subsequent multimodal generation. Taking the request input by the operator, "Please generate a short video script for 'Star Era' for Douyin, highlighting the protagonist Link's moment of obtaining the Star Sword," as an example, the system first needs to perform deep semantic parsing on this natural language string to extract two key information dimensions: the target media form and core knowledge points. The identification of the target media form can be formalized as a text classification task. Let the input request be a string sequence Q={q1, q2, ..., q}. nThe code is encoded using a pre-trained language model and then fed into a classification layer, outputting a probability distribution P(m|Q), where m belongs to a predefined set of media formats M = {short video script, audiobook panel, comic strip panel, related text, ...}. The category corresponding to the highest probability is taken as the parsing result. In this example, the classifier should output "short video script" as the target media format.

[0043] However, simply identifying the media format is far from sufficient to support accurate knowledge subgraph retrieval. The system also needs to extract the core entities and relationships driving the graph query from the request. This process involves the reapplication of named entity recognition and relationship extraction, but the input here is not the original book text, but a short text request generated by the user. Since user requests usually contain highly generalized expressions, such as abstract concepts like "highlight moment," relying solely on keyword matching often fails to accurately anchor specific nodes in the graph. Taking "obtaining the Star Sword" as an example, the corresponding event node name in the IP knowledge graph might be "acquiring the Star Sword" or "the Star Sword recognizes its owner," and this event node is connected to the "Link" node through a "participant" relationship edge. If the system only uses a string-based retrieval method, searching for "obtaining the Star Sword" as a literal keyword in the graph, it is highly likely that the correct node will not be hit due to the difference in expression, resulting in a zero retrieval recall rate. To solve this problem, a semantic retrieval mechanism must be introduced to semantically align the entity mentions in the user request with the normalized entity nodes in the graph. This process, known in academic research as entity linking or referential resolution, is based on the core idea of ​​calculating the similarity between the semantic representation of entity mentions in a user request and the semantic representation of candidate entity nodes in the graph, within a vector space. Let the user request contain entity mentions... The vector representation is obtained after ERNIE encoding: Candidate entity nodes in the graph The corresponding text descriptions (such as entity names, attributes, and related event summaries) are encoded into vector representations: The goal of entity linking is to select elements that make the similarity function... The candidate entities that are maximized are used as the matching results. The similarity function usually uses cosine similarity, and the calculation formula is:

[0044]

[0045] This mechanism ensures that even if the expression used in the request does not exactly match the normalized name in the graph, the system can still achieve accurate alignment through semantic space similarity. For example, although "get the Star Sword" and "acquire the Star Sword" do not match exactly literally, in the semantic space constructed by the pre-trained language model, the vector representations of these two phrases should have a high cosine similarity, thus enabling them to correctly match the target event node.

[0046] After identifying the target media form and semantically linking core entities and events, the system needs to extract a local knowledge subgraph from the global IP knowledge graph based on the parsing results, which serves as input for the subsequent multi-task generation engine. This extraction process is not a simple fixed-depth traversal, but rather requires adaptive subgraph extraction based on the characteristics of the target media form. From an information theory perspective, this process can be viewed as an information source, implicitly representing a need for the set of information required for the generation task. Let R be the set of relevant facts required to complete the generation task in an ideal scenario, and C be the contextual information ultimately retrieved and input into the generation model. According to the basic principles of context engineering, C can be decomposed into three components: relevant facts R that determine the answer Y, redundant information S that is statistically independent of Y, and noisy factual information N that is negatively correlated with or even contradictory to Y. The core objective of context engineering is to maximize the mutual information I(Y;C) between C and the target answer Y, while minimizing the information content of S and N in C. The mathematical definition of mutual information is I(Y; C) = H(Y) - H(Y|C), where H(Y) represents the information entropy of Y, and H(Y|C) represents the conditional entropy of Y given C. By maximizing mutual information, the system can ensure that the context provided to the generative model contains as much decision-related information as possible, while filtering out irrelevant noise. This theoretical framework provides an optimization objective and evaluation criterion for the pruning of knowledge subgraphs—the system should retrieve nodes and edges from the graph that are directly or indirectly related to core entities and events and contribute to the generation of target media content, while excluding redundant information that exists in the graph but is irrelevant to the generation task.

[0047] In practical implementation, knowledge subgraph retrieval and pruning typically employ a multi-stage strategy. The first stage is based on the expansion of the basic subgraph of core entities. That is, starting from the identified core entity nodes, a breadth-first traversal with a limited depth is performed along the relation edges to obtain adjacent nodes and relation edges within a one- or two-hop range.

[0048] Let the graph be G=(V,ε), where V is the set of nodes. G is the set of relation edges; G is the original graph;

[0049] Given a set of core entity nodes:

[0050] Define the set of newly added neighbors at the t-th hop as:

[0051] ;

[0052] and, ;

[0053] Define the set of newly added neighbors at the T-th hop as:

[0054] ;

[0055] The subgraph edge set is defined as:

[0056] .

[0057] This breadth-first traversal-based expansion method can quickly obtain surrounding information directly related to the core entity. For example, a one-hop traversal starting from the "Link" node can obtain the scene node "Star Peak", the item node "Star Sword", the event node "Acquire Star Sword", and the enemy character node "Shadow Guardian". A two-hop traversal can further obtain detailed information such as the geographical attributes of "Star Peak" and the racial attributes of "Shadow Guardian". However, relying solely on fixed-depth traversal has obvious drawbacks. When the graph structure is dense, the number of nodes may increase exponentially with the number of hops, leading to a rapid expansion of the subgraph and the introduction of a large amount of noise. When the graph structure is sparse, fixed-depth traversal may miss nodes that are far away but crucial to generation. This phenomenon is known as the "graph explosion" problem in academic research.

[0058] To address this issue, the second stage requires introducing a graph pruning algorithm based on reward propagation to select the nodes and edges that contribute most to the current generation task from the base subgraph. An effective implementation is a variant of the Prize-Collecting Steiner Tree algorithm. The core idea of ​​this algorithm is to find a subtree connecting all "important" nodes in the graph such that the sum of the subtree's total weight (edge ​​cost) and the penalty for not containing any nodes (node ​​reward) is minimized. Formally, given an undirected graph G = (V, E), where V is the set of nodes, E is the set of edges, and each node has a reward function p : ; indicates a node Importance weights;

[0059] in, surface p Let it be a non-negative real number;

[0060] Each edge has a cost function: The goal of the Prize-Collecting Steiner Tree (PCST) problem is to find a connected subgraph: G′=(V′,E′)⊆G, where E′ is the set of edges contained in the subgraph G′; minimizing the objective function.

[0061]

[0062] The first term represents the total reward lost by nodes that are not included in subgraph G′; e is each edge contained in the subgraph, and c(e) is the cost of edge e.

[0063] By minimizing this objective function, the algorithm seeks a balance between including high-reward nodes and avoiding high-cost edges. In specific applications of knowledge subgraph pruning, the assignment of node rewards p(v) is crucial. It can be allocated based on the semantic relevance of core entities in the user request: the highest reward value is assigned to core entity nodes identified in step one (such as "Link" and "Star Sword"); the second highest reward value is assigned to nodes directly related to the core event within a one-hop neighborhood (such as the event node "Obtain Star Sword" and the scene node "Peak of Stars"); a lower reward value is assigned to nodes providing background information within a two-hop neighborhood (such as the attribute description node of "Shadow Guard"); and zero reward is assigned to completely irrelevant nodes. The edge cost c(e) can be set according to the importance of the relation type; for example, a lower cost is assigned to "participant" relation edges to promote inclusion, a medium cost to "located" relation edges, and a higher cost to "has attributes" relation edges to control the subgraph size. By solving the above optimization problem, the system can automatically prune a moderately sized, information-dense simplified subgraph G_pruned from the basic subgraph. This subgraph contains the core facts required to complete the generation task and effectively filters out redundant and noisy information.

[0064] Compared to the PCST algorithm, other possible subgraph extraction methods, such as fixed-depth traversal or PageRank-based node ranking, have significant shortcomings. While fixed-depth traversal is simple to implement, it cannot differentiate nodes based on their actual contributions, easily introducing a large number of low-value nodes in dense areas leading to context overload, or missing high-value nodes in sparse areas, resulting in insufficient information. PageRank-based methods can identify globally important nodes, but their importance definition is weakly correlated with the current user request. For example, "Azeria" as a core world-building scene may have a high PageRank value in the overall graph, but in the specific event of "Link obtaining the Starsword," the specific scene of "Star Peak" is far more important than the continent itself. In contrast, the PCST algorithm, through its reward function design, can directly quantify the semantic relevance of the user request into a numerical reward for nodes, thereby achieving query-aware personalized subgraph pruning. Experimental studies show that in complex question-answering tasks, using the PCST algorithm for subgraph pruning combined with fine-tuning of a large language model can achieve more than twice the accuracy of benchmark methods on the Hits@1 metric.

[0065] In practical engineering implementations, the retrieval and pruning of knowledge subgraphs are typically accomplished using the query language of graph databases. Taking the widely used Neo4j graph database as an example, its Cypher query language supports declarative graph pattern matching. For scenarios where core entity nodes have been identified, the following Cypher query can be constructed to obtain a basic subgraph centered on the core node with a limited depth of 2:

[0066] "MATCH path = (start:Entity{name:'Link})-[1..2]-(connected)

[0067] WHERE ANY(node ​​IN nodes(path) WHERE node.name IN ['Star Sword', 'Obtain Star Sword', 'Summit of Stars', 'Shadow Guardian'])

[0068] RETURN path

[0069] The query starts with the "Link" node and traverses along any relation edge in one to two hops. The WHERE clause filters the path to ensure it contains at least one core related node, effectively controlling the semantic relevance of the returned subgraph. For scenarios requiring PCST pruning, graph algorithms provided by graph data science libraries (such as Neo4j Graph Data Science Library) can be used to project the basic subgraph into a memory graph and then call the Steiner Tree algorithm for pruning. The resulting pruned subgraph G_pruned is serialized into structured data containing a list of nodes (with attributes such as node name, type, and description) and a list of edges (with attributes such as relation type and direction), serving as input to the multi-task generation engine in step three.

[0070] Through the multi-stage processing of intent parsing, entity linking, basic subgraph expansion, and reward propagation pruning described above, step S2 achieves a precise conversion from the user's natural language request to a structured knowledge subgraph. This process ensures that the contextual information obtained by the subsequent generation engine not only contains the core facts required to complete the generation task but also maintains high information density and low noise, laying a solid foundation for generating high-quality, highly consistent cross-modal content. Compared with directly concatenating the user request and the original text and inputting it into the generation model, this step fundamentally solves the problems of information dilution in long text contexts and fact omissions in multi-hop reasoning through structured querying and optimized pruning of the knowledge graph, demonstrating substantial innovation in information retrieval and context engineering. If the user request implicitly or explicitly requires the generation of content in multiple media formats, the intent parsing and retrieval module will integrate the needs of each target media format and extract a unified, multi-dimensional fusion knowledge subgraph by maximizing the coverage of core entities and relationships to ensure the consistency of the input sources of subsequent multi-path decoders.

[0071] Step S3: Expanded description of the multi-task collaborative generation steps

[0072] In the specific implementation of step S3, the structured knowledge subgraph obtained in step S2 needs to be transformed into an input form that can be processed by a multi-task large model. Through a collaborative mechanism of shared encoding and multi-path decoding, parallel or on-demand generation of multimodal content from a unified knowledge source is achieved. The core of this process lies in solving the semantic gap problem caused by traditional serial calls to multiple models. When the same IP knowledge needs to simultaneously generate short video scripts, audiobook storyboards, and comic panel layouts, if independently trained generative models are called separately, each model will interpret the source knowledge independently from its own perspective. Although the generated text may superficially conform to the input prompts, irreconcilable internal contradictions often arise in deeper dimensions such as narrative rhythm, emotional tone, and visual style. To solve this problem, step S3 adopts a multi-task large model based on the Transformer architecture. Its core design idea is to force all downstream generation tasks to be built on the same semantic understanding foundation through a shared semantic encoding layer, thereby fundamentally ensuring the logical consistency and stylistic coordination of cross-modal content.

[0073] Before inputting the knowledge subgraph into the model, it needs to be vectorized. The knowledge subgraph G_pruned contains a set of nodes V and a set of edges E. Each node v∈V is associated with attribute information such as entity name, entity type, and text description, and each edge e∈E is associated with relation type and direction information. This structured data cannot be directly used as input to a sequence model and needs to be transformed into a continuous vector representation. An effective implementation is to use a graph neural network to encode the knowledge subgraph and aggregate neighborhood information through a multi-round message passing mechanism to obtain the hidden layer representation of each node. Let the nodes... In the The hidden layer vector of the layer is Its update process can be formalized as follows:

[0074]

[0075] in, For nodes The set of neighboring nodes; Connecting nodes eigenvectors of the edges; The aggregation function is a weighted summation in the attention network shown in the figure; after L layers of graph convolution, each node ultimately represents:

[0076] Where V is the set of all nodes along the path.

[0077] Next, the graph pooling operation merges the node-level representations into the entire graph-level representation. This process can be expressed by the following formula:

[0078]

[0079] in, For graph pooling functions, common pooling methods include average pooling, maximization, and attention-based weighted summation. Taking attention pooling as an example, the calculation formula for the pooling operation is:

[0080] ;

[0081] Among them, attention weight The calculation is as follows:

[0082]

[0083] in, It is a learnable attention parameter vector;

[0084] It is a node The hidden layer representation in the final layer L.

[0085] The final result This is the overall vector representation of the knowledge subgraph, which encapsulates the complete semantic information contained within that subgraph—for example, in this case, The code should encode "Link", "Star Sword", "Acquisition" event, "Peak of Stars" scenario, "Shadow Guard" antagonist, and the complex relationships between them.

[0086] In this method, the graph neural network continuously aggregates neighbor information through a multi-layer message passing mechanism to obtain the final representation of each node. Graph pooling then encodes the entire knowledge subgraph by aggregating node representations into a full graph representation. When using attention pooling, by learning the attention weights between nodes, information from important nodes can be effectively aggregated, thereby improving the quality of the graph representation.

[0087] knowledge subgraph vector After inputting into the multi-task large model, the first layer is the shared semantic encoding layer. This layer is essentially a stack of multiple Transformer encoders, its function being to perform deep semantic abstraction on the input vector, extracting universal features valuable for all subsequent generation tasks. Let the shared encoding layer consist of M Transformer blocks, each containing a multi-head self-attention mechanism and a feedforward neural network. The mathematical expression of the self-attention mechanism is:

[0088]

[0089] Where: Q is the query matrix, K is the key matrix, and V is the value matrix. The dimension of the key is used as a scaling factor to avoid excessively large inner products that could cause softmax gradient saturation. Softmax normalizes the matrix on each row to obtain the attention weight matrix. This indicates that the attention score matrix is ​​obtained by multiplying the query matrix Q by the transpose of the key matrix K, and the superscript T indicates the matrix transpose operation.

[0090] For the input representation X of the m-th layer (m) First, the data passes through a self-attention sub-layer (Multi-Head Attention), and then intermediate representations are obtained through residual connections and layer normalization:

[0091] X′=LayerNorm(X (m) +MultiHeadAttention(X (m) ));

[0092] Subsequently, X′ is input into the feedforward network sublayer (FFN), and by combining residual connections and layer normalization, the output of this layer is obtained:

[0093] X (m+1)=LayerNorm(X′+ FFN(X′)).

[0094] After M layers of encoding, the shared encoder outputs the final hidden layer representation Hshared. This representation not only preserves the core semantic information of the knowledge subgraph, but also effectively extracts cross-task general features through multi-layer nonlinear transformation, thus providing a unified and expressive representation foundation for subsequent downstream tasks.

[0095] The design of the shared coding layer has a profound theoretical basis. From an information theory perspective, the core advantage of multi-task learning lies in forcing the model to learn more robust and general feature representations by jointly optimizing multiple related tasks.

[0096] Suppose there are T tasks in total, and their corresponding real data distributions are P1, P2, ..., Pn. T In single-task learning, the goal of each task is to independently learn a path from input X to output. The mapping function. When multiple tasks share a structure in the underlying semantic space, multi-task joint learning can improve overall performance by sharing representations. In this case, the model is usually composed of a shared encoding function. Task-specific decoding functions The composition, and its optimization objective, can be formalized as:

[0097]

[0098] in, Let represent the loss function for the t-th task. ≥0 represents the task weight coefficient, used to balance the contribution of different tasks to the overall goal. Compared to training the model separately for each task, the shared encoding structure can explicitly model the potential correlation between tasks, reduce model complexity while sharing parameters, thereby effectively alleviating the overfitting problem and significantly improving the generalization ability on data-scarce tasks.

[0099] Empirical studies show that models jointly trained on multiple natural language generation tasks have significantly better quality shared representations than single-task models, especially in low-resource scenarios.

[0100] After passing through the shared encoding layer, Hshared is fed in parallel or selectively into multiple task-specific decoding branches. Each decoding branch is responsible for generating content for a specific media format, such as a short video script decoder, an audiobook panel decoder, a comic panel decoder, and a related text decoder. These decoders are all based on the Transformer decoder architecture, but they are customized in terms of output format, loss function design, and training data. Taking the short video script decoder as an example, it needs to generate voice-over text and corresponding on-screen cue word sequences simultaneously, and the two need to maintain alignment on the timeline. To this end, the decoder contains two cooperating sub-modules: a text generation sub-module and an on-screen cue generation sub-module. The two sub-modules share the same input context Hshared, but are decoded through different output headers. To achieve alignment on the timeline, a synchronous generation strategy based on a cross-attention mechanism can be adopted. Let the hidden state of the text generation sub-module at step t be ht^txt, and the hidden state of the on-screen cue generation sub-module at step t be ht^img. The two interact with each other through shared positional encoding and temporal attention to ensure that the generated text fragments correspond one-to-one with the on-screen cue in semantics. The design of the audiobook scene decoder differs. It needs to insert sentiment markers into the generated text for subsequent TTS modules to control the timbre, speech rate, and pitch. To achieve this, the decoder needs to learn during training to insert special markers at appropriate contextual locations. Let the sentiment tag set be E = {neutral, happy, sad, angry, excited, ...}. At each generation step, the decoder needs to predict not only the next text token but also the sentiment tag token with a certain probability. This can be achieved by adding a sentiment classification head above the decoder's output layer, with the loss function being:

[0101] Ltts = Llm + γLemotion, where Llm is the standard language modeling loss (cross-entropy), Lemotion is the cross-entropy loss of sentiment label prediction, and γ is the balance coefficient.

[0102] Comic panel decoders face a more complex task: they need to decompose the continuous events contained in a knowledge subgraph into multiple panel units and generate compositional descriptions and character dialogue for each unit. This process is essentially a sequence-to-sequence generation task, but the output sequence has a multi-level structure.

[0103] A hierarchical decoding strategy is employed for sequence generation: first, a higher-level decoder generates the skeleton representation of the frame unit sequence; then, for each frame unit, a lower-level decoder generates its corresponding fine-grained text description. Let the number of frame units be K, and the skeleton representation of the k-th frame unit be... The t-th generated word unit in this unit is The overall generation probability can then be formalized as a hierarchical conditional decomposition:

[0104]

[0105] in, This represents the word length of the k-th frame unit. This represents the sequence of all skeleton representations preceding the k-th unit. This represents the skeleton context up to the current unit. This represents the sequence of historical lexical units generated in the current unit. This decomposition form reflects the two-level autoregressive structure in the hierarchical generation process: the higher level is responsible for modeling the global dependencies between unit images, while the lower level generates local detail descriptions under the given semantic framework of the unit, thus balancing overall structural consistency with local semantic refinement.

[0106] Training a large multi-task model is a crucial stage for system performance optimization, requiring joint learning using multi-source labeled data such as short video scripts, audiobook text, and comic descriptions. Let there be T tasks in total, and the loss function for the t-th task be Lt(θ), where θ represents the shared parameters of the model. Then, the overall training objective can be expressed as the weighted sum of the losses from each task:

[0107] ,

[0108] in ≥0 represents the weight coefficient of the t-th task, and it usually satisfies... Weight settings have a significant impact on the learning effect of multi-task tasks: if the weight of a certain task is too large, its gradient update will dominate the parameter optimization process, thereby inhibiting the learning of other tasks, leading to training imbalance or performance degradation.

[0109] To alleviate this problem, a dynamic weight adjustment strategy can be adopted to ensure that different tasks contribute relatively evenly to optimization during training. An effective method is an adaptive weighting mechanism based on the gradient norm. Let the gradient of the loss of the t-th task with respect to the shared parameters be...

[0110]

[0111] Let its norm be Therefore, the reciprocal of the gradient norm can be used as an indicator of the task's importance.

[0112]

[0113] Here, ϵ>0 is a numerical stability term used to avoid division by zero when the gradient approaches zero. Then, softmax normalization is used to obtain the dynamic task weights.

[0114]

[0115] Where: τ>0 is a temperature parameter used to control the smoothness of the weight distribution: when τ is small, the weight distribution is sharper; when τ is large, the weights tend to be more uniform.

[0116] This dynamic weighting mechanism can automatically adjust and optimize the contribution based on the current training state of each task, so that tasks with smaller gradients (slower learning progress or higher difficulty) receive higher weights, thereby balancing the convergence speed of each task, avoiding some tasks from converging too early while other tasks are still underfitting, and improving the overall stability and generalization performance of multi-task joint training.

[0117] From the perspective of gradient flow, the core challenge of sharing encoding layers in multi-task learning lies in gradient conflicts between tasks: the gradient update directions generated by different tasks for shared parameters may be inconsistent or even contradictory, making it difficult for the model to simultaneously optimize the objective functions of all tasks. Let the loss function of the t-th task be... Its gradient with respect to the shared parameter θ is:

[0118]

[0119] If weighted joint training is used, the total gradient is:

[0120] ,

[0121] in ≥0 represents the task weight coefficient.

[0122] When there exist two tasks t1 and t2 satisfying

[0123]

[0124] When the angle between two gradients is greater than 90°, it indicates a directional conflict between them in the parameter space. In this case, the weighted gradient vector... The magnitude of the gradient may be significantly smaller than the effective components of the gradient for each individual task, and may even cause the update direction to deviate from the descent direction of any task, thereby reducing the overall optimization efficiency or causing training oscillations.

[0125] To alleviate this problem, a gradient projection strategy can be employed to detect and correct conflicts in the gradients of each task before parameter updates, ensuring that the final update direction simultaneously improves multiple task objectives as much as possible. A typical method is: when a conflict is detected between two task gradients (i.e., their inner product is negative), the conflicting component of one gradient in the direction of the other gradient is removed. Specifically, for task i and task j, if... ,

[0126] Then Projected to The corrected gradient is obtained from the orthogonal complement space:

[0127]

[0128] This transformation will remove exist The negative component in the direction ensures that the corrected gradient satisfies:

[0129]

[0130] This means the update directions of the two tasks no longer conflict. After performing this operation on the gradients of all tasks sequentially, a weighted sum is then performed to obtain the final update direction:

[0131]

[0132] This projection-based gradient coordination mechanism can significantly reduce optimization interference between tasks, enabling shared representations to form more stable collaborative optimization paths across different tasks, thereby improving the convergence stability and overall performance of multi-task training.

[0133] Through the aforementioned multi-task collaborative generation mechanism, step S3 achieves parallel generation from a unified knowledge source to multimodal content. This architecture has significant advantages over traditional methods that sequentially call multiple models: First, the shared encoding layer ensures that all downstream generation tasks are built on the same set of semantic understanding, fundamentally avoiding semantic conflicts across modal content; second, multi-task joint training enables the model to learn shared features and complementary information between tasks, improving data utilization efficiency and generation quality; finally, through dynamic weight adjustment and gradient conflict resolution mechanisms, the balance problem in multi-objective optimization is effectively solved, ensuring that each task can achieve sufficient optimization. In this example, when the knowledge subgraph Hshared is simultaneously input into both the short video script decoder and the audiobook scene decoder, the sequence of visual cue words generated by the former and the sentiment-annotated text generated by the latter are semantically naturally coordinated. For example, the visual cue "Link grips the Star Sword tightly, its light illuminating the night sky" in the short video script and the sentiment annotation "Link (excitedly, his speech quickens): I finally got you!" in the audiobook scene both originate from the same set of semantic understandings of the event "obtaining the Star Sword," thus achieving an intrinsic unity between visual imagination and auditory emotion. This is precisely the core value of the multi-task collaborative generation architecture.

[0134] Step S4: Expanded description of the multimodal content output step

[0135] In the specific implementation of step S4, the decoder outputs generated by the multi-task large model in step S3 need to be transformed into multimodal content that can be directly used by downstream production tools or end users. The core challenge of this process lies in solving the internal alignment problem of the generated content. For short video scripts, it is necessary to ensure that the voice-over text and on-screen cues are precisely synchronized on the timeline; for audiobook storyboards, it is necessary to ensure that the emotional annotations and the narrative context are semantically matched so that the subsequent TTS module can generate audio with emotional fluctuations; for comic panel layouts, it is necessary to ensure that the composition descriptions and character dialogues correspond one-to-one at the screen unit level. If there are deviations in the internal alignment of the generated content, such as the visual elements described by the on-screen cues being out of sync with the plot described by the voice-over text, or the voice emotions indicated by the emotional annotations contradicting the semantics of the text, it will result in a noticeable sense of disjointedness in the final multimodal work, seriously affecting the user experience. To solve this problem, step S4 introduces a collaborative generation and alignment mechanism at the decoder design level to ensure that the outputs of each modality are not only unified at the knowledge source, but also achieve precise correspondence at the presentation level.

[0136] When a user requests a short video script as the media format, the system activates the "Short Video Script Decoder." This decoder contains two cooperating sub-modules: a text generation sub-module and a visual cue generation sub-module. These two sub-modules do not operate independently but achieve information interaction and timeline alignment through a shared attention mechanism. The theoretical basis for this design lies in alignment modeling in sequence-to-sequence generation tasks; its core idea is to transform the alignment problem into the calculation of the implicit alignment probability between two sequences. Let the voice-over text sequence generated by the text generation sub-module be Ytxt = {y1, y2,..., y...}. T The prompt word sequence generated by the on-screen prompt generation submodule is Yimg = {z1, z2, ..., z}. S The two need to satisfy an alignment relationship in terms of temporal semantics: each text segment y T Respond to one or more visual cues z S Furthermore, the visual elements described by these cues should be consistent with the plot semantics of the text fragments. To achieve cross-modal temporal alignment, the decoder introduces an attention-based alignment model, which establishes a probability distribution for all visual cue positions s given a text position t. Its formal expression is:

[0137]

[0138] in: Let t be the hidden state of the text generation submodule at step t.

[0139] This represents the hidden layer representation of the screen prompt generation submodule at step s (to avoid symbol confusion, it is distinguished from the text hidden state and denoted as...). `score(⋅,⋅)` is the alignment score function, used to measure the semantic matching degree between the text state and the visual cue state. By maximizing the alignment probability, the model learns the inherent correspondence between text descriptions and visual cues during training. In the generation phase, the two sub-modules adopt a beam search-based collaborative decoding strategy: for each text segment generated by the text generation sub-module, the visual cue generation sub-module predicts the corresponding visual cue based on the current hidden state and feeds back the semantic information of the visual cue to the text generation sub-module through a cross-attention mechanism to correct the generation direction of subsequent text. This bidirectional information interaction ensures that the final output voice-over text and the visual cue word sequence are naturally aligned semantically and temporally.

[0140] Taking the short video script generated in this example as an example, the voice-over text output by the text generation submodule is "At the summit of Star Peak, Link finally grasped the legendary Star Sword. The light from the sword instantly illuminated the entire night sky, and the Shadow Guard vanished in the light." Simultaneously, the structured cue word sequence generated in parallel by the on-screen cue generation submodule is: [Scene: The summit of Star Peak, night, rotating stars]; [Character: Link, resolute expression, reaching out]; [Item: Star Sword, emitting a dazzling blue light]; [Action: The Shadow Guard dissipates after contact with the light]. This cue word sequence is not randomly arranged, but rather arranged according to the chronological order of the plot development, corresponding to the narrative rhythm of the voice-over text: when the text describes "The summit of Star Peak," the scene cue in the cue word sequence is activated; when the text describes "Link finally grasped it," the character cue is activated; when the text describes "the light from the sword," the item cue is activated; and when the text describes "the Shadow Guard vanishes in the light," the action cue is activated. This precise alignment on the timeline enables subsequent text-based video tools to generate visual images that are highly synchronized with the voice-over text based on the sequence of prompts, thus avoiding the audio-visual asynchrony problem caused by the independent generation of text and images in traditional methods.

[0141] When a user needs to generate an audiobook preview simultaneously, the system can activate the "audiobook decoder" in the same processing round. The core innovation of this decoder lies in the introduction of an automatic sentiment annotation insertion mechanism, inspired by the need for prosody and emotion control in speech synthesis. Traditional sentiment-based speech synthesis typically employs style embedding or global sentiment labeling methods, treating emotion as a holistic attribute of the entire sentence rather than something dynamically changing within it. However, in narrative texts, a character's emotions often change in real-time as the plot unfolds. For example, in this case, Link initially shows excitement upon obtaining the Starsword, but then reveals sadness upon seeing the Shadow Guard disappear. Using global sentiment labels would only assign a single emotion to the entire text, failing to capture this emotional shift and resulting in monotonous audio. To address this issue, the audiobook decoder, during text generation, performs real-time sentiment prediction for each generated text segment based on the hidden layer representation Hshared output from the shared semantic coding layer in step three. Let the word generated in step t be yt. While predicting yt, the decoder also needs to predict the sentiment label et ∈ E with a certain probability, where E is a predefined set of sentiment labels (such as neutral, happy, sad, angry, excited, etc.). The input to the sentiment predictor is the hidden state ht of the current step, and the output is the probability distribution of the sentiment label P(et|ht) = softmax(We ht + be), where We is the weight matrix for sentiment classification, and be is the bias term. During training, the prediction loss of sentiment labels and the language modeling loss are jointly optimized: Lcombined = Llm + λ Lemotion, where Llm is the cross-entropy language modeling loss, Lemotion is the cross-entropy loss for sentiment label prediction, and λ is the balance coefficient. Through this joint training method, the model learns the ability to automatically insert sentiment labels at appropriate contextual positions. During the generation phase, the decoder not only outputs the text sequence but also embeds sentiment labels into the text, for example, "Link (excitedly, speaking quickly): I finally found you!" <sadness> As he watched the guards disappear, a trace of sadness returned to his eyes.< / sadness> 。。The phrases “(excitedly, speaking quickly)” and “ <sadness>"" refers to sentiment markers. The former uses natural language description and is suitable for scripts that require human reading; the latter uses markers and is easy for the subsequent TTS module to parse automatically.

[0142] From an information theory perspective, the insertion of sentiment annotations essentially enhances the conditional information in the speech generation process, reducing uncertainty during speech synthesis. Let the original text sequence be X, and the acoustic feature sequence to be generated be Y. The goal of speech synthesis is to sample from P(Y|X) so that the generated speech conforms to the semantics and emotional content of the text. However, the distribution of P(Y|X) often has high entropy, meaning that given the same text, there can be multiple different speech expressions (different intonations, different emotions). The introduction of the sentiment annotation E transforms the conditional probability into P(Y|X, E). By introducing additional conditional information E, it effectively reduces the conditional entropy H(Y|X, E) < H(Y|X), thus making the generated speech more deterministically consistent with the expected emotional expression. This principle is similar to style control methods in the field of speech synthesis, the core of which lies in guiding the speech generation process through explicit sentiment annotations, avoiding the difficulty of the model implicitly inferring emotion from the text.

[0143] For subsequent TTS modules, the parsing of sentiment markers is a crucial step. Traditional TTS models typically only take text as input and output corresponding acoustic features, failing to adjust speech performance based on emotional changes. To achieve emotion-controlled speech synthesis, a plug-in approach based on Low-Rank Adaptation (LoRA) can be used, inserting sentiment control as a LoRA plug-in into different modules of a pre-trained neutral speech model. The core idea of ​​LoRA is to learn a low-rank adaptation matrix for a specific task without changing the original weights of the pre-trained model. Let the weight matrix of the pre-trained model be W0∈R. d×k LoRA introduces two low-rank matrices A∈R d×r and B∈R r×k The forward propagation process is modified to h = W0x + BAx, where r << min(d,k), and BA is the product of the low-rank adaptation matrices. During training, W0 is frozen and not updated; only A and B are updated. This design allows training for different emotions to learn different low-rank adaptation matrices. During inference, the corresponding LoRA plugin for each emotion can be loaded to switch emotional styles. For example, for "excitement," A_excited and B_excited are loaded; for "sad," A_sad and B_sad are loaded. When parsing emotion markers, the TTS module identifies the emotion tag corresponding to the current text segment and dynamically switches or merges the corresponding LoRA plugin, thus achieving a smooth transition of emotions when generating speech.

[0144] When the system simultaneously activates the short video script decoder and the audiobook scene decoder, it is necessary to address the resource allocation and output synchronization issues when multiple decoders are running in parallel. From a systems engineering perspective, an asynchronous streaming architecture can be adopted, designing the multi-task generation engine as a service that supports multiple concurrent outputs. Each decoder runs in an independent computation stream, sharing the underlying attention key-value cache to avoid redundant computation. Let the hidden layer representation H_shared of the shared encoding layer output be broadcast to all activated decoders, and each decoder independently performs autoregressive generation. When output synchronization is required (e.g., the text and visual cues in the short video script need to be aligned on the timeline), decoders synchronize their states through a cross-attention mechanism; when output is asynchronous (e.g., strict alignment is not required between the short video script and the audiobook scene), decoders run completely independently. This design allows the system to flexibly adapt to different application scenarios—when a user needs to generate multiple media contents simultaneously for the same IP event, the system can output the generation results of multiple modalities at once; when a user only needs a single media content, the system only activates the corresponding decoder, avoiding unnecessary computational overhead.

[0145] Compared to traditional multi-model serial invocation methods, the collaborative generation and alignment mechanism employed in step S4 has significant advantages. In traditional methods, short video script generation and audiobook storyboard generation are typically completed by two independent models, each using only the same text prompts as input. Due to the lack of a unified semantic representation at the underlying level and an alignment mechanism at the output layer, the generated text, while consistent in macro-plot, often deviates in micro-details (such as character tone, scene composition, and emotional shifts). For example, the visual prompt in the short video script when Link obtains the Starsword might emphasize the "dazzling blue light" of the sword, while the corresponding emotional annotation in the audiobook storyboard might only be labeled as "excitement," lacking an inherent semantic connection between the two. In this invention, however, the generation of both decoders is based on the H_shared output of the same shared encoding layer. This vector encapsulates the complete semantics of the event "Link obtains the Starsword" in the knowledge subgraph, including multi-dimensional information such as character emotions, scene atmosphere, and plot tension. Therefore, the sequence of on-screen cues generated by the short video script decoder and the sentiment-annotated text generated by the audiobook scene decoder naturally possess semantic synergy—the former's emphasis on "dazzling blue light" and the latter's annotation of "excitement" both stem from a unified understanding of the "radiance" and "emotional climax" inherent in the "Star Sword" within the same event. This inherent semantic synergy is something that traditional serial multi-model invocation methods cannot achieve.

[0146] Finally, through the multimodal content output mechanism in step S4, the system transforms the intermediate representation generated in step three into a final product usable downstream. The short video script output includes aligned voice-over text and on-screen prompts, which can be directly input into a video generation tool to create a short video; the audiobook storyboard output includes emotionally annotated text, which can be directly input into an emotional TTS module to generate an audiobook with rich emotional nuances; the comic panel output includes structured data on composition descriptions and character dialogue, which can be directly input into a comic generation tool to create a comic; and the peripheral copywriting output can be directly used for product design and promotion. This output process ensures that, starting from a unified IP knowledge source, the resulting multimodal content achieves a high degree of synergy and internal consistency across multiple dimensions, including macro-plot, micro-details, emotional tone, and visual style, thereby achieving the core objective of this invention.

[0147] In summary, the cross-modal content generation method based on IP knowledge graphs provided in this embodiment solves the technical challenges of fragmented and shallow knowledge representation in multimodal content generation from the source by constructing a structured knowledge graph centered on the core IP ontology. Through a knowledge subgraph retrieval mechanism based on intent parsing and reward propagation pruning, it achieves information refinement and noise filtering from user requests to the generation context, effectively addressing the dilemmas of context overload and information omission in traditional methods. By introducing a multi-task large-model architecture of shared encoding and multi-path decoding, and forcing all downstream tasks to establish a unified semantic understanding base at the shared encoding layer, it fundamentally overcomes the semantic gap problem of cross-modal content caused by serially calling multiple models. By introducing an alignment model based on attention mechanisms and an automatic sentiment annotation insertion mechanism at the output stage, it ensures the timeline synchronization of voice-over text and on-screen prompts in short video scripts, and the semantic matching of sentiment annotations and narrative context in audiobook storyboards, achieving accurate correspondence of multimodal content at the presentation level. This method, through the aforementioned interconnected technical means, constructs a complete technical solution from knowledge representation, information retrieval, semantic understanding to content generation and alignment. It significantly improves the consistency, collaboration, and generation efficiency of cross-modal content generation, and has outstanding substantive features and significant technological progress.

[0148] Example 2:

[0149] Corresponding to the above method embodiments, this embodiment provides a cross-modal content generation system based on IP knowledge graphs. The system includes a graph construction module, an intent parsing and retrieval module, a multi-task generation engine, and a content output interface. These modules work collaboratively to achieve integrated automatic generation from original book text to multimodal derived content. The structure, function, working process, and technical effects of each module are described in detail below.

[0150] Mapping module

[0151] The knowledge graph construction module corresponds to step S1 in the method embodiment. Its core function is to extract structured elements from the original book text and construct a knowledge graph that uniformly represents the core ontology of the IP. This module includes a text preprocessing unit, an entity recognition unit, a relation extraction unit, and a knowledge graph storage unit.

[0152] The text preprocessing unit first cleans, segments, and words the input raw book text, and then uses a pre-trained language model (such as ERNIE) to vectorize the text, obtaining context-aware dynamic word vectors for each word. During the encoding process, this unit simultaneously performs dependency parsing and referential resolution, identifying grammatical modification relationships between words in a sentence and cross-sentence referential relationships. For example, it maps different expressions such as "he," "link," and "protagonist" to the same entity identifier, thereby avoiding semantic conflicts caused by entity redundancy in subsequent steps.

[0153] The entity recognition unit, based on the vectorized representation output by the text preprocessing unit, incorporates a fully connected layer and a conditional random field layer for sequence labeling. The fully connected layer maps high-dimensional semantic vectors to the dimensional space of entity labels, while the conditional random field layer learns the transition constraints between output labels, ensuring that the predicted entity label sequence is syntactically and logically valid and optimal. The labeled text is transformed into discrete entity nodes, including character entities (such as "Link"), scene entities within the world-building (such as "Azeria" and "Star Peak"), item entities (such as "Star Sword"), and event entities within key plot elements (such as "Dragon Invasion" and "Obtaining Star Sword").

[0154] The relation extraction unit further establishes semantic connections between entity nodes. This unit employs a model based on an attention mechanism and a bidirectional long short-term memory network to perform deep semantic parsing on sentences containing two entities, focusing on words that best reflect the entity relationship (such as "obtain" and "located in") to predict the type of relationship between entities. For complex logical relationships between event entities (such as temporal and causal relationships), this unit uses a relation propagation mechanism based on a graph neural network. By aggregating information from neighboring nodes to update the representation of the current node, it can effectively capture long-distance dependencies across sentences and construct temporal and causal edges between event nodes.

[0155] The graph storage unit persistently stores the aforementioned entity nodes and relation edges in a graph database (such as Neo4j), forming a structured knowledge graph. This graph not only stores "what exists" in the IP world, but also records "what happened" and "why it happened," providing a logically consistent and deeply narrative knowledge foundation for subsequent multimodal content generation. Through these technical means, the graph construction module overcomes the fragmented and shallow knowledge representation shortcomings of traditional methods, laying a high-quality data foundation for the entire system.

[0156] Intent parsing and retrieval module

[0157] The intent parsing and retrieval module corresponds to step S2 in the method embodiment. Its core function is to receive cross-modal content generation requests expressed by users in natural language, parse the target media format, and retrieve and extract knowledge subgraphs adapted to the target media format from the global IP knowledge graph. This module internally includes an intent classification unit, an entity linking unit, and a subgraph trimming unit.

[0158] The intent classification unit first performs semantic parsing on the user's input request, encodes it through a pre-trained language model, and then inputs it into the classification layer. It outputs a probability distribution and takes the category corresponding to the maximum probability as the target media form output. For example, for the request "Please generate a short video script for 'Star Era' for Douyin", the classifier outputs "short video script".

[0159] The entity linking unit is responsible for extracting the core entities and relationships driving the graph query from the request. Since user requests typically contain highly generalized expressions (such as "highlight moment"), relying solely on keyword matching is insufficient to accurately anchor specific nodes in the graph. Therefore, this unit employs a semantic retrieval mechanism, aligning entity mentions in the request with normalized entity nodes in the graph at the semantic level. By calculating cosine similarity in the vector space, the most matching candidate entity is selected. This mechanism ensures that even if the expression used in the request does not perfectly match the normalized name in the graph, the system can still achieve accurate alignment through semantic similarity; for example, correctly matching "obtain the Star Sword" with the event node "acquire the Star Sword" in the graph.

[0160] The subgraph pruning unit prunes a suitable knowledge subgraph from the global IP knowledge graph based on the target media form and the identified core entities. This unit first performs a depth-limited breadth-first traversal based on the core entity nodes to obtain a base subgraph. Then, it introduces a reward-based graph pruning algorithm—the Prize-Collecting Steiner Tree algorithm—to select the nodes and edges that contribute most to the current generation task from the base subgraph. This algorithm seeks a balance between including high-reward nodes and avoiding high-cost edges by minimizing the objective function. Node rewards are assigned based on the semantic relevance of the entity to the user request, and edge costs are set according to the importance of the relation type. By solving this optimization problem, the subgraph pruning unit outputs a moderately sized, information-dense, concise subgraph that contains the core facts needed to complete the generation task while effectively filtering out redundant and noisy information. Compared to fixed-depth traversal or PageRank-based node sorting methods, this query-aware adaptive pruning mechanism fundamentally solves the dilemmas of context overload and information omission.

[0161] Multi-task generation engine

[0162] The multi-task generation engine corresponds to step S3 in the method embodiment. Its core function is to vectorize the knowledge subgraph obtained by the intent parsing and retrieval module, and through a collaborative mechanism of shared encoding and multi-path decoding, realize the parallel or on-demand generation of multimodal content from a unified knowledge source. The engine internally includes a graph encoding unit, a shared semantic encoding layer, and multiple task-specific decoding branches (short video script decoder, audiobook panel decoder, comic panel decoder, and peripheral text decoder).

[0163] The graph encoding unit is responsible for transforming structured knowledge subgraphs into continuous vector representations. This unit uses a graph neural network to encode the knowledge subgraphs, aggregates neighborhood information through a multi-round message passing mechanism to obtain the hidden layer representation of each node, and then fuses the node-level representations into a graph-level representation of the entire knowledge subgraph through graph pooling operations (such as attention pooling). This vector encapsulates the complete semantic information contained in the subgraph.

[0164] The shared semantic encoding layer, composed of multiple stacked Transformer encoders, performs deep semantic abstraction of the input vector, extracting universal features valuable for all subsequent generation tasks. This layer employs a multi-head self-attention mechanism and a feedforward neural network, optimized through residual connections and layer normalization. After processing by the shared encoding layer, the final hidden layer representation is output, which retains the core semantic information of the knowledge subgraph while also undergoing enhanced extraction of cross-task universal features. From the theoretical perspective of multi-task learning, the shared encoding layer, by jointly optimizing multiple related tasks, forces the model to learn more robust and general feature representations, significantly improving the model's generalization performance on sparse data tasks.

[0165] Multiple task-specific decoding branches are based on the Transformer decoder architecture, but are customized in terms of output format, loss function design, and training data. The short video script decoder includes a text generation submodule and a visual cue generation submodule, which achieve information interaction and timeline alignment through a shared attention mechanism. The audiobook panel decoder predicts sentiment tags based on the hidden state of the current step while generating text, and embeds sentiment annotations into the text; its training employs joint optimization of language modeling loss and sentiment prediction loss. The comic panel decoder uses a hierarchical decoding strategy, first generating a skeleton representation of the sequence of screen units, and then generating a corresponding detailed description for each screen unit. The multi-task generation engine employs a dynamic weight adjustment strategy during training, dynamically adjusting weights according to the magnitude of the gradient for each task to balance the convergence speed of each task. When gradient conflicts occur, the engine uses a gradient projection algorithm to correct the gradient direction of shared parameters, ensuring that the final update direction benefits all tasks. Through these techniques, the multi-task generation engine fundamentally overcomes the cross-modal semantic gap caused by traditional serial calls to multiple models, achieving an inherent unity between visual imagination and auditory emotion.

[0166] Content output interface

[0167] The content output interface corresponds to step S4 in the method embodiment. Its core function is to transform the decoder results output by the multi-task generation engine into multimodal content that can be directly used by downstream production tools or end users, and to ensure the accurate correspondence of each modal output at the presentation level. This interface internally includes a format encapsulation unit, a synchronization alignment unit, and a plugin adaptation unit.

[0168] When the user requests a short video script as the media format, the format encapsulation unit receives the voice-over text sequence and on-screen cue sequence output by the short video script decoder. The synchronization alignment unit, based on an attention-based alignment model, calculates the alignment probability between the text fragments and the on-screen cue, ensuring precise synchronization between the voice-over text and the on-screen cue on the timeline. The final output short video script is structured data containing timestamps, voice-over text, and corresponding on-screen cue arrays, which can be directly input into a video editing tool to generate visually synchronized with the audio.

[0169] When the user requests an audiobook storyboard format, the format encapsulation unit receives a text sequence containing sentiment annotations output by the audiobook decoder. The plugin adaptation unit is responsible for parsing the sentiment annotations in the text and providing plugin support for subsequent TTS modules. This unit adopts a plugin-based approach based on Low-Rank Adaptation (LoRA), inserting sentiment control as a LoRA plugin into a pre-trained neutral speech model. When parsing sentiment annotations, the plugin adaptation unit identifies the sentiment tag corresponding to the current text segment, dynamically loading or fusing the corresponding LoRA plugin, thereby achieving a smooth transition of emotions during speech generation. The final output audiobook storyboard format contains sentiment tags for both text content and timestamps, which can be directly input into the sentiment TTS module to generate audio with rich emotional fluctuations.

[0170] When users need to generate multiple media content simultaneously, the content output interface supports multi-path parallel output. The hidden layer representation of the shared encoding layer output is broadcast to all active decoders, and each decoder performs autoregressive generation independently. When output synchronization is required (such as text and visual cues in a short video script), decoders synchronize their states through a cross-attention mechanism; when output is asynchronous (such as when strict alignment is not required between a short video script and an audiobook storyboard), decoders operate completely independently. This design allows the system to flexibly adapt to different application scenarios—when users need to generate multiple media content simultaneously for the same IP event, the interface can output the generation results of multiple modalities at once; when users only need a single media content, the interface only outputs the result of the corresponding decoder, avoiding unnecessary computational overhead.

[0171] Through the aforementioned technical means, the content output interface ensures that the multimodal content generated, starting from a unified IP knowledge source, achieves a high degree of synergy and inherent consistency across multiple dimensions, including macro-plot, micro-details, emotional tone, and visual style. Compared to traditional multi-model serial invocation methods, the short video scripts and audiobook storyboards output by this interface possess a natural semantic synergy—the visual elements emphasized by the on-screen cues and the vocal emotions indicated by the emotional markers both originate from a unified semantic understanding of the same event. This inherent semantic synergy is something that traditional methods cannot achieve.

[0172] In summary, the cross-modal content generation system based on IP knowledge graphs provided in this embodiment establishes a high-quality structured knowledge foundation through a graph construction module, achieves precise information retrieval with query awareness through an intent parsing and retrieval module, completes the collaborative generation from a unified knowledge source to multimodal content through a multi-task generation engine, and ensures precise alignment at the presentation level through a content output interface. Each module is interconnected and works collaboratively to form a complete, end-to-end cross-modal content generation solution, possessing significant substantive characteristics in terms of technical means and achieving remarkable technological advancements in terms of technical effects.

[0173] Example 3: Electronic Device Example

[0174] This embodiment provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the cross-modal content generation method based on IP knowledge graph as described in Embodiment 1.

[0175] Example 4: Example of a computer-readable storage medium

[0176] This embodiment provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the cross-modal content generation method based on IP knowledge graph as described in Embodiment 1.

[0177] This invention constructs a knowledge representation foundation centered on an IP knowledge graph, systematically extracting structured elements such as characters, worldviews, and plots from original book texts. It also defines multi-dimensional semantic relationships between entities, including participation, location, temporal sequence, and causality. This solves the technical problems of fragmented and superficial knowledge representation in traditional methods, providing a logically consistent and deeply narrative-rich knowledge source for subsequent multimodal generation. Furthermore, by introducing a cross-modal generation request parsing mechanism based on intent classification, semantic linking, and reward propagation pruning, it accurately identifies the target media form from user natural language requests and adaptively prunes information-dense knowledge subgraphs, overcoming the technical difficulties of context overload and key information omission in traditional information retrieval. This invention addresses the semantic gap across modal content caused by traditional serial multi-model calls by designing a multi-task large-scale model architecture that includes a shared semantic encoding layer and multiple task-specific decoding branches. This architecture maps knowledge subgraphs into unified semantic vectors and drives the parallel generation of each decoding branch, ensuring inherent consistency in character settings, plot logic, and emotional tone across different media formats. Furthermore, by introducing an attention-based alignment model and an automatic sentiment annotation insertion mechanism at the output stage, it achieves precise timeline synchronization between voice-over text and on-screen cues in short video scripts, and semantic matching between sentiment annotations and narrative context in audiobook storyboards, thus resolving alignment and collaboration issues at the presentation level for multimodal content. Through these interconnected technical means, this invention achieves high-quality, high-efficiency, and high-consistency generation of content across multiple media formats, from the same IP source, including short video scripts, audiobook storyboards, comic panel layouts, and related copywriting. This significantly improves the automation level of IP development and the collaborative quality of multimodal content. The core of this invention lies in constructing a generative paradigm centered on an IP knowledge graph. First, structured elements such as characters, worldviews, and key plot points are extracted from the original book text to form the IP ontology and construct a knowledge graph. This graph is not only a repository of knowledge but also a "script" and "blueprint" for multimodal generation. Subsequently, upon receiving a generation request, the system parses the target medium (such as short videos or audiobooks) and extracts the most relevant "knowledge subgraph" from the graph. Finally, this knowledge subgraph is input into a specially designed multi-task large model. This model adopts a "unified encoding + multi-path decoding" architecture. A shared encoding layer is responsible for understanding the deep semantics of the knowledge subgraph, while different decoding branches are responsible for generating short video scripts (text + visual cues), audiobook storyboards (voice + emotion annotation), comic panel layouts (text and image alignment), and related copywriting, etc. This mechanism ensures that regardless of the type of content generated, the source is the same structured IP knowledge, fundamentally solving the consistency problem of multimodal content and enabling synchronous or on-demand generation of multiple tasks.

[0178] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.< / sadness>

Claims

1. A cross-modal content generation method based on IP knowledge graph, characterized in that, Includes the following steps: Step S1: IP Knowledge Graph Construction Steps: Parse the original book text, extract character entities, worldview entities and key plot elements from the text, and construct a knowledge graph that uniformly represents the core ontology of the IP by identifying the semantic relationships between entities; among which, semantic relationships include the participation relationship between characters and events, the location relationship between characters and scenes, and the temporal and causal relationships between events; Step S2: Cross-modal content generation request parsing step: In response to the received cross-modal content generation request, perform semantic parsing on the request to determine the target media form, and retrieve and extract the knowledge subgraph adapted to the target media form from the IP knowledge graph; Step S3: Multi-task collaborative generation step: Input the knowledge subgraph into the trained multi-task large model, wherein the multi-task large model contains a shared semantic encoding layer and multiple task-specific decoding branches. The shared semantic encoding layer is used to map the knowledge subgraph into a unified semantic vector, and the multiple decoding branches correspond to content generation tasks of different media forms. Step S4: Multimodal content output step: Activate one or more of the decoding branches corresponding to the target media form, and generate multimodal content adapted to different media forms synchronously or on demand based on a unified semantic vector; wherein, the multimodal content includes at least one of the following: short video scripts with visual prompts, audiobook storyboard text with emotion annotations, comic panel descriptions with aligned text and images, and copywriting for peripheral products associated with entities.

2. The cross-modal content generation method based on IP knowledge graph according to claim 1, characterized in that, Step S1, which involves extracting character entities, world-building entities, and key plot elements from the text, specifically includes: The original book text is vectorized using a pre-trained language model, and dependency parsing and referential resolution are performed based on the encoding results to merge different expressions pointing to the same real-world entity into a unified entity identifier. A fully connected layer and a conditional random field layer are connected above the output layer of the pre-trained language model for sequence labeling. Each word in the text is labeled as the corresponding entity type, thereby identifying the specific constituent elements of character entities, worldview entities including at least scene entities and prop entities, and event entities in key plot elements.

3. The cross-modal content generation method based on IP knowledge graph according to claim 1, characterized in that, The step S1 of identifying semantic relationships between entities specifically includes: A deep learning model based on attention mechanism is used to perform semantic parsing on sentences containing two entities, focusing on words that represent the relationship between the entities, and predicting the type of relationship between the two entities; For the temporal and causal relationships between event entities, a relation propagation mechanism based on graph neural networks is adopted. By aggregating neighbor node information to update the representation of the current node, temporal and causal edges between event nodes are constructed.

4. The cross-modal content generation method based on IP knowledge graph according to claim 1, characterized in that, Step S2, which involves retrieving and extracting a knowledge subgraph from the IP knowledge graph that is compatible with the target media format, specifically includes: The intent of cross-modal content generation requests is classified, and the media form corresponding to the maximum probability is output as the target media form. Semantically link entity mentions in the request, and determine the core entities driving the graph query by calculating the cosine similarity between the entity mention vector and the candidate entity node vector in the graph; Starting from the core entity nodes, perform a breadth-first traversal with a limited depth to obtain the basic subgraph; A graph pruning algorithm based on reward propagation is adopted. Nodes are assigned reward values ​​according to the semantic relevance between entities and requests, and edges are assigned cost values ​​according to the relationship type. By minimizing the objective function, information-dense simplified subgraphs are selected from the basic subgraphs as the knowledge subgraph.

5. The cross-modal content generation method based on IP knowledge graph according to claim 1, characterized in that, Step S3, which involves inputting the knowledge subgraph into the trained multi-task large model, specifically includes: The knowledge subgraph is encoded using a graph neural network. The hidden layer representation of each node is obtained by aggregating neighborhood information through multiple rounds of message passing. Then, the node-level representations are fused into a graph-level vector through graph pooling operations. The graph-level vector is input into the shared semantic coding layer, which is composed of multiple stacked Transformer encoders, and outputs a unified semantic vector after deep semantic abstraction. A unified semantic vector is input in parallel to multiple task-specific decoding branches. Each decoding branch is based on the Transformer decoder architecture, but the output format and training data are customized to the corresponding media form. The multi-task large model is trained using a multi-task learning approach. The training loss is a weighted sum of the loss functions of each task. During training, a dynamic weight adjustment strategy is used to balance the convergence speed according to the gradient magnitude of each task. A gradient projection algorithm is used to correct the gradient direction of shared parameters to eliminate gradient conflicts between tasks.

6. The cross-modal content generation method based on IP knowledge graph according to claim 1, characterized in that, Step S4, which generates a short video script with on-screen prompts, specifically includes: Activate the short video script decoder, which contains a text generation submodule and a video prompt generation submodule; The two sub-modules generate a sequence of voice-over text and a sequence of on-screen prompts in parallel based on a unified semantic vector; The alignment probability between the voice-over text and the on-screen prompts is calculated using an alignment model based on an attention mechanism, ensuring precise synchronization of the two sequences on the timeline. The output includes structured data containing timestamps, voice-over text, and corresponding on-screen cues; the audiobook storyboard text with emotion annotations in step S4 specifically includes: Activate the audiobook scene decoder, which predicts sentiment tags based on the hidden state of the current step while generating the text sequence; Emotional markers are embedded in the generated text, and these markers are used to instruct the subsequent text-to-speech module to control the timbre, speed, and pitch of the speech. The output contains structured data including text content and sentiment annotations.

7. A cross-modal content generation system based on IP knowledge graph, characterized in that, The system for implementing the method according to any one of claims 1 to 6, the system comprising: Knowledge graph construction module: used to parse the original book text, extract character entities, worldview entities and key plot elements from the text, and construct a knowledge graph that uniformly represents the core ontology of the IP by identifying the semantic relationships between entities; Intent parsing and retrieval module: used to receive cross-modal content generation requests, parse the target media form, and retrieve and extract knowledge subgraphs that are compatible with the target media form from the IP knowledge graph; The multi-task generation engine includes a graph encoding unit, a shared semantic encoding layer, and multiple task-specific decoding branches. The graph encoding unit encodes the knowledge subgraph into graph-level vectors. The shared semantic encoding layer maps the graph-level vectors into unified semantic vectors. The multiple decoding branches generate multimodal content in corresponding media formats synchronously or on demand based on the unified semantic vectors. Content output interface: Used to convert the multimodal content output by the multi-task generation engine into a format that can be used by downstream tools, and to ensure the accurate correspondence of each modal output at the presentation level.

8. The cross-modal content generation system based on IP knowledge graph as described in claim 7, characterized in that, The map construction module includes: The text preprocessing unit is used to vectorize the original book text using a pre-trained language model and perform dependency parsing and reference resolution. The entity recognition unit is used to connect a fully connected layer and a conditional random field layer on top of the output layer of the pre-trained language model to perform sequence labeling and identify character entities, scene entities, prop entities and event entities; The relation extraction unit is used to predict the types of relationships between entities using a deep learning model based on an attention mechanism, and to construct temporal and causal edges between event nodes using a relation propagation mechanism based on a graph neural network. Graph storage unit, used to persistently store entity nodes and relation edges in the graph database; The intent parsing and retrieval module includes: The intent classification unit is used to classify cross-modal content generation requests by intent and determine the target media format. The entity linking unit is used to semantically link entity references in the request, and to determine the core entity by calculating the vector cosine similarity. The subgraph pruning unit is used to perform a breadth-first traversal starting from the core entity to obtain the basic subgraph, and to use a graph pruning algorithm based on reward propagation to select a simplified subgraph from the basic subgraph as the knowledge subgraph. The multi-task generation engine is trained using a multi-task learning approach. During training, a dynamic weight adjustment strategy is used to balance the convergence speed of each task, and a gradient projection algorithm is used to eliminate gradient conflicts between tasks. The content output interface includes: The synchronization alignment unit is used to ensure precise synchronization of the voice-over text and on-screen cues on the timeline when generating short video scripts, through an attention-based alignment model. The plugin adaptation unit is used to parse the sentiment markers in the text when generating audiobook storyboards, and to provide plugin-based support for the subsequent text-to-speech module based on low-rank adaptation, so as to achieve a smooth transition of sentiment.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the cross-modal content generation method based on IP knowledge graph as described in any one of claims 1 to 6.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the cross-modal content generation method based on IP knowledge graph as described in any one of claims 1 to 6.

Images