Method and system for multi-modal content generation based on large model and knowledge graph driving

Abstract

The application discloses a multi-modal content generation method and system based on a large model and a knowledge graph, relates to the technical field of multi-modal content generation, and extracts the underlying features of the content generation requirements of a user through a cross-modal semantic encoder based on the neural engine of a multi-modal large model and maps the underlying features to a unified requirement semantic space; the neural engine generates the cross-modal original content sketch corresponding to the content generation requirements through a creative generator; through the architecture of the neural engine creative generation, the symbolic engine knowledge constraint and the dynamic cooperation of the double engines, the application not only inherits the cross-modal understanding and creative ability of the large model, but also plays the fact verification and logical reasoning advantages of the knowledge graph, finally realizes the multi-modal content generation which is both free and anchored to facts, and provides an efficient and reliable intelligent creation tool for the fields of education, media, art and the like.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of multi-modal content generation, in particular to a multi-modal content generation method and system based on a large model and a knowledge graph. BACKGROUND

[0002] Multi-modal content generation aims to automatically generate cross-modal content such as text, images, videos, and 3D models that match user input text, images, and voice, etc. The core challenge is to meet the creativity and factuality of the content.

[0003] Currently, there are two main technical routes for multi-modal content generation technology, but each has significant limitations:

[0004] The generation method based on multi-modal large model is represented by multi-modal large models such as GPT-4V, Midjourney, and DALL·E. Through massive data training, these models have strong cross-modal understanding and creative generation capabilities. This method can flexibly capture the implicit creative intent of user demand and generate cross-modal content with diverse styles and rich forms (such as generating images from text descriptions or generating narrative text from images).

[0005] However, since the generation process relies on data-driven implicit pattern learning and lacks explicit constraints from structured knowledge, there are two major problems: factuality distortion: prone to hallucinated content, i.e., generating information that conflicts with objective facts;

[0006] Cross-modal semantic inconsistency: the core semantics of different modal content (such as text and image) may deviate (for example, the text description "ancient battlefield" contains modern weapons in the image).

[0007] The generation method based on knowledge graph stores domain knowledge through the structured representation of entities, relationships, and attributes, providing explicit factual constraints for content generation (such as "Li Bai is a poet of the Tang Dynasty" and "quantum computers work based on quantum bits"). The generation method based on knowledge graph ensures the accuracy of entity relationships and attributes through logical reasoning, to some extent, solving the factuality problem. However, this method has obvious shortcomings:

[0008] Lack of creative flexibility: limited by the fixed structure and predefined relationships of the knowledge graph, it is difficult to generate creative content that breaks existing knowledge boundaries (such as the fusion of historical figures and future scenarios);

[0009] Poor cross-modal adaptability: there is a semantic gap between the symbolic representation of the knowledge graph and the underlying features of multi-modal content (such as image pixels and voice waveforms), making it difficult to directly support the deep fusion generation of cross-modal content;

[0010] Knowledge update lag: In the face of dynamic creative demand (such as emerging concepts, virtual scene setting), the static structure of the knowledge graph is difficult to expand quickly, resulting in a knowledge gap that cannot be filled in time.

[0011] Therefore, how to construct a multi-modal content generation method with cross-modal creative generation capability and structured knowledge constraint capability, and realize efficient cooperation of neural-symbol dual engine, has become a technical problem to be solved in the field. SUMMARY

[0012] In order to solve the above technical problems, the present application provides a multi-modal content generation method and system driven by large model and knowledge graph. The following technical solutions are adopted:

[0013] The multi-modal content generation method driven by large model and knowledge graph comprises the following steps: the neural engine based on multi-modal large model extracts the underlying features of the user's content generation demand through the cross-modal semantic encoder, and maps them to a unified demand semantic space;

[0014] The neural engine generates a cross-modal original content sketch corresponding to the content generation demand through the creative generator;

[0015] The symbolic engine dynamically extracts the entities, relationships and attributes of the cross-modal original content sketch with the dynamic knowledge graph as the core, forms a context subgraph, and checks the rationality of the content generated by the neural engine through the inference rules of the graph. The symbolic engine adds non-factual but reasonable knowledge nodes based on the creative demand, and automatically associates entities. The multi-modal content generator interacts with the neural engine and the symbolic engine to generate content, and outputs the multi-modal content corresponding to the content generation demand after integrating the generated content.

[0016] Optionally, the neural-symbol collaborative controller is used to realize the real-time interaction and closed-loop optimization of the neural engine and the symbolic engine.

[0017] Optionally, the trigger condition of the neural-symbol collaborative controller is: when the neural engine generates entities and relationships, the collaborative controller automatically calls the symbolic engine to query the graph to confirm the facts; when the symbolic engine finds a knowledge gap, the collaborative controller triggers the neural engine to generate a hypothetical association;

[0018] The conflict resolution strategy of the neural-symbol collaborative controller is: if the neural engine creativity conflicts with the graph facts, the neural-symbol collaborative controller generates a compromise solution and updates the graph to add a temporary hypothesis.

[0019] Optionally, the neural-symbol collaborative controller realizes the interaction between the neural engine and the symbolic engine through semantic-symbol mapping. The mapping loss of the semantic vector z and the knowledge graph G s is expressed as:

[0020]

[0021] where z e is the vector of entity e in semantic space, Enc(e) is the encoding vector of entity e by the neural engine, z r is the vector of relation r in semantic space, and the mapping loss is denoted as L map The objective of the mapping loss L

[0022] Optionally, the creativity boundary of the symbolic engine is controlled by a creativity divergence factor.

[0023] Optionally, the architecture of the neural engine comprises a multi-modal input layer, a cross-modal semantic encoder, a creativity generator, a modal conversion decoder, and a constraint adaptation optimizer.

[0024] The multi-modal input layer is configured to receive a multi-modal content generation requirement of a user and standardize the requirement into a tensor format.

[0025] The cross-modal semantic encoder is configured to map the multi-modal input into a unified semantic space and output a semantic vector.

[0026] The creativity generator is configured to generate a cross-modal raw content sketch based on the semantic vector and knowledge constraints of the symbolic engine.

[0027] The modal conversion decoder is configured to convert an intermediate result of the creativity generator into a target modality and output the cross-modal raw content sketch.

[0028] The constraint adaptation optimizer is configured to receive a fact-checking result of the symbolic engine, correct conflicts in the raw content, and output a final controllable content.

[0029] Optionally, the neural engine further comprises a feedback optimization module, which is configured to receive feedback opinions of the user on the generated multi-modal content and update parameters of the creativity generator based on the feedback opinions.

[0030] Optionally, the architecture of the symbolic engine comprises an entity relation extractor, a dynamic subgraph constructor, a logical reasoner, and a hypothesis extender.

[0031] The entity relation extractor is configured to extract entities, relations, and attributes from the output of the neural engine.

[0032] The dynamic subgraph constructor is configured to construct a context subgraph based on the extracted entity relations and associate the context subgraph with a base knowledge graph.

[0033] The logical reasoner is configured to perform fact-checking and logical reasoning based on the context subgraph and a rule base.

[0034] The hypothesis extender is configured to generate hypothetical entity relations when there is a knowledge gap in the subgraph.

[0035] Optionally, assuming that the extender screens the candidate hypothetical entity relationship by quantifying the rationality of the hypothesis, the rationality score s(h) of the candidate hypothetical entity relationship is calculated using the following formula:

[0036] s(h) = ω1·S sem (h) + ω2·S rul (h) + ω3·S conl (h) + ω4·S nov (h);

[0037] Wherein S sem (h) is the semantic consistency dimension score, S rul (h) is the rule compatibility dimension score, S conl (h) is the scene relevance dimension score, S nov (h) is the creative novelty dimension score, ω1, ω2, ω3, ω4 are the weights of the four dimension scores.

[0038] The multi-modal content generation system driven by a large model and a knowledge graph is used to realize a multi-modal content generation method driven by a large model and a knowledge graph, and the multi-modal content generation system comprises a multi-modal input interface, a cross-modal preprocessing layer, a neural engine module, a symbolic engine module, a neural-symbolic collaborative controller, a multi-modal content generator and a memory.

[0039] The multi-modal input interface is used to receive text, image, voice or 3D model input of a user and convert the input into a standardized data format.

[0040] The cross-modal preprocessing layer extracts multi-modal features based on the encoder of the neural engine, analyzes user demand and maps the demand to a unified semantic space.

[0041] The neural engine module integrates a multi-modal large model and generates cross-modal original content sketches corresponding to the content generation demand.

[0042] The symbolic engine module adds non-factual but reasonable knowledge nodes to the cross-modal original content sketches based on creative demand to generate multi-modal content corresponding to the demand.

[0043] The memory is used to store content generation demand data, basic knowledge base data and generated multi-modal content data of the user.

[0044] In summary, the present application has the following at least one beneficial technical effect:

[0045] The application can provide a multi-modal content generation method and system driven by a large model and a knowledge graph, through a neural engine creative generation, a symbolic engine knowledge constraint, and a dual-engine dynamic coordination architecture, the multi-modal content generation method and system inherit the cross-modal understanding and creative ability of the large model, and play the fact verification and logical reasoning advantages of the knowledge graph, and finally realize multi-modal content generation that is both free and anchored to facts, and provide an efficient and reliable intelligent creation tool for the fields of education, media, art, and the like. BRIEF DESCRIPTION OF DRAWINGS

[0046] Figure 1 is a flowchart of the multi-modal content generation method driven by a large model and a knowledge graph according to the application. DETAILED DESCRIPTION

[0047] The application will be further described in detail below with reference to the accompanying drawings.

[0048] The embodiments of the application disclose a multi-modal content generation method and system driven by a large model and a knowledge graph.

[0049] Referring to Figure 1 , embodiment 1, the multi-modal content generation method driven by a large model and a knowledge graph includes the following steps: a neural engine based on a multi-modal large model extracts the bottom features of the content generation demand of a user through a cross-modal semantic encoder, and maps to a unified demand semantic space;

[0050] The neural engine generates a cross-modal original content sketch corresponding to the content generation demand through a creative generator;

[0051] A symbolic engine dynamically extracts entities, relationships, and attributes of the cross-modal original content sketch with a dynamic knowledge graph as the core, forms a context subgraph, the symbolic engine verifies the rationality of the content generated by the neural engine through the inference rules of the graph, adds non-factual but reasonable knowledge nodes based on the creative demand, and automatically associates the entities, the multi-modal content generator interacts with the neural engine and the symbolic engine to generate content, integrates the generated content, and outputs the multi-modal content corresponding to the content generation demand.

[0052] By adopting the technical scheme, the cross-modal semantic encoder of the engine converts the heterogeneous requirements input by the user, such as the text description of "showing a dinosaur operating a quantum computer with animation" and the reference image, into a unified dimension semantic vector through the pre-trained multi-modal large model, such as the joint feature learning of text, image and voice, and eliminates the modal barrier. For example, "dinosaur" in the text and the dinosaur feature in the image are mapped to the same semantic space, ensuring that the subsequent generation process is consistent in understanding the "core entity". The creative generator, based on the unified semantic vector, combines the context learning and generation capability of the large model, breaks through the traditional rule constraints, and generates a cross-modal original content sketch (such as an animation storyboard + keyframe sketch). This process does not rely on fixed knowledge structure and can freely integrate elements from different fields (such as prehistoric organisms and modern technology) to meet creative needs.

[0053] The symbol engine extracts entities (dinosaur, quantum computer), relationships (operation), and attributes (dinosaur species = tyrannosaurus, quantum computer function = quantum computing) from the original content generated by the neural engine, and constructs a context subgraph strongly related to the current task. This subgraph is not a static reuse of knowledge, but is dynamically generated based on the content sketch to ensure the relevance of knowledge constraints (such as focusing only on the association between "dinosaur-quantum computer", rather than the entire knowledge graph).

[0054] Based on the graph reasoning rules (such as "quantum computer operation requires a logical interface"), the content rationality is checked, and if the original sketch conflicts with the rules (such as "dinosaur directly touching the chip"), a correction signal is triggered. For knowledge gaps (such as "how does a dinosaur understand quantum logic"), the assumption extender adds "non-factual but reasonable" knowledge nodes (such as "dinosaur interacts with quantum computer through brain-computer interface") and automatically associates the implicit relationship between "brain-computer interface-quantum computer", filling the logical gap while preserving creative space.

[0055] The multi-modal content generator serves as a collaborative hub, receiving the creative output from the neural engine and the knowledge constraints from the symbol engine, and achieving closed-loop optimization through the following mechanisms:

[0056] When the symbol engine detects a factual conflict (such as "dinosaur lived in an earlier era than quantum computer"), it feeds back to the neural engine to adjust the content (such as changing to "the digital twin of the dinosaur operates a quantum computer in the metaverse");

[0057] When the creativity of the neural engine exceeds the existing knowledge of the symbol engine (such as "dinosaur language in the metaverse"), the symbol engine dynamically extends the subgraph (adding the assumption node "dinosaur language = quantum coding based on sound waves");

[0058] Finally, the integrated and corrected cross-modal content (such as animation video + explanation text + 3D model) ensures the consistency of different modalities in terms of semantics and logic.

[0059] Through the architecture of neural engine creative generation, symbolic engine knowledge constraint, and dynamic coordination of the two engines, the cross-modal understanding and creative ability of large models are inherited, and the fact verification and logical reasoning advantages of knowledge graphs are also exerted, ultimately realizing multi-modal content generation that is both free and anchored to facts, and providing efficient and reliable intelligent creation tools for education, media, art, and other fields.

[0060] Embodiment 2: Real-time interaction and closed-loop optimization of neural engine and symbolic engine are realized by using a neural-symbolic collaborative controller.

[0061] Embodiment 3: The triggering condition of the neural-symbolic collaborative controller is that when the neural engine generates entities and relationships, the collaborative controller automatically calls the symbolic engine to query the graph to confirm the facts; when the symbolic engine finds a knowledge gap, the collaborative controller triggers the neural engine to generate a hypothetical association.

[0062] The conflict resolution strategy of the neural-symbolic collaborative controller is that if the neural engine's creativity conflicts with the graph facts, the neural-symbolic collaborative controller generates a compromise solution and updates the graph to add a temporary hypothesis.

[0063] Embodiment 4: The neural-symbolic collaborative controller realizes the interaction between the neural engine and the symbolic engine through semantic-symbolic mapping. The mapping loss of the semantic vector z and the knowledge graph G s is expressed as:

[0064]

[0065] where z e is the vector of entity e in the semantic space, Enc(e) is the encoding vector of entity e by the neural engine, z r is the vector of relationship r in the semantic space, and the goal of the mapping loss expression L map is to minimize the mapping error between symbolic knowledge and neural semantics.

[0066] By using the above technical solutions, the core function of the collaborative controller is to break the independent working mode of the neural engine and the symbolic engine, build a closed-loop link of generation-verification-feedback-optimization, and realize the dynamic adaptation of the two engines:

[0067] Interaction timing design: The controller receives the generation state of the neural engine and the knowledge processing result of the symbolic engine in real time through a pre-set communication protocol.

[0068] Based on the status of both parties, the next operation is triggered, avoiding the low efficiency or content deviation caused by blind work of a single engine.

[0069] With content quality indicators (factual accuracy, creative novelty, cross-modal consistency) as feedback targets, the output of the dual engine is adjusted through multiple iterations: for example, if the factual conflict rate of the first generated content exceeds the threshold, the controller will enhance the constraint weight of the symbolic engine; if the creative novelty is insufficient, the divergence factor of the neural engine will be increased.

[0070] The iteration termination condition is that the quality indicators meet the preset threshold or the maximum number of iterations is reached, ensuring a balance between efficiency and effectiveness.

[0071] Embodiment 5: Control the creative boundary of the symbolic engine through the creative divergence factor.

[0072] By adopting the above technical solutions, the creative divergence factor is a key adjustment parameter for balancing the knowledge constraint rigor and creative expansion flexibility of the symbolic engine. Its core role is to dynamically adjust the acceptance threshold and expansion range of the symbolic engine for hypothetical knowledge, achieving precise control of the creative boundary. This mechanism solves the problem of limited creativity caused by fixed knowledge constraints in traditional symbolic engines or content loss caused by unlimited assumption expansion. The creative divergence factor penetrates into the core algorithms of the assumption expander and logical reasoner of the symbolic engine, dynamically adjusts the decision threshold of the two, so that the creative expansion neither deviates from the knowledge base nor breaks through the routine as needed.

[0073] The creative divergence factor is not an isolated parameter, but forms a linkage with the neural engine through the neural-symbolic coordination controller, achieving whole-process creative boundary control.

[0074] Embodiment 6: The architecture of the neural engine includes a multi-modal input layer, a cross-modal semantic encoder, a creative generator, a modal conversion decoder, and a constraint adaptation optimizer.

[0075] The multi-modal input layer is used to receive the user's multi-modal content generation requirements and standardize them into tensor format.

[0076] The cross-modal semantic encoder maps the multi-modal input to a unified semantic space and outputs a semantic vector.

[0077] The creative generator generates a cross-modal original content sketch based on the semantic vector and the knowledge constraints of the symbolic engine.

[0078] The modal conversion decoder converts the intermediate result of the creative generator into the target modality and outputs the cross-modal original content sketch.

[0079] The constraint adaptation optimizer receives the fact checking results of the symbolic engine, corrects the conflicts in the original content, and outputs the final controllable content.

[0080] The neural engine further includes a feedback optimization module. The feedback optimization module of the neural engine receives feedback of a user on the generated multi-modal content, and updates parameters of the creative generator based on the feedback.

[0081] By adopting the technical solution, the modal input layer supports standardized processing of heterogeneous inputs such as text, image, voice, and 3D model, and uniformly converts them into tensor format, breaking through the input limitation of traditional single-modal generation models. For example, a user can input a text description of “cyberpunk style Dunhuang flying goddess” and a reference image at the same time. The system can seamlessly integrate the semantic information of the two inputs, avoiding the demand understanding deviation caused by a single input form, and improving the input compatibility.

[0082] The cross-modal semantic encoder solves the problem of semantic misplacement of text description and image features by mapping multi-modal inputs to a unified semantic space. For example, for the requirement of “quiet seaside sunset”, the “quiet” emotion in the text and the “low saturation tone, gentle ripples” visual features in the image are encoded into consistent semantic vectors, ensuring that the subsequently generated content (such as poetry, video) is highly consistent in emotion and scene, and the cross-modal semantic consistency index is improved to more than 90%.

[0083] The creative generator generates content prototypes based on the unified semantic vectors and the knowledge constraints of the symbolic engine, retaining the divergent creativity of large models while avoiding unbounded fiction through knowledge constraints.

[0084] After receiving the fact-checking results of the symbolic engine (such as the conflict of the physical properties of the quantum light wing), the constraint adaptation optimizer makes targeted corrections to the content (such as adjusting the quantum light wing to an energy visualization effect that conforms to the principles of optics), so that the generated content expresses creativity while improving factual accuracy.

[0085] The modal conversion decoder module accurately converts the intermediate results of the creative generator into the target modality, ensuring the unity of the core semantics of different modal content. For example, for the text description, architectural design drawing, and promotional video generated for “future city coexisting with ancient architecture”, the core creative idea of “spatial and temporal fusion” remains consistent in word metaphor, visual composition, and shot language, and the theme consistency of cross-modal content is improved to 95%.

[0086] The feedback optimization module updates the parameters of the creative generator (such as adjusting the style transfer weight and optimizing the logical reasoning module) by receiving user feedback, so that the system gradually adapts to the preferences or field requirements of specific users. For example, in response to the feedback of an education field user that “the content needs to be more rigorous”, the system can reduce the divergence factor of the creative generator and increase the weight of knowledge constraints. After 3-5 rounds of feedback iteration, the field adaptation accuracy is improved by more than 40%.

[0087] The feedback mechanism enables the system to cope with unknown scenarios or personalized needs. For example, in the face of the generation needs of a niche art style (such as "steam wave + ink painting"), the initial results may have the problem of rigid fusion of styles. Through user feedback on "color matching" and "element proportion", the system can quickly adjust the style feature extraction module of the generator, and after 3 rounds of iteration, it can stably output the expected content, greatly reducing the dependence on pre-training data in specific fields.

[0088] The neural engine architecture achieves precision and universality in cross-modal processing through modular design, and dynamically improves content quality through constraint adaptation and feedback optimization. Ultimately, it surpasses traditional single-modal or non-feedback generation models in terms of generation efficiency, creative accuracy, and user adaptability, providing efficient, controllable, and evolutionary technical support for multi-modal content creation.

[0089] Embodiment 8, the architecture of the symbol engine includes an entity relationship extractor, a dynamic subgraph builder, a logical reasoner, and a hypothesis extender;

[0090] The entity relationship extractor extracts entities, relationships, and attributes from the neural engine output;

[0091] The dynamic subgraph builder constructs a context subgraph based on the extracted entity relationships and associates it with a base knowledge graph;

[0092] The logical reasoner performs fact checking and logical reasoning based on the context subgraph and a rule base;

[0093] The hypothesis extender generates hypothetical entity relationships when there is a knowledge gap in the subgraph.

[0094] Embodiment 9, the hypothesis extender filters candidate hypothetical entity relationships by quantifying the rationality of the hypothesis, and the rationality score s(h) of the candidate hypothetical entity relationship is calculated using the following formula:

[0095] s(h) = ω1·S sem (h) + ω2·S rul (h) + ω3·S conl (h) + ω4·S nov (h);

[0096] Where S sem (h) is the semantic consistency dimension score, S rul (h) is the rule compatibility dimension score, S conl (h) is the context relevance dimension score, S nov (h) is the creative novelty dimension score, and ω1, ω2, ω3, ω4 are the weights of the four dimension scores.

[0097] By adopting the technical scheme, the four core modules of the symbol engine form a closed loop link from knowledge extraction to dynamic expansion, and through division of labor and cooperation, the unstructured creative content generated by the neural engine is converted into structured knowledge constraints, while providing necessary knowledge support for creative generation.

[0098] The entity relationship extractor automatically identifies and extracts core entities (such as "dinosaurs" and "quantum computers"), entity relationships (such as "operation" and "dependence"), and entity attributes (such as "dinosaurs = prehistoric organisms" and "quantum computers = low-temperature environment") from the cross-modal raw content (such as text description, image semantic label, and video script) output by the neural engine.

[0099] The problem of unstructured content being difficult to be directly utilized by the knowledge graph is solved, and through natural language processing (such as BERT-based entity recognition model) and computer vision (such as object detection + relationship prediction), the fuzzy creative expression is converted into computable symbolic knowledge, providing an anchor point for subsequent knowledge processing.

[0100] The dynamic subgraph constructor constructs a context subgraph strongly related to the current generation task based on the extracted entity-relationship-attribute, and associates the basic knowledge graph through entity alignment technology.

[0101] Avoiding the knowledge redundancy and low reasoning efficiency caused by directly using the full knowledge graph. For example, when generating "dinosaur operating quantum computer" related content, the subgraph only retains the core nodes such as "dinosaur", "quantum computer" and "operation" and the associated knowledge (such as "core components of quantum computer = qubits"), rather than loading the entire biology encyclopedia or computer science knowledge graph. At the same time, by associating the basic knowledge graph, necessary background knowledge (such as "characteristics of qubits = superposition state") can be called to enhance the knowledge completeness of the subgraph.

[0102] The logical reasoner solves the problem that creative generation is easy to deviate from the facts based on the context subgraph and the preset rule library. For example, if the neural engine generates "dinosaur operating quantum computer at room temperature", the reasoner can detect the conflict through the rule "quantum computer requires low-temperature environment" and feed back the correction suggestion (such as "add scene design of low-temperature cooling device") to the neural engine.

[0103] The hypothesis extender generates hypothetical entities-relationships that meet the creative demand when there are undefined key associations (such as "how do dinosaurs understand quantum logic") or missing attributes (such as "dinosaur operation mode =?") in the context subgraph, and supplements them to the subgraph after screening through rationality evaluation.

[0104] To address the problem that "existing knowledge cannot cover creative needs," we can ensure the flexibility of knowledge constraints while providing a boundary for "reasonable fiction" for the neural engine, such as assuming that "dinosaurs understand quantum logic through brain-computer interfaces," rather than the unfounded claim that "dinosaurs are born with a knowledge of quantum physics."

[0105] Example 10: A multimodal content generation system driven by a large model and knowledge graph, used to implement a multimodal content generation method driven by a large model and knowledge graph. The multimodal content generation system includes a multimodal input interface, a cross-modal preprocessing layer, a neural engine module, a symbolic engine module, a neural symbolic co-controller, a multimodal content generator, and a memory.

[0106] The multimodal input interface is used to receive text, image, voice or 3D model input from users and convert it into a standardized data format;

[0107] The cross-modal preprocessing layer uses a neural engine-based encoder to extract multimodal features, parse user needs, and map them to a unified semantic space.

[0108] The neural engine module integrates a multimodal large model and generates cross-modal original content prototypes that meet the content generation requirements.

[0109] The symbol engine module adds non-factual but reasonable knowledge nodes to the original cross-modal content prototype based on creative needs to generate multimodal content corresponding to the needs;

[0110] The memory is used to store user content generation requirements data, basic knowledge base data, and generated multimodal content data.

[0111] The following specific embodiments illustrate the implementation principle of the present invention:

[0112] Taking the generation of multimodal popular science content featuring Su Shi discussing poetry with artificial intelligence as an example, this paper details the implementation process of the technical solution, covering the collaborative generation of three modalities: text scripts, illustrations, and short video storyboards. The system architecture and the functions of each module are as follows:

[0113] I. User Requirements and System Input:

[0114] User input:

[0115] Text requirements: "Create a piece of popular science content about Su Shi (a poet of the Song Dynasty) discussing the style of bold and unrestrained poetry with an AI robot. It should include a dialogue script, scene illustrations, and a 30-second short video storyboard, and should balance historical authenticity with technological creativity."

[0116] Reference images: Portrait of Su Shi (traditional ink painting style), concept image of AI robot (minimalist technological style).

[0117] System input processing:

[0118] The multimodal input interface receives text and images, converts them into standardized data (text, word embedding tensors, and image processing into 256×256 pixel feature tensors);

[0119] The cross-modal preprocessing layer calls the encoder of the neural engine to extract core entity features such as "Su Shi", "AI" and "bold words", and maps them to a unified semantic space (outputting a semantic vector with a dimension of 512).

[0120] II. Prototype of Neural Engine Generating Cross-Modal Original Content:

[0121] Cross-modal semantic coding:

[0122] The encoder aligns the textual description of "Su Shi" with the portrait features through contrastive learning, ensuring that the "image of a Song Dynasty literati" forms a differentiated but related representation in the semantic vector with the "technological attributes" of the "AI robot".

[0123] Creative Generator Works:

[0124] Based on the initial knowledge constraints of semantic vectors and symbolic engines ("Su Shi = Song Dynasty poet" and "heroic lyrics = vigorous style" in the basic knowledge graph), the following is generated:

[0125] Text script prototype:

[0126] Su Shi (holding a writing brush): "My poems are based on the line 'The Great River Flows East,' but do you know its heroic spirit?"

[0127] AI (blinking blue light): "Detected 'bold and unrestrained' word characteristics: grand imagery and expressive emotions. But why don't you write on a computer?"

[0128] Illustration prototype:

[0129] The scene is a fusion of a Song Dynasty study and a future laboratory. Su Shi is wearing a long robe, and the AI ​​is a silver-white robot. A calligraphy brush and a stream of code appear in the background.

[0130] Short video storyboard prototype:

[0131] Three keyframes: ① Su Shi wields his brush; ② The AI ​​screen displays a “feature map of bold and unrestrained ci poetry”; ③ Both point to the calligraphy work “The Great River Flows East”.

[0132] Modulation Decoder:

[0133] Convert the intermediate features of the generator into the target modal format.

[0134] III. Symbolic Engine for Constructing Context Subgraphs and Knowledge Constraints:

[0135] Entity relation extraction:

[0136] Extracting triples from the original content:

[0137] (Su Shi, Dynasty, Song Dynasty), (Su Shi, Identity, Poet), (AI, Attribute, Technological Product), (Su Shi and AI, Relationship, Discussion), (Discussion Topic, Attribute, Bold and Unrestrained Poetry).

[0138] Dynamic subgraph construction:

[0139] The contextual subgraph contains core nodes: Su Shi, AI, bold and unrestrained poetry, Song Dynasty, and technological products;

[0140] Connect to the basic knowledge graph: supplement facts such as "the representative work of bold and unrestrained ci poetry = 'Nian Nu Jiao: Reminiscences of Chibi'", "Song Dynasty = 960-1279 years" and "AI = 20th century invention".

[0141] Logic inference engine verification:

[0142] Rule base invocation: "The Song Dynasty predates the invention of AI" and "Su Shi could not have accessed computers";

[0143] Conflict detection: There is a factual conflict in the original script where "AI asks Su Shi why he doesn't use a computer," triggering a correction signal.

[0144] Assuming the extender fills the knowledge gap:

[0145] Knowledge gaps: "How can Su Shi interact with AI across time and space?" "How can AI understand ancient poetry?"

[0146] Generate hypothesis candidates:

[0147] h1: "Su Shi's virtual avatar interacts with AI in the metaverse";

[0148] h2: "AI learns Su Shi's poems through historical documents."

[0149] Reasonableness score s(h1) that meets the threshold (τ=0.6) is retained;

[0150] Subgraph expansion: Add nodes “Metaverse” and “Virtual Avatar”, with the relationships “Su Shi, Virtual Avatar, Universe, AI”.

[0151] IV. Optimization of the Neural-Symbol Co-controller:

[0152] Conflict resolution:

[0153] In response to the "computer" conflict, a compromise solution was generated: the AI's dialogue was revised to "If you were born in this world, would you be willing to use digital devices to record words and phrases?"

[0154] A temporary hypothesis is added to the subgraph: "Historical figures are allowed to converse with modern technology in a virtual setting."

[0155] Semantic mapping loss optimization:

[0156] Optimize the matching degree between the symbol vector of the entity "virtual avatar" and the encoding vector of the neural engine, so that the semantic vector of the "virtual avatar" is closer to the hybrid features of "Su Shi image + digital attributes".

[0157] Regulation of creative divergence factors:

[0158] In the context of popular science, the creative divergence factor is adjusted to 0.3 (low divergence), limiting the expansion of hypotheses to concepts with scientific basis such as "metaverse" and avoiding excessive fiction such as "time travel machine".

[0159] V. Multimodal Content Generator Integrated Output:

[0160] Text script (final version):

[0161] Su Shi (virtual avatar, standing in a digital study): "My poems are based on the line 'The Great River Flows East,' but do you know its unrestrained spirit?"

[0162] AI: "Detected 'bold and unrestrained' word characteristics: grand imagery and expressive emotions. If you were born in this world, would you be willing to use digital devices to record words and phrases?"

[0163] Su Shi said: "Brush and ink have a soul, and numbers also have a spirit, but the artistic conception ultimately resides in the human heart."

[0164] Illustration (Final Version):

[0165] The Song Dynasty-style study is integrated with digital code streams. A virtual avatar of Su Shi (with a semi-transparent ink painting effect) and an AI robot (with brush-shaped decorations) stand opposite each other, with the background containing floating clouds of the poem "Nian Nu Jiao".

[0166] Short video storyboard (final version):

[0167] ①0-10s: Close-up of Su Shi writing calligraphy, with AI-powered screen displaying synchronized text analysis;

[0168] ②10-20s: The two people point to the word cloud, and the camera pulls back to show the metaverse scene;

[0169] ③20-30s: Subtitle “A cross-temporal dialogue between technology and culture”.

[0170] VI. System Module Interaction and Feedback Optimization

[0171] Storage: Stores user requirements, context subgraphs, and generated content for reuse in subsequent similar tasks (e.g., "Li Bai discusses Tang poetry with AI" can invoke the "virtual avatar" hypothesis).

[0172] Feedback optimization: Users reported that "AI dialogue is too stiff". The feedback optimization module adjusted the dialogue style parameters of the creative generator (increasing the "mixing ratio of classical Chinese and vernacular Chinese"). After 3 rounds of iteration, the naturalness of the dialogue improved by 40%.

[0173] Factual accuracy: There are no deviations in core facts such as Su Shi's dynasty and poetic style, with a 100% conflict correction rate; Creative integration: The "virtual avatar + metaverse" hypothesis satisfies both cross-temporal creativity and conforms to technological logic, with an expert review score of 8.5 / 10 for creative novelty.

[0174] Cross-modal consistency: The text, illustrations, and storyboards all focus on the theme of "dialogue between technology and culture," with a semantic matching rate of 93%.

[0175] This embodiment fully demonstrates the dual-engine collaborative mechanism, proving that the technical solution can generate cross-modal and highly creative content while ensuring the rigor of knowledge.

[0176] The above are all preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Therefore, all equivalent changes made in accordance with the structure, shape and principle of the present invention should be covered within the scope of protection of the present invention.

Claims

1. A multimodal content generation method based on large models and knowledge graphs, characterized in that, Includes the following steps: The neural engine based on a multimodal large model extracts the underlying features of users' content generation needs through a cross-modal semantic encoder and maps them to a unified demand semantic space. The neural engine generates cross-modal original content prototypes that meet the content generation requirements through a creative generator. The symbol engine, with dynamic knowledge graph as its core, dynamically extracts entities, relationships, and attributes from the original cross-modal content prototype to form a contextual subgraph. The symbol engine verifies the rationality of the content generated by the neural engine through the reasoning rules of the graph, adds non-factual but reasonable knowledge nodes based on creative needs, and automatically associates them with entities. The multimodal content generator interacts with the neural engine and the symbol engine to generate content, and after integrating the generated content, it outputs the multimodal content corresponding to the content generation needs. The architecture of the neural engine includes a multimodal input layer, a cross-modal semantic encoder, a creative generator, a modality transformation decoder, and a constraint adaptation optimizer; The multimodal input layer is used to receive users' multimodal content generation requests and standardize them into tensor format; A cross-modal semantic encoder maps multimodal inputs to a unified semantic space and outputs semantic vectors; The creative generator generates cross-modal original content prototypes based on knowledge constraints from semantic vectors and symbol engines. The modality conversion decoder converts the intermediate results of the creative generator into the target modality and outputs a cross-modal prototype of the original content. The constraint-adaptive optimizer receives the fact-checking results from the symbol engine, corrects conflicts in the original content, and outputs the final controllable content. The architecture of the symbolic engine includes an entity relation extractor, a dynamic subgraph builder, a logic inferencer, and a hypothesis expander. The entity-relation extractor extracts entities, relations, and attributes from the output of the neural engine. The dynamic subgraph builder constructs contextual subgraphs based on extracted entity relationships and links them to the underlying knowledge graph; The logic inference engine performs fact verification and logical reasoning based on context subgraphs and rule bases; The expander assumes that it generates hypothetical entity relations when there is a knowledge gap in the subgraph.

2. The multimodal content generation method based on large models and knowledge graphs as described in claim 1, characterized in that, A neural-symbolic co-controller is used to achieve real-time interaction and closed-loop optimization between the neural engine and the symbolic engine.

3. The multimodal content generation method based on large models and knowledge graphs as described in claim 2, characterized in that, The triggering conditions for the neural symbolic collaborative controller are: when the neural engine generates entities and relationships, the collaborative controller automatically calls the symbolic engine to query the graph to confirm the facts; when the symbolic engine discovers a knowledge gap, the collaborative controller triggers the neural engine to generate hypothetical associations. The conflict resolution strategy of the neural symbolic co-controller is as follows: if the neural engine's ideas conflict with the graph facts, the neural symbolic co-controller generates a compromise solution and updates the graph by adding temporary hypotheses.

4. The multimodal content generation method based on large models and knowledge graphs as described in claim 3, characterized in that, The creative boundaries of the symbol engine are controlled by the creative divergence factor.

5. The multimodal content generation method based on large models and knowledge graphs as described in claim 4, characterized in that, The neural engine also includes a feedback optimization module, which receives user feedback on the generated multimodal content and updates the parameters of the creative generator based on the feedback.

6. The multimodal content generation method based on large models and knowledge graphs as described in claim 5, characterized in that, The hypothesis expander filters candidate hypothetical entity relationships by quantifying the reasonableness of the hypotheses; the reasonableness score of the candidate hypothetical entity relationship is used for each relationship. Calculate using the following formula: ； in It is the semantic consistency dimension score. It is the score for the rule compatibility dimension. It is the score of the context relevance dimension. It is a score based on the novelty of creativity. , , , These are the weights of the scores in the four dimensions.

7. A multimodal content generation system driven by large models and knowledge graphs, characterized by: To implement the multimodal content generation method based on large model and knowledge graph driven by any one of claims 1-6, the multimodal content generation system includes a multimodal input interface, a cross-modal preprocessing layer, a neural engine module, a symbol engine module, a neural symbol co-controller, a multimodal content generator, and a memory; The multimodal input interface is used to receive text, image, voice or 3D model input from users and convert it into a standardized data format; The cross-modal preprocessing layer extracts multimodal features based on the encoder of the neural engine, parses user needs, and maps them to a unified semantic space; The neural engine module integrates a multimodal large model and generates cross-modal original content prototypes that meet the content generation requirements. The symbol engine module adds non-factual but reasonable knowledge nodes to the original cross-modal content prototype based on creative needs to generate multimodal content corresponding to the needs. The memory is used to store user content generation requirement data, basic knowledge base data, and generated multimodal content data.

Images