A Method and System for Continuous Multimodal Content Generation Based on Knowledge State-Driven and Visual Semantic Navigation
By constructing a structured knowledge space and visual semantic parsing, combined with a multimodal generation model and a consistency evaluation mechanism, the problems of context drift and insufficient user interaction understanding in existing technologies have been solved. This has enabled the coherence, interactive intelligence, and generation reliability of popular science content, and improved the accuracy of scientific facts in the content.
Patent Information
- Application Number
- CN202611364029.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-09-04
- Publication Date
- 2026-11-10
AI Technical Summary
Existing methods for generating popular science content based on generative artificial intelligence lack explicit structured knowledge states, which can easily lead to context drift, knowledge topic jumps, and factual repetition between multiple rounds of generation. User interaction behaviors cannot be effectively understood, the generated content cannot be adapted to the cognitive levels of users of different age groups, and there is a lack of closed-loop evaluation and correction mechanisms for the generated results, making it difficult to guarantee the factual accuracy of popular science content.
By constructing a structured knowledge space, using visual semantic parsing and user interaction behavior to identify user-focused objects, and combining a multimodal generation model and a consistency evaluation mechanism, continuous generation of knowledge nodes and adaptive adjustment of content are achieved. A multidimensional consistency evaluation and failure type-driven condition correction mechanism are adopted to ensure the coherence and accuracy of the generated content.
It achieves knowledge logic self-consistency and interactive intelligence in the multi-round generation process, ensuring the coherence, cognitive adaptability and generation reliability of the content, improving the scientific accuracy of popular science content, and avoiding topic jumps and factual errors.
Smart Images

Figure FT_1 
Figure FT_2 
Figure FT_3
Abstract
Description
Technical Field
[0001] This invention relates to the fields of artificial intelligence and multimodal information processing technology, and more specifically, to a method and system for generating continuous multimodal content based on knowledge state-driven and visual semantic navigation. Background Technology
[0002] Existing methods for generating science popularization content based on generative artificial intelligence mostly adopt two modes: one is that after the user inputs a question, a large language model generates text, and then an image model generates accompanying illustrations; the other is that after the user inputs a topic, multiple pages of content are automatically generated and displayed in a predetermined order. Although the above solutions can reduce content production costs, their core is still one-time content generation.
[0003] Existing technologies mainly suffer from the following problems: the generation process lacks explicit structured knowledge states, making it prone to context drift, knowledge topic jumps, and factual repetition between multiple generation rounds; user interactions are only treated as page-turning events, failing to interpret the user's click position in the image as "what knowledge object the user is paying attention to"; the large language model, visual language model, and image generation model are called independently, lacking unified intermediate state constraints, making it difficult to guarantee semantic, knowledge, visual, and spatial continuity; there is a lack of a next-content decision mechanism based on the user's cognitive state, resulting in generated content that cannot adapt to the cognitive levels of users of different age groups; and there is a lack of a closed-loop evaluation and correction mechanism for the generated results, making it difficult to reliably guarantee the factual accuracy of popular science content. Summary of the Invention
[0004] In view of the above problems, the present invention provides a continuous multimodal content generation method based on knowledge state-driven and visual semantic navigation, the method comprising: Obtain user input information and initial user state, and construct a structured knowledge space based on the user input information. The knowledge space contains multiple knowledge nodes and semantic relationships between nodes. Based on the current knowledge state, a multimodal generation model is invoked to generate current multimodal content, and visual semantic parsing is performed on the current multimodal content to obtain a set of visual entities, and a mapping relationship is established between each visual entity and knowledge nodes in the knowledge space. The visual interaction behavior performed by the user in response to the current multimodal content is obtained, and the visual interaction behavior and the current knowledge state are input into the visual language model to identify the user's current focus and infer the true exploration intention. Based on the object of interest and the actual exploration intent, a set of candidate knowledge nodes is retrieved from the structured knowledge space. The candidate knowledge nodes are then comprehensively scored in conjunction with the initial user state and the current knowledge state to determine the next target knowledge node. Based on the next target knowledge node and the current knowledge state, construct multimodal generation conditions, and call at least one generation model to generate the next multimodal content; A multidimensional consistency evaluation is performed on the next multimodal content. When the evaluation result meets the preset conditions, the knowledge state is updated according to the current interaction result and the generation result, and the next multimodal content is provided to the user for subsequent interaction. When the evaluation result does not meet the preset conditions, the multimodal generation conditions are corrected according to the failure type and regenerated.
[0005] Furthermore, the present invention also provides a continuous multimodal content generation system based on knowledge state-driven and visual semantic navigation, which is used to store and execute the above-described methods.
[0006] Based on the above, the present invention has the following beneficial effects compared with the prior art.
[0007] First, this invention eliminates context drift and knowledge duplication between multiple generation rounds by constructing an explicit, structured knowledge state as the core intermediate variable. In existing solutions, each generation model relies solely on independent prompts, resulting in a lack of shared memory and a tendency for topic jumps or factual inconsistencies. This invention integrates information such as the current knowledge node, knowledge context, and historical exploration paths into state management, ensuring the self-consistency of knowledge logic and the orderly evolution of knowledge during continuous generation.
[0008] Second, this invention transforms user visual interactions such as clicking and selecting into semantic navigation signals within a knowledge space. In existing solutions, user clicks are merely treated as page-turning events, failing to reflect the user's need for knowledge exploration. This invention uses a visual language model to parse the visual entities corresponding to the interaction coordinates and employs a two-stage decoupling algorithm (object recognition plus intent inference) to transform "what the user clicked" into "what the user wants to know," making each interaction a deterministic semantic jump within the knowledge space. This achieves an upgrade in the interaction paradigm from passive page-turning to active exploration.
[0009] Third, this invention decouples knowledge decision-making from model invocation through a multimodal generation condition orchestrator, and ensures visual and spatial continuity across pages through iterative constraints of visual style state and spatial state, thus overcoming the problems of style drift and scene confusion caused by independent invocation of multiple models.
[0010] Fourth, this invention uses an eight-factor comprehensive scoring mechanism for candidate knowledge nodes, which simultaneously considers semantic relevance, knowledge gain, interest matching, age suitability, continuity, curiosity value, difficulty penalty, and repetition penalty. It also adaptively adjusts the content difficulty based on the user's cognitive state, thus achieving "relevant and suitable" content push and avoiding the generation defect of "relevant but age-inappropriate" in existing solutions.
[0011] Fifth, this invention achieves closed-loop verification and targeted repair of generated results through multi-dimensional consistency evaluation and failure type-driven condition correction mechanism, which significantly improves the accuracy of scientific facts in popular science content and overcomes the risk of factual errors caused by "generation and display" in existing solutions.
[0012] In summary, this invention transforms the traditional one-time AI content generation into a state-driven continuous generation process, achieving significant technological advancements in content coherence, interactive intelligence, cognitive adaptability, and generation reliability. Attached Figure Description
[0013] Figure 1 This is a flowchart illustrating the method provided by the present invention; Figure 2 This is an example of the initial knowledge scenario of the present invention: the principle of noon shadow measurement. Ancient people used a gnomon to measure the midday shadow. The summer solstice has the shortest shadow, from which the twenty-four solar terms were defined. On the page, "gnomon," "shortest shadow length," and other terms are visual entities that have been bound to knowledge nodes.
[0014] Figure 3 Example of an interactive knowledge scenario: the "no shadow" gnomon exploration scenario. The areas marked in the figure, such as Solar Angle and Gnomon Principle, Measurement Calibration Toolkit, Local Cultural Rituals, and Historical Shadow Recording, are all clickable visual entities that can be explored.
[0015] Figure 4: Knowledge node organization page for the "Three Phases of Summer Solstice". The three circular scenes correspond to the nodes "Deer Antlers Shedding", "Cicadas Begin to Sing", and "Pinellia Tuber Grows" in the knowledge graph, forming an explorable entry point.
[0016] Figure 5: The "Antler Removal" page. The page also includes knowledge explanations (the hormonal changes in deer are triggered by changes in day length), cultural semantics of yin and yang transformation, and new explorable entities such as "new antlers" and "shedding old antlers" for the next round of interaction.
[0017] Figure 6: "Cicadas Begin to Sing" page. The page includes annotations on the sound-producing mechanism and a diagram of underground molting, both of which are scientifically verifiable constraints for the evaluation module.
[0018] Figure 7: The "Pinellia ternata" page. The three leaflets, green spathe, and underground tubers are all visual entities bound to knowledge nodes.
[0019] Figure 8: Scene page of "Summer Solstice Earth Worship" (including knowledge entities such as Fangze Altar and Earth Worship Ritual).
[0020] Figure 9: Scene page for "Gifting Folding Fans and Cosmetics" (Knowledge entities such as folding fans for cooling off and cosmetics for preventing prickly heat can be further explored).
[0021] Figure 10: Scene page of the "Summer Solstice Noodles" food custom (knowledge nodes such as "Winter Solstice Dumplings, Summer Solstice Noodles", and tasting new wheat).
[0022] Figure 11: Scene page of "Weighing oneself on the summer solstice" (physical objects related to folk customs such as steelyard and prevention of summer heat).
[0023] Figure 12: The "Poetic Summer Solstice" scene page. The poem scroll, sundial, and water clock are all explorable entities bound to knowledge nodes.
[0024] Figure 13: Visual consistency comparison example: another generated result of the same theme gnomon scene, which is consistent with Figure 2 in terms of style, color, characters and composition. Detailed Implementation
[0025] like Figures 1-13 As shown, this technical solution provides a method and system for generating continuous multimodal content based on knowledge state-driven and visual semantic navigation. This invention can be used in scenarios such as children's science picture book generation, digital textbook production, interactive science exhibition content generation, virtual museum guides, and intelligent educational courseware generation. It is particularly suitable for scenarios requiring the continuous generation of multimodal content with semantic coherence, knowledge coherence, and visual coherence based on user interaction behavior.
[0026] The system architecture of this invention includes a multimodal input parsing module, a knowledge space construction module, a knowledge state management module, a multimodal scene planning module, a multimodal generation module, a multimodal semantic understanding module, a user interaction intent understanding module, a next knowledge node planning module, and a content consistency evaluation module.
[0027] The output of the multimodal input parsing module is connected to the input signal of the knowledge space construction module. The output of the multimodal input parsing module is also connected to the first input signal of the user interaction intent understanding module. The output of the knowledge space construction module is connected to the first input signal of the knowledge state management module. The output of the knowledge state management module is connected to the first input signal of the multimodal scene planning module. The output of the knowledge state management module is also connected to the second input signal of the multimodal semantic understanding module, the third input signal of the user interaction intent understanding module, the second input signal of the next knowledge node planning module, and the second input signal of the content consistency evaluation module. The output of the multimodal scene planning module is connected to the input signal of the multimodal generation module. The output of the multimodal generation module is connected to the first input signal of the multimodal semantic understanding module, and the first input signal of the content consistency evaluation module. The output of the multimodal semantic understanding module is connected to the second input signal of the user interaction intent understanding module, and also to the second input signal of the knowledge state management module. The output of the user interaction intent understanding module is connected to the first input signal of the next knowledge node planning module. The output of the next knowledge node planning module is connected to the second input signal of the multimodal scene planning module. The output of the content consistency evaluation module is connected to the third input signal of the knowledge state management module.
[0028] The types of signals transmitted between modules include unified multimodal event signals, knowledge graph data signals, current knowledge state signals, scene planning signals, multimodal content signals, entity knowledge node mapping signals, signals of objects of interest and exploration intentions, signals of the next target knowledge node, and evaluation result feedback signals.
[0029] In this invention, a knowledge node refers to a conceptual unit in a structured knowledge space, which is a vertex in a knowledge graph. Each knowledge node contains at least a node identifier, a node name, a node description, and a knowledge depth attribute.
[0030] A visual entity refers to an interactive visual object identified from multimodal content. Each visual entity includes at least an entity identifier, entity name, bounding box coordinates, semantic description, bounded knowledge node identifier, importance attribute, and explorable marker.
[0031] Knowledge state refers to the unified data structure maintained by the system in each interaction round, which is used to describe the current knowledge node, visual entity set, knowledge context, user cognitive state, visual style state, spatial state, and historical exploration path.
[0032] User cognitive state refers to the set of calculable variables that describe a user's cognitive level, including at least the user's age, current cognitive level, interest status, knowledge mastery status, set of difficult knowledge, and history of interaction behavior.
[0033] Visual style status refers to a set of quantifiable features that describe the visual presentation of multimodal content, including at least the art style category, main color palette, line drawing method, camera angle, composition layout, and main character image characteristics.
[0034] Spatial state refers to a set of quantifiable information describing the spatial layout of entities within a scene, including at least a list of entities, coordinates of each entity, spatial relationships between entities, camera angle, and display scale.
[0035] The method described in this invention is specifically implemented in the following steps. For ease of description, it is indexed by round. Indicates the interaction round, ,in This indicates the initial generation stage.
[0036] Step 1: Knowledge Space Construction The system obtains user input information through a multimodal input parsing module. and initial user status User input information In text format, it contains the topic or question the user hopes to explore, such as "Summer Solstice" or "Why is seawater salty?". Initial user state. It is a six-tuple data structure containing the user's age. Initial cognitive level Initial state of interest Initial knowledge mastery status Initial Difficulty Knowledge Set and the history of initial interaction behavior .
[0037] The knowledge space construction module receives user input information. Next, the structured knowledge space is constructed. This module first obtains relevant information from an external knowledge base. The relevant structured knowledge skeleton contains core concept nodes and known relationships between concepts. Then, the knowledge space construction module calls a large language model to semantically expand the knowledge skeleton, supplementing candidate knowledge nodes and their relationships. This semantic expansion process uses preset prompt word templates to guide the large language model's output. The knowledge graph is then expanded to include relevant nodes and relationships that do not conflict with the knowledge framework. Finally, the knowledge space construction module validates the expanded knowledge graph using knowledge consistency constraints, filtering out nodes that are inconsistent with the framework. Generate a semantic knowledge graph with relation types from irrelevant or conflicting nodes and relationships. .
[0038] knowledge graph middle, It is a collection of knowledge nodes, and each node contains at least a node identifier, a node name, a node description, and a depth attribute. It is a set of edges between nodes with relation types, which include at least causal relationship, temporal order relationship, spatial position relationship, comparative relationship, prerequisite dependency relationship and extended association relationship.
[0039] The knowledge space construction module outputs knowledge graph data signals to the knowledge status management module.
[0040] Step 2: Knowledge State Initialization After receiving the knowledge graph data signal, the knowledge state management module initializes the unified knowledge state. . For a 7-tuple, it is defined as: (1) The meanings of each component are as follows.
[0041] The current knowledge node represents the knowledge nodes displayed on the current page. .
[0042] This is a set of visual entities, representing the visual entities that have been identified and bound on the current page. .
[0043] The knowledge context includes a set of information such as parent nodes, visited nodes, candidate nodes, known nodes, and questionable nodes.
[0044] For user cognitive state, for six-tuple ,in For user age, At the current cognitive level, In an interest state, As a measure of knowledge mastery. It is a set of difficult knowledge and a history of interactive behaviors.
[0045] For visual aspects, it includes stylistic information such as art style, color palette, lines, camera angles, composition, and characters.
[0046] This refers to the spatial state, which includes information about entities within the scene, their locations, spatial relationships, camera angles, and scale.
[0047] This represents the historical exploration state, including the user's exploration path and the generated history. .
[0048] During the initialization phase, For knowledge graphs Zhongyu The most directly corresponding root node, An empty set Include The first-level child node of the root node, To obtain the initial user state, The system's default visual style or the user's pre-selected style. This is the system's default space layout, and it represents the initial empty path.
[0049] in, Corresponding to the current_node field; This is a set of visual entities. Each entity in this set is bound to a knowledge node through the mapping relationship established in step four. The knowledge node bound to the visual entity contains parent_node and visited_nodes information. It contains information about parent_node, visited_nodes, candidate_nodes, mastered_nodes, uncertain_nodes, and semantic_context; middle, Corresponding to the user_interest field, Corresponding to the difficulty_level field, The knowledge mastery status corresponding to mastered_nodes; Corresponding to the visual_context field, Corresponding to the spatial_context field; It contains the generation_history field.
[0050] The state transition equation for the evolution of knowledge state with each interaction round is: (2) in For the user's current multimodal interaction behavior, The results of this round of generation and evaluation, The state update operator.
[0051] Step 3: Initial Multimodal Content Generation The multimodal scene planning module obtains the current knowledge state signal from the knowledge state management module, based on... Generate initial scene planning. Scene planning is structured data, which includes at least the main concept, supporting concepts, a list of visual entities, a set of facts, causal relationships, temporal relationships, layout scheme, camera parameters, style parameters, and interactive area definitions.
[0052] After receiving the scene planning signal, the multimodal generation module calls the multimodal generation model to generate initial multimodal content. The multimodal generation module includes at least one of the following: text generation model, image generation model, video generation model, and speech generation model. For the picture book generation scenario, the multimodal generation module calls the large language model to generate page description text and the image generation model to generate page images based on visual instructions, entity constraints, and style parameters in the scene planning. During the generation process, the multimodal generation module simultaneously outputs a structured set of visual entities, each defined according to the following data structure: (3) in As a unique identifier for an entity, For entity name, These are the bounding box coordinates of the entity within the page. For semantic description, For the bound knowledge node identifier, As importance weight, Mark whether the entity is interactively explorable. Bounding box coordinates. Defined as: (4) in and The x and y coordinates of the top-left corner of the bounding box in the page image. and These are the width and height of the bounding box, respectively, both in pixels.
[0053] For example, in a specific instance, when the system generates the page corresponding to the "Cicadas Begin to Sing" knowledge node, the cicada entity in the visual entity set is represented by the above data structure as follows: entity identifier is E17, entity name is "Cicada", and bounding box coordinates are... The bound knowledge node is "Cicadas Begin to Sing," and the explorable flag is set to 1, indicating that this entity can be clicked by the user to trigger further exploration. (The above...) A value of 1 indicates that the entity can be interactively explored, while a value of 0 indicates that it cannot be interactively explored. The system controls the generation and response of clickable areas on the page based on this flag.
[0054] Step 4: Visual semantic parsing and entity knowledge node alignment The multimodal semantic understanding module receives the multimodal content signal output by the multimodal generation module, performs visual semantic parsing on the generated page image, and obtains a set of visual entities. The multimodal semantic understanding module uses a visual language model to perform object detection and semantic recognition on page images, outputting the name, bounding box coordinates, and semantic description of each visual entity.
[0055] Then, the multimodal semantic understanding module establishes the mapping relationship between each visual entity and knowledge nodes in the knowledge space. Let the visual entity be... The knowledge node is Calculate the four-way weighted similarity: (5) in ,and All are greater than or equal to 0. The meaning and calculation method of each component are as follows.
[0056] The textual semantic similarity between entity names and descriptions and node texts is obtained by encoding entity names and descriptions and knowledge node names and descriptions into semantic vectors and then calculating the cosine similarity.
[0057] The visual feature similarity between entity image region features and node reference visual features is calculated by encoding the image features of the entity bounding box region with the pre-stored reference visual features of the knowledge node.
[0058] The context similarity between the scene context of an entity and the context associated with a node is obtained by comparing the scene description vector of the page where the entity is located with the neighborhood description vector of the knowledge node in the knowledge graph.
[0059] To determine the graph relation matching degree between the adjacency relationships of entities in the knowledge space and the node graph structure, the algorithm calculates the graph relation matching degree between candidate nodes of entity mappings and the current knowledge node. The shortest path distance and relation type matching degree are obtained.
[0060] For each visual entity The multimodal semantic understanding module selects the candidate knowledge nodes with the highest weighted similarity: (6) If the highest similarity exceeds the preset mapping threshold ,Right now Then, a definite mapping is established from the visual entity to the knowledge node. If the highest similarity does not exceed If so, mark the entity as pending manual review or temporarily unbind.
[0061] A preset mapping threshold is set, ranging from 0.6 to 0.8, preferably 0.7. Once the mapping is established, the set of visual entities in the knowledge state... Updated to a complete set of entities containing mapping information, within the entity data structure. The field is filled with the corresponding knowledge node identifier.
[0062] The multimodal semantic understanding module outputs entity knowledge node mapping signals to the knowledge state management module and the user interaction intent understanding module.
[0063] The complete process of the above entity-knowledge node alignment is as follows: First, initialize the mapping set. The set is empty; then the visual entity set is traversed. Each visual entity in Traverse the knowledge graph for this entity Each candidate knowledge node in Calculate the four-way weighted similarity according to formula (5). After traversal, select the candidate knowledge node with the highest similarity. If the highest similarity Greater than the preset mapping threshold Then the mapping Add to map set If the highest similarity does not exceed the threshold, the entity is marked as awaiting manual review or temporarily not bound; after traversing all visual entities, the mapping set is output. .
[0064] Step 5: User Interaction Acquisition and Interaction Behavior Representation The system uses a multimodal input parsing module to acquire the user's visual interaction behaviors with the current multimodal content. Visual interaction behaviors include at least one of clicking, selecting, dragging, and natural language follow-up questions. For clicking, the multimodal input parsing module records the user's click coordinates on the page image. For selection actions, record the bounding box coordinates of the selected area.
[0065] The multimodal input parsing module represents user interaction behavior as unified multimodal event signals. : (7) in The image for the current page. As the current state of knowledge, This is structured interactive event data.
[0066] The multimodal input parsing module outputs a unified multimodal event signal. To the user interaction intent understanding module.
[0067] Step Six: Identifying the User's Currently Focused Interests After receiving the unified multimodal event signal, the user interaction intent understanding module will display the current page image. User interaction coordinates and current knowledge status The visual language model is input together to perform the identification of the user's currently interested object. The visual language model outputs candidate interested entities, semantic descriptions, initial exploration intent, and identification confidence scores. (8) in For user-focused entities identified by the visual language model, For semantic description text, The initial exploration intent output by the visual language model. To identify the confidence level, the value is between 0 and 1.
[0068] The user interaction intent understanding module determines the confidence level based on the recognition. Compared with the preset confidence threshold The size relationship determines how the object of focus is ultimately determined.
[0069] when At that time, the user interaction intent understanding module receives the output of the visual language model and... As the user's current focus. The reliability threshold is preset, and its value ranges from 0.7 to 0.9, preferably 0.8.
[0070] when At this point, the user interaction intent understanding module degenerates into a bounding box hit test. The user interaction intent understanding module then iterates through the bounding box coordinates of all visual entities on the current page. Select the one that satisfies The visual entity with the smallest area is selected as the user's current focus. If no entity's bounding box contains the clicked coordinates, the interaction is considered invalid and will not trigger subsequent generation processes.
[0071] Step 7: Inferring the User's True Exploration Intent After obtaining the user's current focus object, the user interaction intent understanding module performs an inference of the user's true exploration intent. This inference process decouples object recognition and intent inference into two independent stages that are executed sequentially.
[0072] The user interaction intent understanding module obtains the object of interest. In structured knowledge space Location information, access history of knowledge related to the object of interest, and recent continuous interaction patterns of the user. Location information includes the knowledge node corresponding to the object of interest. The knowledge node is defined by its depth, parent nodes, and child nodes. The visited record includes whether the knowledge node has been visited, the number of visits, and the last visit time. The user's recent continuous interaction behavior pattern includes the sequence of objects of interest in the most recent interactions and the duration of each interaction.
[0073] The user interaction intent understanding module combines the current user's age and cognitive state to calculate the true exploration intent: (9) in This is an intent inference function. Its internal logic is a weighted comprehensive judgment based on the position of the object of interest in the knowledge graph, the access status of related knowledge, user interaction patterns, and the user's cognitive state. The same object of interest corresponds to different true exploration intentions under different knowledge states and different interaction histories. When a user clicks on "Cicada" and that knowledge node has not yet been visited, It could be "understanding the basic characteristics of cicadas"; when the basic characteristics of cicadas have been accessed but the "cicada's vocalization mechanism" has not, It might be "understanding the principles behind cicada sound production".
[0074] The user interaction intent understanding module outputs the focus and exploration intent signals to the next knowledge node planning module.
[0075] The core logic of this two-stage separation mechanism lies in the fact that object recognition answers "what the user clicked," while intent inference answers "what the user wants to know." Object recognition only identifies the visual entity to which the user's interaction points, while intent inference, combined with knowledge state, interaction history, and the user's cognitive state, further determines the deeper knowledge direction the user wishes to acquire. Through this two-stage decoupling, the same object of interest can correspond to different genuine exploration intentions under different knowledge states and different interaction histories. This triggers the generation of the next content at different depths and directions in the subsequent retrieval and scoring steps eight to nine, avoiding repetition of previously presented content and enabling continuous and in-depth exploration of knowledge.
[0076] Step 8: Candidate Knowledge Node Retrieval After receiving signals of the object of interest and exploration intent, the next knowledge node planning module starts from the structured knowledge space. Search and Current Interests and the true intention of exploration Relevant candidate knowledge nodes: (10) in The candidate retrieval function has the following internal logic: First, in Mid-positioning and Map the knowledge nodes, then traverse all adjacent nodes within a preset number of hops (preferably 2 hops) using that node as the center, and then... The semantic direction performs preliminary semantic filtering on these adjacent nodes, and the output is consistent with... Candidate node set for semantic direction matching . The number of candidate nodes varies with the density of the knowledge graph.
[0077] Step Nine: Comprehensive Scoring of Candidate Knowledge Nodes and Determination of Target Nodes The next knowledge node planning module combines the initial user state. and current knowledge status For the candidate node set Each candidate knowledge node is comprehensively scored.
[0078] For each candidate knowledge node At the same time, the following eight scoring factors are calculated.
[0079] The semantic relevance factor represents the semantic association strength between the clicked object and the candidate knowledge. It is obtained by calculating the cosine similarity between the embedding vector of the knowledge node corresponding to the clicked object and the embedding vector of the candidate node.
[0080] The knowledge gain factor represents the amount by which a candidate node expands upon the currently learned knowledge system. It is calculated by... The ratio of the number of unvisited neighbor nodes to the total number of neighbor nodes is obtained.
[0081] The interest matching factor represents the match between candidate nodes and the user's current interest state. The matching degree is obtained by calculating the similarity between the topic tag vector of the candidate node and the user interest vector.
[0082] Age-adaptation factor, representing the relationship between the difficulty of candidate node content and the user's age. The suitability is obtained by comparing the preset difficulty level of the candidate node with the recommended difficulty level corresponding to the user's age. The value is higher when the difficulty matches and lower when the difficulty does not match.
[0083] The semantic continuity factor represents the degree of connection between candidate nodes and the knowledge logic of the current page. It is calculated by comparing candidate nodes with the current knowledge nodes. exist The relationship type weight and path length are obtained, and the nodes of directly connected causal or temporal relationships have the highest values.
[0084] The curiosity value factor represents the potential of candidate nodes to inspire further exploration, and its calculation formula is as follows: (11) in The set of visited nodes. Candidate nodes In the depth of a knowledge graph, the more unexplored branches a candidate node has and the shallower its level, the higher its curiosity value.
[0085] This is a difficulty penalty factor, indicating that the difficulty of the candidate node content exceeds the current cognitive level. The degree of difficulty is determined by calculating the preset difficulty level of candidate nodes and the user's current cognitive level. The difference is obtained, and when the difference is positive, the value is greater than 0.
[0086] The duplication penalty factor represents the degree of overlap between candidate nodes and visited or mastered nodes. It is calculated by... and the set of nodes already known The semantic overlap is obtained.
[0087] The next knowledge node planning module performs a weighted summation of the above eight factors to obtain a comprehensive score for each candidate knowledge node: (12) in to Corresponding to , , , , and . to These are preset weighting coefficients, all non-negative, and are calibrated based on the user group and application scenario. For scenarios involving young children, (Age-appropriate) and The weighting of (interest matching) is set relatively high. For scenarios involving older students, (Knowledge gain) and The weighting of (curiosity value) is set relatively high.
[0088] The next knowledge node planning module selects the candidate knowledge node with the highest comprehensive score that has not been visited as the next target knowledge node: (13) in This is the set of visited nodes, ensuring that the selected target node has not been explored, thus avoiding knowledge duplication.
[0089] The next knowledge node planning module outputs the next target knowledge node signal to the multimodal scenario planning module.
[0090] Step 10: Multimodal generation condition orchestration After receiving the next target knowledge node signal, the multimodal scene planning module performs multimodal generation condition orchestration. This module, based on the next target knowledge node... Current knowledge status User status Current visual style status and current spatial state Constructing structured multimodal generation conditions : (14) in To generate conditional programming functions. For structured data, it should include at least semantic instructions, visual instructions, layout instructions, shot instructions, style instructions, entity constraints, factual constraints, age constraints, and interaction constraints.
[0091] Semantic instructions describe the core and supporting concepts that the target knowledge node needs to convey, and are determined by the large language model based on... and in The generation of neighborhood information.
[0092] The visual instructions describe the visual entities and their attributes to be presented, which are determined by the multimodal scene planning module. Generate associated visual entity templates.
[0093] Layout instructions describe how pages are laid out, including the relative positions of entities on the page.
[0094] Camera instructions describe the framing of the shot, including wide shot, medium shot, close-up, or isometric overhead view.
[0095] Style instructions include the current visual style state. All parameters are used to constrain the style output of the image generation model.
[0096] Entity constraints specify a list of visual entities that must appear on the page and their explorability markers.
[0097] Fact constraints specify the scientific facts that must be correctly presented on the page, represented as a structured list of assertions, each of which contains a subject, predicate, object, and relation type.
[0098] Age constraints specify the language difficulty level and explanation depth of the page content, based on the user's age. and current cognitive level Sure.
[0099] Interaction constraints specify the interactive areas on a page and their corresponding triggering behaviors.
[0100] The multimodal scene planning module outputs scene planning signals to the multimodal generation module.
[0101] The above The correspondence between the components and the elements of the ScenePlan is as follows: Semantic instructions include main_concept (core concept) and supporting_concepts (supporting concepts), and are based on... In knowledge graph The neighborhood information provides causal relations and temporal relations; visual instructions include visual entities; fact constraints include facts; layout instructions correspond to layout; camera instructions correspond to camera; style instructions correspond to style; and interaction constraints correspond to interaction regions.
[0102] Step 11: Controlling Visual and Spatial Continuity Across Pages When constructing multimodal generation conditions, the multimodal scene planning module performs cross-page visual continuity and spatial continuity control.
[0103] For visual continuity control, the multimodal scene planning module pre-extracts and maintains the visual style state of the current page. : (15) in As for art style category, Main color scheme For the method of outlining with lines, For lens angle parameters, For the composition and layout, These are the main character traits.
[0104] When constructing the generation conditions for the next multimodal content, the multimodal scene planning module will... As anchoring input conditions for the image generation model, it ensures that the next page is consistent with the current page in terms of style, color, lines, camera angles, composition, and character appearance.
[0105] After the next multimodal content is generated, the multimodal semantic understanding module extracts its actual style features in reverse to obtain the updated visual style state: (16) in This is a style feature extraction function that uses a visual language model to encode the style of the generated image, outputting a structured style state. The updated... Used for the continuous generation of subsequent pages.
[0106] For spatial continuity control, the multimodal scene planning module maintains the spatial state. : (17) in This is a list of entities that appear in the scene. These are the relative position coordinates of each entity in the scene. For spatial relationships between entities, For observation angle parameters, To display the scale ratio.
[0107] When the multimodal generation module generates content with spatial attributes, the multimodal scene planning module forces the spatial state of the next page to remain continuous with the current spatial state, ensuring that the relative positions of entities within the scene do not change abruptly.
[0108] Step 12: Generation of the next multimodal content After receiving the scene planning signal, the multimodal generation module, according to... Different parts are distributed to different generative models to perform multi-model collaborative generation. The large language model is responsible for generating text content, the visual language model is responsible for validating visual conditions, the image generation model is responsible for generating page images, and the video generation model is responsible for generating animation clips.
[0109] After calling the various generated models, the multimodal generation module merges the outputs of each model into multimodal content. : (18) in For multimodal generation functions, It contains at least one page image and its corresponding set of structured visual entities.
[0110] The multimodal generation module outputs multimodal content signals to the multimodal semantic understanding module and the content consistency evaluation module.
[0111] Step Thirteen: Multidimensional Consistency Evaluation After receiving the multimodal content signal, the content consistency evaluation module evaluates the next multimodal content. Perform a multidimensional consistency evaluation. The content consistency evaluation module uses a visual language model or a dedicated evaluation model to calculate the following five-dimensional evaluation vectors: (19) in This is the evaluation function.
[0112] Score semantic consistency and evaluate. Does the text content match the target knowledge node? The core meaning is consistent, and it is achieved by calculating the page description text and... The semantic embedding similarity is obtained.
[0113] Rate and evaluate scientific facts. Are the scientific facts in it correct? The content consistency evaluation module starts from... Extract a set of verifiable facts and combine it with a structured knowledge space. Compare with standard knowledge in the text: (20) in This is a fact extraction function that uses a visual language model to identify and extract factual assertions from page images. This is a fact comparison function that compares the extracted facts with... The system matches the standard knowledge in the data to calculate the proportion of correct facts to all facts.
[0114] Score the visual style consistency and evaluate. Does the visual style match the current visual style status? Consistent, through calculation Style feature vectors and The similarity of the style feature vectors is obtained.
[0115] Age-appropriateness rating, evaluation Is the difficulty of the content related to the user's age? and current cognitive level Adaptation is achieved by calculating the degree of matching between the difficulty level of the page content and the difficulty level recommended by the user.
[0116] Scoring and evaluation of knowledge continuity Relative to the previous page The degree of logical coherence in knowledge is calculated. and exist Based on the relationship type and path distance, the nodes with direct causal or temporal relationships have the highest continuity.
[0117] Step Fourteen: Evaluation Result Determination and Failure-Driven Regeneration The content consistency evaluation module compares the scores for each dimension with corresponding preset thresholds. When the scores for all dimensions are greater than or equal to the corresponding thresholds, i.e. The evaluation was deemed satisfactory. The preset evaluation threshold is set, and its value ranges from 0.7 to 0.9.
[0118] When the evaluation is passed, the content consistency evaluation module outputs an evaluation pass signal to the knowledge status management module.
[0119] When the score in any dimension falls below the corresponding threshold, the content consistency evaluation module determines that the evaluation fails and outputs the specific reason for failure. : (twenty one) The meanings of each failure reason type are as follows.
[0120] This indicates a semantic error, meaning that the text content on the page is inconsistent with the core meaning of the target knowledge node.
[0121] This indicates a scientific fact error, meaning that the facts presented on the page do not match the standard knowledge in the knowledge graph.
[0122] This indicates a visual error, meaning that the page image has visual distortion, missing entities, or misaligned entities.
[0123] This indicates a style error, meaning the visual style of the page is inconsistent with the previous page.
[0124] This indicates a layout error, meaning the page layout is unreasonable or the interactive areas are disordered.
[0125] This indicates a difficulty error, meaning the page content's difficulty level does not match the user's cognitive level.
[0126] This indicates a continuity error, meaning that the knowledge on this page jumps too much from the knowledge on the previous page and lacks logical connection.
[0127] The content consistency evaluation module modifies different sub-items in the multimodal generation conditions according to different failure reason types: (twenty two) in This is a conditional repair function.
[0128] when for hour, Correction The fact constraint sub-items in the text supplement or correct factual assertions.
[0129] when for hour, strengthen The style constraint sub-item enhances the style anchoring parameters.
[0130] when for hour, Correction Adjust the language difficulty level within the age constraint sub-item.
[0131] when for hour, Correction The layout instructions and interactive constraint sub-items in the text.
[0132] The multimodal generation module receives the corrected generation conditions. The multimodal content is then regenerated. The content consistency evaluation module performs the evaluation again until the evaluation passes or the preset maximum number of retries is reached. . It is a preset positive integer, preferably 3.
[0133] If it reaches If the evaluation still fails, the multimodal generation module outputs the most recent generation result and marks it for manual review.
[0134] Step 15: Knowledge Status Update When the evaluation result meets the preset conditions, the knowledge state management module updates the knowledge state according to the current interaction result and the generated result, following the state transition equation. Update.
[0135] The specific updates include: Set as the new current knowledge node, The set of visual entities in the middle is updated to ,Will The set of visited nodes in the data increases User cognitive state Updated to Visual state Updated to , spatial state Updated to , will explore the path Updated to .
[0136] User cognitive status The update is performed as follows: (twenty three) in This includes user behavior data observed by the system, such as page dwell time, whether there are repeated clicks, and whether the user returns to the previous page. When a user's page dwell time exceeds a preset threshold, the system increases the mastery score of the knowledge nodes on that page. When a user clicks on the same entity repeatedly, the system adjusts the interest weight of the related knowledge for that entity.
[0137] The knowledge status management module outputs an updated knowledge status signal for use in the next round of interaction.
[0138] Step Sixteen: Optimize Pre-Inference Caching In generating the current multimodal content Afterwards, the multimodal semantic understanding module performs pre-inference caching optimization in the background.
[0139] The multimodal semantic understanding module uses a visual language model to analyze the current page. Perform click probability pre-analysis on all visual entities to predict the preset number. The most likely interactive entities that users will click: (twenty four) in The click probability prediction function is based on the entity's visual salience on the page, the strength of its semantic association with the current knowledge node, and the entity's explorable markers. And calculate the probability of each entity being clicked based on the user's historical interests. It is a preset positive integer, preferably 3 to 5.
[0140] The multimodal semantic understanding module pre-caches the corresponding semantic description, associated candidate knowledge nodes, and current generation context for each high-probability interactive entity. The cached content includes at least the entity name, entity bounding box coordinates, pre-computed semantic description vector, pre-retrieved candidate knowledge node list, current visual style state, and current spatial state.
[0141] When the user's actual click location hits the bounding box region corresponding to a high-probability interactive entity, the user interaction intent understanding module directly reuses the cached semantic description, associated candidate knowledge nodes, and generation context, skipping the real-time visual language model re-parsing step and directly entering the candidate node scoring and selection stage.
[0142] This caching mechanism reduces interaction latency from seconds to milliseconds and reduces repeated calls to the visual language model, thereby lowering the cost of model calls.
[0143] Step Seventeen: Actively Explore and Recommend In addition to responding to user interactions, the system also performs proactive exploration and recommendation. The next knowledge node planning module selects from the current set of candidate knowledge nodes. The candidate node with the highest score is selected as the active recommendation node: (25) The multimodal scene planning module generates recommendation prompts based on proactively recommended nodes, including the name of the recommended node and a brief exploration guide. The multimodal generation module integrates the recommendation prompts into the current page, prompting the user through visual highlighting or text guidance.
[0144] This active recommendation and passive interaction together form a dual-channel exploration mechanism, guiding users to systematically learn along the optimal path in the knowledge space.
[0145] Algorithm Flow Overview The complete system process can be summarized as the following continuous state-driven generation sequence: (26) Starting from an initial knowledge state, the process involves updating the state after receiving user interactions, determining the next target node, generating multimodal content, performing evaluation and state updates, and repeating this cycle. Throughout the entire process, a unified knowledge state is maintained. As a core intermediate variable, it runs through the entire process of knowledge space construction, multimodal content generation, visual semantic parsing, user interaction understanding, candidate node scoring, generation condition arrangement, and closed-loop evaluation.
[0146] Taking the generation of a popular science picture book about the "Summer Solstice" of the Twenty-Four Solar Terms as an example, after the system receives the user's topic "Summer Solstice" and initial user status, the knowledge space construction module generates a knowledge graph containing heterogeneous knowledge nodes such as astronomy, biology, folklore, and literature. This knowledge graph The top-level structure is as follows: the root node "Summer Solstice" has four first-level branches. The astronomical branch contains three sub-nodes: no shadow cast by a vertical pole, noon shadow measurement, and solar altitude angle; the three pentads branch contains three sub-nodes: deer antlers shedding, cicadas beginning to sing, and pinellia sprouting; the folk customs branch contains four sub-nodes: earth worship, giving folding fans and cosmetics, summer solstice noodles, and weighing oneself; the poetic branch contains sub-nodes containing summer solstice poetry and imagery. The system initializes its knowledge state. Subsequently, the multimodal scene planning module transforms the root node "Summer Solstice" and its first-level child nodes into scene plans, while the multimodal generation module first generates the "Noon Shadow Measurement Principle Page" as the initial multimodal content. This page presents the knowledge of ancient people using a gnomon to measure the midday shadow and the shortest shadow at the summer solstice. The visual entities on the page, such as the gnomon pole, the ruler, and the shadow, have been mapped by the multimodal semantic understanding module to the corresponding knowledge nodes in the knowledge graph. Subsequently, the multimodal generation module generates a "no shadow" gnomon exploration scene. This scene includes visual entities such as a calibrated gnomon, a ruler, a measuring tool kit, and a shadow recording scroll. Each entity is structurally defined according to formula (3) and marked as interactive to explore, providing users with an interactive entry point from the astronomical branch to the three pentads branch or the folk custom branch. Knowledge state management module initialization Subsequently, the multimodal generation module generates the initial page "Noon Shadow Measurement Principle Page". This page contains visual entities such as the gnomon, the ruler, and the shortest shadow length. Each entity has been mapped to the corresponding node in the knowledge graph through the multimodal semantic understanding module.
[0147] When a user clicks on the "antlers" area on the page, the user interaction intent understanding module identifies the object of interest as "antlers" and infers the exploration intent as "understanding the reasons for the changes in antlers at the summer solstice." The next knowledge node planning module retrieves candidate nodes from the knowledge graph and scores them, determining the target node "Antler Solution." The multimodal scene planning module arranges the generation conditions, and the multimodal generation module generates the "Antler Solution" page. The content consistency evaluation module verifies the scientific factual accuracy of the page, confirming that the description of the relationship between antler shedding and changes in day and night length is correct, updates the knowledge status, and proceeds to the next interaction round. Each consecutive page maintains its visual style status. Maintain consistency in art style, color, and characters under iterative constraints.
[0148] The method and system described in this invention eliminate context drift and knowledge duplication through explicit knowledge states, ensuring semantic consistency, knowledge consistency, visual consistency, and spatial consistency throughout the continuous generation process. By transforming user image interactions into semantic navigation signals in the knowledge space, explorable content consumption is achieved. An eight-factor scoring mechanism for candidate nodes balances relevance, age suitability, and knowledge gain, avoiding the recommendation of relevant but unsuitable content. Multi-dimensional evaluation and failure-type-driven conditional correction significantly improve the scientific accuracy and stylistic stability of the generated content. A pre-inference caching mechanism reduces real-time interaction latency and model invocation costs.
[0149] It needs further clarification, firstly, regarding the specific collaborative mechanism by which the knowledge space construction module utilizes a large language model to semantically expand the knowledge skeleton in step one. As described in step one, the knowledge space construction module employs a three-stage collaborative approach to generate the knowledge graph: "external knowledge base defines the skeleton, large language model performs expansion, and consistency constraints perform verification." In this process, the semantic expansion of the large language model is not generated freely, but rather constrained by a structured knowledge skeleton provided by an external knowledge base. The expanded candidate nodes and relationships must undergo knowledge consistency constraint verification and be filtered to match user input information. Only irrelevant or conflicting nodes and relationships are incorporated into the final knowledge graph. This three-stage collaborative mechanism ensures the credibility of the knowledge framework and avoids the risk of unreliable knowledge arising from relying entirely on large language models to freely generate knowledge paths.
[0150] Second, regarding the four-way weighted similarity mentioned in step four. Chinese map relationship matching degree The specific calculation method is shown in formula (5). This represents the graph relation matching degree between the adjacency relationships of entities in the knowledge space and the node graph structure. The specific calculation method is as follows: first, in the knowledge graph... Mid-position entity The mapped candidate knowledge nodes and target knowledge nodes Then calculate the two in The shortest path distance is used to calculate the similarity score, and different matching weights are assigned to different relationship types along the path (such as causal relationships, temporal relationships, etc.). Finally, the combined value of the path distance and relationship type weights is mapped to a similarity score between 0 and 1. The shorter the path and the higher the relationship type matching degree, the higher the similarity score. The higher the value, the better.
[0151] Third, regarding the curiosity value factor mentioned in step nine. Further explanation of formula (11). As shown in formula (11), The numerator of this formula represents the candidate node. In knowledge graph The number of all unvisited neighbor nodes in the denominator reflects the absolute number of "unexplored branches" that the node can provide; Indicates candidate nodes In a knowledge graph, the depth is the length of the shortest path from the root node to that node. The denominator uses... This format allows nodes at shallower levels of the knowledge graph to receive higher curiosity value scores even if they have the same number of unexplored branches. This guides the system to prioritize recommending fundamental and pivotal knowledge nodes, helping users build a more complete knowledge framework.
[0152] Fourth, regarding the weighting coefficients of the eight scoring factors mentioned in step nine. to The calibration method is shown in formula (12). to These are preset weighting coefficients, all non-negative, and are calibrated based on the user group and application scenario. In a preferred embodiment, for science education scenarios for children aged 3 to 6, (Interest matching) and The weight for (age fit) is set to a higher value. (Knowledge gain) and The weight of (curiosity value) is set to a medium value; for the scenario of older students aged 7 to 10, (Knowledge gain) and The weight of (curiosity value) is set to a high value. The weight for (age-appropriateness) is set to a moderate value. The specific values of each weight coefficient can be calibrated through pre-experiments targeting the target user group, or dynamically optimized during system operation through reinforcement learning.
[0153] Fifth, regarding the comprehensive scoring formula (12) mentioned in step nine. to The correspondence with the eight factors is shown in formula (12). .in to Corresponding in sequence (Semantic relevance factor) (Knowledge gain factor) (Interest matching factor) (Age-appropriate factor) (Semantic continuity factor) and (Curiosity Value Factor). (Difficulty penalty factor) and (Repetition penalty factor) is included in the comprehensive score in the form of deduction, and candidate nodes that are more difficult than the user’s current cognitive level or that are duplicates of the knowledge already accessed are penalized.
[0154] Sixth, regarding the user cognitive state update function mentioned in step fifteen above. The specific update logic is shown in formula (23). .in This refers to user behavior data observed by the system. The specific update rules for the function include: when the user stays on a page for more than a preset time threshold (preferably 8 seconds), the system will update the knowledge nodes carried on that page. Mark it as "understood" and improve the user's awareness. The corresponding mastery score When a user clicks on the same visual entity more than twice, the system marks the knowledge node associated with that entity as having "high interest" and increases its visibility. The corresponding interest weights When a user triggers the "return to previous page" action, the system will display the current knowledge node. The status is marked as "questionable," and the user's mastery score is lowered. These update rules together constitute a dynamic adjustment mechanism for the user's cognitive status.
[0155] Seventh, regarding the control method for cross-page semantic continuity described in step eleven above. As described in step eleven, the system performs cross-page visual continuity and spatial continuity control when constructing multimodal generation conditions. Based on this, the system simultaneously performs cross-page semantic continuity control. Specifically, when generating the next page, the system inputs semantic connection constraints, including the current knowledge node. Next target knowledge node Current knowledge context as well as and In knowledge graph Relationship types in The system requires that the generated results satisfy the semantic continuity condition. ,in A preset semantic continuity threshold is set, with a value ranging from 0.6 to 0.8, preferably 0.7, to prevent content from jumping from the current topic to an unrelated topic.
[0156] Eighth, regarding the click probability prediction function mentioned in step sixteen above. The specific calculation logic is shown in formula (24). . The function calculates the click probability of each visual entity based on the following four dimensions: the entity's visual salience on the page (obtained by calculating the proportion of the entity's bounding box area to the total page area and the entity's centering position on the page), the entity's relationship to the current knowledge node, and the relationship between the entity and the current knowledge node. Semantic association strength (by calculating entity name and Semantic embedding similarity of names, and explorable markers of entities. (Entities marked as explorable receive a higher click probability), and the matching degree between user's historical interests and entity topics (by calculating entity topic tags and user interest status). (The degree of matching is obtained). After weighted summation of the scores of the above four dimensions, the top-ranked scores are selected. Entities with a specific bit value are cached as high-probability interaction entities. The preferred value is 3 to 5.
[0157] Ninth, regarding the mechanism of "joint modeling of knowledge graph and page graph and exploration path learning". The system maintains two graph structures simultaneously during operation: the knowledge graph... Describe the semantic relationships between concepts, page diagram This describes the actual sequence of pages accessed by the user and the navigation relationships. The two are mapped together. Establish connections, that is, map each page visited by the user to a corresponding knowledge node. The user's exploration path is as follows: The system builds an individualized interest model based on the history of this path to predict the conditional probability of the next knowledge node. This is used for candidate node scoring optimization in step nine and proactive exploration recommendation in step seventeen. This dual-graph joint modeling mechanism unifies the three layers of "knowledge world," "page world," and "user's actual exploration path," enabling the system to dynamically adjust recommendation strategies based on the user's exploration history.
[0158] The aforementioned steps together constitute the core innovative thread of this invention. Specifically, when the system determines the multimodal content to be generated in the next round of interaction, it is not directly determined by the user's original question, but by the current knowledge state. User interaction semantics Candidate knowledge space User cognitive status and the current generation state (including visual style state) and spatial state This was decided jointly. The entire system was formed. The two-way closed loop works as follows: Before generation, knowledge space modeling and knowledge state initialization are performed through steps one and two; after generation, visual entity recognition and reverse modeling are performed on the generated multimodal content through step four, re-anchoring the visual entities in the image to specific nodes in the knowledge space. The knowledge space modeling before generation and the reverse modeling of visual entities after generation are integrated into a unified knowledge state. (Formula (1)) is closed. This state serves as the core intermediate variable throughout the entire process from step one to step seventeen, making the system different from the traditional one-time generation scheme and realizing state-driven continuous multimodal content generation.
[0159] The technical solution comprised of the aforementioned steps differs fundamentally from ordinary AI content generation solutions. Regarding input, this invention simultaneously acquires the user's topic, user state, and knowledge state, rather than just the topic; regarding generation, this invention uses a unified knowledge state... This invention drives continuous generation, rather than one-time generation; in terms of driving method, it uses knowledge states rather than text prompts as the core driver; user clicks are interpreted as semantic navigation signals in the knowledge space, rather than traditional interface operation events; the generated images simultaneously serve as interactive knowledge interfaces, rather than merely displaying results; and the continuously generated pages are connected through visual style states. and spatial state The iterative constraints maintain knowledge continuity, and pages are not independent of each other; the generation process is influenced by the knowledge space. The content difficulty is determined by constraints, rather than being generated freely by the model; the generated results are presented only after undergoing the multi-dimensional evaluation and failure-driven correction closed loop described in steps thirteen and fourteen, rather than being directly displayed; the content difficulty is determined by the age adaptation factor in step nine. and difficulty penalty factor The multimodal generation module adapts to the user's cognitive state rather than remaining fixed; in step twelf, the generation conditions are... The model is distributed to multiple generative models of different types to perform multi-model collaborative generation, rather than single-model invocation; users achieve dynamic knowledge exploration through the interaction and recommendation mechanisms described in steps five to seven and step seventeen, rather than linear reading.
[0160] In summary, the continuous multimodal content generation method and system based on knowledge state-driven and visual semantic navigation proposed in this invention organically integrates knowledge space modeling, multimodal content generation, visual semantic parsing, user interaction intent understanding, candidate knowledge node scoring, generation condition arrangement, multidimensional consistency evaluation, and failure-driven regeneration by constructing a continuous generation framework with a unified knowledge state as its core. A two-way closed loop.
[0161] The methods and systems provided by this invention can be applied to various scenarios such as children's science picture book generation, digital textbook production, interactive science exhibition content generation, virtual museum navigation, and intelligent educational courseware generation. They can also be extended to fields such as AI-powered scientific exploration, AI-powered interactive textbooks, AI-powered virtual museums, AI-powered digital natural worlds, and general AI-explorable knowledge spaces. Any technical solution that utilizes a unified knowledge state to drive the continuous generation of multimodal content and maps user interaction behavior to the knowledge space through visual semantic parsing to achieve semantic navigation falls within the protection scope of this invention.
Claims
1. A continuous multimodal content generation method based on knowledge state-driven and visual semantic navigation, characterized in that, The method includes: Obtain user input information and initial user state, and construct a structured knowledge space based on the user input information. The knowledge space contains multiple knowledge nodes and semantic relationships between nodes. Based on the current knowledge state, a multimodal generation model is invoked to generate current multimodal content, and visual semantic parsing is performed on the current multimodal content to obtain a set of visual entities, and a mapping relationship is established between each visual entity and knowledge nodes in the knowledge space. The visual interaction behavior performed by the user in response to the current multimodal content is obtained, and the visual interaction behavior and the current knowledge state are input into the visual language model to identify the user's current focus and infer the true exploration intention. Based on the object of interest and the actual exploration intent, a set of candidate knowledge nodes is retrieved from the structured knowledge space. The candidate knowledge nodes are then comprehensively scored in conjunction with the initial user state and the current knowledge state to determine the next target knowledge node. Based on the next target knowledge node and the current knowledge state, construct multimodal generation conditions, and call at least one generation model to generate the next multimodal content; A multidimensional consistency evaluation is performed on the next multimodal content. When the evaluation result meets the preset conditions, the knowledge state is updated according to the current interaction result and the generation result, and the next multimodal content is provided to the user for subsequent interaction. When the evaluation result does not meet the preset conditions, the multimodal generation conditions are corrected according to the failure type and regenerated.
2. The method according to claim 1, characterized in that, The construction of the structured knowledge space specifically includes: The structured knowledge skeleton from an external knowledge base is obtained. A large language model is used to semantically expand the knowledge skeleton to supplement candidate knowledge nodes and relationships between nodes. Then, the expanded knowledge graph is validated by knowledge consistency constraints to filter out nodes and relationships that are irrelevant or conflict with the user input information, thereby generating a semantic knowledge graph with relationship types. The semantic relationships between the nodes include at least causal relationships, temporal order relationships, spatial position relationships, comparative relationships, pre-dependency relationships, and extended association relationships.
3. The method according to claim 1, characterized in that, Establishing the mapping relationship between visual entities and knowledge nodes specifically includes: For each visual entity, a four-way weighted similarity between the entity and the candidate knowledge node is calculated. The four-way weighted similarity includes the textual semantic similarity between the entity name and description and the node text, the visual feature similarity between the entity image region features and the node reference visual features, the contextual similarity between the scene context of the entity and the associated context of the node, and the graph relationship matching degree between the adjacency relationship of the entity in the knowledge space and the node graph structure. For each visual entity, select the candidate knowledge node with the highest weighted similarity. If the highest similarity exceeds the preset mapping threshold, then establish a definite mapping from the visual entity to the knowledge node.
4. The method according to claim 1, characterized in that, Identifying the user's current focus includes: The visual language model is input together with the current multimodal content page image, user interaction coordinates, and current knowledge state, and outputs candidate entities of interest and recognition confidence. When the recognition confidence level is greater than or equal to a preset confidence threshold, the output of the visual language model is accepted as the user's current focus object; When the recognition confidence is less than the preset confidence threshold, it degenerates into a bounding box hit test, traversing the bounding box coordinates of all visual entities on the current page, and selecting the visual entity with the smallest area that contains the interaction coordinates as the user's current focus object.
5. The method according to claim 1, characterized in that, Inferring the true exploration intention specifically includes: Decouple object identification and intent inference into two independent phases that are executed sequentially; After obtaining the user's current focus object, the system further obtains the location information of the focus object in the structured knowledge space, the access records of the knowledge associated with the focus object, and the user's recent continuous interaction behavior patterns. Combined with the current user's age level and cognitive state, the system calculates the true exploration intention. The same object of interest corresponds to different true exploration intentions under different knowledge states and different interaction histories.
6. The method according to claim 1, characterized in that, The comprehensive scoring of candidate knowledge nodes specifically includes: For each candidate knowledge node, the semantic relevance factor, knowledge gain factor, interest matching factor, age fit factor, semantic continuity factor, curiosity value factor, difficulty penalty factor, and repetition penalty factor are calculated simultaneously. The above factors are weighted and summed to obtain the comprehensive score of each candidate knowledge node, and the candidate knowledge node with the highest comprehensive score that has not been visited is selected as the next target knowledge node.
7. The method according to claim 1, characterized in that, Constructing multimodal generation conditions also includes explicit control over cross-page visual continuity: The visual style state of the current page is extracted and maintained in advance. The visual style state includes at least the art style category, main color palette, line drawing method, camera angle, composition layout and main character image characteristics. When constructing the generation conditions for the next multimodal content, the visual style state is used as the anchoring input condition for the image generation model; After the next multimodal content is generated, its actual style features are extracted in reverse to update the visual style state for the continuous generation of subsequent pages.
8. The method according to claim 1, characterized in that, Performing multidimensional consistency evaluation and failure-driven regeneration specifically includes: The next multimodal content is input into the visual language model or evaluation model, and a set of verifiable facts is extracted from it. The set of verifiable facts is compared with the standard knowledge in the structured knowledge space to obtain a scientific fact score. Simultaneously, the semantic theme jump magnitude, visual style deviation distance, and knowledge depth deviation from the user's cognitive level of the next multimodal content and the previous page are calculated. When the score of any dimension is lower than the corresponding threshold, the evaluation is deemed unsuccessful and the specific failure reason type is output. The failure reason type includes at least scientific fact error, visual style drift, layout structure disorder and knowledge difficulty mismatch. Based on the different failure reasons, the fact constraint sub-items, style anchoring sub-items, or age matching sub-items in the multimodal generation conditions are modified accordingly, and the generation model is called again for regeneration until the evaluation is passed or the preset maximum number of retries is reached.
9. The method according to claim 1, characterized in that, After generating the current multimodal content, a pre-inference caching step is also included: Using a visual language model, the click probability of all visual entities on the current page is pre-analyzed to predict a preset number of high-probability interactive entities that are most likely to be clicked by the user. For each high-probability interactive entity, its corresponding semantic description, associated candidate knowledge nodes, and current generation context are pre-cached. When the user's actual click location hits the area corresponding to the high-probability interactive entity, the cached semantic description, associated candidate knowledge nodes, and generation context are directly reused, skipping the real-time visual language model re-parsing step.
10. A continuous multimodal content generation system based on knowledge state-driven and visual semantic navigation, characterized in that, include: The multimodal input parsing module is used to receive user input information and visual interaction behavior, and output a unified multimodal event signal; The knowledge space construction module is signal-connected to the multimodal input parsing module and is used to parse the user input information into a structured knowledge graph and output knowledge graph data. The knowledge state management module is signal-connected to the knowledge space construction module and is used to maintain a unified knowledge state seven-tuple and output the current knowledge state signal. The multimodal scene planning module is signal-connected to the knowledge state management module and is used to convert target knowledge nodes into scene planning signals. A multimodal generation module is signal-connected to the multimodal scene planning module and is used to call at least one generation model to generate multimodal content according to the scene planning signal and output multimodal content signal; The multimodal semantic understanding module is signal-connected to the multimodal generation module and the knowledge state management module, and is used to perform visual entity recognition and knowledge node alignment on the generated multimodal content; The user interaction intent understanding module is signal-connected to the multimodal input parsing module, the multimodal semantic understanding module, and the knowledge state management module, and is used to convert interaction coordinates into attention objects and exploration intent signals; The next knowledge node planning module is signal-connected to the user interaction intent understanding module and the knowledge state management module, and is used to perform candidate knowledge node retrieval and comprehensive scoring selection, and output the next target knowledge node signal to the multimodal scene planning module. The content consistency evaluation module is signal-connected to the multimodal generation module and the knowledge state management module, and is used to perform multidimensional consistency evaluation on the generated multimodal content, and trigger regeneration when the evaluation fails.
