AI-based screen content recognition and knowledge graph construction method and system

Abstract

The present application relates to the technical field of artificial intelligence, and more particularly to a screen content recognition and knowledge graph construction method and system based on AI. By obtaining the screen content sequence and interaction behavior data of the learner, the structured knowledge unit is extracted through multi-modal semantic analysis, the cognitive state is inferred through collaborative analysis based on the unit and interaction behavior data, the personalized measurement is combined with the global knowledge graph to dynamically adjust the topology, the intervention strategy is determined through the time sequence dependence analysis to identify the causal relationship and push the content, and the knowledge graph is updated by collecting feedback. The method can realize precise personalized learning intervention and optimize the learning path.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method and system for screen content recognition and knowledge graph construction based on AI. Background Technology

[0002] With the deepening application of artificial intelligence in education, personalized learning systems based on screen content analysis and knowledge graph construction have gradually become a research hotspot. Current conventional approaches typically rely on screen recording or screenshot technology to obtain learners' visual information, combining this with optical character recognition or image classification models to extract text and image elements. The system further utilizes a pre-set general knowledge graph as a foundation, performing simple statistical analysis on user behavior data (such as clicks, pauses, and scrolling) to infer the learner's level of knowledge mastery. The construction of knowledge graphs is often based on expert annotations or domain ontologies, maintaining a static topology, with only minor adjustments through manual intervention or threshold rules in a few systems. Intervention strategies often employ fixed templates, triggering learning content recommendations based on superficial indicators such as answer accuracy or time consumption, lacking in-depth modeling of the cognitive process.

[0003] The aforementioned conventional approaches have significant drawbacks. On the one hand, existing technologies fail to effectively integrate multimodal semantic information and interactive behavior data from screen content, resulting in knowledge units extracted from the content lacking cognitive hierarchy attributes. Personalized adjustments to knowledge graphs rely solely on behavioral frequency statistics rather than semantic association assessments, making it difficult to accurately reflect learners' dynamically evolving cognitive states. On the other hand, current intervention mechanisms are typically based on isolated behavioral events, failing to identify the causal dependency between current learning difficulties and historical knowledge states. Under the constraint of limited intervention resources, this often leads to inefficient or misleading content delivery, hindering the genuine improvement of learning outcomes. Summary of the Invention

[0004] This invention provides an AI-based method and system for screen content recognition and knowledge graph construction, which can solve the problems in the prior art.

[0005] A first aspect of this invention provides an AI-based method for screen content recognition and knowledge graph construction, comprising:

[0006] Acquire learners' screen display content sequences and interaction behavior data related to those screen displays;

[0007] Multimodal semantic parsing is performed on the sequence of content displayed on the screen to extract structured knowledge units, which contain knowledge representation information and cognitive requirement information;

[0008] Based on the structured knowledge units and the interaction behavior data, the current cognitive state is inferred through the collaborative analysis of content semantic features and interaction behavior patterns, thus obtaining a cognitive state representation;

[0009] Based on the global knowledge graph and the cognitive state representation, knowledge relationships are measured in a personalized manner, and the topological structure of the knowledge graph is dynamically adjusted according to the measurement results to obtain a personalized knowledge topology;

[0010] Based on the personalized knowledge topology, the causal relationship between current learning difficulties and historical knowledge status is identified through temporal dependency analysis. Under the constraint of intervention capacity, the intervention targets and intervention strategies are determined, and appropriate content is generated and pushed to learners.

[0011] The global knowledge graph is updated based on the feedback on the learning results.

[0012] Multimodal semantic parsing is performed on the sequence of content displayed on the screen to extract structured knowledge units. These structured knowledge units contain knowledge representation information and cognitive requirement information, including:

[0013] Semantic analysis is performed on the text content and image content in the screen display content sequence. Key words and definitions are extracted from the text content as text features, and formulas and charts are identified from the image content and symbol complexity and structural features are extracted as visual features.

[0014] The text features and visual features are synthesized into a multimodal knowledge representation vector through a cross-modal fusion mechanism;

[0015] Based on the verb types, sentence structures, and the degree of abstraction of visual elements in the multimodal knowledge representation vector, the Bloom cognitive level of the knowledge point is determined. The Bloom cognitive level includes the memory layer, the understanding layer, the application layer, the analysis layer, the evaluation layer, and the creation layer. The Bloom cognitive level encoding value and the corresponding cognitive operation type are used as cognitive requirement information.

[0016] The multimodal knowledge representation vector and the cognitive requirement information are combined into a structured knowledge unit;

[0017] Using the knowledge points in the structured knowledge units as nodes, edge connections are established based on the pre-existing dependencies and semantic associations between knowledge points to construct a global knowledge graph.

[0018] Based on the structured knowledge units and the interaction behavior data, the current cognitive state is inferred through the collaborative analysis of content semantic features and interaction behavior patterns, resulting in a cognitive state representation, including:

[0019] The cognitive complexity of knowledge points is extracted from the structured knowledge units. The cognitive complexity includes conceptual abstraction, depth of prior dependencies, and symbol density. The cognitive level of the current knowledge point is located in the global knowledge graph based on the cognitive complexity.

[0020] Micro-temporal features are extracted from the interaction behavior data. These micro-temporal features include the first interaction delay duration, operation sequence entropy value, and backtracking behavior frequency. The deviation of each micro-temporal feature from the cognitive complexity is calculated, and features with deviations exceeding a preset deviation threshold are marked as cognitive difficulty signals.

[0021] For the cognitive difficulty signal, query the set of prerequisite knowledge points of the current knowledge point in the global knowledge graph, calculate the semantic similarity between the prerequisite knowledge points and the current interaction behavior pattern, and mark the prerequisite knowledge point with the highest semantic similarity as a suspected defect node.

[0022] The cognitive complexity, the cognitive difficulty signal, and the suspected defective node are encoded into a multi-dimensional vector. Knowledge features and behavioral features are fused through a cross-modal alignment mechanism. The similarity between the fused vector and the predefined cognitive state prototype is calculated. The prototype with the highest similarity is selected as the cognitive state category. The fused vector and the cognitive state category are combined to form a cognitive state representation.

[0023] Extracting micro-temporal features from the interaction behavior data and calculating the deviation includes:

[0024] The time interval from the appearance of the content displayed on the screen to the first interactive operation is recorded as the first interaction delay duration. The corresponding delay benchmark value is queried according to the cognitive complexity, and the normalized deviation between the first interaction delay duration and the delay benchmark value is calculated.

[0025] Extract the learner's operation sequence on the current screen display content, count the frequency of each operation type and calculate the Shannon entropy value of the frequency distribution as the operation sequence entropy value, query the corresponding entropy value benchmark value according to the cognitive complexity, and calculate the normalized deviation between the operation sequence entropy value and the entropy value benchmark value.

[0026] Identify back actions in the operation sequence, including returning to the previous page, rereading viewed content, and undoing operations. Count the number of back actions occurring within a unit time window as the back action frequency. Query the corresponding frequency benchmark value based on the cognitive complexity and calculate the normalized deviation between the back action frequency and the frequency benchmark value.

[0027] When the normalized deviation exceeds a preset deviation threshold, it is marked as a cognitive difficulty signal.

[0028] Based on the global knowledge graph and the cognitive state representation, knowledge relationships are measured in a personalized manner, and the topological structure of the knowledge graph is dynamically adjusted according to the measurement results to obtain a personalized knowledge topology, including:

[0029] For each associated edge in the global knowledge graph, the standard cognitive distance of its connected nodes is calculated. The standard cognitive distance is calculated based on the concept span, cognitive level difference, and dependency chain length.

[0030] Based on the cognitive state representation, the learner's cognitive weak areas and cognitive strong areas are identified. The semantic distance between each knowledge node and the central node of the cognitive weak area and the semantic distance between each knowledge node and the central node of the cognitive strong area are calculated. The ratio of the two is used as the personalized cognitive preference coefficient of the node.

[0031] For each associated edge in the global knowledge graph, a personalized adjustment factor is calculated based on the difference in personalized cognitive preference coefficients of its connected nodes. The standard cognitive distance is nonlinearly mapped to the personalized adjustment factor to obtain the personalized cognitive distance. When the personalized cognitive distance is less than a preset affinity threshold, the weight of the edge is increased; when the personalized cognitive distance is greater than a preset alienation threshold, the weight of the edge is weakened.

[0032] Search for cross-level knowledge node pairs that have not yet established a direct connection but whose concept similarity is greater than a preset similarity threshold between the cognitively weak region and the cognitively strong region. Create temporary connection edges for node pairs that meet the conditions and calculate initial weights based on cross-domain distance and hierarchical differences.

[0033] The adjusted edge weights and newly added temporary edges are superimposed on the global knowledge graph and normalized to obtain a personalized knowledge topology.

[0034] Based on the personalized knowledge topology, causal relationships between current learning difficulties and historical knowledge states are identified through temporal dependency analysis. Under intervention capacity constraints, intervention targets and strategies are determined, and appropriate content is generated and pushed to learners, including:

[0035] Extract the knowledge node corresponding to the current learning difficulty from the personalized knowledge topology, query the set of preceding dependent edges of the knowledge node, traverse backward along the preceding dependent edges to each preceding knowledge node, calculate the cognitive deficiency index of each preceding knowledge node based on the historical cognitive state, and mark it as a candidate root cause node when the cognitive deficiency index exceeds the preset index threshold.

[0036] For each candidate root cause node, the causal transmission attenuation coefficient is calculated based on the path length from the node to the difficult node, the strength weight of the dependent edges on the path, and the time interval. The product of the cognitive deficit index and the causal transmission attenuation coefficient is used as the root cause contribution and sorted.

[0037] The upper limit of the number of nodes for a single intervention is set based on the average number of knowledge points completed in a learner's historical learning session. The nodes are selected sequentially from the sorted candidate root cause nodes until the upper limit is reached, and the selected nodes are used as the set of intervention objects.

[0038] For each node in the set of intervention targets, the level of intervention intensity is determined based on the cognitive deficit index and the root cause contribution, and the type of intervention content is determined based on the knowledge type. These are combined as intervention strategies, and matching materials are retrieved from the content template library according to the intervention strategies to generate appropriate content and push it to the learners.

[0039] The global knowledge graph is updated based on the feedback on the learning results.

[0040] Calculating the causal transmission attenuation coefficient includes:

[0041] Extract the shortest dependency path from candidate root cause nodes to difficult nodes, count the number of hops of the shortest dependency path as the path length, and calculate the reciprocal of the path length as the distance decay factor.

[0042] Extract the strength weight values ​​of each dependent edge on the path, and calculate the arithmetic mean of all strength weight values ​​as the path strength factor;

[0043] Obtain the time difference between the last learning timestamp of the candidate root cause node and the current time. Fit a personalized forgetting curve based on the review interval and test accuracy data in the historical learning record of the node. Substitute the time difference into the exponential decay function corresponding to the personalized forgetting curve to calculate the time decay factor.

[0044] The causal transmission attenuation coefficient is obtained by multiplying the distance attenuation factor, path strength factor, and time attenuation factor.

[0045] A second aspect of this invention provides an AI-based screen content recognition and knowledge graph construction system, comprising:

[0046] The data acquisition unit is used to acquire the learner's screen display content sequence and interactive behavior data related to the screen display content;

[0047] A semantic parsing unit is used to perform multimodal semantic parsing on the sequence of content displayed on the screen and extract structured knowledge units, wherein the structured knowledge units include knowledge representation information and cognitive requirement information;

[0048] The cognitive inference unit is used to infer the current cognitive state based on the structured knowledge unit and the interaction behavior data through the collaborative analysis of content semantic features and interaction behavior patterns, and to obtain a cognitive state representation.

[0049] The topology adjustment unit is used to perform personalized measurement of knowledge associations based on the global knowledge graph and the cognitive state representation, and dynamically adjust the topology of the knowledge graph according to the measurement results to obtain a personalized knowledge topology.

[0050] The intervention push unit is used to identify the causal relationship between the current learning difficulty and the historical knowledge state based on the personalized knowledge topology through temporal dependency analysis, determine the intervention object and intervention strategy under the intervention capacity constraint, generate suitable content and push it to the learner.

[0051] The feedback update unit is used to collect learning effect feedback and update the global knowledge graph.

[0052] A third aspect of the present invention provides an electronic device, comprising:

[0053] processor;

[0054] Memory used to store processor-executable instructions;

[0055] The processor is configured to invoke instructions stored in the memory to execute the aforementioned method.

[0056] A fourth aspect of the present invention provides a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the aforementioned method.

[0057] This invention enables the cognitive state representation to accurately capture learners' depth of understanding and attention distribution through the collaborative analysis of content semantic features and interactive behavior patterns, providing a reliable basis for the construction of personalized learning paths.

[0058] Personalized metrics based on global knowledge graphs and cognitive state representations can dynamically adjust the topological structure of knowledge connections, ensuring that the knowledge graph closely matches the learner's current cognitive level and potential needs. Differentiated processing of knowledge relationships avoids the rigidity of general knowledge graphs, significantly enhancing the relevance of knowledge recommendation and content adaptation, making it particularly suitable for progressive learning of complex knowledge systems.

[0059] By analyzing temporal dependencies, the causal relationship between current learning difficulties and historical knowledge status can be identified. Combined with intervention capacity constraints, intervention targets and strategies can be scientifically determined, achieving precise intervention with limited resources. The generated tailored content directly targets learners' weaknesses, avoiding ineffective pushes. At the same time, the closed-loop feedback of learning effectiveness drives the continuous optimization of the global knowledge graph, forming an adaptive cycle, ultimately improving learning efficiency and knowledge mastery. Attached Figure Description

[0060] Figure 1 This is a flowchart illustrating an AI-based screen content recognition and knowledge graph construction method.

[0061] Figure 2 A flowchart for extracting cognitive state representations. Detailed Implementation

[0062] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0063] The technical solution of the present invention will be described in detail below with reference to specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0064] Figure 1 This is a flowchart illustrating the AI-based screen content recognition and knowledge graph construction method according to an embodiment of the present invention.

[0065] AI-based methods for screen content recognition and knowledge graph construction include:

[0066] Acquire learners' screen display content sequences and interaction behavior data related to those screen displays;

[0067] Multimodal semantic parsing is performed on the sequence of content displayed on the screen to extract structured knowledge units, which contain knowledge representation information and cognitive requirement information;

[0068] Based on the structured knowledge units and the interaction behavior data, the current cognitive state is inferred through the collaborative analysis of content semantic features and interaction behavior patterns, thus obtaining a cognitive state representation;

[0069] Based on the global knowledge graph and the cognitive state representation, knowledge relationships are measured in a personalized manner, and the topological structure of the knowledge graph is dynamically adjusted according to the measurement results to obtain a personalized knowledge topology;

[0070] Based on the personalized knowledge topology, the causal relationship between current learning difficulties and historical knowledge status is identified through temporal dependency analysis. Under the constraint of intervention capacity, the intervention targets and intervention strategies are determined, and appropriate content is generated and pushed to learners.

[0071] The global knowledge graph is updated based on the feedback on the learning results.

[0072] In one optional implementation, multimodal semantic parsing is performed on the screen display content sequence to extract structured knowledge units. These structured knowledge units include knowledge representation information and cognitive requirement information, including:

[0073] Semantic analysis is performed on the text content and image content in the screen display content sequence. Key words and definitions are extracted from the text content as text features, and formulas and charts are identified from the image content and symbol complexity and structural features are extracted as visual features.

[0074] The text features and visual features are synthesized into a multimodal knowledge representation vector through a cross-modal fusion mechanism;

[0075] Based on the verb types, sentence structures, and the degree of abstraction of visual elements in the multimodal knowledge representation vector, the Bloom cognitive level of the knowledge point is determined. The Bloom cognitive level includes the memory layer, the understanding layer, the application layer, the analysis layer, the evaluation layer, and the creation layer. The Bloom cognitive level encoding value and the corresponding cognitive operation type are used as cognitive requirement information.

[0076] The multimodal knowledge representation vector and the cognitive requirement information are combined into a structured knowledge unit;

[0077] Using the knowledge points in the structured knowledge units as nodes, edge connections are established based on the pre-existing dependencies and semantic associations between knowledge points to construct a global knowledge graph.

[0078] For example, the sources of the screen display content sequence include various terminal devices used by learners during the learning process, covering PC browser pages, e-book readers, online course playback interfaces, and document editing tools. For the screen display content sequence, frame-level capture is performed based on fixed time windows or page switching events as trigger conditions, and learner interaction behavior data on the screen content is recorded simultaneously, including mouse hover position, click area, scrolling speed, selected text segments, and dwell time. This interaction behavior data forms time-aligned sequence pairs with the corresponding screen display content frames, providing a complete input data foundation for subsequent multimodal semantic parsing.

[0079] Different semantic parsing paths are used to process the text and image content in the screen display sequence. For the text content, optical character recognition (OCR) technology is first used to convert the text areas in the screenshot into structured text. Then, a named entity recognition model for the education field is used to extract knowledge point keywords, including professional terms, concept names, and their definitions. The recognition of definition statements relies on the joint judgment of sentence pattern matching and semantic role labeling, which can distinguish between declarative definition sentences, explanatory sentences, and illustrative sentences, thereby classifying different types of text fragments into the corresponding knowledge expression categories and forming a set of text features.

[0080] For image content, a specially trained visual analysis model is used to detect objects in non-text areas of screenshots, identifying visual element types such as mathematical formulas, statistical charts, flowcharts, and diagrams. For mathematical formulas, symbolic complexity indicators such as symbol density, nesting level, and operator types are further extracted; for chart elements, structural features such as the number of coordinate axes, the number of data series, and chart type encoding are extracted. These indicators together constitute a visual feature set, reflecting the cognitive processing difficulty of the image content. Both textual and visual features are expressed in vector form, with dimensions determined by the output layer of their respective encoders.

[0081] The core of the cross-modal fusion mechanism lies in mapping text feature vectors and visual feature vectors to a unified semantic space for alignment and synthesis. Specifically, text feature vectors are transformed into a common embedding space through a linear projection layer, and visual feature vectors are similarly transformed into a common embedding space of the same dimension through an independent linear projection layer. Subsequently, a cross-attention mechanism is used, with the text feature vector as the query vector and the visual feature vector as the key vector, to calculate the cross-modal attention weights between them, obtaining the distribution of textual attention to visual content; simultaneously, reverse attention is calculated using the visual feature vector as the query vector and the text feature vector as the key vector. The weighted outputs of the attention in both directions are concatenated with the original projection vector and compressed through a feedforward network to obtain a multimodal knowledge representation vector. This vector simultaneously encodes the textual semantic information and image structural information of the screen content in a single high-dimensional space, serving as the core input for subsequent cognitive level judgments.

[0082] Bloom's Taxonomy of Cognition is based on multimodal knowledge representation vectors and comprehensively evaluated using three categories of criteria. The first criterion is verb type: predicate verbs extracted from the text are categorized according to the verb vocabulary corresponding to Bloom's Taxonomy. For example, "enumerate" and "recall" correspond to the memory level; "explain" and "describe" correspond to the comprehension level; "calculate" and "execute" correspond to the application level; "compare" and "distinguish" correspond to the analysis level; "evaluate" and "judge" correspond to the evaluation level; and "design" and "build" correspond to the creation level. The second criterion is sentence structure: syntactic dependency analysis identifies the presence of complex sentence components such as conditional clauses, hypothetical structures, and causal conjunctions. The presence of complex sentence structures is usually associated with higher cognitive levels. The third criterion is the level of abstraction of visual elements: concrete illustrations and photographs correspond to lower levels of abstraction; flowcharts and concept maps correspond to medium levels of abstraction; and purely symbolic formulas and logical derivation diagrams correspond to higher levels of abstraction. The discrimination results of the above three categories are input into a multilayer perceptron classifier, which outputs a probability distribution of six cognitive levels. The level with the highest probability is taken as the Bloom cognitive level label value for that knowledge point. The Bloom cognitive level encoding value uses integers. It means that among them These correspond to the memory layer to the creation layer, and are supplemented with corresponding cognitive operation type descriptions as textual supplementary fields for cognitive requirement information.

[0083] Structured knowledge units consist of two parts: a multimodal knowledge representation vector and cognitive requirement information, indexed using knowledge point identifiers as the primary key. The multimodal knowledge representation vector preserves the complete semantic encoding of the knowledge point, while the cognitive requirement information stores cognitive level encoding values ​​in structured fields. Together with cognitive operation type tags, these two elements describe the core attributes of the knowledge point in two dimensions: "what it is" and "what level of cognition needs to be achieved." The storage format of structured knowledge units supports direct reading of subsequent graph node attributes, facilitating efficient retrieval during the knowledge graph construction and updating stages.

[0084] Using knowledge points within structured knowledge units as nodes, directed and undirected edges are established based on the pre-existing dependencies and semantic connections between knowledge points. Pre-existing dependencies are identified according to the hierarchical structure of the course knowledge system and the chapter order of the textbook. If understanding knowledge point A presupposes mastery of knowledge point B, then a directed dependency edge from B to A is established in the graph, with edge weights... Dependency strength is determined by expert annotation or statistical estimation from historical learning path data. The establishment of semantic associations between concepts is based on the cosine similarity between multimodal knowledge representation vectors, assuming knowledge points... With knowledge points The multimodal knowledge representation vectors are respectively and The semantic similarity between the two is ,when Exceeding the preset threshold At this time, an undirected edge with semantic association is established between the two nodes, and the edge weight is set to a value. By comprehensively constructing the two types of edges mentioned above, a global knowledge graph containing both pre-dependency topology and semantic nearest neighbor topology is formed. This graph serves as the foundational structure for subsequent personalized knowledge topology generation and the formulation of learning intervention strategies.

[0085] After its initial construction, the global knowledge graph dynamically expands as new screen content sequences are continuously added. New knowledge points undergo the same multimodal semantic parsing process to generate structured knowledge units, which are then incrementally inserted into the graph. The semantic similarity between these new knowledge points and existing nodes is recalculated, and semantically related edges are added to node pairs that meet threshold conditions. This mechanism ensures that the global knowledge graph continuously improves as learning content expands, maintaining complete coverage of the knowledge domains encountered by learners.

[0086] In one optional implementation, based on the structured knowledge units and the interaction behavior data, the current cognitive state is inferred through the collaborative analysis of content semantic features and interaction behavior patterns to obtain a cognitive state representation, including:

[0087] The cognitive complexity of knowledge points is extracted from the structured knowledge units. The cognitive complexity includes conceptual abstraction, depth of prior dependencies, and symbol density. The cognitive level of the current knowledge point is located in the global knowledge graph based on the cognitive complexity.

[0088] Micro-temporal features are extracted from the interaction behavior data. These micro-temporal features include the first interaction delay duration, operation sequence entropy value, and backtracking behavior frequency. The deviation of each micro-temporal feature from the cognitive complexity is calculated, and features with deviations exceeding a preset deviation threshold are marked as cognitive difficulty signals.

[0089] For the cognitive difficulty signal, query the set of prerequisite knowledge points of the current knowledge point in the global knowledge graph, calculate the semantic similarity between the prerequisite knowledge points and the current interaction behavior pattern, and mark the prerequisite knowledge point with the highest semantic similarity as a suspected defect node.

[0090] The cognitive complexity, the cognitive difficulty signal, and the suspected defective node are encoded into a multi-dimensional vector. Knowledge features and behavioral features are fused through a cross-modal alignment mechanism. The similarity between the fused vector and the predefined cognitive state prototype is calculated. The prototype with the highest similarity is selected as the cognitive state category. The fused vector and the cognitive state category are combined to form a cognitive state representation.

[0091] Combination Figure 2 The flowchart for extracting cognitive state representations illustrates that when extracting the cognitive complexity of knowledge points from structured knowledge units, three quantitative indicators need to be considered comprehensively: conceptual abstraction, pre-dependency depth, and symbol density. Conceptual abstraction reflects the distance between the concepts involved in the knowledge point and specific perceptual experience. For example, "Newton's Second Law" has a medium level of abstraction, while "weak convergence in functional spaces" has a high level of abstraction. Pre-dependency depth measures the length of the longest dependency path required to reach the current knowledge node in the global knowledge graph. The longer the path, the higher the level of the knowledge point in the cognitive system. Symbol density counts the frequency of non-natural language symbols such as mathematical symbols, logical operators, and professional abbreviations in the displayed content. High symbol density is usually associated with a strong cognitive processing load. After combining the quantitative values ​​of the above three dimensions, the node closest to the cognitive characteristics of the current knowledge point is retrieved in the global knowledge graph, thereby locating the cognitive level of the current knowledge point. The result of locating the cognitive level will serve as a benchmark reference for subsequent deviation calculations, ensuring that the analysis of behavioral characteristics is always anchored in the specific knowledge context.

[0092] When extracting micro-temporal features from interactive behavior data, three types of behavioral indicators are emphasized. The first interaction delay duration refers to the time interval between the appearance of the current knowledge point content and the first effective interactive operation (such as clicking, scrolling, or annotation). A longer value usually indicates that the learner is experiencing a pause in understanding the content. The operation sequence entropy value quantifies the degree of disorder of operations by calculating the information entropy of the sequence of interactive operation types within a time window. Let the operation sequence contain a total of... Type of operation, number The probability of this operation type occurring is Then the entropy of the operation sequence Defined as The higher the entropy value, the more chaotic the operation and the greater the degree of cognitive confusion. The frequency of backtracking behavior is the number of times learners perform backtracking behaviors such as flipping forward, repeating playback, and undoing operations per unit of time. Frequent backtracking behavior is a direct indication of excessive cognitive load.

[0093] When calculating the deviation of each micro-temporal feature from cognitive complexity, it is necessary to establish a baseline distribution corresponding to the cognitive complexity level for each type of feature. Specifically, in the global sample data, for knowledge points at the same cognitive level, the mean of each micro-temporal feature is calculated. with standard deviation subscript This represents the feature category index. It represents the measured value of the feature observed by the current learner at a given knowledge point. Its deviation Defined as , that is, the standardized absolute deviation. When Exceeding the preset deviation threshold At that time, this feature is labeled as a cognitive difficulty signal. Threshold The settings need to strike a balance between sensitivity and false alarm rate, and the optimal value can be determined through ROC analysis based on historical labeled data. The feature set marked as cognitive difficulty signals will be used as a condition to trigger pre-dependency queries in subsequent steps, thereby avoiding costly graph retrieval for every ordinary interaction.

[0094] For queries targeting pre-dependent knowledge points triggered by cognitive difficulty signals, the global knowledge graph is traversed backwards along the dependency edges, starting from the current knowledge node, to obtain a set of all pre-dependent knowledge points. The traversal depth can be dynamically set based on the pre-dependent depth index; shallower knowledge points only require searching their direct predecessor nodes, while deeper knowledge points need to be expanded to multi-hop neighbors. For each pre-dependent knowledge point obtained, its multimodal knowledge representation is extracted, and semantic similarity is calculated with the feature vector of the current interaction behavior pattern. The semantic similarity calculation method here is consistent with the global knowledge graph construction stage, using cosine similarity to measure the directional proximity between two feature vectors. All pre-dependent knowledge points are sorted in descending order of semantic similarity, and the pre-dependent knowledge point with the highest similarity is marked as a suspected defect node, indicating that the current learning difficulty is most likely due to insufficient mastery of this pre-dependent knowledge point. If the similarity differences among multiple pre-dependent knowledge points are within a tolerable range, several nodes with the highest similarity ranking are included in the candidate set of suspected defect nodes to cover the possibility of multi-source defects.

[0095] When encoding cognitive complexity, cognitive difficulty signals, and suspected defective nodes into a multi-dimensional vector, each information source is independently quantified. The three dimensions of cognitive complexity are directly represented by normalized values; cognitive difficulty signals are represented as binary vectors recording whether each feature is labeled, along with the deviation value of each feature; suspected defective nodes are represented by their node embedding vectors in the knowledge graph. If multiple candidate suspected defective nodes exist, a weighted average of their embedding vectors is taken, with the weight proportional to semantic similarity. The three types of encoded vectors are concatenated to form the original fusion input vector. Subsequently, a cross-modal alignment mechanism is used to align the knowledge feature subspace and the behavioral feature subspace, eliminating differences in scale and semantic space between the two types of features. The core of the cross-modal alignment mechanism is learning a set of projection matrices so that vectors from the knowledge feature subspace and the behavioral feature subspace have comparable geometric structures in the aligned common space. The aligned vectors are further compressed into a fixed-dimensional fusion vector through a nonlinear transformation. .

[0096] Fusion Vector Similarity between the cognitive state prototype and the predefined cognitive state prototype library is calculated using cosine similarity. The predefined cognitive state prototype library contains prototype vectors for several typical cognitive states, such as categories like "fluent mastery," "shallow understanding," "conceptual confusion," and "pre-existing deficiencies." Each prototype vector... It is initialized with the cluster centers of historically labeled samples and continuously updated during system operation. Assume there are a total of... One prototype, for each prototype ( ),calculate and cosine similarity Select The largest corresponding prototype category is taken as the current cognitive state category. Finally, the fused vector... Combined with the selected cognitive state category labels, a complete cognitive state representation is formed. This representation retains both continuous vector representation and discrete category labels. The former is used for fine-grained calculations in subsequent personalized knowledge graph metrics, while the latter is used for rapid matching of intervention strategies and conditional judgments in push logic. This dual encoding method achieves a balance between accuracy and interpretability in cognitive state representation, supporting both precise numerical optimization and providing semantically clear decision-making basis for learning interventions.

[0097] In one optional implementation, extracting micro-temporal features from the interaction behavior data and calculating the deviation includes:

[0098] The time interval from the appearance of the content displayed on the screen to the first interactive operation is recorded as the first interaction delay duration. The corresponding delay benchmark value is queried according to the cognitive complexity, and the normalized deviation between the first interaction delay duration and the delay benchmark value is calculated.

[0099] Extract the learner's operation sequence on the current screen display content, count the frequency of each operation type and calculate the Shannon entropy value of the frequency distribution as the operation sequence entropy value, query the corresponding entropy value benchmark value according to the cognitive complexity, and calculate the normalized deviation between the operation sequence entropy value and the entropy value benchmark value.

[0100] Identify back actions in the operation sequence, including returning to the previous page, rereading viewed content, and undoing operations. Count the number of back actions occurring within a unit time window as the back action frequency. Query the corresponding frequency benchmark value based on the cognitive complexity and calculate the normalized deviation between the back action frequency and the frequency benchmark value.

[0101] When the normalized deviation exceeds a preset deviation threshold, it is marked as a cognitive difficulty signal.

[0102] For example, the process of extracting micro-temporal features from interaction behavior data and calculating deviation involves the independent collection and benchmark comparison of three types of core features. Finally, the cognitive difficulty signal is marked by judging the normalized deviation threshold. These three types of features are the first interaction delay duration, the operation sequence entropy value, and the frequency of backtracking behavior. Each type of feature needs to be dynamically queried in conjunction with the cognitive complexity corresponding to the current screen content to ensure the context sensitivity of deviation calculation.

[0103] The initial interaction delay duration is collected starting from the timestamp of the screen content appearing and ending at the timestamp of the learner's first valid interactive operation (including any recordable operation event such as clicking, swiping, or input). The difference between the two is the initial interaction delay duration. In actual collection, a screen content switching event triggers a timer, which stops immediately after detecting the first interactive event and records the time difference. Cognitive complexity is the quantification result of the cognitive requirement information extracted from structured knowledge units during the multimodal semantic parsing stage, typically corresponding to the level indicated by the Bloom's hierarchy of cognitive levels encoding value. For different cognitive levels, the baseline mean and standard deviation of the delay for each level are pre-calculated using large-scale historical learning behavior data and stored in a baseline lookup table. Given the cognitive level of the current content, the corresponding baseline mean delay is retrieved from the lookup table. Standard deviation from the benchmark The measured delay time of the first interaction is recorded as . The normalized deviation of the first interaction delay Calculated as This deviation reflects the degree to which the learner's delayed behavior deviates from the baseline of the group at the same cognitive level. The greater the deviation, the more abnormal the learner's reaction time is when faced with the current content, which may indicate excessive cognitive load or content comprehension difficulties.

[0104] The extraction of operation sequence entropy focuses on the complete operation sequence corresponding to the currently displayed content. The operation sequence records all interactive events and their type labels generated by the learner during their stay on the content. Operation types include, but are not limited to, page scrolling, content clicking, text selection, note annotation, link navigation, and search triggering. The frequency of each operation type in the operation sequence is calculated, assuming the total number of operation types in the sequence is [value missing]. , No. The probability of each operation type appearing in the operation sequence is Then the Shannon entropy value of the operation sequence according to Calculate the entropy value. This reflects the diversity and disorder of operational behaviors: when entropy is low, learners' operations are concentrated in a few types, and the behavioral patterns are regular; when entropy is high, the operation types are scattered, and the behavior exhibits exploratory or chaotic characteristics. Similarly, for the cognitive level corresponding to the current content, the corresponding baseline mean entropy value is retrieved from the baseline lookup table. Standard deviation from the benchmark The measured entropy value of the operation sequence Substituting into the normalization calculation, we obtain the normalized deviation of the operation sequence entropy value. .when Exceeding the preset deviation threshold This indicates that the complexity of the learner's operational behavior is significantly different from the baseline of the same level group, triggering a cognitive difficulty signal marker.

[0105] Identifying the frequency of back navigation requires semantic-level behavioral pattern matching of the operation sequence. Back navigation is defined as three specific operation patterns: returning to the previous page (i.e., triggering the page back navigation event), rereading previously viewed content (i.e., scrolling the viewport back to the previously viewed content area, determined by comparing the scroll position coordinates with the historical browsing trajectory), and undoing (i.e., triggering undo / back edit events). The operation sequence is scanned event-by-event to identify and label all back navigation events belonging to these three categories. To eliminate the influence of variations in dwell time on frequency statistics, the total number of back navigation events occurring within a unit time window (e.g., 60 seconds) is used as the back navigation frequency. The standardized frequency value is obtained by dividing the total number of backtracking actions by the total duration of the current content (in units of time window length). For the cognitive level corresponding to the current content, the baseline mean frequency of the corresponding backtracking action is retrieved from the baseline lookup table. Standard deviation from the benchmark Calculate the normalized deviation An abnormally high frequency of regression behavior is usually directly related to learners' difficulty in understanding the current content. For example, repeatedly going back to refer to prior knowledge or rereading the same paragraph multiple times are typical manifestations of cognitive impairment.

[0106] Three types of normalization deviation , , Each with a preset deviation threshold Compare them. It is a globally unified judgment threshold, calibrated during system initialization based on the distribution characteristics of historical data, and typically ranges from 1.5 to 2.5 (in multiples of standard deviation). When any deviation exceeds... At that time, the cognitive difficulty signal corresponding to the feature category is activated and marked. The three types of signals can be triggered independently or jointly. The subsequent cognitive state inference module will combine the activation state of the three types of signals with their respective deviation measures to further determine the type and severity of cognitive difficulty. For example, if only the deviation of the first interaction delay exceeds the threshold, it may indicate an initial difficulty in understanding the content; if the entropy value of the operation sequence and the frequency of backtracking behavior both exceed the threshold, it is more likely to indicate that the learner has encountered persistent cognitive difficulties in the deep processing stage of the content. This multi-feature joint analysis mechanism makes the marking of cognitive difficulty signals more accurate and interpretable, avoiding misjudgments caused by occasional fluctuations of a single feature.

[0107] The construction and maintenance of the benchmark lookup table are fundamental to the aforementioned deviation calculation. The benchmark lookup table stores the mean and standard deviation of each feature category hierarchically according to cognitive levels (corresponding to the six levels of Bloom's cognitive hierarchy). The initial version was generated from offline batch historical data statistics. Subsequently, as the global knowledge graph is updated and new learning behavior data accumulates, the benchmark lookup table is periodically re-estimated to ensure the timeliness and representativeness of the benchmark values. At cognitive levels with insufficient data, benchmark values ​​from adjacent levels are used for smooth interpolation to avoid the deviation calculation failing due to missing benchmarks.

[0108] In one optional implementation, based on the global knowledge graph and the cognitive state representation, knowledge relationships are personalized and measured, and the topological structure of the knowledge graph is dynamically adjusted according to the measurement results to obtain a personalized knowledge topology, including:

[0109] For each associated edge in the global knowledge graph, calculate the standard cognitive distance of its connected nodes. The standard cognitive distance is calculated based on the concept span, cognitive level difference, and dependency chain length.

[0110] Based on the cognitive state representation, the learner's cognitive weak areas and cognitive strong areas are identified. The semantic distance between each knowledge node and the central node of the cognitive weak area and the semantic distance between each knowledge node and the central node of the cognitive strong area are calculated. The ratio of the two is used as the personalized cognitive preference coefficient of the node.

[0111] For each associated edge in the global knowledge graph, a personalized adjustment factor is calculated based on the difference in personalized cognitive preference coefficients of its connected nodes. The standard cognitive distance is nonlinearly mapped to the personalized adjustment factor to obtain the personalized cognitive distance. When the personalized cognitive distance is less than a preset affinity threshold, the weight of the edge is increased; when the personalized cognitive distance is greater than a preset alienation threshold, the weight of the edge is weakened.

[0112] Search for cross-level knowledge node pairs that have not yet established a direct connection but whose concept similarity is greater than a preset similarity threshold between the cognitively weak region and the cognitively strong region. Create temporary connection edges for node pairs that meet the conditions and calculate initial weights based on cross-domain distance and hierarchical differences.

[0113] The adjusted edge weights and newly added temporary edges are superimposed on the global knowledge graph and normalized to obtain a personalized knowledge topology.

[0114] For example, after obtaining the cognitive state representation, it is necessary to make personalized adjustments to the topology of the global knowledge graph based on the individual characteristics of the current learner, so that the relationships in the graph can truly reflect the learner's familiarity with the knowledge in the current cognitive state, rather than simply relying on the general subject knowledge structure.

[0115] For each connection edge in the global knowledge graph, the standard cognitive distance between the two knowledge nodes connected by the edge is first calculated. This standard cognitive distance is not simply graph distance, but rather considers three dimensions: conceptual span, cognitive level difference, and dependency chain length. Conceptual span reflects the abstract distance between two knowledge points in the semantic space, obtained through the distance metric of their respective knowledge representation vectors in the semantic embedding space. Cognitive level difference, based on Bloom's taxonomy of cognition, is quantified using the absolute value of the difference in cognitive requirement levels corresponding to the nodes; a larger level difference indicates a greater difference in cognitive complexity between the two knowledge points. Dependency chain length represents the number of nodes along the shortest preceding dependency path from one node to another in the global knowledge graph; a longer path indicates more intermediate knowledge as a foundation between the two knowledge points. These three dimensions are then weighted and fused according to preset weights to obtain the standard cognitive distance for each connection edge. This distance describes the cognitive cost required to connect two knowledge points from the perspective of a general learner.

[0116] Based on cognitive state representation, learners' current cognitive weakness and strength regions are identified. Cognitive weakness regions refer to sets of knowledge nodes with low mastery and dense difficulty signals in the cognitive state representation, while cognitive strength regions are sets of knowledge nodes with high mastery and stable recent interaction performance. For each set, its central node is calculated. The central node is determined by selecting the node with the smallest average semantic distance from all other nodes within the set, ensuring that this node best represents the semantic center of the region. For any knowledge node in the global knowledge graph... Calculate the semantic distance between each node and the central node of the cognitively weak region. and semantic distance to the central node of the cognitive advantage region The ratio of the two is defined as the node. Personalized cognitive preference coefficient ,Right now .when When the value is smaller, it indicates that the node is semantically closer to a weaker region, and the learner's grasp of the related content is relatively weak; when... A larger value indicates that the node is closer to the dominant region, and the learner has a relatively solid grasp of the relevant content of that node.

[0117] For each connection node in the global knowledge graph With nodes The associated edges are used to calculate a personalized adjustment factor based on the difference in personalized cognitive preference coefficients between the two endpoints. The degree of difference is defined as follows: The greater the difference, the more likely the edge connects a node in a weak region to a node in a strong region. From a personalized perspective, such edges have high "cross-domain bridging" value and should receive more attention. (The standard cognitive distance is used as an example.) With personalized adjustment factors Perform a nonlinear mapping to obtain personalized cognitive distance The design principle of nonlinear mapping is as follows: when the difference is large, the mapped personalized cognitive distance is compressed compared to the standard cognitive distance, reflecting that the edge "brings" the two regions closer in the personalized graph; when the difference is small, the mapping result is close to the standard cognitive distance, and the graph structure basically remains unchanged. Specifically, the following mapping form can be adopted: ,in To control the hyperparameters of compression strength, after obtaining the personalized cognitive distance, the edge weights are adjusted according to two preset thresholds: when... Less than the preset affinity threshold When these two knowledge points are considered to have a strong correlation and affinity in the learner's current cognitive structure, the weight of that side is increased; when Greater than the preset alienation threshold When the two knowledge points are considered to be relatively unrelated in the current learner's cognitive structure, the weight of the edge is weakened; when the personalized cognitive distance is between the two thresholds, the edge weight remains unchanged.

[0118] After adjusting the weights of existing edges, it is necessary to search for potential cross-level knowledge node pairs between cognitively weak and cognitively strong regions to discover knowledge connections that are not yet explicitly modeled in the global knowledge graph but actually have significant bridging value. The search scope is limited to: one node belongs to a cognitively weak region, the other to a cognitively strong region, and the two are not yet connected by a direct edge in the global knowledge graph. For node pairs that meet the above conditions, the cosine similarity between the knowledge representation vectors of the two nodes is calculated. If this similarity is greater than a preset similarity threshold, the search is considered complete. If the two nodes are considered to have a potential semantic connection, they can provide a bridge for learners to transfer knowledge from their existing knowledge to their weaker knowledge. For each node pair that meets the condition, a temporary connection edge is created, and the initial weight of the temporary edge is calculated based on the cross-domain semantic distance and cognitive level difference between the two nodes. Specifically, the smaller the cross-domain semantic distance and the smaller the hierarchical difference, the higher the initial weight, reflecting the stronger the feasibility of the temporary connection in cognitive transfer. The initial weight calculation form for temporary edges is as follows: ,in For the semantic distance between cross-domain node pairs, This represents the absolute value of the difference in cognitive levels between the two nodes. and These are the corresponding attenuation coefficients.

[0119] The adjusted original edge weights are combined with the initial weights of the newly added temporary edges and superimposed onto the global knowledge graph to form an enhanced graph structure containing personalized adjustment information. To ensure that all edge weights in the graph are within a uniform dimension, the superimposed graph is normalized so that the sum of the outgoing edge weights of each node is normalized to 1, thus obtaining the final personalized knowledge topology. This topology retains the general subject structure of the global knowledge graph while incorporating the individual characteristics of the learner's current cognitive state. It can more accurately reflect the relative importance and correlation strength between various knowledge points in the learner's current learning stage, providing a structured personalized knowledge foundation for subsequent learning difficulty identification and intervention strategy generation.

[0120] In one optional implementation, based on the personalized knowledge topology, causal relationships between current learning difficulties and historical knowledge states are identified through temporal dependency analysis. Under intervention capacity constraints, intervention targets and strategies are determined, and adapted content is generated and pushed to learners, including:

[0121] Extract the knowledge node corresponding to the current learning difficulty from the personalized knowledge topology, query the set of preceding dependent edges of the knowledge node, traverse backward along the preceding dependent edges to each preceding knowledge node, calculate the cognitive deficiency index of each preceding knowledge node based on the historical cognitive state, and mark it as a candidate root cause node when the cognitive deficiency index exceeds the preset index threshold.

[0122] For each candidate root cause node, the causal transmission attenuation coefficient is calculated based on the path length from the node to the difficult node, the strength weight of the dependent edges on the path, and the time interval. The product of the cognitive deficit index and the causal transmission attenuation coefficient is used as the root cause contribution and sorted.

[0123] The upper limit of the number of nodes for a single intervention is set based on the average number of knowledge points completed in a learner's historical learning session. The nodes are selected sequentially from the sorted candidate root cause nodes until the upper limit is reached, and the selected nodes are used as the set of intervention objects.

[0124] For each node in the set of intervention targets, the level of intervention intensity is determined based on the cognitive deficit index and the root cause contribution, and the type of intervention content is determined based on the knowledge type. These are combined as intervention strategies, and matching materials are retrieved from the content template library according to the intervention strategies to generate appropriate content and push it to the learners.

[0125] The global knowledge graph is updated based on the feedback on the learning results.

[0126] For example, after locating the knowledge node corresponding to the current learning difficulty from the personalized knowledge topology, a reverse traversal is performed along the set of preceding dependency edges of that node, tracing back layer by layer to each preceding knowledge node. During the reverse traversal, for each visited preceding knowledge node, a cognitive deficit index is calculated based on the historical cognitive state record. The Cognitive Deficit Index comprehensively considers the mastery assessment score in historical learning sessions, the degree of forgetting decay, and the recent test error rate at that node. Specifically, it combines the normalized reciprocal of the historical mastery assessment score with a weighted sum of the forgetting decay factor and the recent error rate to obtain a comprehensive quantitative index reflecting the severity of the knowledge deficit at that node. Exceeding the preset index threshold When this happens, the preceding knowledge node is marked as a candidate root cause node and included in the subsequent causal analysis process. This reverse traversal mechanism can effectively break through the obscuring of superficial difficulties and trace the root cause of learning difficulties back to the earlier, less-mastered basic knowledge layer, providing structured causal clues for subsequent precise intervention.

[0127] For each candidate root cause node, calculate its causal transmission attenuation coefficient for the current learning difficulty. . It is determined by three dimensions: the path length from the candidate root cause node to the difficult node. The cumulative product of the strength weights of each dependent edge on the path And the time interval between the last time the node was effectively learned to the current time. The longer the path, the longer the causal transmission chain, and the more significant the attenuation; the lower the strength weight of the dependency edges on the path, the looser the dependency coupling between knowledge points, and the weaker the causal influence; the larger the time interval, the more the direct impact of historical defects on current difficulties is further weakened due to forgetting. Combining these three dimensions... The calculation method is as follows: ;in, This is the path length attenuation coefficient. The time interval decay coefficient is used; both are positive hyperparameters and can be adjusted according to the characteristics of the subject area and the learner group. The cognitive deficit index of candidate root cause nodes is used. With causal transmission attenuation coefficient The product is defined as the root cause contribution. ,Right now For all candidate root cause nodes, sort them out by... Rank the nodes from highest to lowest contribution so that the nodes with the greatest contribution are given priority to enter the intervention candidate queue, ensuring that limited intervention resources are concentrated on the nodes with the most root causes of knowledge defects.

[0128] Intervention capacity constraints are imposed by limiting the number of nodes allowed in a single intervention. To reflect this. The number of knowledge points completed per session in a learner's historical learning conversations is dynamically set. Specifically, the average number of knowledge points reviewed or practiced in each of the learner's past learning conversations is calculated and rounded down to determine the upper limit of the number of nodes that can be covered in this intervention. This design ensures that the amount of intervention tasks matches the learner's actual cognitive capacity, avoiding cognitive overload caused by pushing too much intervention content at once, which could lead to learner resistance or decreased learning efficiency. Nodes are selected sequentially from the sorted list of candidate root cause nodes until the number of selected nodes reaches a certain threshold. The selected nodes form the intervention target set. .

[0129] For the set of intervention targets Each node in the dataset is determined based on its cognitive deficit index. With root cause contribution The intervention intensity level was jointly determined. The intervention intensity level was divided into three levels: when... and When both are in the high range, it is considered a deep intervention, corresponding to systematic explanation and multiple rounds of intensive practice; when one is in the medium range, it is considered a moderate intervention, corresponding to targeted example explanations and focused practice; when both are in the low range, it is considered a mild intervention, corresponding to suggestive review cards or brief reviews. The type of intervention content is determined based on the knowledge type attributes of the knowledge node: conceptual knowledge nodes prioritize matching definition and analogy-based content templates; procedural knowledge nodes prioritize matching step-by-step demonstrations and operational exercises; principle-based knowledge nodes prioritize matching derivation processes and application scenarios. The intervention intensity level and intervention content type are combined to form the intervention strategy for that node. Using the intervention strategy as the search condition, a multi-dimensional matching is performed in the content template library to retrieve materials that best match the node's knowledge representation information, cognitive level, and learner's historical preferences, generating appropriate content.

[0130] After the adapted content is generated, it is processed according to the nodes in the set of intervention objects. The push notifications are arranged in descending order, prioritizing intervention content corresponding to the node with the highest root cause contribution, enabling learners to address their most fundamental knowledge gaps. The push method is tailored to the learner's current learning context: if the learner is actively browsing, the intervention content is seamlessly embedded in the sidebar or pop-up next to the currently displayed content; if the learner is inactive, a notification is sent to guide them to actively access the intervention content interface.

[0131] After the intervention content is delivered, learners' feedback on learning outcomes during the intervention process is continuously collected, including the completion status of the intervention content, changes in test scores after the intervention, and characteristics of subsequent interactive behaviors related to difficult points. After quantifying these feedback signals, the mastery level labeling, dependency edge weights, and cognitive level assessments of corresponding nodes in the global knowledge graph are updated. This allows the global knowledge graph to continuously absorb the latest learning outcome information, maintain a dynamic and accurate portrayal of learners' knowledge states, and provide more reliable foundational data for the next round of cognitive state inference and personalized knowledge topology adjustment. Through this complete closed loop of causal relationship identification, capacity constraint selection, tiered intervention strategy generation, and feedback update, structured tracing of learning difficulties and precise intervention are achieved.

[0132] In one alternative implementation, calculating the causal transmission attenuation coefficient includes:

[0133] Extract the shortest dependency path from candidate root cause nodes to difficult nodes, count the number of hops of the shortest dependency path as the path length, and calculate the reciprocal of the path length as the distance decay factor.

[0134] Extract the strength weight values ​​of each dependent edge on the path, and calculate the arithmetic mean of all strength weight values ​​as the path strength factor;

[0135] Obtain the time difference between the last learning timestamp of the candidate root cause node and the current time. Fit a personalized forgetting curve based on the review interval and test accuracy data in the historical learning record of the node. Substitute the time difference into the exponential decay function corresponding to the personalized forgetting curve to calculate the time decay factor.

[0136] The causal transmission attenuation coefficient is obtained by multiplying the distance attenuation factor, path strength factor, and time attenuation factor.

[0137] For example, in the process of identifying intervention targets, the accurate calculation of the causal transmission attenuation coefficient is one of the core steps. This coefficient is used to quantify the actual influence of candidate root cause nodes on the current learning difficulty nodes, comprehensively considering three dimensions: path topological distance, the strength distribution of dependent edges on the path, and the time forgetting effect, so that the final root cause contribution ranking is more consistent with the learner's true cognitive state.

[0138] Regarding the path length dimension, firstly, in the personalized knowledge topology, using the current learning difficulty node as the endpoint and the candidate root cause node as the starting point, a shortest path search is performed along the directed dependency edges to obtain the shortest dependency path from the candidate root cause node to the difficulty node. The number of directed edges traversed by this path, i.e., the hop count, is counted as the path length. ,in This is the index number of the candidate root cause node. Path length. The larger the value, the farther the candidate root cause node is from the difficult node in the knowledge topology, and the weaker its direct causal influence. The reciprocal of the path length is defined as the distance decay factor. ,Right now When the path length is 1, it indicates that the candidate root cause node is directly adjacent to the difficult node, and the distance decay factor reaches its maximum value of 1. As the path length increases, the distance decay factor decreases monotonically, reflecting the cognitive pattern that the longer the knowledge dependency chain, the lower the direct contribution of the distant node to the difficulty.

[0139] For the path strength dimension, the strength weight value corresponding to each directed dependency edge on the shortest dependency path is extracted, forming a strength weight sequence. Each element in this sequence is the edge weight of the corresponding dependency edge stored in the personalized knowledge topology, reflecting the tightness of the dependency relationship between adjacent knowledge nodes. The path strength factor is obtained by calculating the arithmetic mean of all strength weight values ​​in the sequence. Let the shortest dependency path have a total of Dependency edge, the first The strength weight of the strip edge is Then the path strength factor is The arithmetic mean, rather than the cumulative product, is used to avoid the cumulative product approaching zero excessively and causing a loss of numerical precision when the path is long. Furthermore, the arithmetic mean more robustly reflects the average propagation efficiency of the entire path. A higher path strength factor indicates a stronger connection between knowledge nodes along the dependency chain, and the more effectively the influence of candidate root cause nodes can propagate along the path to difficult nodes.

[0140] It should be noted that the path strength factor here... The calculation uses only the arithmetic mean of the weights of each edge on the path, along with the global parameters. The cumulative product of the strength weights of each dependent edge on the path is semantically fundamentally different: Used for the final composite calculation of root cause contribution, and It is specifically used for evaluating the path strength dimension within the causal transmission attenuation coefficient; the two should not be used interchangeably.

[0141] For the forgetting dimension, obtain candidate root cause nodes. The time difference is obtained by subtracting the timestamp of the last learning session from the current time. (Unit: days). Relying solely on a fixed exponential decay parameter is insufficient to accurately characterize the personalized forgetting patterns of different learners and different knowledge points; therefore, a personalized forgetting curve fitting mechanism is introduced. Two types of data are extracted from the historical learning records of this candidate root cause node: first, the sequence of intervals between each historical review; and second, the sequence of accuracy rates for each corresponding test. With the review interval as the independent variable and test accuracy as the dependent variable, an exponential decay function is used. Perform curve fitting, where These are the initial memory strength parameters. For candidate root cause nodes The personalized forgetting rate parameters obtained through fitting, This represents the time interval since the last learning iteration. The fitting process determines this by minimizing the mean squared error between the predicted accuracy and the measured accuracy. and The optimal value is obtained. After fitting, the measured time difference is... Substituting into the above exponential decay function, we obtain the time decay factor. Its range is limited to Within the range, indicating the passage An estimate of the retention rate of this knowledge node over time. Personalized forgetting rate. It varies from person to person and from knowledge point to knowledge point: for knowledge points that learners have historically reviewed frequently and whose test accuracy has been stable, the fitted result is... A smaller value indicates slower forgetting; conversely, for knowledge points with long review intervals and large fluctuations in accuracy, The larger the value, the faster the time decay factor decreases.

[0142] In obtaining the distance attenuation factor Path intensity factor and time decay factor Then, the three are multiplied to obtain the candidate root cause nodes. Causal transmission attenuation coefficient : The rationale for the product operation lies in the fact that the three factors independently characterize the attenuation effect of causal transmission from different dimensions: path topological distance determines structural attenuation, path strength determines attenuation in transmission efficiency, and time forgetting determines attenuation at the memory level. Only through the combined effect of these three factors can the actual influence of candidate root cause nodes on the current difficulty be fully reflected. When all three factors are close to 1, it indicates that the candidate root cause node is directly adjacent to the difficulty node, has a close dependency relationship, and has recently been effectively learned, and its causal transmission attenuation coefficient is close to its maximum value. If any factor approaches zero, the overall coefficient will be significantly reduced, thereby lowering the priority of that node being selected as the main intervention target in the subsequent root cause contribution ranking.

[0143] Calculated Then, it is compared with the candidate root cause nodes. Cognitive Deficit Index Multiply to obtain the root cause contribution. Therefore, in terms of intervention capacity constraints Next, sort the candidate root cause nodes by Sort in descending order to determine the final set of intervention targets. The introduction of personalized forgetting curves means that the causal transmission decay coefficient is no longer a static graph structure property, but a personalized quantity that changes dynamically with the learner's actual learning behavior, thus ensuring that the intervention strategy is highly adaptable to the learner's current cognitive state.

[0144] A second aspect of this invention provides an AI-based screen content recognition and knowledge graph construction system, comprising:

[0145] The data acquisition unit is used to acquire the learner's screen display content sequence and interactive behavior data related to the screen display content;

[0146] A semantic parsing unit is used to perform multimodal semantic parsing on the sequence of content displayed on the screen and extract structured knowledge units, wherein the structured knowledge units include knowledge representation information and cognitive requirement information;

[0147] The cognitive inference unit is used to infer the current cognitive state based on the structured knowledge unit and the interaction behavior data through the collaborative analysis of content semantic features and interaction behavior patterns, and to obtain a cognitive state representation.

[0148] The topology adjustment unit is used to perform personalized measurement of knowledge associations based on the global knowledge graph and the cognitive state representation, and dynamically adjust the topology of the knowledge graph according to the measurement results to obtain a personalized knowledge topology.

[0149] The intervention push unit is used to identify the causal relationship between the current learning difficulty and the historical knowledge state based on the personalized knowledge topology through temporal dependency analysis, determine the intervention object and intervention strategy under the intervention capacity constraint, generate suitable content and push it to the learner.

[0150] The feedback update unit is used to collect learning effect feedback and update the global knowledge graph.

[0151] A third aspect of the present invention provides an electronic device, comprising:

[0152] processor;

[0153] Memory used to store processor-executable instructions;

[0154] The processor is configured to invoke instructions stored in the memory to execute the aforementioned method.

[0155] A fourth aspect of the present invention provides a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the aforementioned method.

[0156] This invention can be a method, apparatus, system, and / or computer program product. The computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for performing various aspects of the invention.

[0157] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. An AI-based method for screen content recognition and knowledge graph construction, characterized in that, include: Acquire learners' screen display content sequences and interaction behavior data related to those screen displays; Multimodal semantic parsing is performed on the sequence of content displayed on the screen to extract structured knowledge units, which contain knowledge representation information and cognitive requirement information; Based on the structured knowledge units and the interaction behavior data, the current cognitive state is inferred through the collaborative analysis of content semantic features and interaction behavior patterns, thus obtaining a cognitive state representation; Based on the global knowledge graph and the cognitive state representation, knowledge relationships are measured in a personalized manner, and the topological structure of the knowledge graph is dynamically adjusted according to the measurement results to obtain a personalized knowledge topology; Based on the personalized knowledge topology, the causal relationship between current learning difficulties and historical knowledge status is identified through temporal dependency analysis. Under the constraint of intervention capacity, the intervention targets and intervention strategies are determined, and appropriate content is generated and pushed to learners. The global knowledge graph is updated based on the feedback on the learning results.

2. The method according to claim 1, characterized in that, Multimodal semantic parsing is performed on the sequence of content displayed on the screen to extract structured knowledge units. These structured knowledge units contain knowledge representation information and cognitive requirement information, including: Semantic analysis is performed on the text content and image content in the screen display content sequence. Key words and definitions are extracted from the text content as text features, and formulas and charts are identified from the image content and symbol complexity and structural features are extracted as visual features. The text features and visual features are synthesized into a multimodal knowledge representation vector through a cross-modal fusion mechanism; Based on the verb type, sentence structure, and the degree of abstraction of visual elements in the multimodal knowledge representation vector, the Bloom cognitive level of the knowledge point is determined, and the Bloom cognitive level encoding value and the corresponding cognitive operation type are used as cognitive requirement information. The multimodal knowledge representation vector and the cognitive requirement information are combined into a structured knowledge unit.

3. The method according to claim 1, characterized in that, Based on the structured knowledge units and the interaction behavior data, the current cognitive state is inferred through the collaborative analysis of content semantic features and interaction behavior patterns, resulting in a cognitive state representation, including: The cognitive complexity of extracting knowledge points from the structured knowledge unit includes conceptual abstraction, depth of prior dependencies, and symbol density. Micro-temporal features are extracted from the interaction behavior data. These micro-temporal features include the first interaction delay duration, operation sequence entropy value, and backtracking behavior frequency. The deviation of each micro-temporal feature from the cognitive complexity is calculated, and features with deviations exceeding a preset deviation threshold are marked as cognitive difficulty signals. For the cognitive difficulty signal, query the set of prerequisite knowledge points of the current knowledge point in the global knowledge graph, calculate the semantic similarity between the prerequisite knowledge points and the current interaction behavior pattern, and mark the prerequisite knowledge point with the highest semantic similarity as a suspected defect node. The cognitive complexity, the cognitive difficulty signal, and the suspected defective node are encoded into a multi-dimensional vector. Knowledge features and behavioral features are fused through a cross-modal alignment mechanism. The similarity between the fused vector and the predefined cognitive state prototype is calculated. The prototype with the highest similarity is selected as the cognitive state category. The fused vector and the cognitive state category are combined to form a cognitive state representation.

4. The method according to claim 3, characterized in that, Extracting micro-temporal features from the interaction behavior data and calculating the deviation includes: The time interval from the appearance of the content displayed on the screen to the first interactive operation is recorded as the first interaction delay duration. The corresponding delay benchmark value is queried according to the cognitive complexity, and the normalized deviation between the first interaction delay duration and the delay benchmark value is calculated. Extract the learner's operation sequence on the current screen display content, count the frequency of each operation type and calculate the Shannon entropy value of the frequency distribution as the operation sequence entropy value, query the corresponding entropy value benchmark value according to the cognitive complexity, and calculate the normalized deviation between the operation sequence entropy value and the entropy value benchmark value. Identify back actions in the operation sequence, including returning to the previous page, rereading viewed content, and undoing operations. Count the number of back actions occurring within a unit time window as the back action frequency. Query the corresponding frequency benchmark value based on the cognitive complexity, and calculate the normalized deviation between the back action frequency and the frequency benchmark value.

5. The method according to claim 1, characterized in that, Based on the global knowledge graph and the cognitive state representation, knowledge relationships are measured in a personalized manner, and the topological structure of the knowledge graph is dynamically adjusted according to the measurement results to obtain a personalized knowledge topology, including: For each associated edge in the global knowledge graph, the standard cognitive distance of its connected nodes is calculated. The standard cognitive distance is calculated based on the concept span, cognitive level difference, and dependency chain length. Based on the cognitive state representation, the learner's cognitive weak areas and cognitive strong areas are identified. The semantic distance between each knowledge node and the central node of the cognitive weak area and the semantic distance between each knowledge node and the central node of the cognitive strong area are calculated. The ratio of the two is used as the personalized cognitive preference coefficient of the node. For each associated edge in the global knowledge graph, a personalized adjustment factor is calculated based on the difference in personalized cognitive preference coefficients of its connected nodes. The standard cognitive distance is nonlinearly mapped to the personalized adjustment factor to obtain the personalized cognitive distance. When the personalized cognitive distance is less than a preset affinity threshold, the weight of the edge is increased; when the personalized cognitive distance is greater than a preset alienation threshold, the weight of the edge is weakened. Search for cross-level knowledge node pairs that have not yet established a direct connection but whose concept similarity is greater than a preset similarity threshold between the cognitively weak region and the cognitively strong region. Create temporary connection edges for node pairs that meet the conditions and calculate initial weights based on cross-domain distance and hierarchical differences. The adjusted edge weights and newly added temporary edges are superimposed on the global knowledge graph and normalized to obtain a personalized knowledge topology.

6. The method according to claim 1, characterized in that, Based on the personalized knowledge topology, causal relationships between current learning difficulties and historical knowledge states are identified through temporal dependency analysis. Under intervention capacity constraints, intervention targets and strategies are determined, and appropriate content is generated and pushed to learners, including: Extract the knowledge node corresponding to the current learning difficulty from the personalized knowledge topology, query the set of preceding dependent edges of the knowledge node, traverse backward along the preceding dependent edges to each preceding knowledge node, calculate the cognitive deficiency index of each preceding knowledge node based on the historical cognitive state, and mark it as a candidate root cause node when the cognitive deficiency index exceeds the preset index threshold. For each candidate root cause node, the causal transmission attenuation coefficient is calculated based on the path length from the node to the difficult node, the strength weight of the dependent edges on the path, and the time interval. The product of the cognitive deficit index and the causal transmission attenuation coefficient is used as the root cause contribution and sorted. The upper limit of the number of nodes for a single intervention is set based on the average number of knowledge points completed in a learner's historical learning session. The nodes are selected sequentially from the sorted candidate root cause nodes until the upper limit is reached, and the selected nodes are used as the set of intervention objects. For each node in the set of intervention targets, the level of intervention intensity is determined based on the cognitive deficit index and the root cause contribution, and the type of intervention content is determined based on the knowledge type. These are combined as intervention strategies, and matching materials are retrieved from the content template library according to the intervention strategies to generate appropriate content and push it to the learner.

7. The method according to claim 6, characterized in that, Calculating the causal transmission attenuation coefficient includes: Extract the shortest dependency path from candidate root cause nodes to difficult nodes, count the number of hops of the shortest dependency path as the path length, and calculate the reciprocal of the path length as the distance decay factor. Extract the strength weight values ​​of each dependent edge on the path, and calculate the arithmetic mean of all strength weight values ​​as the path strength factor; Obtain the time difference between the last learning timestamp of the candidate root cause node and the current time. Fit a personalized forgetting curve based on the review interval and test accuracy data in the historical learning record of the node. Substitute the time difference into the exponential decay function corresponding to the personalized forgetting curve to calculate the time decay factor. The causal transmission attenuation coefficient is obtained by multiplying the distance attenuation factor, path strength factor, and time attenuation factor.

8. An AI-based screen content recognition and knowledge graph construction system, used to implement the method as described in any one of claims 1-7, characterized in that, include: The data acquisition unit is used to acquire the learner's screen display content sequence and interactive behavior data related to the screen display content; A semantic parsing unit is used to perform multimodal semantic parsing on the screen display content sequence and extract structured knowledge units, wherein the structured knowledge units include knowledge representation information and cognitive requirement information; The cognitive inference unit is used to infer the current cognitive state based on the structured knowledge unit and the interaction behavior data through the collaborative analysis of content semantic features and interaction behavior patterns, and to obtain a cognitive state representation. The topology adjustment unit is used to perform personalized measurement of knowledge associations based on the global knowledge graph and the cognitive state representation, and dynamically adjust the topology of the knowledge graph according to the measurement results to obtain a personalized knowledge topology. The intervention push unit is used to identify the causal relationship between the current learning difficulty and the historical knowledge state based on the personalized knowledge topology through temporal dependency analysis, determine the intervention object and intervention strategy under the intervention capacity constraint, generate suitable content and push it to the learner. The feedback update unit is used to collect learning effect feedback and update the global knowledge graph.

9. An electronic device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to invoke instructions stored in the memory to execute the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1 to 7.

Images