Intelligent consolidation method based on space-time attention mechanism and application thereof

Abstract

The invention discloses an intelligent consolidation method based on a space-time attention mechanism and an application thereof, and relates to the technical field of educational learning methods, and the method comprises the following steps: constructing a space-time attention model: taking a review feature and a video feature as input, constructing a space-time attention mechanism model comprising a time attention layer and a space attention layer, and the model can respectively output attention distribution scores of each time period every day and spatial attention distribution of each review content and knowledge point. Through space-time attention fusion, time-content coupling representation is obtained through cross attention, the defects that a review strategy is monotonous and cannot be explained due to single-dimension modeling are overcome, the situation that a user pays less attention to a certain type of knowledge points in the time period of collusion can be recognized, and the user experience is improved. The generated consolidation plan can adjust the review time window and the review content sequence and duration at the same time, the review hit rate and individuation effect are improved, the system interpretability is improved, and teaching intervention is facilitated.

Description

Technical Field

[0001] This invention relates to the field of educational learning methods, specifically to an intelligent consolidation method based on spatiotemporal attention mechanisms and its application. Background Technology

[0002] Existing technologies in educational learning systems include, but are not limited to: spaced repetition / forgetting curve methods (such as spaced review based on the Ebbinghaus forgetting curve), adaptive learning systems (adjusting question difficulty based on scores / answer history), mastery prediction based on knowledge tracing, and research on estimating attention using behavioral data such as eye movement / facial movements. However, these products have obvious technical shortcomings in practical applications: existing systems either only use answer logs and knowledge tracing, or only use behavioral and attention indicators, lacking the ability to jointly model "when attention is paid" and "what knowledge points and question types are paid attention to" and use them for review decisions.

[0003] Patent CN113380087B discloses a method and system for reciting and memorizing English words based on virtual reality scenes. The above patent solves the problem that existing methods for reciting and memorizing English words are not intelligent enough and cannot evaluate the student's mastery of each English word or provide review suggestions for each English word.

[0004] The aforementioned patent can reasonably set review time based on students' current mastery of English words, and has a high degree of intelligence, but it lacks the ability to jointly model and use it for review decision-making.

[0005] To this end, this application proposes an intelligent consolidation method based on a spatiotemporal attention mechanism that achieves time-content coupled representation through cross-attention and its application. Summary of the Invention

[0006] The purpose of this invention is to provide an intelligent consolidation method and its application based on a spatiotemporal attention mechanism, in order to solve the technical problem mentioned in the background art that existing systems either only use answer logs and knowledge tracking, or only use behavior and attention indicators, and lack the ability to jointly model "when to pay attention" and "what knowledge points and question types were paid attention to" and use them for review decision-making.

[0007] To achieve the above objectives, the present invention provides the following technical solution: an intelligent consolidation method based on a spatiotemporal attention mechanism, wherein the intelligent consolidation method includes the following steps:

[0008] A. Data Collection: Receive and store the review-related data recorded by the user every day. The review-related data includes, but is not limited to: review subjects, review time periods, review content and corresponding duration, incorrect questions, difficult questions, and unfamiliar questions, the user's complete answer process, reference answers, self-test answer accuracy rate and memory retention rate. At the same time, video data of the user's review process is collected through a camera or eye-tracking device. The video data includes eye-tracking data and body movement data.

[0009] B. Preliminary review plan generation: Based on the data recorded daily, a web crawler is used to obtain the knowledge graph of the subject, and knowledge points are extracted from the user's subjective experience, reference answers, and user answers through NLP. The extracted knowledge points are associated with the nodes of the knowledge graph, and a preliminary review plan is generated by combining the user's memory code (forgetting curve).

[0010] C. Feature Extraction and Quantization: Preprocess the video data (denoising, frame extraction and annotation) to extract eye movement features (fixation duration, saccades, scan rate, pupil diameter) and body movement features (head movement frequency, sitting posture, posture change frequency) from the data. Calculate the review efficiency and daily comprehensive review effect based on the review behavior data.

[0011] D. Constructing a spatiotemporal attention model: Using review features and video features as input, construct a spatiotemporal attention mechanism model that includes a time attention layer and a spatial attention layer, so that the model can output the attention distribution score for each time period of the day and the spatial attention distribution of each review content and knowledge point.

[0012] Preferably, the intelligent consolidation method further includes the following steps:

[0013] E. Dynamic Consolidation Plan Generation: Based on the distribution of attention over time and space, the preliminary review plan, and memory codes, the review order, the review time for each item, and the types of recommended learning resources are dynamically adjusted to generate a personalized review and consolidation plan for the next stage.

[0014] F. Iterative Optimization: After the user executes the next stage plan and collects the review data and video data for the new stage, the spatiotemporal attention model is updated online or periodically retrained based on the new data, and steps C to E are executed in a loop.

[0015] Preferably, in the generation of the preliminary review plan, the NLP knowledge point extraction includes: identifying candidate knowledge points from the question stem, reference answer and user answer based on dependency parsing and entity relation extraction, and identifying key knowledge points that the user has missed or has not mastered by matching with nodes of the knowledge graph, thereby including such key knowledge points in the priority list of the preliminary review plan.

[0016] Preferably, in the feature extraction and quantification, the calculation steps for the daily comprehensive review effect Z are as follows: applying the weights a determined by the decision tree model to the answer accuracy D and memory retention J of each subject. 答题准确率 With a 记忆保持率 Z is calculated using linear combinations, and based on Z, multiple subjects are sorted from weakest to strongest to determine the subjects to be reviewed first.

[0017] Z=D×a 答题准确率 +J×a 记忆保持率

[0018] Analysis of weak areas for review: Based on the subjects, content, and duration of review during each review period of the day, analyze the efficiency of redoing questions and reviewing knowledge points.

[0019] Let X be the efficiency of different review topics, and T be the review time. Taking the review efficiency of a certain subject as an example, the review and answering efficiency is:

[0020] X 科目答题 =D 科目 ÷T 科目答题 .

[0021] Preferably, the video feature quantization adopts a unified scoring rule: for each individual indicator, the range is calculated based on the maximum and minimum values ​​of its observed values ​​and a preset number of grades;

[0022] (Maximum value of a single indicator - Minimum value of a single indicator) ÷ Number of grades = Range;

[0023] The metrics are then mapped to an attention score range of 0 to 10 to obtain stable attention measurement results.

[0024] Preferably, the spatiotemporal attention mechanism model is a multi-head self-attention structure, comprising: a temporal attention layer for modeling time series features (using positional encoding and time window segmentation), a spatial attention layer for modeling spatial content attention (performing self-attention calculation on knowledge points and question sequences), and a cross-attention module for fusing temporal and spatial information;

[0025] The model training employs a hybrid loss function combining supervised and contrastive learning to simultaneously optimize the interpretability of the attention distribution and the accuracy of predicting the review effect.

[0026] Preferably, the dynamic consolidation plan generation further includes: identifying user preferences for different explanation formats (text explanation, video explanation, illustration explanation, etc.) based on the spatial attention layer, and using the preferences as the learning resource recommendation weight, so as to output the mapping relationship between review tasks and recommended resources when generating the review plan;

[0027] Iterative optimization includes two parts: online fine-tuning and offline retraining. When the amount of new phased data is less than the threshold, online fine-tuning is used to ensure real-time performance. When the accumulated data reaches the retraining threshold, offline batch retraining is performed and the weight coefficients in the decision tree are re-estimated to update the review effect calculation formula.

[0028] Preferably, the step of constructing the spatiotemporal attention model includes a compensation step for environmental noise and individual differences: before inputting the model, the video and eye-tracking data are subjected to environmental adaptation correction (considering factors such as lighting and camera angle), and the baseline shift caused by individual physiological differences is corrected through user profiles to reduce errors and improve the consistency of attention measurement.

[0029] Preferably, the intelligent consolidation method supports phased analysis based on the monitoring period set by the user's memory password on a time scale. The memory password is the user's forgetting curve sequence for a certain subject (examples are 1, 2, 4, 7, and 15 days). The system determines the phased monitoring duration and data aggregation window based on the memory password to output a consolidation plan aligned with the memory period.

[0030] Preferably, the system includes;

[0031] User information management module: used to receive and store the user's daily review data and video data from the camera or eye-tracking device;

[0032] Knowledge Graph and NLP module: used to capture subject knowledge graphs, analyze questions, extract knowledge points, and associate them with the knowledge graph;

[0033] Feature processing module: used for video preprocessing, extracting eye movement and body motion features and performing unified scoring mapping;

[0034] Spatiotemporal attention model module: used to carry and execute the spatiotemporal attention mechanism model and output the temporal attention distribution and spatial attention distribution;

[0035] The learning resource module and the review and consolidation plan creation module are used to combine attention distribution, preliminary review plans and user memory codes to generate and distribute personalized review and consolidation plans for the next stage and corresponding learning resource recommendations.

[0036] Storage and Iteration Module: Used to save historical review data and model parameters, and perform online fine-tuning or offline retraining according to the set strategy;

[0037] The application system for the consolidation method is configured as an embedded learning software, learning machine, or networked teaching hardware.

[0038] Compared with the prior art, the beneficial effects of the present invention are:

[0039] 1. This invention achieves time-content coupled representation through spatiotemporal attention fusion by obtaining cross-attention, overcoming the shortcomings of monotonous and uninterpretable review strategies caused by single-dimensional modeling. It can identify situations where "users do not pay enough attention to a certain type of knowledge point in a certain time period", enabling the generated consolidation plan to adjust the review time window and the order and duration of review content at the same time, improving the review hit rate and personalization, improving the interpretability of the system, and facilitating teaching intervention.

[0040] 2. This invention, through multimodal behavioral features and knowledge graph-driven priority determination, can accurately locate learners' knowledge weaknesses and provide suggestions for hierarchical knowledge related to the knowledge graph. It solves the problem of "how to directly correspond behavioral attention with content knowledge points and arrange review accordingly", avoiding the bias of making decisions based solely on behavior or answer records, thereby improving the pertinence and coherence of review and reducing ineffective review time.

[0041] 3. This invention models learning resource format preferences and embeds resource recommendations within the review plan, making preferences a weighted factor in the generation of the review plan. It outputs a "review-task-recommended resource" mapping, solving the problem that existing systems only distribute questions without considering the learner's most suitable learning method. It avoids low completion rates caused by mismatches between resources and learner preferences and attention patterns, improves the utilization rate of learning resources and review completion rate, shortens the closed-loop time from recommendation to learning and consolidation, and adopts the optimal interpretation form for different knowledge points, thereby improving memory conversion efficiency.

[0042] 4. This invention achieves alignment between monitoring and planning with the memory password by using a hierarchical iterative training strategy and coupled monitoring, thus satisfying the contradiction between real-time performance and model stability and generalization. It also solves the problem of aligning the consolidation plan with the individual memory cycle. While ensuring personalized real-time response, it avoids model drift through periodic offline retraining, improves long-term performance, and enhances the scientific nature of review timing by coupling with the memory password, thereby significantly improving long-term memory retention and learning efficiency. Attached Figure Description

[0043] Figure 1 This is a schematic diagram of the application system framework of the intelligent consolidation method of the present invention. Detailed Implementation

[0044] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0045] In the description of this invention, it should be noted that the terms "upper," "lower," "inner," "outer," "front end," "rear end," "both ends," "one end," and "the other end," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing this invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention. Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0046] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installed," "equipped with," "connected," etc., should be interpreted broadly. For example, "connection" can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium; it can be a connection within two components. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0047] Please see Figure 1 This invention provides an embodiment of an intelligent consolidation method based on a spatiotemporal attention mechanism and its application, including an intelligent consolidation process based on spatiotemporal attention in a mobile learning app:

[0048] Students open the review task, and the app starts recording with the camera and collecting answer logs locally. Each answer generates a log entry with the start and end times marked. The app performs a quick check on the video locally: if there is insufficient lighting or occlusion, it tags the log and switches to answer log-only mode. On the device, the app performs frame sampling of the video and uses a lightweight model to extract coarse-grained features: presence / absence of face, head lateral angle, and gesture determination (whether the student is looking at the question booklet or touching the screen). If an eye-tracking peripheral is available, it collects the gaze point and pupil diameter sequences. The extracted feature vectors and answer logs are uploaded to the cloud in batches. The server performs NLP on the sensory and reference analysis to obtain a set of candidate knowledge points and a knowledge graph node mapping, generating a question-to-knowledge point mapping table. Time series features (the time window of the most recent N answers, such as daily aggregation within 14 days), spatial features (embedded candidate knowledge points), and behavioral features are input into the spatiotemporal attention model, which outputs time attention distribution (e.g., attention scores in 24-hour segments) and spatial attention scores (0-10 subscales for knowledge points). Using spatial attention level, knowledge point error rate, knowledge point difficulty, and memory password priority as inputs, the review priority score for each knowledge point is calculated. Top-K items are selected according to priority, and review time and resource format (text, short video, interactive questions) are allocated. The APP receives the plan and displays "Why it is recommended" (interpretable text based on attention distribution and knowledge graph) in the UI. Students begin execution, and the system continues to collect execution process data for subsequent iterations.

[0049] Example running scenario:

[0050] User A completed 30 questions between 20:10 and 20:40 on September 1, 2025. The system collected data showing an average fixation time of 450ms and a relatively high head turn (indicating superficial inattention). The user's accuracy rate on "Knowledge Point KP-23" over the past 7 days was 40%.

[0051] Model output: Spatial attention KP-23 score 2.1 (low), temporal attention score 3.0 (low to medium) between 20:00 and 21:00.

[0052] Projection Plan: In the next round of consolidation (scheduled for 2025-09-02, 09:00), KP-23 will be prioritized. A 6-minute short video explanation and 5 targeted exercises are recommended. The reason given is: "Attention is low in the evening, so it is recommended to review briefly the next day to achieve better memory consolidation."

[0053] Please see Figure 1 This invention provides an embodiment of an intelligent consolidation method based on a spatiotemporal attention mechanism and its application, which enables the implementation of knowledge point extraction and knowledge graph mapping.

[0054] The question stem and reference analysis are segmented into sentences, Chinese words are segmented, and stop words are removed, while part-of-speech tagging is performed; the original identifiers of proper terms and mathematical formula fragments are preserved; the core noun phrases and verb-object structures are extracted through dependency parsing, and a candidate list of terms is output through regular dictionary matching; three types of confidence are calculated for each candidate term: dictionary matching confidence (whether it is in the KG thesaurus), semantic similarity (cosine similarity between the candidate sentence vector and the KG node vector), and contextual evidence (frequency of key verbs or modifiers co-occurring with the term in the question stem), and the final confidence is obtained by weighted summation (shown in the figure). (e.g., weights 0.5 / 0.3 / 0.2); if the confidence level is ≥0.75, it is directly mapped to the corresponding KG node; if the confidence level is between 0.6 and 0.75, it is marked as "approximate match" and accumulated in the candidate pool in the background for manual or subsequent model confirmation; if the confidence level is 0.6, it is considered a new concept candidate and needs to be manually supplemented or merged with existing nodes through clustering; three indicators are added to the knowledge point mapped to each question: the average error rate of the knowledge point in the last 30 days, the difficulty coefficient of the knowledge point, and the correlation with the solution path of the current question; and an initial priority score is assigned to each KP according to these indicators;

[0055] Example process:

[0056] Question: "Calculate the acceleration of an object with a mass of 2 kg under a force of 5 N".

[0057] Dependency parsing extracts candidate terms: "acceleration", "force-mass relationship";

[0058] Semantic similarity: Cosine similarity with the knowledge graph node "acceleration" is 0.92 → direct mapping; similarity with "force-mass relationship" is 0.62 → marked as an approximate match and included in the manual confirmation pool;

[0059] Final output.

[0060] Please see Figure 1This invention provides an embodiment of an intelligent consolidation method based on a spatiotemporal attention mechanism and its application. The complete spatiotemporal attention model (main model) is stored in the cloud; distilled smaller models are deployed on edge devices with limited local storage and feature extraction capabilities; differential updates (incremental model packages) and secure model retrieval mechanisms (TLS + signature verification) are supported; knowledge distillation is performed on the large model in the cloud: the large model acts as the teacher, and the smaller models act as the students. The student model is trained by minimizing the KL divergence between student predictions and teacher predictions, aiming to compress the parameters to 8%-12%; the edge devices execute this method in real time. Feature extraction is performed using a distillation model for inference, returning a "high-risk knowledge point" identifier and a small vector instead of a complete attention heatmap, reducing uplink traffic. When the cloud receives sufficient recent behavior data reported by the device (e.g., a user has more than 50 new samples), the cloud performs minor adjustments on the user's data (several epochs, low learning rate), generates a differential weight package, and sends it to the user's device. If the device is offline, a temporary plan is generated using the previously sent attention distribution and rule engine (based on historical priority). Logs during offline periods are cached locally and synchronized in batches when the device is connected to the network.

[0061] Example running scenario:

[0062] When the school network is stable, the learning machine receives differential update packets and updates the local distillation model; during evening self-study, students use the local model to quickly obtain personalized contacts; if the home network is disconnected, the device continues to distribute review tasks locally using the rule engine, and after reconnecting to the network, it synchronizes 3 days of logs to the cloud.

[0063] Please see Figure 1 The present invention provides an embodiment of an intelligent consolidation method based on a spatiotemporal attention mechanism and its application, which is used for class-level and teacher-level aggregation analysis and teaching intervention. Teachers have the right to view class aggregation reports. The system aggregation logic is based on the weighted average of student attention vectors. Student data has been desensitized and processed in accordance with policy compliance.

[0064] The system aligns individual students' spatial attention scores with their temporal attention distribution by knowledge point and time period (unifying them to the same granularity, such as knowledge point granularity and hourly granularity), and calculates a weighted average for all students in the class. It sets alarm rules: when the average attention for a knowledge point is <3 and the average accuracy is <60%, a "class weakness alarm" is triggered, and the number of triggers within a week is recorded. Based on the weakness type and the higher-level nodes in the knowledge graph, the system automatically generates intervention suggestion templates (such as "arrange a 15-minute thematic lecture + two mind map questions + categorized exercises"), and provides suggestions for suitable time periods (prioritizing periods with higher average time attention in the class). Teachers can view reports on their devices and issue basic tasks or create classroom activities with one click. After class, teachers can upload classroom feedback (whether a lecture was given, student participation), and the system uses this information as a monitoring signal for subsequent models.

[0065] Example scenario:

[0066] In the "Differential Application" unit, the system detected that 35% of students had a spatial attention score of <3 on the KP-78 test and an average accuracy rate of 52%, triggering a weakness alert. The system suggested a 12-minute special lecture on this topic during the first period on Monday morning of the following week, along with five targeted practice problems. After the teacher adopted this suggestion, the system tracked the students' answer sheets after class the next day. If the accuracy rate improved to 7%, the system recorded it as an effective intervention.

[0067] Please see Figure 1 The present invention provides an embodiment of an intelligent consolidation method based on spatiotemporal attention mechanism and its application, which is adapted for special populations (attention deficit disorder, visual impairment, etc.).

[0068] Users fill in and select their adaptation needs during registration or first-time use (or these needs are labeled by teachers or parents). The system saves user adaptation tags and adjusts their weights during model input. For users with attention deficit disorder, the system automatically reduces the weight of eye movement and video features (e.g., from 40% to 10%) and increases the weight of answering behavior and short-term test feedback (from 60% to 90%). Short, high-frequency tasks are recommended: limit the duration of each task to 3-5 minutes, increase the frequency of task intervals (e.g., once every 20 minutes), and provide clear, immediate feedback and positive incentives (badges, phrase rewards) after each task. For visually impaired students, prioritize audio explanations and large-font text, and use tactile and voice operations to replace gesture interactions. If video is unavailable, the system uses voice interaction and answer logs as the main input. The system evaluates the adaptation effect (review completion rate, accuracy rate, user and teacher subjective feedback) weekly and automatically adjusts parameters (e.g., further reducing video weight or increasing task segments).

[0069] Example scenario:

[0070] Student B was labeled as having attention deficit disorder. The system broke down his review task into 6 four-minute mini-tasks with a 10-minute break between tasks. After 14 consecutive days, the baseline data showed that the review completion rate increased from 45% to 72%, and the average accuracy rate increased by 9%.

[0071] Please see Figure 1 This invention provides one embodiment: an intelligent consolidation method based on a spatiotemporal attention mechanism and its application, involving staged monitoring and plan generation coupled with a memory code (forgetting curve):

[0072] Each user and subject is configured with a memory password sequence (default [1, 2, 4, 7, 15]), and the system can trigger backtesting and monitoring tasks for the corresponding number of days every day at a set time.

[0073] Using the nth day of the memory code as the trigger point (e.g., Day 1, Day 2, Day 4, etc.), the system automatically generates end-tests or mini-review tasks on the trigger day and collects answer logs and behavioral features. The data collected on the trigger day is scored for temporal and spatial attention, and the spatial attention at this stage is compared with the attention at the initial learning stage to calculate attention drop. If the attention drop of a certain knowledge point on the trigger day is greater than 2 and the answer accuracy drops by more than 10%, then the knowledge point is inserted into the next memory window (e.g., inserting extra review from Day 4 into Day 2) and its priority score is increased. If attention remains stable and the accuracy is greater than 85%, then the future occurrence frequency of that knowledge point is reduced. The results of the interim retests are used as supervision labels to train Lsup to improve the spatiotemporal attention model's ability to predict long-term memory retention.

[0074] Example timeline:

[0075] Day 0: The user completes the learning and passes a formative test. The initial accuracy rate of KP-12 is 78%, and the initial spatial attention is 7.8.

[0076] Day 1 (Trigger): The system pushes another short test. KP-12 accuracy drops to 62 (down 16), spatial attention to 4.5 (down 3.3) → meets the insertion rule → the system arranges priority review and increased resources (videos + practice) on Day 2.

[0077] Day 4 (Re-triggered): If the accuracy rate rises to 80% and the attention rate returns to 7+, the system will restore the normal occurrence frequency of KP-12.

[0078] illustrate:

[0079] 1. During the user's daily review period, first generate an unprocessed review and reinforcement plan based on attention scores from highest to lowest, then rearrange it chronologically to generate a processed review and reinforcement plan:

[0080] Table 1 below shows an example of an unprocessed review and consolidation plan:

[0081] Example table of unprocessed review and consolidation plan

[0082]

[0083] 2. Scoring Rules

[0084] Because users' eye movement data and body movement data are subject to individual differences, in order to reduce the impact of errors caused by individual differences during quantitative analysis, this patent uniformly adopts the method of "(maximum value of a single indicator - minimum value of a single indicator) ÷ number of levels" to determine the grade difference of each level, and then assigns a score (maximum score of 10 points).

[0085] For example, if a user's attention span is a maximum of 2 minutes and a minimum of 1 minute, then the difference between each level is 12 seconds. Therefore, Level 1 is a range of 1 minute 49 seconds to 2 minutes, and is assigned 10 points; Level 2 is a range of 1 minute 37 seconds to 1 minute 49 seconds, and is assigned 8 points... and so on for other data indicators.

[0086] The scoring rules are shown in Table 2 below: Eye-tracking data scoring rules table:

[0087] Eye-tracking data scoring rules table

[0088]

[0089] Working principle: The system collects learners' answer logs, question texts and reference explanations in real time or periodically, as well as optional behavioral and video eye-tracking data. The text information is processed by NLP and compared with the knowledge graph to obtain the question → knowledge point mapping. The behavioral video and eye-tracking data are preprocessed and feature extracted to be converted into stable attention quantification features. These heterogeneous features are uniformly encoded into temporal features and content embeddings to provide structured input for subsequent spatiotemporal attention modeling.

[0090] A spatiotemporal attention mechanism is used to jointly model the input: the temporal attention module describes the attention distribution on different time windows, and the spatial attention module fuses temporal and spatial information to obtain a temporal-content coupled representation. The model training adopts a hybrid loss that combines supervised loss and contrastive learning, and entropy regularization is added to improve the sparsity and interpretability of attention, so that the output can both predict the posterior review effect and provide an interpretable "why recommend" attention proof.

[0091] The system generates dynamic, personalized reinforcement plans based on the model's output of temporal and spatial attention distribution, knowledge graph priority, historical error rate, and user memory passwords, or learnable decision functions. The plans are then distributed to terminals for execution. The system also supports online fine-tuning to quickly adapt to individual behavioral changes and offline retraining to ensure global model stability. By mapping multimodal attention to specific knowledge points and providing visual explanations, the method not only improves review hit rate and memory retention but also facilitates understanding by teachers or learners for teaching intervention and engineering deployment.

[0092] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims.

Claims

1. An intelligent consolidation method based on a spatiotemporal attention mechanism, characterized in that: The method for consolidation includes the following steps: A. Data Collection: Receive and store the review-related data recorded by the user every day, and at the same time collect video data of the user's review process through a camera or eye-tracking device; B. Preliminary review plan generation: Based on the data recorded daily, a web crawler is used to obtain the knowledge graph of the subject, and NLP is used to extract knowledge points from the user's perception, reference answers and user answers. The extracted knowledge points are associated with the knowledge graph nodes, and a preliminary review plan is generated by combining the user's memory code. C. Feature extraction and quantification: Preprocess the video data to extract eye movement features and body movement features, and calculate the review efficiency and daily comprehensive review effect based on the review behavior data; D. Constructing a spatiotemporal attention model: Using review features and video features as input, construct a spatiotemporal attention mechanism model that includes a time attention layer and a spatial attention layer, so that the model can output the attention distribution score for each time period of the day and the spatial attention distribution of each review content and knowledge point.

2. The intelligent consolidation method based on spatiotemporal attention mechanism according to claim 1, characterized in that: The intelligent consolidation method further includes the following steps: E. Dynamic Consolidation Plan Generation: Based on the distribution of attention over time and space, the preliminary review plan, and memory codes, the review order, the review time for each item, and the types of recommended learning resources are dynamically adjusted to generate a personalized review and consolidation plan for the next stage. F. Iterative Optimization: After the user executes the next stage plan and collects the review data and video data for the new stage, the spatiotemporal attention model is updated online or periodically retrained based on the new data, and steps C to E are executed in a loop.

3. The intelligent consolidation method based on spatiotemporal attention mechanism according to claim 2, characterized in that: In the generation of the preliminary review plan, NLP knowledge point extraction includes: identifying candidate knowledge points from the question stem, reference answer and user answer based on dependency parsing and entity relation extraction, and identifying key knowledge points that the user has missed or has not mastered by matching them with nodes of the knowledge graph, thereby including such key knowledge points in the priority list of the preliminary review plan.

4. The intelligent consolidation method based on spatiotemporal attention mechanism according to claim 3, characterized in that: In the feature extraction and quantification process, the calculation steps for the daily comprehensive review effect Z are as follows: For each subject, the accuracy rate D and memory retention J are weighted by a weight determined by the decision tree model. 答题准确率 With a 记忆保持率 Z is calculated using linear combinations, and based on Z, multiple subjects are sorted from weakest to strongest to determine the subjects to be reviewed first. Z=D×a 答题准确率 +J×a 记忆保持率 Analysis of weak areas for review: Based on the subjects, content, and duration of review during each review period of the day, analyze the efficiency of redoing questions and reviewing knowledge points. Let X be the efficiency of different review topics, and T be the review time. Taking the review efficiency of a certain subject as an example, the review and answering efficiency is: X 科目答题 =D 科目 ÷T 科目答题 。 5. The intelligent consolidation method based on spatiotemporal attention mechanism according to claim 4, characterized in that: The video feature quantization adopts a unified scoring rule: for each individual indicator, the range is calculated based on the maximum and minimum values ​​of its observed values ​​and the preset number of grades; (Maximum value of a single indicator - Minimum value of a single indicator) ÷ Number of grades = Range; The metrics are then mapped to an attention score range of 0 to 10 to obtain stable attention measurement results.

6. The intelligent consolidation method based on spatiotemporal attention mechanism according to claim 5, characterized in that: The spatiotemporal attention mechanism model is a multi-head self-attention structure, which includes: a temporal attention layer for modeling time series features, a spatial attention layer for modeling spatial content attention, and a cross-attention module for fusing temporal and spatial information. The model training employs a hybrid loss function combining supervised and contrastive learning to simultaneously optimize the interpretability of the attention distribution and the accuracy of predicting the review effect.

7. The intelligent consolidation method based on spatiotemporal attention mechanism according to claim 2, characterized in that: The dynamic consolidation plan generation further includes: identifying users' preferences for different teaching methods based on the spatial attention layer, and using these preferences as the recommendation weights for learning resources, thereby simultaneously outputting the mapping relationship between review tasks and recommended resources when generating the review plan; Iterative optimization includes two parts: online fine-tuning and offline retraining. When the amount of new phased data is less than the threshold, online fine-tuning is used to ensure real-time performance. When the accumulated data reaches the retraining threshold, offline batch retraining is performed and the weight coefficients in the decision tree are re-estimated to update the review effect calculation formula.

8. The intelligent consolidation method based on spatiotemporal attention mechanism according to claim 1, characterized in that: Before constructing the spatiotemporal attention model, a compensation step for environmental noise and individual differences is also included: before inputting the model, environmental adaptation correction is performed on the video and eye-tracking data, and baseline shift caused by individual physiological differences is corrected through user profiles to reduce errors and improve the consistency of attention measurement.

9. The intelligent consolidation method based on spatiotemporal attention mechanism according to claim 8, characterized in that: The intelligent consolidation method supports phased analysis based on the monitoring period set by the user's memory password. The memory password is the forgetting curve sequence of the user for a certain subject. The system determines the phased monitoring duration and data aggregation window based on the memory password to output a consolidation plan aligned with the memory period.

10. An application system for an intelligent consolidation method based on a spatiotemporal attention mechanism according to claim 9, characterized in that: The system includes; User information management module: used to receive and store the user's daily review data and video data from the camera or eye-tracking device; Knowledge Graph and NLP module: used to capture subject knowledge graphs, analyze questions, extract knowledge points, and associate them with the knowledge graph; Feature processing module: used for video preprocessing, extracting eye movement and body motion features and performing unified scoring mapping; Spatiotemporal attention model module: used to carry and execute the spatiotemporal attention mechanism model and output the temporal attention distribution and spatial attention distribution; The learning resource module and the review and consolidation plan creation module are used to combine attention distribution, preliminary review plans and user memory codes to generate and distribute personalized review and consolidation plans for the next stage and corresponding learning resource recommendations. Storage and Iteration Module: Used to save historical review data and model parameters, and perform online fine-tuning or offline retraining according to the set strategy; The application system for the consolidation method is configured to embed learning software, learning machines, or networked teaching hardware.

Images