A multi-modal ai-based dynamic individualized education plan system and method

Abstract

This invention discloses a dynamic individualized education plan (IEP) system and method based on multimodal AI. The system comprises a perception layer, a fusion layer, a reasoning layer, and an interaction layer, sequentially connected to form a continuous closed loop of perception, fusion, reasoning, intervention, and feedback. The perception layer collects student behavior and interaction data via edge cameras and tablets; the original video is anonymized, outputting only anonymous skeletal pose coordinates. The fusion layer extracts behavioral and interaction features, generating a 192-dimensional behavioral state vector through cross-modal attention fusion, with each dimension corresponding to an IEP evaluation metric. The reasoning layer generates content based on a large language model, constrained by a structured RAG knowledge base. The interaction layer proactively pushes role-based intervention cards via a hybrid trigger engine, updating the legally mandated IEP file after manual review and confirmation. This invention achieves a leap from static assessment to dynamic digital twin for IEP, applicable to individualized education.

Description

Technical Field

[0001] This invention relates to the field of automated generation of individualized education, specifically to a dynamic individualized education plan system and method based on multimodal AI. Background Technology

[0002] Individualized Education Programs (IEPs) are the cornerstone of special education, requiring the development of long-term and short-term learning goals based on each student's current abilities and needs, and regular assessments. However, traditional IEP assessment processes suffer from the following prominent problems:

[0003] First, the assessment method is static and singular. Traditional IEP assessments usually rely on teachers' quarterly or annual scale scores and periodic observations, which cannot capture students' continuous development and changes, and the assessment results are difficult to reflect the true fluctuations in students' performance at different times and in different classroom activities.

[0004] Second, the data sources are limited. Existing assessment methods mainly rely on teachers' subjective observation records, lacking the simultaneous collection of multi-dimensional information such as behavior, interaction, and emotions. This results in limited assessment dimensions and makes it difficult to comprehensively reflect students' overall abilities.

[0005] Third, there is a significant lag in translating assessment results into teaching interventions. Teachers often only take remedial measures after problems are identified, rather than optimizing teaching strategies based on real-time data, resulting in insufficient timeliness and targeted interventions.

[0006] Fourth, student privacy protection is a prominent issue. If a cloud-based video analytics solution is adopted, students' raw video data needs to be uploaded to a cloud server, posing a risk of privacy leaks and making it difficult to meet increasingly stringent data protection regulations.

[0007] In recent years, artificial intelligence technology has begun to be applied in the field of special education. Some studies have attempted to automatically generate IEP (Individual Education Execution) texts using natural language processing techniques, or to perform single-action behavior analysis using computer vision. For example, some studies have used generalized large language models to generate IEP goal suggestions based on student descriptions, but generalized models lack special education expertise constraints and are prone to generating unfounded "illusionary" suggestions. Other studies have used computer vision technology to perform single-action identification of student classroom behavior, but have failed to achieve continuous monitoring and dynamic correlation with IEP goals.

[0008] However, existing technologies remain at the "tool-assisted" level, failing to integrate multimodal perception, dynamic assessment, and proactive intervention into a closed-loop system. In particular, how to achieve continuous multi-source data collection while protecting student privacy, how to automatically quantify the achievement of IEP goals from raw behavioral interaction data, and how to ensure that AI-generated teaching suggestions comply with educational regulations and ethical requirements remain unsolved technical challenges in this field. Summary of the Invention

[0009] The purpose of this invention is to overcome the shortcomings of the prior art and provide a dynamic individualized education program system and method based on multimodal AI to overcome the problems existing in the prior art.

[0010] The present invention adopts the following technical solution to achieve the above objectives. In the first aspect, the present invention provides a dynamic individualized education plan system based on multimodal AI, including a perception layer, a fusion layer, a reasoning layer and an interaction layer, which are connected in sequence to form an information processing closed loop of "perception → fusion → reasoning → intervention". The push results of the interaction layer and the feedback of manual review are returned to the perception layer or the fusion layer as new input data to achieve continuous closed loop iteration.

[0011] The perception layer is deployed at the edge and includes a behavior recognition camera and an interactive tablet. The behavior recognition camera is used to acquire students' depth and RGB images in real time and extract human skeletal key point sequences from them. After extraction, the original image data is anonymized, outputting only an anonymous skeletal pose coordinate sequence that cannot be restored to the original image. The interactive tablet is used to collect students' touch screen interaction logs and screen gaze coordinates.

[0012] The fusion layer receives the skeletal pose coordinate sequence, touch screen interaction log, and screen gaze coordinates output by the perception layer. After aligning them by time window, it extracts behavior embeddings and interaction embeddings through the behavior encoder and interaction encoder, respectively. Then, it generates a Behavior State Vector (BSV) through cross-modal attention fusion. Each dimension of the BSV establishes a one-to-one correspondence with the IEP evaluation metrics.

[0013] The inference layer comprises a domain-adaptive fine-tuned large language model and a Retrieval-Augmented Generation (RAG) module based on a structured knowledge base. The inference layer encodes the current BSV time series, student historical baselines, and existing IEP goals into prompt text, and retrieves knowledge entries related to the current BSV state from the knowledge base. These retrieved knowledge entries serve as evidence to constrain the large language model in generating IEP goal update suggestions and role-based intervention suggestions.

[0014] The interaction layer includes a hybrid trigger engine and a role-based push module. The hybrid trigger engine triggers immediate alerts based on BSV deviations or comprehensive reports based on preset cycles. The role-based push module categorizes the intervention suggestions generated by the inference layer and pushes them to the corresponding terminal devices.

[0015] Furthermore, the Behavioral State Vector (BSV) is a 192-dimensional vector, and the correspondence between its dimensions and the IEP evaluation metrics includes at least the following:

[0016] BSV dimensions 0 to 15 (16 dimensions in total) correspond to the social / shared attention domain, including at least the normalized value of response call-out delay time and the frequency of eye contact;

[0017] Dimensions 16 to 31 of the BSV (a total of 16 dimensions) correspond to the classroom participation / seated behavior domain, and include at least the percentage of time spent seated and the frequency of leaving the seat;

[0018] BSV's 32nd to 47th dimensions (a total of 16 dimensions) correspond to the task focus domain, including at least the percentage of time the gaze lingers on the screen and the frequency of touch screen operations;

[0019] BSV dimensions 48 to 63 (16 dimensions in total) correspond to the emotion / behavior regulation domain, including at least the frequency of stereotyped actions and the number of crying / screaming events;

[0020] BSV dimensions 64 to 79 (16 dimensions in total) correspond to the cognitive / academic domain and include at least task accuracy and response time;

[0021] BSV's 80th to 95th dimensions (16 dimensions in total) correspond to the communication / language domain, and include at least the number of times communication is initiated and the accuracy of instruction response;

[0022] BSV dimensions 96 to 127 (32 dimensions in total) correspond to the motor / fine motor domain, including at least hand movement smoothness and target grasping success rate;

[0023] Dimensions 128 to 191 of the BSV (64 dimensions in total) are reserved for extended dimensions, used for customized mapping based on students' individualized IEP goals.

[0024] Furthermore, the anonymization process is implemented at any of the following levels: at the camera firmware level, the original frames are discarded immediately after skeletal keypoint extraction is completed inside the depth camera; at the edge computing box level, the original video data is deleted after skeletal keypoint extraction is completed on the edge computing device; at the local server level, the original data is cleared after skeletal keypoint extraction is completed in an encrypted state. The anonymized skeletal pose coordinate sequence cannot be restored to the original image or facial features, and all data is synchronized to the school's local server via the NTP protocol, without leaving the campus network.

[0025] Furthermore, the behavior encoder employs a Spatial Temporal Graph Convolutional Network (ST-GCN) pre-trained on the NTU-RGB+D dataset and fine-tuned on a self-built behavior dataset to extract a 128-dimensional behavior embedding from the skeletal sequence. The interaction encoder uses a TemporalTransformer encoder to map the normalized touchscreen displacement sequence and eye-tracking gaze sequence into a 64-dimensional interaction embedding. The cross-modal attention fusion specifically involves using the behavior embedding as a query Q, the interaction embedding as a key K and a value V, calculating the attention output, and then fusing it with the behavior embedding and the interaction embedding to obtain a 192-dimensional BSV.

[0026] Furthermore, the large language model in the inference layer is Qwen-14B-Chat, which is fine-tuned for domain adaptation using the Low-Rank Adaptation (LoRA) method. The training dataset used for fine-tuning consists of time slice pairs extracted from historical IEP records. Each time slice pair contains the student state at the first time step, the change in the student state at the second time step, and the corresponding expert IEP adjustment strategy.

[0027] Furthermore, the construction method of the structured knowledge base includes: segmenting the special education intervention method manual, curriculum standard documents, and ethical norms documents into knowledge fragments according to natural paragraphs or semantic units; having special education experts label each knowledge fragment with triggering conditions and corresponding suggested actions, forming knowledge entries in the form of structured four-tuples, where each four-tuple includes a condition field, an action field, a source field, and a confidence field; and using a text embedding model to vectorize each knowledge entry and store it in a vector database. The retrieval enhancement generation module encodes the current BSV state summary and deviation flag into query vectors during inference, retrieves the k most relevant knowledge entries from the vector database, and constrains the generated content of the large language model with the instruction "based solely on retrieval evidence."

[0028] Furthermore, the knowledge base also includes an "ethics compliance" tag field, used to mark knowledge entries involving restrictive intervention measures; when generating recommendations involving restrictive intervention measures, the large language model is instructed to include complete compliance instructions and applicable conditions.

[0029] Furthermore, the hybrid triggering engine includes: a threshold triggering submodule, which generates an immediate warning when the moving average of a certain BSV dimension falls below a preset baseline threshold; and a periodic triggering submodule, which automatically generates a comprehensive progress report according to a preset period. The calculation window for the moving average and the preset baseline threshold are configured by the IEP team based on individual student circumstances.

[0030] Furthermore, the role-based push module generates the following three types of intervention cards: teacher cards, which contain strategy suggestions and corresponding educational basis, and are pushed to the teacher's terminal; student cards, which contain gamified practice tasks, and are pushed to the student's tablet terminal; and family cards, which contain suggestions for generalized home activities, and are pushed to the parent's terminal.

[0031] Furthermore, the system also includes a manual review module: all intervention suggestions pushed by the interaction layer are in an unbound state before being reviewed and confirmed by the IEP team, and the system records the review status; only after receiving a confirmation instruction from the IEP team terminal will the system update the corresponding suggestion content to the statutory IEP document; the review and confirmation operation is completed through the web management terminal or mobile application, and the review log is persistently stored.

[0032] Furthermore, the fusion layer also includes an incremental update module, which is used to periodically recalculate the BSV baseline distribution of students as the system continues to collect new data, thereby enabling dynamic tracking of student status.

[0033] Secondly, this invention provides a method for generating dynamic individualized education plans based on multimodal AI, implemented using the aforementioned system, and comprising the following steps:

[0034] Step S1: At the edge, a behavior recognition camera is used to collect student depth images and RGB images in real time. After extracting key points of the human skeleton, the original images are desensitized and an anonymous skeleton pose coordinate sequence is output. At the same time, touch screen interaction logs and screen gaze coordinates are collected through an interactive tablet.

[0035] Step S2: Align the skeletal pose coordinate sequence, touch screen interaction log, and screen gaze coordinates according to the time window, extract the behavior embedding and interaction embedding respectively, and generate the behavior state vector (BSV) through cross-modal attention fusion; each dimension of the BSV corresponds one-to-one with the IEP evaluation index.

[0036] Step S3: Encode the current BSV time series, student historical baseline, and existing IEP goals into prompt text, retrieve knowledge entries related to the current BSV state from the structured knowledge base, constrain the large language model with the retrieval results, and generate IEP goal update suggestions and role-based intervention suggestions in JSON format.

[0037] Step S4: Based on the BSV deviation or preset cycle, trigger the intervention push, categorize the intervention suggestions by role and push them to the corresponding terminal devices, and wait for manual review and confirmation by the IEP team.

[0038] Step S5: The manual review results and the new round of sensor data collected by the system are used as feedback inputs. Steps S1 to S4 are executed cyclically to achieve dynamic closed-loop iteration.

[0039] Compared with the prior art, the present invention has the following beneficial technical effects:

[0040] The frequency of assessments has been significantly increased. Traditional IEP assessments rely on teachers' quarterly or annual scale scoring, resulting in long assessment cycles. This invention collects multimodal data in real time at the edge, and the fusion layer quickly generates BSVs through ST-GCN and Transformer, enabling assessment data to be obtained after each lesson. This represents a leap from "static quarterly assessments" to "dynamic real-time tracking," significantly reducing intervention delays.

[0041] The privacy protection effect is significant. This invention performs desensitization processing on the original video immediately after skeletal extraction at the edge, ensuring the video stream never leaves the campus network and the original image cannot be restored. Experimental verification shows that this solution achieves continuous behavior monitoring without original video output, meeting the requirements of multiple regulations such as FERPA, GDPR, and China's Personal Information Protection Law, and avoiding the privacy leakage risks associated with the transmission and storage of original video in traditional cloud-based video analytics solutions.

[0042] The professionalism and compliance of AI-generated content are significantly improved. This invention employs a dual constraint mechanism of "domain-adaptive fine-tuning + structured RAG" to reduce the rate of unfounded suggestions generated by general-purpose large language models from approximately 32% to approximately 4%. By constructing a structured knowledge base containing condition-action quadruples and an ethical compliance filtering mechanism, it ensures that the IEP adjustment suggestions and intervention strategies generated by LLM are professional, traceable, and ethically compliant, thus solving the technical problem of "illusions" generated by general-purpose large models in educational scenarios.

[0043] The behavior recognition accuracy is high. In the proof-of-concept experiment, the weighted F1 score of skeletal behavior recognition reached 89.3%, and experts rated it as 4.2-4.4 / 5 (out of 5) in three dimensions: accuracy, feasibility, and compliance, confirming the feasibility of the technical solution. Attached Figure Description

[0044] Figure 1 This is a schematic diagram of the overall architecture of a dynamic individualized education plan system based on multimodal AI provided in an embodiment of the present invention. Detailed Implementation

[0045] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings.

[0046] Example 1: System Overall Architecture

[0047] like Figure 1As shown, the present invention provides a dynamic individualized education plan system based on multimodal AI, comprising a perception layer, a fusion layer, a reasoning layer, and an interaction layer.

[0048] The perception layer is deployed at the edge of the classroom and includes behavior recognition cameras and interactive whiteboards. The behavior recognition cameras, using Intel RealSense D455 depth cameras, are mounted at the front or side of the classroom to capture depth and RGB images of students. Interactive whiteboards are placed on students' desks to present teaching tasks and collect interactive data.

[0049] The fusion layer receives multimodal data output from the perception layer, and generates BSV after preprocessing, feature extraction, and cross-modal fusion. The fusion layer can be deployed on a local server at the school.

[0050] The inference layer receives the BSV time series output from the fusion layer, combines it with a structured knowledge base, and generates intervention suggestions through a large language model. The inference layer can be deployed on a local server or a private cloud within the university.

[0051] The interaction layer receives the output from the inference layer, determines whether intervention is needed through a hybrid trigger engine, and then pushes intervention suggestions to the teacher's terminal, student's tablet terminal, and parent's terminal via a role-based push module. All push suggestions are only effective after being confirmed by a manual review module.

[0052] The perception layer, fusion layer, reasoning layer, and interaction layer are sequentially connected to form a closed information processing loop. The push results from the interaction layer and the feedback from manual review are returned to the perception layer or fusion layer as new input data, achieving continuous closed-loop iteration.

[0053] Example 2: Specific Implementation of the Perception Layer

[0054] The perception layer includes behavior recognition cameras and interactive panels.

[0055] The behavior recognition camera uses an Intel RealSense D455 depth camera, which simultaneously captures RGB and depth images at a frame rate of 30fps. The MediaPipe pose estimation framework runs on an edge computing device (such as an NVIDIA Jetson Xavier NX) to extract the 3D coordinates of 25 body keypoints (including head, torso, limbs, etc.) and 21 hand keypoints from the depth and RGB images in real time. The skeletal data for each frame is recorded in JSON format.

[0056] Privacy-desensitization processing is implemented as follows: On the edge computing device, after MediaPipe completes the extraction of skeletal key points, the original RGB image and depth image are immediately destroyed, and only the skeletal pose coordinate sequence is output to the fusion layer. This skeletal pose coordinate sequence only contains the three-dimensional spatial coordinate information of the joints and does not contain any data that can be restored to the original image or facial features. Desensitization processing can also be implemented at the camera firmware level (the original frame is discarded after skeletal extraction is completed on the RealSense D455's onboard processor) or at the local server level (the original data is cleared after extraction is completed in encrypted mode), but this embodiment preferably implements it at the edge computing device level to minimize the transmission range of the original data.

[0057] All devices are synchronized to the school's local server via the NTP protocol, and all data remains on the campus network.

[0058] The interactive tablet uses a commercially available tablet device with touch functionality (such as an iPad or Android tablet), with a screen size of at least 10 inches. A web-based teaching application runs on the tablet, using the lightweight eye-tracking library WebGazer.js to estimate students' screen gaze coordinates using the tablet's front-facing camera (sampling rate approximately 10-30fps), while simultaneously recording the coordinates, pressure values, and precise timestamps (accurate to the millisecond level) for each touch event. Touchscreen interaction logs and gaze coordinate data are transmitted to a local server via the campus network and stored synchronously with the skeletal data.

[0059] Example 3: Detailed Implementation of the Fusion Layer

[0060] The fusion layer fuses the multimodal data from the perception layer into BSV in the following manner.

[0061] Data preprocessing: The original skeletal coordinate sequence (30fps) and touchscreen trajectory / gaze coordinates (with inconsistent sampling rates) were first segmented into 5-second time windows. For the skeletal sequence, each 5-second window contained 150 frames, with each frame containing the 3D coordinates of 25 body keypoints and 21 hand keypoints, for a total of 46 keypoints × 3 coordinates = 138-dimensional data. For the touchscreen and eye-tracking data, interpolation and alignment to 30fps were performed within the same 5-second window to obtain the touchscreen displacement sequence and gaze coordinate sequence.

[0062] Behavior Encoding: The behavior encoder employs ST-GCN (Spatiotemporal Graph Convolutional Network). The input to ST-GCN is a sequence of skeletal keypoints, and the output is a 128-dimensional behavior embedding. ST-GCN is first pre-trained on the NTU-RGB+D dataset (containing over 56,000 action samples across 60 action categories), enabling the network to learn to recognize common human action patterns from skeletal coordinates. It is then fine-tuned on a self-built behavior dataset containing 5 classes of IEP-related behaviors (co-attention response, sitting, raising hands, clapping, and task participation), with 20 samples per class, totaling 100 samples. These samples are split into training and test sets in an 8:2 ratio, using 5-fold cross-validation. The fine-tuned ST-GCN can map the input skeletal sequence into a 128-dimensional embedding vector, which encodes the student's behavioral pattern information within a 5-second window.

[0063] Interaction Encoding: The interaction encoder employs a two-layer Temporal Transformer encoder. The input consists of a normalized touchscreen displacement sequence and an eye-tracking gaze sequence. The Transformer encoder's self-attention mechanism captures the temporal dependencies of the interaction behavior, and the output is a 64-dimensional interaction embedding. This vector encodes the student's task operation and attention allocation patterns within the 5-second window.

[0064] Cross-modal attention fusion: The 128-dimensional behavior embedding and the 64-dimensional interaction embedding are concatenated and then input into the cross-modal attention fusion module. Specifically, let the behavior embedding be... Interactive embedding as .Will Obtained by linear projection mapping to 128 dimensions ,make , Calculate attention using the following formula:

[0065] ;

[0066] Where d=128 is the scaling factor. The attention output is a 128-dimensional vector, which is then compared with the original behavior embedding. After performing a residual join, it is then embedded with the original interaction. The data is concatenated and passed through a fully connected layer (192-dimensional input, 192-dimensional output) to obtain the final 192-dimensional BSV. This asymmetric attention design, which uses behavior embeddings as queries and interaction embeddings as keys and values, allows behavior modalities to guide the attention allocation of interaction modalities, making full use of the spatial topological relationships of skeletal data and the temporal dependencies of interaction data.

[0067] BSV Normalization and Mapping: The values ​​of each BSV dimension were normalized to a range of [0,1], where 1 indicates that the indicator has reached the ideal level. The normalization baseline was derived from the distribution of the student's historical data or from norm data of the same age group / disability type. The correspondence between each BSV dimension and the IEP assessment indicators is shown in Table 1.

[0068] Table 1. Definition of Behavior-State Vector (BSV) Structure

[0069]

[0070] Supplementary explanation regarding the calculation of BSV

[49] (number of crying / screaming events): In embodiments with an audio acquisition module deployed (optional), the built-in microphone of the interactive tablet acquires ambient audio. After the audio signal is segmented by speech activity detection (VAD), it is input into a pre-trained audio event classification model (such as a YAMNet-based or self-built crying / screaming binary classifier), and outputs audio event labels and timestamps. These are then combined with the results of the periodic hand movement pattern detection to determine the crying / screaming events. In embodiments without audio acquisition, BSV

[49] can be estimated based solely on the periodic movement patterns of key hand points (such as high-frequency large-amplitude swings), or set as a missing value.

[0071] Custom mapping of BSV reserved extended dimensions: The system provides a BSV configuration management module (deployed on a local server). Special education experts can bind any dimension from BSV dimensions 128 to 191 to a custom IEP indicator through a visual interface on the web management terminal, and set the corresponding normalized baseline and calculation rules (e.g., obtained from a certain sensor data channel through specified calculations). The binding relationship is stored on the local server in the form of a JSON configuration file. The fusion layer dynamically reads this configuration file during inference and maps the corresponding raw data channel to the specified BSV dimension according to the configuration. If a reserved dimension is not bound, the dimension is set to 0 or a null value.

[0072] Incremental Update: The fusion layer also includes an incremental update module. As the system continuously collects new data (e.g., after each teaching day), the incremental update module reads all BSV records for that day, recalculates the mean and standard deviation of each dimension, and updates the student's BSV baseline distribution. Incremental updates use a sliding window approach, employing only data from the most recent 30 teaching days to reflect the student's recent ability status and prevent the baseline from becoming outdated.

[0073] Example 4: Specific Implementation of the Inference Layer

[0074] The reasoning layer includes a large language model and a retrieval enhancement generation module based on a structured knowledge base.

[0075] Base Model and Fine-tuning: For example, the large language model can adopt the Qwen-14B-Chat model open-sourced by Alibaba. This model has 14 billion parameters and supports offline deployment and inference on local servers, eliminating the need to upload student data to cloud APIs, thus meeting privacy compliance requirements in campus scenarios. Sim-IEP Dataset Construction for Domain-Adaptive Fine-tuning: Over 1,200 anonymized historical IEP records were obtained from a special education information platform. Special education experts extracted 200 high-quality time-slice pairs from these records. Each time-slice pair includes a description of the student's state at time step 1 (T1), the corresponding BSV key dimension values, a description of the state change at time step 2 (T2), and expert-made IEP goal adjustments and teaching strategy suggestions. 150 pairs were used for training, and 50 pairs were used for testing. This dataset was used to fine-tune the Qwen-14B-Chat model using LoRA (rank r=8), resulting in a domain-adaptive model. Alternatively, other open-source models with the same localization deployment capabilities, such as Baichuan2-13B-Chat or ChatGLM3-6B, can be used.

[0076] Structured Knowledge Base Construction: Collect school-approved intervention method manuals, curriculum standard documents, and ethical guidelines documents, and construct a structured knowledge base according to the following rules:

[0077] (1) Document segmentation: Each document is segmented into knowledge chunks according to natural paragraphs or semantic units, each chunk being approximately 200-500 words.

[0078] (2) Condition-Action Rule Extraction: Special education experts annotate each segment to clarify its applicable triggering conditions (e.g., "the common attention BSV value is 20% lower than the baseline for 3 consecutive days") and corresponding suggested actions (e.g., "intervention using gesture prompts, 5 minutes each time"), forming knowledge entries in the form of structured quadruples, which include condition field, action field, source field and confidence field.

[0079] (3) Vectorized storage: Each knowledge item is vectorized using a text embedding model (such as text-embedding-ada-002 or bge-large-zh) and stored in the FAISS vector database. The vector dimension is aligned with the LLM embedding dimension.

[0080] (4) Ethical compliance labeling: Add an "ethical compliance" label field to the knowledge entry to mark whether the recommendation involves restrictive interventions (such as physical assistance), and label the applicable conditions and precautions for knowledge entries involving restrictive interventions.

[0081] Reasoning process: The workflow of the reasoning layer is as follows:

[0082] (1) The current BSV time series (containing BSV vectors of the past 5 time windows) output by the fusion layer is input into the inference layer.

[0083] (2) The system encodes the current BSV time series, student historical baseline, and existing IEP target into prompt text. An example of the prompt template is as follows:

[0084] You are a special education expert. Below are the key BSV dimensions of the student [S] over the past 5 time periods (5 minutes each):

[0085] - BSV[0] (Response Name Delay): [0.72, 0.68, 0.55, 0.50, 0.48], 3-day moving average 0.52, 25.7% below baseline (0.70).

[0086] - BSV

[16] (Seated time percentage): [0.80, 0.75, 0.70, 0.65, 0.60], 3-day moving average 0.65, 13.3% lower than baseline (0.75).

[0087] - BSV

[32] (Percentage of eyes remaining): [0.60, 0.55, 0.50, 0.45, 0.40], 3-day moving average 0.45, 30.8% lower than baseline (0.65).

[0088] Current IEP goals: Goal 1 (improving joint attention ability), Goal 2 (extending seated time)

[0089] Based on the retrieved expertise, please output IEP target update recommendations and role-based intervention cards in JSON format.

[0090] (3) Encode the current BSV status summary and deviation flag into a query vector, and retrieve the k most relevant knowledge entries (k=5) in the FAISS vector database.

[0091] (4) The prompt is "based solely on retrieved evidence" and the retrieved knowledge items are input into the large language model as evidence.

[0092] (5) The large language model generates JSON output, including IEP target update suggestions, role-based intervention card content and corresponding evidence sources.

[0093] (6) If the generated content involves restrictive intervention measures, the system will instruct the large language model to include a complete compliance description and applicable conditions in the output based on the “ethical compliance” tag in the knowledge entry.

[0094] Example 5: Detailed Implementation of the Interaction Layer

[0095] The interaction layer includes a hybrid trigger engine, a role-based push module, and a manual review module.

[0096] The hybrid triggering engine includes a threshold triggering submodule and a periodic triggering submodule:

[0097] (1) Threshold Trigger Submodule: Continuously monitors the 3-day moving average of each dimension of BSV. When the 3-day moving average of a certain dimension is lower than 20% of the student's baseline value, an alert is immediately generated and the inference layer is triggered to generate intervention suggestions. The trigger threshold (20%) and the moving average window (3 days) can both be configured by the IEP team through the Web management terminal.

[0098] (2) Periodic Triggering Submodule: Every Friday at 4:00 PM, the inference layer automatically triggers the generation of a comprehensive progress report for the week, including trend charts of changes in each BSV dimension, a list of IEP targets that have been met, a list of targets that need attention, and intervention recommendations for the following week.

[0099] Role-based push module: Upon triggering, the intervention suggestions generated by the inference layer are automatically formatted into three types of cards:

[0100] (1) Teacher Card: It contains strategy suggestions and corresponding educational basis (source fields referenced from the knowledge base), which are pushed to the teacher terminal (teacher mobile app or web terminal) through a secure API. Teachers can view and implement them in the classroom in real time.

[0101] (2) Student Card: Contains automatically scheduled gamified practice tasks, which are pushed to the student's tablet terminal via a secure API. Students can complete these tasks during or after class.

[0102] (3) Family Cards: Include suggestions for generalized home activities (such as exercises that parents can complete with their children at home), which are pushed to the parent's terminal (parent's mobile app or WeChat mini program) via a secure API.

[0103] All card content is transmitted in structured JSON format, and the terminal device renders and displays it according to the character template.

[0104] Manual review module: All intervention suggestions pushed out are in an "unbound" state before being reviewed by the IEP team (including teachers, parents, and special education experts). The system records the review status of each suggestion (pending review, adopted, rejected, modified). IEP team members receive push notifications through the web management terminal or mobile application, view the suggestion content, and click the "adopt," "modify," or "reject" button. After adoption, the system automatically calls the IEP document management interface to update the corresponding suggestion content to the relevant fields in the legal IEP document; after modification, it needs to be resubmitted for review; if rejected, the reason for rejection is recorded. All review operations generate persistent review logs, stored on the local server, to meet compliance audit requirements. The corresponding suggestion content only officially takes effect after receiving a confirmation instruction from the IEP team terminal.

[0105] Example 6: Alternative Implementation

[0106] The following alternatives are all within the scope of protection of this invention.

[0107] Alternatives for perception layer devices: Behavior recognition cameras are not limited to the Intel RealSense D455; other RGB-D sensors such as Azure Kinect and Orbbec Astra can also be used. Pose estimation frameworks are not limited to MediaPipe; OpenPose, BlazePose, or custom convolutional neural network pose estimation networks can also be used.

[0108] Alternatives to behavior encoders: Behavior encoders are not limited to ST-GCN; improved variants (such as ST-GCN++, AGCN) or Transformer-based skeletal behavior recognition models can also be used. Alternatively, 3D convolutional neural networks (3D-CNNs) can be directly used to extract spatiotemporal features from video clips, replacing skeletal keypoint-based approaches.

[0109] Alternatives to interactive encoders: Temporal modeling of interactive encoders is not limited to Temporal Transformers; Long Short-Term Memory (LSTM) networks or Gated Recurrent Units (GRUs) can also be used. The system can also incorporate audio modalities as a third modal input (such as speech emotion recognition and ambient sound analysis), thereby expanding the BSV dimension accordingly.

[0110] Alternatives to the base model: Large language models are not limited to Qwen-14B-Chat. Other open-source large language models such as the LLaMA series, ChatGLM, and Baichuan can also be used, or commercial API models can be used if privacy compliance allows (provided that data is not transmitted outside the campus network).

[0111] Alternatives to fine-tuning methods: LoRA fine-tuning can be replaced by full-parameter fine-tuning to achieve stronger domain adaptability, or other efficient parameter fine-tuning methods such as Adapter and Prefix-Tuning can be used.

[0112] Alternative retrieval schemes: Vector databases like FAISS can be replaced by Milvus, Pinecone, Weaviate, etc. Alternatively, a hybrid retrieval scheme combining sparse retrieval (such as BM25) and dense retrieval can be used, or structured queries based on graph databases can replace vector similarity retrieval.

[0113] Extended triggering mechanism: The hybrid triggering engine can also include a trend prediction triggering submodule (predicting future states based on BSV time series and triggering early warnings), an anomaly detection triggering submodule (using unsupervised anomaly detection algorithms to identify sudden deviations in BSV), and a student-initiated triggering submodule (allowing students to actively request help via tablet terminals).

[0114] Configurable role templates: The three types of role cards can be designed as configurable role templates, dynamically generating more customized content for roles (such as teaching assistant cards and rehabilitation therapist cards) according to the needs of different schools or institutions.

[0115] Tiered review mechanism: The manual review module can introduce a tiered review mechanism, allowing low-risk suggestions (such as minor adjustments to teaching strategies) to take effect automatically and be subject to random checks afterward, while high-risk suggestions (such as those involving restrictive intervention measures) require multi-person countersignature confirmation to balance efficiency and security.

[0116] The above description is merely a preferred embodiment of the present invention. It should be understood that the present invention is not limited to the forms disclosed herein and should not be construed as excluding other embodiments. It can be used in various other combinations, modifications, and environments, and can be altered within the scope of the concept described herein through the above teachings or related technologies or knowledge. Modifications and variations made by those skilled in the art that do not depart from the spirit and scope of the present invention should be within the protection scope of the appended claims.

Claims

1. A dynamic individualized education plan system based on multimodal AI, characterized in that, include: The perception layer, deployed at the edge, includes a behavior recognition camera and an interactive tablet. The behavior recognition camera is used to acquire students' depth images and RGB images in real time, and extract human skeleton key point sequences from them. The original image data is desensitized after extraction, and only anonymized skeleton pose coordinate sequences that cannot be restored to the original images are output. The interactive tablet is used to collect students' touch screen interaction logs and screen gaze coordinates. The fusion layer is used to receive the skeleton pose coordinate sequence, touch screen interaction log and screen gaze coordinate output by the perception layer. After being aligned by time window, the behavior embedding and interaction embedding are extracted by the behavior encoder and interaction encoder respectively, and then the behavior state vector (BSV) is generated by cross-modal attention fusion. The BSV dimensions are established in a one-to-one correspondence with the individualized education program (IEP) evaluation indicators; The inference layer includes a domain-adaptive fine-tuned large language model and a retrieval enhancement generation module (RAG) based on a structured knowledge base. The inference layer encodes the current BSV time series, student historical baseline, and existing IEP targets into prompt text, and retrieves knowledge entries related to the current BSV state from the knowledge base. The retrieved knowledge entries are used as evidence to constrain the large language model to generate IEP target update suggestions and role-based intervention suggestions. The interaction layer includes a hybrid trigger engine and a role-based push module; The hybrid triggering engine triggers an immediate warning based on BSV deviation or a comprehensive report based on a preset cycle. The role-based push module categorizes the intervention suggestions generated by the inference layer by role and pushes them to the corresponding terminal devices. The perception layer, fusion layer, reasoning layer, and interaction layer are sequentially connected to form an information processing closed loop of perception, fusion, reasoning, and intervention. The push results and manual review feedback from the interaction layer are returned to the perception layer or fusion layer as new input data to achieve continuous closed-loop iteration.

2. The dynamic individualized education plan system based on multimodal AI according to claim 1, characterized in that, The Behavioral State Vector (BSV) is a 192-dimensional vector, and the correspondence between its dimensions and IEP evaluation metrics includes at least the following: BSV[0:15] corresponds to the social / shared attention domain, which includes at least the normalized value of the response call-out delay time and the frequency of eye contact; BSV[16:31] corresponds to the classroom participation / seated behavior domain, which includes at least the percentage of time spent seated and the frequency of leaving the seat; BSV[32:47] corresponds to the task focus domain, which includes at least the percentage of time the gaze lingers on the screen and the frequency of touch operations; BSV[48:63] corresponds to the emotion / behavior regulation domain, which includes at least the frequency of stereotyped actions and the number of crying / screaming events; BSV[64:79] corresponds to the cognitive / academic domain and includes at least task accuracy and response time; BSV[80:95] corresponds to the communication / language domain, and includes at least the number of times communication is initiated and the accuracy of command response; BSV[96:127] corresponds to the field of motor / fine motor skills, including at least hand movement smoothness and target grasping success rate; BSV[128:191] is a reserved extended dimension for customized mapping based on the student's individualized IEP goals.

3. The dynamic individualized education plan system based on multimodal AI according to claim 1, characterized in that, The desensitization process is implemented at any of the following levels: At the camera firmware level, the original frames are discarded immediately after the skeletal key points are extracted inside the depth camera. At the edge computing box level, the original video data is deleted after the skeletal key points are extracted on the edge computing device. At the local server level, after extracting the key skeletal points in an encrypted state, the original data is cleared. The anonymous skeletal pose coordinate sequence cannot be restored to the original image or facial features, and all data is synchronized to the school's local server via the NTP protocol without leaving the campus network.

4. The dynamic individualized education plan system based on multimodal AI according to claim 1, characterized in that, The behavior encoder uses a spatiotemporal graph convolutional network ST-GCN pre-trained on the NTU-RGB+D dataset and fine-tunes it on a self-built behavior dataset to extract 128-dimensional behavior embeddings from the skeletal sequence. The interactive encoder uses a Temporal Transformer encoder to map the normalized touch screen displacement sequence and eye movement gaze point sequence into a 64-dimensional interactive embedding. The cross-modal attention fusion specifically involves: using the behavior embedding as a query Q, the interaction embedding as a key K and a value V, calculating the attention output, and then fusing it with the behavior embedding and the interaction embedding to obtain a 192-dimensional BSV.

5. The dynamic individualized education plan system based on multimodal AI according to claim 1, characterized in that, The structured knowledge base is constructed in the following ways: The special education intervention methods manual, curriculum standard documents, and ethical norms documents are divided into knowledge segments according to natural paragraphs or semantic units; Special education experts label each knowledge segment with triggering conditions and corresponding suggested actions, forming knowledge entries in the form of structured four-tuples. The four-tuples include a condition field, an action field, a source field, and a confidence field. The knowledge items are vectorized using a text embedding model and then stored in a vector database. The retrieval enhancement generation module encodes the current BSV state summary and deviation flag into a query vector during inference, retrieves the k most relevant knowledge entries from the vector database, and constrains the generated content of the large language model with the instruction "based solely on retrieval evidence".

6. The dynamic individualized education plan system based on multimodal AI according to claim 1, characterized in that, The hybrid triggering engine includes: The threshold triggering submodule generates an immediate warning when the moving average of a certain BSV dimension falls below a preset baseline threshold. The periodic triggering submodule automatically generates a comprehensive progress report according to a preset cycle. The calculation window for the moving average and the preset baseline threshold are configured by the IEP team based on the individual student circumstances.

7. The dynamic individualized education plan system based on multimodal AI according to claim 1, characterized in that, The role-based push module generates the following three types of intervention cards: Teacher Cards, which include strategy suggestions and corresponding educational basis, are pushed to teachers' terminals. Student Tablet Card, which includes gamified practice tasks, is pushed to the student's tablet terminal; The Family Card includes suggestions for general home activities and is pushed to parents' devices.

8. The dynamic individualized education plan system based on multimodal AI according to claim 1, characterized in that, The individualized education plan system also includes a manual review module: All intervention suggestions pushed by the interaction layer are in an unbound state before being reviewed and confirmed by the IEP team, and the system records the review status. The system will only update the corresponding recommendations to the statutory IEP document after receiving confirmation from the IEP team terminal. The review and confirmation process is completed through a web management terminal or mobile application, and the review logs are persistently stored.

9. The dynamic individualized education plan system based on multimodal AI according to claim 1, characterized in that, The fusion layer also includes an incremental update module, which is used to periodically recalculate the BSV baseline distribution of students as the system continues to collect new data, so as to realize dynamic tracking of student status.

10. A method for generating dynamic individualized education plans based on multimodal AI, applied to the dynamic individualized education plan system based on multimodal AI as described in any one of claims 1-9, characterized in that, Includes the following steps: S1. At the edge, a behavior recognition camera is used to collect student depth images and RGB images in real time. After extracting key points of the human skeleton, the original images are desensitized and an anonymous skeleton pose coordinate sequence is output. At the same time, touch screen interaction logs and screen gaze coordinates are collected through an interactive tablet. S2. Align the skeleton pose coordinate sequence, touch screen interaction log, and screen gaze coordinates according to the time window, extract the behavior embedding and interaction embedding respectively, and generate the behavior state vector (BSV) through cross-modal attention fusion; each dimension of the BSV corresponds one-to-one with the IEP evaluation index; S3. Encode the current BSV time series, student historical baseline, and existing IEP goals into prompt text, retrieve knowledge entries related to the current BSV state from the structured knowledge base, constrain the large language model with the retrieval results, and generate IEP goal update suggestions and role-based intervention suggestions in JSON format; S4. Trigger intervention push based on BSV deviation or preset cycle, push the intervention suggestions to the corresponding terminal devices after classifying them by role, and wait for manual review and confirmation by the IEP team; S5. Take the results of manual review and the new round of sensor data collected by the system as feedback input, and repeat steps S1 to S4 to achieve dynamic closed-loop iteration.

Images