A method for evaluating intervention effect of sound-meaning matching and generating adaptive path

Abstract

The application discloses a method for evaluating intervention effect of sound and meaning matching and generating adaptive path. The method synchronously collects five-way data of visual behavior, tactile interaction, speech feedback, body movement and emotional physiology of a user in a teaching process through a sensor network, jointly trains a modal encoder through a cycle-global joint contrast loss, eliminates dominant channel bias and supports robust inference of a missing channel, constructs a perception- language-semantic interference joint knowledge graph, adds predecessor-successor forgetting propagation coefficients and two-stage semantic confusion interference weights on edges of the graph, introduces individualized forgetting rate and cascade forgetting propagation correction to calculate a comprehensive intervention effect index of each vocabulary node, compresses a global state through a graph attention network, combines a local active vocabulary state to drive a deep Q network to output an individualized teaching path sequence, and finally generates an auxiliary decision report output to a mobile terminal.

Description

Technical Field

[0001] This invention relates to the fields of educational data mining and artificial intelligence-assisted decision-making technology, specifically to a method for multimodal behavioral data fusion assessment and personalized teaching path dynamic planning for family unit intervention scenarios, applicable to early family intervention support systems for children with language development disorders. Background Technology

[0002] Early intervention for language development disorders in children (including spectrum disorders and language delay) is a critical window for improving children's cognitive and language abilities. The family unit intervention model is widely recognized for its high ecological validity and sufficient frequency of interventions. Its core training paradigm is the "sound-meaning-object" triadic matching—establishing stable cognitive anchors between sensory experiences and lexical semantics by repeatedly exposing children to multi-sensory stimuli (tactile texture, olfactory odor, visual color, etc.) of real objects. However, this model relies heavily on the professional judgment of immediate family therapists and lacks objective, quantitative assessment and pathway planning support.

[0003] In terms of multimodal data acquisition and fusion, existing technologies have attempted to introduce single sensors such as eye tracking and speech analysis into children's language training scenarios. However, these systems generally adopt a star-shaped fusion architecture centered on a single dominant modality (usually vision). When a child's lack of cooperation in a home intervention scenario leads to the loss of data from a certain sensor, the failure of the dominant channel will cause the overall assessment system to crash, making it impossible to provide stable and reliable degradation support for immediate family therapists.

[0004] Regarding vocabulary mastery assessment, existing intelligent assisted teaching systems typically only provide static time-point evaluations of children's performance at the end of each lesson, failing to consider the individualized decay patterns of vocabulary cognition over teaching intervals. Different children exhibit significant individual differences in their forgetting rates for different words, and the cognitive decay of preceding words in the knowledge graph has a cascading effect on the stability of subsequent word mastery through logical progression. Ignoring these temporal dynamics leads to a systematic deviation between assessment results and children's actual cognitive state during teaching intervals, resulting in a false sense of advanced learning progress. In terms of teaching path scheduling, existing solutions also fail to incorporate the risk of semantic confusion and interference between words into scheduling decisions. The mutual interference caused by the generalization and transfer of semantically similar words (such as names of similar fruits) during concurrent training cannot be identified and resolved, leading to suboptimal scheduling decisions.

[0005] In summary, the existing technology has two core technical problems: First, the dominant channel bias in the star-shaped fusion architecture of the multimodal assessment framework leads to unreliable degradation of the system when sensor data is missing, and the vocabulary mastery assessment lacks personalized temporal decay and causal forgetting propagation modeling, which cannot reflect the true cognitive state of children during the teaching interval; Second, the teaching path scheduling does not perceive the risk of semantic confusion and interference between words, and lacks an effective prior estimation strategy for interference in the early stage, resulting in suboptimal scheduling decisions. Summary of the Invention

[0006] To address the aforementioned problems in existing technologies, this invention provides a method for evaluating the effectiveness of sound-meaning-object matching intervention and for adaptive path generation. This method aims to solve the technical problems of insufficient robustness of multimodal evaluation systems when sensor channels are missing, lack of personalized temporal dynamic modeling in vocabulary mastery assessment, and failure to perceive semantic confusion and interference risks in teaching path scheduling.

[0007] To achieve the above objectives, the present invention adopts the following technical solution:

[0008] A method for evaluating the effectiveness of sound-semantic object matching intervention and generating adaptive paths includes: synchronously collecting data on multiple indicators of users during the teaching process through a sensor network to obtain five-modal embedding vectors; jointly training multiple encoders with a recurrent-global joint contrastive loss for the five-modal embedding vectors to obtain a recurrent-global contrastive alignment multimodal feature representation set; constructing a directed heterogeneous knowledge graph with teaching vocabulary as nodes and perceptual channel activation relationships and lexical logical relationships as edges, and attaching predecessor-successor forgetting propagation coefficients and semantic confusion interference weights to the graph edges to generate a perceptual-language-semantic interference joint knowledge graph; and calculating the perceptual-language-semantic interference for each vocabulary node based on the multimodal feature representation set and the joint knowledge graph. The perceptual cognitive integrity index, language output accuracy index, and generalization transfer ability index, after being corrected for temporal decay and cascaded forgetting propagation, are normalized and weighted to fuse the emotional physiological stability index, ultimately yielding a comprehensive intervention effect index. Based on the comprehensive intervention effect index and the joint knowledge graph, a global state embedding is aggregated through a graph attention network. This is combined with the evaluation vector of the active vocabulary set, the forgetting urgency index, and the semantic confusion interference submatrix between active vocabulary to construct a local state vector. The global state embedding and the local state vector are used as state inputs to a trained deep Q-network to output a personalized teaching path sequence. Based on the personalized teaching path sequence and the comprehensive intervention effect index of each vocabulary node, an auxiliary decision-making report is generated.

[0009] Furthermore, by synchronously collecting data on multiple indicators of the user during the teaching process through a sensor network, five modal embedding vectors are obtained, including: using distributed cameras and eye-tracking modules as acquisition units to collect the user's gaze heatmap, gaze duration sequence, and eye-switching frequency when facing the target object, obtaining visual behavior data; using a pressure sensor array deployed on the object's surface as acquisition units to collect the user's hand force trajectory and grip stability index when operating the object, obtaining tactile interaction data; and using a directional microphone array as acquisition units to collect the user's voice content text, feedback delay time, and audio output after responding to voice commands. The accuracy of the rhyme is scored to obtain voice feedback data; a wearable inertial sensor is used as the acquisition unit to collect the user's body displacement trajectory and balance control ability index during the teaching activity to obtain proprioceptive motion data; a wristband-type skin conductance sensor is used as the acquisition unit to collect the user's skin conductance response change sequence during the teaching process to obtain emotional physiological data; using the teaching timestamp as the alignment benchmark, the visual behavior data, the tactile interaction data, the voice feedback data, the proprioceptive motion data, and the emotional physiological data are synchronized in time, and then input into the corresponding modal-specific encoder to extract spatiotemporal features to obtain five modal embedding vectors.

[0010] Furthermore, for the five modal embedding vectors, multiple encoders are jointly trained using a cyclic-global joint contrastive loss to obtain a cyclic-global contrast-aligned multimodal feature representation set. This includes: for the five modal embedding vectors, arranging the five modal encoders in a circular topological order; calculating mutual information estimates for the modal embedding vectors output by adjacent encoders; summing the mutual information estimates of all adjacent encoder pairs to obtain a cyclic loss term; for the output embedding vector of each modal encoder, calculating the mean aggregation representation of the remaining four modal embedding vectors; calculating the mutual information estimate between the output embedding vector and the mean aggregation representation; summing the mutual information estimates of the five encoders to obtain a global loss term; weighting the cyclic loss term and the global loss term with a balancing coefficient to obtain a joint contrastive loss; and jointly training the five modal encoders using the joint contrastive loss as the objective function to obtain a cyclic-global contrast-aligned multimodal feature representation set.

[0011] Further, the precursor-successor forgetting propagation coefficient is added to the graph edges, including: for word node pairs with logically progressive relationships in the joint knowledge graph, extracting the first comprehensive intervention effect index time series of the precursor word node and the second comprehensive intervention effect index time series of the successor word node from the historical intervention data, calculating the correlation coefficient between the first comprehensive intervention effect index time series and the second comprehensive intervention effect index time series, and outputting the initial forgetting propagation coefficient; when the number of co-occurrence training samples of word node pairs in the historical intervention data is less than a preset threshold, using the monotonically increasing function of the teaching difficulty difference between the precursor word node and the successor word node as the prior initial value to replace the initial forgetting propagation coefficient; starting from the initial forgetting propagation coefficient or the prior initial value, the precursor-successor forgetting propagation coefficient is updated rollingly with the newly added historical intervention data after each teaching cycle, and the updated precursor-successor forgetting propagation coefficient is added to the corresponding logically progressive edge in the joint knowledge graph to generate a perception-language-semantic interference joint knowledge graph containing the forgetting propagation coefficient.

[0012] Further, semantic confusion interference weights are added to the edges of the graph, including: for any pair of word nodes in the joint knowledge graph, calculating the set intersection-union ratio similarity of the sets of perceptual channels associated with the pair of word nodes to obtain the perceptual channel overlap; extracting the semantic embedding vector of the arbitrary word node, calculating the normalized vector inner product similarity to obtain the semantic embedding similarity; performing a weighted calculation on the perceptual channel overlap and the semantic embedding similarity to obtain a prior estimate of the semantic confusion interference weight; counting the cumulative number of times users misjudged the pair of word nodes in historical teaching records, and when the cumulative number reaches a preset sample size threshold, entering the hot start stage, performing Bayesian posterior fusion with the historical misjudgment frequency and the prior estimate of the semantic confusion interference weight to obtain the semantic confusion interference weight; and attaching the semantic confusion interference weight to the corresponding pair of word nodes in the joint knowledge graph in the form of undirected interference edges to obtain the perceptual-language-semantic interference joint knowledge graph containing the semantic confusion interference weight.

[0013] Furthermore, the perceptual-cognitive integrity index is calculated and a cascaded forgetting propagation correction is introduced, including: for each word node, the immediate activation status of the user on each perceptual channel associated with each word node is statistically analyzed, the activation integrity rate is calculated, and the immediate perceptual-cognitive integrity is obtained; the historical perceptual-cognitive integrity time series of the word node is extracted, and the user's personalized forgetting rate parameter for that word is fitted by maximum likelihood estimation, and initialized with the average forgetting rate parameter of the user group when the historical training samples of the word are insufficient; using the personalized forgetting rate parameter and the time interval since the last teaching as input, the immediate perceptual-cognitive integrity index is adjusted. An exponential decay correction is applied to the perceptual cognitive completeness to obtain the perceptual cognitive completeness after temporal decay. For each predecessor word node of the word node in the joint knowledge graph, the difference between the historical stable reference value of the predecessor word node and the current perceptual cognitive completeness after temporal decay is calculated. When the difference is positive, the difference is weighted by the predecessor-successor forgetting propagation coefficient. The weighted differences of each predecessor word node are summed to obtain the cascade decay correction amount. The perceptual cognitive completeness after temporal decay is subtracted from the cascade decay correction amount and then mapped to the [0,1] interval by a normalization function to obtain the comprehensive intervention effect index.

[0014] Further, the generalization transfer ability index is calculated and semantic interference discount correction is introduced, including: for each word node, by changing the color, size or scene of the target object in the teaching scenario, collecting the number of correct recognitions and the total number of tests in the changed scene, calculating the correct recognition rate, and obtaining the instant generalization transfer ability; retrieving the set of word nodes in the joint knowledge graph that have undirected interference edges with each word node and whose semantic confusion interference weight exceeds a preset interference threshold, and determining whether the word nodes in the set are in the same active training state of the same teaching cycle as the current word node; when there are interference word nodes that are in the same active training state as the current word node, the semantic confusion interference weight is used as the discount coefficient to apply interference discount correction to the instant generalization transfer ability, the discount coefficient is directly assigned by the semantic confusion interference weight, and the corrected result is mapped to the [0,1] interval through a normalization function to obtain the comprehensive intervention effect index.

[0015] Furthermore, after obtaining the comprehensive intervention effect index, the method further includes: obtaining the missing channel label vector to determine the set of sensor channels with missing data in the current teaching evaluation cycle; for each missing channel in the set of sensor channels with missing data, extracting the output embedding vector of the corresponding modal encoder after the completion of cyclic-global joint contrastive training, using the output embedding vector as the cross-modal implicit representation of the missing channel to replace the feature contribution corresponding to the original collected data of the missing channel; using the channel features replaced by the cross-modal implicit representation as a basis to participate in the calculation of the perceptual cognitive integrity index, the language output accuracy index, and the generalization transfer ability index, and the calculation result of the comprehensive intervention effect index remains effective when the number of missing channels does not exceed the preset maximum number of missing channels.

[0016] Furthermore, a global state embedding is aggregated and a local state vector is constructed using a graph attention network. This includes: using the comprehensive intervention effect index, perceptual cognitive integrity index, language output accuracy index, and generalization transfer ability index of all vocabulary nodes as node features, and using the topology of the joint knowledge graph as the graph structure, inputting the graph attention network, and aggregating the neighbor node features layer by layer through a multi-head attention mechanism to obtain a fixed-dimensional global state embedding, the dimension of which is independent of the total number of vocabulary nodes; determining the active vocabulary node set for the current teaching cycle, extracting the comprehensive intervention effect index of each active vocabulary node, constructing a forgetting urgency index by multiplying the personalized forgetting rate parameter by the time interval since the last teaching, and forming a semantic confusion interference sub-matrix between active vocabulary nodes from the semantic confusion interference weights between each pair of nodes in the active vocabulary node set; concatenating the comprehensive intervention effect index, the forgetting urgency index, and the semantic confusion interference sub-matrix between active vocabulary nodes to obtain a local state vector; concatenating the global state embedding with the local state vector, inputting it into the trained deep Q-network, and outputting a personalized teaching path sequence.

[0017] Preferably, the method further includes: after each teaching action is performed, calculating the change in the comprehensive intervention effect index of each vocabulary node, and using the weighted sum of the positive increments in the change as the incremental reward of the comprehensive intervention effect index; detecting the decrease in the semantic confusion interference weight between vocabulary node pairs associated with the interference discrimination training action after the action is performed in the current teaching cycle, and using the monotonically increasing function of the decrease as the semantic interference resolution reward when the decrease exceeds a preset change threshold; for vocabulary nodes whose forgetting urgency index exceeds a preset urgency threshold, using the sum of the differences exceeding the preset urgency threshold as the forgetting cost penalty; weighting the incremental reward of the comprehensive intervention effect index with a first balance coefficient, weighting the semantic interference resolution reward with a second balance coefficient, and weighting the forgetting cost penalty with a third balance coefficient, summing the three weighted results to obtain the reward function value of the current teaching action, and using the reward function value to update the network parameters of the deep Q network using the training signal.

[0018] Compared with the prior art, the present invention has the following technical effects:

[0019] To address the first technical issue, the cyclic-global joint contrastive loss uses the cyclic loss term to constrain adjacent modal alignment and the global loss term to constrain the alignment of each encoder with the global multimodal context, fundamentally eliminating the dominant channel bias problem in the star-shaped fusion architecture. When sensor channels experience data loss due to children's lack of cooperation, the corresponding encoder can perform robust degradation inference based on the learned global cross-modal implicit representation, ensuring a smooth decline in the overall system evaluation performance rather than a collapse. The introduction of personalized forgetting rate parameters and predecessor-successor forgetting propagation coefficients transforms vocabulary mastery assessment from a static time point to a dynamic temporal tracking, accurately reflecting children's true cognitive decline during teaching intervals and avoiding falsely advanced teaching progress.

[0020] To address the second technical issue, the two-stage semantic confusion interference weight estimation strategy initially uses perceptual channel overlap weighted semantic embedding similarity to determine prior estimates, and then uses Bayesian posterior fusion to achieve a smooth transition after sufficient data is available, ensuring the reliability of the estimation throughout the entire lifecycle. The semantic interference perception scheduling and interference discrimination training actions enable the system to actively identify and resolve semantic confusion risks, significantly reducing the failure of generalization transfer caused by vocabulary confusion in children. The design of the graph attention network to compress global state embedding decouples the state dimension from the total number of words, effectively solving the problem of state dimension explosion in large vocabulary scenarios. Attached Figure Description

[0021] Figure 1 A schematic diagram of the overall process of the sound-meaning-object matching intervention effect evaluation and adaptive path generation method provided in an embodiment of the present invention;

[0022] Figure 2This is a schematic diagram of the structure of synchronous acquisition of multimodal behavioral data and cyclic-global joint comparison coding provided in an embodiment of the present invention;

[0023] Figure 3 A schematic diagram illustrating the construction of a joint knowledge graph of perception-language-semantic interference provided in an embodiment of the present invention;

[0024] Figure 4 A schematic diagram of the calculation process for personalized temporal decay and cascaded forgetting propagation correction provided in an embodiment of the present invention;

[0025] Figure 5 A schematic diagram of the structure for constructing compressed state representations and planning deep Q-network paths provided in an embodiment of the present invention;

[0026] Figure 6 A schematic diagram illustrating the composition of the extended reward function provided in an embodiment of the present invention. Detailed Implementation

[0027] Example 1

[0028] like Figure 1 As shown, a method for evaluating the effectiveness of sound-meaning-object matching intervention and for adaptive path generation includes:

[0029] S1: Data on multiple indicators of the user during the teaching process are collected synchronously through a sensor network to obtain five modal embedding vectors.

[0030] In this embodiment of the invention, "user" refers to a child receiving intervention from a family unit. The teaching scenario is a home environment, and the sensor network is deployed using a combination of contactless devices and lightweight wearable devices, eliminating the need for children to actively cooperate in wearing complex devices. The collected indicators cover five perceptual dimensions: visual behavior, tactile interaction, voice feedback, proprioception, and emotional physiology. Data from each dimension is processed by its corresponding modal encoder, outputting five modal embedding vectors, each representing a child's behavioral characteristics across the five perceptual dimensions during the current teaching session.

[0031] S2: For the five modal embedding vectors, multiple encoders are jointly trained using cyclic-global joint contrastive loss to obtain a set of multimodal feature representations aligned by cyclic-global contrastive loss.

[0032] In this invention, traditional star-shaped fusion architectures perform cross-modal alignment centered on a single dominant modality (usually vision). Once the dominant channel data is missing, the overall evaluation system performance deteriorates sharply. This invention employs a recurrent-global joint contrastive loss to jointly train five modal encoders, ensuring that the output feature space of each encoder is aligned not only with adjacent modalities but also with the global multimodal context. This allows each encoder to implicitly encode complementary information from all other modalities. After training, when data from a certain sensor channel is missing, the corresponding encoder can use its learned cross-modal implicit representation for robust inference, resulting in a smooth decline in system evaluation performance rather than a collapse.

[0033] It is worth noting that the star-shaped fusion architecture has an inherent flaw—all modalities align to the central dominant modality. The feature space of the central modality bears too much alignment burden. Once the central modality's channel data is missing, the feature alignment reference for the other modalities disappears, causing the entire feature fusion system to fail. In contrast, the fusion architecture constructed by the recurrent-global joint contrastive loss does not have any privileged central node. The five modal encoders achieve local alignment through the recurrent topology and global alignment through the global loss term. Each encoder experiences alignment constraints from all other modalities equally during training. This design fundamentally eliminates the dominant channel bias, ensuring that the loss of any single or dual channel will not trigger a global feature alignment collapse.

[0034] In the specific operation of the missing channel degradation inference, when a sensor channel is missing data in the current teaching evaluation, the corresponding encoder does not receive the actual acquired data for that channel. Instead, it activates the cross-modal complementary representation that it implicitly learned through the global loss term during the training phase. This complementary representation is essentially a conditional expectation estimate jointly provided by the other four modalities, that is, the optimal inferred value of the missing channel feature given the known features of the other four modalities. Since the five encoders have been fully jointly trained with a large amount of real five-channel complete data during the training phase, the above-mentioned cross-modal implicit representation can capture the deep correlation between channels, thereby providing statistically significant alternative feature contributions in the missing context. This ensures that the overall evaluation system smoothly degrades rather than collapses in the case of missing channels caused by sensor failure or child non-cooperation.

[0035] S3: Using teaching vocabulary as nodes and sensory channel activation relationships and vocabulary logical relationships as edges, construct a directed heterogeneous knowledge graph. Add predecessor-successor forgetting propagation coefficients and semantic confusion interference weights to the graph edges to generate a sensory-language-semantic interference joint knowledge graph.

[0036] Specifically, the node attributes of a knowledge graph include vocabulary categories (nouns, verbs, adjectives, spatial words, emotional words, etc.), associated sensory channels (touch, vision, smell, taste, hearing, proprioception, etc.), pre-dependent vocabulary sets, teaching difficulty levels, and their respective categories (13 categories including fruits, vegetables, furniture, and home appliances). The edge relationships of the graph include sensory synergy edges (e.g., the "cantaloupe" node requires simultaneous activation of the tactile texture, olfactory aroma, and gustatory sweetness channels), logical progression edges (e.g., the concept of "size" depends on the prior recognition of the noun "apple"), generalization equivalence edges (apples of different colors or sizes are mapped to the same vocabulary node), and emotional attachment edges (e.g., "like" is attached to the recognition of a specific noun).

[0037] Specifically, the four types of edge relationships perform different cognitive modeling functions in the knowledge graph. The sensory-coordination edge encodes the necessary activation conditions between vocabulary and sensory channels. For example, the sensory-coordination edge for the vocabulary node "cantaloupe" requires the simultaneous activation of three channels: tactile texture, olfactory aroma, and gustatory sweetness, to determine if the child has established a complete multi-channel sensory-semantic binding. Children with only visual cognition but lacking olfactory and gustatory experiences will have their vocabulary mastery of "cantaloupe" identified by the system as incomplete partial sensory cognition, requiring supplementary training for the missing channels. The logical progression edge encodes the cognitive pre-dependency relationships between vocabulary, reflecting the typical developmental sequence in children's language cognitive development: "concrete before abstract, individual before category, static before dynamic." For example, the establishment of the concept of "size" comparison depends on the child's solidified naming cognition of concrete objects such as "apple," "pear," and "watermelon." Therefore, when arranging the teaching of "size" vocabulary, the system will automatically check whether the comprehensive intervention effect index of the pre-dependent vocabulary has reached the advancement threshold. Generalization equivalence edges encode the semantic mapping relationship of the same word in different physical forms, ensuring that the system can distinguish between children's rote memorization of specific object examples and their generalized cognition of truly mastering the semantics of words. Apples of different colors, sizes, or origins are all mapped to the same "apple" word node in the knowledge graph through generalization equivalence edges, enabling generalization tests to systematically cover multiple dimensions of morphological transformation. Emotional attachment edges encode the semantic anchoring requirements of abstract emotional words for concrete noun cognition. For example, the emotional word "like" requires children to have established a solid naming cognition of specific objects (such as "apple" or "toy"). Emotional attachment edges connect the teaching entry point of abstract emotional words with concrete object word nodes, ensuring that the teaching of emotional words always unfolds on a meaningful semantic carrier.

[0038] By further adding the predecessor-successor forgetting propagation coefficient and semantic confusion interference weight to the above edges, a joint knowledge graph of perception-language-semantic interference is formed, which provides topological constraints and interference priors for subsequent evaluation and scheduling modules.

[0039] S4: Based on the multimodal feature representation set and the joint knowledge graph, calculate the perceptual cognitive integrity index, language output accuracy index and generalization transfer ability index for each word node, and after time decay correction and cascade forgetting propagation correction, normalize and weight the emotional physiological stability index to finally obtain the comprehensive intervention effect index.

[0040] Among them, the Perceptual Cognitive Completeness Index (PCI) measures the degree of activation completeness of children in various perceptual channels associated with target words; the Language Output Accuracy Index (LAI) is based on automatic speech recognition and natural language processing to assess the semantic matching degree and grammatical structure completeness of children's output words and target speech instructions; and the Generalization Transfer Ability Index (GTI) measures the correct recognition rate of children after changing the color, size, or scene of the target object. All three indices incorporate personalized forgetting rate parameters for temporal decay correction, and are superimposed with a cascaded decay correction term of the predecessor node based on the predecessor-successor forgetting propagation coefficient. After normalization, each corrected index is weighted and fused with an emotional physiological stability index (quantified by the stationarity of the GSR change sequence).

[0041] The emotional physiological stability index was quantified by the GSR change sequence collected by a wristband-type electrodermal sensor. Specifically, within each teaching time window, the degree of emotional fluctuation was calculated by combining the baseline drift and peak activation frequency of the GSR signal: the smaller the baseline drift and the lower the peak activation frequency, the more stable and controllable the child's emotional arousal level was within a moderate activation range during the teaching session, corresponding to a higher emotional physiological stability index score; conversely, if the GSR signal showed a sustained baseline increase or high-frequency peak activation, it indicated that the child was in an anxious or over-activated state, corresponding to a lower emotional physiological stability index score. The emotional physiological stability index acts as a moderating coefficient in the weighted fusion of the comprehensive intervention effect index: when the child's emotional physiological state is stable, the weights of the perceptual-cognitive integrity index, language output accuracy index, and generalization transfer ability index of the teaching session are increased, and the comprehensive intervention effect index can better reflect the child's true cognitive level; when the child's emotional fluctuations are significant, the weights of the above three indices are reduced accordingly to avoid the system misjudging a decline in cognitive level due to accidental performance caused by emotional factors, thereby ensuring the stability and reliability of the comprehensive intervention effect index assessment conclusion.

[0042] After weighted fusion, the overall intervention effect index (OIE) of each lexical node is obtained.

[0043] S5: Based on the comprehensive intervention effect index and the joint knowledge graph, the global state embedding is aggregated through a graph attention network, and the local state vector is formed by combining the evaluation vector of the active vocabulary set, the forgetting urgency index and the semantic confusion interference sub-matrix between active vocabulary. The global state embedding and the local state vector are used as state inputs to the trained deep Q network to output a personalized teaching path sequence.

[0044] It should be understood that Graph Attention Networks (GAT) aggregate the evaluation features of all vocabulary nodes on the knowledge graph topology, outputting a fixed-dimensional global state embedding whose dimension is independent of the total number of vocabulary nodes, thus decoupling the dependency between the state space dimension and the vocabulary size. Local state vectors only contain information about the currently active vocabulary set, and their dimension is controllable. Deep Q-Networks take the concatenation of global and local states as input and output the optimal action combination at the end of each teaching cycle, forming a personalized teaching path sequence.

[0045] S6: Generate an auxiliary decision-making report based on the personalized teaching path sequence and the comprehensive intervention effect index of each vocabulary node.

[0046] In this embodiment of the invention, the auxiliary decision-making report is presented to the immediate family therapist through a visual interface on a mobile terminal. The report includes: a radar chart of current stage abilities (time-corrected scores for each dimension: visual, auditory, tactile, olfactory, gustatory, proprioceptive, linguistic, logical, and affective); a list of recommended teaching materials and accompanying verb instruction sets for the next cycle, along with semantic interference risk warnings; early warnings for weak perceptual channels and targeted reinforcement teaching suggestions; a vocabulary forgetting risk warning list (arranged in descending order of forgetting urgency); and predictions of expected language ability milestones and their attainment probabilities. This report presents professional assessment conclusions in an intuitive and visual format, lowering the professional judgment threshold for immediate family therapists.

[0047] Example 2

[0048] like Figure 2 As shown, based on Example 1, multiple indicators of user data during the teaching process are synchronously collected through a sensor network to obtain five modal embedding vectors, including:

[0049] S1.1: Using distributed cameras and eye-tracking modules as the data acquisition units, the system collects a heatmap of the user's gaze point, gaze duration sequence, and eye switching frequency when the user is facing the target object, thereby obtaining visual behavior data.

[0050] In this embodiment of the invention, distributed cameras are deployed at multiple angles on the teaching desktop, and the eye-tracking module operates using non-contact infrared tracking, requiring no device for the child to wear. The gaze point heatmap reflects the distribution of attention intensity for different areas of the target object, the gaze duration sequence records the duration of sustained gaze during each object presentation, and the eye-switching frequency measures the alternating eye contact between the target object and the immediate family member / therapist. These three indicators together constitute the behavioral feature input of the visual channel.

[0051] S1.2: Using a pressure sensor array deployed on the physical object's surface as the data acquisition unit, the system collects the user's hand force trajectory and grip stability index when operating the object, thus obtaining tactile interaction data.

[0052] Specifically, a pressure sensor array is embedded in a dedicated teaching table mat or object tray, recording the temporal changes in pressure values ​​in different areas in real time as children grasp, flip, and press the objects. The hand force trajectory reflects the child's active exploration of the object's texture, and the grasping stability index is quantified by the standard deviation of the pressure signal. It can distinguish between a child's skilled manipulation of the object and the unstable grasping state during initial contact, and is a quantitative representation of the level of proprioceptive and tactile co-development.

[0053] S1.3: Using a directional microphone array as the acquisition unit, the voice content text, feedback delay time, and phonetic accuracy score output by the user after responding to the voice command are acquired to obtain voice feedback data.

[0054] In this embodiment of the invention, a directional microphone array is deployed in front of the child, utilizing beamforming technology to suppress environmental noise and improve the signal-to-noise ratio of the child's speech. The speech content is transcribed by an automatic speech recognition system. The feedback delay time is the time difference between the immediate family member / therapist issuing the speech instruction and the child beginning to output speech. The phonological accuracy score is calculated by acoustically comparing the child's speech with a standard pronunciation template. These three indicators comprehensively reflect the child's speech production ability and language processing speed.

[0055] S1.4: Using wearable inertial sensors as the data acquisition unit, the user's body displacement trajectory and balance control ability index during teaching activities are collected to obtain body motion data.

[0056] The wearable inertial measurement unit (IMU), worn in the form of a lightweight wristband or vest, includes a three-axis accelerometer and a three-axis gyroscope, with a sampling frequency of no less than 50Hz. Body displacement trajectory is estimated using integral acceleration data, reflecting the child's level of active motor participation in educational games; the balance control ability index is calculated by analyzing the power spectral density of trunk acceleration signals in standing or sitting positions, measuring the child's proprioceptive development level.

[0057] S1.5: Using a wristband-type electrodermal sensor as the data acquisition unit, the sequence of changes in the user's electrodermal conductance response during the teaching process is collected to obtain emotional physiological data.

[0058] Optionally, the skin conductance response (GSR) change sequence is continuously recorded at a low-frequency sampling rate of 0.5 Hz. After removing high-frequency noise through sliding window averaging filtering, the peak frequency and average amplitude of the GSR signal within the window are extracted as quantitative indicators of emotional arousal level. High GSR values ​​usually correspond to a higher state of emotional activation (including positive excitement and negative anxiety), and can help assess children's emotional adaptability when combined with the current teaching content.

[0059] S1.6: Using the teaching timestamp as the alignment benchmark, the visual behavior data, the tactile interaction data, the voice feedback data, the proprioceptive motion data, and the emotional physiological data are synchronized in time, and then input into the corresponding modal-specific encoders to extract spatiotemporal features, resulting in five modal embedding vectors.

[0060] Furthermore, each sensor device synchronizes its clock via Network Time Protocol (NTP) upon system startup, using the trigger time of each physical presentation event as the teaching timestamp anchor point. Effective data segments within the teaching window are extracted from the data from each sensor channel along a unified timeline. The five data streams are input to dedicated encoders for five modalities: visual, tactile, speech, motion, and physiological. Each encoder employs a network structure adapted to its corresponding data type to extract spatiotemporal features and outputs a fixed-dimensional modal embedding vector.

[0061] Example 3

[0062] Based on Example 1, multiple encoders are jointly trained using a recurrent-global joint contrastive loss on the five modal embedding vectors to obtain a recurrent-global contrastive aligned multimodal feature representation set, including:

[0063] S2.1: For the five modal embedding vectors, arrange the five modal encoders in a circular topology order, calculate the mutual information estimate of the modal embedding vectors output by adjacent encoders, and sum the mutual information estimates of all adjacent encoder pairs to obtain the cyclic loss term.

[0064] In this embodiment of the invention, five modal encoders are arranged in a circular order of visual → tactile → speech → motor → physiological, forming a cyclic topology. The mutual information estimation of adjacent encoder pairs employs a neural network-based mutual information estimation method (such as MINE or InfoNCE). This method estimates the lower bound of the mutual information between adjacent modalities by comparing positive sample pairs (adjacent modal embeddings at the same teaching time) and negative sample pairs (adjacent modal embeddings at different teaching times). The cyclic loss term is equal to the sum of the mutual information estimates of the five adjacent modal pairs, constraining the feature space output by adjacent encoders to maintain semantic local consistency.

[0065] In this embodiment of the invention, positive sample pairs are composed of embedding vectors of adjacent channels under the same teaching timestamp, reflecting the true co-occurrence association between two adjacent perceptual channels in the same teaching event; negative sample pairs are randomly sampled from the embedding vectors of adjacent channels corresponding to different teaching timestamps in the same batch, representing random pairings where there is no semantic correspondence between the features of two channels after temporal misalignment. By maximizing the mutual information estimate of positive sample pairs and minimizing the mutual information estimate of negative sample pairs, the InfoNCE objective function drives the adjacent encoder to bring the representation of the true co-occurring channels closer in the feature space and push away the representation of the temporally misaligned channels in a contrastive learning manner. It should be noted that the sampling quality of negative samples has a significant impact on the optimization effect of the cyclic loss term. In actual implementation, it should be ensured that each positive sample pair corresponds to a sufficient number of negative sample pairs (e.g., no less than 64) to provide sufficient contrastive learning signals and avoid the model learning overly loose alignment relationships due to insufficient negative samples.

[0066] S2.2: For the output embedding vector of each modal encoder, calculate the mean aggregation representation of the remaining four modal embedding vectors, calculate the mutual information estimate between the output embedding vector and the mean aggregation representation, and sum the mutual information estimates of the five encoders to obtain the global loss term.

[0067] In specific implementation, for the embedding vector output by the $i$-th encoder, the embedding vectors of the other four encoders are average-pooled to obtain a global multimodal context representation. The mutual information estimate between this embedding vector and the global context representation is calculated, and the mutual information estimates calculated for each of the five encoders are summed to obtain the global loss term. The global loss term ensures that the output embedding vector of each encoder not only captures information of its own modality but also perceives the global semantics of all other modalities, enabling each encoder to provide an effective cross-modal implicit representation even when any other modality is missing.

[0068] S2.3: The cyclic loss term and the global loss term are weighted and summed with a balance coefficient to obtain the joint contrastive loss. The five modal encoders are jointly trained using the joint contrastive loss as the objective function to obtain a set of multimodal feature representations aligned with cyclic-global contrast.

[0069] It should be noted that the balance coefficient controls the weight distribution between the cyclic loss term and the global loss term, and can be determined by tuning the prediction accuracy of the comprehensive intervention effect index on the validation set. Joint training uses an end-to-end backpropagation method to simultaneously update the parameters of the five modal encoders. After training convergence, the output embedding vectors of each encoder constitute a set of multimodal feature representations aligned to cyclic-global contrast, which possesses robust inference capabilities even when any single or dual-channel is missing.

[0070] In this embodiment of the invention, the cyclic-global joint contrastive loss is a self-supervised contrastive learning objective function specifically designed for multi-channel heterogeneous sensory signal fusion scenarios. Essentially, it transforms the multimodal feature alignment problem into a cyclic maximum mutual information estimation problem. Through the joint constraints of the cyclic graph topology and the global aggregation term, it drives the output feature space of each modal encoder to simultaneously satisfy both local neighborhood alignment and global multimodal context alignment constraints. This allows each encoder to implicitly encode complementary information from all other modalities while extracting features for its own modality.

[0071] The core insight of this design lies in the existence of two levels of co-occurrence relationships in multimodal information: local synergistic relationships between adjacent sensory channels (e.g., the joint perception of the same object by touch and vision is often strongly correlated), and overall consistency relationships between any modality and the global multimodal context (i.e., the activation pattern of any single channel should be semantically consistent with the teaching event jointly described by all channels). Traditional star-shaped fusion architectures capture the latter through a single central node while completely ignoring the former, resulting in the feature alignment quality of peripheral modalities (such as olfactory and gustatory channels) heavily depending on the feature quality of the central dominant modality. The recurrent-global joint loss captures local synergistic relationships simultaneously through a recurrent graph structure and captures overall consistency relationships through a global loss term, making the two types of alignment constraints complementary rather than interdependent, thereby maintaining the overall robustness of the system when any channel is missing.

[0072] In terms of technical implementation, the construction of the cyclic-global joint contrastive loss is divided into two parallel computational branches. The cyclic loss branch calculates and sums the mutual information estimates between the output embeddings of adjacent encoder channels in a circular order of vision → touch → speech → motion → physiology, forming the cyclic loss term. The global loss branch calculates and sums the mutual information estimates between the output embedding of each encoder channel and the aggregated vector of the mean embeddings of the other four channels, forming the global loss term. The loss values ​​of the two branches are weighted by a balancing coefficient and merged into the joint contrastive loss. This loss is used as the objective function to perform end-to-end joint training on the five encoders. The training data comes from accumulated historical teaching session records. Each training sample is a synchronous data segment of the five sensors within a complete teaching time window. Positive and negative sample pairs are constructed according to the principle of the same timestamp (positive) and randomly misaligned timestamps (negative). During training, the mutual information estimation adopts a lower bound estimation method based on neural networks, which does not require explicit modeling of the joint probability distribution, has high computational efficiency, and is suitable for high-dimensional embedding spaces. The selection of the balance coefficient aims to achieve the optimal prediction accuracy of the comprehensive intervention effect index on the validation set. In one implementation, the weight ratio of the cyclic loss term to the global loss term can be set between 1:0.5 and 1:2, and the specific value is adjusted according to the number of sensor channels and the data quality distribution of each channel.

[0073] Through the above design, this invention achieves the following technical effects. First, the recurrent-global joint architecture fundamentally eliminates the dominant channel bias problem, ensuring that the five sensory channels have a completely symmetrical position in the feature alignment system. Weak channels such as smell and taste, which are easily overlooked in traditional schemes, obtain the same feature alignment quality assurance as the visual channel, making the sensory cognitive integrity assessment results based on these channels more reliable. Second, the cross-modal implicit representation substitution mechanism for missing channels allows the system to smoothly degrade when sensor channel data is missing: under the condition of allowing a maximum of two missing channels, the consistency of the comprehensive intervention effect index assessment conclusion can still be maintained at a reasonable level, avoiding the situation where the entire teaching cycle assessment is invalidated due to a single acquisition failure, and ensuring the continuity of the family intervention data stream. Third, the self-supervised contrastive learning training paradigm does not require manual annotation of the true feature values ​​of missing channels. It can complete the learning of cross-modal implicit representations using only complete five-channel data, significantly reducing the data annotation cost of system deployment and update maintenance.

[0074] Let's take a real-world application scenario as an example. In a teaching session, a child refused to cooperate by wearing a wristband-type electrodermal sensor, resulting in the complete loss of emotional physiological channel data (the fifth bit of the missing channel marker vector was 0). At this time, the physiological encoder corresponding to the emotional physiological channel activated its cross-modal implicit representation. This implicit representation, based on the joint semantics of visual behavioral data (the gaze point heatmap showed that the child's gaze at the target object for a relatively long time), vocal feedback data (short feedback delay and high phonetic accuracy), and proprioceptive motion data (stable body displacement trajectory) from the teaching session, inferred that the child's emotional state was relatively stable during the teaching session. The corresponding implicit feature vector replaced the missing GSR data contribution in the calculation of the comprehensive intervention effect index, allowing the assessment to be completed normally. The immediate family therapist did not need to interrupt the teaching recording due to the sensor's absence.

[0075] Example 4

[0076] Based on Example 1, such as Figure 3 As shown, a directed heterogeneous knowledge graph is constructed using teaching vocabulary as nodes and perceptual channel activation relationships and lexical logical relationships as edges. Precursor-successor forgetting propagation coefficients and semantic confusion interference weights are added to the graph edges to generate a joint perceptual-language-semantic interference knowledge graph. The estimation of the predecessor-successor forgetting propagation coefficient includes:

[0077] S3.1: For the word node pairs with logically progressive relationships in the joint knowledge graph, extract the first comprehensive intervention effect index time sequence of the predecessor word node and the second comprehensive intervention effect index time sequence of the successor word node from the historical intervention data, calculate the correlation coefficient between the first comprehensive intervention effect index time sequence and the second comprehensive intervention effect index time sequence, and output the initial forgetting propagation coefficient.

[0078] In this embodiment of the invention, logically progressive edges in the knowledge graph represent that predecessor words are prerequisites for successor words—for example, mastering the concept of "size" depends on the solid cognition of nouns such as "apple" and "pear." The decay of cognition of predecessor word nodes will affect the stability of mastery of successor words through this logical dependency relationship. To quantify the degree of this influence, the time series of the comprehensive intervention effect index of word node pairs with co-occurrence training records is extracted from historical intervention data, and the Pearson correlation coefficient between the decay of predecessor nodes and the change in the comprehensive intervention effect index of successor nodes is calculated, which is used as the initial forgetting propagation coefficient.

[0079] S3.2: When the number of co-occurrence training samples of word node pairs in the historical intervention data is less than a preset threshold, a monotonically increasing function of the difference in teaching difficulty between the predecessor word node and the successor word node is used as the prior initial value to replace the initial forgetting propagation coefficient.

[0080] Preferably, for newly enrolled children or vocabulary node pairs that have not yet accumulated sufficient co-occurrence training records, the difference in teaching difficulty is used as a proxy variable: the greater the difference in difficulty levels between the preceding and succeeding vocabulary, the stronger the cognitive dependence between them, and the higher the prior initial value of the forgetting propagation coefficient is set. This prior initial value is gradually replaced by a data-driven estimate as historical intervention data accumulates, achieving a smooth transition in the cold start phase.

[0081] S3.3: Starting from the initial forgetting propagation coefficient or the prior initial value, the predecessor-successor forgetting propagation coefficient is updated on a rolling basis with the historical intervention data added after each teaching cycle ends. The updated predecessor-successor forgetting propagation coefficient is attached to the corresponding logical progression edge in the joint knowledge graph to generate a joint knowledge graph of perception-language-semantic interference containing the forgetting propagation coefficient.

[0082] Among them, the rolling update adopts an exponentially weighted moving average strategy, giving higher weight to recent intervention data, so that the forgetting propagation coefficient can dynamically reflect the changes in the strength of knowledge dependence in the child's current cognitive state.

[0083] Example 5

[0084] Based on Example 1, a directed heterogeneous knowledge graph is constructed using taught vocabulary as nodes and perceptual channel activation relationships and vocabulary logical relationships as edges. Precursor-successor forgetting propagation coefficients and semantic confusion interference weights are added to the graph edges to generate a joint perceptual-language-semantic interference knowledge graph. The estimation of the two-stage semantic confusion interference weights includes:

[0085] S3.4: For any pair of word nodes in the joint knowledge graph, calculate the set intersection-union ratio similarity of the sets of perceptual channels associated with the pair of word nodes to obtain the perceptual channel overlap; extract the semantic embedding vector of the arbitrary word node, calculate the normalized vector inner product similarity to obtain the semantic embedding similarity; perform a weighted calculation on the perceptual channel overlap and the semantic embedding similarity to obtain the prior estimate of the semantic confusion interference weight.

[0086] In this embodiment of the invention, the overlap of sensory channels is calculated using the Jaccard similarity coefficient of the sets of sensory channels associated with each word pair. If "apple" is associated with four channels—visual, tactile, olfactory, and gustatory—and "pear" is also associated with these four channels, then the Jaccard similarity coefficient of their sensory channels is 1.0, indicating that children are highly likely to confuse the two in the sensory dimension. Semantic embedding similarity is calculated using the normalized inner product of the semantic vectors of the words. Prior estimation uses a weighted combination of sensory channel overlap as the primary weight component and semantic embedding similarity as the secondary weight component, reflecting the cognitive pattern that children's cognitive confusion is driven more by the similarity of sensory experiences than by semantic similarity.

[0087] S3.5: Count the cumulative number of times users misclassify the word node pair in the historical teaching records. When the cumulative number reaches the preset sample size threshold, enter the hot start stage. Perform Bayesian posterior fusion with the historical misclassification frequency and the prior estimate of the semantic confusion interference weight to obtain the semantic confusion interference weight.

[0088] It is worth noting that Bayesian posterior fusion treats the prior estimate as a parameter of the prior distribution and the historical misjudgment frequency as a likelihood function, calculating the posterior estimate through a conjugate update formula. As the number of misjudged samples continues to accumulate, the confidence interval of the posterior estimate gradually narrows, ultimately becoming dominated by data-driven misjudgment frequency, thus achieving a precise characterization of the patterns of confusion behavior in individual children.

[0089] S3.6: The semantic confusion interference weights are appended to the corresponding word node pairs in the joint knowledge graph in the form of undirected interference edges to obtain the perception-language-semantic interference joint knowledge graph containing semantic confusion interference weights.

[0090] Specifically, undirected interference edges and directed logically progressive edges are identified by different edge types in the graph, allowing for separate retrieval during graph queries to avoid confusion. The semantic confusion interference weights are updated in real time based on the misjudgment statistics at the end of each teaching cycle, maintaining dynamic tracking of the child's current cognitive state.

[0091] The two-stage semantic confusion interference weight estimation method is a prior-posterior fusion estimation mechanism specifically designed for children's vocabulary cognitive confusion behavior. The "semantic confusion interference weight" refers to the quantified probability strength of the negative interference between the active training of one word node pair and the recognition accuracy of the other in a knowledge graph. The value ranges from [0,1], with higher values ​​indicating a greater risk of confusion between the two words in the child's cognitive dimension. Unlike traditional word vector semantic similarity, this weight not only reflects the similarity of words in the adult semantic space but, more importantly, reflects the child's tendency to confuse at the perceptual experience level. Therefore, its calculation must integrate two types of information sources: perceptual channel overlap and actual misjudgment behavior data.

[0092] The necessity of the two-stage design stems from a core contradiction: while data-driven misjudgment frequency statistics best reflect the true patterns of children's confusion behavior, the sample size for misjudgments is zero for newly enrolled children or vocabulary pairs that have not yet undergone comparative teaching, making frequency statistics unusable. If word vector cosine similarity is relied upon as a substitute, the calculation results reflect the similarity of words in the adult language semantic space. However, children's perceptual cognitive confusion is more driven by the similarity of sensory features of objects—for example, the word vector similarity between "apple" and "pear" is not particularly high in the adult semantic space, but for children in the early language acquisition stage, their perceptual features highly overlap in the visual (both round shape, similar size), tactile (similar surface texture), and gustatory (both sweet fruits) channels, easily leading to confusion. Therefore, the cold start stage must use the overlap of perceptual channels as the primary weighting source to provide prior estimates with practical intervention guidance.

[0093] In the cold-start phase, the prior estimation uses the Jaccard similarity coefficient of the word pair's associated perceptual channel set as the first weight component and the normalized inner product of the word semantic embedding vector as the second weight component. When the two components are weighted and summed, the weight of the first component is set to be 2 to 3 times that of the second component. This weight ratio is chosen because, in the early vocabulary acquisition stage of children, the correlation between perceptual channel overlap and actual confusion behavior is significantly higher than that between semantic similarity and confusion behavior. Giving higher weight to perceptual channel overlap can make the prior estimation more consistent with the child's actual confusion risk. In the warm-start phase, the Bayesian posterior fusion uses the Beta distribution as the conjugate prior, transforming the cold-start prior estimate into the shape parameter of the Beta distribution. The number of historical misjudgments and the number of historical correct recognitions are used as the positive and negative sample counts for posterior updates, respectively. The posterior expectation value is calculated in a rolling manner as the updated semantic confusion interference weight. As the number of misjudged samples increases, the confidence interval of the posterior estimate gradually narrows, the weight value converges to the data-driven true misjudgment frequency, the influence of the prior estimate gradually weakens, and finally, the actual confusion behavior pattern of the individual child becomes dominant.

[0094] This method yields the following technical benefits. First, the prior estimation based on the overlap of sensory channels during the cold start phase enables the system to provide meaningful interference risk references for teaching scheduling from the early stages of intervention. This avoids the problem of newly enrolled children being unable to obtain semantic interference sensory scheduling support due to a lack of historical misjudgment data, significantly improving the scheduling quality in the early stages of intervention. Second, the two-stage smooth transition mechanism ensures the stability of weight estimation—when the number of misjudgment samples is small, the posterior estimation has limited correction to the prior; when the sample size is sufficient, data-driven accurate estimation naturally takes over, and there are no abrupt changes in the entire transition process, resulting in good temporal stability of the weight sequence. Third, the dual information sources of sensory channel overlap and misjudgment frequency enable the system to identify and correct some misjudgment noise caused by accidental factors, improving the noise resistance of weight estimation.

[0095] Let's illustrate this with a real-world application scenario. Suppose a child is just beginning intervention, and the system needs to estimate the semantic confusion weights between "apple" and "pear." "Apple" is associated with four sensory channels: vision, touch, smell, and taste. "Pear" is also associated with these four channels, with a Jaccard similarity coefficient of 1.0 for the sensory channels. The normalized inner product similarity of the semantic embeddings of the two words is 0.62. Using a weighted ratio of 2:1, the cold-start prior estimate is (1.0 × 2 + 0.62 × 1) ÷ 3 ≈ 0.87. Based on this, the system identifies "apple" and "pear" as a high-confusion word pair, automatically marking them as unsuitable for simultaneous introduction in the same period in the scheduling plan, and prompting the immediate family therapist to pay attention to differentiation training. After 8 teaching cycles, the system recorded that the child misidentified "pear" as "apple" 3 times and "apple" as "pear" 2 times, for a total of 5 misidentifications and 22 correct identifications. Starting with the prior Beta distribution parameters, the Bayesian posterior fusion, after 5 positive updates (misclassification) and 22 negative updates (correct recognition), adjusts the posterior expectation value to approximately 0.79, reflecting that the child still has a high risk of confusion between the two words. The system continues to retain the priority of training scheduling for interference discrimination of this word pair.

[0096] Example 6

[0097] like Figure 4 As shown, based on Example 1, the perceptual-cognitive integrity index is calculated and a cascade forgetting propagation correction is introduced, including:

[0098] In this embodiment of the invention, the temporally modified multidimensional evaluation index system is a cognitive state dynamic tracking framework that combines personalized parameterization of the Ebbinghaus forgetting curve with the causal structure of a knowledge graph. Unlike existing technologies that only statically assess a child's performance at the end of a lesson, this framework models vocabulary mastery as a dynamic variable that decays continuously over time. By fitting an independent forgetting rate parameter to each child at each vocabulary node, it can provide an optimal estimate of a child's current cognitive state at any given time (including intervals between teaching sessions). Furthermore, this framework incorporates the predecessor-successor topology of the knowledge graph into the forgetting propagation model, explicitly quantifying the cascading effect of the cognitive decay of predecessor words on the stability of successor word mastery, thereby achieving cross-vocabulary node cognitive state linkage tracking.

[0099] The introduction of personalized forgetting rates is based on the following cognitive science: There are significant individual and categorical differences in the forgetting rates of different types of vocabulary among children. Generally, concrete nouns (such as "apple" and "table") are forgotten more slowly due to their rich multi-sensory anchor points, while abstract emotional words (such as "like" and "fear") are forgotten more quickly due to the lack of direct sensory correspondences. Children with higher levels of sensory development typically have lower overall forgetting rates than children with delayed sensory development. Traditional schemes that fix forgetting curve parameters cannot capture these individual differences, leading to systematic biases in the estimation of cognitive states during teaching intervals. The introduction of precursor-successor causal forgetting propagation is based on the following language cognitive laws: There are clear sequential dependencies in the vocabulary cognitive network. When the mastery stability of a precursor word (such as the concrete noun "apple") decreases due to forgetting, the semantic understanding foundation of subsequent words (such as the comparative words "big apple" and "small apple") that depend on that precursor word weakens accordingly, resulting in a predictable decline in the accuracy of responses to subsequent words in the next teaching session. If this cascading effect is not modeled, the system will underestimate the actual degree of forgetting of subsequent vocabulary, resulting in an erroneous judgment that the teaching progress is falsely ahead of schedule.

[0100] In the temporal correction implementation of the Perceptual-Cognitive Completeness Index (PCI), an exponential decay correction is applied using the immediate teaching activation completeness rate as a benchmark and the product of the personalized forgetting rate parameter and the teaching interval as the decay index. Further, based on the difference between the historical stable reference value and the current decay value of the predecessor node, a cascaded decay correction is superimposed after weighting by the forgetting propagation coefficient, and finally normalized to a valid value range using sigmoid, resulting in the temporally corrected PCI. In the temporal correction implementation of the Language Output Accuracy Index (LAI), based on the semantic matching degree and grammatical completeness score of the child's current speech feedback by the automatic speech recognition system, a temporal decay correction is also applied using the personalized forgetting rate parameter corresponding to the lexical node, reflecting the temporal stability of language output ability during the teaching interval. Compared to PCI, the forgetting rate of LAI is generally lower because the neuromuscular memory of language output has stronger temporal stability than the cognitive memory activated by the perceptual channel. In the temporal correction implementation of the Generalization Transfer Ability Index (GTI), a semantic interference discount factor is introduced on top of the instantaneous generalization recognition accuracy. When the target word and words with high interference weights are in an active training state simultaneously, a discount correction is applied to the GTI, so that the assessment results of generalization transfer ability can reflect the actual cognitive challenges faced by children under the current teaching arrangement. The personalized forgetting rate of GTI is generally higher than that of LAI and PCI because generalization transfer ability depends on the deep internalization of word semantics, which is more likely to regress during the interval than the surface sensory activation and speech production. After normalization, the four indices (PCI, LAI, GTI) are combined with the emotional physiological stability regulation coefficient for weighted fusion to obtain the Overall Intervention Effect Index (OIE), which serves as the core state signal and reward signal source for the reinforcement learning path planning engine.

[0101] Through the aforementioned time-series correction assessment index system, this invention achieves the following technical effects. First, the introduction of personalized forgetting rate parameters enables the system to provide an optimal estimate of a child's cognitive state at any teaching interval without requiring real-time teaching data. This allows immediate family therapists to obtain accurate references to the current cognitive state before starting a new round of teaching, and to rationally prioritize teaching content. Second, the correction of precursor-subsequent causal forgetting propagation enables the assessment system to provide early warnings of subsequent vocabulary cognitive risks caused by the forgetting of precursor vocabulary, assisting the teaching scheduling system in proactively arranging the consolidation and review of precursor vocabulary, thus fundamentally blocking the cascading propagation of cognitive regression. Third, the four-dimensional assessment index system decouples the assessment of the four dimensions of sensory cognition, language production, generalization transfer, and emotional adaptation, enabling the system to accurately identify the child's uneven development in each dimension. This provides data support for targeted single-dimensional reinforcement training, avoiding the misallocation of intervention resources caused by a "one-size-fits-all" approach.

[0102] Let's illustrate this with a real-world application scenario. Assume a child's Overall Intervention Effect Index (OIE) after the last "banana" vocabulary lesson was 0.82, and the next lesson is 7 days later. The system fits a personalized PCI forgetting rate parameter of 0.04 (unit: 1 / day) for the child's "banana" vocabulary based on historical data. The calculated PCI after time decay is 0.82×e^(-0.04×7) ≈ 0.82×0.756 ≈ 0.62. Simultaneously, "banana" has a predecessor dependent word "yellow" in the knowledge graph. The current PCI after time decay is 0.55, the historical stable reference value is 0.75, the difference is 0.20, the forgetting propagation coefficient is 0.35, and the cascade decay correction is 0.20×0.35 = 0.07. Finally, the PCI after sigmoid normalization is approximately 0.57. This time-series correction result shows a significant decrease compared to the immediate measurement of 0.82 seven days ago. Based on this, the system judges that the word "banana" has a high risk of being forgotten, and includes it in the priority review queue in the teaching path planning. At the same time, it marks the consolidation training needs of the preceding words in yellow and issues a targeted warning to the immediate family therapist.

[0103] S4.1: For each word node, count the user's real-time activation status on each sensory channel associated with each word node, calculate the activation completeness rate, and obtain the real-time sensory cognitive completeness.

[0104] In this embodiment of the invention, taking the teaching vocabulary word "apple" as an example, its associated sensory channels include four channels: visual (color and shape recognition), tactile (surface texture perception), olfactory (fruit aroma recognition), and gustatory (sweet and sour taste). The criteria for immediate activation are: visual channel activation is defined as a fixation duration exceeding a threshold; tactile channel activation is defined as effective grasping behavior (grasping stability index above baseline); and olfactory and gustatory channels activation is defined as the child actively approaching or ingesting the object. The activation completeness rate is the ratio of the number of activated channels to the total number of associated channels, yielding the immediate sensory cognitive completeness.

[0105] S4.2: Extract the historical perceptual cognitive completeness time series of the word node, fit the user's personalized forgetting rate parameter on the word through maximum likelihood estimation, and initialize it with the average forgetting rate parameter of the user group when the historical training samples of the word are insufficient; use the personalized forgetting rate parameter and the time interval from the last teaching as input, apply exponential decay correction to the immediate perceptual cognitive completeness to obtain the perceptual cognitive completeness after time decay.

[0106] Specifically, the exponential decay correction uses immediate perceptual cognitive integrity as the baseline value and calculates the decay exponent using the product of a personalized forgetting rate parameter and the time interval since the last teaching session. A larger personalized forgetting rate parameter indicates a faster rate of cognitive decay for the word, with a significant decline in mastery occurring even after a short teaching interval. By fitting this parameter to historical time-series data using maximum likelihood estimation, the differentiated forgetting characteristics of different children for different word categories (such as abstract emotional words and concrete fruit nouns) can be captured.

[0107] In the specific implementation of maximum likelihood estimation, the system pairs the perceptual-cognitive completeness assessment values ​​of the vocabulary node in historical teaching records with the corresponding teaching intervals, forming several triples (immediate mastery, teaching interval, post-interval mastery). Assuming an exponential decay model, the optimal forgetting rate parameter is solved by minimizing the root mean square error between the model-predicted post-interval mastery and the actual measured value. The entire estimation process can be efficiently completed using gradient descent or analytical solutions. For vocabulary nodes with insufficient historical training samples (e.g., fewer than 5 triples), the system uses the mean of the forgetting rate parameter estimated by the current child on other vocabulary nodes with sufficient samples as the initial value. This initial value is gradually replaced by vocabulary-specific estimates as subsequent teaching data accumulates. The update frequency of the forgetting rate parameter is set to trigger once after each teaching cycle, ensuring that the parameter can dynamically track the development and changes in children's cognitive abilities—as children's overall language ability improves, their forgetting rate of mastered vocabulary usually gradually decreases. Continuous parameter updates can promptly capture this positive trend, thereby dynamically adjusting the optimization strategy of the teaching interval.

[0108] S4.3: For each predecessor word node of the word node in the joint knowledge graph, calculate the difference between the historical stable reference value of the predecessor word node and the perceptual cognitive integrity after the current time-decayed value. When the difference is positive, the difference is weighted by the predecessor-successor forgetting propagation coefficient. The weighted difference of each predecessor word node is summed to obtain the cascaded decay correction amount. The perceptual cognitive integrity after the time-decayed value is subtracted from the cascaded decay correction amount and then mapped to the [0,1] interval by the normalization function to obtain the comprehensive intervention effect index.

[0109] Furthermore, the historical stable reference value is taken as the moving average of the comprehensive intervention effect index of the preceding vocabulary node over several recent teaching cycles, representing the expected level of the preceding vocabulary under a stable mastery state. When the completeness of the preceding vocabulary after current temporal decay is lower than the historical stable reference value, the difference is positive, triggering cascade decay correction; when the difference is negative (i.e., the current state of the preceding vocabulary is better than the historical average), no correction is applied to avoid erroneously amplifying the evaluation results of subsequent vocabulary. The normalization function adopts the sigmoid transformation to smoothly map the corrected value to the [0,1] interval, ensuring the validity of the comprehensive intervention effect index value.

[0110] Example 7

[0111] Based on Example 1, a generalization transferability index is calculated and a semantic interference discount correction is introduced, including:

[0112] S4.4: For each vocabulary node, by changing the color, size, or scene of the target object in the teaching scenario, the number of times the user correctly identifies the object and the total number of tests are collected, the correct recognition rate is calculated, and the instant generalization transfer capability is obtained.

[0113] In this embodiment of the invention, the generalization test employs a systematic morphological substitution strategy: for noun words (such as "apple"), three generalization scenarios are tested: red apple → green apple (color change), big apple → small apple (size change), and apple on a table → apple in a basket (scene change). The ratio of the number of correct recognitions in each scenario to the total number of tests is calculated to obtain the immediate generalization transfer ability. Generalization transfer ability reflects whether children have moved beyond rote memorization of specific object examples and achieved true internalization of the semantics of the vocabulary.

[0114] S4.5: Retrieve the set of word nodes in the joint knowledge graph that have undirected interference edges with each of the word nodes and whose semantic confusion interference weight exceeds a preset interference threshold, and determine whether the word nodes in the set are in the same active training state of the same teaching cycle as the current word node.

[0115] Optionally, the preset interference threshold can be adjusted according to the system deployment scenario, typically set between 0.6 and 0.8, to filter out a candidate set of interference words that pose a significant risk of confusion with the target words. For each candidate word in this set, it is checked whether it has been included in the teaching plan and actual training has been conducted in the current teaching cycle to determine whether it constitutes a real interference situation.

[0116] S4.6: When there are interfering word nodes that are in active training at the same time as the current word node, the semantic confusion interference weight is used as the discount coefficient to apply interference discount correction to the instant generalization transfer ability. The discount coefficient is directly assigned by the semantic confusion interference weight. The corrected result is mapped to the [0,1] interval through a normalization function to obtain the comprehensive intervention effect index.

[0117] It should be understood that the physical meaning of interference discounting is that when semantically similar words are in an active learning state simultaneously, children's generalization recognition task is actually more difficult (because children need to distinguish between two words under similar perceptual stimuli). Therefore, the actual generalization transfer ability should be discounted based on the immediate measurement value to more realistically reflect the degree of cognitive challenge faced by children under the current teaching arrangement. The discount coefficient is positively correlated with the semantic confusion interference weight; the stronger the interference, the greater the discount.

[0118] Example 8

[0119] Based on Example 1, after obtaining the comprehensive intervention effect index, the following is also included:

[0120] Step A: Obtain the missing channel marker vector to determine the set of sensor channels with missing data in the current teaching evaluation cycle.

[0121] In this embodiment of the invention, the missing channel marker vector is a five-dimensional binary vector, with each dimension corresponding to a sensor channel (visual, tactile, speech, motion, physiological). A value of 1 indicates that the channel data is valid, while a value of 0 indicates that the channel data is missing due to acquisition failure, child non-cooperation, or equipment malfunction. The system automatically checks the data integrity of each channel after each teaching time window ends and generates a real-time missing channel marker vector.

[0122] Step B: For each missing channel in the set of sensor channels with missing data, extract the output embedding vector of the corresponding modal encoder after the completion of the recurrent-global joint contrastive training. Use the output embedding vector as the cross-modal implicit representation of the missing channel to replace the feature contribution corresponding to the original acquired data of the missing channel.

[0123] In this approach, the recurrent-global joint contrastive training implicitly encodes the complementary information of all other modalities into the output embedding vector of each modal encoder. When data for a certain channel is missing, the encoder corresponding to that channel cannot receive the actual data collected in that session. Instead, it uses the cross-modal implicit representation learned in historical training as the feature contribution of that channel in the current evaluation, thus achieving implicit substitution inference for the missing data.

[0124] Step C: Based on the channel features after the cross-modal implicit representation replacement, participate in the calculation of the perceptual cognitive integrity index, the language output accuracy index, and the generalization transfer ability index. The calculation result of the comprehensive intervention effect index remains effective as long as the number of missing channels does not exceed the preset maximum number of missing channels.

[0125] It should be noted that the preset maximum number of missing channels is usually set to 2, meaning that the system can still output valid evaluation results even when a maximum of 2 sensor channels are missing data simultaneously. When the number of missing channels exceeds this threshold, the system issues a data quality warning to the immediate family therapist, indicating that the reliability of the evaluation results has decreased and suggesting that data be collected again.

[0126] Example 9

[0127] like Figure 5 As shown, based on Example 1, a graph attention network is used to aggregate global state embeddings and construct local state vectors to drive a deep Q-network to output a personalized teaching path sequence, including:

[0128] S5.1: The comprehensive intervention effect index, perceptual cognitive integrity index, language output accuracy index, and generalization transfer ability index of all vocabulary nodes are used as node features. The topology of the joint knowledge graph is used as the graph structure. The graph attention network is input and the neighbor node features are aggregated layer by layer through the multi-head attention mechanism to obtain a fixed-dimensional global state embedding. The dimension of the global state embedding is independent of the total number of vocabulary nodes.

[0129] In this embodiment of the invention, the Graph Attention Network (GAT) uses the four-dimensional evaluation vector of each word node as the initial node feature, and uses perceptual synergy edges, logical progression edges, generalization equivalence edges, and emotional attachment edges as the graph structure. Through a multi-head attention mechanism, it aggregates the feature information of neighboring nodes layer by layer on the graph's topology. Finally, a global pooling layer compresses the final layer features of all nodes into a fixed-dimensional global state embedding vector. This vector comprehensively reflects the mastery state of all word nodes at the current moment and their interdependencies in the knowledge graph topology, and its dimension is independent of the total number of words, effectively solving the problem of state dimension explosion in large-vocabulary scenarios.

[0130] S5.2: Determine the set of active vocabulary nodes in the current teaching cycle, extract the comprehensive intervention effect index of each active vocabulary node, construct the forgetting urgency index by multiplying the personalized forgetting rate parameter by the time interval since the last teaching, and form a semantic confusion interference sub-matrix between active vocabulary nodes from the semantic confusion interference weights between each pair of nodes in the set of active vocabulary nodes; concatenate the comprehensive intervention effect index, the forgetting urgency index and the semantic confusion interference sub-matrix between active vocabulary nodes to obtain a local state vector.

[0131] Specifically, the active vocabulary node set typically consists of vocabulary nodes included in the teaching plan within the current teaching cycle, generally not exceeding 10. The forgetting urgency index is represented by the product of a personalized forgetting rate parameter and the time interval since the last teaching session; a larger product indicates a more severe degree of vocabulary cognitive decay and a more urgent need for review. The semantic confusion interference sub-matrix among active vocabulary nodes is a square matrix composed of the semantic confusion interference weights of each node within the active vocabulary node set, with a dimension equal to the square of the number of active vocabulary nodes. Since the number of active vocabulary nodes is controllable, the dimension of this sub-matrix is ​​also controllable.

[0132] S5.3: Concatenate the global state embedding with the local state vector, input it into the trained deep Q network, and output a personalized teaching path sequence.

[0133] In this embodiment of the invention, the action space of the deep Q-network includes seven types of actions: selecting the next target vocabulary for teaching, selecting teaching methods (six scenario modes: shopping, eating and drinking, making, functioning, generalization, and games), adjusting the multimodal stimulus intensity parameters, adjusting the language input complexity (three levels: telegraphic sentences, simple sentences, and complex sentences), triggering stage promotion verification, triggering forgetting review, and performing interference discrimination training. At the end of each teaching cycle, the deep Q-network takes the concatenated state vector as input, outputs the Q-value of each action, and selects the action combination with the highest Q-value as the teaching plan for the next cycle, forming a personalized teaching path sequence.

[0134] In this embodiment of the invention, the semantic interference perception extended DQN teaching path scheduling method based on graph attention compression state is a dynamic teaching sequence generation method that combines deep reinforcement learning with knowledge graph topology perception and is geared towards children's language intervention scenarios. Unlike existing rule-based or expert system-based teaching recommendation schemes, this method does not require manual pre-setting of teaching rules and vocabulary introduction order. Instead, it automatically learns the optimal scheduling strategy from a large amount of teaching interaction data through a deep Q-network. At the same time, it compresses and encodes the topological constraint information of the knowledge graph into the state representation through a graph attention network, enabling the agent to implicitly perceive the logical progression relationship, generalization equivalence relationship, and semantic confusion interference relationship between words when making scheduling decisions. Thus, it generates a personalized optimal teaching path sequence under the constraint of following the laws of children's cognitive development.

[0135] The technical feasibility of this method relies on the synergistic effect of two key designs. First, the Global State Embedding Compression (GAT) on the knowledge graph topology solves the curse of dimensionality problem caused by the linear growth of the state space dimension with the vocabulary size. If the evaluation vectors of all vocabulary nodes are directly concatenated as the state input, the state vector dimension would exceed a thousand dimensions in scenarios where the total number of nodes across 13 major vocabulary categories could reach hundreds, severely impacting the sample efficiency and generalization ability of DQN. GAT performs hierarchical feature aggregation on the graph structure through a multi-head attention mechanism, compressing all vocabulary node information into a fixed-dimensional global embedding vector. This compression process is not a simple loss of information, but rather an information distillation process that preserves the topological relationships between words. Vocabulary node pairs with strong logical progressive dependencies receive higher aggregation priority in GAT's attention weight allocation, enabling the global embedding to highlight key structural information in the knowledge graph. Secondly, the detailed characterization of the current active vocabulary set by the local state vector makes up for the loss of current teaching context details due to the high compression of the global embedding. The forgetting urgency index and the semantic confusion interference submatrix between active vocabulary directly present the dynamic information most relevant to the current decision in the form of low-dimensional dense vectors, so that DQN can take into account both the global knowledge graph state (from the global embedding) and the current local teaching context (from the local state vector) when outputting actions.

[0136] In terms of the design logic of expanding the action space, seven types of actions cover all dimensions of instructional scheduling decisions: selecting the next target vocabulary determines the traversal path on the knowledge graph; selecting the teaching method determines the presentation of sensory stimuli (shopping scenarios emphasize functional cognition, eating and drinking scenarios emphasize taste and smell, making scenarios emphasize tactile and proprioceptive participation, and playing scenarios emphasize emotional activation and motor participation); adjusting the multimodal stimulus intensity parameters and language input complexity jointly determines the cognitive load level of a single instruction; triggering stage progression verification determines whether a child enters the assessment for the next developmental stage; triggering forgetting review is a proactive response to the risk of forgetting; performing interference differentiation training is a proactive mitigation of the risk of semantic confusion. The orderly combination of the seven types of actions forms a complete plan for each instructional cycle. By learning the long-term cumulative benefits of each action under different states, DQN automatically discovers the optimal scheduling balance point under the triple constraints of forgetting patterns, semantic interference, and cognitive development.

[0137] The collaborative working mechanism of the three components in the extended reward function is as follows: The comprehensive intervention effect index increment reward serves as the main signal, driving DQN to prioritize actions that bring positive OIE (Output of Entries) improvement as its core objective. The semantic interference resolution reward serves as an auxiliary signal, providing additional incentives for interference discrimination training actions based on the main signal. This prevents DQN from systematically ignoring the risk of semantic interference resolution because the short-term OIE increment from interference discrimination training may not be as significant as the introduction of new vocabulary. The forgetting cost penalty serves as a constraint signal, imposing penalties on states where the urgency of forgetting exceeds a threshold. This forces DQN to reserve review time for high-forgetting-risk vocabulary when planning the teaching sequence, preventing the agent from falling into a short-sighted strategy of "continuously introducing new vocabulary while neglecting the consolidation of already learned vocabulary." The three components are weighted by their respective balance coefficients, which can be dynamically adjusted according to the different intervention stages of children: in the initial vocabulary accumulation stage of intervention, the weight of the main signal can be increased and the weight of the constraint signal can be decreased; in the consolidation and stabilization stage, the weight of the constraint signal can be increased and the priority of forgetting review in scheduling can be increased; when solving specific vocabulary confusion problems, the weight of the auxiliary signal can be temporarily increased to drive DQN to centrally arrange differentiation training actions.

[0138] Through the above methods, this invention achieves the following technical effects. First, GAT global state compression completely decouples the state dimension of DQN from the total number of words. In actual intervention scenarios with 13 major categories and hundreds of word nodes, the state vector dimension remains controllable (fixed global embedding dimension plus local state vector dimension). The training sample efficiency and generalization ability of DQN are not affected by the expansion of the word size, and the system can support dynamic expansion of the vocabulary without changing the network structure. Second, the three-component collaborative reward function enables DQN to achieve a dynamic balance among the three objectives of rapid word acquisition, semantic interference resolution, and long-term consolidation of learned words, overcoming the inherent limitation of existing rule-based recommendation systems that cannot handle multi-objective optimization due to their single objective. Third, the reinforcement learning framework enables the scheduling strategy to automatically evolve with the continuous progress of children's cognitive development—as children's overall OIE improves, the forgetting rate decreases, and the semantic confusion interference weights converge, the distribution of the state signals received by DQN changes, and the strategy automatically adjusts to a more efficient teaching sequence without manual intervention to modify the scheduling rules, achieving true full-lifecycle adaptive path generation.

[0139] Example 10

[0140] like Figure 6 As shown, based on Example 9, the extended reward function comprises:

[0141] Step A: After each teaching action is performed, calculate the change in the comprehensive intervention effect index of each vocabulary node, and use the weighted sum of the positive increments in the change as the incremental reward of the comprehensive intervention effect index.

[0142] In this embodiment of the invention, the incremental reward of the comprehensive intervention effect index is only weighted and summed for positive changes (i.e., vocabulary nodes whose comprehensive intervention effect index has improved), avoiding interference from negative changes caused by the natural decay of some vocabulary nodes due to forgetting. The increment of each vocabulary node is assigned different weights according to its dependency depth in the knowledge graph: the increment of vocabulary nodes located in deeper layers of the knowledge graph (with more preceding dependent nodes) is given higher weights, reflecting a higher incentive for overcoming difficult vocabulary.

[0143] Step B: Detect the decrease in semantic confusion interference weight between word node pairs associated with the interference discrimination training action after the action is performed in the current teaching cycle. When the decrease exceeds a preset change threshold, use the monotonically increasing function of the decrease as the semantic interference resolution reward.

[0144] The interference discrimination training specifically targets word node pairs whose semantic confusion interference weight exceeds a preset interference threshold. It employs comparative teaching methods (e.g., presenting a red apple and a red tomato simultaneously to guide children in distinguishing their perceptual differences). If, after training, the misjudgment frequency of the word node pair decreases, leading to a reduction in the semantic confusion interference weight after the update, it indicates that the interference discrimination training is effective, and a semantic interference resolution reward is given. The preset change threshold can be set based on the historical fluctuation range of interference weights to avoid triggering erroneous rewards due to normal statistical noise.

[0145] Step C: For vocabulary nodes whose forgetting urgency index exceeds a preset urgency threshold, the sum of the differences exceeding the preset urgency threshold is used as the forgetting cost penalty.

[0146] Furthermore, the forgetting cost penalty applies a penalty to all vocabulary nodes whose forgetting urgency index exceeds the threshold in the current state. The penalty amount is positively correlated with the degree to which the threshold is exceeded, prompting the deep Q network to actively include high-urgency vocabulary into the review queue during path planning. This avoids significant forgetting of already mastered vocabulary due to excessively long teaching intervals, thereby achieving teaching rhythm management driven by forgetting patterns.

[0147] Step D: Weight the incremental reward of the comprehensive intervention effect index with the first balance coefficient, weight the reward of semantic interference reduction with the second balance coefficient, and weight the penalty of forgetting cost with the third balance coefficient. Sum the three weighted results to obtain the reward function value of the current teaching action. Use the reward function value as the training signal to update the network parameters of the deep Q network.

[0148] It should be noted that the first, second, and third balance coefficients control the contribution ratio of the three types of reward signals to the overall reward, and can be adjusted according to the priority requirements of the specific intervention scenario: if the priority is to pursue rapid vocabulary acquisition, the first balance coefficient is increased; if the priority is to solve the vocabulary confusion problem, the second balance coefficient is increased; and if the priority is to ensure the long-term consolidation of acquired vocabulary, the third balance coefficient is increased. After accumulating sufficient training samples, the deep Q-network can implicitly learn the optimal scheduling strategy for dynamically weighing the three types of objectives.

Claims

1. A method for evaluating the effectiveness of sound-meaning-object matching intervention and for adaptive path generation, characterized in that, include: By synchronously collecting data on multiple indicators of users during the teaching process through a sensor network, five modal embedding vectors are obtained. For the five modal embedding vectors, multiple encoders are jointly trained using cyclic-global joint contrastive loss to obtain a set of multimodal feature representations aligned by cyclic-global contrastive loss; Using teaching vocabulary as nodes and sensory channel activation relationships and vocabulary logical relationships as edges, a directed heterogeneous knowledge graph is constructed. Precursor-successor forgetting propagation coefficients and semantic confusion interference weights are added to the graph edges to generate a sensory-language-semantic interference joint knowledge graph. Based on the multimodal feature representation set and the joint knowledge graph, the perceptual cognitive integrity index, language output accuracy index and generalization transfer ability index are calculated for each word node. After time decay correction and cascade forgetting propagation correction, the emotional physiological stability index is normalized and weighted to finally obtain the comprehensive intervention effect index. Based on the comprehensive intervention effect index and the joint knowledge graph, a global state embedding is aggregated through a graph attention network. A local state vector is constructed by combining the evaluation vector of the active vocabulary set, the forgetting urgency index, and the semantic confusion interference submatrix between active vocabulary. The global state embedding and the local state vector are used as state inputs to the trained deep Q network to output a personalized teaching path sequence. A decision support report is generated based on the personalized teaching path sequence and the comprehensive intervention effect index of each vocabulary node.

2. The method according to claim 1, characterized in that, By synchronously collecting data on multiple indicators of the user during the teaching process through a sensor network, five modal embedding vectors are obtained, including: Using distributed cameras and eye-tracking modules as data acquisition units, the system collects heatmaps of user gaze points, gaze duration sequences, and eye-switching frequencies when users face target objects, thereby obtaining visual behavior data. Using a pressure sensor array deployed on the physical object's interface as the data acquisition unit, the system collects the hand force trajectory and grip stability index when the user operates the object, thus obtaining tactile interaction data. Using a directional microphone array as the acquisition unit, the text of the voice content output by the user after responding to the voice command, the feedback delay time, and the phonetic accuracy score are collected to obtain voice feedback data. Wearable inertial sensors are used as data acquisition units to collect the user's body displacement trajectory and balance control ability index during teaching activities, thereby obtaining body motion data; Using a wristband-type electrodermal sensor as the data acquisition unit, the sequence of changes in the user's skin conductance response during the teaching process was collected to obtain emotional physiological data. Using the teaching timestamp as the alignment benchmark, the visual behavior data, the tactile interaction data, the voice feedback data, the proprioceptive motion data, and the emotional physiological data are synchronized in time, and then input into the corresponding modal-specific encoders to extract spatiotemporal features, resulting in five modal embedding vectors.

3. The method according to claim 1, characterized in that, For the five modal embedding vectors, multiple encoders are jointly trained using a recurrent-global joint contrastive loss to obtain a set of recurrent-global contrastively aligned multimodal feature representations, including: For the five modal embedding vectors, five modal encoders are arranged in a circular topology order. The mutual information estimate of the modal embedding vectors output by adjacent encoders is calculated. The mutual information estimates of all adjacent encoder pairs are summed to obtain the cyclic loss term. For the output embedding vector of each modal encoder, the mean aggregation representation of the remaining four modal embedding vectors is calculated, the mutual information estimate between the output embedding vector and the mean aggregation representation is calculated, and the mutual information estimates of the five encoders are summed to obtain the global loss term. The cyclic loss term and the global loss term are weighted and summed with a balancing coefficient to obtain the joint contrastive loss. The five modal encoders are then jointly trained using the joint contrastive loss as the objective function to obtain a set of multimodal feature representations aligned with cyclic-global contrast.

4. The method according to claim 1, characterized in that, Using taught vocabulary as nodes and perceptual channel activation relationships and lexical logical relationships as edges, a directed heterogeneous knowledge graph is constructed. Precursor-successor forgetting propagation coefficients and semantic confusion interference weights are added to the graph edges to generate a joint perceptual-language-semantic interference knowledge graph, including: For the word node pairs with logically progressive relationships in the joint knowledge graph, extract the first comprehensive intervention effect index time sequence of the predecessor word node and the second comprehensive intervention effect index time sequence of the successor word node from the historical intervention data, calculate the correlation coefficient between the first comprehensive intervention effect index time sequence and the second comprehensive intervention effect index time sequence, and output the initial forgetting propagation coefficient. When the number of co-occurrence training samples of word node pairs in the historical intervention data is less than a preset threshold, a monotonically increasing function of the difference in teaching difficulty between the predecessor word node and the successor word node is used as the prior initial value to replace the initial forgetting propagation coefficient. Starting with the initial forgetting propagation coefficient or the prior initial value, the predecessor-successor forgetting propagation coefficient is updated on a rolling basis with the historical intervention data added after each teaching cycle ends. The updated predecessor-successor forgetting propagation coefficient is then attached to the corresponding logical progression edge in the joint knowledge graph to generate a joint knowledge graph of perception-language-semantic interference containing the forgetting propagation coefficient.

5. The method according to claim 1, characterized in that, Using taught vocabulary as nodes and perceptual channel activation relationships and lexical logical relationships as edges, a directed heterogeneous knowledge graph is constructed. Precursor-successor forgetting propagation coefficients and semantic confusion interference weights are added to the graph edges to generate a joint perceptual-language-semantic interference knowledge graph, including: For any pair of word nodes in the joint knowledge graph, calculate the set intersection-union ratio similarity of the sets of sensory channels associated with the pair of word nodes to obtain the sensory channel overlap. Extract the semantic embedding vector of any word node, calculate the normalized vector inner product similarity to obtain the semantic embedding similarity; perform a weighted calculation on the perceptual channel overlap and the semantic embedding similarity to obtain a prior estimate of the semantic confusion interference weight; The cumulative number of times users misclassified the word node pair in the historical teaching records is counted. When the cumulative number reaches the preset sample size threshold, the hot start phase is entered. The historical misclassification frequency and the semantic confusion interference weight prior estimate are fused using Bayesian posterior fusion to obtain the semantic confusion interference weight. The semantic confusion interference weights are appended to the corresponding word node pairs in the joint knowledge graph in the form of undirected interference edges to obtain the perception-language-semantic interference joint knowledge graph containing semantic confusion interference weights.

6. The method according to claim 1, characterized in that, Based on the multimodal feature representation set and the joint knowledge graph, the perceptual-cognitive integrity index, language output accuracy index, and generalization transfer ability index are calculated for each lexical node. After time-series decay correction and cascade forgetting propagation correction, the emotional physiological stability index is normalized and weighted to finally obtain the comprehensive intervention effect index, including: For each word node, the user's real-time activation status on each sensory channel associated with each word node is statistically analyzed, the activation completeness rate is calculated, and the real-time sensory cognitive completeness is obtained. Extract the historical perceptual cognitive completeness time series of the word node, fit the user's personalized forgetting rate parameter for the word through maximum likelihood estimation, and initialize it with the average forgetting rate parameter of the user group when the historical training samples of the word are insufficient; use the personalized forgetting rate parameter and the time interval from the last teaching as input, apply exponential decay correction to the immediate perceptual cognitive completeness to obtain the perceptual cognitive completeness after time decay. For each predecessor word node of the word node in the joint knowledge graph, the difference between the historical stable reference value of the predecessor word node and the perceptual cognitive integrity after the current temporal decay is calculated. When the difference is positive, the difference is weighted by the predecessor-successor forgetting propagation coefficient. The weighted difference of each predecessor word node is summed to obtain the cascade decay correction amount. The perceptual cognitive integrity after the temporal decay is subtracted from the cascade decay correction amount and then mapped to the [0,1] interval by the normalization function to obtain the comprehensive intervention effect index.

7. The method according to claim 1, characterized in that, Based on the multimodal feature representation set and the joint knowledge graph, the perceptual-cognitive integrity index, language output accuracy index, and generalization transfer ability index are calculated for each lexical node. After time-series decay correction and cascade forgetting propagation correction, the emotional physiological stability index is normalized and weighted to finally obtain the comprehensive intervention effect index, including: For each vocabulary node, by changing the color, size, or scene of the target object in the teaching scenario, the number of times the user correctly identifies the object and the total number of tests are collected in the changed scene, the correct recognition rate is calculated, and the instant generalization transfer capability is obtained. Retrieve the set of word nodes in the joint knowledge graph that have undirected interference edges with each of the word nodes and whose semantic confusion interference weight exceeds a preset interference threshold, and determine whether the word nodes in the set are in the same active training state of the same teaching cycle as the current word node. When there are interfering word nodes that are in active training at the same time as the current word node, the semantic confusion interference weight is used as the discount coefficient to apply interference discount correction to the instantaneous generalization transfer ability. The discount coefficient is directly assigned by the semantic confusion interference weight. The corrected result is mapped to the [0,1] interval through a normalization function to obtain the comprehensive intervention effect index.

8. The method according to claim 1, characterized in that, After obtaining the comprehensive intervention effectiveness index, it also includes: Obtain the missing channel marker vector to determine the set of sensor channels with missing data in the current teaching evaluation cycle; For each missing channel in the set of sensor channels with missing data, the output embedding vector of the corresponding modal encoder after the recurrent-global joint contrastive training is extracted. The output embedding vector is used as the cross-modal implicit representation of the missing channel to replace the feature contribution corresponding to the original acquired data of the missing channel. Based on the channel features after the cross-modal implicit representation replacement, the perceptual cognitive integrity index, the language output accuracy index, and the generalization transfer ability index are used in the calculation. The calculation result of the comprehensive intervention effect index remains effective as long as the number of missing channels does not exceed the preset maximum number of missing channels.

9. The method according to claim 1, characterized in that, Based on the comprehensive intervention effect index and the joint knowledge graph, a global state embedding is aggregated through a graph attention network. This is combined with the evaluation vector of the active vocabulary set, the forgetting urgency index, and the semantic confusion interference submatrix between active vocabulary words to construct a local state vector. The global state embedding and the local state vector are then used as state inputs to a trained deep Q-network, outputting a personalized teaching path sequence, including: The comprehensive intervention effect index, perceptual cognition integrity index, language output accuracy index, and generalization transfer ability index of all vocabulary nodes are used as node features. The topology of the joint knowledge graph is used as the graph structure. The graph attention network is input and the neighbor node features are aggregated layer by layer through the multi-head attention mechanism to obtain a fixed-dimensional global state embedding. The dimension of the global state embedding is independent of the total number of vocabulary nodes. The set of active vocabulary nodes in the current teaching cycle is determined, the comprehensive intervention effect index of each active vocabulary node is extracted, the forgetting urgency index is constructed by multiplying the personalized forgetting rate parameter with the time interval since the last teaching, and the semantic confusion interference sub-matrix between active vocabulary nodes is formed from the semantic confusion interference weight between each pair of nodes in the set of active vocabulary nodes. The comprehensive intervention effect index, the forgetting urgency index, and the semantic confusion interference submatrix among active words are concatenated to obtain a local state vector. The global state embedding is concatenated with the local state vector and input into the trained deep Q-network to output a personalized teaching path sequence.

10. The method according to claim 9, characterized in that, The method further includes: After each teaching action is performed, the change in the comprehensive intervention effect index of each vocabulary node is calculated, and the weighted sum of the positive increments in the change is used as the incremental reward of the comprehensive intervention effect index. The decrease in semantic confusion interference weight between word node pairs associated with the interference discrimination training action is detected after the action is performed in the current teaching cycle. When the decrease exceeds a preset change threshold, the monotonically increasing function of the decrease is used as the semantic interference elimination reward. For vocabulary nodes whose forgetting urgency index exceeds a preset urgency threshold, the sum of the differences exceeding the preset urgency threshold is used as the forgetting cost penalty. The incremental reward of the comprehensive intervention effect index is weighted by a first balance coefficient, the reward of semantic interference reduction is weighted by a second balance coefficient, and the penalty of forgetting cost is weighted by a third balance coefficient. The three weighted results are summed to obtain the reward function value of the current teaching action. The reward function value is used as the training signal to update the network parameters of the deep Q network.

Images