A dynamic identification method for multi-modal learning states in an educational robot

Abstract

The application discloses a kind of dynamic identification methods of multi-modal learning state in educational robot, it is related to the field of intelligent education, including: selecting N in target learning task History student and M state dimensions thereof, obtaining state vector of state dimension, carrying out unsupervised clustering, obtain K state cluster, based on K state cluster and its corresponding task score, construct the state cluster sequence of M state dimensions, in the state cluster sequence of M state dimensions, determine the dimension innate weight of each state cluster in the same cluster sequence number, obtain the state vector of target student in M state dimensions, the dimension innate weight of target student in each state dimension is weighted summation with its corresponding task mastery degree, determine the individualized mastery degree of target student;The application effectively overcomes the rigidity of learning state evaluation, improves the adaptability of evaluation result to individual difference, significantly improves the recognition ability of educational robot to individual learning state evolution process.

Description

Technical Field

[0001] This invention relates to the field of intelligent education, specifically to a dynamic recognition method for multimodal learning states in educational robots. Background Technology

[0002] In intelligent education systems, accurately identifying students' learning status is a prerequisite for personalized teaching intervention. Current technologies often rely on educational robots to collect multimodal data (such as cameras capturing facial expressions, microphones recording voice intonation, and system logs recording operational behaviors) to generate a standardized "learning mastery" score, used to assess students' current knowledge acquisition level.

[0003] However, these methods generally employ a "single-point mapping" mechanism, which matches a learner's performance across all dimensions to a uniform level of mastery. This approach implicitly assumes that learners' development across different learning dimensions is synchronous, but this assumption is severely inconsistent with the actual learning process. In reality, learners' progress across different learning dimensions is often asynchronous and uneven. For example:

[0004] Effortful learners: Some learners, despite their high level of concentration (high engagement) and frequent questioning (active participation), are slow to react and make frequent mistakes (high cognitive load), indicating that they are trying hard but have difficulty understanding.

[0005] Introverted learners: Some learners appear calm and have little movement (low physiological activity), but can answer quickly and accurately (high cognitive efficiency), and belong to the "introverted learners".

[0006] False participants: Some participants are highly emotional and physically active (high physiological activity), but their actions are disorganized and they frequently regress (low cognitive control), which is actually "false participation".

[0007] The lack of asynchronous matching capability for multi-dimensional states makes it impossible to identify the complex learning patterns mentioned above, making it difficult to reflect the true multi-dimensional state distribution of individuals, which in turn affects the formulation of teaching strategies. Summary of the Invention

[0008] To address the shortcomings of existing technologies, this invention provides a dynamic recognition method for multimodal learning states in educational robots, which solves the technical problems mentioned in the background by introducing dimensional talent weights and personalized mastery.

[0009] To achieve the above objectives, the present invention provides the following technical solution:

[0010] A dynamic recognition method for multimodal learning states in an educational robot includes the following steps:

[0011] S1. Select N historical learners and their M state dimensions in the target learning task;

[0012] S2. Obtain the state vectors of N historical students in M ​​state dimensions;

[0013] S3. Perform unsupervised clustering on N state vectors along any state dimension to obtain K state clusters;

[0014] S4. Anchor the corresponding historical students within K state clusters and obtain their task scores at the end of the target learning task.

[0015] S5. Determine the task mastery of K state clusters based on the task score, and construct a state cluster sequence on M state dimensions based on the task mastery.

[0016] S6. In a sequence of state clusters with M state dimensions, determine the dimensional talent weight of each state cluster in the same cluster index.

[0017] S7. Obtain the state vectors of the target student in M ​​state dimensions, and find the dimensional talent weights of the target student based on the state vectors.

[0018] S8. Weight the dimensional talent weights of the target learner in each state dimension and their corresponding task mastery, and sum them to determine the individual mastery of the target learner.

[0019] In some specific embodiments, unsupervised clustering is performed on N state vectors, including:

[0020] S3-1. In any N state vectors of any state dimension, randomly select K state vectors as the initial cluster centers;

[0021] S3-2. For each state vector, calculate its Euclidean distance to the K initial cluster centers, and assign it to the initial cluster center with the smallest Euclidean distance.

[0022] S3-3. Traverse all state vectors until each state vector is assigned to the initial cluster center with the smallest Euclidean distance, forming K initial state clusters;

[0023] S3-4. Based on all state vectors within each initial state cluster, calculate their mean vector as the cluster center for the next round.

[0024] S3-5. Calculate the displacement distance between the next cluster center and the previous cluster center;

[0025] S3-6. If there is a displacement distance greater than the preset threshold, then all state vectors are iteratively allocated based on the cluster center of the next round.

[0026] S3-7. Repeat the iterative allocation until all displacement distances are less than a preset threshold, resulting in K state clusters;

[0027] S3-8. Perform the iterative allocation on the state vectors of N historical students in M ​​state dimensions respectively to obtain K state clusters in M ​​state dimensions respectively.

[0028] In some specific embodiments, a sequence of state clusters is constructed across M state dimensions, including:

[0029] S5-1. In the K state clusters, calculate the mean and variance of the task score for each state cluster.

[0030] S5-2. Calculate the task mastery of each state cluster based on the mean and variance of task scores.

[0031] S5-3. Sort the task mastery of each state cluster from high to low, and generate K monotonically increasing cluster numbers;

[0032] S5-4. Based on the K cluster numbers, sort the K state clusters in ascending order until a state cluster sequence with M state dimensions is generated.

[0033] In some specific embodiments, determining the dimensional talent weights of each state cluster within the same cluster index includes:

[0034] S6-1. In a sequence of state clusters with M state dimensions, pair up state clusters with the same cluster number to generate K state cluster tuples.

[0035] S6-2. Calculate the cluster-level modulus and intra-cluster characteristic divergence of the M state clusters in the state cluster tuple;

[0036] S6-3. Based on the cluster-level modulus and intra-cluster feature divergence, calculate the dimensional state strength of the M state clusters in the state cluster tuple.

[0037] S6-4. Normalize the dimensional state strengths of the M state clusters in the state cluster tuple to obtain the dimensional talent weights of each state cluster in the same cluster index.

[0038] In some specific embodiments, the calculation steps for the cluster-level modulus include:

[0039] S6-2-A1. In any state cluster of the same state dimension, select all state vectors within the cluster;

[0040] S6-2-A2: Determine the cluster center vector based on all the state vectors within the cluster;

[0041] S6-2-A3. Calculate the magnitude of the cluster center vector and define it as the cluster-level magnitude.

[0042] In some specific embodiments, the calculation steps of the intra-cluster feature divergence include:

[0043] S6-2-B1. In any state cluster of the same state dimension, select all state vectors and their cluster center vectors within the cluster.

[0044] S6-2-B2, Calculate the squared Euclidean distance between each state vector and the cluster center vector;

[0045] S6-2-B3. Take the mean of the squared Euclidean distances corresponding to all state vectors to obtain the intra-cluster feature divergence.

[0046] In some specific embodiments, the dimensional talent weights of the target learner in each state dimension are found, including:

[0047] S7-1. Based on the state vectors of the M state dimensions, independently match the most similar state cluster in each state dimension in the corresponding state cluster sequence.

[0048] S7-2. For any state dimension, extract the cluster number of the most similar state cluster on the state dimension;

[0049] S7-3. Based on the cluster number of the most similar state cluster in the state dimension, find the dimension talent weight of the target student in the corresponding state cluster sequence.

[0050] In some specific embodiments, the most similar state clusters are independently matched across each state dimension, including:

[0051] S7-1-1. Determine the target cluster sequence from the state cluster sequence with M state dimensions;

[0052] S7-1-2. Extract the cluster center vectors of K state clusters in the target cluster sequence;

[0053] S7-1-3. Based on the state dimension corresponding to the target cluster sequence, obtain the state vector of the target student in the corresponding state dimension;

[0054] S7-1-4. Calculate the similarity between the state vector and the K cluster center vectors, and select the state cluster corresponding to the maximum similarity as the most similar state cluster in the target cluster sequence;

[0055] S7-1-5. Select the most similar state cluster for the state cluster sequence of M state dimensions in turn until the most similar state cluster of the target student in the M state dimensions is obtained.

[0056] This invention provides a dynamic recognition method for multimodal learning states in educational robots, which has the following beneficial effects:

[0057] This invention achieves horizontal comparability of different modalities in the learning process by independently constructing ordered sequences of state clusters across M state dimensions and aligning the cluster numbers in each sequence based on task mastery. For each task mastery level (i.e., the same cluster number), a corresponding dimensional talent weight is calculated and generated based on the intensity of historical learners' state behaviors at that level and group consistency. This weight reflects the relative contribution of each state dimension to the learning state assessment at different mastery levels, thereby dynamically adjusting its influence in the assessment based on the actual performance value of each dimension in historical data. Compared to the "one-size-fits-all" approach of using fixed weights or overall mapping in traditional methods, this effectively overcomes the rigidity of learning state assessment and improves the adaptability of assessment results to individual differences.

[0058] Furthermore, this invention supports target learners in independently matching their most similar historical state clusters across multiple state dimensions. It allows each of their M-dimensional state vectors to find the dimensional talent weights under different cluster indices, and then performs a weighted sum based on the task mastery of the corresponding cluster indices, ultimately generating a personalized mastery score reflecting their multidimensional, unbalanced development characteristics. This personalized mastery score embodies an asynchronous matching strategy, breaking through the implicit assumption in traditional methods that "all dimensions must have uniform labels." It can accurately identify typical learning patterns such as "hard-working learners," "introverted masterers," and "superficially active participants," significantly improving the educational robot's ability to recognize the evolution of individual learning states. Attached Figure Description

[0059] Figure 1 This is a flowchart illustrating a dynamic recognition method for multimodal learning states in an educational robot according to the present invention.

[0060] Figure 2 This is a schematic diagram of the process for generating the state cluster sequence described in this invention;

[0061] Figure 3 This is a schematic diagram of an embodiment of the dimensional talent weighting described in this invention;

[0062] Figure 4 This is a schematic diagram illustrating the calculation process of intra-cluster characteristic divergence as described in this invention;

[0063] Figure 5 This is a schematic diagram of another embodiment of the dimensional talent weighting described in this invention;

[0064] Figure 6 This is a schematic diagram of the process for selecting the most similar state cluster as described in this invention. Detailed Implementation

[0065] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0066] Example 1: Please refer to Figure 1 This invention provides a dynamic recognition method for multimodal learning states in educational robots, comprising the following application steps:

[0067] S1. Select N historical learners and their M state dimensions in the target learning task;

[0068] Specifically, the target learning task represents a specific learning unit supported by the educational robot, such as a knowledge point exercise, a post-class quiz, or a certain stage of an interactive teaching module.

[0069] S2. Obtain the state vector of the historical student in M ​​state dimensions;

[0070] In this embodiment, the state vector includes a physiological activity vector, a cognitive load vector, and a learning engagement vector, which are used to characterize the learner's explicit physiological state, cognitive processing intensity, and degree of active participation, respectively.

[0071] Among them, the physiological activity vector is extracted based on multimodal signals such as the learner's facial expressions, voice tone and body movements; the cognitive load vector is constructed based on log data such as reaction delay, error rate and look-back behavior during the task interaction process; and the learning engagement vector is generated based on participation indicators such as operation frequency, questioning behavior and duration of continuous interaction.

[0072] All state vectors are processed through feature extraction and standardization, and then associated with student ID as a unique identifier to form a mapping dataset that supports cross-querying of state and score.

[0073] In one alternative implementation, the construction of the physiological activity vector includes the following steps:

[0074] ① Facial activation intensity: The video stream captured by the educational robot's camera was analyzed frame by frame using a facial action unit (AU) analysis tool (such as OpenFace 2.0). Core AUs related to emotional expression were selected (such as AU6: cheek lift, AU12: corner of mouth upturn, AU15: corner of mouth pull, AU17: chin lift, AU20: lip stretch, AU25: mouth open, AU45: blink). The average value of the activation intensity (0~5 levels) of each AU was calculated within a 30-second time window and used as the facial activation intensity for that time period.

[0075] ② Pitch fluctuation rate: The speech signal was collected by the built-in microphone of the educational robot (sampling rate 16kHz), the fundamental frequency (F0) was estimated by autocorrelation method, and silent segments were removed by speech activity detection (VAD). The standard deviation of the fundamental frequency per second was calculated, and the mean value within a 30-second sliding window was taken as the pitch fluctuation rate.

[0076] ③ Limb movement frequency: Based on the coordinates of key points of the human body obtained by RGB or depth cameras, the M-dimensional displacement sequence of key parts such as the head and hands is tracked, and the number of times the displacement amplitude exceeds a preset threshold (such as 5cN) per unit time is counted as the limb movement frequency.

[0077] In one alternative implementation, the construction of the cognitive load vector includes the following steps:

[0078] ① Average response delay: Extract the operation records of the learner during the task interaction process from the log of the educational robot system, and calculate the average time interval from the appearance of the prompt to the user's response (click, input, etc.). The longer the delay, the higher the cognitive load.

[0079] ② Error density: The number of incorrect operations per unit time (such as incorrect selection of options, input errors). A high error density reflects cognitive overload.

[0080] ③ Revisit rate: This is the percentage of operations that require returning to the previous page or step out of the total number of operations. Frequent revisits indicate difficulties in information integration.

[0081] In one alternative implementation, the construction of the learning input vector includes the following steps:

[0082] ① Frequency of interaction: The number of times a user effectively interacts with the educational robot interface within a unit of time (such as clicking buttons, dragging elements, submitting answers). The higher the frequency, the stronger the participation.

[0083] ② Number of times students actively ask questions: Record the number of times students actively ask questions without system prompts (such as voice or text input "How is this calculated?"), reflecting their willingness to learn;

[0084] ③ Continuous interaction duration: Calculate the cumulative time of continuous and effective interaction during a student's learning session, excluding interruptions of more than 30 seconds without operation, to measure the level of focus and engagement.

[0085] The above features are standardized by Z-score (i.e., the mean of all historical samples is subtracted and divided by its standard deviation), and then concatenated in a preset order to form the state vector of the corresponding dimension.

[0086] S3. Perform unsupervised clustering on N state vectors along any state dimension to obtain K state clusters;

[0087] S4. Anchor the corresponding historical students within K state clusters and obtain their task scores at the end of the target learning task.

[0088] Furthermore, the task score represents the test score of the historical learner at the end of the target learning task, which is organized by the educational robot. The test covers the learning content of the target learning task, and the test score is normalized and expressed as a value in the range [0,1]. The higher the value, the higher the learner's mastery of the learning content in the target learning task.

[0089] Specifically, the task scores of historical trainees are also associated with trainee IDs and stored in the mapping dataset.

[0090] S5. Determine the task mastery of K state clusters based on the task score, and construct a state cluster sequence on M state dimensions based on the task mastery.

[0091] Specifically, a state cluster sequence refers to an ordered structure formed by arranging K state clusters in a certain state dimension according to their cluster numbers.

[0092] S6. In a sequence of state clusters with M state dimensions, determine the dimensional talent weight of each state cluster in the same cluster index.

[0093] Specifically, the dimensional talent weights in the same cluster are used to characterize the relative contributions of the M state dimensions of physiological activity, cognitive load, and learning engagement to the learning state assessment under the same task mastery level, so that the influence of each dimension can be differentiated and weighted according to the typicality of historical state patterns.

[0094] S7. Obtain the state vectors of the target student in M ​​state dimensions, and find the dimensional talent weights of the target student based on the state vectors; wherein, the dimensional talent weights of each state dimension correspond to the same cluster number or different cluster numbers.

[0095] Unlike the dimensional talent weights within the same cluster number, the dimensional talent weights of a target learner are likely not in the same cluster number. This is because the target learner's state patterns in the M dimensions of physiological activity, cognitive load, and learning engagement may correspond to different levels of mastery, resulting in them being independently matched to state clusters with different cluster numbers for each dimension. Consequently, the retrieved dimensional talent weights are also distributed across different cluster numbers, reflecting the personalization of their multimodal learning state.

[0096] S8. Weight the dimensional talent weights of the target learner in each state dimension and their corresponding task mastery, and sum them to determine the individual mastery of the target learner.

[0097] This embodiment constructs an ordered state cluster sequence based on the historical multimodal states and task scores of trainees, completes independent state matching and dimensional talent weight lookup for target trainees in M ​​state dimensions, and generates personalized mastery through weighted summation, realizing cross-dimensional dynamic mapping from explicit behavior to intrinsic learning level, enabling the educational robot to dynamically identify individual learning states.

[0098] Example 2: See Figures 2 to 6 The technical solution of this embodiment 2 differs from that of embodiment 1 in that it discloses the execution sub-steps of each application step.

[0099] The execution sub-steps of the application step S3 include:

[0100] S3-1. In any N state vectors of any state dimension, randomly select K state vectors as the initial cluster centers;

[0101] S3-2. For each state vector, calculate its Euclidean distance to the K initial cluster centers, and assign it to the initial cluster center with the smallest Euclidean distance.

[0102] S3-3. Traverse all state vectors until each state vector is assigned to the initial cluster center with the smallest Euclidean distance, forming K initial state clusters;

[0103] S3-4. Based on all state vectors within each initial state cluster, calculate their mean vector as the cluster center for the next round.

[0104] S3-5. Calculate the displacement distance between the next cluster center and the previous cluster center;

[0105] S3-6. If there is a displacement distance greater than the preset threshold, then all state vectors are iteratively allocated based on the cluster center of the next round.

[0106] S3-7. Repeat the iterative allocation until all displacement distances are less than a preset threshold, resulting in K state clusters;

[0107] S3-8. Perform the iterative allocation on the state vectors of N historical students in M ​​state dimensions respectively to obtain K state clusters in M ​​state dimensions respectively.

[0108] This embodiment achieves the classification of historical learners' multimodal learning states by independently performing iterative clustering on each state dimension, so that learners with similar physiological, cognitive or engagement patterns are grouped into the same category, providing support for cluster-based task score statistics and cross-dimensional state search.

[0109] Furthermore, the execution sub-steps of the application step S5 are as follows:

[0110] S5-1. In the K state clusters, calculate the mean and variance of the task score for each state cluster.

[0111] Specifically, for any state cluster, the student IDs corresponding to all state vectors contained within it are found through the mapping dataset, and the task score of each student is obtained based on the student ID. Then, the mean of the task scores of each student is calculated, which yields the mean task score of the state cluster. Similarly, the variance of the task score of the state cluster is obtained by calculating the squared standard deviation of the task scores of each student. Finally, by traversing the K state clusters with M state dimensions, the mean and variance of the task scores of each state cluster are obtained.

[0112] S5-2. Calculate the task mastery of each state cluster based on the mean and variance of task scores.

[0113] The formula for calculating the task mastery is:

[0114] ;

[0115] in, It represents the task mastery degree of a state cluster, which reflects the mastery level and consistency of the student group in the corresponding state cluster; This represents the average score of historical trainees within a state cluster, and its value ranges from [0,1] after normalization. It represents the variance of historical student task scores within a state cluster, reflecting the consistency of mastery levels; This is the dynamic adjustment coefficient for the variance of the task score, and its value is determined by the ratio of the mean to the variance of the current state cluster. ,and Represents extremely small positive numbers (such as 10). -6 This is used to prevent the denominator from being zero and to ensure calculation stability.

[0116] Specifically, the dynamic adjustment coefficient The moderating coefficient is directly proportional to the variance and inversely proportional to the mean. This means that when the variance of a student group's scores is large, the moderating coefficient will increase, thus imposing a stronger penalty on the variance. Conversely, when the mean is high, the moderating coefficient will decrease, and the penalty on the variance will be weakened accordingly. The final task mastery takes into account both the average score and the score stability of the student group. That is, a high mean and low variance will result in a higher task mastery, indicating that the student group's task mastery is high and stable. Conversely, a low mean and high variance will result in a lower task mastery, indicating that the student group's mastery is low or unstable.

[0117] S5-3. Sort the task mastery of each state cluster from high to low, and generate K monotonically increasing cluster numbers;

[0118] That is, the state cluster with the highest task mastery is assigned the sequence number 1, the second highest is assigned the sequence number 2, and so on, with the lowest being assigned the sequence number K.

[0119] S5-4. Based on the K cluster numbers, sort the K state clusters in ascending order until a state cluster sequence with M state dimensions is generated.

[0120] This embodiment calculates the mean and variance of the task scores for each state cluster and generates the task mastery by combining a dynamic adjustment coefficient. This enables the state clusters to be sorted in a comprehensive manner based on their mastery level and consistency. State clusters with similar task score distributions are organized into an ordered sequence, so that the same cluster number corresponds to a similar level of mastery across different state dimensions.

[0121] Furthermore, the execution sub-steps of the application step S6 are as follows:

[0122] S6-1. In a sequence of state clusters with M state dimensions, pair up state clusters with the same cluster number to generate K state cluster tuples.

[0123] S6-2. Calculate the cluster-level modulus and intra-cluster characteristic divergence of the M state clusters in the state cluster tuple;

[0124] The calculation process for the cluster-level modulus is as follows;

[0125] S6-2-A1. In any state cluster of the same state dimension, select all state vectors within the cluster;

[0126] S6-2-A2: Determine the cluster center vector based on all the state vectors within the cluster;

[0127] That is, the arithmetic mean of each component of all state vectors in the cluster is calculated, and the arithmetic mean of each component is concatenated in a preset order to construct a vector representing the center position of the cluster in the feature space. This cluster center vector may correspond to the state vector of a real student, or it may be a virtual center point that does not directly correspond to any individual.

[0128] S6-2-A3. Calculate the magnitude of the cluster center vector and define it as the cluster-level magnitude.

[0129] Specifically, the sum of squares and the square root of the eigenvalues ​​of the cluster center vector at each component are performed. The result is the cluster-level modulus of the state cluster, which is used to characterize the intensity of the state behavior of the state cluster in the multidimensional state space.

[0130] The calculation process for the intra-cluster feature divergence is as follows:

[0131] S6-2-B1. In any state cluster of the same state dimension, select all state vectors and their cluster center vectors within the cluster.

[0132] S6-2-B2, Calculate the squared Euclidean distance between each state vector and the cluster center vector;

[0133] S6-2-B3. Take the mean of the squared Euclidean distances corresponding to all state vectors to obtain the intra-cluster feature divergence.

[0134] The cluster feature divergence is used to reflect the inconsistency of historical students in state behavior within the state cluster. The smaller the value, the more concentrated the state within the cluster and the more stable the state pattern. The larger the value, the greater the difference between historical students and the more inconsistent the state.

[0135] S6-3. Based on the cluster-level modulus and intra-cluster feature divergence, calculate the dimensional state strength of the M state clusters in the state cluster tuple.

[0136] The formula for calculating the dimensional state strength is:

[0137] ;

[0138] in, The dimensionality of a state cluster represents its state strength. The cluster-level modulus of the state cluster (reflecting the intensity of state behavior). The intra-cluster characteristic divergence of a state cluster (reflecting the degree of internal dispersion). This is the dynamic adjustment coefficient for the intra-cluster feature divergence. As can be seen from the calculation logic of task mastery, the calculation method for dimensional state strength in this embodiment is also based on the "primary item - dynamic penalty item" structure. However, task mastery is calculated based on the historical learner's task score at the end of the target task, and it is a result-oriented indicator; while dimensional state strength is evaluated based on the intensity of the historical learner's state behavior and state consistency during the learning process. Therefore, from a process-oriented perspective, it better reflects the learner's true state performance at different learning stages.

[0139] Specifically, by comprehensively considering the cluster-level magnitude and intra-cluster feature divergence of state clusters, the representativeness and consistency of each state cluster in the target learning task can be evaluated more comprehensively. A high magnitude indicates that the state behavior of the state cluster is strong, while low variance indicates that the member states within the state cluster are relatively consistent, thus resulting in high dimensional state strength.

[0140] In summary, dimensional state strength, as a process-oriented indicator, characterizes the representativeness and consistency of state patterns in a certain state dimension under the same mastery level. It is jointly determined by cluster-level modulus and intra-cluster feature divergence, providing a more comprehensive assessment of the learning state of educational robots.

[0141] S6-4. Normalize the dimensional state strengths of the M state clusters in the state cluster tuple to obtain the dimensional talent weights of each state cluster in the same cluster index.

[0142] Specifically, the normalization involves summing the dimensional state strengths corresponding to each state cluster with the same cluster index, and then calculating the ratio of each dimensional state strength to the summation, so that the sum of the dimensional talent weights is 1, thereby reflecting the relative weights of each dimensional state under the same task mastery.

[0143] In this embodiment, state cluster tuples are formed by pairing state clusters with the same cluster number, and the dimensional talent weights of each state cluster in the same cluster number are obtained by normalization processing. This makes the relative importance of the M dimensions of physiological activity, cognitive load and learning input determined by the consistency and intensity of their historical states.

[0144] The execution sub-steps of the application step S7 include:

[0145] S7-1. Based on the state vectors of the M state dimensions, independently match the most similar state cluster in each state dimension in the corresponding state cluster sequence.

[0146] S7-2. For any state dimension, extract the cluster number of the most similar state cluster on the state dimension;

[0147] S7-3. Based on the cluster number of the most similar state cluster in the state dimension, find the dimension talent weight of the target student in the corresponding state cluster sequence.

[0148] It should be noted that the dimensional talent weights are pre-constructed based on historical student group behavior patterns, and their values ​​are jointly determined by the state dimension and its matched cluster number. For the target student, this weight is not fixedly assigned, but dynamically searched based on the cluster number actually matched in each dimension: when the target student matches the i-th state cluster in the physiological activity dimension and the j-th state cluster in the cognitive load dimension, the pre-stored weights under the corresponding "dimension-number" combination are extracted respectively. The final weight combination is determined independently by the matching results of each of the M dimensions, reflecting the personalization of learning state assessment.

[0149] This embodiment achieves cross-dimensional talent weight lookup based on the actual state pattern by independently matching the most similar state clusters of the target learner in each state dimension and extracting the corresponding cluster number. This allows the dimensional talent weights of the target learner to be dynamically determined according to the matching results, which can reflect their personalized state in which they may be at different levels of mastery in the physiological, cognitive and investment dimensions.

[0150] The execution sub-step S7-1 includes:

[0151] S7-1-1. Determine the target cluster sequence from the state cluster sequence with M state dimensions;

[0152] S7-1-2. Extract the cluster center vectors of K state clusters in the target cluster sequence;

[0153] S7-1-3. Based on the state dimension corresponding to the target cluster sequence, obtain the state vector of the target student in the corresponding state dimension;

[0154] S7-1-4. Calculate the similarity between the state vector and the K cluster center vectors, and select the state cluster corresponding to the maximum similarity as the most similar state cluster in the target cluster sequence; specifically, the similarity is selected by the cosine similarity of the vectors or the reciprocal of the Euclidean distance.

[0155] S7-1-5. Select the most similar state cluster for the state cluster sequence of M state dimensions in turn until the most similar state cluster of the target student in the M state dimensions is obtained.

[0156] This embodiment calculates the similarity between the target student's state vector and the center vector of each cluster in the state cluster sequence of each state dimension, and selects the state cluster with the highest similarity as the matching result. This enables the location of the most similar state of the target student in each dimension, so that its multimodal state can be independently and accurately mapped to the historical cluster state based on actual observation data.

[0157] In summary, this invention acquires the multimodal state vectors of target learners during the learning process and combines the state behaviors and final task mastery of historical learner groups under the same task to construct a mapping relationship from explicit behavior to intrinsic mastery level. Based on this mapping, the educational robot can independently match the target learner's state in physiological, cognitive, and engagement dimensions and find the dimensional talent weights. Then, it generates personalized mastery through weighted summation, realizing cross-dimensional dynamic evaluation of multimodal learning states, so that the mastery judgment truly reflects the actual state evolution of individual learners during the learning process.

[0158] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; multiple units or components may be combined or integrated into another system, or some features may be omitted or not performed. Furthermore, the mutual couplings or direct couplings or communication connections shown or discussed may be indirect couplings or communication connections between apparatuses or units through some interfaces, and may be electrical, mechanical, or other forms.

[0159] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A dynamic recognition method for multimodal learning states in an educational robot, characterized in that, include: S1. Select N historical learners and their M state dimensions in the target learning task; S2. Obtain the state vectors of N historical students in M ​​state dimensions; The state vector includes a physiological activity vector, a cognitive load vector, and a learning engagement vector, which are used to characterize the learner’s explicit physiological state, cognitive processing intensity, and degree of active participation, respectively. Among them, the physiological activity vector is extracted based on multimodal signals of trainees' facial expressions, voice tone and body movements; the cognitive load vector is constructed based on log data of reaction delay, error rate and look-back behavior during task interaction; and the learning engagement vector is generated based on participation indicators such as operation frequency, questioning behavior and duration of continuous interaction. S3. Perform unsupervised clustering on N state vectors along any state dimension to obtain K state clusters; S4. Anchor the corresponding historical students within K state clusters and obtain their task scores at the end of the target learning task. S5. Determine the task mastery of K state clusters based on the task score, and construct a state cluster sequence on M state dimensions based on the task mastery. S6. In a sequence of state clusters with M state dimensions, determine the dimensional talent weight of each state cluster in the same cluster index. S7. Obtain the state vectors of the target student in M ​​state dimensions, and find the dimensional talent weights of the target student based on the state vectors. S8. Weight the dimensional talent weights of the target learner in each state dimension and their corresponding task mastery to determine the individual mastery of the target learner. The construction of a state cluster sequence across M state dimensions includes: S5-1. In the K state clusters, calculate the mean and variance of the task score for each state cluster. S5-2. Calculate the task mastery of each state cluster based on the mean and variance of task scores. The formula for calculating the task mastery is: ； in, This represents the task mastery of a state cluster. This represents the average task score of historical trainees within a state cluster, and its value ranges from the normalized value. ; This represents the variance of historical student task scores within a state cluster. This is the dynamic adjustment coefficient for the variance of the task score. Represents extremely small positive numbers; S5-3. Sort the task mastery of each state cluster from high to low, and generate K monotonically increasing cluster numbers; S5-4. Based on the K cluster numbers, sort the K state clusters in ascending order until a state cluster sequence with M state dimensions is generated. The determination of the dimensional talent weights of each state cluster within the same cluster index includes: S6-1. In a sequence of state clusters with M state dimensions, pair up state clusters with the same cluster number to generate K state cluster tuples. S6-2. Calculate the cluster-level modulus and intra-cluster characteristic divergence of the M state clusters in the state cluster tuple; S6-3. Based on the cluster-level modulus and intra-cluster feature divergence, calculate the dimensional state strength of the M state clusters in the state cluster tuple. S6-4. Normalize the dimensional state strengths of the M state clusters in the state cluster tuple to obtain the dimensional talent weights of each state cluster in the same cluster index.

2. The dynamic recognition method for multimodal learning states in an educational robot according to claim 1, characterized in that, Unsupervised clustering of N state vectors includes: S3-1. In any N state vectors of any state dimension, randomly select K state vectors as the initial cluster centers; S3-2. For each state vector, calculate its Euclidean distance to the K initial cluster centers, and assign it to the initial cluster center with the smallest Euclidean distance. S3-3. Traverse all state vectors until each state vector is assigned to the initial cluster center with the smallest Euclidean distance, forming K initial state clusters; S3-4. Based on all state vectors within each initial state cluster, calculate their mean vector as the cluster center for the next round. S3-5. Calculate the displacement distance between the next cluster center and the previous cluster center; S3-6. If there is a displacement distance greater than the preset threshold, then all state vectors are iteratively allocated based on the cluster center of the next round. S3-7. Repeat the iterative allocation until all displacement distances are less than a preset threshold, resulting in K state clusters; S3-8. Perform the iterative allocation on the state vectors of N historical students in M ​​state dimensions respectively to obtain K state clusters in M ​​state dimensions respectively.

3. The dynamic recognition method for multimodal learning states in an educational robot according to claim 2, characterized in that, The calculation steps for the cluster-level modulus include: S6-2-A1. In any state cluster of the same state dimension, select all state vectors within the cluster; S6-2-A2: Determine the cluster center vector based on all the state vectors within the cluster; S6-2-A3. Calculate the magnitude of the cluster center vector and define it as the cluster-level magnitude.

4. The dynamic recognition method for multimodal learning states in an educational robot according to claim 3, characterized in that, The steps for calculating the intra-cluster characteristic divergence include: S6-2-B1. In any state cluster of the same state dimension, select all state vectors and their cluster center vectors within the cluster. S6-2-B2, Calculate the squared Euclidean distance between each state vector and the cluster center vector; S6-2-B3. Take the mean of the squared Euclidean distances corresponding to all state vectors to obtain the intra-cluster feature divergence.

5. The dynamic recognition method for multimodal learning states in an educational robot according to claim 4, characterized in that, Find the dimensional talent weights of the target student in each state dimension, including: S7-1. Based on the state vectors of the M state dimensions, independently match the most similar state cluster in each state dimension in the corresponding state cluster sequence. S7-2. For any state dimension, extract the cluster number of the most similar state cluster on the state dimension; S7-3. Based on the cluster number of the most similar state cluster in the state dimension, find the dimension talent weight of the target student in the corresponding state cluster sequence.

6. The method for dynamic recognition of multimodal learning states in an educational robot according to claim 5, characterized in that, Independently match its most similar state clusters across each state dimension, including: S7-1-1. Determine the target cluster sequence from the state cluster sequence with M state dimensions; S7-1-2. Extract the cluster center vectors of K state clusters in the target cluster sequence; S7-1-3. Based on the state dimension corresponding to the target cluster sequence, obtain the state vector of the target student in the corresponding state dimension; S7-1-4. Calculate the similarity between the state vector and the K cluster center vectors, and select the state cluster corresponding to the maximum similarity as the most similar state cluster in the target cluster sequence; S7-1-5. Select the most similar state cluster for the state cluster sequence of M state dimensions in turn until the most similar state cluster of the target student in the M state dimensions is obtained.

Images