A human-computer collaborative personalized learning plan generation system and method thereof

Abstract

This invention relates to the field of human-computer interaction technology, and particularly to a personalized learning plan generation system and method for human-computer collaboration. The system collects multimodal raw data streams from learners; extracts various feature vectors based on the multimodal raw data streams; fuses the feature vectors to obtain a deep fused feature vector; decodes the deep fused feature vector to obtain a basic cognitive-emotional state probability distribution; based on the basic cognitive-emotional state probability distribution, and combined with facial morphological feature vectors, acoustic prosodic feature vectors, and behavioral dynamics feature vectors, performs attribution analysis to generate a real-time state analysis report containing state attribution descriptions, and dynamically adjusts the learning plan. By implementing this invention, personalized and adaptive learning plans can be provided, breaking through the limitations of traditional macro-assessment and general feedback, and supporting more precise and efficient learning intervention.

Description

Technical Field

[0001] This invention relates to the field of human-computer interaction technology, and in particular to a human-computer collaborative personalized learning plan generation system and method. Background Technology

[0002] In existing technologies, learner state analysis based on multimodal information mainly focuses on the recognition of macroscopic behaviors such as head posture and facial orientation, and the assessment of attention in a single dimension. For example, patent application CN116244474A achieves the determination of attention level by fusing facial and voice signals, but its method mainly relies on conventional feature extraction and weighted fusion. The output results are relatively macroscopic and it is difficult to reveal the specific cognitive or emotional causes behind the state, such as the inability to distinguish between conceptual confusion and distraction caused by external interference.

[0003] Such methods lack fine-grained modeling and analysis of the micro-physiological signals with clear psychological significance that accompany cognitive effort during the learning process. For example, they fail to analyze the dynamic characteristics of micro-expressions between the eyebrows and the gaze patterns that represent thinking search. Therefore, the depth of analysis of the learner's true psychological state is limited, and the feedback provided is relatively general, making it difficult to support precise adaptive learning intervention. Summary of the Invention

[0004] This invention addresses the technical problem in existing technologies that have limited depth of analysis of learners' true psychological states and provide rather general feedback, making it difficult to support precise adaptive learning intervention. It provides a human-computer collaborative personalized learning plan generation system and method to solve this problem.

[0005] The technical solution of the present invention to solve the above-mentioned technical problems is as follows: In a first aspect, the present invention provides a human-computer collaborative personalized learning plan generation system, comprising: The real-time data acquisition module is used to simultaneously acquire learners' facial video streams, audio streams, and interactive behavior log streams, and perform time alignment to obtain multimodal raw data streams; The feature vector extraction module is used to extract facial morphological feature vectors, acoustic prosodic feature vectors, and behavioral dynamics feature vectors based on the multimodal raw data stream. The facial morphological feature vectors include mathematical description vectors of gaze focus shift rate, glabellar wrinkle frequency, and mouth corner curvature change curve. The feature vector decoding module is used to fuse the facial morphological feature vector, the acoustic prosodic feature vector, and the behavioral dynamics feature vector to obtain a deep fusion feature vector; based on the deep fusion feature vector, the basic cognitive-emotional state probability distribution is decoded. The analysis report generation module is used to perform attribution analysis based on the basic cognitive-emotional state probability distribution, combined with the facial morphological feature vector, the acoustic prosodic feature vector and the behavioral dynamics feature vector, to generate a real-time state analysis report containing state attribution descriptions. The learning plan adjustment module is used to dynamically adjust the learning plan based on the real-time status analysis report.

[0006] Optionally, learners' facial video streams, audio streams, and interaction behavior log streams are simultaneously acquired and time-aligned to obtain a multimodal raw data stream, including: On the same learning device, the facial video stream is simultaneously captured based on the integrated camera, the voice audio stream is simultaneously captured based on the integrated microphone, and the interactive behavior log stream is simultaneously recorded based on the learning platform client running on the learning device. Each frame of the facial video stream, each preset duration audio segment of the voice audio stream, and each operation event of the interaction behavior log stream are respectively labeled with a high-precision timestamp based on the same system clock source; Based on the high-precision timestamp, the video frame sequence, audio segment sequence, and operation event sequence within the same time window are aligned and packaged to generate the multimodal raw data stream with a unified time reference.

[0007] Optionally, based on the multimodal raw data stream, facial morphological feature vectors are extracted, including: The learning interface of the learning terminal is divided into multiple preset functional areas; Based on the facial video stream, the gaze focus is calculated by tracking the learner's pupils and eye feature points. Based on the gaze focus shift sequence in the preset functional area, the gaze focus offset rate, which represents the attention allocation pattern, is calculated and generated. Based on the facial video stream, a region of interest is defined in the learner's glabella area. By analyzing the texture changes of the region of interest between consecutive frames, the glabella wrinkling frequency, which represents the dynamics of micro-expressions, is calculated and generated. Based on the facial video stream, the movement trajectory of key points at the corners of the learner's mouth is tracked, and a mathematical description vector of the corner curvature change curve is calculated and generated by analyzing the change pattern of the corner curvature over time.

[0008] Optional, also includes: The number of times the learner's gaze switches between different preset functional areas per unit time is counted, and combined with the duration of continuous gaze in a single functional area and the coefficient of variation of the gaze duration, a comprehensive offset rate value is generated as the gaze focus offset rate. Extract the local texture feature time series of the region of interest, perform frequency domain analysis on the local texture feature time series, extract the main frequency component, and use the main frequency component as the frequency of the brow wrinkle; The time series of the corner of the mouth curvature is subjected to first-order and second-order difference calculations to obtain the change rate sequence and acceleration sequence. Multiple mathematical descriptors, including the regularity of the number of positive and negative half-waves and the peak interval, are extracted from the change rate sequence and the acceleration sequence and combined to form the mathematical descriptive vector of the corner of the mouth curvature change curve.

[0009] Optionally, based on the multimodal raw data stream, acoustic prosodic feature vectors and behavioral dynamics feature vectors are extracted, including: The speech audio stream is extracted tones, speech rate variation coefficients, and silence interval lengths, and the three are combined to form the acoustic prosodic feature vector. The number of attempts per unit time, the frequency of undo operations, and the degree of mouse trajectory disorder are calculated from the interaction behavior log, and the three are combined to form the behavior dynamic feature vector.

[0010] Optionally, based on the deep fusion feature vector, the basic cognitive-emotional state probability distribution is decoded, including: The deep fusion feature vector is input into a pre-trained state classification model, which is built based on a deep neural network. The state classification model calculates based on the deep fusion feature vector and outputs a multidimensional probability vector; wherein each dimension of the multidimensional probability vector corresponds to a preset cognitive-emotional basic state. The cognitive-emotional baseline states include at least a state of deep focus, a state of fluent learning, a state of cognitive confusion, and a state of emotional frustration. The value of each dimension in the multidimensional probability vector represents the predicted probability of the existence of the corresponding state, and the sum of the values ​​of all dimensions is 1.

[0011] Optionally, the state classification model is built based on a deep neural network and includes: Multiple sets of sample data were collected. Each set of sample data included sample facial video streams, sample voice audio streams, sample interaction behavior log streams, and sample cognitive-emotional true state labels synchronized with the sample data and annotated by experts. The sample data is preprocessed and features are extracted to obtain a sample deep fusion feature vector; The deep neural network model is trained using the deep fusion feature vector of the sample as the training input and the corresponding cognitive-emotional real state label of the sample as the supervision signal. Finally, the state classification model that has been trained and meets the preset performance standards is obtained.

[0012] Optionally, based on the basic cognitive-emotional state probability distribution, and combined with the facial morphological feature vector, the acoustic prosodic feature vector, and the behavioral dynamics feature vector, attribution analysis is performed to generate a real-time state analysis report containing state attribution descriptions, including: From the aforementioned basic cognitive-emotional state probability distribution, the state with the highest predicted probability value is selected as the dominant state; Based on the facial morphological feature vector, the acoustic prosodic feature vector, and the behavioral dynamics feature vector, the contribution of each feature dimension to the state classification model in predicting the dominant state is calculated. Feature dimensions with a contribution rate higher than a preset threshold are selected as key evidence features; According to a preset mapping rule, the key evidence features are mapped to state attributions described in natural language; wherein, the mapping rule defines the correspondence between specific feature patterns and cognitive or emotional causes. The dominant state, the predicted probability corresponding to the dominant state, and the state attribution are integrated to generate the real-time state analysis report containing the state attribution description.

[0013] Optionally, based on the real-time status analysis report, the learning plan is dynamically adjusted, including: Analyze the real-time status analysis report and extract the dominant status and corresponding status attribution description from the real-time status analysis report; The dominant state and state attribution description are input into a preset learning planning adjustment strategy mapping library for matching; The learning planning adjustment strategy mapping library outputs at least one suitable learning intervention strategy based on the input matching conditions. Based on the aforementioned learning intervention strategy, the subsequent learning content, learning path, presentation method, and interaction rhythm in the current learning plan are adjusted in real time. The adjusted learning plan will be pushed to the learning terminal for execution in real time.

[0014] Secondly, the present invention provides a method for generating personalized learning plans through human-computer collaboration, comprising: Simultaneously collect learners' facial video streams, audio streams, and interactive behavior log streams, and perform time alignment to obtain multimodal raw data streams; Based on the multimodal raw data stream, facial morphological feature vectors, acoustic prosodic feature vectors, and behavioral dynamics feature vectors are extracted respectively. Among them, the facial morphological feature vectors include mathematical description vectors of gaze focus shift rate, glabellar wrinkle frequency, and mouth corner curvature change curve. The facial morphological feature vector, the acoustic prosodic feature vector, and the behavioral dynamics feature vector are fused to obtain a deep fusion feature vector; based on the deep fusion feature vector, the basic cognitive-emotional state probability distribution is decoded. Based on the aforementioned basic cognitive-emotional state probability distribution, and combined with the aforementioned facial morphological feature vector, the aforementioned acoustic prosodic feature vector, and the aforementioned behavioral dynamics feature vector, attribution analysis is performed to generate a real-time state analysis report containing state attribution descriptions. The learning plan is dynamically adjusted based on the real-time status analysis report.

[0015] This invention has at least the following technical effects: 1. By implementing this invention, it is possible to simultaneously collect learners' facial video streams, audio streams, and interactive behavior log streams, and perform time alignment to obtain multimodal raw data streams; ensuring the spatiotemporal consistency of multi-dimensional data, providing high-quality, synchronized raw data support for subsequent multimodal feature fusion and analysis, and avoiding analytical biases caused by data misalignment.

[0016] 2. By implementing this invention, it is possible to extract facial morphological feature vectors, acoustic prosodic feature vectors, and behavioral dynamics feature vectors based on the multimodal raw data stream. The facial morphological feature vectors include mathematical description vectors of gaze focus shift rate, brow wrinkle frequency, and mouth corner curvature change curve. This allows for the mining of key information on learners' microscopic physiological signals and behavioral patterns, breaking through the limitations of traditional macroscopic features, and providing fine-grained, psychologically meaningful feature basis for accurately analyzing psychological states.

[0017] 3. By implementing this invention, it is possible to fuse the facial morphological feature vector, the acoustic prosodic feature vector, and the behavioral dynamics feature vector to obtain a deep fusion feature vector; based on the deep fusion feature vector, decode to obtain the basic cognitive-emotional state probability distribution; integrate the advantages of multi-dimensional features to improve the comprehensiveness and accuracy of state recognition, quantify various states in the form of probability distribution, and provide a clear state orientation for subsequent attribution analysis.

[0018] 4. By implementing this invention, attribution analysis can be performed based on the basic cognitive-emotional state probability distribution, combined with the facial morphological feature vector, the acoustic prosodic feature vector, and the behavioral dynamics feature vector, generating a real-time state analysis report containing state attribution descriptions. This not only clarifies the learner's current state but also reveals the cognitive or emotional causes behind the state, solving the problem of vague traditional feedback, providing a clear basis for precise intervention, enabling real-time adaptive optimization of learning plans, and ensuring that learning arrangements fit the learner's immediate state, thereby improving learning efficiency and experience.

[0019] 5. By implementing this invention, it is possible to achieve accurate and real-time analysis and attribution of learners' cognitive-emotional states, thereby providing personalized and adaptive learning plans, breaking through the limitations of traditional macro-assessment and general feedback, and supporting precise and efficient learning intervention. Attached Figure Description

[0020] Figure 1 A schematic diagram of the structure of a human-computer collaborative personalized learning plan generation system provided by the present invention; Figure 2 This is a flowchart illustrating a human-computer collaborative personalized learning plan generation method provided by the present invention.

[0021] In the attached diagram, the components represented by each number are as follows: Real-time data acquisition module 11, feature vector extraction module 12, feature vector decoding module 13, parsing report generation module 14, learning plan adjustment module 15. Detailed Implementation

[0022] Example 1, as Figure 1 As shown, this embodiment of the invention provides a human-machine collaborative personalized learning plan generation system, including: The real-time data acquisition module 11 is used to synchronously acquire learners' facial video stream, voice audio stream, and interactive behavior log stream, and perform time alignment to obtain multimodal raw data streams; The feature vector extraction module 12 is used to extract facial morphological feature vectors, acoustic prosodic feature vectors, and behavioral dynamics feature vectors based on the multimodal raw data stream, wherein the facial morphological feature vectors include mathematical description vectors of gaze focus shift rate, brow wrinkle frequency, and mouth corner curvature change curve. The feature vector decoding module 13 is used to fuse the facial morphological feature vector, the acoustic prosodic feature vector, and the behavioral dynamics feature vector to obtain a deep fusion feature vector; based on the deep fusion feature vector, the basic cognitive-emotional state probability distribution is decoded. The report generation module 14 is used to perform attribution analysis based on the basic cognitive-emotional state probability distribution and in combination with the facial morphological feature vector, the acoustic prosodic feature vector and the behavioral dynamics feature vector, and generate a real-time state analysis report containing state attribution description. The learning plan adjustment module 15 is used to dynamically adjust the learning plan based on the real-time status analysis report.

[0023] In the real-time data acquisition module 11 of this application embodiment, the learner's facial video stream, voice audio stream, and interaction behavior log stream are simultaneously acquired and time-aligned to obtain a multimodal raw data stream, including: On the same learning device, the facial video stream is simultaneously captured based on the integrated camera, the voice audio stream is simultaneously captured based on the integrated microphone, and the interactive behavior log stream is simultaneously recorded based on the learning platform client running on the learning device. Each frame of the facial video stream, each preset duration audio segment of the voice audio stream, and each operation event of the interaction behavior log stream are respectively labeled with a high-precision timestamp based on the same system clock source; Based on the high-precision timestamp, the video frame sequence, audio segment sequence, and operation event sequence within the same time window are aligned and packaged to generate the multimodal raw data stream with a unified time reference.

[0024] In the real-time data acquisition module 11 of this application embodiment, the purpose of the above steps is to provide a multimodal raw data stream with a unified time reference for the subsequent extraction of facial morphological feature vectors, acoustic prosodic feature vectors, and behavioral dynamics feature vectors, so as to support accurate feature vector decoding and dynamic adjustment of learning planning.

[0025] To achieve the above objectives, it is first necessary to simultaneously capture the facial video stream using an integrated camera and the audio stream using an integrated microphone on the same learning device, and simultaneously record the interactive behavior log stream using a learning platform client running on the learning device. For example, using the same learning device as a carrier, three types of core data are collected collaboratively through the hardware and software integrated into the device. An integrated camera captures the learner's facial image in real time, forming a facial video stream, such as recording dynamic changes in eyebrows and corners of the mouth during the learning process; an integrated microphone continuously collects the learner's voice information, generating an audio stream, including speaking when answering questions and silence when thinking; the learning platform client running on the learning device tracks and records the learner's operation behavior in real time, forming an interactive behavior log stream, such as clicking options, canceling answers, dragging the mouse, etc.

[0026] Next, each frame of the facial video stream, each preset duration audio segment of the voice audio stream, and each operation event of the interaction behavior log stream are respectively labeled with a high-precision timestamp based on the same system clock source; For example, a unified, high-precision timestamp is assigned to the smallest data unit of each of the three types of data streams to ensure time-series traceability. Each frame of the facial video stream is timestamped; for example, the timestamp for frame 100 is 1690000001000. The audio stream is divided into audio segments according to a preset duration, and each segment is timestamped. The preset duration can be set to 0.3 seconds, and the timestamps for a certain segment range from 1690000001000 to 1690000001300. Each operation event in the interactive behavior log stream is individually timestamped; for example, the timestamp for clicking the submit button is 1690000001200. All timestamps are based on the same system clock source, with millisecond-level precision.

[0027] Then, based on the high-precision timestamp, the video frame sequence, audio segment sequence, and operation event sequence within the same time window are aligned and packaged to generate the multimodal raw data stream with a unified time reference.

[0028] For example, based on high-precision timestamps, a fixed time window, such as 5 seconds, is defined. Within this window, the corresponding video frame sequence, audio segment sequence, and operation event sequence are precisely matched to ensure that the three types of data within the same time window reflect the learner's state at the same moment. For instance, within the time window from 1690000001000 to 1690000001500, there are 200 frames of facial video, 17 audio segments, and 3 interaction operation logs. These data are integrated and packaged to generate a multimodal raw data stream with a unified time reference, facilitating temporal correlation analysis during subsequent feature extraction.

[0029] In the feature vector extraction module 12 of this application embodiment, facial morphological feature vectors are extracted based on the multimodal raw data stream, including: The learning interface of the learning terminal is divided into multiple preset functional areas; Based on the facial video stream, the gaze focus is calculated by tracking the learner's pupils and eye feature points. Based on the gaze focus shift sequence in the preset functional area, the gaze focus offset rate, which represents the attention allocation pattern, is calculated and generated. Based on the facial video stream, a region of interest is defined in the learner's glabella area. By analyzing the texture changes of the region of interest between consecutive frames, the glabella wrinkling frequency, which represents the dynamics of micro-expressions, is calculated and generated. Based on the facial video stream, the movement trajectory of key points at the corners of the learner's mouth is tracked, and a mathematical description vector of the corner curvature change curve is calculated and generated by analyzing the change pattern of the corner curvature over time.

[0030] The feature vector extraction module 12 in this embodiment of the application further includes: The number of times the learner's gaze switches between different preset functional areas per unit time is counted, and combined with the duration of continuous gaze in a single functional area and the coefficient of variation of the gaze duration, a comprehensive offset rate value is generated as the gaze focus offset rate. Extract the local texture feature time series of the region of interest, perform frequency domain analysis on the local texture feature time series, extract the main frequency component, and use the main frequency component as the frequency of the brow wrinkle; The time series of the corner of the mouth curvature is subjected to first-order and second-order difference calculations to obtain the change rate sequence and acceleration sequence. Multiple mathematical descriptors, including the regularity of the number of positive and negative half-waves and the peak interval, are extracted from the change rate sequence and the acceleration sequence and combined to form the mathematical descriptive vector of the corner of the mouth curvature change curve.

[0031] In the feature vector extraction module 12 of this application embodiment, the purpose of the above steps is to extract facial morphological feature vectors from the facial video stream of the multimodal raw data stream that can accurately reflect the learner's attention allocation, micro-expression dynamics and emotional tendencies, so as to provide core data support for subsequent deep fusion feature vector construction, cognitive-emotional state decoding and attribution analysis, and solve the problem of insufficient depth of analysis of learner's psychological state in the existing technology.

[0032] To achieve the above objectives, the learning interface of the learning terminal needs to be divided into multiple preset functional areas; that is, the areas should be clearly and reasonably divided according to the functional attributes of the learning terminal interface to ensure that the distribution of the gaze in different learning scenarios can be accurately captured.

[0033] For example, the interface of an online learning app can be divided into five pre-defined functional areas: a course explanation video area, a knowledge point graphic display area, an in-class exercise answer area, a formula editor area, and an error notebook entry area. Each area has clear boundaries and covers the complete learning scenario.

[0034] Next, based on the facial video stream, the gaze focus is calculated by tracking the learner's pupils and eye feature points. According to the gaze focus shift sequence in the preset functional area, the gaze focus offset rate, which represents the attention allocation pattern, is calculated and generated. In this process, the number of times the learner's gaze switches between different preset functional areas per unit time is counted, and a comprehensive offset rate value is generated by combining the continuous gaze duration and the coefficient of variation of the gaze duration in a single functional area, which is used as the gaze focus offset rate. Specifically, the first step is to locate the focal point of the gaze. Based on the facial video stream, computer vision algorithms track the learner's pupil center position, inner corner of the eye, outer corner of the eye, and other eye feature points in real time. Combined with the learning terminal screen coordinate system, the system accurately calculates which preset functional area the focal point of the gaze falls on at each instant.

[0035] For example, the specific method for real-time tracking of eye feature points such as the center position of the pupil, inner corner of the eye, and outer corner of the eye using computer vision algorithms is as follows: First, a lightweight face detection algorithm such as MTCNN is used to outline the face from the real-time image of the camera, quickly filtering the background and focusing only on the facial area; then, based on the face outline, a small area containing the eyes is cropped at a fixed ratio, such as the regular position range of the eyes in the face outline, to narrow the target range for subsequent detection; then, an eye key point model such as MediaPipe Face Mesh is used to analyze the cropped eye area. The MediaPipe Face Mesh has a built-in coordinate template for eye feature points and can directly output the pixel coordinates of the pupil center, inner corner of the eye, and outer corner of the eye.

[0036] Finally, the pixel coordinates of the eye feature points are converted into physical coordinates of the learning terminal screen. Then, the gaze vector is calculated by the relative position of the pupil center with the inner and outer corners of the eye. The screen functional areas are pre-divided and the coordinate range of each area is marked. The gaze vector is projected onto the screen plane in real time to obtain the focus coordinates. Finally, the preset functional area to which the focus coordinates belong is matched.

[0037] The second step is to calculate the number of switching times per unit time to measure the number of times the learner's gaze switches between different preset functional areas per unit time; to calculate the duration of continuous gaze in a single area to measure the duration of continuous gaze in a single functional area per unit time; and to calculate the coefficient of variation of gaze duration to measure the coefficient of variation of the learner's gaze duration per unit time.

[0038] Calculate the number of switching times per unit of time: Set the unit of time to 1 minute, and count the number of times the learner's gaze switches between different preset functional areas within this time period. For example, within 1 minute, switch from the knowledge point graphic display area to the in-class exercise answer area 4 times, from the answer area to the video area 3 times, and from the video area to the error notebook entrance area 1 time, for a total of 8 switching times.

[0039] Calculate the duration of sustained gaze in a single area: Record the duration of the learner's gaze in each preset functional area, such as 30 seconds, 25 seconds, and 35 seconds in the course explanation video area, and 15 seconds, 20 seconds, and 18 seconds in the in-class exercise answer area.

[0040] Calculate the coefficient of variation of gaze duration: Taking the gaze duration in the course explanation video area as an example, first calculate the average (30+25+35) / 3=30 seconds, then calculate the standard deviation (√[(30-30)]). 2 +(25-30) 2 +(35-30) 2 ] / 3)≈5.77 seconds, the coefficient of variation of gaze duration = standard deviation / mean ≈5.77 / 30≈0.19.

[0041] The third step is to generate the overall offset rate. A comprehensive calculation is performed using a preset weighting formula, such as a weight of 0.4 for the number of times the view is switched per unit time, a weight of 0.3 for the duration of continuous gaze in a single area, and a weight of 0.3 for the coefficient of variation of gaze duration. Based on the example values ​​above, the overall offset rate is calculated as: 8 × 0.4 + 30 × 0.3 + 0.19 × 0.3 ≈ 3.2 + 9 + 0.057 ≈ 12.257. The final calculated gaze focus offset rate is 12.26, rounded to two decimal places.

[0042] Then, based on the facial video stream, a region of interest is defined in the learner's glabella area. By analyzing the texture changes of the region of interest between consecutive frames, the glabella wrinkle frequency, which represents the dynamics of micro-expressions, is calculated and generated. Specifically, the local texture feature time series of the region of interest is extracted, and frequency domain analysis is performed on the local texture feature time series to extract the dominant frequency component, which is then used as the glabella wrinkle frequency.

[0043] The first step is to define the region of interest (ROI) between the eyebrows. Based on the facial video stream, the ROI region, specifically the rectangular area between the eyebrows and above the bridge of the nose, is located using a facial landmark detection algorithm. The ROI size is set to 30×20 pixels, and can be adaptively adjusted according to the learner's face size to ensure coverage of all areas between the eyebrows that may produce wrinkles.

[0044] The second step is to extract the local texture feature time series. For 100 consecutive frames of facial video, with a frame rate of 30 frames / second and a duration of about 3.3 seconds, extract the local texture features of the region of interest between the eyebrows frame by frame, such as gray-level co-occurrence matrix features and LBP local binary mode features, to form a texture feature time series of length 100, for example, the feature value corresponding to each frame is [120, 125, 130, ..., 142].

[0045] The third step requires frequency domain analysis and dominant frequency extraction. A Fast Fourier Transform (FFT) is used to transform the time series of local texture features into a frequency domain signal. The time domain signal can be the texture features that change over time, and the frequency domain signal can be the amplitude distribution of different frequency components. Then, the frequency component with the largest amplitude is selected as the dominant frequency component. For example, after FFT transformation, the frequency corresponding to the highest amplitude is 1.8Hz, so the frequency of the brow crease is 1.8Hz. A higher value indicates more frequent brow creases, reflecting a higher level of cognitive effort or confusion in the learner.

[0046] Furthermore, based on the facial video stream, the movement trajectory of key points at the corners of the learner's mouth is tracked. By analyzing the change pattern of the mouth corner curvature time series, a mathematical descriptive vector for the mouth corner curvature change curve is calculated and generated. Specifically, first-order and second-order difference calculations are performed on the mouth corner curvature time series to obtain a change velocity sequence and an acceleration sequence. Multiple mathematical descriptors, including the regularity of the number of positive and negative half-waves and the peak interval, are extracted from the change velocity sequence and the acceleration sequence and combined to form the mathematical descriptive vector for the mouth corner curvature change curve.

[0047] The first step is to obtain the time series of the corner of the mouth's curvature. This involves using a facial keypoint detection algorithm to track feature points on both sides of the corner of the mouth, such as key markers indicating whether the corners of the mouth are upturned or downturned. The angle between the corner of the mouth and the horizontal line is calculated for each frame as the curvature value of the corner of the mouth. This process is repeated for 120 frames to form a time series of the corner of the mouth's curvature, for example, [15°, 16°, 14°, ..., 8°]. Positive values ​​represent upturned corners of the mouth, and negative values ​​represent downturned corners of the mouth.

[0048] The second step requires calculating the velocity sequence and acceleration sequence.

[0049] First, a first-order difference calculation is performed, which involves subtracting the radian value of the previous frame from the radian value of the next frame, and then dividing by the frame interval time to obtain the change rate sequence. For example, if the radian difference between the second frame and the first frame is 1°, then the change rate is 1° / 0.033 seconds ≈ 30° / second; if the radian difference between the third frame and the second frame is -2°, then the change rate is -2° / 0.033 seconds ≈ -60° / second, ultimately forming a change rate sequence of [30, -60, ...].

[0050] Then, a second-order difference calculation is performed, that is, the same first-order difference operation is performed on the velocity sequence to obtain the acceleration sequence. For example, if the difference between the second velocity value and the first velocity value is -90° / second, the acceleration is -90° / second ÷ 0.033 seconds ≈ -2727° / second. 2 This forms an acceleration sequence of [-2727,...].

[0051] The third step involves extracting mathematical descriptors and combining them into a mathematical descriptive vector of the curve showing the change in the corner of the mouth. Specifically, this requires extracting multiple mathematical descriptors that include the number of positive and negative half-waves and the regularity of peak intervals.

[0052] The extraction of the number of positive and negative half-waves is as follows: count the number of positive (positive half-wave) and negative (negative half-wave) values ​​in the velocity sequence and acceleration sequence. For example, the velocity sequence has 35 positive half-waves and 28 negative half-waves, and the acceleration sequence has 32 positive half-waves and 30 negative half-waves.

[0053] The extraction of peak interval regularity involves filtering the peaks in the velocity and acceleration sequences and calculating the time interval between adjacent peaks. For example, if the peak intervals of the velocity sequence are 0.5 seconds, 0.6 seconds, and 0.55 seconds, the average value is 0.55 seconds and the standard deviation is 0.025 seconds. The smaller the standard deviation, the more regular the interval.

[0054] Optionally, the average value of the peak amplitude and the amplitude ratio of the positive and negative half-waves can also be extracted. For example, the average peak amplitude of the velocity sequence is 50° / second, and the amplitude ratio of the positive half-wave is 60%.

[0055] Finally, all mathematical descriptors are combined in a preset order to generate a mathematical descriptive vector of the curve of mouth corner curvature, such as [35,28,32,30,0.55,0.025,50,60,...].

[0056] In the feature vector extraction module 12 of this application embodiment, based on the multimodal raw data stream, acoustic prosodic feature vectors and behavioral dynamics feature vectors are extracted, including: The speech audio stream is extracted tones, speech rate variation coefficients, and silence interval lengths, and the three are combined to form the acoustic prosodic feature vector. The number of attempts per unit time, the frequency of undo operations, and the degree of mouse trajectory disorder are calculated from the interaction behavior log, and the three are combined to form the behavior dynamic feature vector.

[0057] In the feature vector extraction module 12 of this application embodiment, the purpose of the above steps is to extract acoustic prosodic feature vectors that reflect the learner's emotional state, expression stability, and operational behavior characteristics from the speech audio stream and interaction behavior log of the multimodal raw data stream, as well as behavioral dynamic feature vectors that reflect learning participation, operational proficiency, and thinking clarity. In conjunction with facial morphological feature vectors, these feature vectors provide comprehensive multi-dimensional data support for subsequent deep fusion feature vector construction and cognitive-emotional state decoding, making state analysis more three-dimensional and accurate.

[0058] To achieve the above objectives, it is first necessary to extract intonation smoothness, speech rate variation coefficient and silence interval length from the speech audio stream, and combine the three to form the acoustic prosodic feature vector; The first step is to extract intonation smoothness: this involves preprocessing the audio stream to remove background noise and then analyzing the amplitude of pitch changes. For example, a pitch change threshold of 50Hz can be set, and the number of times the pitch fluctuation exceeds this threshold within a unit of time (e.g., 30 seconds) is counted. The fewer the number of times, the smoother the intonation. For example, if the pitch fluctuation exceeds the threshold 3 times within 30 seconds, the smoothness is calculated as (total duration - fluctuation duration) / total duration, resulting in an intonation smoothness of 0.92.

[0059] The second step requires calculating the coefficient of variation of speech rate: first, the audio stream is segmented into multiple sentence segments, and the speech rate (in words per second) of each sentence is calculated. For example, if the speech rates of the 5 sentences are 18 words per second, 20 words per second, 17 words per second, 22 words per second, and 19 words per second, the average is calculated as (18+20+17+22+19) / 5=19.2 words per second. Then, the standard deviation is calculated to be approximately 1.79 words per second. Therefore, the coefficient of variation of speech rate = standard deviation / average ≈ 1.79 / 19.2 ≈ 0.093.

[0060] The third step requires calculating the length of the silence interval: that is, identifying segments in the audio stream that have no valid speech signal. For example, if the silence threshold is set to -40dB, the total duration of all silent segments within a unit of time is accumulated. For example, if the silent segments within 1 minute are 2 seconds, 3 seconds, 1.5 seconds, and 2.5 seconds, the total silence interval length is 9 seconds.

[0061] The fourth step is to combine the feature vectors: for example, combine the intonation smoothness (0.92), speech rate variation coefficient (0.093), and silence interval length (9 seconds) in sequence to form an acoustic prosodic feature vector, such as [0.92, 0.093, 9].

[0062] Next, the number of attempts per unit time, the frequency of undo operations, and the degree of mouse trajectory disorder are calculated from the interaction behavior log, and the three are combined to form the behavior dynamics feature vector.

[0063] The first step is to calculate the number of attempts per unit of time: for example, setting the unit of time as 2 minutes, and counting the number of times learners attempt the same task from the interaction behavior log. For example, in the exercise answering task, the total number of times an answer is submitted, modified, and resubmitted within 2 minutes is 6, that is, the number of attempts per unit of time is 6.

[0064] The second step is to calculate the frequency of undo operations: count the total number of undo operations per unit time, and divide by the unit time to get the frequency. For example, if there are 4 undo operations in 1 minute, the frequency of undo operations = 4 times / 60 seconds ≈ 0.067 times / second.

[0065] The third step is to calculate the mouse trajectory disorder: extract mouse coordinate data from the interaction behavior log and calculate the curvature change and deviation from a straight line in the trajectory. For example, if the straight-line distance of the trajectory is set to 100 pixels and the actual trajectory length is 150 pixels, the disorder = (actual trajectory length - straight-line distance) / straight-line distance × 100%. Combining this with the dispersion of the coordinate points, the final mouse trajectory disorder is calculated to be 0.45.

[0066] The fourth step is to combine the feature vectors: for example, combine the number of attempts per unit time (6), the frequency of undo operations (0.067 times / second), and the mouse trajectory disorder (0.45) in order to form the behavioral dynamics feature vector [6, 0.067, 0.45].

[0067] In the feature vector extraction module 12 of this application embodiment, the facial morphological feature vector, the acoustic prosodic feature vector, and the behavioral dynamics feature vector are also fused to obtain a deep fused feature vector.

[0068] In this embodiment, the purpose of obtaining the deep fusion feature vector is to eliminate the limitations of a single dimension, form a more comprehensive deep fusion feature vector, and provide unified and efficient core input data for subsequent accurate decoding of cognitive-emotional states.

[0069] Specifically, feature standardization is required first, which means scaling the parameters of each dimension of the three types of feature vectors to the range of 0-1. For example, the gaze focus offset rate of 12.26 in the facial morphology feature vector is standardized to 0.61, and the silence gap length of 9 seconds in the acoustic prosody feature vector is standardized to 0.15.

[0070] Next, feature concatenation and weighted fusion are performed. This involves concatenating the three types of feature vectors after standardization in a preset order, and then summing them using pre-trained weights. For example, the weights for facial morphology features are 0.4, vocal prosody features are 0.3, and behavioral dynamics features are 0.3. For instance, the concatenated vector is [0.61, 0.18, 0.52, 0.92, 0.093, 9, 6, 0.067, 0.45], and the weighted calculation yields a deep fusion feature vector of [0.58, 0.16, 0.49, ...].

[0071] In the feature vector decoding module 13 of this application embodiment, based on the deep fusion feature vector, the basic cognitive-emotional state probability distribution is decoded and obtained, including: The deep fusion feature vector is input into a pre-trained state classification model, which is built based on a deep neural network. The state classification model calculates based on the deep fusion feature vector and outputs a multidimensional probability vector; wherein each dimension of the multidimensional probability vector corresponds to a preset cognitive-emotional basic state. The cognitive-emotional baseline states include at least a state of deep focus, a state of fluent learning, a state of cognitive confusion, and a state of emotional frustration. The value of each dimension in the multidimensional probability vector represents the predicted probability of the existence of the corresponding state, and the sum of the values ​​of all dimensions is 1.

[0072] In the feature vector decoding module 13 of this application embodiment, the state classification model is constructed based on a deep neural network, including: Multiple sets of sample data were collected. Each set of sample data included sample facial video streams, sample voice audio streams, sample interaction behavior log streams, and sample cognitive-emotional true state labels synchronized with the sample data and annotated by experts. The sample data is preprocessed and features are extracted to obtain a sample deep fusion feature vector; The deep neural network model is trained using the deep fusion feature vector of the sample as the training input and the corresponding cognitive-emotional real state label of the sample as the supervision signal. Finally, the state classification model that has been trained and meets the preset performance standards is obtained.

[0073] In the feature vector decoding module 13 of this application, the purpose of the above steps is to analyze and calculate the deep fusion feature vector through a pre-trained deep neural network model, output the probability distribution of various cognitive-emotional basic states, accurately quantify the learner's current psychological state, and provide core judgment basis for subsequent attribution analysis and learning plan adjustment.

[0074] To achieve the above objectives, it is first necessary to construct the state classification model based on a deep neural network.

[0075] For example, the state classification model can adopt a hybrid architecture of CNN + fully connected layers, with the following specific structure: Input layer: Used to receive deep fusion feature vectors. The dimension can be set to 256, for example, facial morphology feature vector 64-dimensional + acoustic prosody feature vector 32-dimensional + behavioral dynamics feature vector 32-dimensional. After standardization and weighted fusion, it is expanded to 256 dimensions. The input data format is float32, and the batch size is set to 32.

[0076] Convolutional layer 1: 64 kernels, 3×3 kernel size, stride 1, same padding, ReLU activation function, output feature map dimension 256×64.

[0077] Pooling layer 1: Max pooling is used, with a pooling kernel size of 2×2, a step size of 2, and an output dimension of 128×64.

[0078] Convolutional layer 2: 128 kernels, 3×3 kernel size, stride 1, same padding method, ReLU activation function, output feature map dimension 128×128.

[0079] Pooling layer 2: Max pooling is used, with a pooling kernel size of 2×2, a step size of 2, and an output dimension of 64×128.

[0080] Flattening layer: Used to convert the two-dimensional feature map output by pooling layer 2 into a one-dimensional vector with a dimension of 64×128=8192.

[0081] Fully connected layer 1: 512 neurons, ReLU activation function, dropout rate set to 0.3 to prevent overfitting, output dimension 512.

[0082] Fully connected layer 2: 256 neurons, ReLU activation function, dropout rate 0.2, output dimension 256.

[0083] Output layer: 4 neurons, corresponding to 4 cognitive-emotional basic states. The activation function is Softmax. The output dimension is 4-dimensional, corresponding to the deep focus state, the fluent learning state, the cognitive confusion state, and the emotional frustration state.

[0084] For training the state classification model, the Adam optimizer was used with an initial learning rate of 0.001. The learning rate decayed to 0.9 times its original value every 10 epochs. The cross-entropy loss function was used to calculate the difference between the predicted probability vector and the true label.

[0085] The training consists of 50 epochs, and an early stop mechanism is triggered when the accuracy on the validation set fails to improve for 5 consecutive epochs.

[0086] Performance evaluation metrics include: test set accuracy ≥ 92%, recall ≥ 90%, and F1 score ≥ 90%. All of these metrics must be met simultaneously to be considered to meet the preset performance standards.

[0087] This requires collecting multiple sets of sample data, for example, 10,000 sets of sample data, covering learners of different ages and learning scenarios, such as math problem-solving, language learning, and programming practice. Each set of samples includes synchronized sample facial video streams, sample audio streams, sample interaction behavior log streams, and sample cognitive-emotional true state labels jointly annotated by at least 3 experts.

[0088] Then, using the previously described high-precision timestamp annotation method, the sample facial video stream, voice audio stream, and interaction behavior log stream are time-aligned to generate the sample multimodal raw data stream.

[0089] Using the aforementioned feature extraction method, facial morphological feature vectors, acoustic prosodic feature vectors, and behavioral dynamics feature vectors of the samples are extracted respectively.

[0090] The feature vectors of the three types of samples are standardized, and then weighted and fused to generate a deep fusion feature vector. The specific processing principle is the same as the method of obtaining the deep fusion feature vector.

[0091] Furthermore, the sample deep fusion feature vectors are divided into training set, validation set, and test set in a 7:2:1 ratio.

[0092] The deep neural network model is iteratively trained by taking the deep fusion feature vector of the training set as input and the corresponding sample cognitive-emotional real state label as supervision signal.

[0093] During training, metrics such as the accuracy of the validation set are monitored in real time. If overfitting occurs, the dropout rate is adjusted or a regularization term is added. If the convergence speed is too slow, the learning rate is appropriately increased. After training, the model is evaluated using a test set. If it meets the preset performance criteria, it is determined to be a successfully trained state classification model.

[0094] Furthermore, the deep fusion feature vector obtained in the aforementioned steps is input into the trained state classification model.

[0095] The state classification model extracts high-order correlation information of features through convolutional layers, transforms it through fully connected layers, and finally calculates the output multidimensional probability vector through the Softmax activation function of the output layer.

[0096] For example, suppose the output multidimensional probability vector is [0.05, 0.12, 0.78, 0.05], where the first dimension corresponds to the state of deep focus with a probability of 0.05; the second dimension corresponds to the state of fluent learning with a probability of 0.12; the third dimension corresponds to the state of cognitive confusion with a probability of 0.78; and the fourth dimension corresponds to the state of emotional frustration with a probability of 0.05. The sum of all the dimensions is 1, clearly showing the probability of existence of each cognitive-emotional basic state.

[0097] In the report generation module 14 of this application embodiment, based on the basic cognitive-emotional state probability distribution and combined with the facial morphological feature vector, the acoustic prosodic feature vector, and the behavioral dynamics feature vector, attribution analysis is performed to generate a real-time state analysis report containing a state attribution description, including: From the aforementioned basic cognitive-emotional state probability distribution, the state with the highest predicted probability value is selected as the dominant state; Based on the facial morphological feature vector, the acoustic prosodic feature vector, and the behavioral dynamics feature vector, the contribution of each feature dimension to the state classification model in predicting the dominant state is calculated. Feature dimensions with a contribution rate higher than a preset threshold are selected as key evidence features; According to a preset mapping rule, the key evidence features are mapped to state attributions described in natural language; wherein, the mapping rule defines the correspondence between specific feature patterns and cognitive or emotional causes. The dominant state, the predicted probability corresponding to the dominant state, and the state attribution are integrated to generate the real-time state analysis report containing the state attribution description.

[0098] In this embodiment of the application, the purpose of the above-mentioned analysis report generation module 14 is to clarify the learner's current dominant cognitive-emotional state and core causes, and generate a real-time state analysis report that combines quantitative data and common attribution, so as to provide a clear and practical basis for subsequent precise adjustment of the learning plan, and solve the problem that the feedback of the existing technology is general and cannot trace the root cause of the state.

[0099] To achieve the above objectives, it is first necessary to select the state with the highest predicted probability value from the basic cognitive-emotional state probability distribution as the dominant state. This involves selecting the state with the highest predicted probability from the basic cognitive-emotional state probability distribution as the dominant state. For example, if the probability distribution is: deep focus state 0.05, fluent learning state 0.12, cognitive confusion state 0.78, and emotional frustration state 0.05, then the cognitive confusion state has the highest probability and is therefore the dominant state, with a corresponding predicted probability of 0.78.

[0100] Then, based on the facial morphology feature vector, the acoustic prosodic feature vector, and the behavioral dynamics feature vector, the contribution of each feature dimension to the state classification model's prediction of the dominant state is calculated. For example, using SHAP values ​​or gradient attribution, the contribution of each dimension in the three types of feature vectors—facial morphology, acoustic prosody, and behavioral dynamics—to the dominant state prediction result can be calculated. For instance, in facial morphology features, the contribution of the frequency of brow folds is 0.35, and the contribution of the gaze focus shift rate is 0.28; in acoustic prosody features, the contribution of the speech rate variation coefficient is 0.12; in behavioral dynamics features, the contribution of the undo operation frequency is 0.20, and the contributions of other dimensions are all below 0.05.

[0101] Next, feature dimensions with a contribution rate higher than a preset threshold are selected as key evidence features; For example, a preset threshold for contribution is set at 0.15, and feature dimensions with contribution values ​​higher than this threshold are selected as key evidence features. In the example above, the frequency of brow creases (0.35), gaze focus shift rate (0.28), and undo operation frequency (0.20) are all higher than 0.15, thus qualifying as key evidence features.

[0102] Furthermore, according to a preset mapping rule, the key evidence features are mapped to state attributions described in natural language; wherein, the mapping rule defines the correspondence between specific feature patterns and cognitive or emotional causes. The specific feature pattern refers to a quantitative combination pattern composed of key evidence features such as the learner's physiological characteristics and operational behavior. For example, features such as the frequency of frown lines, the rate of gaze focus shift, and the frequency of operational behavior reach specific values ​​and meet the contribution threshold. For example, specific feature pattern 1 can be a frown line frequency higher than 2Hz and a contribution ≥ 0.2; specific feature pattern 2 can be a gaze focus shift rate ≥ 0.6 and a contribution ≥ 0.2; specific feature pattern 3 can be a cancel operation frequency ≥ 0.06 times / second and a contribution ≥ 0.15.

[0103] The specific feature patterns mentioned above each represent a psychological state of the learner. These specific feature patterns can be determined by collecting raw data such as learner's brow furrows, gaze focus, and operational behavior, combined with the learner's subjective psychological questionnaire results in the corresponding data scenario. Examples of such subjective psychological questionnaire results include: difficulty in understanding concepts, lack of concentration, and hesitation in problem-solving approaches.

[0104] For example, the mapping rules may include a brow crease frequency higher than 2Hz and a contribution of ≥0.2, corresponding to conceptual comprehension difficulties leading to confusion; a gaze focus shift rate of ≥0.6 and a contribution of ≥0.2, corresponding to frequent switching of attention across multiple areas and difficulty in focusing; and a cancellation operation frequency of ≥0.06 times / second and a contribution of ≥0.15, corresponding to uncertainty in problem-solving approaches and repeated revisions.

[0105] Based on the key features of the example, the state attribution description can be: current cognitive confusion is mainly due to obstructed understanding of knowledge points, accompanied by difficulty in focusing attention and uncertainty in problem-solving strategies, leading to frequent undoing operations.

[0106] Finally, the dominant state, the predicted probability corresponding to the dominant state, and the state attribution are integrated to generate the real-time state analysis report containing the state attribution description.

[0107] The system integrates the dominant state of cognitive confusion, the corresponding predicted probability of 0.78, and the state attribution description to generate a real-time state analysis report. An example of a real-time state analysis report is as follows: The dominant state is cognitive confusion, with a predicted probability of 78%. The state is attributed to current cognitive confusion, primarily due to obstructed understanding of knowledge points, accompanied by difficulty focusing attention and frequent undoing of actions due to uncertainty in problem-solving strategies.

[0108] Learning plan adjustment module 15 dynamically adjusts the learning plan based on the real-time status analysis report, including: Analyze the real-time status analysis report and extract the dominant status and corresponding status attribution description from the real-time status analysis report; The dominant state and state attribution description are input into a preset learning planning adjustment strategy mapping library for matching; The learning planning adjustment strategy mapping library outputs at least one suitable learning intervention strategy based on the input matching conditions. Based on the aforementioned learning intervention strategy, the subsequent learning content, learning path, presentation method, and interaction rhythm in the current learning plan are adjusted in real time. The adjusted learning plan will be pushed to the learning terminal for execution in real time.

[0109] In the learning plan adjustment module 15 of this application embodiment, the purpose of the above steps is to match appropriate learning intervention strategies based on the dominant state and state attribution in the real-time state analysis report, dynamically adjust the core elements of the learning plan, realize personalized and precise learning intervention, help learners alleviate adverse states, and improve learning efficiency and effectiveness.

[0110] To achieve the above objectives, it is first necessary to parse the real-time state analysis report and extract the dominant state and its corresponding state attribution description; that is, to perform structured parsing of the real-time state analysis report and extract the dominant state and its corresponding state attribution description. For example, if the dominant state obtained after parsing is cognitive confusion, the state attribution description is that the current cognitive confusion is mainly due to obstructed understanding of knowledge points, accompanied by difficulty in focusing attention and uncertainty in problem-solving strategies, leading to frequent undoing operations.

[0111] Next, the dominant state and state attribution description are input into a preset learning planning adjustment strategy mapping library for matching; The dominant state and state attribution description to be extracted are input into a pre-defined learning planning adjustment strategy mapping library. This learning planning adjustment strategy mapping library first stores the association between different state-attribution combinations and corresponding intervention strategies. For example, for a combination of cognitive confusion state + knowledge point comprehension obstruction + lack of focus + uncertainty in problem-solving approach, the matching intervention strategies include breaking down difficult knowledge points, increasing guided questioning, slowing down the interaction pace, and optimizing the content presentation.

[0112] Then, the learning planning adjustment strategy mapping library outputs at least one suitable learning intervention strategy based on the input matching conditions. The learning plan adjustment strategy mapping library outputs at least one suitable learning intervention strategy based on the input matching conditions. Using the example above, the output intervention strategies are: 1. Break down the current learning knowledge point, decomposing complex concepts into three basic sub-knowledge points, each accompanied by a 5-minute micro-video explanation; 2. After explaining each sub-knowledge point, insert two basic reinforcement questions, using guided pop-up questions to help organize the thinking; 3. Extend the interaction interval between each learning segment from the original 30 seconds to 60 seconds, providing ample time for reflection; 4. Change the pure text knowledge point analysis to a presentation method combining text and images with animation demonstrations.

[0113] Furthermore, based on the aforementioned learning intervention strategy, the subsequent learning content, learning path, presentation method, and interaction rhythm in the current learning plan are adjusted in real time. For example, in subsequent learning content, based on the learning intervention strategy, the originally planned complex knowledge points are replaced with three basic sub-knowledge points, accompanied by micro-videos and reinforcement questions.

[0114] Regarding the learning path, the original learning path involved directly learning the complex knowledge points as planned, accompanied by pure text explanations. This has been adjusted to learning sub-knowledge point 1, watching micro-videos, and completing reinforcement exercises; learning sub-knowledge point 2, watching micro-videos, and completing reinforcement exercises; learning sub-knowledge point 3, watching micro-videos, and completing reinforcement exercises; and finally returning to the original knowledge points for comprehensive application.

[0115] In terms of presentation, the original plain text parsing has been changed to a combination of text, images, and animations, with key content highlighted.

[0116] In terms of the interactive rhythm, the interval between interactions has been adjusted from 30 seconds to 60 seconds, and the time for answering reinforcement questions has been extended from 1 minute to 2 minutes.

[0117] Finally, the adjusted learning plan is immediately pushed to the learning terminal for execution. This means that the adjusted learning plan is instantly pushed to the learner's learning terminal, such as a learning app or tablet learning software. Upon receiving the plan, the terminal immediately updates its current learning task list and executes it according to the new plan.

[0118] Example 2, as Figure 2 As shown, based on the human-computer collaborative personalized learning plan generation system provided in Embodiment 1 and the same inventive concept, this embodiment of the invention also provides a human-computer collaborative personalized learning plan generation method, including: S100: Simultaneously collects learners' facial video stream, voice audio stream, and interaction behavior log stream, and performs time alignment to obtain multimodal raw data stream; S200: Based on the multimodal raw data stream, extract facial morphological feature vectors, acoustic prosodic feature vectors, and behavioral dynamics feature vectors respectively. The facial morphological feature vectors include mathematical description vectors of gaze focus shift rate, glabellar wrinkle frequency, and mouth corner curvature change curve. S300: The facial morphological feature vector, the acoustic prosodic feature vector, and the behavioral dynamics feature vector are fused to obtain a deep fusion feature vector; based on the deep fusion feature vector, the basic cognitive-emotional state probability distribution is decoded; S400: Based on the basic cognitive-emotional state probability distribution, and combined with the facial morphological feature vector, the acoustic prosodic feature vector and the behavioral dynamics feature vector, perform attribution analysis to generate a real-time state analysis report containing state attribution descriptions; S500: Based on the real-time status analysis report, dynamically adjust the learning plan.

[0119] In embodiment S100 of this application, the learner's facial video stream, voice audio stream, and interaction behavior log stream are simultaneously acquired and time-aligned to obtain a multimodal raw data stream, including: On the same learning device, the facial video stream is simultaneously captured based on the integrated camera, the voice audio stream is simultaneously captured based on the integrated microphone, and the interactive behavior log stream is simultaneously recorded based on the learning platform client running on the learning device. Each frame of the facial video stream, each preset duration audio segment of the voice audio stream, and each operation event of the interaction behavior log stream are respectively labeled with a high-precision timestamp based on the same system clock source; Based on the high-precision timestamp, the video frame sequence, audio segment sequence, and operation event sequence within the same time window are aligned and packaged to generate the multimodal raw data stream with a unified time reference.

[0120] In embodiment S200 of this application, facial morphological feature vectors are extracted based on the multimodal raw data stream, including: The learning interface of the learning terminal is divided into multiple preset functional areas; Based on the facial video stream, the gaze focus is calculated by tracking the learner's pupils and eye feature points. Based on the gaze focus shift sequence in the preset functional area, the gaze focus offset rate, which represents the attention allocation pattern, is calculated and generated. Based on the facial video stream, a region of interest is defined in the learner's glabella area. By analyzing the texture changes of the region of interest between consecutive frames, the glabella wrinkling frequency, which represents the dynamics of micro-expressions, is calculated and generated. Based on the facial video stream, the movement trajectory of key points at the corners of the learner's mouth is tracked, and a mathematical description vector of the corner curvature change curve is calculated and generated by analyzing the change pattern of the corner curvature over time.

[0121] In embodiment S200 of this application, it further includes: The number of times the learner's gaze switches between different preset functional areas per unit time is counted, and combined with the duration of continuous gaze in a single functional area and the coefficient of variation of the gaze duration, a comprehensive offset rate value is generated as the gaze focus offset rate. Extract the local texture feature time series of the region of interest, perform frequency domain analysis on the local texture feature time series, extract the main frequency component, and use the main frequency component as the frequency of the brow wrinkle; The time series of the corner of the mouth curvature is subjected to first-order and second-order difference calculations to obtain the change rate sequence and acceleration sequence. Multiple mathematical descriptors, including the regularity of the number of positive and negative half-waves and the peak interval, are extracted from the change rate sequence and the acceleration sequence and combined to form the mathematical descriptive vector of the corner of the mouth curvature change curve.

[0122] In embodiment S200 of this application, based on the multimodal raw data stream, acoustic prosodic feature vectors and behavioral dynamics feature vectors are extracted, including: The speech audio stream is extracted tones, speech rate variation coefficients, and silence interval lengths, and the three are combined to form the acoustic prosodic feature vector. The number of attempts per unit time, the frequency of undo operations, and the degree of mouse trajectory disorder are calculated from the interaction behavior log, and the three are combined to form the behavior dynamic feature vector.

[0123] In embodiment S300 of this application, based on the deep fusion feature vector, the basic cognitive-emotional state probability distribution is decoded and obtained, including: The deep fusion feature vector is input into a pre-trained state classification model, which is built based on a deep neural network. The state classification model calculates based on the deep fusion feature vector and outputs a multidimensional probability vector; wherein each dimension of the multidimensional probability vector corresponds to a preset cognitive-emotional basic state. The cognitive-emotional baseline states include at least a state of deep focus, a state of fluent learning, a state of cognitive confusion, and a state of emotional frustration. The value of each dimension in the multidimensional probability vector represents the predicted probability of the existence of the corresponding state, and the sum of the values ​​of all dimensions is 1.

[0124] In embodiment S300 of this application, the state classification model is constructed based on a deep neural network, including: Multiple sets of sample data were collected. Each set of sample data included sample facial video streams, sample voice audio streams, sample interaction behavior log streams, and sample cognitive-emotional true state labels synchronized with the sample data and annotated by experts. The sample data is preprocessed and features are extracted to obtain a sample deep fusion feature vector; The deep neural network model is trained using the deep fusion feature vector of the sample as the training input and the corresponding cognitive-emotional real state label of the sample as the supervision signal. Finally, the state classification model that has been trained and meets the preset performance standards is obtained.

[0125] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A human-computer collaborative personalized learning plan generation system, characterized in that, The system includes: The real-time data acquisition module is used to simultaneously acquire learners' facial video streams, audio streams, and interactive behavior log streams, and perform time alignment to obtain multimodal raw data streams; The feature vector extraction module is used to extract facial morphological feature vectors, acoustic prosodic feature vectors, and behavioral dynamics feature vectors based on the multimodal raw data stream. The facial morphological feature vectors include mathematical description vectors of gaze focus shift rate, glabellar wrinkle frequency, and mouth corner curvature change curve. The feature vector decoding module is used to fuse the facial morphological feature vector, the acoustic prosodic feature vector, and the behavioral dynamics feature vector to obtain a deep fusion feature vector; based on the deep fusion feature vector, the basic cognitive-emotional state probability distribution is decoded. The analysis report generation module is used to perform attribution analysis based on the basic cognitive-emotional state probability distribution, combined with the facial morphological feature vector, the acoustic prosodic feature vector and the behavioral dynamics feature vector, to generate a real-time state analysis report containing state attribution descriptions. The learning plan adjustment module is used to dynamically adjust the learning plan based on the real-time status analysis report.

2. The human-computer collaborative personalized learning plan generation system according to claim 1, characterized in that, Simultaneously acquire learners' facial video streams, audio streams, and interaction behavior log streams, and perform time alignment to obtain a multimodal raw data stream, including: On the same learning device, the facial video stream is simultaneously captured based on the integrated camera, the voice audio stream is simultaneously captured based on the integrated microphone, and the interactive behavior log stream is simultaneously recorded based on the learning platform client running on the learning device. Each frame of the facial video stream, each preset duration audio segment of the voice audio stream, and each operation event of the interaction behavior log stream are respectively labeled with a high-precision timestamp based on the same system clock source; Based on the high-precision timestamp, the video frame sequence, audio segment sequence, and operation event sequence within the same time window are aligned and packaged to generate the multimodal raw data stream with a unified time reference.

3. The human-computer collaborative personalized learning plan generation system according to claim 1, characterized in that, Based on the multimodal raw data stream, facial morphological feature vectors are extracted, including: The learning interface of the learning terminal is divided into multiple preset functional areas; Based on the facial video stream, the gaze focus is calculated by tracking the learner's pupils and eye feature points. Based on the gaze focus shift sequence in the preset functional area, the gaze focus offset rate, which represents the attention allocation pattern, is calculated and generated. Based on the facial video stream, a region of interest is defined in the learner's glabella area. By analyzing the texture changes of the region of interest between consecutive frames, the glabella wrinkling frequency, which represents the dynamics of micro-expressions, is calculated and generated. Based on the facial video stream, the movement trajectory of key points at the corners of the learner's mouth is tracked, and a mathematical description vector of the corner curvature change curve is calculated and generated by analyzing the change pattern of the corner curvature over time.

4. The human-computer collaborative personalized learning plan generation system according to claim 3, characterized in that, Also includes: The number of times the learner's gaze switches between different preset functional areas per unit time is counted, and combined with the duration of continuous gaze in a single functional area and the coefficient of variation of the gaze duration, a comprehensive offset rate value is generated as the gaze focus offset rate. Extract the local texture feature time series of the region of interest, perform frequency domain analysis on the local texture feature time series, extract the main frequency component, and use the main frequency component as the frequency of the brow wrinkle; The time series of the corner of the mouth curvature is subjected to first-order and second-order difference calculations to obtain the change rate sequence and acceleration sequence. Multiple mathematical descriptors, including the regularity of the number of positive and negative half-waves and the peak interval, are extracted from the change rate sequence and the acceleration sequence and combined to form the mathematical descriptive vector of the corner of the mouth curvature change curve.

5. The human-computer collaborative personalized learning plan generation system according to claim 1, characterized in that, Based on the aforementioned multimodal raw data stream, acoustic prosodic feature vectors and behavioral dynamics feature vectors are extracted, including: The speech audio stream is extracted tones, speech rate variation coefficients, and silence interval lengths, and the three are combined to form the acoustic prosodic feature vector. The number of attempts per unit time, the frequency of undo operations, and the degree of mouse trajectory disorder are calculated from the interaction behavior log, and the three are combined to form the behavior dynamic feature vector.

6. The human-computer collaborative personalized learning plan generation system according to claim 1, characterized in that, Based on the deep fusion feature vector, the basic cognitive-emotional state probability distribution is decoded, including: The deep fusion feature vector is input into a pre-trained state classification model, which is built based on a deep neural network. The state classification model calculates based on the deep fusion feature vector and outputs a multidimensional probability vector; wherein each dimension of the multidimensional probability vector corresponds to a preset cognitive-emotional basic state. The cognitive-emotional baseline states include at least a state of deep focus, a state of fluent learning, a state of cognitive confusion, and a state of emotional frustration. The value of each dimension in the multidimensional probability vector represents the predicted probability of the existence of the corresponding state, and the sum of the values ​​of all dimensions is 1.

7. The human-computer collaborative personalized learning plan generation system according to claim 6, characterized in that, The state classification model is built based on a deep neural network and includes: Multiple sets of sample data were collected. Each set of sample data included sample facial video streams, sample voice audio streams, sample interaction behavior log streams, and sample cognitive-emotional true state labels synchronized with the sample data and annotated by experts. The sample data is preprocessed and features are extracted to obtain a sample deep fusion feature vector; The deep neural network model is trained using the deep fusion feature vector of the sample as the training input and the corresponding cognitive-emotional real state label of the sample as the supervision signal. Finally, the state classification model that has been trained and meets the preset performance standards is obtained.

8. The human-computer collaborative personalized learning plan generation system according to claim 7, characterized in that, Based on the aforementioned basic cognitive-emotional state probability distribution, and combined with the aforementioned facial morphological feature vector, the aforementioned acoustic prosodic feature vector, and the aforementioned behavioral dynamics feature vector, attribution analysis is performed to generate a real-time state analysis report containing state attribution descriptions, including: From the aforementioned basic cognitive-emotional state probability distribution, the state with the highest predicted probability value is selected as the dominant state; Based on the facial morphological feature vector, the acoustic prosodic feature vector, and the behavioral dynamics feature vector, the contribution of each feature dimension to the state classification model in predicting the dominant state is calculated. Feature dimensions with a contribution rate higher than a preset threshold are selected as key evidence features; According to a preset mapping rule, the key evidence features are mapped to state attributions described in natural language; wherein, the mapping rule defines the correspondence between specific feature patterns and cognitive or emotional causes. The dominant state, the predicted probability corresponding to the dominant state, and the state attribution are integrated to generate the real-time state analysis report containing the state attribution description.

9. A human-computer collaborative personalized learning plan generation system according to claim 1, characterized in that, Based on the real-time status analysis report, the learning plan is dynamically adjusted, including: Analyze the real-time status analysis report and extract the dominant status and corresponding status attribution description from the real-time status analysis report; The dominant state and state attribution description are input into a preset learning planning adjustment strategy mapping library for matching; The learning planning adjustment strategy mapping library outputs at least one suitable learning intervention strategy based on the input matching conditions. Based on the aforementioned learning intervention strategy, the subsequent learning content, learning path, presentation method, and interaction rhythm in the current learning plan are adjusted in real time. The adjusted learning plan will be pushed to the learning terminal for execution in real time.

10. A method for generating personalized learning plans through human-computer collaboration, characterized in that, The method is applied to a human-computer collaborative personalized learning plan generation system according to any one of claims 1-9, the method comprising: Simultaneously collect learners' facial video streams, audio streams, and interactive behavior log streams, and perform time alignment to obtain multimodal raw data streams; Based on the multimodal raw data stream, facial morphological feature vectors, acoustic prosodic feature vectors, and behavioral dynamics feature vectors are extracted respectively. Among them, the facial morphological feature vectors include mathematical description vectors of gaze focus shift rate, glabellar wrinkle frequency, and mouth corner curvature change curve. The facial morphological feature vector, the acoustic prosodic feature vector, and the behavioral dynamics feature vector are fused to obtain a deep fusion feature vector; based on the deep fusion feature vector, the basic cognitive-emotional state probability distribution is decoded. Based on the aforementioned basic cognitive-emotional state probability distribution, and combined with the aforementioned facial morphological feature vector, the aforementioned acoustic prosodic feature vector, and the aforementioned behavioral dynamics feature vector, attribution analysis is performed to generate a real-time state analysis report containing state attribution descriptions. The learning plan is dynamically adjusted based on the real-time status analysis report.

Images