An education evaluation method and system based on multi-modal time sequence regularization

Abstract

The application relates to the field of education evaluation and data processing, and discloses an education evaluation method and system based on multi-modal time sequence normalization, which comprises the following steps: a data access module receives classroom teaching audio and video data, objective test scores and subjective evaluation texts; an algorithm engine module performs time sequence alignment and feature extraction on the audio and video data, respectively generates a teaching stimulus intensity sequence and a cognitive load micro-motion sequence, calculates a text semantic complexity index to generate a dynamic time window coefficient, performs dynamic time normalization calculation on the two-end sequences based on the coefficient to generate a rhythm mismatch index, and an evaluation output module performs polarity reasoning on the evaluation texts by using a recurrent neural network, combines the test scores to generate a static basic score, and finally performs exponential calibration operation on the static basic score according to the rhythm mismatch index to generate a comprehensive subject score and output the result. The application breaks the limitation of a fixed constraint window and improves the accuracy of multi-modal education evaluation.

Description

Technical Field

[0001] This invention relates to the field of educational assessment and data processing, and in particular to an educational assessment method and system based on multimodal temporal regularization. Background Technology

[0002] With the deepening of educational informatization, teaching quality assessment is gradually evolving from traditional manual lectures and paper-and-pencil exams to data-driven intelligent methods.

[0003] A search revealed that the "Automatic Generation and Intelligent Analysis Method for Teaching Evaluation Reports Driven by Multimodal Data" published in Chinese Patent Announcement No. CN120046863A collects multimodal data, including student exam scores, classroom interaction data, learning behavior data, and teacher feedback data, from academic management systems and online learning platforms. After standardization and cleaning, an evaluation model is constructed through a multi-layer neural network, and an adaptive weighted scoring system is used to generate teaching evaluation reports. This solution improves the automation level and data coverage dimensions of teaching evaluation to a certain extent.

[0004] In addition, according to the search, the "Classroom Attention Assessment System and Method Based on Multimodal Data Perception" disclosed in Chinese Patent Announcement No. CN122335486A automatically marks the time sequence nodes of teaching links by using the voiceprint features, speech rate changes, pause duration and semantic keywords of the teacher's teaching voice, and outputs the attention data of individual students and the class as a whole in each teaching link, thus realizing the segmented quantitative assessment of classroom attention.

[0005] However, the aforementioned existing technologies still have the following shortcomings:

[0006] The data sources are mainly post-event statistical data from the academic affairs system or single-dimensional student attention data. There is a lack of joint modeling of the dynamic temporal relationship between teaching stimuli from the teacher's side and cognitive responses from the student's side, and it is impossible to quantify the degree of matching between teaching pace and student cognitive load.

[0007] Classroom timing analysis often uses fixed windows or pre-defined segment divisions, without adaptively adjusting the tolerance for time deviation based on the semantic complexity of the teaching content, resulting in a systematic bias in the accuracy of rhythm matching assessment for teaching segments of different difficulty.

[0008] The evaluation and scoring mechanisms are mostly static weighted integration models, which have not established an exponential decay mapping relationship between the mismatch of teaching rhythm and teaching quality, making it difficult to accurately reflect the substantial impact of classroom timing misalignment on the overall teaching effect.

[0009] Therefore, there is an urgent need for an intelligent education assessment system that can dynamically match the temporal features of both teachers and students, adaptively adjust assessment parameters based on semantic complexity, and calibrate scores based on rhythm mismatch, so as to improve the accuracy and adaptability of teaching quality assessment. Summary of the Invention

[0010] To overcome the above deficiencies, this invention provides an educational assessment method and system based on multimodal temporal regularization. It aims to improve the shortcomings of existing educational assessment systems, such as the lack of an absolute time reference leading to imprecise temporal fusion of multimodal data, the use of a fixed constraint window resulting in low accuracy of rhythm matching, and the lack of a calibration and penalty mechanism for dynamic temporal misalignment, which make it difficult to accurately reflect the quality of real teaching interactions.

[0011] To achieve the above objectives, the present invention adopts the following technical solution: an educational assessment method based on multimodal temporal regularization, comprising the following steps: receiving classroom teaching audio and video data, objective test scores, and subjective evaluation text;

[0012] The classroom teaching audio and video data is decoded and separated into independent video frame sequence data and audio stream data, and the audio stream data is converted into discrete text data fragments.

[0013] Perform temporal alignment and feature extraction on the independent video frame sequence data and the audio stream data to obtain the original stimulus parameters from the teacher's end and the physiological feature vector from the student's end;

[0014] The original stimulus parameters from the teacher's end are input into a pre-trained deep autoencoder to perform a dimensionality reduction mapping operation, generating a corresponding teaching stimulus intensity sequence from the teacher's end.

[0015] The physiological feature vector of the student terminal is input into a pre-trained feature fusion multilayer perceptron to perform nonlinear spatial mapping operation, thereby generating a corresponding cognitive load micro-motion sequence for the student terminal;

[0016] Natural language processing is performed on the text data fragment to calculate the semantic complexity index, and a dynamic time window coefficient is generated based on the semantic complexity index.

[0017] Based on the dynamic time window coefficient, dynamic time warping calculation is performed on the teaching stimulus intensity sequence and the cognitive load micro-motion sequence to obtain the global cumulative cost and average phase deviation, and then input into a pre-trained classification and evaluation network to perform mapping and normalization processing to generate a rhythm mismatch index.

[0018] The subjective evaluation text is input into a configured recurrent neural network to extract text feature sequences and perform polarity inference operations, outputting a sentiment polarity classification probability value. The objective test score and the configured weight coefficients are then combined to perform a weighted calculation to generate a static basic score.

[0019] An exponential calibration operation is performed on the static basic score based on the rhythm mismatch index to generate a comprehensive subject score, and the final evaluation result is output based on the comprehensive subject score.

[0020] As a further description of the above technical solution:

[0021] The process involves performing temporal alignment and feature extraction on the independent video frame sequence data and the audio stream data to obtain the original stimulus parameters from the teacher's end and the physiological feature vector from the student's end. The original stimulus parameters from the teacher's end are then input into a pre-trained deep autoencoder to perform dimensionality reduction mapping operations, generating a corresponding teaching stimulus intensity sequence for the teacher's end.

[0022] The independent video frame sequence data and the hardware physical timestamp carried in the audio stream data are analyzed, and an absolute time coordinate system is established with the hardware physical timestamp as the calculation basis.

[0023] According to a preset baseline sampling frame rate constant, the independent video frame sequence data and the audio stream data are resampled at equal intervals, and the frame data with equal time spans are extracted and combined to construct a discrete time step sequence.

[0024] Image spatial region calibration is performed on the independent video frame sequence data to separate the screen frames corresponding to the front area of ​​the classroom and the screen frames corresponding to the student seat area;

[0025] The screen frames corresponding to the front-end area of ​​the classroom and the corresponding audio stream data are aggregated and encapsulated to generate a multimodal data source for the teacher, and the screen frames corresponding to the student seat area are extracted and encapsulated separately to generate a multimodal data source for the student.

[0026] For the speech stream data and video frame data corresponding to the discrete time step sequence of the multimodal data source on the teacher's end, the semantic information density value, speech emphasis feature vector and visual guidance value are extracted as the original stimulus parameters of the teacher's end.

[0027] The semantic information density value, the speech emphasis feature vector, and the visual guidance value are concatenated to construct a high-dimensional feature tensor, and the high-dimensional feature tensor is input into the configured deep autoencoder;

[0028] The deep autoencoder performs a dimensionality reduction mapping operation on the high-dimensional feature tensor to output scalar values, and then concatenates the scalar values ​​of each independent time node according to the chronological order of the discrete time step sequence to form the teaching stimulus intensity sequence.

[0029] As a further description of the above technical solution:

[0030] The step of inputting the student's physiological feature vector into a pre-trained feature fusion multilayer perceptron to perform nonlinear spatial mapping operations and generate a corresponding cognitive load micro-motion sequence for the student includes:

[0031] For the frame data of independent time nodes corresponding to the discrete time step sequence in the multimodal data source of the student terminal, extract the micro-saccade occurrence frequency value, core motion unit switching frequency value, and posture micro-amplitude value.

[0032] The physiological feature vector is constructed by concatenating the micro-saccade occurrence frequency value, the core action unit switching frequency value, and the posture micro-amplitude value, and the physiological feature vector is input into the configured feature fusion multilayer perceptron.

[0033] The feature fusion multilayer perceptron performs a nonlinear spatial mapping operation on the physiological feature vector to output a fused scalar value, and then splices the fused scalar values ​​of each independent time node in the order of the discrete time step sequence to form the cognitive load micro-motion sequence.

[0034] As a further description of the above technical solution:

[0035] The extraction of semantic information density values, speech emphasis feature vectors, and visual guidance values ​​includes:

[0036] The semantic information density value is generated by combining the distribution probability of each word element contained in the text data segment within the independent time node with the preset vocabulary space, and performing logarithmic weighted summation calculation on the distribution probability according to the Shannon information entropy principle.

[0037] The speech stream data is subjected to time-domain and frequency-domain transformation operations to generate a frequency-domain feature sequence. The fundamental frequency value of the speech signal is extracted from the frequency-domain feature sequence, and the root mean square value of the short-time energy of the speech stream data amplitude is calculated in the time domain. The fundamental frequency value and the root mean square value of the short-time energy are concatenated to generate the speech emphasis feature vector.

[0038] Dense optical flow calculation is performed between adjacent frame data on the time axis to extract the pixel motion velocity vector field in the frame data. The first derivative operation of the time dimension is performed on the pixel motion velocity vector field to generate an optical flow acceleration vector. The Euclidean scalar norm of the optical flow acceleration vector is calculated to generate the visual guidance degree value.

[0039] As a further description of the above technical solution:

[0040] The step of performing natural language processing on the text data fragment, calculating the semantic complexity index, and generating a dynamic time window coefficient based on the semantic complexity index includes:

[0041] Named entity recognition is performed on the text data fragment to generate an entity set containing candidate words. The entity set is then matched and traversed with the knowledge concept nodes within a preset subject knowledge graph to count the number of valid entities. The ratio of the number of valid entities to the total number of words contained in the text data fragment is calculated to generate an entity mapping density value.

[0042] The text data fragment is subjected to dependency parsing operations to construct a syntactic dependency tree containing hierarchical topological logic, and the maximum number of connected edges extending from the root node to the terminal leaf node of the syntactic dependency tree is calculated to generate a numerical value of syntactic structure depth.

[0043] The entity mapping density value is multiplied by the configured entity density weight coefficient to obtain a first product value, the syntactic structure depth value is multiplied by the configured syntactic depth weight coefficient to obtain a second product value, and an addition aggregation operation is performed on the first product value and the second product value to generate the semantic complexity index.

[0044] The semantic complexity index is multiplied by the configured window scaling adjustment coefficient to obtain the time constraint extension variable. The time constraint extension variable is added by the configured basic allowable delay window step constant to obtain the intermediate value of the time window. The intermediate value of the time window is rounded up to convert it into discrete integer boundary parameters. The dynamic time window coefficient is generated based on the discrete integer boundary parameters.

[0045] As a further description of the above technical solution:

[0046] Based on the dynamic time window coefficient, dynamic time warping calculation is performed on the teaching stimulus intensity sequence and the cognitive load micro-motion sequence to obtain the global cumulative cost and average phase deviation, which are then input into a pre-trained classification and evaluation network for mapping and normalization processing to generate a rhythm mismatch index, including:

[0047] Normalization is performed on the teaching stimulus intensity sequence and the cognitive load micro-motion sequence, and the absolute difference of continuous scalar values ​​at different independent time nodes of the normalized sequence is calculated to construct a two-dimensional distance matrix.

[0048] The dynamic time window coefficient is set as a constraint boundary condition. A cumulative regularization cost matrix is ​​constructed within the range of the difference between the horizontal and vertical coordinates defined by the constraint boundary condition. For the matrix coordinate nodes that exceed the range defined by the constraint boundary condition, the corresponding numerical parameters are set to infinity.

[0049] Dynamic programming optimization is performed within the cumulative normalization cost matrix to find the minimum cumulative cost path from the starting point to the ending point of the connecting matrix coordinate system.

[0050] The global cumulative cost is generated by extracting the values ​​recorded at the end coordinate nodes of the minimum cumulative cost path, and the average phase deviation is generated by averaging the absolute values ​​of the differences between the horizontal and vertical coordinates of all nodes included in the minimum cumulative cost path.

[0051] The global cumulative cost and the average phase deviation are concatenated to construct a cost feature vector. The cost feature vector is then input into a configured classification evaluation network to perform mapping and nonlinear activation function normalization processing to generate the rhythm mismatch index.

[0052] As a further description of the above technical solution:

[0053] The subjective evaluation text includes teacher comments, student comments, and parent comments. The subjective evaluation text is input into a configured recurrent neural network to extract text feature sequences and perform polarity inference operations, outputting a sentiment polarity classification probability value. This value is then combined with the objective exam score and configured weighting coefficients to perform a weighted calculation, generating a static basic score, including:

[0054] Semantic-level preprocessing and character correction operations are performed on the subjective evaluation text, and the identity of the submitting subject of the subjective evaluation text is extracted and bound as the corresponding original source tag to generate a standard evaluation text set.

[0055] The standard evaluation text set is input into the configured recurrent neural network engine, the text feature sequence of the standard evaluation text set is extracted, and polarity inference operation is performed on the text feature sequence in combination with the original source label, and the sentiment polarity classification probability value is output.

[0056] Extract the target probability value representing positive emotional polarity from the emotional polarity classification probability values, multiply the target probability value by a preset full score evaluation base, and calculate the teacher evaluation score corresponding to the teacher's comment text, the student evaluation score corresponding to the student's comment text, and the parent evaluation score corresponding to the parent's comment text.

[0057] The objective test score is multiplied by the configured test score weighting coefficient to obtain the first score product. The teacher evaluation score, the student evaluation score, and the parent evaluation score are multiplied by the configured teacher rating weighting coefficient, student rating weighting coefficient, and parent rating weighting coefficient to obtain the second score product, the third score product, and the fourth score product, respectively.

[0058] An additive aggregation operation is performed on the first score product, the second score product, the third score product, and the fourth score product to generate the static basic score.

[0059] As a further description of the above technical solution:

[0060] The process of performing an exponential calibration operation on the static baseline score based on the rhythm mismatch index to generate a comprehensive subject score, and outputting the final evaluation result based on the comprehensive subject score, includes:

[0061] The rhythm mismatch index is multiplied by the mismatch penalty sensitivity coefficient pre-configured within the system to obtain the penalty index variable, and the natural exponential function is performed on the penalty index variable to generate the exponential decay multiplier of the decay mapping.

[0062] The static basic score is multiplied and calibrated with the exponential decay multiplier to generate the comprehensive subject score;

[0063] Perform a size comparison operation between the comprehensive subject score and the numerical boundaries contained in the subject quality rating interval set to determine the target numerical interval in which the comprehensive subject score falls;

[0064] Extract the intelligent assessment level label mapped to the target numerical range, encapsulate the comprehensive subject score and the intelligent assessment level label into the final assessment result, and archive and save it.

[0065] As a further description of the above technical solution:

[0066] After receiving classroom teaching audio and video data, objective test scores, and subjective evaluation texts, it also includes:

[0067] Identify non-standard control characters and semantically meaningless garbled characters within the subjective evaluation text, and perform character truncation and deletion operations;

[0068] The system identifies null values ​​in the objective test scores and sends a data retransmission request signal to the corresponding input terminal when the null value is identified.

[0069] The objective test scores are converted into a uniform precision floating-point numerical format, and the subjective evaluation text is standardized by character encoding to convert it into a universal multi-octet character set format.

[0070] As a further description of the above technical solution:

[0071] A multimodal time-series regularization-based educational assessment system, used for a multimodal time-series regularization-based educational assessment method, the system comprising:

[0072] The data access module is used to receive classroom teaching audio and video data, objective test scores, and subjective evaluation texts;

[0073] The algorithm engine module, communicatively connected to the data access module, internally includes a preprocessing unit, a feature extraction unit, a semantic parsing unit, and a temporal warping unit. The preprocessing unit performs decoding and separation operations on the classroom teaching audio-visual data, splitting it into independent video frame sequence data and audio stream data, and converting the audio stream data into discrete text data fragments. The feature extraction unit performs temporal alignment and feature extraction on the independent video frame sequence data and audio stream data, obtaining the teacher's original stimulus parameters and the student's physiological feature vector. The teacher's original stimulus parameters are input into a pre-trained deep autoencoder to perform dimensionality reduction mapping operations, generating corresponding teacher... The teaching stimulus intensity sequence of the student terminal is used to input the physiological feature vector of the student terminal into a pre-trained feature fusion multilayer perceptron to perform nonlinear spatial mapping operation, thereby generating a corresponding cognitive load micro-motion sequence of the student terminal. The semantic parsing unit is used to perform natural language processing on the text data segment, calculate the semantic complexity index, and generate a dynamic time window coefficient based on the semantic complexity index. The time warping unit is used to perform dynamic time warping calculation on the teaching stimulus intensity sequence and the cognitive load micro-motion sequence based on the dynamic time window coefficient to obtain the global cumulative cost and average phase deviation, and input them into a pre-trained classification evaluation network to perform mapping and normalization processing to generate a rhythm mismatch index.

[0074] The evaluation output module, which is communicatively connected to the algorithm engine module, is used to input the subjective evaluation text into a configured recurrent neural network to extract text feature sequences and perform polarity inference operations, output sentiment polarity classification probability values, perform weighted calculations by combining the objective test scores with configured weight coefficients to generate a static basic score, and perform exponential calibration operations on the static basic score according to the rhythm mismatch index to generate a comprehensive subject score, and output the final evaluation result based on the comprehensive subject score.

[0075] The present invention has the following beneficial effects:

[0076] 1. In this invention, the feature extraction unit parses the hardware physical timestamps carried in the independent video frame sequence data and audio stream data. The audio and video acquisition device writes the hardware physical timestamps into the underlying media information protocol during the signal acquisition stage according to the global clock pulse maintained by the system. The feature extraction unit establishes a unified absolute time coordinate system based on the hardware physical timestamps to offset the asynchronous transmission delays generated by the wide-angle video acquisition unit, local video acquisition unit and omnidirectional audio acquisition unit during signal transcoding. By constructing an absolute coordinate system based on the global clock and physical timestamps, the asynchronous delays of multiple sensors in the acquisition and transcoding process are effectively eliminated, ensuring the time rigor of multimodal data fusion.

[0077] 2. In this invention, the semantic parsing unit performs an up-rounding operation on the middle value of the time window, converting the middle value of the time window into discrete integer boundary parameters. Based on the discrete integer boundary parameters, it generates dynamic time window coefficients for corresponding independent time nodes. The dynamic time window coefficients define the upper limit of the maximum allowable time offset window in the dynamic time warping algorithm. After the semantic parsing unit completes the calculation of the dynamic time window coefficients for all independent time nodes under the discrete time step sequence, it directly outputs the dynamic time window coefficients to the time warping unit, providing time range constraints for subsequent algorithm sequence alignment operations. This breaks the limitation of the traditional DTW algorithm using a fixed constraint window, enabling the time offset tolerance to be dynamically and adaptively adjusted according to the semantic complexity of the current teaching content, significantly improving the fault tolerance and accuracy of audio and video rhythm matching.

[0078] 3. In this invention, the evaluation output module performs a multiplication operation on the rhythm mismatch index and the mismatch penalty sensitivity coefficient to obtain the penalty index variable, and then performs a natural exponential function calculation to generate the exponential decay multiplier. The evaluation output module performs a multiplication calibration operation on the static base score and the exponential decay multiplier, and uses the penalty mechanism caused by the dynamic misalignment of the time sequence to adjust the static base score numerically downward, generating a comprehensive subject score. Attached Figure Description

[0079] Figure 1This is a diagram showing the functional modules and internal unit logic structure of the system of the present invention;

[0080] Figure 2 This is a diagram illustrating the overall execution steps of the multimodal education assessment process of this invention.

[0081] Figure 3 This is a data flow diagram for feature extraction and sequence generation in this invention;

[0082] Figure 4 This is a flowchart of the semantic parsing and dynamic window generation process of the present invention;

[0083] Figure 5 This is a flowchart of the dynamic time warping and rhythm mismatch calculation process of the present invention;

[0084] Figure 6 This is a flowchart of the comprehensive evaluation, scoring, and rating process for this invention. Detailed Implementation

[0085] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0086] Reference Figures 1-6 This invention provides an educational assessment system based on multimodal temporal regularization, including a data access module, an algorithm engine module, and an assessment output module.

[0087] The deep learning-based intelligent education assessment system adopts a distributed hardware architecture in its physical deployment, which is divided into edge acquisition nodes deployed at the teaching site and cloud computing center deployed at the back end. The system has several input terminals (such as teacher input terminals, student input terminals, and parent input terminals) for receiving multi-dimensional external evaluation data.

[0088] The edge acquisition node includes audio and video acquisition devices, which perform real-time recording of the teaching scene's images and sounds to acquire classroom teaching audio and video data. The cloud computing center provides computing power support and runs the data access module, algorithm engine module, and evaluation output module.

[0089] The data access module communicates with the audio and video acquisition equipment to receive classroom teaching audio and video data. The data access module also receives objective test scores and subjective evaluation texts sent from external input terminals. The data access module has a data storage unit inside. The data storage unit performs classification storage and format standardization operations on classroom teaching audio and video data, objective test scores and subjective evaluation texts, and establishes a data interaction interface.

[0090] The algorithm engine module retrieves classroom teaching audio and video data from the data access module. The algorithm engine module internally includes a feature extraction unit, a semantic parsing unit, and a temporal regularization unit.

[0091] The feature extraction unit performs frame-level alignment processing on the audio and video data of classroom teaching, and extracts multi-dimensional features (including semantic, speech and visual multi-dimensional parameters) from the teacher's side and physiological response features from the student's side respectively. The feature extraction unit performs dimensionality reduction mapping operation on the extracted multi-dimensional features from the teacher's side and outputs the teaching stimulus intensity sequence. The feature extraction unit performs nonlinear fusion operation on the extracted physiological response features from the student's side and outputs the cognitive load micro-motion sequence.

[0092] The semantic parsing unit retrieves text data fragments extracted by the feature extraction unit, performs natural language processing, extracts entity mapping density and syntactic structure depth, calculates the semantic complexity index, and generates dynamic time window coefficients based on the semantic complexity index.

[0093] The temporal warping unit receives the teaching stimulus intensity sequence and cognitive load micro-motion sequence output by the feature extraction unit. The temporal warping unit combines the dynamic time window coefficient generated by the semantic parsing unit to perform dynamic time warping calculation, solve the global cumulative cost and average phase deviation of sequence matching, and output the rhythm mismatch index.

[0094] The evaluation output module receives the rhythm mismatch index output by the timing warping unit and extracts the objective test scores and subjective evaluation texts stored in the data storage unit. The evaluation output module performs sentiment polarity analysis and weighted calculation on the objective test scores and subjective evaluation texts to generate a static basic score. The evaluation output module performs an exponential calibration operation on the static basic score based on the rhythm mismatch index to generate a comprehensive subject score and output the evaluation result.

[0095] The specific steps of a deep learning-based intelligent education assessment system in executing a multimodal education assessment process are as follows.

[0096] The audio and video acquisition equipment records the images and sounds of the teaching scene, generating classroom teaching audio and video data. The audio and video acquisition equipment includes a wide-angle video acquisition unit facing the front area of ​​the classroom and a local video acquisition unit arranged in the student area, and is also equipped with an omnidirectional audio acquisition unit. The audio and video acquisition equipment performs continuous signal acquisition operations according to a preset fixed frame rate and fixed audio sampling rate, converts the acquired physical analog signals into digital signals, and encapsulates them to form classroom teaching audio and video data.

[0097] The data access module communicates with the audio and video acquisition equipment to receive classroom teaching audio and video data. The data access module receives objective test scores and subjective evaluation texts sent by the input terminals. The input terminals include teacher input terminals, student input terminals, and parent input terminals. The objective test scores include digitally formatted periodic subject test scores. The subjective evaluation texts include teacher comments submitted by the teacher input terminal, student comments submitted by the student input terminal, and parent comments submitted by the parent input terminal. The input terminals package and send the objective test scores and subjective evaluation texts to the data access module through network transmission protocols such as Hypertext Transfer Protocol (HTTP), Secure Hypertext Transfer Protocol (HTTPS), or Transmission Control Protocol / Internet Protocol (TCP / IP).

[0098] After acquiring the data, the data access module performs preliminary data filtering operations at the network transmission layer for the objective exam scores and subjective evaluation text. The data access module identifies non-standard control characters and semantically meaningless garbled characters in the subjective evaluation text caused by network packet loss or communication anomalies, and performs character truncation and deletion operations. The data access module identifies null data in the objective exam scores and sends a data retransmission request signal to the corresponding input terminal when null data is identified.

[0099] The data storage unit inside the data access module receives classroom teaching audio and video data, filtered objective test scores, and filtered subjective evaluation text. It performs unified format conversion and classified storage operations. The data storage unit performs decoding and separation operations on the classroom teaching audio and video data, splitting it into independent video frame sequence data and audio stream data. The data storage unit converts the objective test scores into a unified precision floating-point numerical format and performs character encoding standardization operations on the subjective evaluation text, converting it into a universal multi-octet encoded character set format. The data storage unit writes the standardized classroom teaching audio and video data, objective test scores, and subjective evaluation text into an independent physical partition to build a data interaction interface.

[0100] The algorithm engine module retrieves independent video frame sequence data and audio stream data from the data storage unit through the data interaction interface. The feature extraction unit inside the algorithm engine module receives the independent video frame sequence data and audio stream data and performs physical frame-level timing alignment operations.

[0101] The feature extraction unit parses the hardware physical timestamps carried in the independent video frame sequence data and audio stream data. The audio and video acquisition device writes the hardware physical timestamps into the underlying media information protocol during the signal acquisition stage according to the global clock pulse maintained by the system. The feature extraction unit establishes a unified absolute time coordinate system based on the hardware physical timestamps to offset the asynchronous transmission delays generated by the wide-angle video acquisition unit, local video acquisition unit and omnidirectional audio acquisition unit during signal transcoding.

[0102] The feature extraction unit sets the global baseline sampling frame rate constant of the system, and performs equal-interval resampling on independent video frame sequence data and audio stream data according to the baseline sampling frame rate constant during the duration of classroom teaching. It extracts frame data combinations with equal time spans to construct a discrete time step sequence, which contains a finite number of independent time nodes arranged in ascending order of absolute time sequence.

[0103] The feature extraction unit performs image spatial region calibration on independent video frame sequence data, separating physical space images of different dimensions. Specifically, based on the physical deployment location of the equipment, the feature extraction unit identifies the frames captured by the wide-angle video acquisition unit as the classroom front area image and the frames captured by the local video acquisition unit as the student seating area image. The feature extraction unit aggregates and encapsulates the frames associated with the classroom front area with the corresponding audio stream data to generate a multimodal data source for the teacher. The feature extraction unit extracts and encapsulates the frames associated with the student seating area separately to generate a multimodal data source for the student.

[0104] The feature extraction unit maps the teacher-side and student-side multimodal data sources to independent time nodes in the discrete time step sequence according to absolute time synchronization. Through numerical truncation and interpolation compensation operations, the feature extraction unit constrains the teacher-side and student-side multimodal data sources to maintain a consistent time dimension at any independent time node in the discrete time step sequence. The feature extraction unit completes strict alignment of the two-end data of the teacher-side and student-side multimodal data sources, generates an aligned dual-end synchronized data structure, and stores it in the system cache inside the algorithm engine module. Based on the global clock and physical timestamp, an absolute coordinate system is constructed, which effectively eliminates the asynchronous delay of multiple sensors in the acquisition and transcoding process, and ensures the temporal rigor of multimodal data fusion.

[0105] The feature extraction unit reads the aligned dual-end synchronous data structure from the system cache inside the algorithm engine module, extracts the teacher-side multimodal data source contained in the dual-end synchronous data structure, and extracts multi-dimensional raw stimulus parameters (specifically including semantic information density values, speech emphasis feature vectors, and visual guidance values) from the speech stream data and image frame data of the independent time nodes of the corresponding discrete time step sequence in the teacher-side multimodal data source.

[0106] For speech stream data, the feature extraction unit uses an automatic speech recognition algorithm (in this embodiment, the automatic speech recognition algorithm can adopt existing mature models such as Whisper; this invention does not limit its specific internal network structure) to convert the speech stream data into discrete text data fragments. It then combines this with a preset vocabulary space to statistically analyze the distribution probability of each word in the text data fragments at independent time nodes. Based on the Shannon information entropy principle, it performs a logarithmic weighted summation calculation on the distribution probability to generate a semantic information density value that quantifies the discreteness of the text information distribution. This calculation process is specifically represented as follows:

[0107]

[0108] in, The semantic information density value represents the summation over all lexical units within an independent time node. Represents the first time point within that independent time node A specific word element, For the first The probability distribution of each word at an independent time point, and the sum of the probability distributions of all words is 1.

[0109] The feature extraction unit performs time-domain and frequency-domain transformation operations on the speech stream data to generate a frequency-domain feature sequence. Within the frequency-domain feature sequence, it extracts the fundamental frequency value of the speech signal and calculates the root mean square value of the short-time energy of the speech stream data amplitude in the time domain. Then, it concatenates the fundamental frequency value and the root mean square value of the short-time energy to generate a speech emphasis feature vector.

[0110] For frame data, the feature extraction unit performs dense optical flow calculations between adjacent frame data on the time axis (which can be implemented using existing optical flow algorithms such as Farneback), extracts the pixel motion velocity vector field in the frame data, performs the first derivative operation of the time dimension on the pixel motion velocity vector field to generate an optical flow acceleration vector, and finally calculates the Euclidean scalar norm of the optical flow acceleration vector to generate a visual guidance degree value.

[0111] The feature extraction unit is internally configured with a deep autoencoder. The feature extraction unit performs uniform dimension normalization on the semantic information density value, speech emphasis feature vector, and visual guidance value. The normalized semantic information density value, speech emphasis feature vector, and visual guidance value are concatenated to construct a high-dimensional feature tensor, which is input into the encoding network of the deep autoencoder. The encoding network of the deep autoencoder performs a dimensionality reduction mapping operation based on a non-linear activation function on the high-dimensional feature tensor, outputting a low-dimensional latent vector. Subsequently, the low-dimensional latent vector is mapped to the scalar value of the corresponding independent time node through the built-in fully connected regression layer. The feature extraction unit concatenates the scalar values ​​of each independent time node according to the chronological order of the discrete time step sequence to form a one-dimensional teaching stimulus intensity sequence.

[0112] After constructing a one-dimensional sequence of teaching stimulus intensity, the feature extraction unit extracts the student-side multimodal data source from the dual-end synchronous data structure, and performs physiological response feature extraction operations on the frame data of the independent time nodes of the discrete time step sequence in the student-side multimodal data source.

[0113] The feature extraction unit performs gaze tracking calculations on the frame data (using existing open-source eye-tracking algorithms) to locate the pupil center coordinate trajectory, calculates the displacement change of the pupil center coordinate trajectory and compares it with a preset micro-motion threshold, thereby statistically analyzing the frequency of micro-saccades within independent time nodes. Simultaneously, the feature extraction unit runs a facial motion coding model (such as the existing FACS model) to extract facial feature points, maps the core motion unit state based on the facial feature points, and statistically analyzes the core motion unit state switching frequency within independent time nodes. In addition, the feature extraction unit uses a pose estimation network (such as the existing OpenPose model) to identify body skeleton key points in the frame data, calculates the spatial displacement values ​​of the body skeleton key point coordinates between adjacent frame data on the time axis, and performs an averaging operation on the spatial displacement values ​​to generate pose micro-variation values.

[0114] The feature extraction unit is equipped with a feature fusion multilayer perceptron. The feature extraction unit concatenates the micro-saccade frequency values, core action unit switching frequency values, and posture micro-amplitude values ​​to construct a physiological feature vector, which is then input into the feature fusion multilayer perceptron. The feature fusion multilayer perceptron performs a nonlinear spatial mapping operation on the physiological feature vector and outputs the fused scalar values ​​corresponding to the independent time nodes. The feature extraction unit concatenates the fused scalar values ​​of each independent time node according to the chronological order of the discrete time step sequence to form a one-dimensional cognitive load micro-motion sequence.

[0115] After the feature extraction unit constructs a one-dimensional cognitive load micro-motion sequence, the semantic parsing unit retrieves text data fragments of independent time nodes of the corresponding discrete time step sequence generated by the feature extraction unit. The semantic parsing unit internally has a fixed configuration of a preset subject knowledge graph, which contains concept nodes representing subject domain knowledge and a network data structure that maps the relationships between concept nodes.

[0116] The semantic parsing unit performs named entity recognition on the text data fragment, extracts candidate words contained in the text data fragment to form an entity set, and performs character matching traversal operation on the candidate words contained in the entity set with the knowledge concept nodes in the preset subject knowledge graph. The number of valid entities successfully mapped to the preset subject knowledge graph is counted. The semantic parsing unit extracts the total number of words contained in the text data fragment, performs a division operation, calculates the ratio of the number of valid entities to the total number of words, and generates an entity mapping density value.

[0117] The semantic parsing unit performs dependency parsing operations on the text data fragments, extracts the grammatical master-detail relationship between each word in the text data fragment, and constructs a syntactic dependency tree containing hierarchical topological logic based on the grammatical master-detail relationship. The semantic parsing unit runs a depth-first search algorithm to traverse the hierarchical topological logic of the syntactic dependency tree, calculates the maximum number of connected edges extending from the root node to the terminal leaf node of the syntactic dependency tree, and generates a numerical value of the syntactic structure depth.

[0118] It should be noted that the various weight coefficients involved in the embodiments of the present invention (such as entity density weight coefficient, syntactic depth weight coefficient, etc.) as well as the system hyperparameters such as the basic allowable delay window step constant, window scaling adjustment coefficient, and mismatch penalty sensitivity coefficient mentioned later, can all be pre-calibrated by using the analytic hierarchy process combined with expert experience on the historical education evaluation dataset, or by using multiple linear regression fitting to obtain the optimal system configuration and then storing it in the system.

[0119] The semantic parsing unit stores the entity density weight coefficient and syntactic depth weight coefficient pre-configured by the system. The semantic parsing unit performs a multiplication calculation on the entity mapping density value and the entity density weight coefficient to obtain the first product value, and performs a multiplication calculation on the syntactic structure depth value and the syntactic depth weight coefficient to obtain the second product value. The semantic parsing unit performs an addition aggregation operation on the first product value and the second product value to generate the semantic complexity index corresponding to the independent time node.

[0120] After generating the semantic complexity index for the corresponding independent time node, the semantic parsing unit retrieves the basic allowed delay window step constant and window scaling adjustment coefficient pre-configured by the system. The basic allowed delay window step constant defines the baseline time offset margin under the normal semantic environment, and the window scaling adjustment coefficient defines the mapping slope of the time offset boundary as the semantic dimension changes.

[0121] The semantic parsing unit performs a multiplication operation on the semantic complexity index and the window scaling adjustment coefficient to obtain the time constraint extension variable, and performs an addition operation on the time constraint extension variable and the basic allowable delay window step constant to obtain the floating-point intermediate value of the time window. Then, the semantic parsing unit performs an up-rounding operation on the intermediate value of the time window to convert the intermediate value of the time window into discrete integer boundary parameters, and generates the dynamic time window coefficients for the corresponding independent time nodes based on the discrete integer boundary parameters.

[0122] The dynamic time window coefficient defines the upper limit of the maximum allowable time offset window in the dynamic time warping algorithm. After the semantic parsing unit completes the calculation of the dynamic time window coefficient for all independent time nodes under the discrete time step sequence, it directly outputs the dynamic time window coefficient to the time warping unit, providing time range constraints for subsequent algorithm sequence alignment operations. This breaks the limitation of the traditional DTW algorithm using a fixed constraint window, allowing the time offset tolerance to be dynamically and adaptively adjusted according to the semantic complexity of the current teaching content, significantly improving the fault tolerance and accuracy of audio and video rhythm matching.

[0123] The temporal warping unit receives the teaching stimulus intensity sequence and cognitive load micro-motion sequence output by the feature extraction unit, and extracts the dynamic time window coefficients of the corresponding independent time nodes directly output by the semantic parsing unit.

[0124] After obtaining the teaching stimulus intensity sequence and the cognitive load micro-motion sequence, the temporal warping unit first performs min-max normalization on both to map them to the same numerical interval (e.g., [0,1]). Then, it extracts the continuous scalar values ​​contained in the normalized sequence and uses Euclidean distance or Manhattan distance algorithms to calculate the absolute difference of the continuous scalar values ​​at different independent time nodes of the two sequences, constructing a two-dimensional distance matrix. The temporal warping unit sets the dynamic time window coefficient as a constraint boundary condition. Within the range of horizontal and vertical coordinate differences defined by the constraint boundary condition, it extracts the absolute difference of the current coordinate node in the distance matrix and adds this absolute difference to the minimum cumulative cost value among the adjacent preceding horizontal node, vertical node, and diagonal node. The sum is then assigned as the cumulative warping cost of the current coordinate node, and the entire cumulative warping cost matrix is ​​constructed recursively in this way.

[0125] For matrix coordinate nodes that exceed the limits of the constraint boundary conditions, the temporal warping unit forces the numerical parameters of the corresponding matrix coordinate nodes to be set to infinity, thereby blocking the sequence alignment matching that exceeds the constraint boundary. Then, dynamic programming optimization is performed within the cumulative warping cost matrix to solve for the minimum cumulative cost path connecting the start and end points of the matrix coordinate system.

[0126] The timing warping unit extracts the values ​​recorded by the coordinate nodes at the end of the minimum cumulative cost path to generate the global cumulative cost, and calculates the average phase deviation by averaging the absolute values ​​of the differences between the horizontal and vertical coordinates of all nodes included in the minimum cumulative cost path.

[0127] The temporal warping unit is configured with a classification evaluation network. The temporal warping unit concatenates the global cumulative cost and the average phase deviation to construct a cost feature vector, and inputs the cost feature vector into the classification evaluation network. The temporal warping unit uses the classification evaluation network to perform fully connected weight mapping and nonlinear activation function normalization on the cost feature vector to generate a rhythm mismatch index with values ​​in the range of zero and one, and then passes the rhythm mismatch index to the evaluation output module.

[0128] After receiving the rhythm mismatch index from the timing warping unit, the evaluation output module reads the objective test scores and subjective evaluation texts from the data storage unit through the data interaction interface. The subjective evaluation texts include teacher comments, student comments, and parent comments, which are stored separately.

[0129] The evaluation output module performs semantic-level preprocessing and character correction operations on the subjective evaluation text before NLP (Natural Language Processing). It identifies and removes repeated punctuation marks and special control characters within the subjective evaluation text, replaces non-standard words with standard synonyms from a preset thesaurus, performs dependency parsing on the replaced subjective evaluation text, corrects erroneous grammatical structures to correct grammatical structures according to standard grammar mapping rules, and extracts and binds the submitter's identity (teacher, student, or parent) as the corresponding original source tag, ultimately generating a set of standard evaluation texts.

[0130] The evaluation output module is internally configured with a recurrent neural network engine. The evaluation output module inputs the standard evaluation text set into the recurrent neural network engine, which performs word segmentation and part-of-speech tagging on the standard evaluation text set, extracts the text feature sequence of the standard evaluation text set, and uses context state memory nodes combined with the original source labels to perform semantic polarity inference operation on the text feature sequence, outputting continuous sentiment polarity classification probability values.

[0131] The evaluation output module extracts the target probability value representing positive emotional polarity from the emotional polarity classification probability value, multiplies the target probability value (ranging from 0 to 1) with a preset full score base (e.g., 100 points), and linearly maps it to a percentage-based evaluation score, thereby calculating the teacher evaluation score for the corresponding teacher comment text, the student evaluation score for the corresponding student comment text, and the parent evaluation score for the corresponding parent comment text.

[0132] The evaluation output module simultaneously retrieves the weighted coefficients of the exam score, teacher evaluation, student evaluation, and parent evaluation, all stored in the system and weighted to a sum of one. The module multiplies the objective exam score with the exam score weight coefficient to obtain the first score product. It then multiplies the teacher evaluation score with the teacher evaluation weight coefficient, the student evaluation score with the student evaluation weight coefficient, and the parent evaluation score with the parent evaluation weight coefficient to obtain the second, third, and fourth score products, respectively. Finally, it performs an addition aggregation operation on the first, second, third, and fourth score products to generate a static basic score.

[0133] After generating the static base score, the evaluation output module extracts the rhythm mismatch index transmitted by the timing warping unit and retrieves the mismatch penalty sensitivity coefficient pre-configured within the system. The mismatch penalty sensitivity coefficient limits the decay mapping slope of the rhythm mismatch index to the evaluation score.

[0134] The evaluation output module multiplies the tempo mismatch index and the mismatch penalty sensitivity coefficient to obtain the penalty index variable, and then calculates the exponential decay multiplier using the natural exponential function. The specific calculation formula is as follows:

[0135]

[0136] in, It is the exponential decay multiplier. The rhythm mismatch index is mentioned above. Let the mismatch penalty sensitivity coefficient be, and set as follows: This ensures that the function is a decay mapping function.

[0137] The evaluation output module performs a multiplication calibration operation on the static basic score and the exponential decay multiplier, and uses the penalty mechanism caused by the dynamic misalignment of time series to adjust the static basic score downward to generate a comprehensive subject score.

[0138] The assessment output module internally stores a set of subject quality rating intervals. The assessment output module performs a size comparison operation between the comprehensive subject score and the numerical boundaries contained in the set of subject quality rating intervals to determine the target numerical interval in which the comprehensive subject score falls. The assessment output module extracts the intelligent assessment level label mapped to the target numerical interval and encapsulates the comprehensive subject score and the intelligent assessment level label into the final assessment result. The assessment output module outputs the final assessment result to the data storage unit for archiving and saving through the data interaction interface, thus completing the teaching assessment workflow.

[0139] Furthermore, to ensure the accurate output of the deep learning-based intelligent education assessment system, the various deep learning models and neural networks involved in this invention underwent a thorough training process before deployment. The specific training configuration is as follows:

[0140] For deep autoencoders: a combination of unsupervised learning and supervised fine-tuning is used for training. Unlabeled multi-dimensional raw stimulus parameters are extracted from historical classroom teaching videos and concatenated into high-dimensional feature tensor samples to construct the training set. The loss function in the unsupervised pre-training stage is set as the mean squared error (MSE) reconstruction loss function. The deep autoencoder consists of an encoder network and a decoder network. During training, the encoder network compresses the high-dimensional feature tensor into low-dimensional latent variables, and the decoder network attempts to restore the low-dimensional latent variables to the original high-dimensional features. The network weights are continuously adjusted through the backpropagation algorithm to minimize the reconstruction error. After training convergence, the decoder network is removed, and only the encoder network is retained. A fully connected regression layer is then cascaded on it. Supervised fine-tuning training enables the fully connected regression layer to have the reasoning ability to map low-dimensional latent variables into scalar values, which is used for dimensionality reduction mapping and scalar calculation during system inference.

[0141] For the feature fusion multilayer perceptron: supervised learning is used for training. Micro-saccades, facial feature switching, and posture displacement features are extracted from historical data as physiological feature vector samples. Educational experts quantify the "cognitive load state" corresponding to the above sample time periods into a 0-1 range as the ground truth. Mean squared error or Huber loss function is selected, and gradient descent optimization algorithm (such as Adam optimizer) is used to iteratively update the neuron weights of the multilayer fully connected layers inside the perceptron until the error between the fused scalar value output by the model and the expert score label tends to be minimized.

[0142] For the classification evaluation network: a supervised binary classification or regression learning mode is adopted to construct a historical sequence alignment dataset containing positive and negative samples. Positive samples represent scenarios with highly matched teacher-student interaction rhythms, while negative samples represent scenarios with severely mismatched rhythms. The global cumulative cost and average phase deviation generated by the DTW algorithm for each sample are extracted to construct the input training vector. The mismatch degree (0 to 1) is manually labeled as a label. The network structure includes an input layer, several hidden layers, and an output layer with a sigmoid activation function. The binary cross-entropy loss function is used to perform backward iterative training, enabling the network to have the reasoning ability to map the sequence alignment cost to the rhythm mismatch index.

[0143] For the recurrent neural network engine: Supervised training is performed using a bidirectional long short-term memory network (Bi-LSTM) or gated recurrent unit (GRU) architecture. A massive amount of open-source educational review texts and historical system archive comments are collected. The label dataset is composed of positive, neutral, and negative sentiment polarity labels performed manually. During training, the word segmentation sequence is first mapped to a continuous vector representation through a pre-trained word embedding model (such as Word2Vec), which is then input into the recurrent neural network engine. The error between the network output class probability and the true polarity label is calculated using the cross-entropy loss function. Backpropagation (BPTT) is performed along the time step to update the network parameters. After training, the network weights are fixed to support the subsequent output of sentiment polarity classification probability values.

[0144] Furthermore, at the physical implementation level, the deep learning-based intelligent education assessment system described in this embodiment of the invention can be embodied in one or more computer devices. The computer device includes at least a processor, a memory, and a communication interface. The memory is used to store computer programs. The processor executes the calculation and processing logic corresponding to the data access module, the algorithm engine module, and the assessment output module and their internal units by calling the computer programs stored in the memory, thereby realizing the above-mentioned multimodal education assessment method flow.

[0145] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. An educational assessment method based on multimodal temporal regularization, characterized in that, Includes the following steps: Receive classroom teaching audio and video data, objective test scores, and subjective evaluation texts; The classroom teaching audio and video data is decoded and separated into independent video frame sequence data and audio stream data, and the audio stream data is converted into discrete text data fragments. Perform temporal alignment and feature extraction on the independent video frame sequence data and the audio stream data to obtain the original stimulus parameters from the teacher's end and the physiological feature vector from the student's end; The original stimulus parameters from the teacher's end are input into a pre-trained deep autoencoder to perform a dimensionality reduction mapping operation, generating a corresponding sequence of teaching stimulus intensities from the teacher's end. The physiological feature vector of the student terminal is input into a pre-trained feature fusion multilayer perceptron to perform nonlinear spatial mapping operation, thereby generating a corresponding cognitive load micro-motion sequence for the student terminal; Natural language processing is performed on the text data fragment to calculate the semantic complexity index, and a dynamic time window coefficient is generated based on the semantic complexity index. Based on the dynamic time window coefficient, dynamic time warping calculation is performed on the teaching stimulus intensity sequence and the cognitive load micro-motion sequence to obtain the global cumulative cost and average phase deviation, and then input into a pre-trained classification evaluation network to perform mapping and normalization processing to generate a rhythm mismatch index. The subjective evaluation text is input into a configured recurrent neural network to extract text feature sequences and perform polarity inference operations to output the emotional polarity classification probability value. Combined with the objective test score and the configured weight coefficient, a weighted calculation is performed to generate a static basic score. An exponential calibration operation is performed on the static basic score based on the rhythm mismatch index to generate a comprehensive subject score, and the final evaluation result is output based on the comprehensive subject score.

2. The educational assessment method based on multimodal temporal regularization according to claim 1, characterized in that, The process involves performing temporal alignment and feature extraction on the independent video frame sequence data and the audio stream data to obtain the original stimulus parameters from the teacher's end and the physiological feature vector from the student's end. The original stimulus parameters from the teacher's end are then input into a pre-trained deep autoencoder to perform dimensionality reduction mapping operations, generating a corresponding teaching stimulus intensity sequence for the teacher's end. The independent video frame sequence data and the hardware physical timestamp carried in the audio stream data are analyzed, and an absolute time coordinate system is established with the hardware physical timestamp as the calculation basis. According to a preset baseline sampling frame rate constant, the independent video frame sequence data and the audio stream data are resampled at equal intervals, and the frame data with equal time spans are extracted and combined to construct a discrete time step sequence. Image spatial region calibration is performed on the independent video frame sequence data to separate the screen frames corresponding to the front area of ​​the classroom and the screen frames corresponding to the student seat area; The screen frames corresponding to the front-end area of ​​the classroom and the corresponding audio stream data are aggregated and encapsulated to generate a multimodal data source for the teacher, and the screen frames corresponding to the student seat area are extracted and encapsulated separately to generate a multimodal data source for the student. For the speech stream data and video frame data corresponding to the discrete time step sequence of the multimodal data source on the teacher's end, the semantic information density value, speech emphasis feature vector and visual guidance value are extracted as the original stimulus parameters of the teacher's end. The semantic information density value, the speech emphasis feature vector, and the visual guidance value are concatenated to construct a high-dimensional feature tensor, and the high-dimensional feature tensor is input into the configured deep autoencoder; The deep autoencoder performs a dimensionality reduction mapping operation on the high-dimensional feature tensor to output scalar values, and then concatenates the scalar values ​​of each independent time node according to the chronological order of the discrete time step sequence to form the teaching stimulus intensity sequence.

3. The educational assessment method based on multimodal temporal regularization according to claim 2, characterized in that, The step of inputting the student's physiological feature vector into a pre-trained feature fusion multilayer perceptron to perform nonlinear spatial mapping operations and generate a corresponding cognitive load micro-motion sequence for the student includes: For the frame data of independent time nodes corresponding to the discrete time step sequence in the multimodal data source of the student terminal, extract the micro-saccade occurrence frequency value, core motion unit switching frequency value, and posture micro-amplitude value. The physiological feature vector is constructed by concatenating the micro-saccade occurrence frequency value, the core action unit switching frequency value, and the posture micro-amplitude value, and the physiological feature vector is input into the configured feature fusion multilayer perceptron. The feature fusion multilayer perceptron performs a nonlinear spatial mapping operation on the physiological feature vector to output a fused scalar value, and then splices the fused scalar values ​​of each independent time node in the order of the discrete time step sequence to form the cognitive load micro-motion sequence.

4. The educational assessment method based on multimodal temporal regularization according to claim 2, characterized in that, The extraction of semantic information density values, speech emphasis feature vectors, and visual guidance values ​​includes: The semantic information density value is generated by combining the distribution probability of each word element contained in the text data segment within the independent time node with the preset vocabulary space, and performing logarithmic weighted summation calculation on the distribution probability according to the Shannon information entropy principle. The speech stream data is subjected to time-domain and frequency-domain transformation operations to generate a frequency-domain feature sequence. The fundamental frequency value of the speech signal is extracted from the frequency-domain feature sequence, and the root mean square value of the short-time energy of the speech stream data amplitude is calculated in the time domain. The fundamental frequency value and the root mean square value of the short-time energy are concatenated to generate the speech emphasis feature vector. Dense optical flow calculation is performed between adjacent frame data on the time axis to extract the pixel motion velocity vector field in the frame data. The first derivative operation of the time dimension is performed on the pixel motion velocity vector field to generate an optical flow acceleration vector. The Euclidean scalar norm of the optical flow acceleration vector is calculated to generate the visual guidance degree value.

5. The educational assessment method based on multimodal temporal regularization according to claim 1, characterized in that, The step of performing natural language processing on the text data fragment, calculating the semantic complexity index, and generating a dynamic time window coefficient based on the semantic complexity index includes: Named entity recognition is performed on the text data fragment to generate an entity set containing candidate words. The entity set is then matched and traversed with the knowledge concept nodes within a preset subject knowledge graph to count the number of valid entities. The ratio of the number of valid entities to the total number of words contained in the text data fragment is calculated to generate an entity mapping density value. The text data fragment is subjected to dependency parsing operations to construct a syntactic dependency tree containing hierarchical topological logic, and the maximum number of connected edges extending from the root node to the terminal leaf node of the syntactic dependency tree is calculated to generate a numerical value of syntactic structure depth. The entity mapping density value is multiplied by the configured entity density weight coefficient to obtain a first product value, the syntactic structure depth value is multiplied by the configured syntactic depth weight coefficient to obtain a second product value, and an addition aggregation operation is performed on the first product value and the second product value to generate the semantic complexity index. The semantic complexity index is multiplied by the configured window scaling adjustment coefficient to obtain the time constraint extension variable. The time constraint extension variable is added by the configured basic allowable delay window step constant to obtain the intermediate value of the time window. The intermediate value of the time window is rounded up to convert it into discrete integer boundary parameters. The dynamic time window coefficient is generated based on the discrete integer boundary parameters.

6. The educational assessment method based on multimodal temporal regularization according to claim 1, characterized in that, Based on the dynamic time window coefficient, dynamic time warping calculation is performed on the teaching stimulus intensity sequence and the cognitive load micro-motion sequence to obtain the global cumulative cost and average phase deviation, which are then input into a pre-trained classification and evaluation network for mapping and normalization processing to generate a rhythm mismatch index, including: Normalization is performed on the teaching stimulus intensity sequence and the cognitive load micro-motion sequence, and the absolute difference of continuous scalar values ​​at different independent time nodes of the normalized sequence is calculated to construct a two-dimensional distance matrix. The dynamic time window coefficient is set as a constraint boundary condition. A cumulative regularization cost matrix is ​​constructed within the range of the difference between the horizontal and vertical coordinates defined by the constraint boundary condition. For the matrix coordinate nodes that exceed the range defined by the constraint boundary condition, the corresponding numerical parameters are set to infinity. Dynamic programming optimization is performed within the cumulative normalization cost matrix to find the minimum cumulative cost path from the starting point to the ending point of the connecting matrix coordinate system. The global cumulative cost is generated by extracting the values ​​recorded at the end coordinate nodes of the minimum cumulative cost path, and the average phase deviation is generated by averaging the absolute values ​​of the differences between the horizontal and vertical coordinates of all nodes included in the minimum cumulative cost path. The global cumulative cost and the average phase deviation are concatenated to construct a cost feature vector. The cost feature vector is then input into a configured classification evaluation network to perform mapping and nonlinear activation function normalization processing to generate the rhythm mismatch index.

7. The educational assessment method based on multimodal temporal regularization according to claim 1, characterized in that, The subjective evaluation text includes teacher comments, student comments, and parent comments. The subjective evaluation text is input into a configured recurrent neural network to extract text feature sequences and perform polarity inference operations, outputting a sentiment polarity classification probability value. This value is then combined with the objective exam score and configured weighting coefficients to perform a weighted calculation, generating a static basic score, including: Semantic-level preprocessing and character correction operations are performed on the subjective evaluation text, and the identity of the submitting subject of the subjective evaluation text is extracted and bound as the corresponding original source tag to generate a standard evaluation text set. The standard evaluation text set is input into the configured recurrent neural network engine, the text feature sequence of the standard evaluation text set is extracted, and polarity inference operation is performed on the text feature sequence in combination with the original source label, and the sentiment polarity classification probability value is output. Extract the target probability value representing positive emotional polarity from the emotional polarity classification probability values, multiply the target probability value by a preset full score evaluation base, and calculate the teacher evaluation score corresponding to the teacher's comment text, the student evaluation score corresponding to the student's comment text, and the parent evaluation score corresponding to the parent's comment text. The objective test score is multiplied by the configured test score weighting coefficient to obtain the first score product. The teacher evaluation score, the student evaluation score, and the parent evaluation score are multiplied by the configured teacher rating weighting coefficient, student rating weighting coefficient, and parent rating weighting coefficient to obtain the second score product, the third score product, and the fourth score product, respectively. An additive aggregation operation is performed on the first score product, the second score product, the third score product, and the fourth score product to generate the static basic score.

8. The educational assessment method based on multimodal temporal regularization according to claim 1, characterized in that, The process of performing an exponential calibration operation on the static baseline score based on the rhythm mismatch index to generate a comprehensive subject score, and outputting the final evaluation result based on the comprehensive subject score, includes: The rhythm mismatch index is multiplied by the mismatch penalty sensitivity coefficient pre-configured within the system to obtain the penalty index variable, and the natural exponential function is performed on the penalty index variable to generate the exponential decay multiplier of the decay mapping. The static basic score is multiplied and calibrated with the exponential decay multiplier to generate the comprehensive subject score; Perform a size comparison operation between the comprehensive subject score and the numerical boundaries contained in the subject quality rating interval set to determine the target numerical interval in which the comprehensive subject score falls; Extract the intelligent assessment level label mapped to the target numerical range, encapsulate the comprehensive subject score and the intelligent assessment level label into the final assessment result, and archive and save it.

9. The educational assessment method based on multimodal temporal regularization according to claim 1, characterized in that, After receiving classroom teaching audio and video data, objective test scores, and subjective evaluation texts, it also includes: The system identifies non-standard control characters and semantically meaningless garbled characters within the subjective evaluation text and performs character truncation and deletion operations; it also identifies null values ​​in the objective exam scores and sends a data retransmission request signal to the corresponding input terminal when null values ​​are identified. The objective test scores are converted into a uniform precision floating-point numerical format, and the subjective evaluation text is standardized by character encoding to convert it into a universal multi-octet character set format.

10. An educational assessment system based on multimodal temporal regularization, characterized in that, The system is used to implement the multimodal temporal regularization-based educational assessment method as described in any one of claims 1 to 9, the system comprising: The data access module is used to receive classroom teaching audio and video data, objective test scores, and subjective evaluation texts; The algorithm engine module, communicatively connected to the data access module, internally includes a preprocessing unit, a feature extraction unit, a semantic parsing unit, and a temporal warping unit. The preprocessing unit performs decoding and separation operations on the classroom teaching audio-visual data, splitting it into independent video frame sequence data and audio stream data, and converting the audio stream data into discrete text data fragments. The feature extraction unit performs temporal alignment and feature extraction on the independent video frame sequence data and audio stream data, obtaining the teacher's original stimulus parameters and the student's physiological feature vector. The teacher's original stimulus parameters are input into a pre-trained deep autoencoder to perform dimensionality reduction mapping operations, generating corresponding teacher... The teaching stimulus intensity sequence of the student terminal is used to input the physiological feature vector of the student terminal into a pre-trained feature fusion multilayer perceptron to perform nonlinear spatial mapping operation, thereby generating a corresponding cognitive load micro-motion sequence of the student terminal. The semantic parsing unit is used to perform natural language processing on the text data segment, calculate the semantic complexity index, and generate a dynamic time window coefficient based on the semantic complexity index. The time warping unit is used to perform dynamic time warping calculation on the teaching stimulus intensity sequence and the cognitive load micro-motion sequence based on the dynamic time window coefficient to obtain the global cumulative cost and average phase deviation, and input them into a pre-trained classification evaluation network to perform mapping and normalization processing to generate a rhythm mismatch index. The evaluation output module, which is communicatively connected to the algorithm engine module, is used to input the subjective evaluation text into a configured recurrent neural network to extract text feature sequences and perform polarity inference operations, output sentiment polarity classification probability values, perform weighted calculations by combining the objective test scores with configured weight coefficients to generate a static basic score, and perform exponential calibration operations on the static basic score according to the rhythm mismatch index to generate a comprehensive subject score, and output the final evaluation result based on the comprehensive subject score.

Images