Classroom multi-modal data processing method and system based on deep learning

Abstract

The invention discloses a classroom multi-modal data processing method and system based on deep learning, and aims to solve the composite technical problem of poor classroom multi-modal data fusion analysis effect caused by global asynchronism of a data source, a nonlinear sequential relationship between modals and lack of perception for teaching semantics in the prior art. The method comprises the following steps: acquiring a classroom multi-modal data stream; innovative hierarchical cross-modal time alignment processing is carried out, coarse-grained global alignment is carried out firstly to correct initial time migration between devices, then fine-grained local alignment is carried out, and the core of the processing is that an innovative dynamic time warping algorithm is adopted. Higher alignment weights are given to semantic key segments such as teacher explanation key points and teacher-student interaction, and nonlinear alignment conforming to the teaching rhythm is achieved; and finally, the aligned features are sent to a fusion network to generate a unified fusion feature vector.

Description

Technical Field

[0001] This invention relates to the field of information technology, and in particular to a method and system for processing multimodal classroom data based on deep learning. Background Technology

[0002] Against the backdrop of rapid development in educational informatization, multimodal data processing in the classroom environment has become a crucial supporting area for improving teaching quality and intelligence. This field, by integrating various types of data such as sound, images, and behavior, provides a key foundation for instructional analysis and personalized education. However, its importance is also accompanied by complex technical challenges, urgently requiring innovative methods to overcome existing bottlenecks.

[0003] Currently, although some methods have attempted to comprehensively process classroom data, most suffer from insufficient adaptability, especially when faced with diverse devices and environments, lacking a unified coordination mechanism. These methods often struggle to handle the complex differences between data from different sources, leading to low efficiency in practical applications and failing to meet dynamically changing classroom needs. Focusing on specific challenges, the processing of multimodal data in the classroom environment first faces the problem of alignment in the time dimension. Due to differences in latency when different devices collect data, the timestamps of the data often cannot be directly correlated, making it difficult to maintain consistency when integrating and analyzing multiple types of data. The start time of the data stream is almost impossible to be completely consistent, and clock drift will occur due to long-term operation. If this global time offset is not corrected, all subsequent analyses will be based on an incorrect time baseline from the beginning. Secondly, even if the global offset is resolved, the temporal relationship between modalities is not a simple linear correspondence. For example, a teacher may first give a complete explanation and then write the corresponding keywords on the whiteboard, or the speed of their gestures may not be consistently matched with their speaking speed. This natural, non-linear "speak first, then do" or "speak and do simultaneously" temporal relationship cannot be effectively handled by traditional alignment methods based on fixed time windows, leading to a mismatch between teaching content and behavior at the feature level. Finally, classroom teaching activities have a clear purpose, and not all time segments are equally important. The moments when the teacher emphasizes knowledge points, raises key questions, and provides effective student responses are the core of instructional analysis.

[0004] Existing alignment algorithms, including the classic Dynamic Time Warping (DTW), treat all data points in the time series equally when calculating alignment paths. They fail to understand the illustrative meaning of the data and therefore cannot guarantee the highest accuracy alignment at these semantically critical moments. This indiscriminate alignment strategy means that even the most valuable information fragments may fail to match precisely due to algorithmic limitations, thus restricting the performance ceiling of higher-level intelligent analysis models. These three factors are intertwined, making it difficult for data to achieve effective synergy during the integration process.

[0005] In summary, existing multimodal data processing technologies are insufficient to address the problems at all three levels simultaneously. There is an urgent need for a technical solution that can first establish a global time reference, secondly flexibly adapt to nonlinear temporal relationships, and thirdly perform intelligent alignment guided by pedagogical semantics. Summary of the Invention

[0006] The purpose of this invention is to provide a deep learning-based method and system for processing multimodal classroom data, in order to solve the complex technical problems of low accuracy and poor effect of multimodal data fusion analysis caused by the asynchronous global time due to heterogeneous devices, the nonlinear temporal relationship between teaching behavior and language expression, and the lack of perception of key teaching semantics in the alignment process.

[0007] To achieve the above objectives, this invention provides a deep learning-based method for processing multimodal classroom data, comprising the following steps: First, a classroom multimodal data stream is acquired, which includes at least first modal data and second modal data, both of which contain timestamp information.

[0008] Next, hierarchical cross-modal time alignment processing is performed to generate a time-synchronized multimodal feature set. This core step includes two sub-steps: First, coarse-grained global alignment is performed. This step determines a global time offset by calculating the cross-correlation between the global energy characteristics of the first and second modal data, and uses this offset to perform overall time shift compensation on one of the modal data streams.

[0009] Secondly, based on global alignment, fine-grained local alignment is performed. This step employs an innovative dynamic time warping algorithm to perform non-linear path alignment on the modal data stream after time translation compensation.

[0010] Finally, based on the time-synchronized multimodal feature set, feature fusion is performed, and a unified fusion feature vector with condensed information is generated through a fusion neural network model for subsequent classroom analysis.

[0011] Preferably, the step of performing fine-grained local alignment specifically includes: Semantic information is extracted from the first modal data and the second modal data respectively, and the semantic importance attention weight of each data point at the time step is calculated using a preset cross-modal attention network. Based on the feature sequences of the first modal data and the second modal data, a local cost matrix is ​​constructed between the first modal data and the second modal data, where each element value is the difference between data points at the corresponding time step; The semantic importance weights are applied to the local cost matrix to generate a weighted cost matrix; Based on the weighted cost matrix, a dynamic programming algorithm is used to search for an optimal regularization path, and the time axis of the second modality data is distorted according to the optimal regularization path to match the time axis of the first modality data.

[0012] Preferably, the step of performing fine-grained local alignment specifically includes: Semantic information is extracted in parallel from the first modality data and the second modality data, and a semantic importance attention weight is generated for each data time step using a cross-modal attention network; Based on the feature sequences of the first modality data and the second modality data, a cost matrix is ​​constructed, where the cost value of any point is calculated based on the feature distance at the corresponding time step and the semantic importance attention weight. A dynamic programming algorithm is used to search for an optimal regularized path with the minimum cumulative cost on the cost matrix, and a nonlinear mapping is performed on the time axis of one of the data streams based on the path.

[0013] Accordingly, this invention provides a deep learning-based classroom multimodal data processing system, mainly comprising: The data acquisition module is used to acquire a classroom multimodal data stream that includes at least first modal data and second modal data, both of which contain timestamp information; A time alignment module is used to perform hierarchical cross-modal time alignment processing to generate a time-synchronized multimodal feature set. The time alignment module includes: The global alignment unit is used to determine a global time offset by calculating the cross-correlation between the global energy characteristics of the first modal data and the second modal data, and to perform time shift compensation on one of the modal data streams. The local alignment unit is used to perform nonlinear local time alignment of the first modal data and the second modal data based on the modal data stream after time translation compensation, using a dynamic time warping algorithm. The feature fusion module is used to perform feature fusion based on the time-synchronized multimodal feature set to generate a unified fusion feature vector for classroom analysis.

[0014] The core concept of this invention lies in abandoning the traditional fragmented approach of physical alignment followed by content analysis in data processing, and proposing a semantically guided physical alignment approach. By constructing a hierarchical alignment framework, it achieves coarse-to-fine alignment. Its greatest innovation lies in the design of a DTW algorithm that incorporates semantic importance weighting. This algorithm is the first to introduce cross-modal instructional semantic attention into the cost calculation of DTW, making the temporal alignment process no longer a blind pursuit of distance minimization, but rather a purposeful and focused semantic matching.

[0015] This invention uses a coarse-grained global alignment step and a method of calculating the cross-correlation of energy envelopes to accurately calibrate and compensate for the initial startup delay and clock drift between devices, thus establishing a reliable and unified time reference for all subsequent processing.

[0016] This invention employs the Dynamic Time Warping (DTW) algorithm as its basic framework for fine-grained local alignment. The algorithm's non-linear warping capability allows for flexible stretching or compression of the time axis to match two modal sequences along the optimal path, effectively addressing naturally occurring non-linear temporal relationships such as the difference between the teacher's speaking speed and the speed of their actions.

[0017] This invention proposes a DTW algorithm that incorporates semantic importance weighting. It introduces an attention model to identify key semantic segments in the classroom (such as teacher emphasis and teacher-student interaction), and uses these as weights to adjust the cost matrix of the DTW algorithm. This ensures that the algorithm prioritizes high-precision matching of these semantic key points when searching for the optimal alignment path. This "focused and differentiated" intelligent alignment method ensures that the most valuable teaching information is accurately linked before fusion, greatly improving the quality and fidelity of the fused features.

[0018] In summary, this invention provides an end-to-end, high-precision classroom multimodal data processing solution by combining global calibration, nonlinear matching, and semantically aware intelligent weighting. The fusion features generated by this invention can provide unprecedented high-quality data input for downstream teaching analysis tasks (such as student attention assessment and teaching effectiveness evaluation), thereby significantly improving the accuracy and reliability of these applications. Attached Figure Description

[0019] Figure 1 The overall processing flowchart of the method provided in Embodiment 1 of the present invention.

[0020] Figure 2 : Internal processing flowchart of the hierarchical semantic awareness temporal alignment module of the present invention.

[0021] Figure 3 The overall processing flowchart of the method provided in Embodiment 2 of the present invention.

[0022] Figure 4 The flowchart of the dynamic time warping (SW-DTW) algorithm combining semantic importance weighting of this invention is shown below.

[0023] Figure 5 The system functional modules and their interrelationships provided in this embodiment of the invention are shown in the diagram.

[0024] Figure 6 : Performance comparison of the present invention system and the baseline system on the student distraction detection task: precision-recall curve.

[0025] Figure 7 A visual schematic diagram of a cross-modal dynamic time-warping weighted process according to an embodiment of the present invention. Detailed Implementation

[0026] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this specification, and not all embodiments. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this specification.

[0027] Example 1; This example uses the processing of teacher audio (first modality) and panoramic classroom video (second modality) in the classroom as an example, referring to... Figure 1 , Figure 2 and Figure 5 The complete execution steps of the method of the present invention are described in detail.

[0028] Step S100: Acquire and preprocess the multimodal data stream.

[0029] After system startup, the data acquisition module receives audio and video data in real time from the front-end microphone array and high-definition camera via the network interface. Assume the audio stream is PCM encoded at a 16kHz sampling rate and 16-bit depth, and the video stream is H.264 encoded at 30fps and 1920x1080 resolution. This module parses and decodes the received data packets to obtain the raw audio waveform data and video image frames. Crucially, this module adds a high-precision system timestamp (accurate to microseconds) from a unified source to each decoded data unit (e.g., a 10-millisecond audio frame containing 160 samples and a complete video image frame), forming two parallel, timestamped raw data sequences. It then extracts preliminary features, such as the Mel-frequency cepstral coefficients (MFCC) sequence for the audio and the frame-level feature sequence for the video extracted by a pre-trained ResNet-50 network.

[0030] Step S200: Perform hierarchical semantic-aware temporal alignment. Its internal process is as follows: Figure 3 .

[0031] Before proceeding with global time reference calibration, it is necessary to clarify the technical principles of this invention. In the specific application scenario of classroom teaching, there is a strong semantic and behavioral coupling, i.e., a high correlation, between the macroscopic energy characteristics of the audio stream and the macroscopic motion characteristics of the video stream. This correlation is the technical prerequisite for achieving coarse-grained global alignment in this application. Specifically, research has found that: 1. Regarding the energy characteristics of audio, this application preferably uses Short-Time Energy (STE), which physically represents the loudness of sound. In a classroom environment, it primarily reflects the rhythm of teaching activities. For example, when the teacher raises their voice to emphasize key points, asks questions, or students answer questions collectively, the STE sequence will exhibit significant peaks. However, during calm explanations or when students are studying independently, the STE sequence remains relatively stable.

[0032] 2. Regarding the motion characteristics of the video, this application preferably uses Mean Frame Difference (MFD), which physically represents the intensity of motion in the image. In a classroom environment, it also reflects the rhythm of teaching activities. For example, when the teacher turns to write on the blackboard, uses large gestures, walks down from the podium to interact, or when students stand up or become restless, the MFD sequence will show significant peaks; while when the teacher is standing still lecturing or the students are attentively listening, the MFD sequence remains stable.

[0033] In typical teaching activities, a teacher's vocalizations and body language are often synchronized or closely related. For example, the teaching semantic of "emphasizing key points" is simultaneously manifested through two modalities: "increasing volume" (resulting in a STE peak) and "gestures / writing on the blackboard" (resulting in an MFD peak). Similarly, a lively classroom discussion is accompanied by both noisy voices (high STE) and frequent body movements (high MFD). Therefore, although STE and MFD are two different physical quantities, in the specific spatiotemporal context of classroom teaching, their energy fluctuations form a highly similar macroscopic profile on the timeline.

[0034] Based on the above technical principles, this application can accurately find the time delay point that maximizes the correlation between these two one-dimensional energy feature sequences by calculating the cross-correlation between them. This delay point is the global time offset between the devices.

[0035] Sub-step S210: Global time base calibration (coarse-grained alignment). This step is performed by the global alignment unit ( Figure 5This unit is designed to resolve the fixed time difference caused by asynchronous device startup. Assume that due to network issues, the first data frame of the video stream arrives at the server 1250 milliseconds later than the first data frame of the audio stream. This unit extracts 30 seconds of data from both streams. For the audio, it calculates a short-time energy sequence with a 100-millisecond window; for the video, it calculates the average inter-frame difference (representing motion intensity) sequence within the same window. These two one-dimensional energy sequences reflect the macroscopic rhythm of classroom activities. Then, a cross-correlation algorithm based on Fast Fourier Transform (FFT) is used to efficiently calculate the correlation between these two energy sequences as time delays. The calculation results show that the correlation reaches its peak when the timestamps of the video sequences are shifted forward by 1255 milliseconds. This value of 1255 milliseconds is determined as the global time offset. The system then uniformly subtracts 1255 milliseconds from all timestamps of the video streams to complete global calibration, bringing the two data streams into a macroscopically synchronized state.

[0036] Specifically, the global alignment unit first extracts a fixed duration of data, such as the first 30 seconds, from the beginning of two multimodal data streams (e.g., an audio stream of the first modality and a video stream of the second modality). Then, the unit calculates the global energy feature sequence for each of the two data streams, which reflects the macroscopic rhythm of the classroom activity.

[0037] Specifically, the two preferred methods for calculating energy characteristics are provided with detailed mathematical definitions: 1. Energy characteristics of audio streams: Short-Time Energy (STE) For audio streams, their energy characteristics are preferably obtained by calculating a "short-time energy" sequence. The calculation process is as follows: First, the continuous audio signal x(n) is divided into multiple data frames, where n is the sampling point index. For the m-th frame, its short-time energy E(m) is defined as the sum of the squares of the signal amplitudes at all sampling points within that frame. To reduce spectral leakage, a window function w(k) (e.g., a Hamming window or a Heining window) is typically applied to each frame before calculation. The specific calculation formula is as follows: .in: E(m) is the short-time energy value of the m-th time frame.

[0038] x(·) is the original discrete audio signal.

[0039] L is the length of each frame (for example, for audio with a sampling rate of 16kHz, it can be set to 400 sampling points, or 25 milliseconds).

[0040] R is the frame shift (Hop Length), which is the number of sampling points between the start points of adjacent frames (for example, it can be set to 160 sampling points, or 10 milliseconds, which means there is 60% overlap between frames).

[0041] w(k) is a window function of length L.

[0042] By performing the above calculations on consecutive frames of the audio stream, a one-dimensional short-time energy sequence [E(1), E(2), ..., E(M)] that reflects the fluctuations in sound volume can be obtained.

[0043] 2. Energy characteristics of video streams: Mean Frame Difference (MFD) For video streams, their energy characteristics are preferably obtained by calculating the "average inter-frame difference" sequence. This characteristic can effectively characterize the intensity of motion in the frame. The calculation process is as follows: First, to reduce computational complexity, it is preferable to convert each frame of the video sequence from color to grayscale. For any two consecutive frames f_t and f_{t-1} (where t is the frame index) in the video sequence, calculate the absolute value of the difference between the grayscale values ​​of all corresponding pixels between them, and then calculate the average value of the entire frame. The specific calculation formula is as follows: .in: MFD(t) is the average inter-frame difference between frame t and frame (t-1).

[0044] H and W are the height and width of the video frame, respectively (in pixels).

[0045] I_t(i, j) is the gray value at pixel coordinates (i, j) in the t-th frame.

[0046] |·| represents the absolute value operation.

[0047] By performing the above calculations on consecutive frames of the video stream, a one-dimensional average inter-frame difference sequence [MFD(1), MFD(2), ..., MFD(T)] that reflects changes in the content of the video can be obtained.

[0048] Through the above calculations, the original multimodal data stream is transformed into two one-dimensional energy feature sequences. These two sequences together reflect the macroscopic rhythm of classroom activities. Subsequently, the system employs a cross-correlation algorithm based on Fast Fourier Transform (FFT) to efficiently calculate the change in correlation between these two energy feature sequences over time delay. The time delay corresponding to the peak correlation is determined as the global time offset (e.g., 1255 milliseconds). The system then applies this offset to all timestamps of one of the data streams to complete global calibration, bringing the two data streams into a macroscopic synchronization state, laying a solid foundation for subsequent fine-grained local alignment.

[0049] Sub-step S220: Semantic-aware nonlinear regularization (fine-grained alignment).

[0050] Based on global synchronization, this step is performed by local alignment units ( Figure 5 The implementation aims to solve complex nonlinear temporal relationships in teaching activities. Its core is a dynamic time warping (CSA-DTW) algorithm that combines cross-modal semantic attention.

[0051] S221: Semantic Analysis and Attention Weight Generation: In parallel, the semantic analysis module ( Figure 5 The system processes two raw data streams. It utilizes a set of pre-trained deep learning models to extract high-level semantic information from each modality in parallel. For the audio stream, automatic speech recognition (ASR) and natural language processing (NLP) models identify teaching keywords such as "key points," "difficult points," and "please pay attention," as well as verbal behaviors such as asking questions and summarizing. For the video stream, object detection and behavior recognition models identify key teaching behaviors such as teacher blackboard writing, gestures, and interactions with specific students. Subsequently, a cross-modal attention network receives these semantic labels and, combined with a pre-defined classroom teaching model, generates a "semantic importance" attention weight for each time step of each modality. For example, when a teacher verbally emphasizes a key point and simultaneously writes it on the whiteboard, the attention weight at the corresponding audio and video time point will significantly increase.

[0052] Suppose that at 5 minutes and 10 seconds, the audio stream, analyzed by ASR and NLP models, identifies that the teacher says, "This question is a must-know point for the college entrance examination; everyone must master it." The video stream, analyzed by a behavior recognition model, detects that the teacher simultaneously outlines the question on the whiteboard with red chalk. After receiving these two strongly correlated teaching events, the cross-modal attention network will output a very high semantic importance attention weight (e.g., 5.0) for both the audio stream during the time period corresponding to the phrase "must-know point" and the video stream during the time period corresponding to the teacher's "outlining" action. During non-teaching moments such as when the teacher pauses or drinks water, the weight will be the baseline value of 1.0.

[0053] S222: Constructing a semantically weighted cost matrix: Construct a cost matrix between the first modality feature sequence and the second modality feature sequence. In this matrix, the cost value of any point (i, j) depends not only on the physical distance (e.g., Euclidean distance) between the i-th audio feature and the j-th video feature, but also on the inverse modulation of the semantic importance weights corresponding to these two points. That is, when the semantic importance of both time points is extremely high, the cost of their matching is artificially and significantly reduced.

[0054] Specifically, the local alignment unit receives the audio MFCC feature sequence and the video ResNet feature sequence, as well as the aforementioned attention weight sequence. It constructs a cost matrix D', where the element D'(i, j) is calculated using the following formula: , where ||.||2 represents the Euclidean distance. Since the weight W is 5.0 for both the teacher's explanation of "must-know points" and "drawing frames," the cost of matching feature vectors at these moments will be divided by 25 from the original distance, artificially and drastically reduced.

[0055] S223: Search for the optimal regular path: On this weighted cost matrix, a dynamic programming algorithm is used to find a path with the minimum cumulative cost from the lower left corner to the upper right corner. Since the "cost" of passing through the aforementioned key areas is extremely low, the path will inevitably "bend" to pass through these areas, thus forcing a precise alignment between the teacher's voice saying "must-know points" and his "drawing the frame" action, even if there is a natural delay of a fraction of a second in physical time.

[0056] The final output of this step is the optimal regularized path that defines the best nonlinear mapping relationship between audio and video sequences.

[0057] Step S300: Deep fusion of multimodal features.

[0058] The feature fusion module, based on the optimal path output from the previous step, distorts and resamples the video feature sequence along the time axis to ensure strict alignment with each time point of the audio feature sequence. At each new, synchronized time point, the aligned audio MFCC vector and video ResNet vector are concatenated. This concatenated long feature vector sequence (e.g., length T, dimensions D_audio + D_video) is then input into a fusion network containing multiple gated recurrent units (GRUs) and a final fully connected layer.

[0059] Step S400: Output a unified fused feature vector.

[0060] The fully connected fusion network encodes and extracts the temporal information of the entire sequence, ultimately outputting a unified fusion feature vector sequence with a fixed dimension (e.g., 256 dimensions). This sequence is the final output of this method and can be used for downstream classroom analysis tasks.

[0061] Example 2; This example still uses the processing of classroom audio (first modality) and video (second modality) as an example, referring to... Figure 3 The overall execution flow of the method of the present invention is described in detail.

[0062] Step S201: Acquire multimodal data streams. After system startup, audio and video streams are received in real time from the microphone array and high-definition camera, respectively. For example, the audio stream is PCM data with a sampling rate of 16kHz, and the video stream is H.264 encoded data at 30fps. After decoding each received audio data block and video frame, the system attaches a high-precision system timestamp, forming two parallel, timestamped raw data streams.

[0063] Step S202: Perform hierarchical time alignment. The specific breakdown is as follows: Sub-step S202a: Global alignment.

[0064] This step aims to correct a fixed initial offset between the devices. Assume that, upon detection, the timestamp of the first valid frame of the video stream is 780 milliseconds later than the timestamp of the first valid sample of the audio stream. The system locates the delay point with the highest correlation by calculating the cross-correlation function of the energy envelopes (short-time energy of the audio and the sum of the absolute values ​​of the inter-frame differences of the video) of the two streams within the initial 15 seconds. The value of this delay point (e.g., 785 milliseconds) is determined as the global time offset. The system then uniformly subtracts 785 milliseconds from all timestamps of the video stream to complete the global calibration, bringing the two streams to a state of near synchronization.

[0065] Sub-step S202b: Local alignment.

[0066] Building upon global synchronization, this step addresses non-linear temporal relationships. The system employs a "Semantic Importance Weighted Dynamic Time Warping (SW-DTW)" algorithm for fine-tuning the two streams. The output of this step is an optimal matching path between two feature sequences (e.g., an MFCC sequence of audio and a CNN feature sequence of video), indicating which segment of the audio stream each frame of the video stream should be matched with.

[0067] Specifically, this involves combining semantic-weighted dynamic time warping (SW-DTW), such as... Figure 4 As shown, its technical details are explained in detail.

[0068] Suppose we have two globally aligned feature sequences, an audio feature sequence and a digitally aligned feature sequence. and video feature sequences .

[0069] 1. Feature extraction and semantic weight generation: Feature extraction: It is the audio feature vector (such as MFCC) at the i-th time step, b j It is the video feature vector at the j-th time step (such as features extracted from a CNN model).

[0070] Semantic weight generation: A parallel semantic analysis module analyzes the original audio and video streams and generates a weight sequence synchronized with the feature sequence. and The weight values ​​range from [1, α], where α > 1. For example, when the teacher utters the keyword "key point," the corresponding weight... The value is set to a higher α (e.g., α=5); during smooth transitions, the weight is 1.

[0071] 2. Construction of the weighted cost matrix: The algorithm first constructs an n*m cost matrix D'. Each element D'(i, j) in the matrix represents the "cost" of matching the i-th time point of the audio with the j-th time point of the video. The calculation method of this cost is one of the key aspects of this algorithm: .

[0072] in: It is a distance metric between two feature vectors, usually using Euclidean distance or cosine distance.

[0073] and These are the semantic importance weights corresponding to the two time points.

[0074] ε is a very small positive number (e.g., 1e-9) used to prevent the denominator from being zero.

[0075] The core idea of ​​this formula is: when the semantic importance of both time points (i and j) is high (i.e. and (Since both are very large), the cost D'(i, j) of matching them will be significantly reduced. This makes the algorithm strongly inclined to prioritize matching these moments that both sides consider "important" when searching for the optimal path later.

[0076] 3. Dynamic programming to find the optimal path: Next, dynamic programming is used to compute a cumulative cost matrix γ. Each element γ(i, j) in the matrix represents the minimum cumulative cost of the optimal path from the starting point (1, 1) to point (i, j). The recurrence relation is as follows: .

[0077] Starting with γ(1, 1) = D'(1, 1), fill the entire γ matrix in row or column order. Finally, γ(n, m) is the minimum alignment cost between the two sequences.

[0078] 4. Path backtracking and timeline mapping: Starting from the endpoint (n, m), by backtracking backwards according to the recurrence relation, we can find the optimal regularized path that minimizes the cumulative cost. Each pk is a coordinate pair (i, j). This path P provides a non-linear mapping from the timeline of the audio sequence to the timeline of the video sequence, guiding subsequent feature resampling and fusion.

[0079] Through the above steps, the SW-DTW algorithm achieves intelligent and precise alignment guided by teaching content.

[0080] Step S203: Feature fusion.

[0081] Based on the optimal matching path output by S202b, the system resamples the video feature sequence. For example, if the path indicates that the 100th frame of the video should match the 3000th to 3020th sample points of the audio, the system associates the video feature vector of the 100th frame with the feature vector of this audio segment. In this way, the system generates a new, temporally fully aligned multimodal feature sequence. Then, at each time point, the audio and video feature vectors are concatenated along the feature dimension to form a longer, high-dimensional pre-fusion feature vector.

[0082] Step S204: Generate a unified fusion feature vector.

[0083] The high-dimensional feature sequence generated by S203 is input into a pre-trained fusion network (e.g., a sequence model based on Transformer or LSTM). This network learns and captures intra-modal and inter-modal temporal dependencies, and through dimensionality reduction and information extraction, outputs a fixed-length (e.g., 128-dimensional) unified fusion feature vector for each time point. This vector is a highly condensed representation of the classroom state at that time step and can be directly used for downstream tasks.

[0084] Example 3; This example refers to Figure 5This invention provides a detailed description of the specific implementation methods of each functional module of the system described herein. The system can be deployed on edge computing devices in a classroom or on a cloud server and consists of purely software modules.

[0085] 1. Data Acquisition Module: This module serves as the system's entry point. Internally, it is implemented as a multi-threaded I / O manager. Each thread is responsible for receiving data from one data source. For example, one thread connects to the camera via the RTSP protocol stack, pulls the H.264 stream, and decodes it using the FFmpeg library. Another thread listens for PCM packets sent by the microphone array via a UDP socket. After decoding the data into a usable format, all threads call the operating system kernel-level high-precision clock (such as clock_gettime(CLOCK_MONOTONIC)) to obtain the current timestamp, encapsulate it together with the data block, and place it in a shared, thread-safe data queue for downstream modules to consume.

[0086] 2. Semantic Analysis Module: Contains multiple sub-units that process data in parallel. Audio Semantic Unit: Built-in end-to-end Automatic Speech Recognition (ASR) service based on the Conformer architecture, converting real-time audio streams into text streams. The text stream is then fed into a BERT-based Natural Language Processing (NLP) pipeline trained to perform two tasks: ① classroom-specific keyword recognition (e.g., "key points", "difficult points", "exam", "question"); ② speech behavior classification (e.g., "explain", "question", "evaluate", "instruction").

[0087] Video Semantic Unit: Built-in YOLOv8 for real-time detection of teachers and students in the video. For teachers, the image blocks within their detection boxes are fed into a behavior recognition model based on a SlowFast network, which is trained to recognize specific teaching actions (such as "writing on the blackboard," "pointing at the screen," and "gesturing"). For students, their head posture and global behaviors (such as "raising their hand," "leaning on the table," and "turning around") are recognized.

[0088] Cross-modal attention network: This is a small Transformer encoder model. Its input is a sequence of semantic labels extracted from audio and video semantic units within the same time window. The network is pre-trained to learn the importance of different semantic combinations. For example, it learns that when (audio: "asking a question") and (video: "student raises their hand") occur simultaneously, it is a very important teacher-student interaction segment, and therefore outputs high attention weights for the time points corresponding to these two events.

[0089] The final output of this module is a multi-channel attention weight sequence that is precisely aligned with the timestamps of the original data stream.

[0090] Specifically, the input processing, architecture, and training process of this small Transformer encoder model are as follows: Input encoding and vectorization: To input discrete, symbolic semantic information into a neural network, a series of encoding and vectorization processes are required. This process preferably includes the following steps: 1.1 Generation of semantic label sequences: First, the system processes data streams from different modalities (such as audio and video) in parallel. The audio stream is processed by ASR and NLP units (such as the Conformer+BERT pipeline described in Example 3) to extract semantic tags at each time step t, such as "explaining," "asking a question," "answering," and "silence." The video stream is processed by a visual analysis unit (such as YOLOv8+SlowFast) to extract semantic tags at the same time step t, such as "writing on the blackboard," "gesture pointing," "students raising their hands," and "no activity." This results in two parallel, time-aligned semantic tag sequences: S_audio = [L_a1, L_a2, ...] and S_video = [L_v1, L_v2, ...].

[0091] 1.2 Semantic Dictionary and Embedding Layer: The system pre-defines a global dictionary (Vocabulary) containing all possible audio and video semantic tags. Each unique tag in this dictionary corresponds to a learnable, high-dimensional feature vector, namely a "word embedding".

[0092] Specifically, an embedding layer (nn.Embedding) can be constructed, whose dictionary size is the number of unique labels in the global dictionary, and the embedding dimension (d_embed) can be set to, for example, 64.

[0093] 1.3 Construction of cross-modal input vectors: At each time step t, the feature vectors E_at and E_vt corresponding to the audio label L_at and video label L_vt in the embedding layer are obtained respectively. In order for the network to comprehensively consider the information of the two modalities, the two vectors are concatenated to form a fused input vector X_t with a dimension of 2 * d_embed (i.e., 128 dimensions).

[0094] X_t = Concatenate(E_at, E_vt) Thus, the original two label sequences are transformed into a fused vector sequence X = [X_1, X_2, ...,X_T], where T is the sequence length.

[0095] 1.4 Positional Encoding: Since the Transformer model itself does not have the ability to process sequence order, positional information needs to be added to each element of the input vector sequence X. This invention uses a standard sine / cosine positional encoding function to generate a positional encoding vector P_t with the same dimension as the input vector (128 dimensions) for each position in the sequence, and adds it to the corresponding input vector X_t to obtain the final input Z_t fed into the Transformer encoder.

[0096] Z_t = X_t + P_t Network Architecture: The core of this small Transformer encoder model is a standard Transformer encoder, designed to efficiently capture contextual dependencies in semantic sequences with low computational cost. A preferred, specific architecture parameter is as follows: Model dimension (d_model): 128. This dimension is present throughout all parts of the model and is consistent with the dimension of the input vector concatenated in step 1.3.

[0097] Number of encoder layers (N): 2 layers. It consists of two identical Transformer encoder layers stacked together. This shallow number of layers is sufficient to capture short-term semantic associations common in classroom teaching scenarios, while maintaining the model's lightweight nature.

[0098] Number of multi-head attention heads (h): 4. In each encoder layer, the self-attention module is divided into 4 parallel attention "heads", each with a dimension of d_model / h = 128 / 4 = 32. The multi-head mechanism allows the model to focus on information from different locations and from different representation subspaces.

[0099] Feedforward network dimension (d_ff): 512. In each encoder layer, after the attention module is a feedforward network consisting of two linear layers and a ReLU activation function. The dimension d_ff of its inner layers is typically four times the model dimension, i.e., 4 * 128 = 512.

[0100] Output Layer: The output of the Transformer encoder is still a sequence Y = [Y_1, Y_2, ..., Y_T], where each Y_t is a 128-dimensional vector. To obtain the "semantic importance weight" (a scalar) for each time step, a linear layer is connected to the end of the model, which maps the 128-dimensional feature vector to a scalar value, Linear(in_features=128, out_features=1).

[0101] To ensure that the output weights have a clear physical meaning (e.g., weights greater than or equal to 1, where 1 represents baseline importance), the output of the linear layer is passed through an activation function, such as the Softplus function, and a bias of 1.0 is added: Weight_t = 1.0 + Softplus(Linear(Y_t)).

[0102] The Softplus function is a smooth approximation of ReLU, which guarantees that the output is always positive, avoids the gradient vanishing problem, and makes the final weights smooth and always greater than 1.

[0103] Training Process: Training this attention network is a typical supervised learning task, the goal of which is to enable the network to learn to accurately output importance weights that are consistent with human expert cognition based on the semantic sequence of the input.

[0104] 3.1 Training Dataset: This network does not use a general pre-trained model, but is trained from scratch for classroom teaching scenarios or fine-tuned on a small dataset. The training data comes from a human-annotated classroom multimodal dataset (e.g., Edu-MultiModal-10h in this application).

[0105] Annotation process: Several education experts or experienced teachers were invited to watch the audio and video in the dataset. They were asked to assign an importance score (e.g., an integer from 1 to 5, where 1 represents "unimportant / non-teaching activity" and 5 represents "extremely critical teaching activity") to each teaching activity at each time step (e.g., per second), based on its importance to achieving the teaching objectives. The annotations from multiple experts were then averaged or tested for consistency to form the final "true value" weight sequence W_gt.

[0106] 3.2 Training Objective (Loss Function): The goal of training is to minimize the difference between the weight sequence W_pred predicted by the network and the ground truth weight sequence W_gt labeled by experts. Therefore, this invention adopts the Mean Squared Error (MSE Loss) function commonly used in regression tasks.

[0107] Loss = (1 / T) * Σ (W_pred_t - W_gt_t)² Where T is the sequence length. This loss function penalizes the deviation between the predicted value and the true value, driving the model to learn an accurate mapping relationship from semantics to importance.

[0108] 3.3 Training Hyperparameters: Training is performed using standard deep learning frameworks in the field, such as PyTorch or TensorFlow. Typical hyperparameter settings are as follows: Optimizer: AdamW, learning rate 1e-4.

[0109] Batch Size: 32, meaning that 32 semantic sequence fragments are processed in each iteration.

[0110] Training epochs: Multiple iterations are performed on the training set until the loss of the model on the independent validation set no longer decreases significantly, in order to prevent overfitting.

[0111] Through the detailed input processing, network architecture design, and targeted training process described above, the "cross-modal attention network" of this invention is implemented through a small Transformer encoder, providing reliable and quantified intelligent weights for the subsequent semantically weighted dynamic time warping (SW-DTW) step.

[0112] 3. Timing Alignment Module: This module is the heart of the system and is responsible for performing timing alignment.

[0113] Global Alignment Unit: This unit is relatively simple to implement, mainly relying on signal processing libraries (such as NumPy and SciPy in Python). It obtains initial data from the data queue, calculates the energy envelope, and uses the scipy.signal.correlate function to calculate cross-correlation and delay. Then, it uses the calculated offset as a global variable for use in all subsequent timestamp calibrations.

[0114] Local Alignment Unit: This is the most computationally intensive unit in the system, and its core is an efficient implementation of the CSA-DTW or SW-DTW algorithm. To handle long sequences, this unit typically employs approximation algorithms such as FastDTW to reduce its O(N²) complexity. It obtains preliminary features (such as MFCC and ResNet features) from the data queue and attention weights from the semantic analysis module. Then, it constructs a weighted cost matrix in an optimized C++ or CUDA kernel and performs dynamic programming backtracking to compute the optimal regularization path at extremely high speed.

[0115] 4. Feature Fusion Module: This module is the system's output. Based on the regularized path output by the local alignment unit, it interpolates and resamples the feature sequence of one modality, ensuring that the features of all modalities are aligned at each time point. Then, it concatenates these aligned feature vectors and inputs the result into a pre-trained GRU or Transformer fusion model implemented in TensorFlow or PyTorch for final feature extraction and dimensionality reduction. Finally, it provides the fused feature vectors to external classroom analysis applications via an RPC or RESTful API interface.

[0116] Through the precise coordination of the above modules, this system can automatically and efficiently transform raw, chaotic multimodal data streams into structured, high-quality fusion features containing rich temporal and semantic information.

[0117] Example 4: In a specific application scenario of "detecting student distraction in class", this example verifies the performance advantages of the method and system described in this invention compared with the prior art through a quantitative comparative experiment.

[0118] 1. Verification methods and experimental setup Dataset: The experiment used a self-built classroom behavior analysis dataset called "Edu-MultiModal-10h". This dataset contains a total of 10 hours of multimodal data recorded from real high school classrooms in five different subjects (mathematics, physics, English, history, and chemistry). The data includes: Teacher's explanation audio (16kHz, mono).

[0119] High-definition video (30fps, 1080p) of the teacher's blackboard writing and main activities.

[0120] Wide-angle surveillance video covering the entire class (30fps, 1080p).

[0121] The dataset has been meticulously annotated by three educational psychology experts. The annotations are titled "Student Distraction Events," with the start and end time frames of each distraction event for each student being accurate. Distraction types include "whispering with a deskmate," "looking down at a phone or fidgeting," and "staring out the window or daydreaming for a long time."

[0122] Experimental Groups: Baseline Group: Employs a common "fixed-time-window alignment" method. This method divides the audio and video streams into windows of equal length (e.g., 2 seconds) and directly considers the audio and video features within the same window as aligned.

[0123] The Invention Group adopts the complete "hierarchical semantic-aware temporal alignment" method proposed in this invention, including global alignment and the core "combined semantic importance weighted DTW (CSA-DTW)" local alignment.

[0124] Downstream Task and Model: To ensure fairness in the comparison, both experiments used the exact same downstream distraction detection model. This model is a classifier based on a Long Short-Term Memory (LSTM) network, whose input is a fused sequence of feature vectors, and whose output is each student's listening status at each time step ("focused" or "distracted"). The only variable is the quality of the fused features input to the model, which directly depends on the upstream alignment method.

[0125] Evaluation metrics: The standard evaluation metrics used in classification tasks are precision, recall, and F1 score to measure the model’s performance in detecting “distracting events”.

[0126] 2. Verification methods and results The experiments were conducted on a server equipped with an NVIDIA RTX 4090 GPU, using the PyTorch deep learning framework. The "Edu-MultiModal-10h" dataset was divided into training and test sets in an 8:2 ratio. After preprocessing the data using the baseline system and the system of this invention, the downstream distraction detection models were trained and tested. The performance comparison on the test set is shown in the table below: Table 1: Performance Comparison of Distraction Event Detection ; As shown in Table 1, compared with the baseline system, the precision, recall and F1 score of the downstream distraction detection model were improved by about 19 percentage points after data preprocessing using the method of this invention.

[0127] 3. Visualized Results Display To more intuitively illustrate the performance differences between the two methods at different decision thresholds, we plotted their precision-recall curves on the test set, as shown below. Figure 6 As shown in the diagram. In the PR curve, the closer the curve is to the upper right corner (1,1), the better the model's performance.

[0128] PR curve of the "baseline system" ( Figure 6 The relatively flat (middle dashed line) indicates that while it improves recall, precision drops rapidly. This means that in order to find more distracting events, the system will generate a large number of false positives.

[0129] PR curve of the "invention system" ( Figure 6 The solid line is significantly higher than the baseline system and has a fuller shape, which indicates that the system of the present invention can maintain a much higher precision at any recall level, achieving a good balance between accurate and complete retrieval.

[0130] The area under the curve (AUC-PR) is a metric for measuring the overall performance of the PR curve. Calculations show that the AUC-PR value of the system of this invention reaches 0.93, significantly higher than the baseline system's 0.75, which quantitatively demonstrates the overwhelming advantage of this invention.

[0131] Through the quantitative table comparisons and visual curve analyses described above, it can be concluded that the hierarchical, semantically aware cross-modal time alignment method proposed in this invention can effectively solve the problems of asynchrony, nonlinearity, and semantic mismatch in classroom multimodal data, generating high-quality, high-fidelity fusion features. These high-quality features can provide a solid data foundation for downstream intelligent analysis tasks (such as distraction detection in this example), thereby fundamentally improving their accuracy and reliability, and possessing practical application value.

[0132] Example 5; Figure 7 This is a visual diagram illustrating the cross-modal dynamic time warping weighting process according to an embodiment of the present invention. The diagram aims to more intuitively demonstrate the core mechanism and effect of the dynamic time warping (SW-DTW) algorithm incorporating semantic importance weighting. The diagram includes four sub-diagrams: The top-left subplot is a "Cross-modal Dynamic Temporal Warping Weighted Schematic Diagram," showing the weighted cost matrix D' between the audio feature sequence (horizontal axis) and the visual feature sequence (vertical axis). The color intensity of each point in the matrix represents the magnitude of the alignment cost; the darker the color (e.g., dark brown area), the lower the cost calculated by the weighting algorithm according to this invention. The superimposed red dashed path in the figure is the "optimal warping path P" searched by the dynamic programming algorithm. This path tends to pass through low-cost regions, indicating that the algorithm successfully identified and prioritized the alignment of semantically strongly related key event points.

[0133] The top right subplot is a "cross-modal sequence comparison plot," which schematically shows how a certain feature value of two modalities (such as audio and vision) changes over time before alignment.

[0134] The lower left subplot is a "weight distribution diagram," which uses a bar chart to show the distribution of different semantic importance weights (such as high, medium, and low weights) throughout the entire sequence, intuitively reflecting the sparsity of key events in the teaching process.

[0135] The bottom right sub-figure is the "algorithm flowchart," which briefly summarizes the core computational steps of the SW-DTW algorithm, including inputting sequence data, calculating the feature distance matrix, applying dynamic time warping, calculating the weight distribution, generating the optimal path, and outputting the weighted alignment result.

[0136] Figure 7 The joint demonstration showed how the present invention can intelligently adjust alignment costs by quantifying semantic importance and ultimately find a high-quality alignment path that conforms to the teaching rhythm, thus verifying the effectiveness of the core algorithm of the present invention.

[0137] In addition, in another embodiment, the first modal data can be the audio stream of a teacher's lecture, and the second modal data can be the screen recording video stream of an electronic whiteboard or PPT. The present invention can achieve precise alignment between the teacher's spoken content and the screen operation, that is, the idea of ​​the present invention is also applicable to the alignment of text and audio.

[0138] Those skilled in the art will understand that the core idea of ​​adjusting the cost matrix through semantic importance weights proposed in this invention can also be applied to other variants of the dynamic time warping algorithm (such as FastDTW, SparseDTW, etc.), or other sequence alignment algorithms that rely on cost matrix path search. These variants all fall within the scope of this invention's protection. Based on the embodiments of the present invention described above, and through the above description, those skilled in the art can make various changes and modifications without departing from the technical concept of the present invention. The technical scope of the present invention is not limited to the contents of the specification, but must be determined according to the scope of the claims.

Claims

1. A method for processing multimodal classroom data based on deep learning, characterized in that, Includes the following steps: Acquire a classroom multimodal data stream that includes at least first modal data and second modal data, both of which contain timestamp information; Perform hierarchical cross-modal time alignment processing to generate a time-synchronized multimodal feature set. The hierarchical cross-modal time alignment processing includes: S1: Perform coarse-grained global alignment by calculating the cross-correlation between the global energy characteristics of the first modal data and the second modal data, determining a global time offset, and performing time shift compensation on one of the modal data streams; S2: Perform fine-grained local alignment. Based on the modal data stream after time translation compensation, perform nonlinear local time alignment on the first modal data and the second modal data through a dynamic time warping algorithm. Based on the time-synchronized multimodal feature set, feature fusion is performed to generate a unified fusion feature vector for classroom analysis.

2. The method according to claim 1, characterized in that, The step of performing fine-grained local alignment specifically includes: Semantic information is extracted from the first modal data and the second modal data respectively, and the semantic importance attention weight of each data point at the time step is calculated using a preset cross-modal attention network. Based on the feature sequences of the first modal data and the second modal data, a local cost matrix is ​​constructed between the first modal data and the second modal data, where each element value is the difference between data points at the corresponding time step; The semantic importance weights are applied to the local cost matrix to generate a weighted cost matrix; Based on the weighted cost matrix, a dynamic programming algorithm is used to search for an optimal regularization path, and the time axis of the second modality data is distorted according to the optimal regularization path to match the time axis of the first modality data.

3. The method according to claim 1, characterized in that, The step of performing fine-grained local alignment specifically includes: Semantic information is extracted in parallel from the first modality data and the second modality data, and a semantic importance attention weight is generated for each data time step using a cross-modal attention network; Based on the feature sequences of the first modality data and the second modality data, a cost matrix is ​​constructed, where the cost value of any point is calculated based on the feature distance at the corresponding time step and the semantic importance attention weight. A dynamic programming algorithm is used to search for an optimal regularized path with the minimum cumulative cost on the cost matrix, and a nonlinear mapping is performed on the time axis of one of the data streams based on the path.

4. The method according to claim 2 or 3, characterized in that, The first modal data is classroom audio data, whose semantic information includes teaching keywords or discourse behaviors extracted through speech recognition and natural language processing; the second modal data is classroom video data, whose semantic information includes teacher blackboard writing, gestures or teacher-student interaction events extracted through behavior recognition.

5. The method according to claim 1, wherein The feature fusion process includes: concatenating the time-aligned feature vectors of each modality, then performing dimensionality reduction and deep feature extraction through a fusion neural network model, and outputting the unified fusion feature vector.

6. A classroom multimodal data processing system based on deep learning, characterized in that, include: The data acquisition module is used to acquire a classroom multimodal data stream that includes at least first modal data and second modal data, both of which contain timestamp information; A time alignment module is used to perform hierarchical cross-modal time alignment processing to generate a time-synchronized multimodal feature set. The time alignment module includes: The global alignment unit is used to determine a global time offset by calculating the cross-correlation between the global energy characteristics of the first modal data and the second modal data, and to perform time shift compensation on one of the modal data streams. The local alignment unit is used to perform nonlinear local time alignment of the first modal data and the second modal data based on the modal data stream after time translation compensation, using a dynamic time warping algorithm. The feature fusion module is used to perform feature fusion based on the time-synchronized multimodal feature set to generate a unified fusion feature vector for classroom analysis.

7. The system according to claim 6, characterized in that, The local alignment unit is specifically used for: Semantic information is extracted from the first modal data and the second modal data respectively, and the semantic importance weight of each time step is calculated using a preset attention model; Construct a local cost matrix between the first modal data and the second modal data, where each element value is the difference between data points at the corresponding time step; The semantic importance weights are applied to the local cost matrix to generate a weighted cost matrix; Based on the weighted cost matrix, a dynamic programming algorithm is used to search for an optimal regularization path, and the time axis of the second modality data is distorted according to the optimal regularization path to match the time axis of the first modality data.

8. The system according to claim 6, characterized in that, The local alignment unit is specifically used for: Semantic information is extracted in parallel from the first modality data and the second modality data, and a semantic importance attention weight is generated for each data time step using a cross-modal attention network; Based on the feature sequences of the first modality data and the second modality data, a cost matrix is ​​constructed, where the cost value of any point is calculated based on the feature distance at the corresponding time step and the semantic importance attention weight. A dynamic programming algorithm is used to search for an optimal regularized path with the minimum cumulative cost on the cost matrix, and a nonlinear mapping is performed on the time axis of one of the data streams based on the path.

9. The system according to claim 7 or 8, characterized in that, The system also includes: A semantic analysis module, connected to the local alignment unit, is used to extract semantic information from the data stream and generate semantic importance attention weights for use by the local alignment unit.

10. The system according to claim 6, characterized in that, The feature fusion module is specifically used to: concatenate the time-aligned feature vectors of each modality, then perform dimensionality reduction and deep feature extraction through a fusion neural network model, and output the unified fusion feature vector.

Images