Deep learning-based classroom multi-modal data processing method and system

Abstract

The application discloses a kind of based on deep learning's classroom multi-modal data processing method and system, to solve the compound type technical problem that existing technology is caused by global asynchronous of data source, nonlinear time sequence relationship between modal and lack of perception to teaching semantics, leading to poor classroom multi-modal data fusion analysis effect.The method of the application comprises: obtaining classroom multi-modal data stream;Innovative hierarchical cross-modal time alignment processing is executed, the processing first carries out coarse-grained global alignment to correct the initial time offset between devices, then fine-grained local alignment is executed, and the core is to adopt an innovative dynamic time warping algorithm, to give higher alignment weight to semantic key fragments such as teacher explanation emphasis and teacher-student interaction, to realize nonlinear alignment in line with teaching rhythm;Finally, the features after alignment are sent into fusion network to generate uniform fusion feature vector.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of information technology, and particularly to a classroom multi-modal data processing method and system based on deep learning. BACKGROUND

[0002] Under the background of rapid development of educational informatization, multi-modal data processing in the classroom environment has become an important support field for improving teaching quality and intelligent level. This field integrates sound, image, behavior and other multi-type data to provide key foundation for teaching analysis and personalized education. However, its importance is accompanied by complex technical challenges, and innovative methods are needed to break through existing bottlenecks.

[0003] Currently, although some methods have tried to comprehensively process classroom data, there are still problems of insufficient adaptability, especially when facing diversified devices and environments, and lack of unified coordination mechanism. These methods often have difficulty in dealing with the complex differences of data from different sources, resulting in low efficiency in actual application and difficulty in meeting the dynamic changes of classroom needs. Focusing on specific challenges, the processing of multi-modal data in the classroom environment first faces the problem of alignment in the time dimension. Due to the time delay difference of different devices in collecting data, the timestamps of data often cannot be directly corresponded, which makes it difficult to maintain consistency in the fusion analysis of multi-type data. The starting time of data flow is almost impossible to be completely consistent, and clock drift will occur due to long-term operation. If this global time offset is not corrected, all subsequent analysis will be based on the wrong time baseline from the beginning. Secondly, even if the global offset is solved, the temporal relationship between modalities is not simply linearly corresponding. For example, the teacher may first speak a complete explanation, and then write the corresponding keywords on the whiteboard, or the speed of his gestures does not match the speech speed constantly. This natural, nonlinear "speak first, do later" or "speak while doing" temporal relationship cannot be effectively handled by traditional alignment methods based on fixed time window, which will lead to mismatching of teaching content and behavior in the feature level. Finally, classroom teaching activities have a clear purpose, and not all time segments are equally important. The moments when the teacher emphasizes the knowledge points, raises key questions, and students effectively answer are the core of teaching analysis.

[0004] Existing alignment algorithms, including the classic dynamic time warping (DTW), treat all data points in the time series equally when calculating the alignment path. They cannot understand the teaching meaning of the data, so they cannot guarantee the highest accuracy of alignment at these semantic key moments. This indiscriminate alignment strategy may not accurately match the most valuable information segments due to the limitations of the algorithm, thereby limiting the performance upper limit that the upper intelligent analysis model can achieve. These three factors are intertwined, making it difficult for data to form effective synergy in the integration process.

[0005] In summary, the existing multi-modal data processing technology is difficult to cope with the above three levels of problems at the same time, and there is an urgent need for a technical solution that can first establish a global time reference, secondly can flexibly adapt to nonlinear temporal relationships, and can intelligently align teaching semantics as the guide. SUMMARY

[0006] The purpose of the present application is to provide a deep learning-based classroom multi-modal data processing method and system to solve the problems of global time asynchrony caused by device heterogeneity, nonlinear temporal relationship between teaching behavior and language expression, and lack of perception of key teaching semantics in the alignment process in the prior art, thereby causing the composite technical problem of low multi-modal data fusion analysis precision and poor effect.

[0007] To achieve the above-mentioned purpose, the present application provides a deep learning-based classroom multi-modal data processing method, comprising the following steps:

[0008] First, obtain a classroom multi-modal data stream comprising at least first modal data and second modal data, both of which contain timestamp information.

[0009] Next, perform hierarchical cross-modal time alignment processing to generate a time-synchronized multi-modal feature set. This core step includes two sub-steps:

[0010] First, perform coarse-grained global alignment. This step determines a global time offset by calculating the cross-correlation between the global energy features of the first modal data and the second modal data, and uses the offset to perform overall time translation compensation on one of the modal data streams.

[0011] Second, on the basis of global alignment, perform fine-grained local alignment. This step uses an innovative dynamic time warping algorithm to perform nonlinear path alignment on the modal data stream after time translation compensation.

[0012] Finally, based on the time-synchronized multi-modal feature set, perform feature fusion to generate a unified fusion feature vector for subsequent classroom analysis through a fusion neural network model.

[0013] Preferably, the step of performing fine-grained local alignment specifically includes:

[0014] Extract semantic information from the first modal data and the second modal data respectively, and calculate the semantic importance attention weight of each data time step using a pre-set cross-modal attention network;

[0015] constructing a local cost matrix between the first modality data and the second modality data based on the feature sequences of the first modality data and the second modality data, wherein each element value is a data point difference between corresponding time steps;

[0016] applying the semantic importance weight to the local cost matrix to generate a weighted cost matrix;

[0017] based on the weighted cost matrix, searching an optimal warping path by using a dynamic programming algorithm, and warping a time axis of the second modality data according to the optimal warping path to match a time axis of the first modality data.

[0018] Preferably, the step of performing fine-grained local alignment specifically comprises:

[0019] extracting semantic information from the first modality data and the second modality data in parallel, and generating a semantic importance attention weight for each time step of the data by using a cross-modality attention network;

[0020] constructing a cost matrix based on the feature sequences of the first modality data and the second modality data, wherein a cost value of any point is calculated according to a feature distance of corresponding time steps and the semantic importance attention weight;

[0021] searching an optimal warping path with minimum cumulative cost on the cost matrix by using a dynamic programming algorithm, and performing nonlinear mapping on a time axis of one of the data streams according to the path.

[0022] Correspondingly, the present application provides a deep learning-based classroom multi-modal data processing system, mainly comprising:

[0023] a data acquisition module for acquiring classroom multi-modal data streams including at least first modality data and second modality data, wherein the first modality data and the second modality data both contain timestamp information;

[0024] a time alignment module for performing hierarchical cross-modality time alignment processing to generate a time-synchronized multi-modal feature set, wherein the time alignment module comprises:

[0025] a global alignment unit for determining a global time offset by calculating cross-correlation between global energy features of the first modality data and the second modality data, and performing time translation compensation on one of the modality data streams;

[0026] a local alignment unit for performing nonlinear local time alignment on the first modality data and the second modality data by using a dynamic time warping algorithm based on the modality data stream after the time translation compensation;

[0027] A feature fusion module is configured to perform feature fusion based on the time-synchronized multi-modal feature set to generate a unified fusion feature vector for classroom analysis.

[0028] The core idea of the present application is to abandon the traditional data processing mode of physical alignment first and content analysis later, and to propose a semantic-guided physical alignment idea. By constructing a hierarchical alignment framework, coarse-to-fine calibration is achieved. The greatest creativity lies in the design of the DTW algorithm combined with semantic importance weighting. It first introduces cross-modal teaching semantic attention into the cost calculation of DTW, so that the time alignment process is no longer blind distance minimization, but purposeful and focused semantic matching.

[0029] The present application accurately marks and compensates for the initial start-up delay and clock drift between devices through the coarse-grained global alignment step, using the method of calculating the energy envelope cross-correlation, and establishes a reliable and unified time reference for all subsequent processing.

[0030] The present application adopts the dynamic time warping (DTW) algorithm as the basic framework in the fine-grained local alignment. The nonlinear warping capability of this algorithm can flexibly stretch or compress the time axis to match two modal sequences with the optimal path, thereby effectively solving the natural nonlinear timing relationship between the teacher's speech speed and the speed of actions.

[0031] The present application proposes a DTW algorithm combined with semantic importance weighting. By introducing an attention model to identify semantic key segments in the classroom (such as teacher emphasis and teacher-student interaction), the weight is used to adjust the cost matrix of the DTW algorithm. This makes the algorithm prioritize the high-precision matching of these semantic key points when searching for the optimal alignment path. This intelligent alignment method with focus and distinction ensures that the most valuable teaching information can be accurately associated before fusion, greatly improving the quality and fidelity of the fusion features.

[0032] In summary, the present application combines global calibration, nonlinear matching, and semantic-aware intelligent weighting to provide an end-to-end, high-precision classroom multi-modal data processing solution. The fusion features generated by the present application can provide unprecedented high-quality data input for downstream teaching analysis tasks (such as student concentration evaluation and teaching effectiveness evaluation), thereby significantly improving the accuracy and reliability of these applications. BRIEF DESCRIPTION OF DRAWINGS

[0033] Figure 1 The overall processing flowchart of the method provided by the embodiment one of the present application.

[0034] Figure 2 The internal processing flowchart of the hierarchical semantic-aware time alignment module of the present application.

[0035] Figure 3 The method provided by the embodiment two of the present application has the overall process flow diagram.

[0036] Figure 4 The present application combines the semantic importance weighted dynamic time warping (SW-DTW) algorithm flow diagram.

[0037] Figure 5 The system function modules provided by the embodiment of the present application and their interrelationship diagram.

[0038] Figure 6 The present application system and baseline system performance comparison precision-recall (P-R) curve diagram on the student distraction detection task.

[0039] Figure 7 The visualization schematic diagram of the cross-modal dynamic time warping weighting process of one embodiment of the present application. DETAILED DESCRIPTION

[0040] In order to make the person skilled in the art better understand the technical solutions in the specification, the technical solutions in the specification will be described clearly and completely in the specification, and obviously, the described embodiments are only part of the embodiments of the specification, not all the embodiments. Based on the embodiments in the specification, all other embodiments obtained by the person skilled in the art without creative labor should belong to the scope of protection of the specification.

[0041] Embodiment one; the present embodiment takes processing teacher audio (first modality) and classroom panoramic video (second modality) in the classroom as an example, and refers to Figure 1 , Figure 2 and Figure 5 , the complete execution steps of the method of the present application are described in detail.

[0042] Step S100: obtaining and preprocessing multi-modal data stream.

[0043] After the system is started, the data acquisition module receives audio and video data from the front-end microphone array and high-definition camera in real time through the network interface. Assuming that the audio stream is PCM encoded with a sampling rate of 16 kHz and a depth of 16 bits, and the video stream is H.264 encoded with a resolution of 30 fps and 1920x1080. This module parses and decodes the received data packets to obtain the original audio waveform data and video image frames. It is crucial that this module stamps each decoded data unit (for example, a 10-millisecond audio frame containing 160 sampling points, and a complete video image frame) with a high-precision system timestamp (accurate to microseconds) of uniform origin, forming two parallel, timestamped original data sequences, and extracting preliminary features such as the Mel Frequency Cepstral Coefficient (MFCC) sequence of the audio and the frame-level feature sequence extracted by the pre-trained ResNet-50 network of the video.

[0044] Step S200: Perform hierarchical semantic-aware timing alignment. Its internal process is referred to Figure 3 .

[0045] Before the present application enters global time reference calibration, it is necessary to clarify the technical principle of the present application. In the specific application scenario of classroom teaching, there is a strong semantic and behavioral coupling between the macro energy features of the audio stream and the macro motion features of the video stream, i.e. a high degree of correlation. This correlation is the technical premise for the coarse-grained global alignment implemented by the present application. Specifically, it has been found through research that:

[0046] 1. The energy feature of the audio, which is preferably the Short-Time Energy (STE) in the present application, has the physical meaning of the loudness of sound. In a classroom environment, it mainly reflects the rhythm of teaching activities, such as moments when the teacher raises the volume to emphasize key points, initiates questions, and students collectively answer questions, etc., at which time the STE sequence will present significant peaks; while in a smooth explanation or student self-study, the STE sequence is in a relatively stable state.

[0047] 2. The motion feature of the video, which is preferably the Mean Frame Difference (MFD) in the present application, has the physical meaning of the motion intensity of the picture. In a classroom environment, it also reflects the rhythm of teaching activities, such as moments when the teacher turns around to write on the board, uses large gestures, walks down the podium to interact, students stand up or stir, etc., at which time the MFD sequence will also present significant peaks; while in a teacher standing to lecture or students concentrating to listen, the MFD sequence is in a stable state.

[0048] In typical teaching activities, a teacher's vocalizations and body language are often synchronized or closely related. For example, the teaching semantic of "emphasizing key points" is simultaneously manifested through two modalities: "increasing volume" (resulting in a STE peak) and "gestures / writing on the blackboard" (resulting in an MFD peak). Similarly, a lively classroom discussion is accompanied by both noisy voices (high STE) and frequent body movements (high MFD). Therefore, although STE and MFD are two different physical quantities, in the specific spatiotemporal context of classroom teaching, their energy fluctuations form a highly similar macroscopic profile on the timeline.

[0049] Based on the above technical principles, this application can accurately find the time delay point that maximizes the correlation between these two one-dimensional energy feature sequences by calculating the cross-correlation between them. This delay point is the global time offset between the devices.

[0050] Sub-step S210: Global time base calibration (coarse-grained alignment). This step is performed by the global alignment unit ( Figure 5 This unit is designed to resolve the fixed time difference caused by asynchronous device startup. Assume that due to network issues, the first data frame of the video stream arrives at the server 1250 milliseconds later than the first data frame of the audio stream. This unit extracts 30 seconds of data from both streams. For the audio, it calculates a short-time energy sequence with a 100-millisecond window; for the video, it calculates the average inter-frame difference (representing motion intensity) sequence within the same window. These two one-dimensional energy sequences reflect the macroscopic rhythm of classroom activities. Then, a cross-correlation algorithm based on Fast Fourier Transform (FFT) is used to efficiently calculate the correlation between these two energy sequences as time delays. The calculation results show that the correlation reaches its peak when the timestamps of the video sequences are shifted forward by 1255 milliseconds. This value of 1255 milliseconds is determined as the global time offset. The system then uniformly subtracts 1255 milliseconds from all timestamps of the video streams to complete global calibration, bringing the two data streams into a macroscopically synchronized state.

[0051] Specifically, the global alignment unit first extracts a fixed duration of data, such as the first 30 seconds, from the beginning of two multimodal data streams (e.g., an audio stream of the first modality and a video stream of the second modality). Then, the unit calculates the global energy feature sequence for each of the two data streams, which reflects the macroscopic rhythm of the classroom activity.

[0052] Specifically, the two preferred methods for calculating energy characteristics are provided with detailed mathematical definitions:

[0053] 1. Energy characteristics of audio streams: Short-Time Energy (STE)

[0054] For audio stream, its energy feature is preferably obtained by computing a "short-time energy" sequence. The computation process is as follows:

[0055] First, the continuous audio signal x(n) is segmented into multiple data frames, where n is the sample point index. For the m-th frame, its short-time energy E(m) is defined as the sum of the square of the signal amplitude of all sample points within the frame. To reduce spectral leakage, a window function w(k) (such as Hamming window or Hanning window) is usually applied to each frame signal before computation. The specific computation formula is as follows:

[0056] wherein:

[0057] E(m) is the short-time energy value of the m-th time frame.

[0058] x(·) is the original discrete audio signal.

[0059] L is the length of each frame (for example, for audio with a sampling rate of 16 kHz, it can be set to 400 sample points, i.e. 25 milliseconds).

[0060] R is the frame shift (Hop Length), i.e. the number of sample points between the starting points of adjacent frames (for example, it can be set to 160 sample points, i.e. 10 milliseconds, which means there is a 60% overlap between frames).

[0061] w(k) is a window function with length L.

[0062] By performing the above calculation on consecutive frames of the audio stream, a one-dimensional short-time energy sequence [E(1), E(2),..., E(M)] that can reflect the fluctuations in sound volume can be obtained.

[0063] 2. Energy feature of video stream: Mean Frame Difference (MFD)

[0064] For video stream, its energy feature is preferably obtained by computing a "Mean Frame Difference" sequence. This feature can effectively represent the degree of motion of the picture. The computation process is as follows:

[0065] First, to reduce the computational complexity, each frame of color image of the video sequence is preferably converted into a grayscale image. For any two consecutive frames f_t and f_{t-1} in the video sequence (where t is the frame index), the absolute value of the difference between the grayscale values of all corresponding pixel points is calculated, and then the average value of the entire frame is calculated. The specific computation formula is as follows:

[0066] wherein:

[0067] MFD(t) is the average inter-frame difference value between the t-th frame and the (t-1)-th frame.

[0068] H and W are the height and width of the video frame in pixels, respectively.

[0069] I_t(i, j) is the gray value of the t-th frame at pixel coordinate (i, j).

[0070] |·| denotes the absolute value operation.

[0071] By performing the above calculation on the consecutive frames of the video stream, a one-dimensional average inter-frame difference sequence [MFD(1), MFD(2),..., MFD(T)] that can reflect the changes in the video content is obtained.

[0072] Through the above calculation, the original multi-modal data stream is converted into two one-dimensional energy feature sequences. These two sequences together reflect the macro-rhythm of the classroom activity. Subsequently, the system uses a cross-correlation algorithm based on fast Fourier transform (FFT) to efficiently calculate the change of correlation between the two energy feature sequences with time delay. When the correlation reaches the peak value, the time delay corresponding to the peak value is determined as the global time offset (e.g. 1255 milliseconds). The system then applies this offset to all the time stamps of one of the data streams, completes the global calibration, and makes the two data streams enter the macro-synchronous state, laying a solid foundation for subsequent fine-grained local alignment.

[0073] Sub-step S220: semantic-aware nonlinear warping (fine-grained alignment).

[0074] On the basis of global synchronization, this step is performed by the local alignment unit ( Figure 5 ), which aims to solve the complex nonlinear time sequence relationship in teaching activities. The core of this step is a dynamic time warping CSA-DTW algorithm combined with cross-modal semantic attention.

[0075] S221: semantic analysis and attention weight generation: in parallel, the semantic analysis module ( Figure 5The two original data streams are processed. The system utilizes a set of pre-trained deep learning models to extract high-level semantic information from each modality data in parallel. For the audio stream, through automatic speech recognition (ASR) and natural language processing (NLP) models, teaching keywords such as "key points", "difficult points", "please pay attention" and speech behaviors such as questions and summaries are identified. For the video stream, through object detection and behavior recognition models, key teaching behaviors such as teacher's board writing, gesture pointing, and interaction with specific students are identified. Subsequently, a cross-modal attention network receives these semantic labels and combines a preset classroom teaching model to generate a "semantic importance" attention weight for each time step of each modality. For example, when the teacher emphasizes the key points orally and writes the key points on the whiteboard at the same time, the attention weight of the corresponding audio and video time points will be significantly increased.

[0076] Suppose at 5 minutes and 10 seconds, the audio stream is analyzed by the ASR and NLP models to identify that the teacher says "This question is a must-try point for the college entrance examination, everyone must master it"; the video stream is analyzed by the behavior recognition model to detect that the teacher simultaneously frames the question on the whiteboard with a red chalk. After receiving these two strongly correlated teaching events, the cross-modal attention network outputs a very high semantic importance attention weight, for example, 5.0, for the audio stream in the time period corresponding to the "must-try point" sentence and for the video stream in the time period corresponding to the teacher's "framing" action. During the teacher's pause, drinking water, and other non-teaching moments, the weight is the baseline value 1.0.

[0077] S222: Construct a semantic-weighted cost matrix: construct a cost matrix between the first modality feature sequence and the second modality feature sequence. The cost value of any point (i, j) in this matrix depends not only on the physical distance (such as Euclidean distance) between the i-th audio feature and the j-th video feature, but also on the inverse modulation of the semantic importance weight corresponding to these two points. That is, when the semantic importance of the two time points is very high, the cost of matching each other is artificially significantly reduced.

[0078] Specifically, the local alignment unit receives the audio MFCC feature sequence and the video ResNet feature sequence, as well as the above attention weight sequence. It constructs a cost matrix D', where the calculation formula of the element D'(i, j) is: where ||.||2 represents the Euclidean distance. Since the weights W of the teacher's "must-try point" and "framing" moments are both 5.0, the cost of matching the feature vectors at these moments will be artificially greatly reduced by dividing the original distance by 25.

[0079] S223: Search the optimal warping path: on this weighted cost matrix, a path with the minimum accumulated cost is found from the bottom-left corner to the top-right corner by dynamic programming algorithm. Since the "cost" of passing through the key region is extremely low, this path will inevitably "bend" to pass through the region, thus enforcing the precise alignment of the teacher's voice and his "frame-drawing" action, even if there is a natural delay of zero point several seconds in physical time.

[0080] The final output of this step is the optimal warping path that defines the best non-linear mapping relationship between the audio and video sequences.

[0081] Step S300: Multimodal feature deep fusion.

[0082] The feature fusion module warps and resamples the video feature sequence on the time axis according to the optimal path output in the previous step, so that it is strictly aligned with each time point of the audio feature sequence. At each new, synchronized time point, the aligned audio MFCC vector and video ResNet vector are spliced. Then, this spliced long feature vector sequence (e.g., length T, dimension D_audio + D_video) is input into a fusion network containing multiple layers of gated recurrent units (GRU) and a final fully connected layer.

[0083] Step S400: Output unified fusion feature vector.

[0084] The fully connected layer fusion network encodes and refines the time series information of the entire sequence, and finally outputs a unified fusion feature vector sequence with a fixed dimension (e.g., 256 dimensions). This sequence is the final output of the method and can be used for downstream classroom analysis tasks.

[0085] Example two; this example still takes the processing of classroom audio (first modality) and video (second modality) as an example, and refers to Figure 3 , which describes the overall execution process of the method in detail.

[0086] Step S201: Obtain multimodal data stream. After the system starts, audio stream and video stream are accessed in real time from the microphone array and high-definition camera respectively. For example, the audio stream is 16kHz sampling rate PCM data, and the video stream is 30fps H.264 encoded data. The system decodes each audio data block and video frame received and attaches a high-precision system timestamp to form two parallel, timestamped original data streams.

[0087] Step S202: Perform hierarchical time alignment. The specific decomposition is as follows:

[0088] Sub-step S202a: Global alignment.

[0089] This step aims to correct a fixed initial offset between the devices. Assume that, upon detection, the timestamp of the first valid frame of the video stream is 780 milliseconds later than the timestamp of the first valid sample of the audio stream. The system locates the delay point with the highest correlation by calculating the cross-correlation function of the energy envelopes (short-time energy of the audio and the sum of the absolute values ​​of the inter-frame differences of the video) of the two streams within the initial 15 seconds. The value of this delay point (e.g., 785 milliseconds) is determined as the global time offset. The system then uniformly subtracts 785 milliseconds from all timestamps of the video stream to complete the global calibration, bringing the two streams to a state of near synchronization.

[0090] Sub-step S202b: Local alignment.

[0091] Building upon global synchronization, this step addresses non-linear temporal relationships. The system employs a "Semantic Importance Weighted Dynamic Time Warping (SW-DTW)" algorithm for fine-tuning the two streams. The output of this step is an optimal matching path between two feature sequences (e.g., an MFCC sequence of audio and a CNN feature sequence of video), indicating which segment of the audio stream each frame of the video stream should be matched with.

[0092] Specifically, this involves combining semantic-weighted dynamic time warping (SW-DTW), such as... Figure 4 As shown, its technical details are explained in detail.

[0093] Suppose we have two globally aligned feature sequences, an audio feature sequence and a digital feature sequence. and video feature sequences .

[0094] 1. Feature extraction and semantic weight generation:

[0095] Feature extraction: It is the audio feature vector (such as MFCC) at the i-th time step, b j It is the video feature vector at the j-th time step (such as features extracted from a CNN model).

[0096] Semantic weight generation: A parallel semantic analysis module analyzes the original audio and video streams and generates a weight sequence synchronized with the feature sequence. and The weight values ​​range from [1, α], where α > 1. For example, when the teacher utters the keyword "key point," the corresponding weight... The value is set to a higher α (e.g., α=5); during smooth transitions, the weight is 1.

[0097] 2. Weighted cost matrix construction:

[0098] The algorithm first constructs an n*m cost matrix D'. Each element D'(i, j) in the matrix represents the "cost" of matching the i-th time point of the audio sequence with the j-th time point of the video sequence. The calculation of this cost is one of the keys of the algorithm: .

[0099] where: is the distance measure between two feature vectors, usually using Euclidean distance or cosine distance.

[0100] and are the semantic importance weights of the two time points, respectively.

[0101] ε is a very small positive number (such as 1e-9) to prevent the denominator from being zero.

[0102] The core idea of this formula is: when the semantic importance of two time points (i and j) is very high (i.e. and are both large), the matching cost D'(i, j) will be significantly reduced. This makes the algorithm strongly prefer to match these "important" moments when finding the optimal path in the subsequent.

[0103] 3. Dynamic programming to solve the optimal path:

[0104] Next, dynamic programming is used to calculate a cumulative cost matrix γ. Each element γ(i, j) in the matrix represents the minimum cumulative cost of the optimal path from the starting point (1, 1) to the point (i, j). Its recursive relationship is as follows: .

[0105] Starting from γ(1, 1) = D'(1, 1), fill the entire γ matrix in row or column order. Finally, γ(n, m) is the minimum alignment cost between the two sequences.

[0106] 4. Path backtracking and time axis mapping:

[0107] Starting from the end point (n, m), according to the recursive relationship, we can find the optimal alignment path that minimizes the cumulative cost , where each pk is a coordinate pair (i, j). This path P provides a non-linear mapping relationship from the time axis of the audio sequence to the time axis of the video sequence, guiding the subsequent feature resampling and fusion.

[0108] Through the above steps, the SW-DTW algorithm realizes intelligent and accurate alignment guided by teaching content.

[0109] Step S203: Feature fusion.

[0110] According to the optimal matching path output by S202b, the system resamples the video feature sequence. For example, if the path shows that the 100th frame of the video should match the 3000th to 3020th sample points of the audio, the system associates the video feature vector of the 100th frame with the feature vectors of this segment of audio. In this way, the system generates a new, completely time-aligned multi-modal feature sequence. Then, at each time point, the audio feature vector and the video feature vector are concatenated along the feature dimension to form a longer high-dimensional pre-fusion feature vector.

[0111] Step S204: Generating unified fusion feature vector.

[0112] The high-dimensional feature sequence generated by S203 is input into a pre-trained fusion network (e.g., a sequence model based on Transformer or LSTM). The network can learn and capture intra-modal and inter-modal temporal dependencies, and finally output a fixed-length (e.g., 128-dimensional) unified fusion feature vector for each time point through dimension reduction and information extraction. This vector is a highly condensed representation of the state of the classroom at that moment, which can be directly used for downstream tasks.

[0113] Embodiment three; this embodiment refers to Figure 5 , describes the specific implementation of each functional module of the system described in the present application. The system can be deployed on the edge computing device or cloud server of the classroom and is composed of pure software modules.

[0114] 1. Data acquisition module: This module is the entrance of the system. It is implemented as a multi-threaded I / O manager internally. Each thread is responsible for receiving a data source. For example, a thread connects to the camera through the RTSP protocol stack, pulls the H.264 code stream, and decodes it using the FFmpeg library. Another thread listens to the PCM data packets sent by the microphone array through the UDP socket. After all threads decode the data into a usable format, they all call the high-precision clock (such as clock_gettime(CLOCK_MONOTONIC)) in the operating system kernel to get the current timestamp, and package it together with the data block, and put it into a shared, thread-safe data queue for downstream modules to consume.

[0115] 2. Semantic analysis module: contains multiple parallel processing subunits:

[0116] Audio Semantic Unit: Built-in end-to-end Automatic Speech Recognition (ASR) service based on the Conformer architecture, converting real-time audio streams into text streams. The text stream is then fed into a BERT-based Natural Language Processing (NLP) pipeline trained to perform two tasks: ① classroom-specific keyword recognition (e.g., "key points", "difficult points", "exam", "question"); ② speech behavior classification (e.g., "explain", "question", "evaluate", "instruction").

[0117] Video Semantic Unit: Built-in YOLOv8 for real-time detection of teachers and students in the video. For teachers, the image blocks within their detection boxes are fed into a behavior recognition model based on a SlowFast network, which is trained to recognize specific teaching actions (such as "writing on the blackboard," "pointing at the screen," and "gesturing"). For students, their head posture and global behaviors (such as "raising their hand," "leaning on the table," and "turning around") are recognized.

[0118] Cross-modal attention network: This is a small Transformer encoder model. Its input is a sequence of semantic labels extracted from audio and video semantic units within the same time window. The network is pre-trained to learn the importance of different semantic combinations. For example, it learns that when (audio: "asking a question") and (video: "student raises their hand") occur simultaneously, it is a very important teacher-student interaction segment, and therefore outputs high attention weights for the time points corresponding to these two events.

[0119] The final output of this module is a multi-channel attention weight sequence that is precisely aligned with the timestamps of the original data stream.

[0120] Specifically, the input processing, architecture, and training process of this small Transformer encoder model are as follows:

[0121] Input encoding and vectorization:

[0122] To input discrete, symbolic semantic information into a neural network, a series of encoding and vectorization processes are required. This process preferably includes the following steps:

[0123] 1.1 Generation of semantic label sequences:

[0124] First, the system processes data streams from different modalities (e.g., audio and video) in parallel. The audio stream is processed by the ASR and NLP units (e.g., the Conformer+BERT pipeline as described in Embodiment Three) to extract semantic labels at each time step t, such as "lecture," "question," "answer," "silence," and so on. The video stream is processed by the visual analysis unit (e.g., YOLOv8+SlowFast) to extract semantic labels at the same time step t, such as "board writing," "hand pointing," "student raising hand," "no activity," and so on. As a result, two parallel, time-aligned semantic label sequences are formed: S_audio = [L_a1, L_a2,...] and S_video = [L_v1, L_v2,...].

[0125] 1.2 Vocabulary and Embedding Layer:

[0126] The system is pre-configured with a global vocabulary that contains all possible audio and video semantic labels. Each unique label in the vocabulary corresponds to a learnable, high-dimensional feature vector, known as an "embedding."

[0127] Specifically, an embedding layer (nn.Embedding) can be constructed, with its vocabulary size equal to the number of unique labels in the global vocabulary, and the embedding dimension (d_embed) can be set to, for example, 64.

[0128] 1.3 Construction of Cross-Modality Input Vector:

[0129] At each time step t, the audio label L_at and the video label L_vt are obtained, and their corresponding feature vectors E_at and E_vt in the embedding layer are obtained. To allow the network to consider information from both modalities, the two vectors are concatenated to form a fusion input vector X_t with a dimension of 2 * d_embed (i.e., 128 dimensions).

[0130] X_t = Concatenate(E_at, E_vt)

[0131] Thus, the original two label sequences are converted into a fusion vector sequence X = [X_1, X_2,...,X_T], where T is the sequence length.

[0132] 1.4 Positional Encoding:

[0133] Since the Transformer model itself does not have the ability to handle sequence order, position information needs to be added for each element of the input vector sequence X. The invention uses a standard sine / cosine position encoding function to generate a position encoding vector P_t of the same dimension (128 dimensions) as the input vector for each position in the sequence, and adds it to the corresponding input vector X_t to obtain the final input Z_t sent to the Transformer encoder.

[0134] Z_t = X_t + P_t

[0135] Network Architecture: The core of this small Transformer encoder model is a standard Transformer encoder (Encoder), which is designed to efficiently capture the contextual dependencies in semantic sequences with lower computational cost. A preferred, specific architecture parameter is as follows:

[0136] Model dimension (d_model): 128. This dimension runs through all parts of the model and is consistent with the dimension of the input vector after splicing in step 1.3.

[0137] Number of encoder layers (N): 2 layers. Two identical Transformer encoder layers are stacked to form a 2-layer encoder. This shallow layer number is sufficient to capture the short-term semantic associations commonly seen in classroom teaching scenarios, while maintaining the lightweight nature of the model.

[0138] Number of multi-head attention heads (h): 4. In each encoder layer, the self-attention module (Self-Attention) is divided into 4 parallel attention "heads", each with a dimension of d_model / h = 128 / 4 = 32. The multi-head mechanism allows the model to jointly focus on information from different positions and different representation subspaces.

[0139] Feed-forward network dimension (d_ff): 512. In each encoder layer, after the attention module, there is a feed-forward network (Feed-Forward Network) composed of two linear layers and a ReLU activation function. The dimension d_ff of the internal layer is usually 4 times the model dimension, i.e. 4 * 128 = 512.

[0140] Output Layer:

[0141] The output of the Transformer encoder is still a sequence Y = [Y_1, Y_2,..., Y_T], where each Y_t is a 128-dimensional vector. To get the “semantic importance weight” (a scalar) for each time step, a linear layer is attached at the end of the model to map the 128-dimensional feature vector to a scalar value, Linear(in_features=128, out_features=1).

[0142] To ensure that the output weight value has a clear physical meaning (for example, the weight value is greater than or equal to 1, where 1 represents the baseline importance), the output of the linear layer will pass through an activation function, such as the Softplus function, and add a bias of 1.0, Weight_t = 1.0 + Softplus(Linear(Y_t)).

[0143] The Softplus function is a smooth approximation of ReLU that ensures the output is always positive, avoiding the problem of gradient vanishing, making the final weight smooth and always greater than 1.

[0144] Training Process:

[0145] The training of this attention network is a typical supervised learning task, which aims to enable the network to accurately output importance weights consistent with human expert cognition based on the input semantic sequence.

[0146] 3.1 Training Dataset:

[0147] Instead of using a general pre-trained model, this network is trained from scratch or fine-tuned on a small-scale dataset for the classroom teaching scenario. The training data comes from a fine-labeled classroom multi-modal dataset (such as Edu-MultiModal-10h in this application).

[0148] Labeling process: Invite several education experts or experienced teachers to watch the audio and video in the dataset. They are required to give an importance score (for example, an integer from 1 to 5, 1 representing “not important / non-teaching activity”, 5 representing “extremely critical teaching session”) for each time step (for example, every second) of teaching activity according to its importance to the teaching goal. Average or consistency check the labels of multiple experts to form the final “ground truth” weight sequence W_gt.

[0149] 3.2 Training Objective (Loss Function):

[0150] The goal of training is to minimize the difference between the predicted weight sequence W_pred and the ground truth weight sequence W_gt annotated by experts. Therefore, the invention adopts the Mean Squared Error (MSE Loss) commonly used in regression tasks.

[0151] Loss = (1 / T) * Σ (W_pred_t - W_gt_t)²

[0152] Where T is the sequence length. This loss function will penalize the deviation between the predicted value and the true value, driving the model to learn the accurate mapping relationship from semantics to importance.

[0153] 3.3 Training hyperparameters:

[0154] Training is done using standard deep learning frameworks in the field (such as PyTorch or TensorFlow). Typical hyperparameter settings are as follows:

[0155] Optimizer: AdamW, learning rate 1e-4.

[0156] Batch size: 32, i.e. 32 semantic sequence segments are processed each iteration.

[0157] Epochs: Multiple iterations are performed on the training set until the model's loss on an independent validation set no longer significantly decreases to prevent overfitting.

[0158] Through the above detailed input processing, network architecture design and target-oriented training process, the "cross-modal attention network" described in the invention is implemented through a small Transformer encoder, providing reliable and quantitative intelligent weights for the subsequent semantic weighted dynamic time warping (SW-DTW) step.

[0159] 3. Time alignment module: This module is the heart of the system, responsible for performing time alignment.

[0160] Global alignment unit: The implementation of this unit is relatively simple, mainly relying on signal processing libraries (such as NumPy and SciPy in Python). It obtains the initial data from the data queue, calculates the energy envelope, and uses the scipy.signal.correlate function to calculate the cross-correlation and delay. The calculated offset is then used as a global variable for all subsequent timestamp calibration.

[0161] Local alignment unit: This is the most computationally intensive unit in the system, whose core is an efficient implementation of the CSA-DTW algorithm or the SW-DTW algorithm. To handle long sequences, this unit usually employs an approximate algorithm such as FastDTW to reduce the O(N²) complexity. It takes the preliminary features (such as MFCC and ResNet features) from the data queue and the attention weights from the semantic analysis module. Then, it constructs a weighted cost matrix in an optimized C++ or CUDA kernel and performs dynamic programming backtracking to calculate the optimal warping path at extremely high speed.

[0162] 4. Feature fusion module: This module is the outlet of the system. It interpolates and resamples the feature sequence of one modality according to the warping path output by the local alignment unit, ensuring that at each time point, the features of all modalities are aligned. Then, it concatenates these aligned feature vectors and inputs the result into a pre-trained GRU or Transformer fusion model implemented with TensorFlow or PyTorch for the final feature extraction and dimensionality reduction, and provides the final fusion feature vector to external classroom analysis applications through an RPC or RESTful API interface.

[0163] Through the precise coordination of the above modules, the system can automatically and efficiently convert raw, chaotic multi-modal data streams into structured, high-quality fusion features containing rich temporal and semantic information.

[0164] Example Four: In a specific "student classroom distraction detection" application scenario, this example verifies the performance advantages of the method and system described in the invention compared to existing technologies through quantitative comparative experiments.

[0165] 1. Verification method and experimental setup

[0166] Dataset: The experiment uses a self-built classroom behavior analysis dataset named "Edu-MultiModal-10h". This dataset contains a total of 10 hours of multi-modal data recorded from real high school classrooms of 5 different subjects (mathematics, physics, English, history, and chemistry). The data includes:

[0167] Teacher's explanation audio (16kHz, mono).

[0168] Teacher's board writing and high-definition video of main activities (30fps, 1080p).

[0169] Wide-angle monitoring video covering all students (30fps, 1080p).

[0170] The dataset has been finely annotated by three educational psychology experts with the content of "student distraction events" to the starting and ending time frames of each student's each distraction event, and the distraction types include "whispering with the same table", "lowering the head to play mobile phones or make small actions", "looking out the window or staring for a long time", etc.

[0171] Experimental grouping:

[0172] Baseline system group (Baseline Group): A common "alignment based on fixed time window" method is adopted. This method divides the audio stream and the video stream into windows of equal length (for example, 2 seconds), and directly regards the audio and video features in the same window as aligned.

[0173] Invention system group (Invention Group): The complete "hierarchical semantic-aware temporal alignment" method proposed in the present application is adopted, including global alignment and core "DTW combined with semantic importance weighting (CSA-DTW)" local alignment.

[0174] Downstream task and model: In order to ensure the fairness of comparison, the same downstream distraction detection model is adopted in both groups of experiments. This model is a classifier based on long short-term memory network (LSTM), whose input is the fused feature vector sequence, and whose output is the listening state of each student at each time point ("concentrated" or "distracted"). The only variable is the data quality of the fused features input to the model, which directly depends on the upstream alignment method.

[0175] Evaluation index: The standard evaluation index in the classification task: precision (Precision), recall (Recall) and F1 score (F1-Score) are used to measure the detection performance of the model on "distraction events".

[0176] 2. Verification means and results

[0177] The experiment was carried out on a server equipped with NVIDIA RTX 4090 GPU, using the PyTorch deep learning framework. The "Edu-MultiModal-10h" dataset was divided into training set and test set according to the ratio of 8:2. After preprocessing the data using the baseline system and the system of the present application, the downstream distraction detection model was trained and tested. The performance comparison on the test set is shown in the following table:

[0178] Table 1: Comparison of distraction event detection performance

[0179] ;

[0180] As can be seen from Table 1, compared with the baseline system, the precision, recall and F1 score of the downstream distraction detection model are all improved by about 19 percentage points after data preprocessing by the method of the application.

[0181] 3. Effect visual display

[0182] In order to more intuitively show the performance difference of the two methods under different judgment thresholds, we draw their Precision-Recall Curve (P-R Curve) on the test set, as shown in Figure 6 In the P-R curve, the closer the curve is to the upper right corner (1, 1) point, the better the performance of the model.

[0183] The P-R curve of the "baseline system" (dotted line in Figure 6 ) is relatively flat, indicating that while improving the recall rate, the precision rate will decrease rapidly, which means that in order to find more distraction events, the system will produce a large number of false positives.

[0184] The P-R curve of the "application system" (solid line in Figure 6 ) is significantly higher than the baseline system, and the shape is more full, which means that at any recall rate level, the application system can maintain much higher precision, achieving a good balance between accuracy and completeness.

[0185] The Area Under the Curve (AUC-PR) is an indicator to measure the overall performance of the P-R curve. After calculation, the AUC-PR value of the application system reaches 0.93, which is much higher than the 0.75 of the baseline system, which quantitatively proves the overwhelming advantage of the application.

[0186] Through the above quantitative table comparison and visual curve analysis, it can be concluded that the hierarchical and semantic-aware cross-modal time alignment method proposed in the application can effectively solve the asynchronous, nonlinear and semantic mismatch problems in classroom multi-modal data, and generate high-quality and high-fidelity fusion features. Such high-quality features can provide a solid data foundation for downstream intelligent analysis tasks (such as distraction detection in this example), thereby fundamentally improving their accuracy and reliability, and have practical application value.

[0187] Example five; Figure 7 is a visual diagram of the cross-modal dynamic time warping weighting process according to an embodiment of the application. The diagram is intended to more intuitively show the core mechanism and effect of the dynamic time warping algorithm combined with semantic importance weighting (SW-DTW). The diagram includes four sub-diagrams:

[0188] The upper left subgraph is a "cross-modal dynamic time warping weighting diagram", which shows the weighted cost matrix D' between the audio feature sequence (horizontal axis) and the visual feature sequence (vertical axis). The color depth of each point in the matrix represents the size of the alignment cost. The darker the color (such as the dark brown area), the lower the cost value calculated by the weighting algorithm according to the application. The red dashed path superimposed in the figure is the "optimal warping path P" searched by the dynamic programming algorithm, which tends to pass through the low-cost area, indicating that the algorithm successfully identifies and preferentially aligns those key event points with strong semantic correlation.

[0189] The upper right subgraph is a "cross-modal sequence comparison diagram", which schematically shows the change of a certain feature value over time before alignment of two modalities (such as audio and vision).

[0190] The lower left subgraph is a "weight distribution diagram", which shows the distribution of different semantic importance weights (such as high, medium and low weights) in the entire sequence in the form of a bar chart, directly reflecting the sparsity of key events in the teaching process.

[0191] The lower right subgraph is an "algorithm flowchart", which briefly summarizes the core calculation steps of the SW-DTW algorithm, including input sequence data, calculate feature distance matrix, apply dynamic time warping, calculate weight distribution, generate optimal path, and output weighted alignment results, etc.

[0192] Figure 7 Jointly shows how the application intelligently adjusts the alignment cost by quantifying semantic importance and ultimately finds a high-quality alignment path that meets the teaching rhythm, verifying the effectiveness of the core algorithm of the application.

[0193] In addition, in another embodiment, the first modality data can be an audio stream of a teacher's lecture, and the second modality data can be a screen recording video stream of an electronic whiteboard or PPT. The application can realize accurate alignment of the teacher's spoken content and screen operation, that is, the idea of the application is also applicable to the alignment of text and audio.

[0194] Those skilled in the art can understand that the core idea of the application of adjusting the cost matrix by semantic importance weight can also be applied to other variants of the dynamic time warping algorithm (such as FastDTW, SparseDTW, etc.), or other sequence alignment algorithms that rely on path search of the cost matrix, and these variants fall within the protection concept of the application.

[0195] With the above examples according to the present application as the inspiration, through the above description, relevant staff can make various changes and modifications without deviating from the scope of the technical idea of the present application. The technical scope of the present application is not limited to the content of the specification, and must be determined according to the scope of the claims.

Claims

1. A method for processing multimodal classroom data based on deep learning, characterized in that, Includes the following steps: Acquire a classroom multimodal data stream that includes at least first modal data and second modal data, both of which contain timestamp information; Perform hierarchical cross-modal time alignment processing to generate a time-synchronized multimodal feature set. The hierarchical cross-modal time alignment processing includes: S1: Perform coarse-grained global alignment by calculating the cross-correlation between the global energy characteristics of the first modal data and the second modal data, determining a global time offset, and performing time shift compensation on one of the modal data streams; S2: Perform fine-grained local alignment. Based on the modal data stream after time translation compensation, perform nonlinear local time alignment on the first modal data and the second modal data through a dynamic time warping algorithm. Based on the time-synchronized multimodal feature set, feature fusion is performed to generate a unified fusion feature vector for classroom analysis; The step of performing fine-grained local alignment specifically includes: Semantic information is extracted in parallel from the first modality data and the second modality data, and a semantic importance attention weight is generated for each data time step using a cross-modal attention network; Based on the feature sequences of the first modality data and the second modality data, a cost matrix is ​​constructed, where the cost value of any point is calculated based on the feature distance at the corresponding time step and the semantic importance attention weight. A dynamic programming algorithm is used to search for an optimal regularized path with the minimum cumulative cost on the cost matrix, and a nonlinear mapping is performed on the time axis of one of the data streams based on the path.

2. The method according to claim 1, characterized in that, The step of performing fine-grained local alignment specifically includes: Semantic information is extracted from the first modal data and the second modal data respectively, and the semantic importance attention weight of each data point at the time step is calculated using a preset cross-modal attention network. Based on the feature sequences of the first modal data and the second modal data, a local cost matrix is ​​constructed between the first modal data and the second modal data, where each element value is the difference between data points at the corresponding time step; The semantic importance attention weights are applied to the local cost matrix to generate a weighted cost matrix; Based on the weighted cost matrix, a dynamic programming algorithm is used to search for an optimal regularization path, and the time axis of the second modality data is distorted according to the optimal regularization path to match the time axis of the first modality data.

3. The method according to claim 1 or 2, characterized in that, The first modal data is classroom audio data, whose semantic information includes teaching keywords or discourse behaviors extracted through speech recognition and natural language processing; the second modal data is classroom video data, whose semantic information includes teacher blackboard writing, gestures or teacher-student interaction events extracted through behavior recognition.

4. The method according to claim 1, characterized in that, The feature fusion steps include: concatenating the time-aligned feature vectors of each modality, then performing dimensionality reduction and deep feature extraction through a fusion neural network model, and outputting the unified fusion feature vector.

5. A classroom multimodal data processing system based on deep learning, characterized in that, include: The data acquisition module is used to acquire a classroom multimodal data stream that includes at least first modal data and second modal data, both of which contain timestamp information; A time alignment module is used to perform hierarchical cross-modal time alignment processing to generate a time-synchronized multimodal feature set. The time alignment module includes: The global alignment unit is used to determine a global time offset by calculating the cross-correlation between the global energy characteristics of the first modal data and the second modal data, and to perform time shift compensation on one of the modal data streams. The local alignment unit is used to perform nonlinear local time alignment of the first modal data and the second modal data based on the modal data stream after time translation compensation, using a dynamic time warping algorithm. The feature fusion module is used to perform feature fusion based on the time-synchronized multimodal feature set to generate a unified fusion feature vector for classroom analysis. Specifically, the local alignment unit is used for: Semantic information is extracted in parallel from the first modality data and the second modality data, and a semantic importance attention weight is generated for each data time step using a cross-modal attention network; Based on the feature sequences of the first modality data and the second modality data, a cost matrix is ​​constructed, where the cost value of any point is calculated based on the feature distance at the corresponding time step and the semantic importance attention weight. A dynamic programming algorithm is used to search for an optimal regularized path with the minimum cumulative cost on the cost matrix, and a nonlinear mapping is performed on the time axis of one of the data streams based on the path.

6. The system according to claim 5, characterized in that, The local alignment unit is specifically used for: Semantic information is extracted from the first modal data and the second modal data respectively, and the semantic importance attention weights for each time step are calculated using a preset attention model. Construct a local cost matrix between the first modal data and the second modal data, where each element value is the difference between data points at the corresponding time step; The semantic importance attention weights are applied to the local cost matrix to generate a weighted cost matrix; Based on the weighted cost matrix, a dynamic programming algorithm is used to search for an optimal regularization path, and the time axis of the second modality data is distorted according to the optimal regularization path to match the time axis of the first modality data.

7. The system according to claim 5 or 6, characterized in that, The system also includes: A semantic analysis module, connected to the local alignment unit, is used to extract semantic information from the data stream and generate semantic importance attention weights for use by the local alignment unit.

8. The system according to claim 5, characterized in that, The feature fusion module is specifically used to: concatenate the time-aligned feature vectors of each modality, then perform dimensionality reduction and deep feature extraction through a fusion neural network model, and output the unified fusion feature vector.

Images