An online multi-terminal interconnection production and teaching integration precise management system based on big data analysis

Abstract

This invention relates to an online, multi-terminal interconnected, industry-education integration precision management system based on big data analytics. Specifically, it relates to the field of big data analytics, utilizing dynamic interference compensation technology to effectively overcome issues such as lighting fluctuations, noise interference, and network jitter in complex teaching environments, ensuring high-fidelity collection of classroom teaching behavior data. Through a spatiotemporal fusion mechanism, it precisely aligns the correlation between teachers' body language and verbal emotions, deeply exploring implicit teaching patterns in teacher-student interactions. Based on behavioral feature correlation maps, it quantitatively analyzes the causal impact of teaching strategies, providing teachers with personalized improvement directions. Finally, it combines reinforcement learning to optimize multimodal decision weights, generating forward-looking teaching trend predictions and structured assessment reports, achieving a closed-loop empowerment from data collection to professional development, comprehensively promoting the upgrading of teaching quality and the high-quality development of teacher capabilities in industry-education integration scenarios.

Description

Technical Field

[0001] This invention relates to the field of big data analytics, and more specifically, to a precision management system for online multi-terminal interconnection of industry and education based on big data analytics. Background Technology

[0002] In the practical scenario of teaching ability competitions in vocational colleges, the multimodal data collection and analysis of classroom teaching processes has become a key link in improving teaching quality and assessment level. The system usually uses customized audio and video terminals deployed in classrooms, such as high-definition cameras and multi-channel microphone arrays, to capture multidimensional information about teachers' teaching behavior, students' interactive responses, and the classroom environment in real time. Internal factors include the dynamism of teaching content, the diversity of teacher-student interaction modes, and the nonlinear changes in individual behavior; external factors involve the influence of complex physical environments, such as fluctuations in light intensity, background noise interference, network transmission stability, and the heterogeneity of multi-terminal access. This data is aggregated and preliminarily processed through an online multi-terminal interconnection platform to form a multi-source heterogeneous data stream, including voice waveforms, video frame sequences, and text interaction logs, thereby providing a data foundation for subsequent teaching behavior analysis and digital profile construction.

[0003] However, existing technologies have significant shortcomings in multimodal data fusion processing, making it difficult to guarantee the accuracy and robustness of classroom teaching assessments. Current systems generally adopt shallow fusion strategies based on weighted averaging or feature splicing, failing to fully consider the spatiotemporal correlation and semantic consistency between multi-source data. For example, the teacher's gestures in the video stream and the emotional features of the voice in the audio stream should have a strong temporal correlation, but existing fusion models lack dynamic context awareness mechanisms and cannot effectively align and integrate this information. Especially in low-light or high-noise environments, the quality of single-modal data drops sharply, and the system struggles to recover complete semantics through complementary information, resulting in biases in the generated teaching digital profiles. In addition, existing AI analysis engines mostly rely on static pre-trained models and lack the ability to adapt to real-time environmental changes, further exacerbating the problem of multimodal data silos. This technological limitation directly affects the credibility of teaching assessment indicators such as grade assessments and data evidence, causing biases in teacher professional development guidance and ultimately restricting the overall effectiveness improvement of the industry-education integration management system. Summary of the Invention

[0004] This invention addresses the technical problems existing in the prior art by providing an online multi-terminal interconnected precision management system for industry-education integration based on big data analysis. Through modules XX and XX, it solves the problems mentioned in the background.

[0005] The technical solution of this invention to solve the above-mentioned technical problems is as follows: specifically including: a data preprocessing module, a multimodal fusion module, a correlation graph construction module, and a weight optimization module, wherein;

[0006] Data preprocessing module: When collecting multimodal raw data streams of classroom teaching through customized audio and video terminals, the Long Short-Term Memory Network prediction model trained based on historical environmental data performs dynamic interference compensation on video frame sequences, audio waveforms and text logs, and outputs calibrated feature vectors;

[0007] Multimodal fusion module: After receiving the calibrated feature vector, it adopts a Transformer architecture with multi-head self-attention mechanism, and aligns the spatiotemporal dependency between video action and speech emotion through dynamic time offset weights to generate a fused feature tensor;

[0008] Association Graph Construction Module: Maps the fused feature tensor to graph structure data, dynamically generates an adjacency matrix based on cosine similarity, and extracts the dependency weights between nodes through a graph convolutional network to construct a behavioral feature association graph;

[0009] Weight optimization module: Constructs a state space for reinforcement learning based on the behavioral feature association graph, dynamically adjusts the multimodal fusion weights through a reward feedback mechanism, and outputs a teaching behavior trend prediction and evaluation report;

[0010] In a preferred embodiment, the dynamic interference compensation in the data preprocessing module includes performing multimodal feature extraction and spatiotemporal alignment operations, specifically:

[0011] First, the original data stream of classroom teaching is captured in real time through a customized audio and video terminal. This data stream contains three types of heterogeneous data: video frame sequences, audio waveforms, and text logs.

[0012] Next, for the video frame sequence, a pre-trained 3D convolutional neural network is used to extract spatial features from the video frame sequence, generating a spatiotemporal feature vector containing the teacher's body movements and blackboard writing content.

[0013] Meanwhile, for audio waveforms, the Mel frequency cepstral coefficient conversion method is applied to convert the audio waveform into time-frequency spectral features, and the time derivative features are superimposed to generate audio feature vectors;

[0014] For text log data, a bidirectional encoder representation model is used for text embedding processing, outputting a fixed-dimensional text feature vector;

[0015] Finally, all feature vectors are aligned to the same time step in the time dimension to form a spatiotemporally aligned feature group.

[0016] In a preferred embodiment, the specific operation of training the long short-term memory network prediction model based on historical environmental data is as follows:

[0017] The intensity of light fluctuations, background noise in decibels, and network jitter frequency in historical datasets were used as training samples.

[0018] A dynamic prediction model is constructed through a gated cyclic unit network. This model receives real-time collected environmental parameters as input and outputs a three-dimensional prediction vector containing light interference intensity, noise interference intensity, and network interference intensity.

[0019] Meanwhile, a nonlinear mapping function is designed to convert the predicted interference intensity into a feature confidence weight vector. Each element in the weight vector corresponds to a reliability evaluation value of a specific feature dimension, with a value range of 0 to 1. A value close to 1 indicates high feature confidence, while a value close to 0 indicates that a compensation mechanism needs to be activated.

[0020] In a preferred embodiment, the process of generating the output calibrated feature vector is as follows:

[0021] Based on the three-dimensional prediction vector output by the dynamic prediction model, a compensation matrix corresponding to each feature dimension is generated.

[0022] A confidence-weighted fusion mechanism is adopted: the spatiotemporal aligned feature group and the compensation matrix are linearly combined according to the confidence weight. Specifically, the original data is retained for feature dimensions with high confidence, and the compensation is replaced for feature dimensions with low confidence.

[0023] The text feature vectors are directly concatenated to the final output, resulting in a calibrated feature vector with uniform dimensions and corresponding time steps.

[0024] In a preferred embodiment, the multimodal fusion module aligns the spatiotemporal dependency between video actions and speech emotions using dynamic time offset weights. This includes performing feature dimension decomposition, modal recalibration, and dynamic time offset weight generation. Specifically, when a calibrated feature vector is received from the data preprocessing module, the feature vector is first decomposed into video components, audio components, and text components according to modal type. The video component represents the extracted video spatial features, the audio component represents the converted temporal-spectral features, and the text component represents the encoded text embedding features. Then, each modal component is mapped to a unified embedding space using a trainable projection matrix. The video component is processed using a video projection matrix, the audio component is processed using an audio projection matrix, and the text component is processed using a text projection matrix. The output is a unified video embedding feature, an audio embedding feature, and a text embedding feature.

[0025] Subsequently, based on video and audio embedding features, a gated recurrent unit model is used to learn the spatiotemporal offset pattern. This gated recurrent unit model iteratively processes the concatenated vector of video and audio features at time steps, outputs the hidden state vector, and generates dynamic time offset weights through a fully connected layer and a sigmoid activation function. The weights take values ​​between zero and one to quantify the spatiotemporal dependence strength between video actions and speech emotions. A weight close to one indicates a strong dependence, and a weight close to zero indicates a weak dependence.

[0026] In a preferred embodiment, the Transformer architecture employing a multi-head self-attention mechanism to generate fused feature tensors includes performing a multi-head attention mechanism and spatiotemporal fusion operation. Specifically, it involves taking video embedding features, audio embedding features, and text embedding features with unified output dimensions, along with dynamic time offset weights, as inputs. First, the video embedding features, audio embedding features, and text embedding features are concatenated along the feature dimensions to form fused embedding features. Then, query vectors, key vectors, and value vectors are generated using query weight matrices, key weight matrices, and value weight matrices, respectively. Next, for each attention head, an attention score matrix is ​​calculated. This matrix fuses semantic similarity and spatiotemporal relevance. Semantic similarity is obtained by dividing the dot product of the query vector and the key vector by a scaling factor. Spatiotemporal relevance is obtained by multiplying the product of the dynamic time offset weights and the time distance function by the offset enhancement coefficient. The time distance function uses an exponential decay form to calculate the weights of neighboring time steps.

[0027] Subsequently, the attention score matrix is ​​normalized using the Softmax function, and the output of a single attention head is obtained by adding a weighted vector. The outputs of all attention heads are concatenated along the feature dimension, and a fused feature tensor is generated by the output projection matrix. The dimension of this fused feature tensor matches the time step.

[0028] In a preferred embodiment, the association graph construction module dynamically generates an adjacency matrix based on cosine similarity, which includes performing node vector generation and graph structure initialization operations as well as dynamic adjacency matrix generation operations. Specifically, when the fusion feature tensor output from the multimodal fusion module is received, the fusion feature tensor is first decomposed into multiple node vectors according to the time step dimension.

[0029] Next, an all-zero matrix is ​​initialized as the initial state of the adjacency matrix. The dimension of this matrix is ​​determined by the number of nodes and is a square matrix. Then, the cosine similarity values ​​between all node vectors are calculated to form the original similarity matrix. Each element in this matrix represents the original association strength between the features at the corresponding time steps.

[0030] Based on the numerical distribution of the original similarity matrix, a Gaussian mixture model is used for fitting analysis, and a dynamic threshold parameter is automatically calculated. This dynamic threshold is determined by the mean component and standard deviation component of the Gaussian mixture model combined with an adjustment factor.

[0031] The original similarity matrix is ​​binarized based on a dynamic threshold to generate a binarized adjacency matrix. A value of 1 indicates a significant association between nodes, while a value of 0 indicates no significant association. The final output is a binarized adjacency matrix and a set of node vectors.

[0032] In a preferred embodiment, the step of extracting the dependency weights between nodes through a graph convolutional network includes performing a spatiotemporal graph convolutional network operation to extract dependency weights and a behavioral feature association graph construction operation. Specifically, when a binary adjacency matrix and a set of node vectors are received, the node vector set is first linearly transformed using a trainable weight matrix to generate projected node features.

[0033] Next, a multi-head graph attention mechanism is used to calculate the attention coefficients between nodes. This mechanism concatenates the projected feature vectors of each pair of nodes and processes them with a trainable parameter vector and a non-linear activation function to obtain an unnormalized attention score. The attention score is then standardized to transform it into a probability distribution and multiplied with the corresponding elements in the binary adjacency matrix to generate a weighted adjacency matrix.

[0034] Subsequently, a time decay mechanism is introduced, which calculates the decay factor based on the difference between the time steps of the nodes and multiplies the factor by the corresponding element in the weighted adjacency matrix.

[0035] Finally, by integrating the node vector set, the weighted adjacency matrix, and the non-zero edge set, a weighted directed behavioral feature association graph is constructed.

[0036] In a preferred embodiment, the weight optimization module constructs the state space of reinforcement learning based on the behavioral feature association graph, which includes performing state space construction operations and action space design operations. Specifically, when a behavioral feature association graph is received from the association graph construction module, the topological structure information of the graph is first encoded using a multi-head graph attention network based on the node feature vectors and adjacency matrix of the graph. The network generates state representation vectors of key nodes by calculating normalized attention weights between nodes and aggregating the features of neighboring nodes. The state representation vectors of all key nodes together constitute the state space of reinforcement learning.

[0037] Next, the action space is designed as the adjustment amount of the multimodal fusion weights, which corresponds to the three dimensions of video modality weights, audio modality weights, and text modality weights. Based on the state representation vector, action vectors are generated through a policy network. This policy network includes a state encoding layer and a policy output layer. The state encoding layer uses the hyperbolic tangent activation function to process the input state, and the policy output layer uses the normalized exponential function to constrain the output to a probability distribution between zero and one, thereby obtaining the action vector representing the adjustment amount of each modality weight.

[0038] In a preferred embodiment, the step of dynamically adjusting the multimodal fusion weights through a reward feedback mechanism and outputting a teaching behavior trend prediction and evaluation report includes performing reward calculation and course learning operations and prediction and report generation operations. Specifically, when the state representation vector and action vector are received, a reward function is first designed. This reward function contains three components: an accuracy reward component is calculated based on the similarity between the behavior prediction sequence generated by the policy network and the real behavior sequence derived from the labeled data; a sparsity penalty component is calculated based on the first norm of the action vector; and a course reward component is calculated based on the training progress indication function. The reward value is the weighted sum of the three components.

[0039] Based on the state sequence composed of state representation vectors arranged in chronological order, a temporal convolutional network model is used to predict teaching behavior trends. This temporal convolutional network processes the state sequence and outputs the behavior probability distribution for future time steps. Finally, based on the final fusion weight vector obtained after multiple rounds of iterative optimization of the fusion weight adjustment, it is combined with the behavior trend prediction results and the behavior feature correlation graph to generate an assessment report through a structured report template. The report includes three parts: level assessment, data evidence, and conclusion recommendations. The data evidence comes from the dependency weights in the behavior feature correlation graph, and the conclusion recommendations combine the trend prediction results to provide directions for teaching improvement.

[0040] The beneficial effects of this invention are as follows: Through multimodal data collaborative processing and adaptive optimization mechanisms, the accuracy and guiding value of vocational school teaching ability assessment are significantly improved. The system effectively overcomes problems such as light fluctuations, noise interference, and network jitter in complex teaching environments by utilizing dynamic interference compensation technology, ensuring high-fidelity collection of classroom teaching behavior data. Through spatiotemporal fusion mechanisms, the correlation between teachers' body movements and verbal emotions is accurately aligned, deeply exploring the implicit teaching patterns in teacher-student interactions. Based on behavioral feature correlation maps, the causal impact of teaching strategies is quantitatively analyzed, providing teachers with personalized improvement directions. Finally, by combining reinforcement learning to optimize multimodal decision weights, forward-looking teaching trend predictions and structured assessment reports are generated, realizing a closed-loop empowerment from data collection to professional development, and comprehensively promoting the upgrading of teaching quality and high-quality development of teacher capabilities in the context of industry-education integration. Attached Figure Description

[0041] Figure 1 This is a flowchart of the method of the present invention;

[0042] Figure 2 This is a block diagram of the system structure of the present invention. Detailed Implementation

[0043] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0044] In the description of this application, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the stated features. In the description of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0045] In the description of this application, the term "for example" is used to mean "used as an example, illustration, or description." Any embodiment described as "for example" in this application is not necessarily to be construed as being more preferred or advantageous than other embodiments. The following description is provided to enable any person skilled in the art to make and use the invention. Details are set forth in the following description for purposes of explanation. It should be understood that those skilled in the art will recognize that the invention can be made without using these specific details. In other instances, well-known structures and processes will not be described in detail to avoid obscuring the description of the invention with unnecessary detail. Therefore, the invention is not intended to be limited to the embodiments shown, but is consistent with the broadest scope of the principles and features disclosed in this application.

[0046] Example 1

[0047] This embodiment provides, for example Figure 1-2 The above describes an online, multi-terminal interconnected, industry-education integration precision management system based on big data analytics, specifically including: a data preprocessing module, a multimodal fusion module, a correlation graph construction module, and a weight optimization module, wherein;

[0048] Data preprocessing module: When collecting multimodal raw data streams of classroom teaching through customized audio and video terminals, a long short-term memory network prediction model trained based on historical environmental data performs dynamic interference compensation on video frame sequences, audio waveforms, and text logs, and outputs calibrated feature vectors. In vocational school teaching and competition scenarios, customized audio and video terminals (such as multi-channel microphone arrays and wide dynamic range cameras) collect raw multimodal data streams (video frame sequences, audio waveforms, and text logs) in real time. Due to environmental interference (light fluctuations, noise, network jitter) causing data distortion, this module generates calibrated feature vectors through dynamic interference compensation technology, providing high-quality input for subsequent multimodal fusion.

[0049] Multimodal fusion module: After receiving the calibrated feature vectors, it adopts a Transformer architecture with a multi-head self-attention mechanism to align the spatiotemporal dependencies between video actions and speech emotions through dynamic time offset weights, and generates a fusion feature tensor. This module can solve the spatiotemporal mismatch problem between video actions and speech emotions, and replace the traditional static weighting or simple splicing method.

[0050] The association graph construction module maps the fused feature tensor to graph structure data, dynamically generates an adjacency matrix based on cosine similarity, and extracts the dependency weights between nodes through a graph convolutional network to construct a behavioral feature association graph. Compared with the static graph construction or simple correlation coefficient method that traditional methods rely on, this module achieves adaptive extraction of teaching behavior associations through the fusion of dynamic similarity calculation and spatiotemporal graph convolution.

[0051] Weight optimization module: Based on the behavioral feature association graph, a state space for reinforcement learning is constructed. The multimodal fusion weights are dynamically adjusted through a reward feedback mechanism, and teaching behavior trend prediction and evaluation report are output. Since traditional methods mostly use static weights or heuristic rules, this module achieves adaptive adjustment of fusion weights and forward-looking prediction of teaching behavior by coupling graph neural networks with the course learning reinforcement mechanism.

[0052] In this embodiment, it should be specifically noted that in the data preprocessing module, dynamic interference compensation includes performing multimodal feature extraction and spatiotemporal alignment operations, specifically:

[0053] First, the raw data stream of classroom teaching is captured in real time using a customized audio-visual terminal. This data stream contains three types of heterogeneous data: video frame sequences, audio waveforms, and text logs (X). raw ={V frame， A wave T log});

[0054] Next, for the video frame sequence, a pre-trained 3D convolutional neural network is used to extract spatial features from the video frame sequence, generating a spatiotemporal feature vector containing the teacher's body movements and blackboard writing. The expression for the spatiotemporal feature vector is:

[0055]

[0056] Where, φ v V represents the spatiotemporal feature vector. frame W represents a video frame sequence. v The weight parameters of the C3D model, obtained through pre-training, are used for feature extraction. v The bias parameters of the C3D model, obtained through pre-training, are used to adjust the output. v The video feature dimension is represented by 256, and t represents the time step (i.e., the total length of the sequence, with a value range of 0.1 seconds per step). The function of this formula is to capture the teacher's body movements and blackboard writing by extracting the spatiotemporal features of the video.

[0057] Simultaneously, for the audio waveform, the Mel-frequency cepstral coefficients (MFCC) conversion method is applied to convert the audio waveform into time-frequency characteristics, and time derivative characteristics are superimposed to generate an audio feature vector. The expression for the audio feature vector is as follows:

[0058]

[0059] Where, φ a A represents the audio feature vector. wave Represents an audio waveform, d a The value of MFCC represents the feature dimension (set to 40). MFCC stands for Mel-frequency cepstral coefficient function, used to generate the basic time-spectral features. ΔMFCC stands for time derivative function, used to calculate the dynamic changes of the spectrum. The purpose of this formula is to enhance the temporal information of audio features and ensure alignment with video features.

[0060] For text log data, the Bidirectional Encoder Representation (BERT) model is used for text embedding, outputting a fixed-dimensional text feature vector. The expression for the text feature vector is:

[0061]

[0062] Where, φ q T represents the text feature vector. log Represents text log data, d q[CLS] represents the BERT output dimension (its value is set to 768), BERT stands for Bidirectional Encoder Representation Model, used for semantic encoding, and [CLS] represents the classification label, used to output aggregated text features. The function of this formula is to extract deep semantic features of the text and fix the dimension for alignment.

[0063] Finally, all feature vectors are aligned to the same time step t in the time dimension to ensure that video action features and audio features are strictly synchronized in the time dimension, thereby forming a spatiotemporally aligned feature group (Φ={φ v ,φ a ,φ q This provides high-precision input for subsequent environmental interference prediction, thereby avoiding data distortion caused by time misalignment;

[0064] The specific steps for training a long short-term memory network prediction model based on historical environmental data are as follows:

[0065] Using historical datasets, the intensity of light fluctuations, background noise in decibels, and network jitter frequency ( X hist The historical environment dataset is a collection of N samples used to train the prediction model; ΔL i This represents the intensity of the light fluctuation in the i-th sample. This represents the background noise value in decibels for the i-th sample. The i-th sample represents the network jitter frequency or latency fluctuation value, where i represents the sample index (from 1 to N, and N represents the total number of samples in the historical dataset) and is used as a training sample.

[0066] A dynamic prediction model is constructed using a gated recurrent unit network. This model receives real-time environmental parameters as input and outputs a three-dimensional prediction vector containing the intensity of light interference, noise interference, and network interference. The expression for the dynamic prediction model is as follows:

[0067]

[0068] Where, ε t Let ε represent the perturbation vector (i.e., the three-dimensional prediction vector) predicted at time step t, and ε t =[ε L , ε Q , ε J ],ε L ε represents the predicted value of the intensity of light interference. Q ε represents the predicted value of noise interference intensity. J The predicted value representing the intensity of network interference, GRU env This represents a gated recurrent unit (GRU) network, a specific model used for predicting environmental disturbances. W gru ,bgru The trainable parameters of a GRU network, namely the model's weights and biases, are obtained by analyzing a historical dataset X. hist The training yielded ΔL t , This represents the environmental parameters collected in real time at the current time step t;

[0069] Simultaneously, a nonlinear mapping function is designed to convert the predicted interference intensity into a feature confidence weight vector. Each element in this weight vector corresponds to a reliability evaluation value for a specific feature dimension, with a value range from 0 to 1. Values ​​approaching 1 indicate high feature confidence, while values ​​approaching 0 indicate the need to activate a compensation mechanism. The formula for the feature confidence weight vector is:

[0070]

[0071] Among them, Λ t This represents the feature confidence weight vector calculated at time step t, where each element's value is between 0 and 1, used to evaluate the reliability of the corresponding feature dimension: approaching 1 indicates that the feature dimension is reliable and should be retained; approaching 0 indicates that the feature dimension is unreliable and a compensation mechanism needs to be activated. σ represents the Sigmoid activation function, used to compress the output value to the interval (0,1), and tanh represents the hyperbolic tangent activation function, used to optimize the input vector ε. t Perform nonlinear transformation, W λe This represents a trainable weight matrix with dimension (d). v +d a )×3, which maps the 3-dimensional perturbation vector to (d v +d a ) dimensional space, b λe This represents a trainable bias vector with dimension (d) v +d a ), d v d represents the video feature dimension (its value is set to 256). a This represents the feature dimension of MFCC (its value is set to 40);

[0072] The process of generating the calibrated feature vector is as follows:

[0073] Based on the 3D prediction vector output by the dynamic prediction model, a compensation matrix corresponding to each feature dimension is generated. The expression for the compensation matrix is ​​as follows:

[0074] C t =W c ·ε t +b c ;

[0075] Among them, C tLet represent the compensation matrix, which shows the correction amount for video and audio features at time step t, with dimension . ε t Let ε represent the perturbation vector (i.e., the three-dimensional prediction vector) predicted at time step t, and ε t =[ε L ,ε Q ,ε J ],b c This represents a trainable bias vector used to adjust the compensation matrix, with dimensions of . W c This represents a trainable weight matrix used to map a 3D prediction vector to a compensation matrix, with dimensions of 1.

[0076] A confidence-weighted fusion mechanism is adopted: the spatiotemporally aligned feature groups and the compensation matrix are linearly combined according to their confidence weights. Specifically, the original data is retained for feature dimensions with high confidence, while the compensation value is replaced for feature dimensions with low confidence. The expression is as follows:

[0077]

[0078] in, This represents the calibrated feature vector, which is the output of the data preprocessing module. Λ t φ represents the feature confidence weight vector calculated at time step t. v Represents the spatiotemporal eigenvector, φ a C represents the audio feature vector. t This represents the compensation matrix, and ⊙ represents the element-wise multiplication operator. This indicates that the original data is retained for feature dimensions with high confidence, (1-Λ) t )⊙C t This indicates that feature dimensions with low confidence levels are replaced with compensation amounts;

[0079] Because it is unaffected by the physical environment, the text feature vectors are directly concatenated to the final output, resulting in calibrated feature vectors with uniform dimensions and strictly corresponding time steps. (dimension is) Its data structure is compatible with the input specifications of subsequent multimodal fusion modules, and the expression for the calibration feature vector is:

[0080]

[0081] Where, φ q Represents the text feature vector. This represents all calibrated video and audio features from time step 1 to t.

[0082] In this embodiment, it is specifically necessary to explain that in the multimodal fusion module, aligning the spatiotemporal dependency between video actions and speech emotions through dynamic time offset weights includes performing feature dimension decomposition, modal recalibration, and dynamic time offset weight generation. Specifically, when the calibrated feature vector from the data preprocessing module is received... First, the feature vector is decomposed into video components according to modality type. audio components and text components The video component represents the extracted video spatial features, the audio component represents the transformed temporal-spectral features, and the text component represents the encoded text embedding features. All components maintain a consistent time step. Then, a trainable projection matrix is ​​used... Mapping each modal component to a unified embedding space, the expression is:

[0083]

[0084] Among them, E m Let represent the embedded features, i.e., the output result obtained after projection transformation, and m represent the modality type index (v for video, a for audio, and q for text), with dimensions of . (Taking 512) means that it unifies the original features of different modalities to the same embedding dimension, preparing for subsequent fusion calculations. The trainable projection matrix is ​​used to linearly transform high-dimensional or low-dimensional raw features to a specified unified embedding space. M represents the original modal feature matrix, which is the input data and represents the original modal components decomposed from the calibrated feature vector. In the formula, M specifically refers to any one of Ve, Ae, and Te, and its expression is M = Ve, Ae, Te. The video component is processed by the video projection matrix, the audio component is processed by the audio projection matrix, and the text component is processed by the text projection matrix. The output is video embedding features, audio embedding features, and text embedding features with unified dimensions.

[0085] Subsequently, based on video and audio embedding features, a gated recurrent unit (GRU) model is used to learn the spatiotemporal offset pattern. This GRU model iteratively processes the concatenated vector of video and audio features at time steps, outputting a hidden state vector. The expression for the hidden state vector is as follows:

[0086] h τe =GRU offset ([E v [:,τe];E a [:,τe]];W gru b gru );

[0087] Among them, hτe The hidden state vector represents the hidden state of the Gated Recurrent Unit (GRU) model at time step τe. It is the encoding result of the GRU model of the information at the current and previous time steps τe, with dimensions of 1. GRU offset This represents a gated recurrent unit (GRU) model used to learn the spatiotemporal offset pattern between video and audio. The subscript offset is used to distinguish it from the GRU model used earlier. v E represents the video embedding features, specifically the video modal feature vector after processing with the projection matrix. a W represents the audio embedding features, i.e., the audio modal feature vector after processing by the projection matrix. τe represents the time step, i.e., the position index of the currently processed data in the time series, τe∈[1,t], where t represents the time step length (total sequence length). gru ,b gru This represents the trainable parameters of the GRU network, namely the weights and biases of the GRU model. Dynamic temporal offset weights are generated through a fully connected layer and a sigmoid activation function. These weights range from zero to one and are used to quantify the spatiotemporal dependency strength between video actions and speech emotions. Weights approaching one indicate a strong dependency, and those approaching zero indicate a weak dependency. The output of this step is a unified embedding feature and dynamic temporal offset weights, providing the input basis for subsequent multi-head attention fusion, thereby ensuring intermodal dimensionality compatibility and spatiotemporal alignment accuracy. The expression for the dynamic temporal offset weights is:

[0088]

[0089] Among them, W td Let b represent the trainable weight matrix used to map the hidden states to a scalar. td σ represents the trainable bias term, used to adjust the baseline of the weight output, and σ represents the Sigmoid activation function, which maps any real number to the interval (0,1). The dynamic time offset weight of time step τe is represented by its value ∈ [0,1], which quantifies the dependence strength between video and audio at that time point;

[0090] The Transformer architecture employing a multi-head self-attention mechanism generates a fused feature tensor by performing multi-head attention and spatiotemporal fusion operations. Specifically, this involves unifying the output dimensions of video embedding features, audio embedding features, and text embedding features (E...). m =[E v E a E qUsing dynamic time offset weights as input, the video embedding features, audio embedding features, and text embedding features are first concatenated along the feature dimensions to form a fused embedding feature. Then, query vector, key vector, and value vector are generated by query weight matrix, key weight matrix, and value weight matrix, respectively. The expression for the query vector is:

[0091] Qc = W Qc ·E m ;

[0092] The expression for the key vector is:

[0093] Kc = W Kc ·E m ;

[0094] The expression for a value vector is:

[0095] Vc = W Vc ·E m ;

[0096] Among them, E m W represents the embedded features. Qc W Kc W Vc This represents a trainable weight matrix used to project embedded features onto a query, key, and value space, with dimension d. head ×3d embed , including 3D embed Let Qc represent the dimension of the attention head (set to 64), Qc represent the query vector (the query component in the attention mechanism), Kc represent the key vector (the key component in the attention mechanism), and Vc represent the value vector (the value component in the attention mechanism). Then, for each attention head, an attention score matrix is ​​calculated. This matrix integrates semantic similarity and spatiotemporal relevance. Semantic similarity is obtained by taking the dot product of the query vector and the key vector and dividing by a scaling factor. Spatiotemporal relevance is obtained by multiplying the dynamic time offset weights by the time distance function and then multiplying by the offset enhancement coefficient. The time distance function uses an exponential decay method to calculate the weights of neighboring time steps. The formula for the attention score matrix is:

[0097]

[0098] Among them, S y [τe,j] represents the score matrix of the attention head y at time steps τe and j (semantic and spatiotemporal correlation), Qc y Kc represents the query vector of the attention head y. y This represents the key vector of the attention head y, where τe and j represent time step indices (the current processing time point), ranging from 1 to t, and d... headλc represents the dimension of the attention head, used to scale the dot product result and prevent gradient vanishing. λc represents the offset enhancement coefficient, used to balance the contribution of the attention mechanism and spatiotemporal constraints; it is typically set to 0.5. The dynamic time offset weight of time step τe is represented, and its value is ∈ [0,1]. δc(τe,j) represents the time distance function, which is used to calculate the proximity between time steps τe and j. For example, δc(τe,j)=exp(-|τe-j|) gives higher weight to neighboring time steps.

[0099] Subsequently, the attention score matrix is ​​normalized using the Softmax function, and the output of each attention head is obtained by weighting the vector. The outputs of all attention heads are concatenated along the feature dimension, and a fused feature tensor is generated through the output projection matrix. The dimension of this fused feature tensor is strictly matched to the time step and is directly used as input to the association graph construction module, thus supporting subsequent behavior dependency analysis. This operation ensures high-precision alignment between video actions and speech emotions, improving the robustness and efficiency of multimodal fusion. The formula for the fused feature tensor is:

[0100]

[0101] Where Gc represents the fused feature tensor, d hidden Represents the hidden layer dimension, head y The output of attention head y is obtained by normalizing the score using Softmax and then weighting it into a vector. Hv represents the total number of attention heads, and W... O This represents the output projection matrix, used to map the outputs of all attention heads to a unified space.

[0102] In this embodiment, it is specifically necessary to explain that in the association graph construction module, the dynamic generation of the adjacency matrix based on cosine similarity includes performing node vector generation and graph structure initialization operations as well as dynamic adjacency matrix generation operations. Specifically, when the fusion feature tensor Gc output from the multimodal fusion module is received, the fusion feature tensor Gc is first decomposed into multiple node vectors Uc={u1,u2,…,u t}(in d represents the behavioral characteristics at the first time step. hidden The dimension of the fusion feature is inherited from the previous module and its value is set to 1024. Each node vector represents the behavioral feature representation of a specific time step. The dimension of all node vectors is consistent and inherited from the hidden dimension of the fusion feature tensor.

[0103] Next, initialize an all-zero matrix as the initial state Lc of the adjacency matrix. (0) ∈R t×tThe dimension of this matrix is ​​determined by the number of nodes and it is a square matrix. Then, the cosine similarity values ​​between all pairwise node vectors are calculated to form the original similarity matrix. Each element in this matrix represents the original association strength between features at the corresponding time step. The expression for the original similarity matrix is:

[0104]

[0105] Among them, S raw [ci,cj] represents the element in the ci-th row and cj-th column of the original similarity matrix (the original association strength between the behavioral feature node at time step ci and the behavioral feature node at time step cj), with a value range of [-1,1]. Values ​​closer to 1 indicate stronger association because the vector directions of the two nodes are more similar; values ​​closer to -1 indicate opposite directions; and 0 indicates no association. ci Let u represent the vector of the ci-th node (the behavioral feature representation at the ci-th time step). cj This represents the vector of the cj-th node (the behavioral feature representation of the cj-th time step);

[0106] Based on the numerical distribution of the original similarity matrix, a Gaussian mixture model is used for fitting analysis, and a dynamic threshold parameter is automatically calculated. This dynamic threshold is determined by the mean and standard deviation components of the Gaussian mixture model combined with an adjustment factor, and is used to distinguish between strong and weak associations. The formula for calculating the dynamic threshold is:

[0107]

[0108] Where, θ dyn This represents a dynamic threshold, used to determine the strength of a correlation. Similarity values ​​above this value are considered significant and retained; values ​​below this value are filtered out. The value range is [-1, 1], μe. GMM The mean parameter represents the Gaussian mixture model. αc represents the standard deviation parameter of the Gaussian mixture model, and αc represents the adjustment factor used to control the degree of deviation of the threshold relative to the center of the distribution. For example, when αc is 1.5, the threshold is set to the mean plus 1.5 times the standard deviation, which will filter out about 93% of the (MISSING) weak associations in the distribution (assuming the distribution is approximately normal).

[0109] The original similarity matrix is ​​binarized using a dynamic threshold to generate a binarized adjacency matrix. In this matrix, a value of 1 indicates a significant association between nodes, while a value of zero indicates no significant association. The expression for the binarized adjacency matrix is:

[0110]

[0111] Among them, Lc binary[ci,cj] represents the element in the ci-th row and cj-th column of the binary adjacency matrix (whether there is a significant association between nodes ci and cj). The value range is only 1 (significant association exists) or 0 (no significant association). The function of this expression is to replace the fixed threshold and adaptively capture significant behavioral dependencies (such as only keeping the top 20% of strongly related edges). Finally, it outputs a binary adjacency matrix and a set of node vectors, providing the input basis for subsequent graph convolutional network processing.

[0112] Extracting dependency weights between nodes through graph convolutional networks includes performing dependency weight extraction operations using spatiotemporal graph convolutional networks and constructing behavioral feature association graphs. Specifically, when a binary adjacency matrix and a set of node vectors are received, the node vector set is first linearly transformed using a trainable weight matrix to generate projected node features.

[0113] Next, a multi-head graph attention mechanism is used to calculate the attention coefficients between nodes. This mechanism concatenates the projected feature vectors of each pair of nodes and processes them with a trainable parameter vector and a non-linear activation function to obtain unnormalized attention scores. The attention scores are then standardized to transform them into a probability distribution and multiplied by the corresponding elements in the binary adjacency matrix. This refines the dependency weights while preserving significant associations, generating a weighted adjacency matrix. The expression for the weighted adjacency matrix is ​​as follows:

[0114]

[0115] Where Lc[ci,cj] represents the element in the ci-th row and cj-th column of the weighted adjacency matrix (the final dependency weight of node ci on node cj), with a value range of [0,1]. A larger value indicates a stronger dependency. Wy represents the trainable weight matrix with dimension 1. (where d) head The value is set to 512), which is used to perform linear transformation on the node features to extract higher-level feature representations. ay represents the attention mechanism parameter vector (dimension: ), ci,cj represent the node feature vectors (i.e., the feature representations of the ci-th and cj-th nodes), and LeakyReLU represents the activation function of the linear rectified unit with leakage (expressed as...). And αv=0.01), softmax cj This indicates that the standardized index is calculated for the dimension containing the variable cj, and ⊥ represents the matrix transpose symbol. The purpose of this formula is to refine the weights on the dynamically selected edge set and capture non-linear dependencies (such as the causal strength of the teacher's action on the student's response).

[0116] Subsequently, a time decay mechanism is introduced. A decay factor is calculated based on the difference between the time steps of corresponding nodes, and this factor is multiplied by the corresponding element in the weighted adjacency matrix to highlight the dependency relationship between temporally adjacent nodes. The expression is as follows:

[0117] Lc final [ci,cj]=Lc[ci,cj]·exp(-βv|ci-cj|);

[0118] Among them, Lc final [ci,cj] represents the element in the ci-th row and cj-th column of the final adjacency matrix (the final dependency strength of node ci on node cj). exp(·) represents the natural exponential function, which is an exponential operation with the natural constant e (approximately 2.71828) as the base. In this formula, its role is to generate a decay coefficient. The characteristic of the exponential function is that the output value decreases rapidly as the input value (-βv|ci-cj|) increases, but it is always a positive number. This ensures that the larger the time difference between node pairs, the more severely their association weight is decayed. βv represents the time decay coefficient, which is set to 0.2. The larger the βv value, the faster the decay, meaning that the system emphasizes very close behavioral dependencies; the smaller the βv value, the slower the decay, meaning that the system will consider dependencies over a longer time range. |ci-cj| represents the absolute difference between the time step indices corresponding to node ci and node cj.

[0119] Finally, by integrating the node vector set, the weighted adjacency matrix, and the set of non-zero edges, a weighted directed behavioral feature association graph is constructed. Ec represents the non-zero edge set (defined by non-zero elements of Lc). This graph fully represents the dependence strength and temporal constraints between teaching behavior features. This graph is directly used as the input of the weight optimization module, supporting dynamic weight adjustment based on reinforcement learning.

[0120] In this embodiment, it should be specifically explained that in the weight optimization module, the construction of the reinforcement learning state space based on the behavioral feature association graph includes performing state space construction operations and action space design operations. Specifically, when the behavioral feature association graph is received from the association graph construction module... First, based on the node feature vectors and adjacency matrix of the graph, a multi-head graph attention network is used to encode the topological structure information of the graph. This network generates the state representation vector of key nodes by calculating the normalized attention weights between nodes and aggregating the features of neighboring nodes. The expression of the state representation vector is as follows:

[0121]

[0122] Among them, s k This represents the state representation vector of the k-th key node. This represents a multi-head splicing operation, where Hv represents the total number of attention heads, σ represents the non-linear activation function, and ∑ cj∈N(ci) This indicates that the summation is performed on all neighboring nodes cj of node ci (as defined by Lc). Wy represents the normalized attention weight (calculated based on Lc[ci,cj]) between nodes ci and cj in the y-th attention head. hv Let u represent the trainable weight matrix for the y-th attention head. cj The original feature vector of node cj represents the state representation vector of all key nodes, which together constitute the state space Sv = {s1, s2, ..., sj} for reinforcement learning. Kv (Kv represents the number of key nodes);

[0123] Next, the action space is designed as the adjustment amount for the multimodal fusion weights. This adjustment amount corresponds to three dimensions: video modality weights, audio modality weights, and text modality weights. Based on the state representation vector, action vectors are generated through a policy network. This policy network includes a state encoding layer and a policy output layer. The state encoding layer uses a hyperbolic tangent activation function to process the input state, and the policy output layer uses a normalized exponential function to constrain the output to a probability distribution between zero and one, thus obtaining the action vector representing the adjustment amount of each modality weight. The output of this step is the state space and action vectors, providing the foundation for the subsequent reward feedback mechanism. The formula for generating the fusion weight adjustment amount through the policy network is:

[0124] a k =π lm (sk) = softmax(W) π ·tanh(W enc s k ));

[0125] Among them, a k This represents the fusion weight adjustment (i.e., the output of the policy network, a vector containing three elements, each representing the adjustment to the video, audio, and text modal weights, respectively). Its value is constrained to the range [0,1] by the softmax function, and the sum of the three elements is 1, which meets the requirements of weight allocation. π lm Let lm represent the policy network, where lm indicates that the policy network has a set of trainable parameters (i.e., W). π and W enc ), s k Let W represent the state representation vector of the k-th key node, where softmax represents the standardized exponential function. π W represents the policy output weight matrix. enc This represents the state-encoded weight matrix. The purpose of this formula is to map the state to a weight adjustment strategy, supporting dynamic rebalancing of multimodal contributions.

[0126] The reward feedback mechanism dynamically adjusts the multimodal fusion weights and outputs teaching behavior trend predictions and evaluation reports. This includes performing reward calculations and course learning operations, as well as prediction and report generation operations. Specifically, when the state representation vector and action vector are received, a reward function is designed. This reward function consists of three components: an accuracy reward component calculated based on the similarity between the behavior prediction sequence generated by the policy network and the real behavior sequence derived from labeled data; a sparsity penalty component calculated based on the L1 norm of the action vector; and a course reward component calculated based on the training progress indicator function. The reward value is a weighted sum of the three components. The expression of the reward function is as follows:

[0127] r k =ηv·Sim(P pred ,P true )-λl||a k ||1+γl·I l ;

[0128] Where, r k Let represent the reward value obtained at the k-th key node, and ηv, λl, and γl represent weight coefficients (set to 1.0, 0.1, and 0.5 respectively), used to adjust the relative importance of the three components in the reward function (ηv is the accuracy reward weight, which determines the contribution of the similarity between the predicted behavior and the actual behavior to the total reward; the larger the value, the more the system tends to improve prediction accuracy; λl is the sparsity penalty weight, used to control the penalty for model complexity; the larger the value, the more the system tends to choose a simpler (more zero values) weight allocation scheme to prevent overfitting; γl is the course reward weight, used to determine the influence of the course learning mechanism on the total reward; the larger the value, the more the system follows the course learning progress from easy to difficult). Sim represents the similarity function, used to calculate the similarity between two vectors (P). pred and P true The similarity between the two directions is calculated, and the output is a value between -1 and 1. The closer the value is to 1, the more accurate the prediction. pred P represents the predicted behavioral sequence. true Represents the actual sequence of actions, ||a k ||1 represents the fusion weight adjustment amount a k L1 norm, I l This represents the course indicator function (a function that changes according to the training process (such as the current training round)). Its purpose is to achieve "course learning", that is, to gradually increase the difficulty of the learning task as training progresses, thereby accelerating the convergence process of the model and improving the final performance.

[0129] Based on the state sequence composed of state representation vectors arranged in chronological order, a temporal convolutional network model is used to predict teaching behavior trends, expressed as follows:

[0130] P trend =TCN s1, s2, ..., s Kv ;

[0131] Among them, P trend This represents a teaching behavior trend prediction sequence, used to predict the probability or likelihood of various teaching behaviors (such as "explanation", "questioning", "interaction") occurring within a future period. For example, it predicts that "the probability of teacher-student interaction will increase from 30% to 70% in the next 5 minutes". TCN stands for Temporal Convolutional Network, which processes the state sequence and outputs the behavior probability distribution for future time steps. Finally, based on the final fusion weight vector obtained after multiple rounds of iterative optimization by adjusting the fusion weights, it is combined with the behavior trend prediction results and the behavior feature association graph to generate an evaluation report through a structured report template. The report includes three parts: level assessment, data evidence, and conclusion recommendations. The data evidence comes from the dependency weights in the behavior feature association graph, and the conclusion recommendations combine the trend prediction results to provide directions for teaching improvement. This step outputs optimized weights, behavior trend predictions, and the evaluation report, completing the closed-loop optimization.

[0132] It should be noted that the descriptions of each embodiment in the above embodiments have different focuses. For parts that are not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0133] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0134] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0135] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0136] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0137] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0138] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. An online multi-terminal interconnection production and teaching integration precision management system based on big data analysis, characterized in that, Specifically, it includes: The module includes a data preprocessing module, a multimodal fusion module, a correlation graph construction module, and a weight optimization module. Data preprocessing module: When collecting multimodal raw data streams of classroom teaching through customized audio and video terminals, the gated recurrent unit network prediction model trained based on historical environmental data performs dynamic interference compensation on video frame sequences, audio waveforms and text logs, and outputs calibrated feature vectors; Dynamic interference compensation includes performing multimodal feature extraction and spatiotemporal alignment operations, specifically: First, the original data stream of classroom teaching is captured in real time through a customized audio and video terminal. This data stream contains three types of heterogeneous data: video frame sequences, audio waveforms, and text logs. Next, for the video frame sequence, a pre-trained 3D convolutional neural network is used to extract spatial features from the video frame sequence, generating a spatiotemporal feature vector containing the teacher's body movements and blackboard writing content. Meanwhile, for audio waveforms, the Mel frequency cepstral coefficient conversion method is applied to convert the audio waveform into time-frequency spectral features, and the time derivative features are superimposed to generate audio feature vectors; For text log data, a bidirectional encoder representation model is used for text embedding processing, outputting a fixed-dimensional text feature vector; Finally, all feature vectors are aligned to the same time step in the time dimension to form a spatiotemporally aligned feature group. The specific steps for training a gated recurrent unit network prediction model based on historical environmental data are as follows: The intensity of light fluctuations, background noise in decibels, and network jitter frequency in historical datasets were used as training samples. A dynamic prediction model is constructed through a gated cyclic unit network. This model receives real-time collected environmental parameters as input and outputs a three-dimensional prediction vector containing light interference intensity, noise interference intensity, and network interference intensity. Meanwhile, a nonlinear mapping function is designed to convert the predicted interference intensity into a feature confidence weight vector. Each element in the weight vector corresponds to a reliability evaluation value of a specific feature dimension, with a value range of 0 to 1. A value close to 1 indicates high feature confidence, while a value close to 0 indicates that a compensation mechanism needs to be activated. The process of generating the calibrated feature vector is as follows: Based on the three-dimensional prediction vector output by the dynamic prediction model, a compensation matrix corresponding to each feature dimension is generated. A confidence-weighted fusion mechanism is adopted: the spatiotemporal feature vectors and audio feature vectors in the spatiotemporal aligned feature group are linearly combined with the compensation matrix according to the confidence weight. Specifically, the original data is retained for feature dimensions with high confidence, and the compensation is replaced for feature dimensions with low confidence. The text feature vector is directly concatenated to the compensated spatiotemporal feature vector and audio feature vector to generate a calibrated feature vector with uniform dimensions and corresponding time steps. Multimodal fusion module: After receiving the calibrated feature vector, it adopts a Transformer architecture with multi-head self-attention mechanism, and aligns the spatiotemporal dependency between video action and speech emotion through dynamic time offset weights to generate a fused feature tensor; Aligning the spatiotemporal dependency between video actions and speech emotions through dynamic time offset weights involves performing feature dimension decomposition, modality recalibration, and dynamic time offset weight generation. Specifically, when a calibrated feature vector is received from the data preprocessing module, the feature vector is first decomposed into video, audio, and text components according to modality type. The video component represents the extracted video spatial features, the audio component represents the transformed temporal-spectral features, and the text component represents the encoded text embedding features. Then, each modality component is mapped to a unified embedding space through a trainable projection matrix. The video component is processed through a video projection matrix, the audio component through an audio projection matrix, and the text component through a text projection matrix, outputting video embedding features, audio embedding features, and text embedding features with unified dimensions. Subsequently, based on video and audio embedding features, a gated recurrent unit model is used to learn the spatiotemporal offset pattern. This gated recurrent unit model iteratively processes the concatenated vector of video and audio features at time steps, outputs the hidden state vector, and generates dynamic time offset weights through a fully connected layer and a sigmoid activation function. The weights are between zero and one and are used to quantify the spatiotemporal dependence strength between video actions and speech emotions. A weight close to one indicates a strong dependence, and a weight close to zero indicates a weak dependence. Association Graph Construction Module: Maps the fused feature tensor to graph structure data, dynamically generates an adjacency matrix based on cosine similarity, and extracts the dependency weights between nodes through a graph convolutional network to construct a behavioral feature association graph; The dynamic generation of adjacency matrix based on cosine similarity includes the execution of node vector generation and graph structure initialization operations as well as dynamic adjacency matrix generation operations. Specifically, when the fusion feature tensor output from the multimodal fusion module is received, the fusion feature tensor is first decomposed into multiple node vectors according to the time step dimension. Next, an all-zero matrix is ​​initialized as the initial state of the adjacency matrix. The dimension of this matrix is ​​determined by the number of nodes and is a square matrix. Then, the cosine similarity values ​​between all node vectors are calculated to form the original similarity matrix. Each element in this matrix represents the original association strength between the features at the corresponding time steps. Based on the numerical distribution of the original similarity matrix, a Gaussian mixture model is used for fitting analysis, and a dynamic threshold parameter is automatically calculated. This dynamic threshold is determined by the mean component and standard deviation component of the Gaussian mixture model combined with an adjustment factor. The original similarity matrix is ​​binarized based on a dynamic threshold to generate a binarized adjacency matrix. In this matrix, a value of 1 indicates a significant association between nodes, and a value of 0 indicates no significant association. The final output is a binarized adjacency matrix and a set of node vectors. Weight optimization module: Constructs a state space for reinforcement learning based on the behavioral feature association graph, dynamically adjusts the multimodal fusion weights through a reward feedback mechanism, and outputs a teaching behavior trend prediction and evaluation report; Constructing the state space of reinforcement learning based on behavioral feature association graphs includes performing state space construction operations and action space design operations. Specifically, when a behavioral feature association graph is received from the association graph construction module, the topological structure information of the graph is first encoded using a multi-head graph attention network based on the node feature vectors and adjacency matrix of the graph. This network generates the state representation vectors of key nodes by calculating the normalized attention weights between nodes and aggregating the features of neighboring nodes. The state representation vectors of all key nodes together constitute the state space of reinforcement learning. Next, the action space is designed as the adjustment amount of the multimodal fusion weights, which corresponds to the three dimensions of video modal weights, audio modal weights, and text modal weights. Based on the state representation vector, action vectors are generated through a policy network. This policy network includes a state encoding layer and a policy output layer. The state encoding layer uses the hyperbolic tangent activation function to process the input state, and the policy output layer uses the normalized exponential function to constrain the output to a probability distribution between zero and one, thereby obtaining the action vector representing the adjustment amount of each modal weight. The reward feedback mechanism dynamically adjusts the multimodal fusion weights and outputs teaching behavior trend prediction and evaluation reports, including reward calculation and course learning operations, and prediction and report generation operations. Specifically, when the state representation vector and action vector are received, a reward function is designed first. This reward function consists of three components: the accuracy reward component is calculated based on the similarity between the behavior prediction sequence generated by the policy network and the real behavior sequence from the labeled data; the sparsity penalty component is calculated based on the first norm of the action vector; and the course reward component is calculated based on the training progress indication function. The reward value is the weighted sum of the three components. Based on the state sequence composed of state representation vectors arranged in chronological order, a temporal convolutional network model is used to predict teaching behavior trends. This temporal convolutional network processes the state sequence and outputs the behavior probability distribution for future time steps. Finally, based on the final fusion weight vector obtained after multiple rounds of iterative optimization of the fusion weight adjustment, it is combined with the behavior trend prediction results and the behavior feature correlation graph to generate an assessment report through a structured report template. The report includes three parts: level assessment, data evidence, and conclusion recommendations. The data evidence comes from the dependency weights in the behavior feature correlation graph, and the conclusion recommendations combine the trend prediction results to provide directions for teaching improvement. 2.The online multi-end interconnected production and teaching fusion precise management system based on big data analysis according to claim 1, characterized in that: The generation of the fused feature tensor includes performing a multi-head attention mechanism and spatiotemporal fusion operation. Specifically, the video embedding feature, audio embedding feature, and text embedding feature with a unified output dimension, as well as dynamic time offset weights, are taken as input. First, the video embedding feature, audio embedding feature, and text embedding feature are concatenated along the feature dimension to form a fused embedding feature. Then, query vector, key vector, and value vector are generated by query weight matrix, key weight matrix, and value weight matrix, respectively. Next, for each attention head, the attention score matrix is ​​calculated. This matrix fuses semantic similarity and spatiotemporal correlation. Semantic similarity is obtained by taking the dot product of the query vector and the key vector and dividing by the scaling factor. Spatiotemporal correlation is obtained by multiplying the product of the dynamic time offset weights and the time distance function and then multiplying by the offset enhancement coefficient. The time distance function uses an exponential decay form to calculate the weights of neighboring time steps. Subsequently, the attention score matrix is ​​normalized using the Softmax function, and the output of a single attention head is obtained by adding a weighted vector. The outputs of all attention heads are concatenated along the feature dimension, and a fused feature tensor is generated by the output projection matrix. The dimension of this fused feature tensor matches the time step.

3. The online multi-terminal interconnected industry-education integration precision management system based on big data analysis according to claim 2, characterized in that: The extraction of dependency weights between nodes through graph convolutional networks includes performing a spatiotemporal graph convolutional network operation to extract dependency weights and a behavioral feature association graph construction operation. Specifically, when a binary adjacency matrix and a set of node vectors are received, the node vector set is first linearly transformed using a trainable weight matrix to generate projected node features. Next, a multi-head graph attention mechanism is used to calculate the attention coefficients between nodes. This mechanism concatenates the projected feature vectors of each pair of nodes and processes them with a trainable parameter vector and a non-linear activation function to obtain an unnormalized attention score. The attention score is then standardized to transform it into a probability distribution and multiplied with the corresponding elements in the binary adjacency matrix to generate a weighted adjacency matrix. Subsequently, a time decay mechanism is introduced, which calculates the decay factor based on the difference between the time steps of the nodes and multiplies the factor by the corresponding element in the weighted adjacency matrix. Finally, by integrating the node vector set, the weighted adjacency matrix, and the non-zero edge set defined by the non-zero elements of the weighted adjacency matrix, a weighted directed behavioral feature association graph is constructed.

Images