Classroom teaching digital portrait construction method based on time domain multi-modal data fusion

Abstract

The invention provides a classroom teaching digital portrait construction method based on time domain multi-modal data fusion. The method comprises the following steps: acquiring multi-modal data in classroom teaching; carrying out preprocessing on the multi-modal data; according to a time axis, carrying out synchronous processing on the preprocessed multi-modal data; performing feature extraction on the multi-modal data after the synchronization processing to obtain a plurality of feature vectors in the multi-modal data; performing label labeling on the feature vector to obtain a feature label; carrying out fusion processing on the plurality of feature vectors to obtain a fused feature vector; and carrying out fusion processing on the feature tag and the fused feature vector to obtain a classroom digital portrait on the time axis.

Description

Technical Field

[0001] The present invention relates to the field of data processing, and in particular to a method for constructing a digital portrait of classroom teaching based on time-domain multimodal data fusion. Background Art

[0002] With the rapid development of educational informatization, the demand for intelligent evaluation of classroom teaching process is growing. However, the teaching activities and teacher-student behaviors in the classroom teaching process are complex and diverse, and a single data source cannot fully and accurately reflect the whole picture of this process. Traditional teaching evaluation methods mainly rely on manual classroom observation and delayed analysis, which makes it difficult to achieve real-time tracking and intelligent analysis of the teaching process, and face the dilemma of high labor cost, low efficiency and high subjectivity. In addition, in the process of technology driving classroom teaching analysis from manual classroom observation to semi-automatic and automated analysis, problems such as single data source, weak applicability of identification results, and poor educational interpretability have been highlighted, and there is a lack of multi-dimensional, in-depth, and interpretable comprehensive analysis of classroom teaching elements and process elements. Therefore, how to integrate multimodal data, build a comprehensive digital portrait of classroom teaching, and improve the coupling of technical logic and educational theory has become an important direction for improving teaching quality and personalized teaching.

[0003] In recent years, multimodal data fusion technology has gradually attracted attention in the field of education. By integrating multiple devices such as cameras, microphones and sensors, data such as teachers' and students' facial expressions, voiceprints, and movements can be collected in real time. These multimodal data can not only provide rich original classroom information, but also be deeply analyzed through deep learning and image processing technologies to mine key features and behavior patterns in the teaching process. However, the existing multimodal data fusion methods still have deficiencies in consistency in the time dimension and complementarity in the spatial dimension, as well as the lack of a theoretical framework to guide data feature mining and educational interpretation, making it difficult to fully and dynamically present the classroom teaching process and reflect the underlying teaching rules of the classroom. Therefore, a digital portrait construction method based on timeline-based classroom multimodal data fusion came into being, aiming to solve the limitations of existing technologies and achieve intelligent analysis and detailed characterization of the classroom teaching process. Summary of the invention

[0004] The purpose of the embodiment of the present invention is to provide a method for constructing a digital portrait of classroom teaching based on time-domain multimodal data fusion, so as to solve the problem in the prior art that the overall picture of the classroom teaching process cannot be intelligently analyzed.

[0005] In order to solve the above problems, in a first aspect, the present invention provides a method for constructing a digital portrait of classroom teaching based on time domain multimodal data fusion, the method comprising:

[0006] Obtain multimodal data from classroom instruction;

[0007] Preprocessing the multimodal data;

[0008] According to the time axis, the pre-processed multimodal data is synchronously processed;

[0009] Perform feature extraction on the synchronously processed multimodal data to obtain multiple feature vectors in the multimodal data;

[0010] Labeling the feature vector to obtain a feature label;

[0011] Performing fusion processing on the plurality of feature vectors to obtain a fused feature vector;

[0012] The feature labels and the fused feature vectors are fused to obtain a digital portrait of the classroom on the timeline.

[0013] In one possible implementation, the multimodal data includes classroom video data, classroom audio data, teaching content data and teaching media data; the classroom video data is a first time series signal, the classroom audio data is a second time series signal, the teaching content data is a third time series signal, and the teaching media data is a fourth time series signal.

[0014] In a possible implementation manner, the preprocessing of the multimodal data specifically includes:

[0015] Performing denoising on the multimodal data by wavelet transform or Fourier transform;

[0016] Standardize the denoised multimodal data;

[0017] The standardized multimodal data is subjected to outlier detection processing to obtain preprocessed multimodal data.

[0018] In a possible implementation, the multimodal data includes classroom video data, classroom audio data, teaching content data and teaching media data; the classroom video data is a first time series signal, the classroom audio data is a second time series signal, the teaching content data is a third time series signal, and the teaching media data is a fourth time series signal; the synchronous processing of the pre-processed multimodal data according to the time axis specifically includes:

[0019] According to the formula Calculate the minimum distance between any two time series signals among the first time series signal, the second time series signal, the third time series signal and the fourth time series signal; where d(x i ,y j ) is the distance between two time series signals, xi is any time series signal, y j For x i Different time series signals; DTW(X,Y) is the minimum distance;

[0020] According to the formula R XY (τ)=∑ t X(t)Y(t+τ) calculates the correlation between any two time series signals.

[0021] In a possible implementation, extracting features from the synchronously processed multimodal data to obtain a feature vector specifically includes:

[0022] According to the long short-term memory network LSTM, through i t =σ(W i ·[h t-1 ,x t ]+b i ), f t =σ(W f ·[h t-1 ,x t ]+b f ), o t =σ(W o ·[h t-1 ,x t ]+b o ), h t =o t *tanh(C t ) calculates the eigenvector; where i t 、f t , o t are the activation values ​​of the input gate, forget gate, and output gate, respectively, and C t is the cell state, h t is the hidden state and σ is the Sigmoid function.

[0023] In a possible implementation, labeling the feature vector to obtain the feature label specifically includes:

[0024] According to the deep learning algorithm, the classroom behaviors are labeled to obtain behavior labels; the behavior labels include classroom behaviors including teacher behaviors and student behaviors, the teacher behaviors include teacher lecturing, teacher asking questions, and teacher writing on the blackboard; the student behaviors include student raising hands, student standing, student listening, and student demonstration;

[0025] According to the order of occurrence of classroom teaching activities, teaching events are marked to obtain event labels; the event labels include creating situations, teaching new knowledge, practicing and evaluating, group exploration, organization and management, and homework;

[0026] According to the expression recognition algorithm, the facial expressions of teachers and students are extracted to obtain expression labels;

[0027] According to the audio analysis algorithm and expression label recognition algorithm, the emotional state of teachers and students is extracted to obtain emotional labels;

[0028] The behavior label, the event label, the expression label and the emotion label constitute a feature label.

[0029] In a possible implementation, the fusing the plurality of feature vectors to obtain a fused feature vector specifically includes:

[0030] according to Calculate the weight of each feature vector; where f i is the eigenvector, α i is the weight, w is the attention weight vector;

[0031] according to Calculate the fused feature vector; where N is the number of feature vectors and F′ is the fused feature vector.

[0032] In a possible implementation, the fusing process of the feature label and the fused feature vector to obtain a digital portrait of the classroom on the timeline specifically includes:

[0033] pass Modeling multimodal feature data with feature labels, where X = {x1, x2, …, x N} is the training set of multimodal feature vectors, Y = {y1,y2,…,y N} is the corresponding feature label set, θ is the model parameter, and L is the loss function;

[0034] According to the timeline, according to the start time of each event End time Build Timeline

[0035] For the feature vector at each moment, the principal component analysis method is used to reduce the dimension.

[0036] In a possible implementation, the method further includes:

[0037] According to the digital classroom portrait, the teaching content summary is refined through Summary = Τ (A (t), C (t)); where Τ (·) represents the text mining function, A (t) is the classroom audio data A (t), and C (t) is the teaching content data;

[0038] according to and EventTimeDistributio n={(t i ,E i |i=1,2,...,N} to judge the teaching mode, where E i is the teaching event, Ι(·) represents the indicator function, N represents the total number of events, t i Indicates the time when the event occurred;

[0039] according to Analyze the changing trend of the emotional state of teachers or students; where S(t) is the emotional state, obtained through emotional labels;

[0040] according to Analyze the frequency of various behaviors of teachers and students in the classroom; B i For behavior, it is obtained through behavior tags;

[0041] according to Analyze students' participation in class; TotalBehaivorCount is the total number of participations, and ActiveBehaviorCount is the number of active behavior participations;

[0042] according to Analyze students’ answers in class; among them, S j is the knowledge point score, K represents the number of knowledge points;

[0043] according to and Horizontal comparison of classroom portraits of the same grade and the same subject D i , identify the commonalities and differences between different classrooms; among them, classroom portrait D i Create a portrait of the classroom;

[0044] According to Individual Change = {D t1 ,D t2 ,...,D tn}and Vertical comparison of classroom portraits of the same teacher in different periods or classes t , where t n Indicates different time points.

[0045] In a second aspect, the present invention provides a device for constructing a classroom teaching digital portrait based on time domain multimodal data fusion, the device comprising:

[0046] An acquisition module, which is used to acquire multimodal data in classroom teaching;

[0047] A preprocessing module, the preprocessing module is used to preprocess the multimodal data;

[0048] A synchronization processing module, the synchronization processing module is used to synchronize the pre-processed multimodal data according to the time axis;

[0049] A feature extraction module, wherein the feature extraction module is used to extract features from the synchronously processed multimodal data to obtain a plurality of feature vectors in the multimodal data;

[0050] A label marking module, wherein the label marking module is used to label the feature vector to obtain a feature label;

[0051] A first fusion processing module, the first fusion processing module is used to perform fusion processing on the multiple feature vectors to obtain a fused feature vector;

[0052] The second fusion processing module is used to perform fusion processing on the feature labels and the fused feature vectors to obtain a digital portrait of the classroom on a timeline.

[0053] In a third aspect, the present invention provides a device including a memory and a processor, wherein the memory is used to store programs, and the processor is used to execute any method described in the first aspect.

[0054] In a fourth aspect, the present invention provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute any method described in the first aspect.

[0055] In a fifth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the method described in any one of the first aspects is implemented.

[0056] By applying the classroom teaching digital portrait construction method based on time domain multimodal data fusion provided by the embodiment of the present invention, the following technical effects are achieved:

[0057] 1. Propose a method for constructing a digital portrait of classroom teaching by integrating multimodal data (video data, audio data, teaching content data, and teaching media data), establish a unified timeline framework and data mapping model, and improve the consistency and complementarity of multimodal data fusion in the classroom.

[0058] 2. Propose a new idea for the intelligent construction of classroom portraits based on the combination of timeline and teaching events, use teaching events as a starting point to carefully depict the evolutionary pattern of the classroom teaching process, and improve the readability and applicability of classroom teaching analysis.

[0059] 3. It is proposed to realize the mapping from teaching behavior to teaching events through classification or time series analysis, so as to improve the educational interpretability of classroom teaching analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 A schematic diagram of a flow chart of a method for constructing a digital portrait of classroom teaching based on time-domain multimodal data fusion provided in the first embodiment of the present invention;

[0061] Figure 2 for Figure 1 A specific embodiment of

[0062] Figure 3 for Figure 1 The scene graph;

[0063] Figure 4 A schematic diagram of the structure of a device for constructing a digital portrait of classroom teaching based on time-domain multimodal data fusion provided in the second embodiment of the present invention. DETAILED DESCRIPTION

[0064] The present application will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the relevant invention, rather than to limit the invention. It should also be noted that, for ease of description, only the parts related to the relevant invention are shown in the accompanying drawings.

[0065] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0066] The first, second, third and fourth numbers are for distinction only and have no other meaning.

[0067] Figure 1 The flowchart of the method for constructing a classroom teaching digital portrait based on time domain multimodal data fusion provided by the first embodiment of the present invention is shown in FIG. The method is applied in a teaching scenario, and the execution subject of the method is a device with processing functions such as a server, a processor, and a terminal. Figure 1 As shown, the method comprises the following steps:

[0068] Step 110, obtaining multimodal data in classroom teaching;

[0069] Among them, multimodal data includes classroom video data, classroom audio data, teaching content data and teaching media data; classroom video data is the first time series signal, classroom audio data is the second time series signal, teaching content data is the third time series signal, and teaching media data is the fourth time series signal.

[0070] The classroom video data collected by the camera in the classroom can be obtained, which is represented as the first time series signal V(t). The classroom audio data collected by the microphone in the classroom is obtained, which is represented as the second time series signal A(t). The image taken by the camera can be obtained for image text analysis, so as to extract text information as teaching content data, which is represented as the third time series signal C(t). The teaching content data includes but is not limited to PPT, blackboard writing, etc. The operation data of teachers and students on teaching equipment (such as smart large screen, dot matrix smart pen, etc.) in the smart classroom is obtained, which is represented as the fourth time series signal M(t).

[0071] Step 120, preprocessing the multimodal data;

[0072] Among them, step 120 specifically includes: denoising the multimodal data through wavelet transform or Fourier transform; standardizing the denoised multimodal data so that different data sources have the same dimension and range; performing outlier detection on the standardized multimodal data to obtain preprocessed multimodal data.

[0073] Specifically, wavelet transform or Fourier transform can be used to remove the noise frequency component, for example, by smoothing the signal through a low-pass filter H(s):

[0074]

[0075] Among them, the transfer function of the low-pass filter H(s) can be expressed as Among them, ω c is the cutoff frequency and n is the order of the filter.

[0076] We detect outliers by calculating the standard deviation σ of the data points from the mean μ. If a data point x i If the standard deviation exceeds a certain threshold multiple of the mean, it is considered an outlier, that is:

[0077] |x i -μ|>kσ

[0078] in, k is a predefined constant.

[0079] Step 130, performing synchronization processing on the pre-processed multimodal data according to the time axis;

[0080] Specifically, in order to ensure the time synchronization of multimodal data, a unified timestamp is required. Time alignment algorithms such as Dynamic Time Warping (DTW) or Cross-Correlation Function (CCF) are used to ensure the temporal consistency of data.

[0081] According to the formula Calculate the minimum distance between any two time series signals among the first time series signal, the second time series signal, the third time series signal and the fourth time series signal; where d(x i ,y j ) is the distance between any two time series signals, x i is any time series signal, y j For x i Different time series signals; DTW(X,Y) is the minimum distance.

[0082] According to the formula R XY (τ)=∑ t X(t)Y(t+τ) calculates the correlation between any two time series signals, where X(t) is any time series signal, Y(t) is a time series signal different from X(t), and τ is the time delay.

[0083] Step 140, performing feature extraction on the synchronously processed multimodal data to obtain a plurality of feature vectors in the multimodal data;

[0084] Specifically, feature extraction uses deep learning models, such as long short-term memory (LSTM) or self-attention, to extract meaningful information from multimodal data. Assume that X is the input data set, that is, the multimodal data after synchronous processing, and f is the feature extraction function. According to the long short-term memory network LSTM, through i t =σ(W i ·[h t-1 ,x t ]+b i ), f t =σ(W f ·[h t-1 ,x t ]+b f ), o t =σ(W o ·[h t-1 ,x t ]+b o ), h t =o t *tanh(Ct ) calculates the eigenvector; where i t 、f t , o t are the activation values ​​of the input gate, forget gate, and output gate, respectively, and C t is the cell state, h t is the hidden state, σ is the Sigmoid function, and these parameters constitute the feature vector.

[0085] Step 150, labeling the feature vectors of the multimodal data to obtain feature labels;

[0086] Among them, the construction of the labeling system involves an automatic or semi-automatic labeling process, which specifically includes:

[0087] According to the deep learning algorithm, classroom behaviors are annotated to obtain behavior labels; behavior labels include classroom behaviors, including teacher behaviors and student behaviors. Teacher behaviors include teacher lectures, teacher questions, and teacher blackboard writing; student behaviors include students raising their hands, students standing, students listening, and students demonstrating;

[0088] According to the order of classroom teaching activities, teaching events are marked to obtain event labels; event labels include creating situations, teaching new knowledge, practicing and evaluating, group exploration, organization and management, and homework;

[0089] According to the expression recognition algorithm, the facial expressions of teachers and students are extracted to obtain expression labels;

[0090] According to the audio analysis algorithm and expression label recognition algorithm, the emotional state of teachers and students is extracted to obtain emotional labels;

[0091] Behavior labels, event labels, expression labels and emotion labels constitute feature labels.

[0092] In one example, the tagging system can be represented as:

[0093] L={T,B,E,S}

[0094] Among them, T represents event label, B represents behavior label, E represents expression category, and S represents emotion label.

[0095] Each tag in the tag system has its specific meaning and scope of application. The following will explain the definition, applicable scenarios and categories of each tag in detail and illustrate them with examples.

[0096] 1. Event Tag (T)

[0097] Definition: Event labels (T) are used to represent specific teaching events in the classroom teaching process. Event labels are realized by time series analysis of multiple behavior labels and are used to describe key teaching activities in the classroom teaching process.

[0098] Applicable scenarios: Mainly used to describe the important nodes and activities in the classroom teaching process as a whole. For example, teaching activities that take place within a certain period of time (such as teaching new knowledge and group exploration).

[0099] Event tags include: Setting the Scene, Teaching New Knowledge, Practice and Assessment, Group Inquiry, Classroom Management, and Homework.

[0100] Example: At the beginning of the class, the teacher guides students into the learning state through stories or cases, which is marked as the Setting the Scene event. Before the end of the class, the teacher issues exercises and students answer them, which is marked as the Practice and Assessment event.

[0101] 2. Behavior label (B)

[0102] Definition: Behavior labels (B) are used to represent the behavior status of an individual (teacher or student) in a specific period of time (such as teaching new knowledge events). Behavior labels are implemented through behavior recognition technology and are suitable for analyzing the specific behavior of individuals in the classroom.

[0103] Applicable scenarios: Mainly used to identify and record various behaviors of teachers and students in the classroom. For example, teachers' teaching behaviors (such as explaining and asking questions) and students' learning behaviors (such as raising hands and standing).

[0104] Behavior tags include: Teacher behavior: Teaching, Questioning, Walking Around, Writing on Blackboard, Operating PPT, Operating smart screen, Pushing exercises, Picking people; Student behavior: Raising Hand, Standing Up, Bending Over, Listening, Presentation, Answering with dot-matrix pen

[0105] Example: In the group exploration event stage, the system recognizes that the teacher frequently walks around, indicating that the teacher is actively paying attention to the students' learning status. In the teaching new knowledge event stage, many students raise their hands, indicating that the students have questions that need to be answered by the teacher.

[0106] 3. Expression category (E)

[0107] Definition: Expression category (E) is used to represent the facial expression state of an individual (teacher or student) within a specific period of time. The recognition of expression category is usually achieved through facial expression recognition technology, which is suitable for individual-level analysis.

[0108] Applicable scenarios: Mainly used to analyze the emotional state of individuals, including the facial expressions of teachers and students. For example, it can be used to detect the psychological activities of teachers when explaining a certain knowledge point, or the reactions of students when listening to a lecture.

[0109] Expression labels include: Happy, Angry, Sad, Surprised, Disgusted, Fearful, and Neutral.

[0110] Example: During the teaching of new knowledge event, when the teacher is teaching new knowledge, the facial expression recognition system detects that the teacher maintains a happy expression most of the time, which indirectly indicates that the teacher is actively interacting with the students; when the students are listening to the lecture, they show a surprised expression for a period of time, which may be because they have an interest in or doubt about a certain knowledge point.

[0111] 4. Sentiment label (S)

[0112] Definition: Emotional labels (S) are used to represent the overall emotional state of individual teachers and student groups in a specific period of time. The identification of emotional labels is usually achieved through multimodal data fusion such as voice emotion analysis and facial expression analysis, reflecting the emotional investment of teachers and students in teaching.

[0113] Applicable scenarios: Emotional labels are applicable to both individual-level emotional state analysis (such as the teacher's emotional state) and group-level emotional state analysis (such as the emotional state of the entire class). In a classroom environment, emotional labels can help understand students' overall learning emotions and reactions.

[0114] Emotional labels include: Positive, Negative, Neutral, Excited, Bored, and Anxious.

[0115] Example: In the group exploration event phase, the voice emotion analysis system detected that most students showed a positive emotional state, indicating that the students were interested in the exploration content. In the teaching new knowledge event phase, the teacher showed an anxious emotional state, which may be because the questions were difficult or the students' feedback was poor.

[0116] Below, we will explain the sample application of the label system:

[0117] Here is an example of how these labels might be applied in a real classroom:

[0118] Situation description: The teacher is teaching new knowledge points and the students are listening.

[0119] Label system application:

[0120] Event label (T): The current time period is marked as the Teaching New Knowledge event.

[0121] Behavior label (B): When the teacher is teaching new knowledge points, he / she asks questions and writes on the blackboard. When the students are listening, some of them raise their hands to express their questions.

[0122] Expression category (E): The teacher maintained a happy expression during the explanation; some students showed surprised expressions during the lecture.

[0123] Emotional label (S): Teachers and students showed positive emotional states, indicating that teachers and students had a high level of emotional investment in teaching.

[0124] In this way, tags can make it possible to more comprehensively and intuitively track and analyze the various behaviors and emotional states of teachers and students during classroom teaching, thereby providing a basis for improving teaching quality and personalized teaching.

[0125] Step 160, fusing multiple feature vectors to obtain a fused feature vector;

[0126] Step 170, fuse the feature labels and the fused feature vectors to obtain a digital portrait of the classroom on the timeline.

[0127] Constructing a portrait involves combining the feature vector F and the label system L to form a comprehensive description of the evolution pattern of the classroom teaching process and visualize the events. It involves multimodal fusion models, such as the Multi-Modal Attention Network, which outputs a digital representation D that includes classroom teaching events, teacher and student behaviors and emotions, and goal achievement.

[0128] The calculation formula of the multimodal attention network is:

[0129]

[0130]

[0131] r i =∑ j α ij h j

[0132] Among them, e ij and e ik are the attention scores, representing the attention scores of modality i and modality k, respectively, and v α It is used to convert the output after nonlinear transformation into a scalar attention vector. The tanh hyperbolic tangent function introduces nonlinear transformation, enabling the network to capture the complex relationship and nonlinear dependency between the features of modality i and modality j. α Is a learnable weight matrix responsible for the input concatenation vector Perform a linear transformation, which maps the input concatenated vector to the new feature space. is the eigenvector h of mode i i and the eigenvector s of mode j j The concatenation operation connects two feature vectors into a longer vector by dimension. ij is the attention weight, b αis a learnable bias vector used to adjust the offset after each linear transformation. Its function is to adjust the input to make the model more flexible and prevent the linear transformation from being too rigid. i is the fused feature vector, h i and j are the eigenvectors of different modes. When there are multiple modes, r i Represents the result of weighted summation of feature vectors of all modalities. Regardless of the number of modalities, the attention mechanism will model the features of different modalities and generate the fused feature vector r through weighted summation. i , used for downstream tasks.

[0133] The goal of this module is to generate a comprehensive description of the evolution pattern of the classroom teaching process based on the results of the previous processing and feature extraction, and to construct a digital portrait of the classroom including a timeline. The specific implementation steps are as follows:

[0134] 1. Multimodal data fusion:

[0135] After feature extraction, data from different modalities need to be effectively fused in time and space. A multi-modal attention network (MAN) is used to weight features to ensure that key information is highlighted.

[0136] Mathematical description:

[0137] Let F be the feature matrix extracted from multimodal data, containing N feature vectors f i , i = 1, 2, ..., N. Using the attention mechanism, calculate the weight α of each feature vector i :

[0138]

[0139] Among them, w is the attention weight vector. The fused feature vector F′ is expressed as:

[0140]

[0141] 2. Model training:

[0142] Using labeled multimodal data, classroom behaviors are modeled using machine learning algorithms. Commonly used models include support vector machines (SVM), random forests (RF), and neural networks (such as LSTM, Transformer, etc.).

[0143] Mathematical description:

[0144] Let X = {x1, x2, …, x N} is the training set of multimodal feature vectors, Y = {y1,y2,…,y N} is the corresponding label set, the training process of model M can be expressed as:

[0145]

[0146] Among them, θ is the model parameter, L is the loss function, and N is the number of multimodal data in the training set and label set.

[0147] 3. Behavior recognition and classification:

[0148] Use the trained model to perform behavior recognition and classification on new classroom data, and identify the specific behaviors of teachers and students, such as teachers lecturing, teachers patrolling, teachers asking questions, teachers writing on the blackboard, teachers operating PPT, teachers operating smart screens, teachers pushing exercises, teachers randomly selecting people, students standing, students raising their hands, students demonstrating, students listening, students bending over desks, and students answering with dot pens.

[0149] Mathematical description:

[0150] For the input feature vector x, the output of model M is the predicted behavior category

[0151]

[0152] 4. Event division and annotation:

[0153] Based on the identified behavior, the classroom teaching process is divided and labeled through classification or time series analysis, classification algorithms such as SVM or RF, and sequence analysis algorithms such as Time Series Clustering (TSC). Event division includes creating situations, teaching new knowledge, practicing and evaluating, group exploration, organization and management, and homework.

[0154] Mathematical description:

[0155] Suppose that each event E i A set of behavioral sequences The annotation of the event can be expressed as:

[0156]

[0157] 6. Timeline construction and visualization:

[0158] According to the timeline framework, the identified classroom activities and teaching events are arranged and visualized in time. The timeline not only shows the time point of the event, but also includes the event duration and related feature information.

[0159] Mathematical description:

[0160] Assume that for each event E i With start time and end time The duration of the event is Build timeline T:

[0161]

[0162] 7. Multimodal feature fusion visualization:

[0163] The data and features of different modes are presented on a unified visualization platform to provide multi-level analysis of the classroom teaching process. The visualization includes teacher behavior (such as asking questions, patrolling, and writing on the blackboard), student behavior (such as raising hands, standing, and listening), emotional state (such as positive and negative), and teaching content (such as PPT and writing on the blackboard).

[0164] Mathematical description:

[0165] The multimodal features at each time t are fused and visualized. Let f t is the feature vector at time t, and the principal component analysis (PCA) or t-SNE method is used to reduce the dimension to two-dimensional or three-dimensional space:

[0166] f t ′=PCA(f t )or f t ′=t-SNE(f t )

[0167] Through the above steps, the final digital portrait of the classroom can not only comprehensively depict various events and behaviors in the classroom teaching process, but also provide educators with detailed analysis and feedback to help them optimize teaching strategies, improve teaching quality and students' learning experience.

[0168] Furthermore, the application also includes:

[0169] According to the digital classroom portrait, the teaching content summary is extracted through Summary = Τ(A(t), C(t)); where Τ(·) represents the text mining function, A(t) is the classroom audio data, and C(t) is the teaching content data;

[0170] according to and EventTimeDistributio n={(t i ,E i |i=1,2,...,N} to judge the teaching mode, where E i is the teaching event, Ι(·) represents the indicator function, N represents the total number of events, ti Indicates the time when the event occurred;

[0171] according to Analyze the changing trend of the emotional state of teachers or students; where S(t) is the emotional state, obtained through emotional labels;

[0172] according to Analyze the frequency of various behaviors of teachers and students in the classroom; B i For behavior, it is obtained through behavior tags;

[0173] according to Analyze students' participation in class; TotalBehaivorCount is the total number of participations, and ActiveBehaviorCount is the number of active behavior participations;

[0174] according to Analyze students’ answers in class; among them, S j is the knowledge point score, K represents the number of knowledge points;

[0175] according to and Horizontal comparison of classroom portraits of the same grade and the same subject D i , identify the commonalities and differences between different classrooms; among them, classroom portrait D i Create a portrait of the classroom;

[0176] According to Individual Change = {D t1 ,D t2 ,...,D tn}and Vertical comparison of classroom portraits of the same teacher in different periods or classes t , where t n Indicates different time points.

[0177] The following is a detailed description of how to use classroom teaching digital portraits for evaluation:

[0178] Based on the constructed classroom digital portrait, a comprehensive and fine-grained analysis of the classroom teaching process is conducted, and a specific analysis report is generated ( Figure 3 ). It not only includes the evaluation of teacher and student behavior in a class, but also involves dynamically tracking the evolution trend of classroom portraits of different subjects, grades, and teachers. Through in-depth analysis of classroom activities, it helps educators better understand the dynamic evolution of the classroom, improve teaching effectiveness, and provide personalized teaching suggestions. Specific report indicator examples:

[0179] 1. Classroom portrait static analysis report:

[0180] Based on a single classroom digital portrait, the entire classroom teaching process is comprehensively analyzed, and reports are intelligently generated using text mining and multimodal data analysis results. Specific analysis dimensions include but are not limited to:

[0181] Teaching content summary: Based on classroom teaching audio data A(t) and teaching content data C(t), the teaching content summary is extracted through text mining method. The mathematical description is as follows:

[0182] Summary = Τ(A(t), C(t))

[0183] Where Τ(·) represents the text mining function.

[0184] Teaching events: Analyze different teaching events in the whole class i The number, time and sequence of occurrences of teaching events (such as teaching new knowledge, practicing and evaluating, etc.) are analyzed, and the key links in the teaching process are identified based on the distribution of teaching events, and the teaching mode is judged. The mathematical description is as follows:

[0185]

[0186] EventTimeDistribution = {(t i ,E i |i=1,2,...,N}

[0187] Where Ι(·) represents the indicator function, N represents the total number of events, and t i Indicates the time when the event occurred.

[0188] Teaching performance:

[0189] 1) Teaching emotion: Analyze the changing trend of the emotional state S(t) of teachers and students. The mathematical description is as follows:

[0190]

[0191] 2) Teacher-student behavior: Analyze various behaviors of teachers and students in the classroom i The frequency of occurrence and its rationality. The mathematical description is as follows:

[0192]

[0193] Among them, M represents the total number of behaviors and T represents the total time.

[0194] 3) Teacher-student dialogue: Analyze the teacher and student responses in class, such as the content of the questions, the content of the follow-up questions, the number of questions asked, the number of follow-up questions, the waiting time for answers, etc.

[0195] 4) Student participation: Analyze students’ participation and enthusiasm in class, including raising hands, answering questions, raising heads, etc. The mathematical description is as follows:

[0196]

[0197] 5) Classroom answers: Analyze students’ classroom answers and knowledge point scores based on teaching equipment data during classroom teaching j Etc. The mathematical description is as follows:

[0198]

[0199] Among them, K represents the number of knowledge points.

[0200] 6) Teacher’s teaching style: Analyze the changes and rationality of teachers’ teaching style during the teaching process.

[0201] 7) Teacher’s speaking speed: Analyze the speed and rationality of the teacher’s teaching.

[0202] 8) Blackboard writing standards: Analyze the high-frequency words, character standards, line spacing, etc. in the teacher’s blackboard writing.

[0203] Goal Achieved:

[0204] Comprehensive teaching performance and teaching event indicators are used to analyze the degree of achievement of classroom goals.

[0205] 2. Classroom portrait dynamic analysis report:

[0206] Based on the comparative analysis of multiple classroom digital portraits, the underlying teaching rules and mechanisms of the classroom are explored from both the global and individual levels:

[0207] Global level: horizontal comparison of classroom portraits of the same grade and the same subject D i , identify the commonalities and differences between different classes, and provide targeted suggestions for teachers to improve their teaching. The mathematical description is as follows:

[0208]

[0209] Where P represents the number of different classes.

[0210] Individual level: longitudinal comparison of classroom portraits of the same teacher in different periods or classes t , analyze the changes in teaching effects and help teachers continuously optimize teaching strategies. The mathematical description is as follows:

[0211] IndividualChange={D t1 ,D t2 ,...,D tn}

[0212]

[0213] Among them, t n Indicates different time points.

[0214] By applying the classroom teaching digital portrait construction method based on time domain multimodal data fusion provided by the embodiment of the present invention, the following technical effects are achieved:

[0215] 1. Propose a method for constructing a digital portrait of classroom teaching by integrating multimodal data (video data, audio data, teaching content data, and teaching media data), establish a unified timeline framework and data mapping model, and improve the consistency and complementarity of multimodal data fusion in the classroom.

[0216] 2. Propose a new idea for the intelligent construction of classroom portraits based on the combination of timeline and teaching events, use teaching events as a starting point to carefully depict the evolutionary pattern of the classroom teaching process, and improve the readability and applicability of classroom teaching analysis.

[0217] 3. It is proposed to realize the mapping from teaching behavior to teaching events through classification or time series analysis, so as to improve the educational interpretability of classroom teaching analysis.

[0218] Figure 4 A schematic diagram of the structure of a classroom teaching digital portrait construction device based on time domain multimodal data fusion provided in the second embodiment of the present invention, as shown in FIG. Figure 4 As shown, the device includes: an acquisition module 410, a preprocessing module 420, a synchronization processing module 430, a feature extraction module 440, a label marking module 450, a first fusion processing module 460 and a second fusion processing module 470.

[0219] The acquisition module 410 is used to acquire multimodal data in classroom teaching;

[0220] The preprocessing module 420 is used to preprocess the multimodal data;

[0221] The synchronization processing module 430 is used to synchronize the pre-processed multimodal data according to the time axis;

[0222] The feature extraction module 440 is used to extract features from the synchronously processed multimodal data to obtain multiple feature vectors in the multimodal data;

[0223] The labeling module 450 is used to label the feature vector to obtain a feature label;

[0224] The first fusion processing module 460 is used to perform fusion processing on multiple feature vectors to obtain a fused feature vector;

[0225] The second fusion processing module 470 is used to perform fusion processing on the feature labels and the fused feature vectors to obtain a digital portrait of the classroom on the timeline.

[0226] Among them, multimodal data includes classroom video data, classroom audio data, teaching content data and teaching media data; classroom video data is the first time series signal, classroom audio data is the second time series signal, teaching content data is the third time series signal, and teaching media data is the fourth time series signal.

[0227] Furthermore, the preprocessing module 420 preprocesses the multimodal data, specifically including:

[0228] De-noising multimodal data through wavelet transform or Fourier transform;

[0229] Standardize the denoised multimodal data;

[0230] The standardized multimodal data is subjected to outlier detection processing to obtain preprocessed multimodal data.

[0231] Furthermore, the synchronization processing module 430 performs synchronization processing on the pre-processed multimodal data according to the time axis, specifically including:

[0232] According to the formula Calculate the minimum distance between any two time series signals among the first time series signal, the second time series signal, the third time series signal and the fourth time series signal; where d(x i ,y j ) is the distance between two time series signals, x i is any time series signal, y j For x i Different time series signals; DTW(X,Y) is the minimum distance;

[0233] According to the formula R XY (τ)=∑ t X(t)Y(t+τ) calculates the correlation between any two time series signals.

[0234] Furthermore, the feature extraction module 440 extracts features from the synchronously processed multimodal data to obtain feature vectors including:

[0235] According to the long short-term memory network LSTM, through i t =σ(W i ·[h t-1 ,x t ]+b i ), f t =σ(W f·[h t-1 ,x t ]+b f ), o t =σ(W o ·[h t-1 ,x t ]+b o ), h t =o t *tanh(C t ) calculates the eigenvector; where i t 、f t , o t are the activation values ​​of the input gate, forget gate, and output gate, respectively, and C t is the cell state, h t is the hidden state and σ is the Sigmoid function.

[0236] Furthermore, the labeling module 450 labels the feature vector, and the obtained feature labels specifically include:

[0237] According to the deep learning algorithm, classroom behaviors are annotated to obtain behavior labels; behavior labels include classroom behaviors, including teacher behaviors and student behaviors. Teacher behaviors include teacher lectures, teacher questions, and teacher blackboard writing; student behaviors include students raising their hands, students standing, students listening, and students demonstrating;

[0238] According to the order of classroom teaching activities, teaching events are marked to obtain event labels; event labels include creating situations, teaching new knowledge, practicing and evaluating, group exploration, organization and management, and homework;

[0239] According to the expression recognition algorithm, the facial expressions of teachers and students are extracted to obtain expression labels;

[0240] According to the audio analysis algorithm and expression label recognition algorithm, the emotional state of teachers and students is extracted to obtain emotional labels;

[0241] Behavior labels, event labels, expression labels and emotion labels constitute feature labels.

[0242] Furthermore, the first fusion processing module 460 performs fusion processing on multiple feature vectors to obtain a fused feature vector specifically including:

[0243] according to Calculate the weight of each feature vector; where f i is the eigenvector, α i is the weight, w is the attention weight vector;

[0244] according to Calculate the fused feature vector; where N is the number of feature vectors and F′ is the fused feature vector.

[0245] Furthermore, the second fusion processing module 470 performs fusion processing on the feature labels and the fused feature vectors to obtain a classroom digital portrait on the time axis, which specifically includes:

[0246] pass Modeling multimodal feature data with feature labels, where X = {x1, x2, …, x N} is the training set of multimodal feature vectors, Y = {y1,y2,…,y N} is the corresponding feature label set, θ is the model parameter, and L is the loss function;

[0247] According to the timeline, according to the start time of each event End time Build Timeline

[0248] For the feature vector at each moment, the principal component analysis method is used to reduce the dimension.

[0249] Furthermore, the device further includes a classroom analysis module 480, which is used to:

[0250] According to the digital classroom portrait, the teaching content summary is refined through Summary = Τ (A (t), C (t)); where Τ (·) represents the text mining function, A (t) is the classroom audio data A (t), and C (t) is the teaching content data;

[0251] according to and EventTimeDistributio n={(t i ,E i |i=1,2,...,N} to judge the teaching mode, where E i is the teaching event, Ι(·) represents the indicator function, N represents the total number of events, t i Indicates the time when the event occurred;

[0252] according to Analyze the changing trend of the emotional state of teachers or students; where S(t) is the emotional state, obtained through emotional labels;

[0253] according to Analyze the frequency of various behaviors of teachers and students in the classroom; B i For behavior, it is obtained through behavior tags;

[0254] according to Analyze students' participation in class; TotalBehaivorCount is the total number of participations, and ActiveBehaviorCount is the number of active behavior participations;

[0255] according to Analyze students’ answers in class; among them, S j is the knowledge point score, K represents the number of knowledge points;

[0256] according to and Horizontal comparison of classroom portraits of the same grade and the same subject D i , identify the commonalities and differences between different classrooms; among them, classroom portrait D i Create a portrait of the classroom;

[0257] According to Individual Change = {D t1 ,D t2 ,...,D tn}and Vertical comparison of classroom portraits of the same teacher in different periods or classes t , where t n Indicates different time points.

[0258] Embodiment 3 of the invention provides a device, including a memory and a processor, wherein the memory is used to store a program, and the memory can be connected to the processor via a bus. The memory can be a non-volatile memory, such as a hard disk drive and a flash memory, and the memory stores a software program and a device driver. The software program can perform various functions of the above method provided in the embodiment of the present invention; the device driver can be a network and interface driver. The processor is used to execute the software program, and when the software program is executed, the method provided in embodiment 1 of the present invention can be implemented.

[0259] Embodiment 4 of the present invention provides a computer program product including instructions. When the computer program product is run on a computer, the computer executes the method provided in Embodiment 1 of the present invention.

[0260] Embodiment 5 of the present invention provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the method provided in Embodiment 1 of the present invention is implemented.

[0261] The professionals should further realize that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to the function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0262] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0263] The above specific implementation methods further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above are only specific implementation methods of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for constructing a classroom teaching digital portrait based on time domain multimodal data fusion, characterized in that: The method comprises: Obtain multimodal data from classroom instruction; Preprocessing the multimodal data; According to the time axis, the pre-processed multimodal data is synchronously processed; Perform feature extraction on the synchronously processed multimodal data to obtain multiple feature vectors in the multimodal data; Labeling the feature vector to obtain a feature label; Performing fusion processing on the plurality of feature vectors to obtain a fused feature vector; The feature labels and the fused feature vectors are fused to obtain a digital portrait of the classroom on the timeline.

2. The method according to claim 1, characterized in that The multimodal data includes classroom video data, classroom audio data, teaching content data and teaching media data; the classroom video data is a first time series signal, the classroom audio data is a second time series signal, the teaching content data is a third time series signal, and the teaching media data is a fourth time series signal.

3. The method according to claim 1, characterized in that The preprocessing of the multimodal data specifically includes: Performing denoising on the multimodal data by wavelet transform or Fourier transform; Standardize the denoised multimodal data; The standardized multimodal data is subjected to outlier detection processing to obtain preprocessed multimodal data.

4. The method according to claim 2, characterized in that: The synchronous processing of the pre-processed multimodal data according to the time axis specifically includes: According to the formula Calculate the minimum distance between any two time series signals among the first time series signal, the second time series signal, the third time series signal and the fourth time series signal; where d(x i ,y j ) is the distance between two time series signals, x i is any time series signal, y j For x i Different time series signals; DTW(X,Y) is the minimum distance; According to the formula R XY (τ)=∑ t X(t)Y(t+τ) calculates the correlation between any two time series signals.

5. The method according to claim 1, characterized in that The step of extracting features from the synchronously processed multimodal data to obtain a feature vector specifically includes: According to the long short-term memory network LSTM, through i t =σ(W i ·[h t-1 ,x t ]+b i ), f t =σ(W f ·[h t-1 ,x t ]+b f ), o t =σ(W o ·[h t-1 ,x t ]+b o ), h t =o t *tanh(C t ) calculates the eigenvector; where i t 、f t , o t are the activation values ​​of the input gate, forget gate, and output gate, respectively, and C t is the cell state, h t is the hidden state, and σ is the Sigmoid function.

6. The method according to claim 1, characterized in that The labeling of the feature vector to obtain the feature label specifically includes: According to the deep learning algorithm, the classroom behaviors are labeled to obtain behavior labels; the behavior labels include classroom behaviors including teacher behaviors and student behaviors, the teacher behaviors include teacher lecturing, teacher asking questions, and teacher writing on the blackboard; the student behaviors include student raising hands, student standing, student listening, and student demonstration; According to the order of occurrence of classroom teaching activities, teaching events are marked to obtain event labels; the event labels include creating situations, teaching new knowledge, practicing and evaluating, group exploration, organization and management, and homework; According to the expression recognition algorithm, the facial expressions of teachers and students are extracted to obtain expression labels; According to the audio analysis algorithm and expression label recognition algorithm, the emotional state of teachers and students is extracted to obtain emotional labels; The behavior label, the event label, the expression label and the emotion label constitute a feature label.

7. The method according to claim 1, characterized in that The fusing of the plurality of feature vectors to obtain a fused feature vector specifically includes: according to Calculate the weight of each feature vector; where f i is the eigenvector, α i is the weight, w is the attention weight vector; according to Calculate the fused feature vector; where N is the number of feature vectors and F′ is the fused feature vector.

8. The method according to claim 1, characterized in that The fusion processing of the feature labels and the fused feature vectors to obtain a digital portrait of the classroom on the time axis specifically includes: pass Modeling multimodal feature data with feature labels, where X = {x1, x2, …, x N } is the training set of multimodal feature vectors, Y = {y1,y2,…,y N } is the corresponding feature label set, θ is the model parameter, and L is the loss function; According to the timeline, according to the start time of each event End time Build Timeline For the feature vector at each moment, the principal component analysis method is used to reduce the dimension.

9. The method according to claim 1, characterized in that: The method then further comprises: According to the digital classroom portrait, the teaching content summary is extracted through Summary = Τ(A(t), C(t)); where Τ(·) represents the text mining function, A(t) is the classroom teaching audio data A(t), and C(t) is the teaching content data; according to and EventTimeDistributio n={(t i ,E i |i=1,2,...,N} to judge the teaching mode, where E i is the teaching event, Ι(·) represents the indicator function, N represents the total number of events, t i Indicates the time when the event occurred; according to Analyze the changing trend of the emotional state of teachers or students; where S(t) is the emotional state, obtained through emotional labels; according to Analyze the frequency of various behaviors of teachers and students in the classroom; B i For behavior, it is obtained through behavior tags; according to Analyze students' participation in class; TotalBehaivorCount is the total number of participations, and ActiveBehaviorCount is the number of active behavior participations; according to Analyze students' classroom answers; among them, S j is the knowledge point score, K represents the number of knowledge points; according to and Horizontal comparison of classroom portraits of the same grade and the same subject D i , identify the commonalities and differences between different classes; among them, the classroom portrait D i Create a portrait of the classroom; According to Individual Change = {D t1 ,D t2 ,...,D tn }and Vertical comparison of classroom portraits of the same teacher in different periods or classes t , where t n Indicates different time points.

10. A classroom teaching digital portrait construction device based on time domain multimodal data fusion, characterized in that: The device comprises: An acquisition module, which is used to acquire multimodal data in classroom teaching; A preprocessing module, the preprocessing module is used to preprocess the multimodal data; A synchronization processing module, the synchronization processing module is used to synchronize the pre-processed multimodal data according to the time axis; A feature extraction module, wherein the feature extraction module is used to extract features from the synchronously processed multimodal data to obtain a plurality of feature vectors in the multimodal data; A label marking module, wherein the label marking module is used to label the feature vector to obtain a feature label; A first fusion processing module, the first fusion processing module is used to perform fusion processing on the multiple feature vectors to obtain a fused feature vector; The second fusion processing module is used to perform fusion processing on the feature labels and the fused feature vectors to obtain a digital portrait of the classroom on a timeline.

Images