Method for realizing experiment teaching behavior analysis based on deep convolutional neural network

Abstract

The invention discloses a method for realizing experiment teaching behavior analysis based on a deep convolutional neural network, and the method comprises the steps: collecting physical, chemical and student experiment teaching data, carrying out the preprocessing, and segmenting a physical, chemical and student experiment teaching process; extracting a plurality of features from the physical, chemical and biological experiment teaching data, and performing multi-modal feature fusion to form comprehensive feature representation; constructing a deep convolutional neural network model by using the fused features; training a deep convolutional neural network model, adjusting model parameters, and identifying teaching behaviors; analyzing the model identification result, and evaluating the frequency and duration of different teaching behaviors and the reaction and participation degree of the students; and constructing a description framework, converting a model identification result and evaluation data into a report, and outputting the report to a teacher. According to the method, the deep learning technology is combined, multi-modal data is comprehensively analyzed, key features are extracted, modal information is integrated through the feature fusion technology, a deep convolutional neural network model is constructed, and therefore teaching improvement and strategy adjustment are supported.

Description

Technical Field

[0001] The present invention relates to the technical field of methods specifically suitable for analyzing experimental teaching behaviors, and in particular to a method for analyzing experimental teaching behaviors based on a deep convolutional neural network. Background Art

[0002] With the continuous development of educational technology, experimental teaching, as an important part of cultivating students' practical ability and innovative thinking, has attracted more and more attention for its effect and quality. However, traditional teaching behavior analysis methods often rely on manual observation and evaluation, which is not only time-consuming and laborious, but also easily affected by subjective factors, making it difficult to ensure the accuracy and objectivity of the analysis. Therefore, using modern information technology, especially deep learning technology, to automate and intelligently analyze multimodal data in experimental teaching has become an important way to improve teaching quality and efficiency.

[0003] For physics, chemistry and biology experimental teaching, existing technologies mainly rely on single-modal data and are unable to fully capture the multimodal information in the experimental teaching process, resulting in one-sided analysis results; existing methods cannot achieve automatic teaching behavior recognition and are not suitable for large-scale data analysis; and there is a lack of combining teaching behavior with specific information on individual students' performance to implement more personalized teaching.

[0004] Therefore, it is necessary to improve the experimental teaching behavior analysis method in the prior art to solve the above problems. Summary of the invention

[0005] The present invention overcomes the shortcomings of the prior art and provides a method for analyzing experimental teaching behavior based on a deep convolutional neural network, aiming to solve the problem of relying on single-modality data in the prior art and the problem of automated teaching behavior recognition.

[0006] To achieve the above object, the technical solution adopted by the present invention is: a method for analyzing experimental teaching behavior based on a deep convolutional neural network, comprising:

[0007] S1. Collect physics, chemistry and biology experimental teaching data, perform preprocessing, and segment the physics, chemistry and biology experimental teaching process;

[0008] S2, extract multiple features from the physics, chemistry and biology experimental teaching data, perform multimodal feature fusion, and form a comprehensive feature representation;

[0009] S3, using the fused features to build a deep convolutional neural network model;

[0010] S4, train the deep convolutional neural network model, adjust the model parameters, and identify teaching behaviors;

[0011] S5. Analyze the model identification results to evaluate the frequency and duration of different teaching behaviors, as well as students’ responses and engagement;

[0012] S6. Build a description framework, convert the model recognition results and evaluation data into a report, and output it to the teacher.

[0013] 2. According to claim 1, a method for analyzing experimental teaching behavior based on a deep convolutional neural network is characterized in that: in step S1,

[0014] S11. Collect physics, chemistry and biology experimental teaching data from actual teaching scenarios, including: video, audio and text; ensure that these data cover different teaching scenarios and key teaching behaviors;

[0015] S12, preprocessing the physics, chemistry and biology experimental teaching data;

[0016] S13. Label the pre-processed physics, chemistry and biology experiment teaching data, and mark the occurrence time and category of key teaching behaviors.

[0017] In a preferred embodiment of the present invention, in step S12, the collected data is preprocessed:

[0018] Process the video data: extract each frame of the image from the video data; select the frames containing important experimental operations or teaching behaviors as key frames through image analysis technology; perform background blurring on the video frames;

[0019] Process the audio data: Use the noise reduction algorithm to remove the background noise in the audio data; Use the VAD algorithm to detect the speech segments in the audio and remove the silent parts; When text analysis is required, convert the voice data into text data and use speech recognition technology to convert the speech in the audio into text form;

[0020] Process text data: split text data into words or phrases; remove stop words in the text; perform part-of-speech tagging on words in the text; convert words in the text into word vectors through word embedding models, including Word2Vec, GloVe, and BERT.

[0021] In a preferred embodiment of the present invention, in step S2, feature extraction includes:

[0022] Extract spatial features from video frames through deep convolutional neural networks; for video sequences, use information capture techniques in the temporal dimension, including optical flow networks or 3D CNNs, to extract features in the temporal dimension; combine spatial features with temporal dimension features to extract visual features;

[0023] Extract auditory features from audio data through convolutional neural networks or recurrent neural networks; when converting speech to text, extract semantic features from the converted text;

[0024] Semantic features are further extracted from word vectors in text data through convolutional neural networks or recurrent neural networks.

[0025] In a preferred embodiment of the present invention, in step S2, three modal data are provided: video feature V, audio feature A, and text feature T; after each modal feature is extracted, it is converted into a corresponding fixed-length feature vector d V ,d A ,d T ; Perform z-score normalization on the extracted feature vector to obtain V norm ,A norm ,T norm ;

[0026] Define an attention model, assign a weight to each modality, and use a single-layer feedforward network to calculate the attention score: v =W v V norm +b v , e a =W a A norm +b a , e t =W t T norm +b t ; Among them, W v ,W a ,W t is the weight matrix of the corresponding video feature, audio feature and text feature vector, b v ,b a ,b t is the corresponding bias term; the softmax function is used to convert the attention score into a probability distribution:

[0027] According to the attention weight, the modal features are weighted to obtain the comprehensive vector F = α v V norm +α a A norm +α t T norm .

[0028] In a preferred embodiment of the present invention, in step S3, the step of constructing a deep convolutional neural network model includes:

[0029] S31, the input layer receives the fused feature vector F, and sets its dimension to D; the input layer has no specific calculation formula, but only defines the shape of the input;

[0030] S32, several fully connected layers form a hidden layer, and the calculation formula of each fully connected layer is as follows: For the lth layer: Z (l) =W (l) A (l-1) +b (l) ,A (l) =BN(Z (l) ),H (l) =ReLU(A (l) ), Where: Z (l) is the result before the linear activation of the lth layer, W (l) is the weight matrix of the lth layer, b (l) is the bias vector of the lth layer, A (l-1) and A (l) is the activation output of the l-1th and lth layers, BN represents batch normalization, ReLU is the activation function, and Dropout(H, p) represents a random dropout layer with a dropout rate of p;

[0031] S33. Use the Softmax activation function to generate probability distribution and construct the output layer: Z (L) =W (L) A (L-1) +b (L) ,P=Softmax(Z (L) ), where: L is the last layer, Z (L) It is the result before the linear activation of the last layer, and P is the probability distribution of the output.

[0032] In a preferred embodiment of the present invention, in step S4, during the training process, the model parameters are updated using the gradient descent method to minimize the loss function:

[0033] Loss function Γ = -∑ i y i log(p i ), where y i is the true label, p i is the probability predicted by the deep convolutional neural network model;

[0034] Update the weights and bias vectors to modify the deep convolutional neural network model:

[0035] Where l′ is the updated lth layer, η is the learning rate; the updated weights and bias vectors are substituted into the deep convolutional neural network model, and the output of the updated deep convolutional neural network model is Z (L′) =W (L′)A (L′-1) +b (L′) ,P′=Softmax(Z (L′) ), L′ is the updated Lth layer.

[0036] In a preferred embodiment of the present invention, in step S4, the deep convolutional neural network model is trained with the labeled data set, including video, audio and text data, and the data set includes: the teacher's standard operation, the student's experimental attempt, and the correction of incorrect operation.

[0037] In a preferred embodiment of the present invention, in step S5, the key teaching behavior category to which the input feature vector belongs is determined by the maximum value in the probability distribution P′. in, is the predicted teaching behavior category, P c ′ is the element in the probability distribution output by the updated deep convolutional neural network model, corresponding to the probability of each teaching behavior category;

[0038] The prediction results are counted to calculate the frequency of each key teaching behavior category: Among them, K is the set of all time points, is the teaching behavior category at time point k, is the indicator function, when If it is equal to category i, it returns 1, otherwise it returns 0;

[0039] Use the model prediction results and timestamps to mark the start and end time of each teaching behavior, with a duration of k j -k j-1 , j is the jth time period.

[0040] The present invention solves the defects existing in the background technology and has the following beneficial effects:

[0041] (1) The present invention combines deep learning technology to conduct a comprehensive analysis of multimodal data such as video, audio and text collected in experimental teaching. It uses deep convolutional neural networks and recurrent neural networks to extract key features from different modalities, integrates information from each modality through feature fusion technology, and enhances the model's understanding of teaching behavior. The constructed model is trained and optimized to automatically identify and classify key behaviors in teaching. Through empirical analysis and a unified description framework, the recognition results are converted into in-depth insights into teaching behaviors, thereby supporting teaching improvements and strategy adjustments.

[0042] (2) The present invention extracts multiple features from video, audio and text data, and performs multimodal feature fusion to form a comprehensive feature representation. It extracts spatial features and temporal dimension features from video frames, auditory features from audio data, and semantic features from text data. The present invention effectively integrates complementary information from different modalities and improves the comprehensive expression ability of features. Compared with the prior art, the combination of this step makes the teaching behavior analysis more comprehensive and accurate.

[0043] (3) The present invention uses the fused features to construct a deep convolutional neural network model, and trains it through annotated data sets, adjusting model parameters to identify teaching behaviors. The model gradually extracts high-level features through structures such as fully connected layers, batch normalization, activation functions, and random inactivation layers, and uses the Softmax activation function to generate probability distribution. Compared with the prior art, the deep convolutional neural network model used in the present invention has stronger feature extraction capabilities and classification performance, and can more accurately identify key teaching behavior categories.

[0044] (4) The present invention analyzes the model recognition results to evaluate the frequency, duration, and student response and participation of different teaching behaviors. Combining the probability distribution and timestamp information predicted by the model, the present invention can accurately mark the start and end time of each teaching behavior and calculate relevant statistical indicators. Compared with the prior art, the present invention provides a more detailed and comprehensive teaching behavior analysis report, which helps teachers to have a deeper understanding of the teaching process and students' learning situation. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art are briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative work.

[0046] Figure 1 is a flow chart of a preferred embodiment of the present invention;

[0047] Figure 2 It is a step diagram of constructing a deep convolutional neural network model according to a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0048] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0049] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein. Therefore, the protection scope of the present invention is not limited to the specific embodiments disclosed below.

[0050] like Figure 1 As shown, a method for analyzing experimental teaching behavior based on a deep convolutional neural network includes the following steps:

[0051] S1. Collect physics, chemistry and biology experimental teaching data, perform preprocessing, and segment the physics, chemistry and biology experimental teaching process;

[0052] S2, extract multiple features from the physics, chemistry and biology experimental teaching data, perform multimodal feature fusion, and form a comprehensive feature representation;

[0053] S3, using the fused features to build a deep convolutional neural network model;

[0054] S4, train the deep convolutional neural network model, adjust the model parameters, and identify teaching behaviors;

[0055] S5. Analyze the model identification results to evaluate the frequency and duration of different teaching behaviors, as well as students’ responses and engagement;

[0056] S6. Build a description framework to convert the model recognition results and evaluation data into reports and output them to teachers;

[0057] Specifically, in step S1,

[0058] S11. Collect physics, chemistry and biology experimental teaching data from actual teaching scenarios, including: video, audio and text; ensure that these data cover different teaching scenarios and key teaching behaviors;

[0059] S12, preprocessing the physics, chemistry and biology experimental teaching data;

[0060] S13, annotating the pre-processed physics, chemistry and biology experiment teaching data, marking the occurrence time and category of key teaching behaviors;

[0061] Specifically, in step S11, the video data includes: the teacher's demonstration and the students' experimental operation; the audio data includes: the teacher's explanation and guidance and the students' questions and discussions; the text data includes: experimental reports, handouts, and observation records;

[0062] The physics, chemistry and biology experimental teaching data covers many aspects of physics, chemistry and biology experimental teaching, and can comprehensively reflect the actual situation of experimental teaching. By analyzing these data, we can have an in-depth understanding of teachers' teaching behaviors and students' learning behaviors, and provide strong support for teaching improvement.

[0063] Specifically, in step S12, the collected data is preprocessed:

[0064] (1) Processing video data: extract each frame of the video data for subsequent image analysis; select frames containing important experimental operations or teaching behaviors as key frames through image analysis technology; blur the background of the video frames to highlight the experimental operation area and reduce background interference; process the video data through frame extraction, key frame selection, and background blurring.

[0065] (2) Processing audio data: Use noise reduction algorithms to remove background noise from audio data and improve speech clarity; use VAD algorithms to detect speech segments in the audio and remove silent parts to reduce the amount of computation required for subsequent processing; when text analysis is required, convert speech data into text data and use speech recognition technology to convert speech in the audio into text for subsequent text mining and analysis; process audio data through noise reduction, voice activity detection, and optional speech-to-text conversion.

[0066] (3) Processing text data: Segment text data into words or phrases for subsequent text analysis; remove stop words in the text to reduce the complexity of subsequent processing; tag the words in the text with parts of speech for subsequent semantic analysis; convert the words in the text into word vectors through word embedding models, including Word2Vec, GloVe, and BERT, for subsequent machine learning or deep learning processing; process text data through word segmentation, stop word removal, part-of-speech tagging, and word embedding conversion.

[0067] Through the above preprocessing steps, the original video, audio, and text data can be converted into a format that is more suitable for machine learning or deep learning model processing, thereby improving the accuracy and efficiency of subsequent teaching behavior analysis.

[0068] Specifically, in step S13, the categories of key teaching behaviors include:

[0069] Teacher demonstration: the demonstration behavior performed by the teacher during the experiment, such as showing the experimental equipment and operating the experimental steps;

[0070] Student experimental operations: the operations performed by students during the experiment, such as using experimental equipment, carrying out experimental steps, etc.

[0071] Explain key concepts: The behavior of teachers explaining key concepts in the experiment;

[0072] Guidance and correction: teachers provide guidance and correct wrong behaviors during students’ experimental operations;

[0073] Student questions and discussions: students’ behavior of asking questions and engaging in discussions during the experiment;

[0074] Presentation and interpretation of experimental results: The teacher's behavior of presenting and explaining the experimental results.

[0075] By labeling these key teaching behavior categories, the model will be able to learn important behavioral patterns in the experimental teaching process and provide strong support for subsequent teaching behavior analysis.

[0076] Specifically, in step S13, the pre-processed video and audio data are used to segment the physics, chemistry and biology experiment teaching process, realize automatic segmentation of classroom events, and identify the start and end of teaching behavior.

[0077] Further analyze the key frames in the preprocessed video data, and identify the category of teaching behavior in the key frames through image recognition technology; timestamp the start and end frames of the identified teaching behavior; sort the teaching behaviors in the key frames in chronological order, construct a teaching behavior sequence, analyze the patterns in the teaching behavior sequence, and identify the start and end of the teaching behavior, thereby realizing the automatic segmentation of classroom events;

[0078] The speech segments in the preprocessed audio data are further analyzed. Through speech recognition technology, the speech segments are converted into text, and the content related to the teaching behavior is extracted. The audio segments related to the identified teaching behavior are timestamped, and the corresponding teaching behavior is matched with the video data. Combined with the timestamp of the video data, classroom events can be accurately segmented, thereby identifying the transition points between different teaching behaviors.

[0079] Specifically, in step S2, feature extraction includes:

[0080] Using deep convolutional neural networks to extract spatial features from video frames, including character posture, facial expressions, and gestures, plays an important role in understanding teaching behavior;

[0081] For video sequences, use information capture technology in the time dimension, including optical flow network or 3D CNN, to extract features in the time dimension, including the change and speed of the action. These features can reflect the dynamic changes of teaching behavior.

[0082] By combining the above-mentioned spatial features and temporal dimension features, more specific visual features can be extracted, including: experimental equipment operation and chemical change process. These features are directly related to the specific operations and phenomena in the physics, chemistry and biology experimental teaching process.

[0083] Using convolutional neural networks or recurrent neural networks to extract auditory features from audio data, including speaking speed, intonation, and emotional state, can reflect the changes in the teacher's tone and emotions during the explanation process; when speech-to-text is converted, semantic features are extracted from the converted text to further enrich the understanding of teaching behavior.

[0084] Use convolutional neural networks or recurrent neural networks to further extract semantic features from word vectors in text data, including key concepts and descriptions of operation steps.

[0085] Deep convolutional neural networks include: ResNet, VGG, 3D CNN; Deep convolutional neural networks are a type of neural network that is particularly suitable for processing grid-structured data, i.e., video frame images. Through structures such as convolutional layers, pooling layers, and fully connected layers, useful features can be effectively extracted from the original image for various image recognition, classification, and other tasks; optical flow networks can estimate the motion between two frames, while 3D CNN can extract features in the spatiotemporal domain.

[0086] Through the above feature extraction process, rich multimodal features can be extracted from the physics, chemistry and biology experimental teaching data. These features cover multiple aspects such as vision, hearing and semantics, and can fully reflect various teaching behaviors in the experimental teaching process.

[0087] Multimodal feature fusion is the process of integrating different features from video, audio and text. By combining the complementary information of different modal data, the accuracy and comprehensiveness of teaching behavior recognition can be improved.

[0088] Specifically, in step S2, there are three modal data: video feature V, audio feature A, and text feature T. After each modal feature is extracted, it is converted into a corresponding fixed-length feature vector d V ,d A ,d T ; Perform z-score normalization on the extracted feature vector to obtain V norm ,A norm ,T norm ;

[0089] Define an attention model, assign a weight to each modality, and use a single-layer feedforward network to calculate the attention score: v =W v V norm +b v , e a =W a A norm +b a , e t =W t T norm +b t ; Among them, W v ,W a ,W t is the weight matrix of the corresponding video feature, audio feature and text feature vector, b v ,b a ,b t is the corresponding bias term; the softmax function is used to convert the attention score into a probability distribution:

[0090] According to the attention weight, the modal features are weighted to obtain the comprehensive vector F = α v V norm +α a A norm +α t T norm .

[0091] like Figure 2 As shown, in step S3, the steps of constructing a deep convolutional neural network model include:

[0092] S31, the input layer receives the fused feature vector F, and sets its dimension to D; the input layer has no specific calculation formula, but only defines the shape of the input;

[0093] S32, several fully connected layers form a hidden layer, and the calculation formula of each fully connected layer is as follows: For the lth layer: Z (l) =W (l) A (l-1) +b (l) ,A (l) =BN(Z (l) ),H (l) =ReLU(A (l) ), Where: Z (l) is the result before the linear activation of the lth layer, W (l) is the weight matrix of the lth layer, b (l) is the bias vector of the lth layer, A (l-1) and A(l) is the activation output of the l-1th and lth layers, BN stands for batch normalization, ReLU is the activation function, and Dropout(H, p) stands for a random dropout layer with a dropout rate of p; the random dropout layer randomly drops the output of some neurons to reduce overfitting;

[0094] S33. Use the Softmax activation function to generate probability distribution and construct the output layer: Z (L) =W (L) A (L-1) +b (L) ,P=Softmax(Z (L) ), where: L is the last layer, Z (L) It is the result before the linear activation of the last layer, and P is the probability distribution of the output.

[0095] Specifically, in step S4, the labeled data set, including video, audio and text data, is used to train the deep convolutional neural network model, and the data set includes: the teacher's standard operation, the student's experimental attempt, and the correction of the wrong operation;

[0096] During the training process, the gradient descent method is used to update the model parameters and minimize the loss function:

[0097] Loss function Γ = -∑ i y i log(p i ), where y i is the true label, p i is the probability predicted by the deep convolutional neural network model;

[0098] Update the weights and bias vectors to modify the deep convolutional neural network model: Where l′ is the updated lth layer, η is the learning rate; the updated weights and bias vectors are substituted into the deep convolutional neural network model, and the output of the updated deep convolutional neural network model is Z (L′) =W (L′) A (L′-1) +b (L′) ,P′=Softmax(Z (L′) ), L′ is the updated Lth layer.

[0099] Use the trained model to automatically identify key teaching behavior categories.

[0100] Specifically, in step S5, the key teaching behavior category to which the input feature vector belongs is determined by the maximum value in the probability distribution P′. in, is the predicted teaching behavior category, P c′ is the element in the probability distribution output by the updated deep convolutional neural network model, corresponding to the probability of each teaching behavior category;

[0101] The prediction results are counted to calculate the frequency of each key teaching behavior category: Among them, K is the set of all time points, is the teaching behavior category at time point k, is the indicator function, when If it is equal to category i, it returns 1, otherwise it returns 0;

[0102] Use the model prediction results and timestamps to mark the start and end time of each teaching behavior, with a duration of k j -k j-1 , j is the jth time period;

[0103] Calculate student participation as the total number of student questions and discussions divided by the total lab time;

[0104] Calculate the students' enthusiasm, that is, the number of active questions and discussions divided by the total number of questions and discussions; this can be calculated by combining the teaching behaviors determined by the above-mentioned deep convolutional neural network model with the frequency of occurrence of the teaching behavior categories.

[0105] Through the above steps, the updated deep convolutional neural network model is used to predict new experimental teaching data and identify the categories of key teaching behaviors, which can help teachers understand the teaching effect in a timely manner and make corresponding adjustments.

[0106] The above is based on the ideal embodiment of the present invention. Through the above description, relevant personnel can make various changes and modifications without departing from the technical concept of the present invention. The technical scope of the present invention is not limited to the content in the specification, and the technical scope must be determined according to the scope of the claims.

Claims

1. A method for analyzing experimental teaching behavior based on deep convolutional neural network, characterized in that: Includes steps: S1. Collect physics, chemistry and biology experimental teaching data, perform preprocessing, and segment the physics, chemistry and biology experimental teaching process; S2, extract multiple features from the physics, chemistry and biology experimental teaching data, perform multimodal feature fusion, and form a comprehensive feature representation; S3, using the fused features to build a deep convolutional neural network model; S4, train the deep convolutional neural network model, adjust the model parameters, and identify teaching behaviors; S5. Analyze the model identification results to evaluate the frequency and duration of different teaching behaviors, as well as students’ responses and engagement; S6. Build a description framework, convert the model recognition results and evaluation data into a report, and output it to the teacher.

2. According to claim 1, a method for analyzing experimental teaching behavior based on a deep convolutional neural network is characterized in that: In step S1, S11. Collect physics, chemistry and biology experimental teaching data from actual teaching scenarios, including: video, audio and text; ensure that these data cover different teaching scenarios and key teaching behaviors; S12, preprocessing the physics, chemistry and biology experimental teaching data; S13. Label the pre-processed physics, chemistry and biology experiment teaching data, and mark the occurrence time and category of key teaching behaviors.

3. According to claim 2, a method for analyzing experimental teaching behavior based on a deep convolutional neural network is characterized in that: In step S12, the collected data is preprocessed: Process the video data: extract each frame of the image from the video data; select the frames containing important experimental operations or teaching behaviors as key frames through image analysis technology; perform background blurring on the video frames; Process the audio data: use the noise reduction algorithm to remove the background noise in the audio data; use the VAD algorithm to detect the speech segments in the audio and remove the silent parts; When text analysis is required, voice data is converted into text data and speech recognition technology is used to convert the speech in the audio into text form; Process text data: split text data into words or phrases; remove stop words in the text; Perform part-of-speech tagging on words in the text; The words in the text are converted into word vectors through word embedding models, including Word2Vec, GloVe, and BERT.

4. The method for analyzing experimental teaching behavior based on a deep convolutional neural network according to claim 3 is characterized in that: In step S2, feature extraction includes: Extract spatial features from video frames through deep convolutional neural networks; for video sequences, use information capture techniques in the temporal dimension, including optical flow networks or 3D CNNs, to extract features in the temporal dimension; combine spatial features with temporal dimension features to extract visual features; Extract auditory features from audio data through convolutional neural networks or recurrent neural networks; when converting speech to text, extract semantic features from the converted text; Semantic features are further extracted from word vectors in text data through convolutional neural networks or recurrent neural networks.

5. The method for analyzing experimental teaching behavior based on a deep convolutional neural network according to claim 1, characterized in that: In step S2, three modal data are assumed: video feature V, audio feature A, and text feature T. After each modal feature is extracted, it is converted into a corresponding fixed-length feature vector d V ,d A ,d T ; Perform z-score normalization on the extracted feature vector to obtain V norm ,A norm ,T norm ; Define an attention model, assign a weight to each modality, and use a single-layer feedforward network to calculate the attention score: v =W v V norm +b v , e a =W a A norm +b a , e t =W t T norm +b t ; Among them, W v ,W a ,W t is the weight matrix of the corresponding video feature, audio feature and text feature vector, b v ,b a ,b t is the corresponding bias term; the softmax function is used to convert the attention score into a probability distribution: According to the attention weight, the modal features are weighted to obtain the comprehensive vector F = α v V norm +α a A norm +α t T norm .

6. The method for analyzing experimental teaching behavior based on a deep convolutional neural network according to claim 5, characterized in that: In step S3, the steps of constructing a deep convolutional neural network model include: S31, the input layer receives the fused feature vector F, and sets its dimension to D; the input layer has no specific calculation formula, but only defines the shape of the input; S32, several fully connected layers form a hidden layer, and the calculation formula of each fully connected layer is as follows: For the lth layer: Z (l) =W (l) A (l-1) +b (l) ,A (l) =BN(Z (l) ),H (l) =ReLU(A (l) ), Where: Z (l) is the result before the linear activation of the lth layer, W (l) is the weight matrix of the lth layer, b (l) is the bias vector of the lth layer, A (l-1) and A (l) is the activation output of the l-1th and lth layers, BN represents batch normalization, ReLU is the activation function, and Dropout(H, p) represents a random dropout layer with a dropout rate of p; S33. Use the Softmax activation function to generate probability distribution and construct the output layer: Z (L) =W (L) A (L-1) +b (L) ,P=Softmax(Z (L) ), where: L is the last layer, Z (L) It is the result before the linear activation of the last layer, and P is the probability distribution of the output.

7. The method for analyzing experimental teaching behavior based on a deep convolutional neural network according to claim 6 is characterized in that: In step S4, during the training process, the model parameters are updated using the gradient descent method to minimize the loss function: Loss function Γ = -∑ i y i log(p i ), where y i is the true label, p i is the probability predicted by the deep convolutional neural network model; Update the weights and bias vectors to modify the deep convolutional neural network model: Where l′ is the updated lth layer, η is the learning rate; the updated weights and bias vectors are substituted into the deep convolutional neural network model, and the output of the updated deep convolutional neural network model is Z (L′) =W (L′) A (L′-1) +b (L′) ,P′=Softmax(Z (L′) ), L′ is the updated Lth layer.

8. The method for analyzing experimental teaching behavior based on a deep convolutional neural network according to claim 1, characterized in that: In step S4, the deep convolutional neural network model is trained with the labeled data set, including video, audio and text data. The data set includes: the teacher's standard operation, the student's experimental attempt, and the correction of incorrect operation.

9. The method for analyzing experimental teaching behavior based on a deep convolutional neural network according to claim 1, characterized in that: In step S5, the key teaching behavior category to which the input feature vector belongs is determined by the maximum value in the probability distribution P′. in, is the predicted teaching behavior category, P c ′ is the element in the probability distribution output by the updated deep convolutional neural network model, corresponding to the probability of each teaching behavior category; The prediction results are counted to calculate the frequency of each key teaching behavior category: Among them, K is the set of all time points, is the teaching behavior category at time point k, is the indicator function, when If it is equal to category i, it returns 1, otherwise it returns 0; Use the model prediction results and timestamps to mark the start and end time of each teaching behavior, with a duration of k j -k j-1 , j is the jth time period.

Images