Multi-view middle school experiment step detection method and system based on differential feature enhancement

Abstract

The present invention discloses a multi-view middle school experimental step detection method and system based on differential feature enhancement. First, the video is preprocessed to extract the original and differential video frame segments, and the visual timing features and differential motion features are obtained through a feature extractor. The input encoder learns the spatiotemporal dependency relationship, and the cross-attention mechanism is used to fuse the features. Finally, the fused multi-view visual timing features are output through a decoder to output the classification results. This patent can more accurately capture the changes and action characteristics of the experimental steps, improve the accuracy and robustness of the classification model, and can effectively solve the problems that manual detection of middle school experimental steps consumes a lot of human resources, the existing methods do not fully explore the timing relationship, and the efficiency is low. The detection accuracy is improved, so it has a high application value and application prospect.

Description

Technical Field

[0001] The present invention belongs to the technical field of deep learning, and mainly relates to a multi-perspective middle school experiment step detection method and system based on differential feature enhancement. Background Art

[0002] Experimental teaching is one of the important teaching contents stipulated in the curriculum plan and curriculum standards, and is an important way to cultivate students' scientific literacy and innovation ability. Through experimental teaching, students can personally participate in scientific practice, explore scientific laws, and cultivate experimental design and operation capabilities, thereby deepening their understanding and application of course knowledge. Therefore, experimental teaching has an irreplaceable position and role in primary and secondary education, and is of great significance to the comprehensive development and quality improvement of students.

[0003] However, the current experimental evaluation model faces multiple challenges. First, with limited teaching time and a large number of students, traditional evaluation methods may not be able to timely and comprehensively evaluate the experimental operation ability of each student. Secondly, traditional evaluation methods may be subjective and inconsistent, which leads to doubts about the fairness of the evaluation results. Artificial intelligence technology has the ability to analyze and evaluate student experiments in real time, can automatically identify and evaluate students' experimental operations, and provide objective and accurate evaluation results. In addition, artificial intelligence technology can also provide support for the establishment of a comprehensive and standardized evaluation framework, thereby improving the consistency and fairness of the evaluation. Therefore, using artificial intelligence to evaluate middle school experiments can more comprehensively and efficiently evaluate students' experimental operation capabilities, which will help improve the quality and effectiveness of science education.

[0004] At present, the mainstream research direction focuses on using action recognition algorithms to learn the temporal relationship in videos to achieve accurate detection of action categories. However, these methods have some shortcomings when processing video data such as middle school experiments: First, since middle school experimental videos are usually long and contain rich semantic information, traditional action recognition algorithms often cannot fully capture the key information, resulting in insufficient understanding of experimental steps and actions, which may lead to overfitting of scene-related clues, thereby affecting the generalization ability and detection accuracy of the model; secondly, the current action recognition method is inefficient and not suitable for real-time detection, which limits its application in teaching practice and experimental operation. In practical applications, especially in teaching scenarios, real-time is crucial, and current methods are difficult to meet this requirement, limiting their application effect in actual teaching. Summary of the invention

[0005] The present invention is aimed at the problems of inaccurate and poor real-time detection of experimental steps in the prior art, and provides a multi-view middle school experimental step detection method and system based on differential feature enhancement. First, the video is preprocessed to extract the original and differential video frame fragments, and the visual timing features and differential motion features are obtained through the feature extractor. The encoder is input to learn the spatiotemporal dependency relationship, and the cross-attention mechanism is used to fuse the features. Finally, the fused multi-view visual timing features are output through the decoder to output the classification results. This patent can more accurately capture the changes and action characteristics of the experimental steps, improve the accuracy and robustness of the classification model, and can effectively solve the problems that manual detection of middle school experimental steps consumes a lot of human resources, the existing methods do not fully explore the timing relationship, and the efficiency is low. It improves the detection accuracy, and therefore has a high application value and application prospects.

[0006] In order to achieve the above object, the technical solution adopted by the present invention is: a multi-view middle school experiment step detection method based on differential feature enhancement, comprising the following steps:

[0007] S1: Input a middle school experimental video with multiple perspectives, front view and top view, pre-process the video frames of the two perspectives, obtain a group of 16 video frame segments with each frame in each perspective as the anchor point center, and obtain differential video frame segments by frame difference;

[0008] S2: Use the visual large model VideoMAE to extract features from the video frame segments of the two perspectives, and obtain the visual timing features and differential motion features at the video frame level of different perspectives;

[0009] S3: Input the visual temporal features and differential motion features of the two-view video frame levels extracted in step S2 into the encoder, and use the multi-head self-attention mechanism to learn the spatiotemporal dependency;

[0010] S4: The encoded visual temporal features and differential motion features of the two perspectives are input into the decoder for fusion;

[0011] S5: Use the multi-view features of the visual timing features fused with the differential motion features to perform classification and output the classification results of the video frames.

[0012] As an improvement of the present invention, the step S1 specifically includes the following steps:

[0013] S11, video frame preprocessing: converting the middle school experiment videos of the front view and the top view into a video frame sequence, setting the anchor point of the video segment as the key frame in the segment, sampling the video frame, and obtaining a group of 16 video frame segments with the anchor point as the center; the size of the video frame sequence is 224*224;

[0014] S12, video frame difference: In the original video frame segment, the first frame x0 is not subjected to difference operation, and each of the other frames x i Subtract the first frame x0 to get the differential video frame, and get the differential video frame x′ i ;

[0015] x′ i =x i -x0;

[0016] S13, data standardization processing: standardizing the original video frames and differential frames, converting the data into a distribution with a mean of 0 and a standard deviation of 1, and converting the original video frame segments and differential video frame segments into a tensor format so as to facilitate input into a deep learning model for training;

[0017]

[0018]

[0019]

[0020] Among them, x i represents the video frame, μ and σ represent the mean and standard deviation of all video frames, z i is the normalized video frame.

[0021] As another improvement of the present invention, in step S2, a visual large model VideoMAE mask self-encoding method is adopted, and the visual temporal features and differential motion feature maps at the video frame level of two perspectives output by the feature extractor enter the encoder:

[0022]

[0023] Among them, for time T, we feed the video frames of the previous t time periods into the feature extractor u to obtain the feature f of the current time period T .

[0024] As another improvement of the present invention, the encoder of step S3 includes at least a spatial attention module and a temporal attention module;

[0025] The spatial attention module is composed of a multi-head self-attention layer, including attention projection, attention weight calculation, weighted summation and multi-head mechanism; multi-head attention is to use multiple query vectors Q i , in parallel from the input information K i Select multiple sets of information from the dataset, calculate the attention scores, apply scaled dot products and softmax to normalize the range of the attention scores, and finally add them to the value V iThe weighted sum is used to obtain the output. The weight of each spatial position is modeled, and the multi-head self-attention mechanism is used to capture the dependencies between different regions in the same frame.

[0026]

[0027] The temporal attention module: consists of a multi-head self-attention layer, which captures the temporal relationship between different frames in the video sequence and emphasizes the key moments or periods related to the task;

[0028]

[0029] Among them, F T represents the input video sequence features, Represents the new features obtained after applying multi-head self-attention.

[0030] As another improvement of the present invention, the step S4 specifically includes the following steps:

[0031] S41: Use the cross attention mechanism to query useful context information from the two viewpoint differential features respectively;

[0032] S42: performing weighted summation on the fused visual temporal features and the differential motion features;

[0033] S43: Get the final visual temporal multi-view feature map of fused differential motion features:

[0034]

[0035] Among them, the output characteristics of the encoder As a query, use the multi-head attention mechanism to select the view difference feature S c Find useful information to enrich itself and obtain the final fusion features

[0036] In order to achieve the above-mentioned purpose, the present invention also adopts the technical solution of: a multi-perspective middle school experiment step detection system based on differential feature enhancement, including a computer program, which implements the steps of any of the above-mentioned methods when executed by a processor.

[0037] Compared with the prior art, the present invention has the following beneficial effects:

[0038] (1) The present invention adopts a differential feature enhancement method to more accurately capture the changes in experimental steps and action characteristics, which helps to improve the accuracy and robustness of the classification model;

[0039] (2) The method of the present invention can more effectively learn the spatiotemporal dependencies in the video, which helps to capture important information of the experimental steps and improve the detection accuracy;

[0040] (3) The method of the present invention can effectively solve the problems of manual detection, such as the experimental steps consuming a lot of human resources, the existing methods not fully exploring the time relationship, and low efficiency, thereby improving the detection accuracy. Therefore, it has high application value and application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 is a flow chart of the steps of the method of the present invention;

[0042] Figure 2 is a flow chart of the steps of video frame preprocessing in step S1 of the method of the present invention;

[0043] Figure 3 is a flow chart of an encoder and a decoder of the present invention;

[0044] Figure 4 It is a diagram of the test results of the middle school experimental steps of the present invention. DETAILED DESCRIPTION

[0045] The present invention will be further explained below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention.

[0046] Example 1

[0047] like Figure 1 As shown, the multi-view middle school experimental step detection method based on differential feature enhancement includes the following steps:

[0048] Step S1: input a multi-view middle school experimental video, pre-process the video frames of the two views respectively, take each frame in each view as the anchor point center to obtain a group of 16 video frame segments, and obtain differential video frame segments by frame difference; Figure 2 As shown, this step specifically includes the following steps:

[0049] S11, video frame preprocessing: First, read the video file or read the network video stream to convert the video into a video frame sequence; then scale the size of each frame to 224*224 to ensure that the input of the model has a consistent size; set the anchor point of the video segment as the key frame in the segment; sample the video frame to obtain a group of 16 video frame segments with the anchor point as the center;

[0050] S12, video frame difference: Frame difference extracts motion information in the video by calculating the difference between frames, thereby helping to identify actions; in the original video frame segment, the first frame x0 is not subjected to difference operation, and each of the other frames x i Subtract the first frame x0 to get the differential video frame, and get the differential video frame x′ i ;

[0051] x′ i =x i -x0:

[0052] S13, data standardization processing: the original video frames and differential frames are standardized. In RGB images, there are usually three channels (red, green, and blue). The mean of each channel is set to 0.485, 0.456, and 0.406, and the standard deviation is set to 0.229, 0.224, and 0.225. Through standardization, the data is converted into a distribution with a mean of 0 and a standard deviation of 1, so that the data is easier to converge during training, thereby improving the training efficiency and generalization ability of the model; finally, the original video frame fragments and the differential video frame fragments are converted into tensor format for input into the deep learning model for training.

[0053]

[0054]

[0055]

[0056] Among them, x i represents the video frame, μ and σ represent the mean and standard deviation of all video frames, z i is the normalized video frame.

[0057] Step S2: Use the visual large model VideoMAE to extract features from the video frame segments of the two perspectives to obtain visual timing features and differential motion features at the video frame level of different perspectives;

[0058] S21: the original video frame segments and the differential video frame segments of the two perspectives enter the feature extractor respectively;

[0059] S22: The feature extractor adopts the visual large model VideoMAE mask self-encoding method to capture the visual timing information at the video frame level in different perspectives of the clip;

[0060] S23: The visual temporal features and differential motion feature maps at the video frame level of the two perspectives output from the feature extractor enter the encoder:

[0061]

[0062] Among them, for time T, we feed the video frames of the previous t time periods into the feature extractor u to obtain the feature f of the current time period T .

[0063] Step S3: Input the extracted visual temporal features and differential motion features at the video frame level of the two perspectives into the encoder to learn the spatiotemporal dependency using a multi-head self-attention mechanism;

[0064] like Figure 3 As shown, the encoder module includes two attention mechanism modules: first, a spatial attention module, and then a temporal attention module; the spatial attention module is used to capture the importance of different regions in the video frame so as to focus on the regions related to the task; by modeling the weight of each spatial position, it can selectively focus on a specific region in the video frame; using a multi-head self-attention mechanism, the dependency between different regions in the same frame is captured; the temporal attention module is used to capture the temporal relationship between different frames in the video sequence, and emphasize the key moments or time periods related to the task, so as to focus on important time points more effectively;

[0065] The spatial attention module is used to capture the importance of different regions in the video frame. It is mainly composed of a multi-head self-attention layer, including attention projection, attention weight calculation, weighted summation and multi-head mechanism. Multi-head attention is to use multiple query vectors Q i , in parallel from the input information K i Select multiple sets of information from the dataset, calculate the attention scores, apply scaled dot products and softmax to normalize the range of the attention scores, and finally add them to the value V i The weighted sum is used to get the output. In addition, there are Dropout layers, nonlinear activation functions, and fully connected layers;

[0066] The temporal attention module is used to capture the temporal relationship between different frames in a video sequence and is mainly composed of a multi-head self-attention layer.

[0067]

[0068]

[0069] Among them, F T represents the input video sequence features, Represents the new features obtained after applying multi-head self-attention.

[0070] Step S4: The encoded visual temporal features and differential motion features are input into the decoder for fusion;

[0071] S41: Query useful contextual information from differential features using cross-attention mechanism;

[0072] S42: performing weighted summation on the fused visual temporal features and the differential motion features;

[0073] S43: Obtain the final visual temporal multi-view feature map that integrates the differential motion features.

[0074]

[0075] Among them, the output characteristics of the encoder As a query, use the multi-head attention mechanism to select the view difference feature S c Find useful information to enrich itself and obtain the final fusion features

[0076] The decoder is used to fuse the visual timing features with the differential motion features, enhance the visual timing features with the differential motion features, and finally map the features to each category through a fully connected layer, obtain the probability of each category after normalization, and output the category with the highest probability as the classification result of the video frame.

[0077] Step S5: Use the multi-view features of the visual timing features fused with the differential motion features to perform classification and output the classification results of the video frames.

[0078] The Adam method is used for stochastic gradient descent, the batch size is set to 128, the learning rate is set to 0.0001, and the weight decay is set to 0.0005. Other settings are completely based on the settings mentioned in the literature of related action recognition methods to train the entire network. In order to speed up the training process, the feature extraction speed is set to 1fps, and the end-to-end classification network is fully trained.

[0079] The method of the present invention was applied to a carbon dioxide experimental data set for experiment. The carbon dioxide experimental data set contains 222 groups of videos and 11 steps of producing carbon dioxide from limestone. The experiment is relatively complex, and each action lasts for a short time, which challenges the accuracy and sensitivity of the algorithm. In order to measure the gap between the segmentation result and the actual result, the present invention uses the average precision (mAP) per frame to evaluate the effectiveness of online action detection. mAP is a commonly used indicator, and the average precision (AP) of each action category needs to be averaged. The higher the mAP result, the higher the classification accuracy.

[0080] The visual comparison results of the middle school experimental steps are as follows Figure 4 As shown in the figure, it can be seen that the detection result obtained by the present invention is basically consistent with the real result, except that there is a deviation in the classification of individual frames, and it can be regarded as a reliable result overall.

[0081] The comparison of the results of the detection using this method is shown in Table 1.

[0082] Table 1

[0083]

[0084] It can be seen from the table that the detection accuracy of the present invention on the carbon dioxide data set is better than that of other methods, and the efficiency is also improved, which is more suitable for practical applications.

[0085] In the description of this specification, the description with reference to the terms "one embodiment", "example", "specific example", etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0086] It should be noted that the above content only illustrates the technical idea of ​​the present invention and cannot be used to limit the protection scope of the present invention. For ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications all fall within the protection scope of the claims of the present invention.

Claims

1. A multi-view middle school experimental step detection method based on differential feature enhancement, characterized in that: The steps include: S1: Input a middle school experimental video with multiple perspectives, front view and top view, pre-process the video frames of the two perspectives, obtain a group of 16 video frame segments with each frame in each perspective as the anchor point center, and obtain differential video frame segments by frame difference; S2: Use the visual large model VideoMAE to extract features from the video frame segments of the two perspectives, and obtain the visual timing features and differential motion features at the video frame level of different perspectives; use the visual large model VideoMAE mask self-encoding method, and the visual timing features and differential motion feature maps at the video frame level of the two perspectives output by the feature extractor enter the encoder: Among them, u is the feature extractor, T is a selected moment, and t is the number of frames offset forward from the selected moment T; S3: Input the visual temporal features and differential motion features of the two-view video frame levels extracted in step S2 into an encoder, and learn the spatiotemporal dependency using a multi-head self-attention mechanism; the encoder includes at least a spatial attention module and a temporal attention module; The spatial attention module is composed of a multi-head self-attention layer, including attention projection, attention weight calculation, weighted summation and multi-head mechanism; multi-head attention is to use multiple query vectors Q i , in parallel from the input information K i Select multiple sets of information from the dataset, calculate the attention scores, apply scaled dot products and softmax to normalize the range of the attention scores, and finally add them to the value V i The output is obtained by weighted summation; the weight of each spatial position is modeled, and the multi-head self-attention mechanism is used to capture the dependencies between different regions in the same frame; in, For input information, d k To indicate input information Dimensions; The temporal attention module: consists of a multi-head self-attention layer, which captures the temporal relationship between different frames in the video sequence and emphasizes the key moments or periods related to the task; Among them, F T represents the input video sequence features, Represents the new features obtained after applying multi-head self-attention; S4: The encoded visual temporal features and differential motion features of the two perspectives are input into the decoder for fusion; S41: Use the cross attention mechanism to query useful context information from the two viewpoint differential features respectively; S42: performing weighted summation on the fused visual temporal features and the differential motion features; S43: Get the final visual temporal multi-view feature map of fused differential motion features: Among them, the output characteristics of the encoder As a query, use the multi-head attention mechanism to select the view difference feature S c Find useful information and get the final fusion features S5: Use the multi-view features of the visual timing features fused with the differential motion features to perform classification and output the classification results of the video frames.

2. The multi-view middle school experiment step detection method based on differential feature enhancement as claimed in claim 1 is characterized in that: The step S1 specifically includes the following steps: S11, video frame preprocessing: converting the middle school experiment videos of the front view and the top view into a video frame sequence, setting the anchor point of the video segment as the key frame in the segment, sampling the video frame, and obtaining a group of 16 video frame segments with the anchor point as the center; the size of the video frame sequence is 224*224; S12, video frame difference: In the original video frame segment, the first frame x0 is not subjected to difference operation, and each of the other frames x i Subtract the first frame x0 to get the differential video frame, and get the differential video frame x′ i ; x′ i =x i -x0; S13, data standardization processing: standardizing the original video frames and differential frames, converting the data into a distribution with a mean of 0 and a standard deviation of 1, and converting the original video frame segments and differential video frame segments into a tensor format so as to facilitate input into a deep learning model for training; Among them, x i represents the video frame, μ and σ represent the mean and standard deviation of all video frames, z i is the normalized video frame, and n represents the number of video frames.

3. A multi-view secondary school experimental procedure detection system based on differential feature enhancement, including a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to claim 1 or 2 are implemented.

Images