Multimodal middle school experiment step detection method and system based on motion target semantic enhancement

Abstract

The present invention discloses a multimodal middle school experimental step detection method and system based on motion target semantic enhancement. First, the video frame is preprocessed, the motion area is obtained by frame difference, the motion target is obtained by target detection technology, and the semantic timing features are extracted by using the BERT model. Then, the video features are modeled for temporal dependency in the encoder to obtain the visual timing features at the step level, which are then fused with the motion target semantic features in the decoder to build the connection between the experimental steps and the corresponding targets, so as to realize the accurate detection of the experimental steps in the middle school experimental video. The patented method can more effectively capture the unique motion features of the experimental steps, effectively distinguish different steps, realize the accurate judgment of the experimental steps, and has broad application prospects.

Description

Technical Field

[0001] The present invention belongs to the technical field of intelligent education of artificial intelligence, and mainly relates to a multimodal middle school experiment step detection method and system based on motion target semantic enhancement. Background Art

[0002] Experimental teaching is an important teaching content stipulated in the national curriculum plan and curriculum standards, an important way to cultivate innovative talents, and an important part of comprehensively improving the quality of education. It is necessary to include experimental teaching in the education quality evaluation and monitoring system, and include students' experimental operation ability performance in the comprehensive quality evaluation.

[0003] However, the current evaluation model of middle school experiments faces many challenges. With limited teaching time and a large number of students, how to reform the experimental operation evaluation method to comprehensively and efficiently assess students' experimental operation ability has become an urgent issue. In response to these challenges, it is crucial to utilize the transformative potential of artificial intelligence technology. Artificial intelligence has the ability to analyze and evaluate student experiments in real time, providing a basis for establishing a comprehensive and standardized evaluation framework. By exploring and using artificial intelligence to strengthen middle school experimental evaluation, the quality and effectiveness of science education can ultimately be improved.

[0004] Current research methods mainly use online action detection algorithms to analyze the temporal relationship of actions in videos to achieve accurate detection of action categories. However, there are still some bottlenecks when processing middle school experimental videos: First, the scenes of middle school experiments are usually more complex, involving the use of multiple experimental instruments and different experimental steps. This requires the algorithm to accurately capture the interactive relationship between the experimenter and the experimental instrument in order to better understand the experiment. However, the existing methods fail to make full use of these feature information, resulting in inaccurate recognition and understanding of the experimental steps, affecting the in-depth analysis of the video content; secondly, the current action detection method usually adopts a dual-stream structure, in which optical flow information occupies an important position in the extraction process. However, the extraction process of optical flow is quite time-consuming, and in practical applications, it is easily affected by video quality and complex scenes, resulting in low algorithm operation efficiency. In addition, optical flow information has an important impact on the performance of action detection. Removing optical flow will lead to a decrease in detection accuracy, which limits the application of current methods in actual teaching scenarios. Summary of the invention

[0005] The present invention is aimed at the problems existing in the prior art, and provides a multimodal middle school experimental step detection method and system based on motion target semantic enhancement. First, the video frame is preprocessed, the motion area is obtained by frame difference, the motion target is obtained by target detection technology, and the semantic timing features are extracted using the BERT model. Then, the video features are modeled for temporal dependency in the encoder to obtain the visual timing features at the step level, which are then fused with the semantic features of the motion target in the decoder to build the connection between the experimental steps and the corresponding targets, so as to achieve accurate detection of the experimental steps in the middle school experimental video. This patented method can more effectively capture the unique motion features of the experimental steps, effectively distinguish different steps, and achieve accurate judgment of the experimental steps, and has broad application prospects.

[0006] In order to achieve the above object, the technical solution adopted by the present invention is: a multimodal middle school experiment step detection method based on motion target semantic enhancement, comprising the following steps:

[0007] S1, data preprocessing: preprocess the input middle school experiment video data, convert the video into a video frame sequence, and use each frame as the anchor point center to obtain a group of 16 video frame segments;

[0008] S2, semantic temporal feature extraction: obtain the moving target and use YOLOV8 to obtain the semantics of the moving target, and use the BERT model to encode the semantics of the moving target to generate the semantic temporal features of the moving target;

[0009] S3, acquisition of visual temporal features at the video frame level: use the visual large model VideoMAE to extract features from video frame segments and obtain visual temporal features at the video frame level;

[0010] S4, step-level visual timing feature acquisition: input the video frame-level timing features obtained in step S3 into the encoder to learn the step-level visual timing features; the encoder at least includes a timing fusion module to form the step-level visual timing features from the time relationship between different frames in the video sequence;

[0011] S5, feature fusion: inputting the step-level visual temporal features obtained in step S4 and the moving target semantic temporal features obtained in step S2 into the decoder for fusion; the decoder enhances the visual features with the moving target semantic features, and outputs the classification results of the video frames through the fully connected layer;

[0012] S6, result output: Use the multimodal features of the fusion step-level visual temporal features and the semantic temporal features of the moving target for classification, and output the classification results of the video frame.

[0013] As an improvement of the present invention, the data preprocessing in step S1 also includes data standardization processing, setting the mean of each channel of red, green and blue of the RGB image of the original video frame to 0.485, 0.456 and 0.406, and the standard deviation to 0.229, 0.224 and 0.225, respectively calculating the mean μ and standard deviation σ of all frames, and setting each video frame x i After normalization, we get z i , so that the data is easier to converge during training, improving the training efficiency and generalization ability of the model; then convert the original video frame fragments into tensor format, specifically:

[0014]

[0015]

[0016]

[0017] Among them, the mean of all frames is μ, the standard deviation is σ, and x i represents the video frame, z i represents the normalized video frame, and n is the number of video frames.

[0018] As another improvement of the present invention, the step S2 specifically includes the following steps:

[0019] S21, obtaining motion area: using the inter-frame difference calculation method to analyze the pixel differences between adjacent video frames to capture the changes in the motion area in the video, and by comparing and analyzing the changes in each pixel point, determining the changed area, thereby accurately extracting the dynamic motion area in the video:

[0020] v i =(z i+1 -z i )∧(z i -z i -1)

[0021] Among them, z i Indicates the current frame, z i+1 and z i-1 They represent two adjacent frames, and the results of pairwise difference can be obtained to get the motion area v of this segment. i ;

[0022] S22, moving target detection: Use the target detection technology YOLOv8 to detect the target in the intermediate frame and obtain the corresponding experimental instrument target. In the experiment, each step usually requires the use of specific experimental instruments. Identifying the experimental instruments involved can provide an accurate basis for the detection and analysis of subsequent experimental steps;

[0023] S23, obtain the semantics of the moving target: binarize the moving area obtained in step S21, filter out the moving target by calculating the intersection and union ratio of the target's coordinate box and the moving area, and then use the BERT model to extract the semantic temporal features of the moving target.

[0024] As another improvement of the present invention, in step S23, when binarizing the moving area, pixel values ​​greater than 30 are set to 255, and pixel values ​​less than or equal to 30 are set to 0; when screening moving targets, the threshold is set to 0.7.

[0025] As another improvement of the present invention, in step S3, the visual large model VideoMAE mask self-encoding method is used. For time T, the video frames of the previous t time points are sent to the feature extractor u together to obtain the feature f at the current time point. T :

[0026]

[0027] As another improvement of the present invention, step S4 is specifically as follows: the temporal fusion module is used to capture the temporal relationship between different frames in the video sequence to form a visual temporal feature at the learning step level, which is mainly composed of a multi-head self-attention layer, including attention projection, calculation of attention weights, weighted summation and multi-head mechanism. The multi-head attention is to use multiple query vectors Q i , in parallel from the input information K i Select multiple sets of information from the dataset, calculate the attention scores, apply scaled dot products and softmax to normalize the range of the attention scores, and finally add them to the value V i The weighted sum is used to get the output. During the query process, each query vector will focus on different parts of the input information, that is, analyze the current input information from different angles; in addition, there are Dropout layers, nonlinear activation functions, and fully connected layers:

[0028]

[0029] The operation process of the temporal fusion module is as follows: first, a position-encoded tensor is created according to the input sequence length and feature dimension, then the input and position encoding are added to fuse the position information, and then the multi-head self-attention mechanism is used to capture the video sequence F T The temporal relationship between different frames in the image is calculated, and finally the output features are normalized and the new features are returned.

[0030]

[0031] As a further improvement of the present invention, step S5 specifically includes the following steps:

[0032] S51: Using multi-head self-attention mechanism to fuse the semantics of multi-frame moving objects;

[0033] S52: Use the cross attention mechanism to query context information from the fused semantic features of step S51; specifically, the output features of the encoder As a query, a multi-head attention mechanism is used to integrate the semantic features of the moving target S c Find useful information to enrich yourself;

[0034]

[0035] S53: Obtain a multimodal feature map that integrates the visual temporal features at the step level and the semantic temporal features of the moving target:

[0036] In order to achieve the above-mentioned purpose, the present invention also adopts the technical solution of: a multimodal middle school experiment step detection system based on motion target semantic enhancement, including a computer program, which implements the steps of any of the above-mentioned methods when executed by a processor.

[0037] Compared with the prior art, the present invention has the following beneficial effects:

[0038] (1) The present invention adopts the motion target semantic enhancement technology, which can more effectively capture the key information in the middle school experiment video, so that the model can understand the experimental steps and actions more accurately, thereby improving the detection accuracy and generalization ability;

[0039] (2) This paper proposes a new action recognition network to cope with the complex scenes and diverse experimental steps in middle school experimental videos. Through motion target semantic enhancement, it more effectively learns the connection between steps and instruments, thereby improving the recognition ability of experimental steps.

[0040] (3) This patent can effectively solve the problems of manual detection, such as the experimental steps consuming a lot of human resources, the existing methods not fully exploring the temporal relationship, and low efficiency. It eliminates the time-consuming optical flow operation and improves the detection accuracy. Therefore, it has high application value and application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 is a flow chart of the steps of the method of the present invention;

[0042] Figure 2 is a flowchart of the steps of obtaining the semantics of the moving target in step S2 of the method of the present invention;

[0043] Figure 3 is a schematic diagram of the structure of the encoder in step S4 and the decoder in step S5 of the method of the present invention;

[0044] Figure 4 It is a comparison result diagram of the experimental steps detected in the test example of the present invention. DETAILED DESCRIPTION

[0045] The present invention will be further explained below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention.

[0046] Example 1

[0047] A multimodal middle school experimental step detection method based on motion target semantic enhancement, such as Figure 1 As shown, the following steps are included:

[0048] S1: Input a middle school experiment video, pre-process the video frames, and use each frame as the anchor point center to obtain a group of 16 video frame segments.

[0049] First, read the video file and convert the video into a sequence of video frames; then scale the size of each frame to 224*224 to ensure that the input of the model has a consistent size; set the anchor point of the video segment as the key frame in the segment; sample the video frames and obtain a group of 16 video frame segments centered on the anchor point; standardize the original video frames. In RGB images, there are usually three channels (red, green, and blue). The mean of each channel is set to 0.485, 0.456, and 0.406, and the standard deviation is set to 0.229, 0.224, and 0.225. Through standardization, the data is converted into a distribution with a mean of 0 and a standard deviation of 1, so that the data is easier to converge during training, thereby improving the training efficiency and generalization ability of the model; finally, convert the original video frame segments into a tensor format for input into the deep learning model for training.

[0050]

[0051]

[0052]

[0053] S2: Get the moving target and use YOLOV8 to get the semantics of the moving target, and use the BERT model to encode the semantics of the moving target to generate the semantic temporal features of the moving target. Figure 2 As shown, Figure 2 The flowchart of steps to obtain the semantics of moving targets is as follows:

[0054] S21, obtaining motion regions: using an inter-frame difference calculation method to analyze pixel differences between adjacent video frames to capture changes in motion regions in the video, and by comparing and analyzing changes in each pixel point to determine which regions have changed, thereby accurately extracting dynamic motion regions in the video;

[0055] v i =(z i+1 -z i )∧(z i -z i-1 )

[0056] S22, target detection: Use the target detection technology YOLOv8 to detect the target in the intermediate frame and obtain the corresponding experimental instrument target. In the experiment, each step usually requires the use of specific experimental instruments. Identifying the experimental instruments involved can provide an accurate basis for the detection and analysis of subsequent experimental steps;

[0057] S23, obtain the semantics of moving targets: first, binarize the moving area, set pixel values ​​greater than 30 to 255, and set pixel values ​​less than or equal to 30 to 0. Then, filter out the moving targets by calculating the intersection and union ratio of the target's coordinate box and the moving area. Set the threshold to 0.7. Finally, use the BERT model to extract the semantic temporal features of the moving targets.

[0058] S3: Use the visual large model VideoMAE to extract features from video frame fragments and obtain visual timing features at the video frame level.

[0059] S31: The original video frame segment enters the feature extractor;

[0060] S32: The feature extractor adopts the visual large model VideoMAE mask self-encoding method to capture the visual timing information at the video frame level in the clip;

[0061] S33: The original feature map output from the feature extractor enters the encoder:

[0062]

[0063] S4: The extracted video frame-level temporal features are input into the encoder to learn the step-level visual temporal features;

[0064] like Figure 3 As shown, the encoder includes at least one temporal fusion module: first, a position-encoded tensor is created according to the input sequence length and feature dimension, then the input and the position code are added to fuse the position information, then a multi-head self-attention mechanism is used to capture the temporal relationship between different frames in the video sequence, and finally the output features are normalized and new features are returned;

[0065]

[0066]

[0067] The temporal fusion module is used to capture the temporal relationship between different frames in the video sequence to form visual temporal features at the learning step level. It is mainly composed of a multi-head self-attention layer, including attention projection, calculation of attention weights, weighted summation and multi-head mechanism. In addition, there are dropout layers, nonlinear activation functions and fully connected layers.

[0068] S5: The processed step-level visual temporal features and the semantic temporal features of the moving target are input into the decoder for fusion.

[0069] S51: Using multi-head self-attention mechanism to fuse the semantics of multi-frame moving objects;

[0070] S52: Query useful contextual information from fused semantic features using cross-attention mechanism;

[0071] S53: Obtain a multimodal feature map that fuses the visual temporal features at the step level with the semantic temporal features of the moving target.

[0072]

[0073] The decoder is used to fuse the step-level visual temporal features with the moving target semantic temporal features, enhance the visual features with the moving target semantic features, and finally output the classification results of the video frames through the fully connected layer.

[0074] S6: Use the multimodal features of the fusion step-level visual temporal features and the semantic temporal features of the moving target to perform classification and output the classification results of the video frame.

[0075] The Adam method is used for stochastic gradient descent, the batch size is configured to 128, the learning rate is set to 0.0001, and the weight decay is set to 0.0005. The object detection algorithm uses YOLOv8, and the semantic extraction model uses BERT, all with default settings. In order to speed up the training process, the feature extraction speed is set to 1fps, and the end-to-end classification network is fully trained.

[0076] Test Case

[0077] The method of the present invention was applied to two real experimental data sets for experiments, and the selected middle school experiments were oxygen experiments and carbon dioxide experiments. The oxygen experiment data set includes more than 90 hours of videos of potassium permanganate oxygen production, and 32 steps are annotated. The carbon dioxide experiment data set contains 222 groups of videos and 11 steps of limestone producing carbon dioxide. In comparison, the former is more complex, and the latter has a shorter action duration. In order to measure the gap between the segmentation results and the actual results, this test example uses the average precision (mAP) per frame to evaluate the effectiveness of online action detection. mAP is a commonly used indicator, and the average precision (AP) of each action category needs to be averaged. The higher the mAP result, the higher the classification accuracy.

[0078] The visual comparison results of the middle school experimental steps are as follows Figure 4 As shown in the figure, it can be seen that the detection result obtained by the present invention is basically consistent with the real result, except that there is a deviation in the classification of individual frames, and it can be regarded as a reliable result overall.

[0079] The comparison of the results of the detection using this method is shown in Table 1.

[0080] Table 1

[0081]

[0082] It can be seen from the above table that the detection accuracy of the present invention on the oxygen data set and the carbon dioxide data set is better than that of other methods, and the efficiency is also improved, which is more suitable for practical applications.

[0083] In the description of this specification, the description with reference to the terms "one embodiment", "example", "specific example", etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0084] It should be noted that the above content only illustrates the technical idea of ​​the present invention and cannot be used to limit the protection scope of the present invention. For ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications all fall within the protection scope of the claims of the present invention.

Claims

1. A multimodal middle school experimental step detection method based on motion target semantic enhancement, characterized in that: The steps include: S1, data preprocessing: preprocess the input middle school experiment video data, convert the video into a video frame sequence, and use each frame as the anchor point center to obtain a group of 16 video frame segments; S2, semantic temporal feature extraction: obtain the moving target and use YOLOV8 to obtain the semantics of the moving target, and use the BERT model to encode the semantics of the moving target to generate the semantic temporal features of the moving target; S3, acquisition of visual temporal features at the video frame level: use the visual large model VideoMAE to extract features from video frame segments and obtain visual temporal features at the video frame level; S4, step-level visual timing feature acquisition: input the video frame-level timing features obtained in step S3 into the encoder to learn the step-level visual timing features; the encoder at least includes a timing fusion module to form the step-level visual timing features from the time relationship between different frames in the video sequence; S5, feature fusion: input the step-level visual temporal features obtained in step S4 and the moving target semantic temporal features obtained in step S2 into the decoder for fusion; the decoder enhances the visual features with the moving target semantic features, and outputs the classification results of the video frames through the fully connected layer; S6, result output: Use the multimodal features of the fusion step-level visual temporal features and the semantic temporal features of the moving target for classification, and output the classification results of the video frame.

2. The multimodal middle school experiment step detection method based on motion target semantic enhancement as claimed in claim 1 is characterized in that: The data preprocessing in step S1 also includes data standardization processing. The mean of each channel of red, green and blue of the RGB image of the original video frame is set to 0.485, 0.456 and 0.406, and the standard deviation is set to 0.229, 0.224 and 0.

225. The mean μ and standard deviation σ of all frames are calculated respectively. i After normalization, we get z i , and then convert the original video frame fragment into a tensor format, specifically: Among them, x i represents the video frame, z i represents the normalized video frame, and n is the number of video frames.

3. The multimodal middle school experiment step detection method based on motion target semantic enhancement as claimed in claim 1 is characterized in that: The step S2 specifically includes the following steps: S21, obtaining the motion area: using the inter-frame difference calculation method, analyzing the pixel differences between adjacent video frames, and extracting the dynamic motion area in the video: in i =(from i+1 -With i )∧(of i -With i-1 ) Among them, z i Indicates the current frame, z i+1 and z i-1 They represent two adjacent frames, and the results of pairwise difference can be obtained to get the motion area v of this segment. i ; S22, target detection: Use the target detection technology YOLOv8 to detect the target in the intermediate frame and obtain the corresponding experimental instrument target; S23, obtain the semantics of the moving target: binarize the moving area obtained in step S21, filter out the moving target by calculating the intersection and union ratio of the target's coordinate box and the moving area, and then use the BERT model to extract the semantic temporal features of the moving target.

4. The multimodal middle school experiment step detection method based on motion target semantic enhancement as claimed in claim 3 is characterized in that: In step S23, when binarizing the moving area, pixel values ​​greater than 30 are set to 255, and pixel values ​​less than or equal to 30 are set to 0; when screening moving targets, the threshold is set to 0.

7.

5. The multimodal middle school experiment step detection method based on motion target semantic enhancement as claimed in claim 4 is characterized in that: In step S3, the video frames of the previous t time points are sent to the feature extractor u at time T to obtain the feature f at the current time point. T :

6. The multimodal middle school experiment step detection method based on motion target semantic enhancement as claimed in claim 1, characterized in that: In step S4, the temporal fusion module is used to capture the temporal relationship between different frames in the video sequence to form visual temporal features at the learning step level. It is mainly composed of a multi-head self-attention layer, including attention projection, calculation of attention weights, weighted summation and multi-head mechanism. The multi-head attention uses multiple query vectors Q i , in parallel from the input information K i Select multiple sets of information from the , calculate the attention score, and then apply the scaled dot product and softmax to normalize the range of the attention score, and the value V i The weighted summation gives the output: in, For input information, d k To indicate input information Dimensions; The operation process of the temporal fusion module is as follows: according to the input sequence length and feature dimension, a position encoding tensor is created, the input and position encoding are added together, and then the multi-head self-attention mechanism is used to capture the video sequence F T The temporal relationship between different frames in the image is calculated, and finally the output features are normalized and the new features are returned.

7. The multimodal middle school experiment step detection method based on motion target semantic enhancement as claimed in claim 3 or 4, characterized in that: The step S5 specifically includes the following steps: S51: Using multi-head self-attention mechanism to fuse the semantics of multi-frame moving objects; S52: query context information from the fused semantic features of step S51 using a cross attention mechanism; S53: Based on the output characteristics of the encoder As a query, a multi-head attention mechanism is used to integrate the semantic features of the moving target S c Find useful information and obtain a multimodal feature map that integrates the step-level visual temporal features and the semantic temporal features of the moving target:

8. A multimodal secondary school experimental procedure detection system based on motion target semantic enhancement, including a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method as described in any one of claims 1 to 7 are implemented.

Images