A method for identifying physics and chemistry experimental operation behaviors based on spatiotemporal characteristics

Abstract

The present invention relates to the field of behavior recognition technology, and is a method for identifying physical and chemical experiment operation behaviors based on spatiotemporal features. The method utilizes a lightweight deep learning network to identify the state of experimental equipment, establishes a physical and chemical experiment operation key step model according to the experimental equipment transition state, divides a video into a number of experimental operation video segments through the physical and chemical experiment operation key step model, dynamically samples the video segments according to the experimental operation key step model to obtain key experimental operation behavior images, extracts spatiotemporal features from the key experimental operation behavior images through deep learning temporal networks and spatial networks, obtains recognition results of students' experimental operation behaviors in each video segment, and identifies physical and chemical experiment operation behaviors.

Description

Technical Field

[0001] The present invention relates to the technical field of behavior recognition, and in particular to a method for identifying physical and chemical experimental operation behaviors based on spatiotemporal features. Background Art

[0002] Promoting student lab performance exams and strengthening student practical skills are essential requirements for quality-oriented education in the process of educational reform. The unique nature of lab performance exams dictates that their reliability and efficiency are closely linked not only to the quality of the test questions and the scientific rationality of the scoring criteria, but also to the accuracy and reliability of the scoring methods. To ensure the accuracy and reliability of lab performance exams, it is necessary to accurately identify various student lab behaviors and further quantify and evaluate these behaviors.

[0003] Student action recognition is crucial for identifying student behavior in experimental experiments. Currently, action recognition models the spatiotemporal information of pre-segmented time-domain sequences, learning the representational and motion information contained in videos. This approach establishes a mapping between video content and action categories, enabling computers to effectively perform video understanding tasks. Compared to image classification tasks, action recognition in videos involves an additional temporal dimension. Therefore, capturing the temporal information contained between video frames is a key issue in action recognition. Action recognition methods primarily fall into two categories: traditional handcrafted feature-based methods and deep learning-based methods. Traditional action recognition methods can be categorized based on feature extraction methods, such as those based on human joints, spatiotemporal points of interest, and dense trajectories. These methods encode the manually extracted features into high-dimensional feature vectors, which are then used to train a classifier for action recognition. However, the extraction and encoding of feature vectors often results in high spatiotemporal complexity, which significantly impacts algorithm performance. With the successful application of convolutional neural networks in image classification, their accuracy has significantly surpassed that of traditional handcrafted methods, and action recognition has shifted from traditional handcrafted feature-based methods to deep learning-based methods. Action recognition methods based on deep learning use an end-to-end approach to complete classification by autonomously learning behavioral representations in videos through the network. Action recognition methods based on deep learning mainly revolve around a two-stream network, which consists of a spatial stream network and a temporal stream network. The disadvantages of traditional two-stream networks are: (1) Since optical flow only represents the motion information between adjacent frames, the two-stream network has very limited access to temporal context, which is not conducive to modeling some actions with a large time span; (2) The two-stream network simply combines the predictions of the spatial stream network and the temporal stream network classifiers. When the scenes or movements in the video are extremely similar, it is easy for one network to misjudge and cause errors in the entire video-level prediction. At the same time, the temporal network requires pre-calculation of optical flow, but the pre-calculation and storage of optical flow are very expensive, and optical flow will cause device delay, which will also limit its performance in real-time detection. Summary of the Invention

[0004] In order to overcome the deficiencies in the prior art, the present invention aims to solve the following technical problem: providing a method for identifying physical and chemical experimental operation behaviors based on spatiotemporal features. A lightweight deep learning network is used to identify the state of experimental equipment. A model of key steps in physical and chemical experimental operations is established based on the state transitions of the experimental equipment. The video is divided into several experimental operation video segments using the model of key steps in physical and chemical experimental operations. The video segments are dynamically sampled based on the model of key steps in experimental operations to obtain images of key experimental operation behaviors. Deep learning of the temporal and spatial networks of the key experimental operation behavior images is used to extract spatiotemporal features. The recognition results of the student's experimental operation behaviors in each video segment are obtained, thereby identifying the physical and chemical experimental operation behaviors.

[0005] The technical solution adopted by the present invention is: a method for identifying physical and chemical experimental operation behaviors based on spatiotemporal characteristics, which is carried out in the following steps:

[0006] Step 1: Determine the physical and chemical experiment of the experimental operation behavior to be identified and all the experimental equipment involved, use a camera device to capture the video of the physical and chemical experimental process completed by the experimenter, and sample the physical and chemical experimental operation video at time intervals to obtain a series of experimental result images;

[0007] Step 2: Use the trained deep learning network YOLOv5 on the computer to identify the experimental results of the experimenter, and identify the experimenter's physical and chemical experimental equipment, their spatial location, and the time point of the experimental operation;

[0008] Step 3: Based on the identified experimental equipment, spatial locations, and experimental operation time points of the experimenter's physical and chemical experimental operation, determine the event Em activated by the experimental operation and the degree of significant change in the experimental state Vmij, where m is the experimental equipment number. All involved experimental equipment are numbered starting from 0 and m is a consecutive integer. i is the start time of event E, and j is the end time of event E.

[0009] Step 4: Count the number of experimental operation activation events TE and the degree of change in the state of the experimental equipment TV within a certain time period k. When TE>TH E And TV>TH V When , this time period is considered to be the key step of the experimental operation, TH E TH is the threshold for the activation quantity of experimental operation events in key experimental steps. V It is the threshold for judging the degree of equipment status change in key experimental steps.

[0010] Step 5: Divide the physical and chemical experiment video into several video segments based on the number of key steps and the time period in which these steps occur. The number of video segments is the number of key steps, and the time period in which these steps occur is the duration of the video segments. Dynamically sample each of these segmented video segments to obtain a dataset of key experimental operation information. The dynamic sampling rate is determined by the number of experimental operation activation events and the degree of change in the experimental equipment.

[0011] Step 6: Use the deep learning dual-stream network model that has been trained in the computer to identify the experimental operation of the experimenter and identify the experimental operation behavior of the experimenter.

[0012] The deep learning network YOLOv5 trained on a computer means that before the deep learning network YOLOv5 recognizes the experimental result images of the experimenter, a large number of correctly completed physical and chemical experimental processes are video-captured using a camera device to form correct experimental process videos. The correct experimental process operation videos are sparsely sampled at equal time intervals to obtain a series of experimental operation result images. The time intervals for the equal time interval sampling of the experimenter's experimental process videos are the same as the time intervals for the equal time interval sampling of the correct experimental process operation videos. The image calibration tool software LabelImg is used to label each correct experimental process experimental equipment, i.e., the original dataset of experimental equipment information, with the annotation content including name, center coordinates, length, and width, to obtain an experimental equipment information label dataset. The original dataset of experimental equipment information is divided into an original training dataset of experimental equipment information and an original verification dataset of experimental equipment information. Correspondingly, the experimental equipment label dataset is divided into an experimental equipment label training dataset and an experimental equipment label verification dataset. The deep learning network YOLOv5 is used to train a recognition model on the original training dataset of experimental equipment information and the experimental equipment label training dataset. The trained recognition model is verified using the original verification dataset of experimental equipment information and the experimental equipment label training dataset, thereby obtaining a recognition model that can correctly identify experimental equipment.

[0013] In step 3, the event Em activated by the experimental operation and the degree of significant change in the experimental state Vmij are determined by identifying the experimental equipment for the physical and chemical experiment of the experimenter and its spatial position and the experimental operation time point to obtain the center coordinates and the length and width of the experimental equipment, which are recorded as Bmi=<Xmi,Ymi,Wmi,Hmi> , where Xmi represents the normalized horizontal coordinate of the center of the m experimental equipment at time i, Ymi represents the normalized vertical coordinate of the center of the m equipment at time i, Wmi represents the normalized width of the m equipment at time i, and Hmi represents the normalized height of the m equipment at time i; calculate the state change SCm of the m experimental equipment at different times. The state change SCm of the experimental equipment includes the spatial position change Dmij and the aspect ratio change WHmij, SCm=a*Dmij+b*WHmij, where a and b are the weight coefficients of the spatial position change and the aspect ratio change, and their values ​​are determined according to the usage of different experimental equipment in the experimental process. The initial values ​​are a=0.5 and b=0.5 respectively, which can be adjusted for different physical and chemical experiments, where Dmij=abs(Xmi-Xmj)+abs(Ymi-Ymj), WHmij=(Wmi*Hmj) / (Wmj*Hmi), and abs is the absolute value operation. If SCm>THsc, it is considered that the equipment state has changed significantly and there is a potential experimental operation behavior. The experimental operation event Em is activated, that is, Em=1, otherwise Em=0, and the equipment state change significance Vmij is assigned to Vmij=min{1 / ,(SCm-THsc) / THsc}, where THsc is the judgment threshold for the experimental equipment state change.

[0014] In step five, the dynamic sampling rate is determined based on the number of experimental operation activation events and the degree of change of experimental equipment. The dynamic sampling rate R is related to the number of experimental operation activation events TE and the degree of change TV of experimental equipment. If no experimental operation activation event occurs, the sampling rate is lowered, and if an experimental operation activation event occurs, the sampling rate is increased. The degree of increase in the value of the sampling rate R is proportional to the number of experimental operation activation events TE and the degree of change TV of experimental equipment; let the average sampling rate be Ra, and the sampling rate is lowered if no experimental operation activation event occurs in the time period, and its sampling rate R = Ra / N, N is a positive integer greater than 1, determined according to the specific physical and chemical experiment, and the initial value is 3. If an experimental operation activation event occurs in the time period, the sampling rate R is increased, and the degree of increase in the value of the sampling rate R is proportional to the number of experimental operation activation events and the degree of change of experimental equipment. Let the total time of a certain time frequency segment be TT, and calculate the average number of experimental operation activation events TE in the time period TT. a and the average change degree TV of experimental equipment a , find the adjustment coefficients α and β, α = TE ij / TE a ,β=TV ij / TV a, sampling rate R = 1.5*(α+β)*R a .

[0015] The deep learning two-stream network model that has been trained in the computer means that before the deep learning two-stream network model recognizes the experimental operation behavior images of the experimenter, a large number of correctly completed physical and chemical experimental processes are video-captured by a camera device to form a correct experimental process, and steps one to five are performed on the correct experimental process operation video to obtain a series of key experimental operation behavior result images. The image calibration tool software LabelImg is used to label each key experimental operation behavior, i.e., a key experimental operation behavior information dataset, to obtain a key experimental operation behavior label dataset. The key experimental operation behavior information dataset is divided into a key experimental operation behavior information training dataset and a key experimental operation behavior information verification dataset. Correspondingly, the key experimental operation behavior label dataset is divided into a key experimental operation behavior label training dataset and a key experimental operation behavior label verification dataset. The deep learning two-stream network is used to train the recognition model on the key experimental operation behavior information training dataset and the key experimental operation behavior label training dataset. The trained recognition model is verified using the key experimental operation behavior information verification dataset and the key experimental operation behavior label verification dataset to obtain a recognition model that can correctly recognize experimental operation behaviors.

[0016] The beneficial effects of the present invention compared to the existing technology are as follows: Currently, the technology for object recognition through deep learning is mature and has high recognition accuracy. The present invention first uses a lightweight deep learning model to correctly identify physical and chemical experimental equipment and their status, establishes a model of key operation steps based on the experimental equipment and status, and completes the preliminary division of the operation video by experimental steps, which is more conducive to the further identification of experimental operational behaviors. The segmented video clips are dynamically sampled based on the number of experimental operation behavior activation events and the degree of change in the experimental equipment obtained by the correct identification of the experimental equipment and status. Videos containing experimental operation behaviors are densely sampled, while videos without experimental operation behaviors are sparsely sampled, which is more conducive to improving the accuracy of the dual-stream network and achieving real-time implementation. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a schematic flow diagram of the present invention. DETAILED DESCRIPTION

[0018] The following describes the technical solution of the present invention in detail with reference to an embodiment: The present invention utilizes a lightweight deep learning network to identify the state of experimental equipment, establishes a model of key steps in physical and chemical experimental operations based on the experimental equipment's transition state, and divides the video into several experimental operation video segments using the key step model. Based on the experimental operation prior information and the key step model, the video segments are dynamically sampled to obtain a key experimental operation image dataset. The key experimental operation image dataset is then fed into a deep learning temporal network and spatial network to extract spatiotemporal features, thereby obtaining recognition results for student experimental operation behaviors in each video segment. The following explanation uses the example of preparing a sodium chloride solution of a certain concentration in a chemical experiment in a physical and chemical experimental operation exam.

[0019] The preparation of sodium chloride solution of a certain concentration mainly includes five key steps: (1) Weighing solid sodium chloride, the main experimental equipment is: tray balance, solid sodium chloride, medicine spoon and beaker, etc.; (2) Measuring distilled water, the main experimental equipment is: measuring cylinder, rubber-tipped dropper, distilled water and beaker, etc.; (3) Dissolving solid sodium chloride, the main experimental equipment is: glass rod and beaker, etc.; (4) Pour the prepared solution into a narrow-necked bottle, the main experimental equipment is: narrow-necked bottle and beaker, etc.; (5) Fill in and paste labels, the main experimental equipment is: narrow-necked bottle and label paper.

[0020] The specific detection and classification steps are as follows:

[0021] Step 1: Use a camera device to collect videos of a large number of correctly completed experimental processes of preparing sodium chloride solution of a certain concentration to form a video of the correct experimental process of preparing sodium chloride solution of a certain concentration. Sparse and equal time interval sampling is performed on the correct experimental process operation video to obtain a series of experimental operation result pictures. For example, the student experimental video is evenly sampled at a sampling rate of 5 frames / second.

[0022] Step 2: Use the image calibration tool software LabelImg to label each correct experimental process experimental equipment, that is, the original data set of experimental equipment information. The annotation content includes name, center coordinates, length and width. The experimental equipment dataset for preparing a sodium chloride solution of a certain concentration is obtained. The experimental equipment dataset list is {tray balance, laboratory paper, weights, wide-mouth bottle (containing sodium chloride), medicine spoon, measuring cylinder, rubber-tipped dropper, beaker, glass rod, narrow-necked bottle and label}. The experimental equipment dataset for preparing sodium chloride solution of a certain concentration is divided into a training dataset of experimental equipment for preparing sodium chloride solution of a certain concentration and a verification dataset of experimental equipment for preparing sodium chloride solution of a certain concentration. Correspondingly, the label dataset for preparing sodium chloride solution of a certain concentration is divided into a label training dataset for preparing sodium chloride solution of a certain concentration and a label verification dataset for preparing sodium chloride solution of a certain concentration. The deep learning network YOLOv5 is used to train the recognition model on the experimental equipment training dataset for preparing sodium chloride solution of a certain concentration and the label training dataset for preparing sodium chloride solution of a certain concentration. The trained recognition model is verified using the experimental equipment verification dataset for preparing sodium chloride solution of a certain concentration and the label verification dataset for preparing sodium chloride solution of a certain concentration, and a recognition model that can correctly identify the experimental equipment for preparing sodium chloride solution of a certain concentration is obtained.

[0023] Step 3: The time interval for sampling the video of the student's experimental process is the same as the time interval for sampling the video of the correct experimental process. The sampled image is obtained. The experimental equipment recognition model for preparing a sodium chloride solution of a certain concentration is used to identify the experimental equipment for this experimental operation image. The main experimental equipment such as the tray balance, experimental paper, weights, wide-mouth bottle (containing sodium chloride), medicine spoon, measuring cylinder, rubber-tipped dropper, beaker, glass rod, narrow-mouth bottle and label are identified, as well as the center coordinates of each experimental equipment at a certain time i and the length and width of the experimental equipment, recorded as Bmi=<Xmi,Ymi,Wmi,Hmi> , where Xmi represents the normalized horizontal coordinate of the center of m equipment at time i, Ymi represents the normalized vertical coordinate of the center of m equipment at time i, Wmi represents the normalized width of m equipment at time i, and Hmi represents the normalized height of m equipment at time i.

[0024] Step 4: Calculate the state change SCm of the experimental equipment at different moments m. The state change SCm of the experimental equipment includes the spatial position change Dmij and the aspect ratio change WHmij. SCm = a*Dmij+b*WHmij, where a and b are the weight coefficients of the spatial position change and the aspect ratio change, which are determined according to the usage of different experimental equipment in the experimental process. The initialization values ​​are a=0.5 and b=0.5, respectively. They can be adjusted for different physical and chemical experiments. Dmij = abs(Xmi-Xmj)+abs(Ymi-Ymj), WHmij = (Wmi*Hmj) / (Wmj*Hmi), and abs is the absolute value operation. If SCm>THsc, THsc is the experimental equipment state change setting value obtained based on experience, it is considered that the equipment state has changed significantly and there is a potential experimental operation behavior. This experimental operation event Em is activated, that is, Em=1, otherwise Em=0, and the equipment state change significance Vmij is assigned to Vmij=min{1 / ,(SCm-THsc) / THsc}, THsc is the experimental equipment state change judgment threshold.

[0025] Step 5: Count the number of events and degree of change of the key step equipment that has changed significantly in a certain period of time. For example, the number of events TE and degree of change TV of the main equipment used to weigh sodium chloride solid, including the tray balance, sodium chloride solid, medicine spoon and beaker, that have changed significantly in a certain period of time are calculated using the following formulas: When TE>TH E And TV>TH V If the time period is , then the time period is considered to be the key step of the experimental operation for weighing the sodium chloride solid; otherwise, the time period is considered not to be the key step of the experimental operation for weighing the sodium chloride solid.

[0026] Step 6: Based on the statistical data from step 5, the five key steps for preparing a sodium chloride solution of a certain concentration are obtained. The total experimental operation video is divided into five experimental operation video segments based on the five key steps obtained.

[0027] Step 7: Dynamically sample the five video clips to obtain a dataset of key experimental operation behavior information. The dynamic sampling rate R is related to the number of experimental operation activation events TE and the degree of change TV of the experimental equipment. If no experimental operation activation event occurs, the sampling rate is reduced; if an experimental operation activation event occurs, the sampling rate is increased. The degree of increase in the sampling rate R is proportional to the number of experimental operation activation events TE and the degree of change TV of the experimental equipment. Assuming the average sampling rate is R a , no experimental operation activation event occurs during the time period, reducing the sampling rate, and its sampling rate R=R a / N, where N is a positive integer greater than 1, determined according to the specific physical and chemical experiment, and its initial value is 3. The sampling rate R is increased when an experimental operation activation event occurs in the time period. The degree of increase in the sampling rate R is proportional to the number of experimental operation activation events and the degree of change in the experimental equipment. Assuming that the total time of a certain time frequency segment is TT, calculate the average number of experimental operation activation events TE in the time period TT. a and the average change degree TV of experimental equipment a , find the adjustment coefficients α and β, α = TE ij / TE a ,β=TV ij / TV a , sampling rate R = 1.5*(α+β)*R a .

[0028] Step 8: Use the image calibration tool software LabelImg to label the key experimental operation behavior information data set obtained by dynamic sampling of 5 video clips, mainly labeling the experimental operation behavior, and obtain the key experimental operation behavior label data set. The experimental action information label data set list {level the balance, place the experimental paper, place the weights, place the sodium chloride solid, pour the sodium chloride solid, take the measuring cylinder, pour the distilled water, add the liquid with a rubber-tipped dropper, observe the concave liquid surface, add distilled water into the beaker, stir, add the sodium chloride solution into the narrow-necked bottle, write labels, label, and organize the laboratory table}. The key experimental operation behavior information dataset for preparing a sodium chloride solution of a certain concentration is divided into a training dataset for the key experimental operation behavior information for preparing a sodium chloride solution of a certain concentration and a verification dataset for the key experimental operation behavior information for preparing a sodium chloride solution of a certain concentration. Correspondingly, the key experimental operation behavior label dataset for preparing a sodium chloride solution of a certain concentration is divided into a training dataset for the key experimental operation behavior label for preparing a sodium chloride solution of a certain concentration and a verification dataset for the key experimental operation behavior label for preparing a sodium chloride solution of a certain concentration. A deep learning two-stream network is used to train a sodium chloride solution experimental operation behavior recognition model using the key experimental operation behavior information training dataset and the key experimental operation behavior label training dataset. The trained sodium chloride solution experimental operation behavior recognition model is verified using the key experimental operation behavior information verification dataset and the key experimental operation behavior label verification dataset, resulting in a recognition model that can correctly identify the experimental operation behavior of preparing a sodium chloride solution of a certain concentration. The recognition network architecture is shown in the attached figure.

[0029] Step 9: Identify the student's experimental operation of preparing a sodium chloride solution of a certain concentration through the trained recognition model of the experimental operation behavior of preparing a sodium chloride solution of a certain concentration, and identify the student's experimental operation behavior of preparing a sodium chloride solution of a certain concentration.

[0030] The above content is a further detailed description of the present invention in combination with specific implementation methods. It cannot be determined that the specific implementation methods of the present invention are limited to these. For ordinary technicians in the technical field to which the present invention belongs, they can make several simple deductions or substitutions without departing from the present invention, which should be regarded as belonging to the scope of patent protection determined by the submitted claims of the present invention.

Claims

1. A method for identifying physical and chemical experimental operation behaviors based on spatiotemporal characteristics, characterized by: Proceed as follows: Step 1: Determine the physical and chemical experiment and all the experimental equipment involved in the experimental operation behavior to be identified, use a camera to capture the video of the physical and chemical experimental process completed by the experimenter, perform sparse and equal time interval sampling on the physical and chemical experimental operation video, and obtain a series of experimental result images; Step 2: Use the trained deep learning network YOLOv5 on the computer to identify the experimental results of the experimenter, and identify the experimenter's physical and chemical experimental equipment, their spatial location, and the time point of the experimental operation; Step 3: Based on the identified experimental equipment used in the experimenter's physical and chemical experiments, their spatial locations, and the experimental operation time points, determine the event Em activated by the experimental operation and the degree of significant change in the experimental state Vmij, where m is the experimental equipment number. All involved experimental equipment are numbered starting from 0, m is a consecutive integer, i is the start time of the event Em, and j is the end time of the event Em; Step 4: Count the number of experimental operation activation events TE and the degree of change in the state of the experimental equipment TV within a certain time period k. When TE>TH E And TV>TH V When , this time period is considered to be the key step of the experimental operation, TH E The threshold for the number of activations of experimental operation events for key steps of the experiment, TH V The threshold for judging the degree of change in the equipment status in key experimental steps; Step 5: Divide the physical and chemical experiment video into several video segments based on the number of key steps in the physical and chemical experiment and the time period in which the key steps occur. The number of key steps is the number of divided video segments, and the time period in which the key steps occur is the duration of the video segment. Dynamic sampling is performed on the divided video segments to obtain a key experimental operation information dataset. The dynamic sampling rate is determined by the number of experimental operation activation events and the degree of change in the experimental equipment. Step 6: Use the deep learning dual-stream network model that has been trained in the computer to identify the experimental operation of the experimenter and identify the experimental operation behavior of the experimenter.

2. The method for identifying physical and chemical experimental operation behaviors based on spatiotemporal characteristics according to claim 1 is characterized in that: The deep learning network YOLOv5 that has been trained in the computer means that before the deep learning network YOLOv5 recognizes the experimental result image of the experimenter, a large number of correctly completed physical and chemical experimental processes are video-captured using a camera device to form correct experimental process videos, and the correct experimental process operation videos are sparsely sampled at equal time intervals to obtain a series of experimental operation result images. The time interval when the experimenter's experimental process video is sampled at equal time intervals is the same as the time interval when the correct experimental process operation video is sampled at equal time intervals. The image calibration tool software LabelImg is used to label each correct experimental process experimental equipment, that is, the original dataset of experimental equipment information, and the labeling content includes the name, center coordinates, length and width, to obtain an experimental equipment information label dataset; the original dataset of experimental equipment information is divided into an original training dataset of experimental equipment information and an original verification dataset of experimental equipment information, and the corresponding experimental equipment label dataset is divided into an experimental equipment label training dataset and an experimental equipment label verification dataset; the deep learning network YOLOv5 is used to train a recognition model on the original training dataset of experimental equipment information and the experimental equipment label training dataset, and the trained recognition model is verified using the original verification dataset of experimental equipment information and the experimental equipment label training dataset to obtain a recognition model that can correctly identify the experimental equipment.

3. The method for identifying physical and chemical experimental operation behaviors based on spatiotemporal characteristics according to claim 1 is characterized in that: In step 3, the event Em activated by the experimental operation and the degree of significant change in the experimental state Vmij are determined by identifying the experimental equipment for the physical and chemical experiment of the experimenter and its spatial position and the experimental operation time point to obtain the center coordinates and the length and width of the experimental equipment, which are recorded as Bmi=<Xmi,Ymi,Wmi,Hmi> , where Xmi represents the normalized horizontal coordinate of the center of experimental equipment m at time i, Ymi represents the normalized vertical coordinate of the center of equipment m at time i, Wmi represents the normalized width of equipment m at time i, and Hmi represents the normalized height of equipment m at time i. Calculate the state change SCm of experimental equipment m at different times. The state change SCm of experimental equipment m includes the spatial position change Dmij and the aspect ratio change WHmij. SCm = a*Dmij + b*WHmij, where a and b are the weight coefficients of spatial position change and aspect ratio change. Their values ​​are determined according to the usage of different experimental equipment during the experiment, and the initial value is a = 0. 5, b = 0.5, adjusted for different physical and chemical experiments, where Dmij = abs(Xmi-Xmj) + abs(Ymi-Ymj), WHmij = (Wmi*Hmj) / (Wmj*Hmi), abs is the absolute value operation; if SCm>THsc, it is considered that the equipment state has changed significantly and there is a potential experimental operation behavior, and this experimental operation event Em is activated, that is, Em = 1, otherwise Em = 0, and the degree of equipment state change Vmij is assigned to Vmij = min{1, (SCm-THsc) / THsc}, THsc is the judgment threshold for the experimental equipment state change.

4. The method for identifying physical and chemical experimental operation behaviors based on spatiotemporal characteristics according to claim 1, characterized in that: In step 5, the dynamic sampling rate is determined according to the number of experimental operation activation events and the degree of change of experimental equipment. The dynamic sampling rate R is related to the number of experimental operation activation events TE and the degree of change of experimental equipment TV. If no experimental operation activation event occurs, the sampling rate is reduced, and if an experimental operation activation event occurs, the sampling rate is increased. The degree of increase in the value of the sampling rate R is proportional to the number of experimental operation activation events TE and the degree of change of experimental equipment TV. Let the average sampling rate be R a , no experimental operation activation event occurs during the time period, reducing the sampling rate, and its sampling rate R=R a / N, N is a positive integer greater than 1, determined according to the specific physical and chemical experiments, with an initial value of 3. The sampling rate R is increased when an experimental operation activation event occurs during the time period. The degree of increase in the sampling rate R is proportional to the number of experimental operation activation events and the degree of change in the experimental equipment. Let the total time of a certain frequency segment be TT, and calculate the average number of experimental operation activation events TE within the TT time period. a and the average change degree TV of experimental equipment a , find the adjustment coefficients α and β, α = TE ij / TE a ,β=TV ij / TV a , sampling rate R = 1.5*(α+β)*R a .

5. The method for identifying physical and chemical experimental operation behaviors based on spatiotemporal characteristics according to claim 1 is characterized in that: The deep learning two-stream network model that has been trained in the computer means that before the deep learning two-stream network model recognizes the experimental operation behavior picture of the experimenter, a large number of correctly completed physical and chemical experimental processes are video captured by a camera device to form a correct experimental process, and steps one to five are performed on the correct experimental process operation video to obtain a series of key experimental operation behavior result pictures. The image calibration tool software LabelImg is used to label each key experimental operation behavior, that is, the key experimental operation behavior information dataset, to obtain a key experimental operation behavior label dataset, and the key experimental operation behavior information dataset is divided into a key experimental operation behavior information training dataset and a key experimental operation behavior information verification dataset. Correspondingly, the key experimental operation behavior label dataset is divided into a key experimental operation behavior label training dataset and a key experimental operation behavior label verification dataset. The deep learning two-stream network is used to train the recognition model on the key experimental operation behavior information training dataset and the key experimental operation behavior label training dataset, and the trained recognition model is verified using the key experimental operation behavior information verification dataset and the key experimental operation behavior label verification dataset to obtain a recognition model that correctly recognizes the experimental operation behavior.

Images