Virtual-real fusion chemical experiment platform for real-time feedback based on multi-modal perception

Abstract

The invention discloses a virtual-real fusion chemical experiment platform for real-time feedback based on multi-modal perception, and the platform comprises a video capturing module which carries out the video collection of a student experiment process, and captures the operation process of a student using an experiment apparatus; the target detection module is used for detecting various experimental apparatuses and chemical reagents involved in the chemical experiment process in real time; the HOI detection module is used for identifying an interaction behavior between the student and the experiment equipment in an experiment process and judging whether the interaction behavior is aligned with a key point in a preset experiment step or not; the feedback module is used for judging the correctness of the operation in the experiment process of the student according to the detection results of the target detection and the HOI detection, and carrying out feedback prompt in real time for wrong operation; through the video capture module, video acquisition can be carried out on the experiment process when a student independently carries out chemical experiment operation, and according to key nodes in set experiment steps, results of the target detection module and the HOI detection module are compared with those of the nodes to judge whether the operation of the student is correct or not.

Description

Technical Field

[0001] The present invention relates to the technical field of chemical experiment simulation, and in particular to a virtual-reality fusion chemical experiment platform with real-time feedback based on multimodal perception. Background Art

[0002] Chemical experiments can combine theoretical knowledge from books with practical operations to help students better understand and master chemical concepts. During the experiment, since chemical experiments are often accompanied by intuitive phenomena such as color changes and gas generation, students can understand chemical knowledge more intuitively. Moreover, by personally conducting experimental operations, students' hands-on ability can be exercised, which is crucial for cultivating practical talents. However, the operation process of chemical experiments requires strict adherence to experimental steps and the use of correct experimental techniques (such as stirring, mixing, heating, filtering, etc.) to ensure the correctness of the experiment.

[0003] Since some chemical experiments are complicated, students are prone to make mistakes in experimental steps, fail to use correct experimental techniques (such as stirring, mixing, heating, filtering, etc.), and fail to use experimental instruments correctly when performing chemical experiments independently. This will not only cause experimental failures, but also easily cause some serious safety accidents (such as incorrect mixing of chemicals, such as acids and alkalis, which may produce exothermic reactions, causing splashes and burns). For this reason, the present application proposes a method that can analyze and judge whether the student's experimental operation is correct in real time when the student performs an independent chemical experiment, and give correct experimental operation feedback prompts in real time. Summary of the invention

[0004] The embodiment of the present application provides a virtual-reality fusion chemistry experiment platform with real-time feedback based on multimodal perception. Through the video capture module, the experimental process of students can be captured by video when they independently perform chemical experiments. The results of the target detection module and the HOI detection module are compared with the key nodes in the set experimental steps to determine whether the student's operation is correct. If it is detected that the student's operation sequence is wrong or key actions are missing, the student can be prompted by voice or screen.

[0005] The embodiment of the present application provides a virtual-reality fusion chemistry experiment platform with real-time feedback based on multimodal perception, including:

[0006] The video capture module collects videos of students’ experimental process and captures the operation process of students using experimental equipment;

[0007] Target detection module, real-time detection of various experimental equipment and chemical reagents involved in the chemical experiment process;

[0008] The HOI detection module identifies the interaction between students and experimental equipment during the student experiment and determines whether it is aligned with the key points in the preset experimental steps;

[0009] The feedback module judges the correctness of students' experimental operations based on the detection results of the target detection module and the HOI detection module, and provides real-time feedback prompts in the event of incorrect operations.

[0010] The video capture module uses a camera, through which the video of the student's experimental process can be collected in real time.

[0011] The target detection module is used to construct feature detection and extraction of the interaction between students and various experimental equipment and chemical reagents;

[0012] The object detection module uses DETR and CLIP as the main object detection models, and uses the powerful bounding box detection capability of DETR and the multimodal feature extraction capability of CLIP to achieve accurate detection of hands and chemical experimental equipment. The pre-trained model is fine-tuned on the data set of chemical experimental equipment to improve the detection effect of specific experimental equipment.

[0013] The process of feature detection and extraction by the target detection module is as follows:

[0014] Create a data set of chemical experimental equipment, pre-process the images in the data set, and then annotate the images;

[0015] Pre-train the DTER and CLIP models on the dataset of chemical experimental equipment;

[0016] Use the encoder of the CLIP model to extract image and text features respectively;

[0017] The image to be detected is input into the DTER model for target feature detection. The text and image features generated by the CLIP model can enhance the detection capability of the DTER model.

[0018] Finally, NMS (non-maximum suppression) is used for the bounding box output by the DTER model to eliminate overlapping boxes and retain the best detection results.

[0019] The dataset of chemical experimental equipment is constructed by annotating the bounding boxes and category labels of various chemical experimental equipment and chemical reagent targets in the key frames of the collected chemical experiment video data.

[0020] The CLIP model contains an image encoder and a text encoder to extract image features or generate text embeddings of images.

[0021] During the experiment, the interaction between students and experimental equipment, such as picking up the beaker, pouring the solution, mixing the solution and other key steps of the experimental operation, the HOI detection module judges whether the interaction between students and experimental equipment is correct based on the feature information extracted by the target detection module;

[0022] The HOI detection module determines whether the interaction between students and experimental equipment is correct in the following process:

[0023] Constructing the interaction pair between the student and the chemical experiment equipment (the relative position relationship between the student and the chemical experiment equipment) through the HOI detector;

[0024] The HOI decoder is constructed using the Transformer layer to generate paired HOI visual features.

[0025] Using the HOI classifier, a convolutional neural network (CNN) is used in the HOI classifier to further refine the image feature information;

[0026] Set the confidence threshold to post-process the predictions of the interaction pair types and retain the best or most correct prediction results;

[0027] Finally, the HOI detection results are output, and the model performance is evaluated using the evaluation parameters so that the model performance can be adjusted in time according to the evaluation results.

[0028] The Transformer layer contains a criss-cross attention block, a self-attention block, and a feed-forward network (FFN) block.

[0029] Non-maximum suppression (NMS) is used to retain the best or most correct predictions of the interaction pair type.

[0030] The trained HOI model is exported to ONNX or other formats for deployment on different platforms.

[0031] The feedback module is based on the key nodes in the set experimental steps (such as picking up the beaker, pouring the solution, mixing the solution and other key steps of the experimental operation). It compares the results of the target detection module and the HOI detection module with these nodes to determine whether the student's operation is correct. If it is detected that the student's operation sequence is wrong or key actions are missing, the student can be prompted through voice or screen.

[0032] The specific process of real-time detection of students' experimental process during the experiment is as follows:

[0033] (1) Data Collection

[0034] Use a camera to collect video or image data of the experimental scene;

[0035] (2) Object Detection

[0036] Standardize the collected videos or images, such as resizing and denoising. Use the DTER model and CLIP model to extract image features and determine the category of experimental equipment. The Decoder of the DETR model uses the self-attention mechanism to associate the output of the Encoder to generate accurate bounding boxes and category predictions, and output a set of candidate bounding boxes and corresponding category probabilities.

[0037] (3) HOI detects the interaction between students and chemical experimental equipment

[0038] The HOI detection model identifies the interaction between people and objects in the image and determines the type of interaction action, such as "taking" and "operating".

[0039] (4) Behavioral analysis

[0040] Formulate the judgment rules of positive and negative feedback according to the experimental operation procedures (that is, the correct operation process of chemical experiments), and compare the behaviors detected in real time with the judgment rules of positive and negative feedback;

[0041] (5) Feedback Generation

[0042] Positive feedback process: when the detected operation behavior conforms to the experimental procedures, the feedback module gives positive feedback and can inform the operator of the correct behavior through sound prompts;

[0043] In the negative feedback process, when the detected operation behavior does not conform to the experimental procedures, the feedback module gives negative feedback and informs the operator of the behavior error through warning sound or display on the display screen;

[0044] (6) Feedback communication

[0045] Feedback information is displayed on the display screen in the experimental operation area and broadcast through speech synthesis technology so as to inform students in time.

[0046] One or more technical solutions provided in the embodiments of the present application have at least the following technical effects or advantages: through the video capture module, when students perform chemical experiments independently, their experimental process can be captured by video; based on the key nodes in the set experimental steps (such as picking up a beaker, pouring solutions, mixing solutions and other key steps of experimental operations), the results of the target detection module and the HOI detection module are compared with these nodes to determine whether the student's operation is correct; if it is detected that the student's operation sequence is wrong or key actions are missing, the student can be prompted by voice or screen to ensure the correctness of the student's independent chemical experiment operation. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 This is a system flow chart of the virtual-real fusion chemical experiment teaching platform used in this application;

[0048] Figure 2 Flowchart of feature detection and extraction process for the target detection module of this application;

[0049] Figure 3 A flowchart of the process of judging whether the interaction between students and experimental equipment is correct for the HOI detection module of this application;

[0050] Figure 4 This is the output result flow chart of the feedback module for this application. DETAILED DESCRIPTION

[0051] In order to facilitate the understanding of the present invention, the present invention will be described more fully below with reference to the relevant drawings. The preferred embodiments of the present invention are given in the drawings. However, the present invention can be implemented in many different forms and is not limited to the embodiments described herein. On the contrary, the purpose of providing these embodiments is to make the disclosure of the present invention more thoroughly understood.

[0052] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used herein in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The term "and / or" used herein includes any and all combinations of one or more related listed items.

[0053] Embodiment 1

[0054] See also Figure 1-4 , a virtual-real fusion chemical experiment platform based on multimodal perception for real-time feedback, including:

[0055] The video capture module collects videos of students’ experimental process and captures the operation process of students using experimental equipment;

[0056] Target detection module, real-time detection of various experimental equipment and chemical reagents involved in the chemical experiment process;

[0057] The HOI detection module identifies the interaction between students and experimental equipment during the student experiment and determines whether it is aligned with the key points in the preset experimental steps;

[0058] The feedback module judges the correctness of students' experimental operations based on the detection results of the target detection module and the HOI detection module, and provides real-time feedback prompts in the event of incorrect operations.

[0059] The video capture module uses a camera, through which the video of the student's experimental process can be collected in real time;

[0060] The camera uses a binocular camera, which consists of two cameras, which can simulate the human binocular vision system and provide stereoscopic vision to facilitate the subsequent measurement of the distance between students and chemical experimental equipment;

[0061] Choosing a high-resolution camera can provide clearer images, which helps the HOI detection module to more accurately identify students and experimental equipment; a wide dynamic range can help the camera maintain image quality under different lighting conditions;

[0062] In order to capture fast-moving objects or actions, the camera needs to have sufficient frame rate, and the camera needs to be stably mounted to avoid image jitter, which is critical for accurate HOI detection modules. The camera needs to cover the entire experimental table, and multiple cameras are required to achieve full coverage of the operating experimental process; finally, the camera needs to be compatible with the video capture device and HOI detection system, facilitating the selection of effective video compression formats for storage and transmission of high-resolution video data.

[0063] In order to achieve all-round collection of students' experimental process and ensure that the light changes in the laboratory have the least impact on the camera, multiple binocular cameras can be installed at different positions on the experimental table to achieve 360-degree coverage of the complete experimental operation process without blind spots; if the experimental area is large, a mobile camera platform can be used, such as a robot carrying a camera, to achieve dynamic tracking and coverage;

[0064] At the same time, control the light in the experimental environment to reduce the impact of reflections and shadows on video capture; use uniform and stable light sources in the laboratory as much as possible, avoid direct sunlight or strong and changing artificial light, use blackout curtains or blinds to control natural light entering from the windows, or use artificial lighting systems, such as lights with adjustable brightness and color temperature, to maintain consistency of light in the laboratory.

[0065] The object detection module uses DETR and CLIP as the main object detection models, and uses the powerful bounding box detection capability of DETR and the multimodal feature extraction capability of CLIP to achieve accurate detection of hands and chemical experimental equipment. The pre-trained model is fine-tuned on the data set of chemical experimental equipment to improve the detection effect of specific experimental equipment.

[0066] The process of feature detection and extraction by the target detection module is as follows:

[0067] 1.1 Data Preparation

[0068] The collected chemical experiment videos are used to annotate the key frames in the chemical experiment video data with the bounding boxes and category labels of various chemical experiment equipment and chemical reagent targets to create a data set of chemical experiment equipment;

[0069] The images in the chemical experimental equipment dataset are preprocessed by using operations such as scaling and normalization;

[0070] Then, the processed image is annotated with correct information using bounding boxes and category labels to facilitate feature analysis by the target detection module and the HOI detection module.

[0071] 1.2 Model Ensemble

[0072] The DETR model is pre-trained on the basis of the chemical laboratory equipment dataset, so that it can detect students and laboratory equipment in images or videos, and the parameters of the DETR model are adjusted to adapt to the special requirements of the laboratory environment;

[0073] At the same time, using the chemical experimental equipment dataset to pre-train the CLIP model can improve the accuracy of feature detection.

[0074] 1.3 Visual Feature Extraction

[0075] The CLIP model consists of an image encoder and a text encoder;

[0076] The image encoder of the CLIP model extracts features of the input image;

[0077] The text encoder of the CLIP model generates text embeddings for each target category. These features are used to enhance the feature representation of the DETR model and improve the detection accuracy.

[0078] 1.4 Object Detection

[0079] The image to be detected is input into the DETR model. The DETR model uses the Transformer architecture to extract and refine image features, and uses its Encoder-Decoder architecture to generate region proposals. The Decoder of the DETR model uses the self-attention mechanism to associate the output of the Encoder, generate accurate bounding boxes and category predictions, and output a set of candidate bounding boxes and corresponding category probabilities.

[0080] The text and image features generated by the CLIP model are used to enhance the detection ability of the DETR model, and the DETR model is trained by taking the similarity between text embedding and image features as an additional loss function.

[0081] 1.5 Post-processing

[0082] Apply NMS (non-maximum suppression) to the bounding boxes output by DETR to eliminate overlapping boxes and retain the best detection results;

[0083] Filter the detection results according to the confidence threshold and only keep the detection boxes with high confidence (first set an initial value such as 10-20 cm, and then fine-tune the initial threshold according to the accuracy of the detection results to improve the correctness of the detection results).

[0084] The chemical experiment videos collected in the chemical experiment equipment dataset are videos of chemical experiments that are correctly operated in a laboratory environment. They can be used to construct the chemical experiment equipment dataset for the target detection module and the dataset of students interacting with the experimental equipment for the HOI detection module.

[0085] The dataset of chemical experimental equipment needs to be accurately labeled, including the location of the experimental equipment, the location of the students' hands, and the categories of students' interaction with the experimental equipment (such as picking up the beaker, pouring the solution, mixing the solution, and other key steps of the experimental operation).

[0086] During the experiment, the interaction between students and experimental equipment, such as picking up the beaker, pouring the solution, mixing the solution and other key steps of the experimental operation, the HOI detection module judges whether the interaction between students and experimental equipment is correct based on the feature information extracted by the target detection module;

[0087] The HOI detection module determines whether the interaction between students and experimental equipment is correct in the following process:

[0088] 2.1 Build an interactive relationship between students and experimental equipment

[0089] The HOI detector is selected based on the target frames of students and chemical experimental equipment detected by the target detection module. All interaction pairs are processed by the HOI detector and all possible interaction pairs are constructed. The spatial relationship filtering strategy is used to screen out spatially close interaction pairs. Specifically, a threshold is first set. Only when the distance between the center points of the bounding boxes of students and chemical experimental equipment is less than the threshold, we consider that there may be interaction between them (first set an initial value such as 10-20 cm, and then fine-tune the initial threshold according to the accuracy of the detection results to improve the accuracy of the detection results); secondly, the relative position between students and chemical experimental equipment is analyzed. For example, when a certain chemical experimental equipment is above or behind the student, such interaction is considered unreasonable. Such relative position is used to further screen or prioritize certain interaction pairs. After the spatial relationship screening, the remaining interaction pairs are retained. The HOI detector uses interaction features and spatial information injection to generate HOI tokens. These HOI tokens represent potential interaction behaviors between students and experimental equipment. Based on the detected bounding boxes of students and chemical experimental equipment, all possible interaction pairs are constructed.

[0090] 2.2 Generating paired HOI visual features

[0091] Using the HOI encoder, the constructed pairs of HOI tokens are input into the HOI decoder, which generates paired HOI visual features by aggregating contextual visual cues from the CLIP image encoder and text encoder as well as relevant semantic information.

[0092] 2.3 Prediction of interaction pair types

[0093] Using the HOI classifier, a convolutional neural network (CNN) is used in the HOI classifier to further refine the category text features of the CLIP text encoder, the visual features of the CLIP image encoder, and the decoded paired HOI visual features; then feature fusion is used to combine the features of students, chemical laboratory equipment, and interactions, and an attention mechanism is introduced to weight the features of different regions, so that the model pays more attention to the regions that are important for judging the interaction type; after feature fusion, a fully connected layer (FFN) is used to predict the interaction type. The cross entropy loss of the classification stage is calculated to train the HOI classifier to identify different interaction types, such as students holding a beaker and pouring liquid. The extracted features are input into the HOI classifier to obtain the interaction type and confidence score of each interaction pair.

[0094] 2.4 The best or most correct prediction result is retained

[0095] Set a confidence threshold, and filter out low-confidence interaction type prediction results based on the preset confidence threshold; predict multiple interaction types for the same pair of students and chemical laboratory equipment, and use non-maximum suppression (NMS) to retain the best or most correct prediction results (first set an initial value such as 10-20 cm, and then fine-tune the initial threshold based on the accuracy of the test results to improve the accuracy of the test results).

[0096] 2.5 Result Output

[0097] Finally, the HOI detection results are output, including the bounding boxes of students and chemical experimental equipment, the interaction type, and the corresponding confidence scores.

[0098] The HOI decoder consists of multiple customized Transformer layers, each of which contains a criss-cross attention block, a self-attention block, and a feed-forward network (FFN) block.

[0099] After the model training is completed, the average precision (AP) and mAP can be used ( mean A verageThe performance of the model is evaluated by using indicators such as precision and accuracy. The DETR model and CLIP model used are combined and compared with the traditional DETR model. The performance of the DETR model and CLIP model is improved by combining image and text feature analysis.

[0100] Use Precision, Recall, and F1-Score to evaluate the effect of the HOI detection model on interaction recognition.

[0101] Later, the model's hyperparameters, such as learning rate, bounding box matching threshold, etc., can be adjusted based on the evaluation results; data enhancement techniques, such as random cropping and color transformation, can be applied to improve the generalization ability of the model.

[0102] The trained model is exported to ONNX or other formats for deployment on different platforms.

[0103] The object detection module and the HOI detection module are compared with the key nodes of the experimental steps (such as picking up the beaker, pouring the solution, mixing the solution, and other key steps of the experimental operation). The feedback model determines whether the student's operation is correct based on the detection results;

[0104] If an error in the operation sequence or missing key actions is detected, students will be prompted by voice playback through a voice player or by playing a correct operation video on a display screen.

[0105] The display screen can highlight the correct operation path in real time in the video stream by overlaying images.

[0106] Collect usage feedback from students and teachers, analyze the model's false detections and missed detections, continuously iterate the model, continuously improve the data set and model performance, and improve the accuracy and robustness of detection.

[0107] The technical solution in the above-mentioned embodiment of the present application has at least the following technical effects or advantages: through the video capture module, when the student performs a chemical experiment independently, the experimental process can be captured by video, and according to the key nodes in the set experimental steps (such as picking up the beaker, pouring the solution, mixing the solution and other key steps of the experimental operation), the results of the target detection module and the HOI detection module are compared with these nodes to determine whether the student's operation is correct. If it is detected that the student's operation sequence is wrong or the key action is missing, the student can be prompted by voice or screen to ensure the correctness of the student's independent chemical experiment operation.

[0108] Embodiment 2

[0109] A chemical experiment teaching platform for virtual-real integration is equipped with a computer device, which consists of at least one processor, a memory, and a computer program stored in the memory and can be run on the processor; when the processor executes the computer program, the chemical experiment teaching platform for virtual-real integration can judge whether the student's experimental process operation is correct;

[0110] The computer device is equipped with a computer-readable storage medium, which stores a computer program. When the computer program is executed by the processor, it can realize the processing work between the video capture module, the target detection module, the HOI detection module and the feedback module.

[0111] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A virtual-real fusion chemistry experiment platform with real-time feedback based on multimodal perception, characterized by: include, The video capture module collects videos of the students' experimental process and captures the operation process of the students using the experimental equipment. The video capture module uses a camera, and the camera uses a binocular camera; Target detection module, real-time detection of various experimental equipment and chemical reagents involved in the chemical experiment process; The HOI detection module identifies the interaction between students and experimental equipment during the student experiment and determines whether it is aligned with the key points in the preset experimental steps; The feedback module judges the correctness of students' experimental operations based on the detection results of the target detection module and the HOI detection module, and provides real-time feedback prompts in the event of incorrect operations.

2. The virtual-reality fusion chemistry experimental platform with real-time feedback based on multimodal perception as claimed in claim 1, characterized in that: The target detection module uses DETR and CLIP as the main target detection models, and uses the powerful bounding box detection capability of DETR and the multimodal feature extraction capability of CLIP to achieve accurate hand and chemical laboratory equipment detection.

3. The virtual-real fusion chemical experiment platform with real-time feedback based on multimodal perception as claimed in claim 1, characterized in that: The process of feature detection and extraction by the target detection module is as follows: Create a data set of chemical laboratory equipment, use scaling and normalization operations to pre-process the images in the data set, and then use bounding boxes and category labels to correctly annotate the images; Pre-train the DTER and CLIP models on the dataset of chemical experimental equipment; Use the encoder of the CLIP model to extract image and text features respectively; The image to be detected is input into the DETR model for target feature detection. The DETR model uses the Transformer architecture to extract and refine image features, and uses its Encoder-Decoder architecture to generate region proposals. The Decoder of the DETR model uses the self-attention mechanism to associate the output of the Encoder, generate accurate bounding boxes and category predictions, and output a set of candidate bounding boxes and corresponding category probabilities. The text and image features generated by the CLIP model are used to enhance the detection capability of the DETR model. The DETR model is trained by taking the similarity between text embedding and image features as an additional loss function. Finally, NMS (non-maximum suppression) is used for the bounding box output by the DTER model, and the detection results are filtered according to the confidence threshold, retaining only the detection boxes with high confidence.

4. The virtual-reality fusion chemistry experimental platform with real-time feedback based on multimodal perception as claimed in claim 3, characterized in that: The chemical experiment videos collected in the chemical experiment equipment dataset are videos of chemical experiment operations performed correctly in a laboratory environment. The key frames in the chemical experiment video data are annotated with the bounding boxes and category labels of various chemical experiment equipment and chemical reagent targets. This can be used to construct a chemical experiment equipment dataset for a target detection module and a dataset of student interaction with experiment equipment for a HOI detection module.

5. The virtual-reality fusion chemistry experimental platform with real-time feedback based on multimodal perception as claimed in claim 3, characterized in that: The content of the data set annotation of the chemical experimental equipment includes the position of the experimental equipment, the position of the student's hand, and the category of the student's interaction behavior with the experimental equipment.

6. The virtual-reality fusion chemistry experimental platform with real-time feedback based on multimodal perception as claimed in claim 3, characterized in that: The CLIP model includes an image encoder and a text encoder; The image encoder of the CLIP model extracts features of the input image; The text encoder of the CLIP model generates text embeddings for each target category. These features are used to enhance the feature representation of the DETR model and improve the detection accuracy.

7. The virtual-real fusion chemical experiment platform with real-time feedback based on multimodal perception as claimed in claim 1, characterized in that: The HOI detection module determines whether the interaction between students and experimental equipment is correct in the following process: The target boxes of students and chemical experimental equipment detected by the HOI detector process all interaction pairs through the HOI detector and construct all possible interaction pairs. Use the spatial relationship filtering strategy to filter out spatially close interaction pairs; The HOI decoder is constructed using the Transformer layer, and the constructed pairs of HOI tokens are input into the HOI decoder. The HOI decoder generates paired HOI visual features by aggregating the contextual visual cues and related semantic information of the CLIP image encoder and the text encoder. Using the HOI classifier, a convolutional neural network (CNN) is used in the HOI classifier to further refine the category text features of the CLIP text encoder, the visual features of the CLIP image encoder, and the decoded paired HOI visual features; then feature fusion is used to combine the features of students, chemical experimental equipment, and interactions, and an attention mechanism is introduced to weight the features of different regions, so that the model pays more attention to the areas that are important for judging the interaction type; After feature fusion, a fully connected layer (FFN) is used to predict the interaction type. The cross entropy loss of the classification stage is calculated to train the HOI classifier to identify different interaction types, such as students holding a beaker and pouring liquid. The extracted features are input into the HOI classifier to obtain the interaction type and confidence score of each interaction pair; The confidence threshold is set to post-process the prediction of the interaction pair type, and the low-confidence interaction type prediction results are filtered out according to the preset confidence threshold; multiple interaction types of the same pair of students and chemical experimental equipment are predicted, and non-maximum suppression (NMS) is used to retain the best or most correct prediction results; Finally, the HOI detection results are output, and the model performance is evaluated using the evaluation parameters Precision, Recall, and F1-Score, so that the model performance can be adjusted in time according to the evaluation results.

8. The virtual-reality fusion chemistry experimental platform with real-time feedback based on multimodal perception as claimed in claim 7, characterized in that: The specific construction process of the interaction between students and chemical experimental equipment, First, we set a threshold. Only when the distance between the center point of the bounding box of the student and the chemical experimental equipment is less than the threshold, we consider that there may be interaction between them. Secondly, we analyze the relative position between the student and the chemical experimental equipment. For example, when a chemical experimental equipment is above or behind the student, such interaction is considered unreasonable. We use such relative position to further screen or prioritize certain interaction pairs. After screening the spatial relationship, the remaining interaction pairs are retained. The HOI detector uses interaction features and spatial information injection to generate HOI tokens. These HOI tokens represent potential interaction behaviors between students and experimental equipment. Based on the detected bounding boxes of students and chemical experimental equipment, all possible interaction pairs are constructed.

9. The virtual-reality fusion chemistry experimental platform with real-time feedback based on multimodal perception as claimed in claim 1, characterized in that: The feedback module compares the key nodes of the experimental steps through the target detection module and the HOI detection module, and the feedback model determines whether the student's operation is correct based on the detection results; If an error in the operation sequence or missing key actions is detected, students will be prompted by voice playback through a voice player or by playing a correct operation video on a display screen.

10. The virtual-real fusion chemical experiment platform with real-time feedback based on multimodal perception as claimed in claim 1, characterized in that: The specific process of real-time detection of students' experimental process during the experiment is as follows: (1) Data Collection Use a camera to collect video or image data of the experimental scene; (2) Object Detection Standardize the collected videos or images; The DTER model and CLIP model are used to extract image features and determine the category of experimental equipment. The Decoder of the DETR model uses the self-attention mechanism to associate the output of the Encoder to generate accurate bounding boxes and category predictions, and output a set of candidate bounding boxes and corresponding category probabilities. (3) HOI detection of the interaction between students and chemical experimental equipment. The HOI detection model identifies the interaction relationship between people and objects in the image and determines the type of interaction action. (4) Behavioral analysis Formulate the judgment rules of positive and negative feedback according to the experimental operation procedures, and compare the behaviors detected in real time with the judgment rules of positive and negative feedback; (5) Feedback Generation Positive feedback process: when the detected operation behavior conforms to the experimental procedures, the feedback module gives positive feedback and can inform the operator of the correct behavior through sound prompts; In the negative feedback process, when the detected operation behavior does not conform to the experimental procedures, the feedback module gives negative feedback and informs the operator of the wrong behavior through warning sounds or display screens; (6) Feedback communication Feedback information is displayed on the display screen in the experimental operation area and broadcast through speech synthesis technology so as to inform students in time.

Images