A remote virtual experiment system and method combined with internet of things

Abstract

The application belongs to the technical field of internet distance education, and discloses a remote virtual experiment system and method combined with the internet of things, which comprises a near-end virtual reality module, a cloud-end management and analysis module and a remote internet of things experiment module; the near-end virtual reality module is used for restoring the modeling remote laboratory scene, experiment action and experiment visual data, and executing element control instructions through a control handle; the cloud-end management and analysis module is used for realizing the centralized management of the internet of things equipment, data collection, analysis and judgment of the control instructions issued to students, recording of student experiment information and judgment of experiment results; and the remote internet of things experiment module is used for realizing sensor data uploading and control instruction issuing, and controlling and identifying the action of the laboratory executing element. The application has strong experiment safety, and all instruction operations are completed through network communication, so that the action of other laboratory equipment is reduced, the safety and reliability of student experiments are ensured, and dangerous experiments are avoided.

Description

Technical Field

[0001] This invention belongs to the field of Internet-based distance education technology, and in particular relates to a remote virtual experiment system and method that combines the Internet of Things. Background Technology

[0002] In recent years, with the continuous development of information technology and internet technology, remote experiments have been widely used in university teaching. Remote experimental teaching breaks the limitations of time and space, bringing great convenience to students. The focus of student experiments is to verify theories through hands-on practice, which requires that virtual experimental data be authentic and reliable. Moreover, with numerous laboratory equipment, how to effectively collect experimental data and safely execute experimental actions is the key to remote experiments.

[0003] The application of virtual reality technology, Internet of Things technology, and artificial intelligence provides new solutions for remote experimental teaching. The integration of these methods can enhance the authenticity and effectiveness of experimental practice. By combining experimental equipment resources with students' experimental needs, the laboratory can be made "visible, controllable, perceptible, and judgeable," effectively improving experimental efficiency and the quality of experimental teaching. Summary of the Invention

[0004] The purpose of this invention is to provide a remote virtual experiment system and method that integrates the Internet of Things to solve the above-mentioned technical problems.

[0005] To address the aforementioned technical problems, the specific technical solution of the present invention, which integrates a remote virtual experiment system and method based on the Internet of Things, is as follows:

[0006] A remote virtual experiment system integrating the Internet of Things (IoT) includes a near-end virtual reality module, a cloud management and analysis module, and a remote IoT experiment module.

[0007] The near-end virtual reality module is used to restore and model the remote laboratory scene, experimental actions and experimental visualization data through multi-angle video data and sensor data, and to convert the execution intention into the control command of the execution element through the control handle.

[0008] The cloud-based management and analysis module is used to centrally manage IoT devices, collect data, analyze and judge control commands issued by students, record student experimental information, and evaluate experimental results through various algorithms and communication technologies.

[0009] The remote IoT experimental module is used to network laboratory equipment through a smart gateway, construct an experimental equipment network, realize sensor data uploading and control command issuance, and control and identify the actions of laboratory actuators.

[0010] Furthermore, the near-end virtual reality module is at least one system including a processor for 3D modeling and positioning, a virtual reality device with communication functions, and an operating handle with a multi-angle rotating joystick.

[0011] Furthermore, the cloud management and analysis module is at least one cloud server that integrates processing algorithms, communication functions, an Internet of Things platform, and database storage functions.

[0012] Furthermore, the processing algorithm functions of the cloud management and analysis module include computer vision target detection, 3D reconstruction, threshold judgment and analysis for fault and danger early warning, scoring and judging student experimental behavior by combining target detection results and threshold judgment results, and generating experimental analysis reports.

[0013] score=(detect_objisobj1?)*w1+(detect_obj_i <obj1_i?)*w2…(3)

[0014] Where w1 and w2 are parameter weights, detect_obj is the experimental equipment and its number detected in this step, obj1 is the reference action equipment and its number entered by the teacher, and the equipment score is obtained by judging whether it conforms to the specified experimental equipment, detect_obj_i is the current value of the experimental equipment in this step, and obj1 is the reference data entered by the teacher, and the data score is obtained by judging whether it conforms to the specified value range.

[0015] Furthermore, the remote IoT experimental module is at least one system including communication, smart gateway, experimental equipment with unique IP addresses, multi-angle binocular stereo vision camera, positioning tag, acoustic-optical-electric sensor and control server.

[0016] Furthermore, the communication hardware of the near-end virtual reality module, cloud management and analysis module, and remote IoT experimental module includes a WIFI module and an Ethernet networking device. The communication protocol uses the MQTT protocol, and the control command transmission format is: command code parameter 1, parameter 2, parameter 3, etc. The format is processed and decoded through a communication algorithm model.

[0017] The communication function only accepts instructions sent from a specified server or IP address, restricts access to IoT devices, and is limited to authorized teachers and specific devices. The higher the priority obtained by equation (4), the greater the privilege, and vice versa.

[0018] Priority = IP * w1 + (uid && password) * w2 + firewall * w3 (4)

[0019] Where w1, w2, and w3 are parameter weights, IP is the IP address parameter, obtained by checking if it matches a specified server or IP address, uid is the user ID, password is the user password, and the two are ANDed, obtained by checking if it matches a specific user, and firewall is the firewall authentication parameter, obtained by checking if the user has passed the firewall security authentication.

[0020] This invention also discloses a remote virtual experiment method for a remote virtual experiment system, including an initialization process, a forward transmission process, and a reverse feedback process. The initialization process is used to start the experimental equipment and preset experimental information. The forward transmission process is used to update the experimental data and laboratory model, generate new control commands, and control the actions of the remote experimental equipment. The reverse feedback process is used to collect and process data from the experimental equipment and generate processing result feedback.

[0021] Furthermore, the specific steps of the initialization process are as follows:

[0022] Step A1: Laboratory environment calibration: The binocular stereo camera captures images of the remote experimental platform, and the near-end virtual reality module performs target recognition and 3D reconstruction of the equipment in the area to establish a visible 3D experimental space;

[0023] Step A2: Experimental Equipment Configuration: Configure and enter the IP address of the required experimental equipment for network communication, and confirm the cloud status of the experimental equipment;

[0024] Step A3: Pre-set teaching information input: Teachers input reference experimental data, thresholds, and experimental operation procedures to establish a reference information database.

[0025] Furthermore, the specific steps of the forward transmission process are as follows:

[0026] Step B1: Experimental Status Update: Based on the data information obtained from decoding, update the modeling effect of the experimental equipment's movements to present the virtual laboratory in the virtual reality headset;

[0027] Step B2: Control command generation: Based on the position information and electrical signals of the control handle, the position and direction are detected in real time using a three-dimensional spatial model and converted into control commands through a cloud processor. The commands include information such as the controlled device number, address, and manipulation action, and are encoded in a format that conforms to the MQTT protocol.

[0028] Step B3: Remote IoT Experiment: The generated control commands are downloaded and transmitted to the corresponding remote experimental devices through the cloud IoT platform, the operation commands are decoded, and the devices are driven to perform the corresponding experimental operations.

[0029] Furthermore, the reverse feedback process comprises the following steps:

[0030] Step C1: Experimental data feedback: The remote laboratory is equipped with acoustic, optical, electrical, and gas signal sensors to read real-time experimental data and transmit it to the cloud server;

[0031] Step C2: Experimental Actions and Processing: A multi-point binocular stereo camera is installed in the remote laboratory to capture and track the motion equipment, and the motion parameters are transmitted to the near end.

[0032] Step C3: Experimental Data Processing: Decode the transmitted information, perform image recognition and data analysis through the cloud processor, transmit real-time data and correct / incorrect prompts back to the local end, and update the student's score for this step after analysis.

[0033] The remote virtual experiment system and method of the present invention, which combines the Internet of Things, has the following advantages:

[0034] (1) This invention can meet the remote experiment needs of students. Students do not need to go to the real laboratory in person. They can conduct remote experiments anytime and anywhere and obtain real experimental data.

[0035] (2) The present invention has strong experimental safety. All instructions and operations are completed through network communication, which reduces the operation of other laboratory equipment, ensures the safety and reliability of student experiments, and avoids dangerous experiments.

[0036] (3) This invention improves the effectiveness of experimental equipment management and makes more rational use of experimental resources;

[0037] (4) The network security of the present invention is strong. Through multiple verifications, the device is protected from illegal operation. Attached Figure Description

[0038] Figure 1 This is a block diagram of the remote virtual experiment system combining the Internet of Things of the present invention;

[0039] Figure 2 This is an example scenario diagram of the remote virtual experiment system combining the Internet of Things according to the present invention;

[0040] Figure 3 This is a flowchart of the remote virtual experiment method combining the Internet of Things according to the present invention;

[0041] Figure 4 This is a schematic diagram of an embodiment of the remote virtual experiment method combining the Internet of Things of the present invention. Detailed Implementation

[0042] To better understand the purpose, structure, and function of this invention, the following detailed description of a remote virtual experiment system and method incorporating the Internet of Things, in conjunction with the accompanying drawings, is provided.

[0043] like Figure 1As shown, the present invention provides a remote virtual experiment system and method incorporating the Internet of Things, including a near-end virtual reality module, a cloud management and analysis module, and a remote IoT experiment module.

[0044] The near-end virtual reality module is at least one system including a processor for 3D modeling and positioning, a virtual reality device with communication capabilities, and an operating handle with a multi-angle rotating joystick. The near-end virtual reality module is used to recreate and model remote laboratory scenes, experimental actions, and experimental visualization data through multi-angle video data and sensor data. The operating handle can convert the execution intention into control commands for the actuators.

[0045] The cloud management and analysis module consists of at least one cloud server that integrates processing algorithms, communication functions, an IoT platform, and database storage. The cloud management and analysis module is used to centrally manage IoT devices, collect data, analyze and judge control commands issued by students, record student experimental information, and evaluate experimental results through various algorithms and communication technologies.

[0046] The cloud-based management and analysis module's processing algorithms include computer vision target detection, 3D reconstruction, threshold judgment and analysis for fault and hazard warning, scoring and analyzing students' experimental behavior based on the combined target detection results and threshold judgment results, and generating experimental analysis reports.

[0047] score = (detect_obj is obj1 ?) * w1 + (detect_obj_i < obj1_i ?)*w2… (3)

[0048] Where w1 and w2 are parameter weights, detect_obj is the experimental equipment and its number detected in this step, obj1 is the reference action equipment and its number entered by the teacher, and the equipment score is obtained by judging whether it conforms to the specified experimental equipment, detect_obj_i is the current value of the experimental equipment in this step, and obj1 is the reference data entered by the teacher, and the data score is obtained by judging whether it conforms to the specified value range.

[0049] The remote IoT experimental module is at least one system including communication, smart gateway, experimental equipment with unique IP address, multi-angle binocular stereo vision camera, positioning tag, acoustic-optical-electric sensor and control server; the remote IoT experimental module is used to network laboratory equipment through smart gateway, construct experimental equipment network, realize sensor data uploading and control command issuance, control and identify the actions of laboratory actuators.

[0050] The communication hardware of the near-end virtual reality module, cloud management and analysis module, and remote IoT experiment module requires WIFI module, Ethernet networking equipment, etc. The communication protocol uses MQTT protocol, and the control command transmission format is: command code parameter 1 parameter 2 parameter 3, etc. The format is processed and decoded by the communication algorithm model.

[0051] The communication function only accepts instructions sent from a specified server or IP address, restricts access to IoT devices, and is limited to authorized teachers and specific devices. The higher the priority obtained by equation (4), the greater the privilege, and vice versa.

[0052] Priority = IP * w1 + (uid && password) * w2 + firewall * w3 (4)

[0053] Where w1, w2, and w3 are parameter weights, IP is the IP address parameter, obtained by checking if it matches a specified server or IP address, uid is the user ID, password is the user password, and the two are ANDed, obtained by checking if it matches a specific user, and firewall is the firewall authentication parameter, obtained by checking if the user has passed the firewall security authentication.

[0054] like Figure 2 The diagram shown is an example scene of the remote virtual experiment system combined with the Internet of Things of the present invention. 1 is a near-end virtual reality device, 2 is a motor, 3 is a robotic arm, 4 is a PLC, 5 is a multi-angle binocular camera, 6 is a gas sensor, and 7 is a smart Internet of Things gateway.

[0055] like Figure 3 As shown, the present invention provides a remote virtual experiment method combining the Internet of Things, including an initialization process, a forward transmission process, and a reverse feedback process. The initialization process completes the startup of the experimental equipment and preset experimental information, etc.; the forward transmission process completes the updating of experimental data and laboratory models, generates new control commands, and controls the actions of the remote experimental equipment; the reverse feedback process completes the data acquisition and data processing of the experimental equipment, and generates processing result feedback.

[0056] The specific steps of the initialization process are as follows:

[0057] (1) Laboratory environment calibration: The binocular stereo camera takes pictures of the remote experimental platform, and the near-end virtual reality module performs target recognition and three-dimensional reconstruction of the equipment in the area to establish a visible three-dimensional experimental space.

[0058] (2) Experimental equipment configuration: Configure and enter the IP address of the required experimental equipment for network communication, and confirm the cloud status of the experimental equipment;

[0059] (3) Pre-set teaching information input: Teachers input experimental data, thresholds, and experimental operation procedures that can be referenced to establish a reference information database.

[0060] The specific steps of the forward transmission process are as follows:

[0061] (1) Experimental status update: Based on the data information obtained from decoding, update the modeling effect of the experimental equipment's actions to present the virtual laboratory in the virtual reality headset;

[0062] (2) Control command generation: Based on the position information and electrical signals of the control handle, the position and direction are detected in real time using a three-dimensional spatial model and converted into control commands through a cloud processor. The commands include the controlled device number, address and control actions, and are encoded in a format that conforms to the MQTT protocol.

[0063] (3) Remote IoT experiment: The generated control commands are downloaded and transmitted to the corresponding remote experimental devices through the cloud IoT platform, the operation commands are decoded, and the devices are driven to perform the corresponding experimental operations.

[0064] The reverse feedback process involves the following steps:

[0065] (1) Experimental data feedback: The remote laboratory is equipped with acoustic, optical, electrical and gas signal sensors to read real-time experimental data and transmit it to the cloud server;

[0066] (2) Experimental actions and processing: A multi-point binocular stereo camera is installed in the remote laboratory to capture and track the motion device and transmit the motion parameters to the near end;

[0067] (3) Experimental data processing: Decode the transmitted information, perform image recognition and data analysis through the cloud processor, transmit real-time data and correct / incorrect prompts back to the local end, and update the student's score for this step after analysis.

[0068] The forward transmission process and the reverse feedback process occur simultaneously and do not affect each other.

[0069] Example 1:

[0070] The following example, a PLC-controlled motor start-up experiment, illustrates the application method of this invention.

[0071] like Figure 4As shown in this embodiment, 1 represents the operation point, 2 represents the menu bar, 3 represents the camera, 4 represents the motor, 5 represents the indicator light being on, 6 represents the indicator light being off, and 7 represents the motor rotation direction. The PLC input terminals I0.0 are the motor start button, I0.1 is the motor forward button, I0.2 is the motor reverse button, and I0.3 is the motor stop button, each connected to the corresponding motor terminal. I0.1 and I0.2 are not interlocked, and Q0.0 and Q0.1 are also not interlocked. The PLC output terminal Q0.0 is the forward contactor, and Q0.1 is the reverse contactor, completing the motor's operation of first rotating forward and then reverse.

[0072] In the laboratory, completing the designated experimental procedures involves the following steps:

[0073] (1) First, calibrate the equipment required for the experiment, obtain the features of the equipment through multi-angle shooting for modeling, configure the IP address of the equipment, ensure the data of the equipment is uploaded to the cloud, and power on the experimental equipment to complete the process.

[0074] (2) In the virtual reality device, press I0.0 and I0.1 with the controller. The command is transmitted to the remote experimental device via the network and the data is returned. The Q0.0 indicator light can be seen to light up through the VR display. The motor shows a forward rotation trend. Experimental data such as voltage, current, and PLC input / output status are displayed. The status of the motor is updated. After judgment, the student's operation is correct and the student's score is recorded.

[0075] (3) Press I0.2 with the controller in the virtual reality device. The command is transmitted to the remote experimental device via the network and the data is returned. You can see the Q0.1 indicator light on the VR display. At this time, both Q0.0 and Q0.1 are lit up, displaying experimental data such as voltage, current, and PLC input / output status. According to the algorithm, it does not meet the actual experimental standards and displays an error message. The program controls the device to stop urgently. At this time, both Q0.0 and Q0.1 are dimmed. I0.0 should be turned off before proceeding to the next step. The student's operation in this step is wrong and points will be deducted.

[0076] (4) Students continue to perform the experiment until the experiment ends, and record the experimental data, experimental images, and student scores.

[0077] It is understood that the present invention has been described through some embodiments, and those skilled in the art will recognize that various changes or equivalent substitutions can be made to these features and embodiments without departing from the spirit and scope of the invention. Furthermore, under the teachings of the present invention, these features and embodiments can be modified to adapt to specific situations and materials without departing from the spirit and scope of the invention. Therefore, the present invention is not limited to the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application are within the protection scope of the present invention.

Claims

1. A remote virtual experiment system integrating the Internet of Things, characterized in that, It includes a near-end virtual reality module, a cloud management and analysis module, and a remote IoT experimental module; The near-end virtual reality module is used to restore and model the remote laboratory scene, experimental actions and experimental visualization data through multi-angle video data and sensor data, and to convert the execution intention into the control command of the execution element through the control handle. The cloud-based management and analysis module is used to centrally manage IoT devices, collect data, analyze and judge control commands issued by students, record student experimental information, and evaluate experimental results through various algorithms and communication technologies. The remote IoT experimental module is used to network laboratory equipment through a smart gateway, construct an experimental equipment network, realize sensor data uploading and control command issuance, and control and identify the actions of laboratory actuators. The communication hardware components of the near-end virtual reality module, cloud management and analysis module, and remote IoT experiment module include a WIFI module and an Ethernet networking device. The communication protocol uses the MQTT protocol, and the control command transmission format is: command code parameter 1, parameter 2, parameter 3, etc. The format is processed and decoded through a communication algorithm model. The communication function only accepts instructions sent from a specified server or IP address, restricts access to IoT devices, and is limited to authorized teachers and specific devices. The higher the priority obtained by equation (4), the greater the privilege, and vice versa. Priority = IP * w1 + (uid && password) * w2 + firewall * w3 (4) where w1, w2, and w3 are parameter weights, IP is the IP address parameter, which is obtained by judging whether it matches the specified server or IP address, uid is the user ID, password is the user password, and the two are in an AND relationship. This parameter is obtained by judging whether it matches a specific user, and firewall is the firewall authentication parameter, which is obtained by judging whether the user has passed the firewall security authentication.

2. The remote virtual experiment system according to claim 1, characterized in that, The near-end virtual reality module is at least one system including a processor for 3D modeling and positioning, a virtual reality device with communication functions, and an operating handle with a multi-angle rotating joystick.

3. The remote virtual experiment system according to claim 1, characterized in that, The cloud management and analysis module is at least one cloud server that integrates processing algorithms, communication functions, an Internet of Things platform, and database storage functions.

4. The remote virtual experiment system according to claim 1, characterized in that, The processing algorithm functions of the cloud management and analysis module include computer vision target detection, 3D reconstruction, threshold judgment and analysis of fault and danger warning, scoring and judging student experimental behavior by combining target detection results and threshold judgment results, and generating experimental analysis reports. score = (detect_objis obj1 ?) * w1 + (detect_obj_i < obj1_i ?)*w2 ...(3) Where w1 and w2 are parameter weights, detect_obj is the experimental equipment and its number detected in this step, obj1 is the reference action equipment and its number entered by the teacher, and the equipment score is obtained by judging whether it conforms to the specified experimental equipment, detect_obj_i is the current value of the experimental equipment in this step, and obj1 is the reference data entered by the teacher, and the data score is obtained by judging whether it conforms to the specified value range.

5. The remote virtual experiment system according to claim 1, characterized in that, The remote IoT experimental module is at least one system including communication, smart gateway, experimental equipment with unique IP addresses, multi-angle binocular stereo vision camera, positioning tag, acoustic-optical-electric sensor and control server.

6. A remote virtual experiment method for a remote virtual experiment system as described in any one of claims 1-5, characterized in that, The process includes an initialization process, a forward transmission process, and a reverse feedback process. The initialization process is used to start the experimental equipment and preset experimental information. The forward transmission process is used to update the experimental data and laboratory model, generate new control commands, and control the actions of the remote experimental equipment. The reverse feedback process is used to collect and process data from the experimental equipment and generate processing result feedback.

7. The remote virtual experiment method according to claim 6, characterized in that, The specific steps of the initialization process are as follows: Step A1: Laboratory environment calibration: The binocular stereo camera captures images of the remote experimental platform, and the near-end virtual reality module performs target recognition and 3D reconstruction of the equipment in the area to establish a visible 3D experimental space; Step A2: Experimental Equipment Configuration: Configure and enter the IP address of the required experimental equipment for network communication, and confirm the cloud status of the experimental equipment; Step A3: Pre-set teaching information input: Teachers input reference experimental data, thresholds, and experimental operation procedures to establish a reference information database.

8. The remote virtual experiment method according to claim 6, characterized in that, The specific steps of the forward transmission process are as follows: Step B1: Experimental Status Update: Based on the data information obtained from decoding, update the modeling effect of the experimental equipment's movements to present the virtual laboratory in the virtual reality headset; Step B2: Control command generation: Based on the position information and electrical signals of the control handle, the position and direction are detected in real time using a three-dimensional spatial model and converted into control commands through a cloud processor. The commands include information such as the controlled device number, address, and manipulation action, and are encoded in a format that conforms to the MQTT protocol. Step B3: Remote IoT Experiment: The generated control commands are downloaded and transmitted to the corresponding remote experimental devices through the cloud IoT platform, the operation commands are decoded, and the devices are driven to perform the corresponding experimental operations.

9. The remote virtual experiment method according to claim 6, characterized in that, The reverse feedback process involves the following steps: Step C1: Experimental data feedback: The remote laboratory is equipped with acoustic, optical, electrical, and gas signal sensors to read real-time experimental data and transmit it to the cloud server; Step C2: Experimental Actions and Processing: A multi-point binocular stereo camera is installed in the remote laboratory to capture and track the motion equipment, and the motion parameters are transmitted to the near end. Step C3: Experimental Data Processing: Decode the transmitted information, perform image recognition and data analysis through the cloud processor, transmit real-time data and correct / incorrect prompts back to the local end, and update the student's score for this step after analysis.

Images