A virtual remote experiment system and method based on bioelectric sensing

Abstract

The present invention belongs to the field of Internet technology, and discloses a virtual remote experiment system and method based on bioelectric perception. The environmental sensing module captures the real-time video of the laboratory and transmits it to the virtual display module, which is displayed to the user in three dimensions. The user conducts the experiment according to the real-time three-dimensional picture. The electromyographic sensing module collects the hand posture of the user during the action to perform accurate hand modeling, which is wirelessly transmitted to the control machine module after encoding and encryption by the remote information module. The machine device performs the specified action according to the decrypted instruction, and transmits the touch information received by the smart skin back to the electromyographic sensing module to give the user a response. At the same time, the effect information of the experiment is fed back to the virtual display module in real time, and the experiment is evaluated through the target recognition of the remote information module. The present invention realizes early prediction by extracting the characteristics of electromyographic signals to identify gestures, and further improves the efficiency of remote control. It truly realizes "visible and controllable" remote operation.

Description

Technical Field

[0001] The present invention belongs to the field of Internet technology, and in particular relates to a virtual remote experiment system and method based on bioelectric perception. Background Art

[0002] Experimental class is an essential course for students to test their comprehensive abilities. It not only requires students' systematic organizational ability, but also tests their hands-on ability. On the basis of comprehensive assessment of students, higher requirements are also placed on experimental equipment. Traditional laboratories are built on the basis of physical equipment. Students make appointments to use, participate in the operation throughout the process, and wait for the experimental results. Although such a process is real and convenient for observation, it brings with it problems such as long time consumption, the need to arrive at the laboratory in person, and the inability to effectively use experimental resources, which all trouble people.

[0003] In recent years, the rapid development of remote communication technology has made it easier to complete office work and study online, which has triggered our thinking about remote experiments. Students can control the laboratory through a complete remote operating system, anytime and anywhere, and get the same sense of reality as offline. In fact, there is already a function of remotely controlling the start and stop of equipment through software, but it has not yet achieved true immersive operation. The real experience not only helps students master the experiment, but also improves the efficiency of resource space utilization. Summary of the invention

[0004] The purpose of the present invention is to provide a virtual remote experiment method based on bioelectric perception to solve the above-mentioned technical problems.

[0005] In order to solve the above technical problems, the specific technical solutions of a virtual remote experiment system and method based on bioelectric sensing of the present invention are as follows:

[0006] A virtual remote experiment system based on bioelectric perception, including an electromyographic sensing module, a virtual display module, a remote information module, a control machine module, and an environmental sensing module;

[0007] The electromyographic sensing module is used to collect the user's arm electromyographic signals, process and identify them, and output accurate hand posture modeling results in real time;

[0008] The virtual display module is used to display the video screen of the laboratory to the user in three dimensions;

[0009] The telematics module is used for information processing and communication between the virtual end and the real end, and for experimental evaluation;

[0010] The control machine module is used to follow the gesture recognized by electromyography to perform the same action and use the smart skin to obtain the touch parameters of the experimental supplies;

[0011] The environmental sensing module is used to sense the environmental parameters of the laboratory in all directions and provide operation responses and effect feedback.

[0012] Furthermore, the electromyographic sensing module includes an electromyographic signal acquisition device, an inertial measurement unit, an angle sensor, a microprocessor, a tactile generator and a wireless communication module. The electrical signal acquisition device, the inertial measurement unit and the angle sensor are connected to the microprocessor, the microprocessor is connected to the tactile generator, and the wireless communication module is communicatively connected to the microprocessor; the inertial measurement unit is installed at the center of the experimenter's elbow; the angle sensor is installed on the fingers; the tactile generator is installed on the fingertips of both hands of the experimenter; the electromyographic signal acquisition device is used to collect muscle electrical signals; the inertial measurement unit is used to measure the inertia of the moving joints; the angle sensor is used to measure the angular direction of the fingers; the microprocessor is used to process the signals of each module; and the wireless communication module is used to transmit signals.

[0013] Furthermore, the tactile generator includes a micro heater, a vibrating sheet, and a micro humidity generator. Furthermore, the electromyographic signal acquisition device includes an 8-channel surface electrode, a high-pass filter circuit, a differential amplifier, a low-pass filter circuit, and an analog-to-digital converter. The 8-channel surface electrodes are connected to the high-pass filter circuit, the high-pass filter circuit is connected to the differential amplifier, the differential amplifier is connected to the low-pass filter circuit, and the low-pass filter circuit is connected to the analog-to-digital converter. The electromyographic signals of the 8 channels are subjected to wavelet transform to extract preliminary features, and deep features are mined through the autoencoder. The time domain features are obtained through the long short-term memory model, and finally continuous gesture classification is achieved, and accurate prediction is performed in combination with the finger angle sensor data. Furthermore, the virtual display module includes holographic imaging glasses, headphones, and a wireless communication module. Furthermore, the remote information module is a processor and a wireless communication module loaded with control, encryption, and evaluation algorithms.

[0014] Furthermore, the control machine module is at least one set of a robot system device equipped with smart skin, which can sense the touch parameters of the material, hardness, weight and pressure of the experimental tools in real time.

[0015] Furthermore, the environmental sensing module is a system comprising at least one multi-directional binocular stereo camera, a microphone, sensors for measuring parameters of temperature, concentration, and light intensity, and a wireless communication module.

[0016] Furthermore, the working process of the system is as follows: the environmental sensing module captures the real-time video of the laboratory and transmits it to the virtual display module wirelessly, and displays it to the user in three dimensions. The user conducts experiments based on the real-time three-dimensional images. The electromyography sensing module collects the hand posture of the user during the action to perform accurate hand modeling, which is encoded and encrypted by the remote information module and wirelessly transmitted to the control machine module. The machine device performs the specified action according to the decrypted instruction, and transmits the touch information received by the smart skin back to the electromyography sensing module to give the user a response. At the same time, the effect information of the experiment is fed back to the virtual display module in real time, and the experiment is evaluated through target recognition by the remote information module. The present invention also discloses a control method for a virtual remote experiment system based on bioelectric perception, including a forward information transmission process and a reverse perception feedback process. The specific steps are as follows:

[0017] S1: Holographic imaging: The binocular camera in the environmental sensing module is fixed above the manipulator device to capture the depth map of the experimental table, which is transmitted to the virtual display module in real time through wireless communication. The holographic imaging technology is used to perform real-time three-dimensional imaging of the video, which is displayed to the user through holographic glasses. The user performs virtual experimental operations based on the holographic image viewed;

[0018] S2: Hand movement recognition: When offline, the electromyography sensor module obtains the action electrical signal of the arm muscle and uses the wavelet transform to extract multiple features after filtering, denoising, rectification and smoothing by the circuit. A pattern recognition network is built for training. When operating online, the predicted muscle strength E and gesture category G are output. 1 i , G 2 i (i=1~40), at the same time, an inertial measurement unit is installed at the center of the elbow to record the movement direction D, and combined with the angle sensor to obtain the bending angle A of the wrist and ten fingers j (j=0-10), the hand shape is modeled by fuzzy gestures and precise angles, so as to accurately predict gesture movements;

[0019] S3: Remote information transmission: When the virtual end is mapped to the real end, the position coordinate B of the robot elbow is transformed from the direction D through the coordinate system, involving formula (1). During online operation, the instruction m that the virtual end needs to transmit to the real end is in the form of @_t_D_G 1 i _G 2 i _E_A j _#, where @ and # are start and end characters, t is time, G 1 i , G 2 iRepresent the gesture categories of the left and right hands respectively, use the SCP protocol for remote transmission, and encrypt the instructions using the RSA algorithm, as shown in formula (2);

[0020]

[0021] in, is the coordinate B of the robot elbow, is the elbow IMU direction D, represents acceleration and angular velocity respectively, R is the rotation matrix, and T is the translation matrix;

[0022] c=m e mod N (2)

[0023] Among them, c is the encrypted instruction, m is the plaintext instruction before encryption, and the key pair (e, N) is randomly generated;

[0024] S4: Controlled operation of the real end: A dexterous five-finger manipulator is set in front of the experimental table for follow-up experimental operation. The manipulator adopts a six-axis vertical 14-joint structure. After receiving the remote transmission command of the virtual end, it performs continuous trajectory control according to the decrypted robot elbow position B, and performs continuous trajectory control according to the gesture category G and the finger bending angle A. j Decode the hand model and control the posture of the manipulator. Obtain the actual position C through the depth image obtained by the binocular camera. Use the PID controller to adjust according to the error between the current position C and the target position B, as shown in formula (3).

[0025]

[0026] S5: Experimental feedback and evaluation: The feedback includes operation feedback and effect feedback. The operation feedback mechanism uses smart skin to cover the ten ends of the manipulator and adjust the temperature T S The physical quantities of pressure P, humidity H are remotely transmitted to the tactile generator on the virtual end to provide real-time touch feedback to the user. The effect feedback mechanism is to identify the target in the depth video captured by the binocular camera, and feed back the physical quantities transmitted by the gas sensor, electrical signal sensor, etc. to the virtual display module and introduce it into the equipped evaluation system; the evaluation system monitors the operation and experimental effect of each step, and makes a final evaluation of the experiment according to the set scoring mechanism.

[0027] The virtual remote experiment system and method based on bioelectric perception of the present invention have the following advantages:

[0028] 1. Experimenters do not need to rely on offline physical laboratories. Remote experiments not only improve learning efficiency, but also save time and effort and improve space utilization.

[0029] 2. The three-dimensional holographic display provides the experimenter with a real experience as if they were in a real laboratory, giving all-round visual and auditory feedback and creating an immersive experimental atmosphere.

[0030] 3. By extracting the features of electromyographic signals to identify gestures, advance prediction is achieved, further improving remote efficiency.

[0031] 4. Efficient information encryption transmission not only enables rapid response and synchronous experiments, but also improves security.

[0032] 5. The multi-sensor setting adds the function of operation and effect feedback, allowing users to grasp the action response of the laboratory in time and get an immersive operation feel.

[0033] 6. It truly realizes "visible and controllable" remote operation, which greatly facilitates the experimenter's comprehensive control of the laboratory online.

[0034] 7. Using a manipulator to follow the experiment can effectively protect the personal safety of the user and avoid life safety problems caused by experimental errors.

[0035] 8. For some experiments with a long time period, there is no need to wait for a long time. The alarm system can be set up through the monitoring camera to remotely notify the user, which greatly improves time efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 This is a block diagram of the equipment composition of the remote experiment system based on bioelectric signals;

[0037] Figure 2 This is the block diagram of the electromyographic sensing module;

[0038] Figure 3 This is a schematic diagram of the operation of the remote following experiment based on bioelectric signals;

[0039] Figure 4 It is a network diagram of continuous gesture recognition and prediction algorithm based on bioelectric signals;

[0040] Figure 5 It is the performance evaluation and teaching flow chart of the remote experiment system. DETAILED DESCRIPTION

[0041] In order to better understand the purpose, structure and function of the present invention, the following is a further detailed description of a virtual remote experiment system and method based on bioelectric perception of the present invention in conjunction with the accompanying drawings.

[0042] A virtual remote experiment system based on bioelectric perception, the system includes an electromyographic sensing module, a virtual display module, a remote information module, a control machine module, and an environmental sensing module;

[0043] The electromyographic sensing module is used to collect the electromyographic signals of the user's arm, process and identify them, and output accurate hand posture modeling results in real time. The electromyographic sensing module includes an electromyographic signal acquisition device, an inertial measurement unit, an angle sensor, a microprocessor, a touch generator and a wireless communication module. The electrical signal acquisition device, the inertial measurement unit, and the angle sensor are connected to the microprocessor, the microprocessor is connected to the touch generator, and the wireless communication module is connected to the microprocessor for communication. The inertial measurement unit is installed at the center of the experimenter's elbow; the angle sensor is installed on the finger; the touch generator is installed on the fingertips of the experimenter's hands; the touch generator includes a micro heater, a vibrating sheet, and a micro humidity generator. The electromyographic signal acquisition device is used to collect muscle electrical signals; the inertial measurement unit is used to measure the inertia of the moving joint; the angle sensor is used to measure the angle direction of the finger; the microprocessor is used to process the signals of each module; and the wireless communication module is used to transmit signals. The electromyographic signal acquisition device includes 8-channel surface electrodes, high-pass filter circuit, differential amplifier, low-pass filter circuit, and analog-to-digital converter. The 8-channel surface electrodes are connected to the high-pass filter circuit, the high-pass filter circuit is connected to the differential amplifier, the differential amplifier is connected to the low-pass filter circuit, and the low-pass filter circuit is connected to the analog-to-digital converter. The 8-channel electromyographic signal is subjected to wavelet transform to extract preliminary features, and the deep features are mined through the autoencoder. The time domain features are obtained through the long short-term memory (LSTM) model, and finally continuous gesture classification is realized, and accurate prediction is performed in combination with the finger angle sensor data.

[0044] The virtual display module includes holographic imaging glasses, headphones and a wireless communication module; and is used to display the video images of the laboratory to the user in three dimensions.

[0045] The telematics module is a processor and wireless communication module loaded with control, encryption and evaluation algorithms; it is used for information processing and communication between the virtual end and the real end, and for experimental evaluation;

[0046] The control machine module is at least one set of robot system devices equipped with smart skin, which can sense the material, hardness, weight, pressure and other touch parameters of the experimental tools in real time. The control machine module is used to follow the gestures recognized by electromyography to perform the same actions and use the smart skin to obtain the touch parameters of the experimental tools;

[0047] The environmental sensing module is a system that includes at least one set of multi-directional binocular stereo cameras, microphones, various sensors for measuring parameters such as temperature, concentration, light intensity, etc., and wireless communication modules; the environmental sensing module is used to perceive the environmental parameters of the laboratory in all directions and give operation responses and effect feedback.

[0048] The working process of the virtual remote experiment system based on bioelectric perception is as follows: the environmental sensing module captures the real-time video of the laboratory and transmits it to the virtual display module wirelessly, and displays it to the user in three dimensions. The user conducts experiments based on the real-time three-dimensional picture. The electromyography sensing module collects the hand posture of the user during movement to perform accurate hand modeling, which is encoded and encrypted by the remote information module and wirelessly transmitted to the control machine module. The machine device performs the specified action according to the decrypted instruction, and transmits the touch information received by the smart skin back to the electromyography sensing module to give the user a response. At the same time, the effect information of the experiment is fed back to the virtual display module in real time, and the experiment is evaluated through target recognition by the remote information module.

[0049] A virtual remote experiment method based on bioelectric perception includes not only a forward information transmission process, but also a reverse perception feedback process. The specific steps are as follows:

[0050] (1) Holographic imaging: The binocular camera in the environmental sensing module is fixed above the manipulator device to capture the depth map of the experimental platform, which is transmitted to the virtual display module in real time through wireless communication. The video is imaged in real time using holographic imaging technology and displayed to the user through holographic glasses. The user performs virtual experimental operations based on the holographic image viewed;

[0051] (2) Hand movement recognition: When offline, the electromyography sensor module obtains the arm muscle movement electrical signal and filters it through circuits to remove noise, rectify and smooth it. It then uses wavelet transform to extract multiple features and builds a pattern recognition network for training. When online, it outputs the predicted muscle strength E and gesture category G. 1 i , G 2 i (i=1~40), at the same time, an inertial measurement unit (IMU) is installed at the center of the elbow to record the movement direction D, and the bending angle A of the wrist and ten fingers is obtained in combination with the angle sensor j (j=0-10), the hand shape is modeled by fuzzy gestures and precise angles, so as to accurately predict gesture movements;

[0052] (3) Remote information transmission: When the virtual end is mapped to the real end, the position coordinate B of the robot elbow is transformed from the direction D through the coordinate system, involving formula (1). During online operation, the instruction m that the virtual end needs to transmit to the real end is in the form of @_t_D_G 1 i _G 2 i _E_A j _#, (@, # are start and end characters, t is time, G 1 i , G 2 iRepresenting the gesture categories of the left and right hands respectively), the SCP protocol is used for remote transmission, and the instructions are encrypted by the RSA algorithm, as shown in formula (2);

[0053]

[0054] in, is the coordinate B of the robot elbow, is the elbow IMU direction D, represents acceleration and angular velocity respectively, R is the rotation matrix, and T is the translation matrix;

[0055] c=m e mod N (2)

[0056] Among them, c is the encrypted instruction, m is the plaintext instruction before encryption, and the key pair (e, N) is randomly generated;

[0057] (4) Controlled operation of the real end: A dexterous five-finger manipulator is set up in front of the experimental table for follow-up experimental operation. The manipulator adopts a six-axis vertical 14-joint structure. After receiving the remotely transmitted command from the virtual end, it performs continuous trajectory control according to the decrypted robot elbow position B, and performs continuous trajectory control according to the gesture category G and the finger bending angle A. j Decode the hand model and control the posture of the manipulator. Obtain the actual position C through the depth image obtained by the binocular camera. Use the PID controller to adjust according to the error between the current position C and the target position B, as shown in formula (3).

[0058]

[0059] (5) Experimental feedback and evaluation: The feedback includes operation feedback and effect feedback. The operation feedback mechanism uses smart skin to cover the ten ends of the manipulator and adjust the temperature T S The physical quantities of pressure P, humidity H are remotely transmitted to the tactile generator on the virtual end to provide real-time touch feedback to the user. The effect feedback mechanism is to identify the target in the depth video captured by the binocular camera, and combine the physical quantities transmitted by the gas sensor, electrical signal sensor, etc. to feedback to the virtual display module and introduce it into the equipped evaluation system. The evaluation system monitors the operation and experimental results of each step, and makes a final evaluation of the experiment according to the set scoring mechanism.

[0060] Example 1: Basic experiment using the virtual remote experiment system based on bioelectric signals of the present invention

[0061] like Figure 1 As shown, the remote experiment system in this embodiment includes holographic glasses, headphones, electromyographic sensor modules, tactile generators, microprocessors, five wireless communication modules, smart skin, a seven-degree-of-freedom manipulator system, a binocular camera, a microphone, and a multi-directional sensor. Figure 2 and Figure 3 As shown in the figure, the myoelectric signal acquisition device of the electromyographic sensing module is fixed on the upper arm of the experimenter, the tactile generator is fixed on the fingertips of both hands of the experimenter, the microprocessor is placed in the holographic glasses, and the five wireless communication modules are respectively placed on the myoelectric signal acquisition device of both hands, the tactile generator of both hands, the holographic glasses and the robot device in the laboratory. Figure 3 As shown, 1 is an electromyographic sensing module, 2 is an electromyographic collection electrode, 3 is an inertial measurement unit, 4 is an angle sensor, 5 is a touch generator, 6 is a binocular camera, and 7 is a manipulator.

[0062] The built-in binocular camera in the laboratory captures the depth video of the experimental table and transmits it to the holographic glasses in real time. The user performs experimental operations based on the stereoscopic images. The electromyographic signal acquisition device on the arm sends the pre-processed muscle electroelectric signals to the processor of the remote information module for accurate hand posture modeling. The gesture recognition network used is as follows: Figure 4 As shown in the figure, the 8-channel electromyographic signal is subjected to wavelet transform to extract preliminary features, and the deep features are mined through the autoencoder. The time domain features are obtained through the long short-term memory (LSTM) model, and finally continuous gesture classification is realized, and the finger angle sensor data is combined for accurate prediction. In the processor, it is encrypted and wirelessly transmitted to the manipulator, and the manipulator responds according to the converted coordinates and decryption instructions to realize the remote control experiment. After the depth video obtained by the binocular camera is remotely fed back to the microprocessor, the target is recognized frame by frame, and the YOLOv4 (You Only Look Once version4) algorithm is used to identify the needle numbers, reagent colors or the placement of experimental tools, etc., and finally the results are input into the performance evaluation system.

[0063] Example 2: Performance evaluation of a virtual remote experiment system based on bioelectric signals

[0064] This embodiment uses the same remote experiment acquisition device as the first embodiment, in which the microprocessor is used for holographic image generation and instruction encryption generation, information transmission, and evaluation system. Figure 5 As shown, the specific steps to complete the experiment are as follows:

[0065] (1) Equipment Readiness Calibration: The robot returns to its original position. The experimenter makes a classic gesture before the experiment to observe the robot's follow-up sensitivity. When the gesture indicating the start is made, the timer starts the total experiment time.

[0066] (2) Experimental operation: When performing experimental actions, if a fault occurs and the robot fails to respond in time, the yellow light will flash. At this time, the robot system will start the automatic teaching function and repeat the previous operation. If the recognition algorithm detects that the risk factor of the current operation is higher than the threshold, a red light alarm will be triggered and safe operation will be automatically performed;

[0067] (3) Experimental record and judgment: Under the premise of normal operation response and no dangerous operation, the gesture recognition results and binocular camera monitoring results are combined to convert them into a text record experimental report. When the gesture indicating termination is detected, the robot arm returns to its position and the experiment ends.

[0068] (4) Score calculation and evaluation: The images acquired by the binocular camera are used for target detection to extract effect information, including the readings of various meters, objects grasped by the robot, and the color and volume changes produced by the experiment. At the same time, multiple sensors such as sound, temperature, and gas are used to assist in monitoring. The evaluation system automatically gives a comprehensive score after the experiment.

[0069] (5) Experimental correction teaching: If the comprehensive score is not full marks, the evaluation system will automatically call up the error correction mechanism, compare each step with the correct operation in the database, point out the wrong steps and demonstrate the correct operation method, and finally stop the equipment and return it to its original position.

[0070] It is to be understood that the present invention is described by some embodiments, and it is known to those skilled in the art that various changes or equivalent substitutions may be made to these features and embodiments without departing from the spirit and scope of the present invention. In addition, under the teachings of the present invention, these features and embodiments may be modified to adapt to specific circumstances and materials without departing from the spirit and scope of the present invention. Therefore, the present invention is not limited by the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application are within the scope of protection of the present invention.

Claims

1. A control method for a virtual remote experiment system based on bioelectric sensing, wherein the virtual remote experiment system based on bioelectric sensing comprises an electromyographic sensing module, a virtual display module, a remote information module, a control machine module, and an environmental sensing module; The electromyographic sensing module is used to collect the user's arm electromyographic signals, process and identify them, and output accurate hand posture modeling results in real time; The virtual display module is used to display the video screen of the laboratory to the user in three dimensions; The telematics module is used for information processing and communication between the virtual end and the real end, and for experimental evaluation; The control machine module is used to follow the gesture recognized by electromyography to perform the same action and use the smart skin to obtain the touch parameters of the experimental supplies; The environmental sensing module is used to sense the environmental parameters of the laboratory from all directions and give operation responses and effect feedback; It is characterized in that The control method includes a forward information transmission process and a reverse perception feedback process, and the specific steps are as follows: S1: Holographic imaging: The binocular camera in the environmental sensing module is fixed above the manipulator device to capture the depth map of the experimental table, which is transmitted to the virtual display module in real time through wireless communication. The holographic imaging technology is used to perform real-time three-dimensional imaging of the video, which is displayed to the user through holographic glasses. The user performs virtual experimental operations based on the holographic image viewed; S2: Hand movement recognition: When offline, the electromyography sensor module obtains the action electrical signals of the arm muscles and uses the wavelet transform to extract multiple features after filtering, denoising, rectification and smoothing by the circuit. A pattern recognition network is then built for training. When online, the predicted muscle strength E and gesture category G are output. 1 i , G 2 i (i=1~40), at the same time, an inertial measurement unit is installed at the center of the elbow to record the movement direction D, and the bending angle A of the wrist and ten fingers is obtained in combination with the angle sensor j (j=0-10), the hand shape is modeled by fuzzy gestures and precise angles, so as to accurately predict gesture movements; S3: Remote information transmission: When the virtual end is mapped to the real end, the position coordinate B of the robot elbow is transformed from the direction D through the coordinate system, involving formula (1). During online operation, the instruction m that the virtual end needs to transmit to the real end is in the form of @_t_D_G 1 i _G 2 i _E_A j _#, where @ and # are start and end characters, t is time, G 1 i , G 2 i Represent the gesture categories of the left and right hands respectively, use the SCP protocol for remote transmission, and encrypt the instructions using the RSA algorithm, as shown in formula (2); in, is the coordinate B of the robot elbow, is the elbow IMU direction D, represents acceleration and angular velocity respectively, R is the rotation matrix, and T is the translation matrix; c=m e mod N (2) Among them, c is the encrypted instruction, m is the plaintext instruction before encryption, and the key pair (e, N) is randomly generated; S4: Controlled operation of the real end: A dexterous five-finger manipulator is set in front of the experimental table for follow-up experimental operation. The manipulator adopts a six-axis vertical 14-joint structure. After receiving the remote transmission command of the virtual end, it performs continuous trajectory control according to the decrypted robot elbow position B, and performs continuous trajectory control according to the gesture category G and the finger bending angle A. j Decode the hand model and control the posture of the manipulator. Obtain the actual position C through the depth image obtained by the binocular camera. Use the PID controller to adjust according to the error between the current position C and the target position B, as shown in formula (3). S5: Experimental feedback and evaluation: The feedback includes operation feedback and effect feedback. The operation feedback mechanism uses smart skin to cover the ten ends of the manipulator and adjust the temperature T S The physical quantities of pressure P, humidity H are remotely transmitted to the tactile generator on the virtual end to provide real-time touch feedback to the user. The effect feedback mechanism is to identify the target in the depth video captured by the binocular camera, and feed back the physical quantities transmitted by the gas sensor, electrical signal sensor, etc. to the virtual display module and introduce it into the equipped evaluation system; the evaluation system monitors the operation and experimental effect of each step, and makes a final evaluation of the experiment according to the set scoring mechanism.

2. The control method according to claim 1, characterized in that: The electromyographic sensing module includes an electromyographic signal acquisition device, an inertial measurement unit, an angle sensor, a microprocessor, a tactile generator and a wireless communication module. The electrical signal acquisition device, the inertial measurement unit and the angle sensor are connected to the microprocessor, the microprocessor is connected to the tactile generator, and the wireless communication module is communicatively connected to the microprocessor; the inertial measurement unit is installed at the center of the experimenter's elbow; the angle sensor is installed on the fingers; the tactile generator is installed on the fingertips of both hands of the experimenter; the electromyographic signal acquisition device is used to collect muscle electrical signals; the inertial measurement unit is used to measure the inertia of the moving joints; the angle sensor is used to measure the angular direction of the fingers; the microprocessor is used to process the signals of each module; and the wireless communication module is used to transmit signals.

3. The control method according to claim 2, characterized in that: The touch generator comprises a micro heater, a vibration sheet and a micro humidity generator.

4. The control method according to claim 2, characterized in that: The electromyographic signal acquisition device includes 8-channel surface electrodes, a high-pass filter circuit, a differential amplifier, a low-pass filter circuit, and an analog-to-digital converter. The 8-channel surface electrodes are connected to the high-pass filter circuit, the high-pass filter circuit is connected to the differential amplifier, the differential amplifier is connected to the low-pass filter circuit, and the low-pass filter circuit is connected to the analog-to-digital converter. The electromyographic signals of the 8 channels are subjected to wavelet transform to extract preliminary features, and deep features are mined through an autoencoder. Time domain features are obtained through a long short-term memory model, and finally continuous gesture classification is achieved, and accurate prediction is performed in combination with finger angle sensor data.

5. The control method according to claim 1, characterized in that: The virtual display module includes holographic imaging glasses, headphones and a wireless communication module.

6. The control method according to claim 1, characterized in that: The telematics module is a processor and a wireless communication module loaded with control, encryption and evaluation algorithms.

7. The control method according to claim 1, characterized in that: The control machine module is at least one set of a robot system device equipped with smart skin, which can sense the touch parameters of the material, hardness, weight and pressure of the experimental tools in real time.

8. The control method according to claim 1, characterized in that: The environmental sensing module is a system comprising at least one multi-directional binocular stereo camera, a microphone, sensors for measuring parameters such as temperature, concentration, and light intensity, and a wireless communication module.

9. The control method according to claim 1, characterized in that: The working process of the system is as follows: the environmental sensing module captures real-time video of the laboratory and transmits it wirelessly to the virtual display module, and displays it in three dimensions to the user. The user conducts experiments based on the real-time three-dimensional images. The electromyography sensing module collects the hand posture of the user during movement to perform accurate hand modeling, which is encoded and encrypted by the remote information module and wirelessly transmitted to the control machine module. The machine device performs specified actions according to the decrypted instructions, and transmits the touch information received by the smart skin back to the electromyography sensing module to give the user a response. At the same time, the effect information of the experiment is fed back to the virtual display module in real time, and the experiment is evaluated through target recognition by the remote information module.

Images