Design method of VR electrical and electronic experiment system based on Leap Motion gesture recognition

Abstract

The present invention relates to a VR electrical and electronic experiment system design method, a VR electrical and electronic experiment system design method based on Leap Motion gesture recognition, comprising the following steps: (1) system hardware selection (2) system 3D virtual model modeling (3) system engine development (4) system algorithm development (5) system multi-person online construction. Compared with the existing virtual simulation experiment technology, the present invention adopts the latest VR3D display technology, with high restoration, more realistic experimental scenes, and strong visual impact; provides a gesture interaction method to replace the handle interaction method, and the experimental interaction is natural and accurate; adopts a multi-person online system, and multiple people conduct real-time online experiments, which is more interactive.

Description

Technical Field

[0001] The present invention relates to a VR electrical and electronic experiment system design method, and more specifically, to a VR electrical and electronic experiment system design method based on Leap Motion gesture recognition. Background Art

[0002] Virtual reality is a 3D virtual simulation technology that uses computers, sensors and various auxiliary equipment to create a virtual world to reproduce the real world. The most typical feature of virtual reality is immersion, which acts on users through vision, hearing, touch, etc., making them feel as if they are in the scene. The cost and difficulty of using virtual reality technology to create realistic virtual scenes and environments are often much lower than building the real world. Therefore, it has broad application prospects in many fields such as military training, experimental teaching, industrial design, architectural display, medical anatomy and virtual games. As an important part of human-computer interaction, the research and development of gesture recognition technology also affects the naturalness and flexibility of VR interaction.

[0003] Immersive virtual simulation experiments are innovative experimental modes developed based on virtual reality technology and human-computer interaction technology. Experimenters learn and train in this virtual experimental environment, just like in a real environment, and perform experimental operations according to the set steps. This experimental mode achieves far better results than the real environment at the same experimental cost. In higher education teaching experiments, the country vigorously advocates teaching reform, and a large number of cheap and convenient new virtual simulation experimental teaching methods have emerged. However, most of these virtual simulation experimental methods are difficult to meet the requirements of the existing teaching experimental system due to low restoration, insufficient sense of reality, low immersion, and poor interactivity. At the same time, the original teaching experimental equipment is expensive, and the rapid replacement of student equipment will cause a serious economic burden on the laboratory. Therefore, in order to improve the current level of university teaching experiments, it is urgent to develop an economical, highly simulated, and interactive virtual simulation experimental platform. Summary of the invention

[0004] In order to overcome the deficiencies in the prior art, the purpose of the present invention is to provide a VR electrical and electronic experiment system design method based on LeapMotion gesture recognition. From the perspective of human-computer interaction, the present invention is based on virtual reality technology and gesture recognition technology, based on clear teaching and education goals, combined with actual application scenarios, and uses MAYA modeling software and next-generation game engine UE4 to create a VR electrical and electronic experiment system based on gesture recognition. The system includes hardware selection, 3D virtual model modeling, engine development, gesture recognition algorithm development, and multi-person online mechanism, etc., and has good three-dimensional interactivity, operability, functional perfection, and scene realistic interaction and real characteristics. The experimenter only needs a VR device to break the limitation of space, remotely log in to the experimental system, select the corresponding experimental table to carry out relevant virtual experiments, the experimental environment is realistic, and human-computer interaction is carried out through gesture recognition. The experimental operation is real, and it can also participate in the experiment with other online experimenters, communicate and learn from each other, and have a strong sense of interaction, perfectly reproducing the real experimental scene.

[0005] In order to achieve the above-mentioned invention purpose and solve the problems existing in the prior art, the technical solution adopted by the present invention is: a VR electrical and electronic experiment system design method based on Leap Motion gesture recognition, comprising the following steps:

[0006] Step 1: System hardware selection, including the following two parts:

[0007] (a) VR head display device, HTC Vive is selected, a complete set of VR head display equipment, including a helmet, two handles and two locators. The handles and locators are wireless. The helmet is connected to the computer through the HDMI interface. HTC Vive uses LightingHouse indoor positioning technology. The two rows of LED lights in the locator emit scanning beams 6 times per second, scanning the 15×15 foot positioning space in the horizontal and vertical directions respectively. The HTC VIVE helmet and handles are equipped with more than 70 photosensitive sensors. When the beam scans, the helmet starts counting. After the sensor receives the scanning beam, it uses the relationship between the sensor position and the laser reception time to calculate the exact position relative to the locator;

[0008] (b) Gesture recognition sensor, which is a gesture recognition sensor released by Leap for PC and Mac. Similar to Kinect, it can recognize the somatosensory movements of the hand and form an interactive 3D space. Whether it is subtle movements of the fingers or large movements of the arms, it can accurately detect;

[0009] Step 2: Build a 3D virtual model of the system. The modeling software is MAYA, which includes the following sub-steps:

[0010] (a) 3D virtual laboratory model building: investigate the structure and length, width and height ratio of the real laboratory, record the structural data of each part according to the investigated structural characteristics, draw a 3D layout, and use MAYA software to create a 3D virtual laboratory model according to the layout, restoring the real laboratory environment 1:1;

[0011] (b) 3D virtual laboratory bench modeling: research the structure of the real electrical and electronic laboratory bench, which usually includes the laboratory bench, instruments, jacks, voltage regulators, ballasts, capacitors, resistors, power supplies, indicator lights, switches, light bulbs and wires. According to the investigated structural characteristics, record the structural data of each part, draw a 3D layout, and use MAYA software to create a 3D virtual laboratory bench model according to the layout, restoring the real laboratory bench 1:1;

[0012] Step 3: System engine development. Select UE4 as the VR development engine, which includes the following sub-steps:

[0013] (a) Scene construction: Place the created 3D virtual laboratory model and 3D virtual laboratory bench model in the level scene of the UE4 engine, and arrange the virtual laboratory scene with reference to the real laboratory environment;

[0014] (b) Material creation: select each 3D virtual model in the virtual laboratory scene, make and assign corresponding material balls, so that each 3D virtual model shows corresponding accurate and realistic art effects;

[0015] (c) Post-processing: In the created virtual laboratory scene, add various lighting, fog effects and post-processing boxes, adjust the corresponding parameters, and create a real space atmosphere;

[0016] (d) Bake lighting, select and adjust the lightmap resolution for each 3D virtual model, select production-level rendering, build lighting, and produce realistic light and shadow effects;

[0017] (e) Blueprint script, using the blueprint editor / C++ programming language to create the relevant logic of the 3D virtual laboratory bench experiment, including experimental circuit connection logic, experimental effect triggering logic, experimental instrument measurement line connection logic, and experimental instrument measurement effect triggering and display logic;

[0018] Step 4: System algorithm development, mainly gesture recognition algorithm development, which includes the following sub-steps:

[0019] (a) Collection and feature calculation of hand somatosensory data. The Leap Motion sensor is used to track and collect dynamic hand somatosensory data, and the data is feature calculated and preprocessed as training samples for the gesture recognition algorithm model. The hand data information contained in the data frame collected by Leap Motion includes palm position, palm normal vector, palm direction, grasping force, fingertip position, finger length and radius of the hand-held sphere. The above collected hand data are selected to calculate various gesture features, namely palm displacement, gesture speed and finger change characteristics. Palm displacement represents the Euclidean distance between the palm coordinates of the target hand at two moments, which is described by formula (1).

[0020]

[0021] Where P represents the palm displacement, C1(x i ,y i , z i ) represents the spatial coordinates of the palm at time i, C2(x j ,y j , z j ) represents the spatial coordinates of the palm at time j;

[0022] The gesture speed is the speed of the gesture movement. Here, the frame rate is taken as the gesture speed and described by formula (2):

[0023]

[0024] Where V k represents the rate of gesture in the kth frame, C3(x k ,y k , z k ) represents the spatial coordinates of the palm at the kth frame, C4(x k-1 ,y k-1 , z k-1 ) represents the spatial coordinates of the palm at the k-1th frame, and Δt represents the frame time difference;

[0025] The finger change feature is described by the fingertip distance feature, fingertip angle feature and fingertip height feature. The fingertip distance feature refers to the distance between the fingertip and the palm, which is described by formula (3):

[0026]

[0027] Among them, D i Indicates fingertip distance, F i represents the coordinates of the fingertip position, i represents the finger number, C represents the coordinates of the palm position, S represents the radius of the sphere held by the hand,

[0028] The fingertip angle feature refers to the angle between the palm and fingertip projection vector and the hand direction vector, which is described by formula (4):

[0029]

[0030] Among them, A i represents the fingertip angle, ∠ represents the angle symbol, and F i represents the fingertip position coordinates, Indicates F i Projection on the hand plane, C represents the palm position coordinates, h represents the hand direction vector, and i represents the finger number;

[0031] The fingertip height feature refers to the fingertip position coordinate F i The distance from the palm plane is described by formula (5),

[0032]

[0033] Among them, E i Indicates fingertip height, F i represents the fingertip position coordinates, Indicates F i The projection on the hand plane, n represents the normal vector of the palm plane, S represents the radius of the sphere held by the hand, i represents the finger number, and sgn(·) represents the sign function; the gesture features calculated above are combined to create different gesture feature sets, that is, to train the corresponding gestures, which are used as training samples of the gesture recognition algorithm model after preprocessing. Different gesture feature sets are used to train different gestures;

[0034] (b) Training of gesture recognition algorithm model. The gesture recognition algorithm model adopts hidden Markov model (HMM), which is a statistical model with Markov properties. A complete HMM consists of five parameters (π, A, B, S, V), where π represents the initial distribution probability of the hidden state, A represents the hidden state transfer matrix, B represents the probability matrix of the observed state, S represents the hidden state set, and V represents the observed state set. In the process of using the HMM model, three problems generally need to be solved, namely learning problem, probability calculation problem and prediction problem. The model training part mainly solves the first two problems, namely learning problem and probability calculation problem. The learning problem is manifested as the condition that the observed state sequence O is known. Under the condition, the model parameter λ that maximizes P(O|λ) is obtained through repeated training and iteration. Here, the Baum-Welch algorithm is used to solve the optimal model parameter. The model parameter λ=(π, A, B) of the model is calculated and updated through repeated iterations until the optimal model parameter λ is obtained; the probability calculation problem is expressed as calculating the output probability P(O|λ) of all models under the condition of known parameters λ=(π, A, B, S, V) and observation state sequence O. Here, the forward algorithm is used to solve the probability calculation problem. By extracting features from the unknown gesture observation sequence O, and then calculating the matching probability of the feature vector and each HMM, the gesture model with the largest matching probability P(O|λ) is selected as the recognition result;

[0035] (c) Gesture recognition algorithm model detection. The model detection part mainly solves the prediction problem among the three problems of the HMM model. The prediction problem is to find the hidden state Q with the highest matching degree with the observed state sequence O under the condition of known parameters λ=(π, A, B, S, V) and the observed state sequence O;

[0036] (d) Importing the gesture recognition algorithm model: importing the trained HMM model into the VR electrical and electronic experiment system, replacing the VR handle as a new human-computer interaction method, and combining the blueprint / C++ script to give each trained gesture a corresponding operation function, namely grabbing, sliding, zooming in, and zooming out, and triggering the corresponding gesture events to realize gesture interaction;

[0037] Step 5: Multi-person online system setup, including the following sub-steps:

[0038] (a) Create a multiplayer online UI interface. Create a new scene in UE4 and create a multiplayer online UI interface in it, which contains three main UI buttons, namely, Create Lab, Search Lab, and Join Lab;

[0039] (b) Create multiplayer online script logic, design corresponding logic functions for corresponding UI buttons through Blueprint Editor / C++ script, click Create Lab to execute CreateSession to create a new multiplayer online virtual laboratory, that is, create a new virtual laboratory scene; click Search Lab to execute FindSessions to view the currently created multiplayer online virtual laboratory scenes; click Join Lab to execute JoinSession to select the searched multiplayer online virtual laboratory to join.

[0040] The beneficial effect of the present invention is: a method for designing a VR electrical and electronic experiment system based on Leap Motion gesture recognition, comprising the following steps: (1) system hardware selection (2) system 3D virtual model building (3) system engine development (4) system algorithm development (5) system multi-person online construction. Compared with the existing virtual simulation experiment technology, the present invention adopts the latest VR3D display technology, with higher restoration, more realistic experimental scenes, and stronger visual impact; provides a gesture interaction method to replace the handle interaction method, and the experimental interaction is more natural and more accurate; adopts a multi-person online system, and multiple people conduct real-time online experiments, which is more interactive. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 It is a flow chart of the steps of the method of the present invention.

[0042] Figure 2 It is a flow chart of the algorithm of the system of the present invention. DETAILED DESCRIPTION

[0043] The present invention will be further described below in conjunction with the accompanying drawings.

[0044] like Figure 1 As shown, a VR electrical and electronic experiment system design method based on Leap Motion gesture recognition includes the following steps:

[0045] Step 1: System hardware selection, including the following two parts:

[0046] (a) VR head display device, HTC Vive is selected, a complete set of VR head display equipment, including a helmet, two handles and two locators. The handles and locators are wireless. The helmet is connected to the computer through the HDMI interface. HTC Vive uses LightingHouse indoor positioning technology. The two rows of LED lights in the locator emit scanning beams 6 times per second, scanning the 15×15 foot positioning space in the horizontal and vertical directions respectively. The HTC VIVE helmet and handles are equipped with more than 70 photosensitive sensors. When the beam scans, the helmet starts counting. After the sensor receives the scanning beam, it uses the relationship between the sensor position and the laser reception time to calculate the exact position relative to the locator;

[0047] (b) Gesture recognition sensor, which is a gesture recognition sensor released by Leap for PC and Mac. Similar to Kinect, it can recognize the somatosensory movements of the hand and form an interactive 3D space. Whether it is subtle movements of the fingers or large movements of the arms, it can accurately detect;

[0048] Step 2: Build a 3D virtual model of the system. The modeling software is MAYA, which includes the following sub-steps:

[0049] (a) 3D virtual laboratory model building: investigate the structure and length, width and height ratio of the real laboratory, record the structural data of each part according to the investigated structural characteristics, draw a 3D layout, and use MAYA software to create a 3D virtual laboratory model according to the layout, restoring the real laboratory environment 1:1;

[0050] (b) 3D virtual laboratory bench modeling: research the structure of the real electrical and electronic laboratory bench, which usually includes the laboratory bench, instruments, jacks, voltage regulators, ballasts, capacitors, resistors, power supplies, indicator lights, switches, light bulbs and wires. According to the investigated structural characteristics, record the structural data of each part, draw a 3D layout, and use MAYA software to create a 3D virtual laboratory bench model according to the layout, restoring the real laboratory bench 1:1;

[0051] Step 3: System engine development. Select UE4 as the VR development engine, which includes the following sub-steps:

[0052] (a) Scene construction: Place the created 3D virtual laboratory model and 3D virtual laboratory bench model in the level scene of the UE4 engine, and arrange the virtual laboratory scene with reference to the real laboratory environment;

[0053] (b) Material creation: select each 3D virtual model in the virtual laboratory scene, make and assign corresponding material balls, so that each 3D virtual model shows corresponding accurate and realistic art effects;

[0054] (c) Post-processing: In the created virtual laboratory scene, add various lighting, fog effects and post-processing boxes, adjust the corresponding parameters, and create a real space atmosphere;

[0055] (d) Bake lighting, select and adjust the lightmap resolution for each 3D virtual model, select production-level rendering, build lighting, and produce realistic light and shadow effects;

[0056] (e) Blueprint script, using the blueprint editor / C++ programming language to create the relevant logic of the 3D virtual laboratory bench experiment, including experimental circuit connection logic, experimental effect triggering logic, experimental instrument measurement line connection logic, and experimental instrument measurement effect triggering and display logic;

[0057] Step 4: System algorithm development, mainly gesture recognition algorithm development. The algorithm flow is as follows: Figure 2 As shown, it specifically includes the following sub-steps:

[0058] (a) Collection and feature calculation of hand somatosensory data. The Leap Motion sensor is used to track and collect dynamic hand somatosensory data, and the data is feature calculated and preprocessed as training samples for the gesture recognition algorithm model. The hand data information contained in the data frame collected by Leap Motion includes palm position, palm normal vector, palm direction, grasping force, fingertip position, finger length and radius of the hand-held sphere. The above collected hand data are selected to calculate various gesture features, namely palm displacement, gesture speed and finger change characteristics. Palm displacement represents the Euclidean distance between the palm coordinates of the target hand at two moments, which is described by formula (1).

[0059]

[0060] Where P represents the palm displacement, C1(x i ,y i , z i ) represents the spatial coordinates of the palm at time i, C2(x j ,y j , z j ) represents the spatial coordinates of the palm at time j;

[0061] The gesture speed is the speed of the gesture movement. Here, the frame rate is taken as the gesture speed and described by formula (2):

[0062]

[0063] Where V k represents the rate of gesture in the kth frame, C3(x k ,yk , z k ) represents the spatial coordinates of the palm at the kth frame, C4(x k-1 ,y k-1 , z k-1 ) represents the spatial coordinates of the palm at the k-1th frame, and Δt represents the frame time difference;

[0064] The finger change feature is described by the fingertip distance feature, fingertip angle feature and fingertip height feature. The fingertip distance feature refers to the distance between the fingertip and the palm, which is described by formula (3):

[0065]

[0066] Among them, D i Indicates fingertip distance, F i represents the coordinates of the fingertip position, i represents the finger number, C represents the coordinates of the palm position, S represents the radius of the sphere held by the hand,

[0067] The fingertip angle feature refers to the angle between the palm and fingertip projection vector and the hand direction vector, which is described by formula (4):

[0068]

[0069] Among them, A i represents the fingertip angle, ∠ represents the angle symbol, and F i represents the fingertip position coordinates, Indicates F i Projection on the hand plane, C represents the palm position coordinates, h represents the hand direction vector, and i represents the finger number;

[0070] The fingertip height feature refers to the fingertip position coordinate F i The distance from the palm plane is described by formula (5),

[0071]

[0072] Among them, E i Indicates fingertip height, F i represents the fingertip position coordinates, Indicates F i The projection on the hand plane, n represents the normal vector of the palm plane, S represents the radius of the sphere held by the hand, i represents the finger number, and sgn(·) represents the sign function; the gesture features calculated above are combined to create different gesture feature sets, that is, to train the corresponding gestures, which are used as training samples of the gesture recognition algorithm model after preprocessing. Different gesture feature sets are used to train different gestures;

[0073] (b) Training of gesture recognition algorithm model. The gesture recognition algorithm model adopts hidden Markov model (HMM), which is a statistical model with Markov properties. A complete HMM consists of five parameters (π, A, B, S, V), where π represents the initial distribution probability of the hidden state, A represents the hidden state transfer matrix, B represents the probability matrix of the observed state, S represents the hidden state set, and V represents the observed state set. In the process of using the HMM model, three problems generally need to be solved, namely learning problem, probability calculation problem and prediction problem. The model training part mainly solves the first two problems, namely learning problem and probability calculation problem. The learning problem is manifested as the condition that the observed state sequence O is known. Under the condition, the model parameter λ that maximizes P(O|λ) is obtained through repeated training and iteration. Here, the Baum-Welch algorithm is used to solve the optimal model parameter. The model parameter λ=(π, A, B) of the model is calculated and updated through repeated iterations until the optimal model parameter λ is obtained; the probability calculation problem is expressed as calculating the output probability P(O|λ) of all models under the condition of known parameters λ=(π, A, B, S, V) and observation state sequence O. Here, the forward algorithm is used to solve the probability calculation problem. By extracting features from the unknown gesture observation sequence O, and then calculating the matching probability of the feature vector and each HMM, the gesture model with the largest matching probability P(O|λ) is selected as the recognition result;

[0074] (c) Gesture recognition algorithm model detection. The model detection part mainly solves the prediction problem among the three problems of the HMM model. The prediction problem is to find the hidden state Q with the highest matching degree with the observed state sequence O under the condition of known parameters λ=(π, A, B, S, V) and the observed state sequence O;

[0075] (d) Importing the gesture recognition algorithm model: importing the trained HMM model into the VR electrical and electronic experiment system, replacing the VR handle as a new human-computer interaction method, and combining the blueprint / C++ script to give each trained gesture a corresponding operation function, namely grabbing, sliding, zooming in, and zooming out, and triggering the corresponding gesture events to realize gesture interaction;

[0076] Step 5: Multi-person online system setup, including the following sub-steps:

[0077] (a) Create a multiplayer online UI interface. Create a new scene in UE4 and create a multiplayer online UI interface in it, which contains three main UI buttons, namely, Create Lab, Search Lab, and Join Lab;

[0078] (b) Create multiplayer online script logic, design corresponding logic functions for corresponding UI buttons through Blueprint Editor / C++ script, click Create Lab to execute CreateSession to create a new multiplayer online virtual laboratory, that is, create a new virtual laboratory scene; click Search Lab to execute FindSessions to view the currently created multiplayer online virtual laboratory scenes; click Join Lab to execute JoinSession to select the searched multiplayer online virtual laboratory to join.

Claims

1. A VR electrical and electronic experiment system design method based on Leap Motion gesture recognition, Features The following steps are involved: Step 1: System hardware selection, including the following two parts: (a) VR head display device, HTC Vive is selected, a complete set of VR head display equipment, including a helmet, two handles and two locators. The handles and locators are wireless. The helmet is connected to the computer through the HDMI interface. HTC Vive uses LightingHouse indoor positioning technology. The two rows of LED lights in the locator emit scanning beams 6 times per second, scanning the 15×15 foot positioning space in the horizontal and vertical directions respectively. The HTC VIVE helmet and handles are equipped with more than 70 photosensitive sensors. When the beam scans, the helmet starts counting. After the sensor receives the scanning beam, it uses the relationship between the sensor position and the laser reception time to calculate the exact position relative to the locator; (b) Gesture recognition sensor, which is a gesture recognition sensor released by Leap for PC and Mac. Similar to Kinect, it can recognize the somatosensory movements of the hand and form an interactive 3D space. Whether it is subtle movements of the fingers or large movements of the arms, it can accurately detect; Step 2: Build a 3D virtual model of the system. The modeling software is MAYA, which includes the following sub-steps: (a) 3D virtual laboratory model building: investigate the structure and length, width and height ratio of the real laboratory, record the structural data of each part according to the investigated structural characteristics, draw a 3D layout, and use MAYA software to create a 3D virtual laboratory model according to the layout, restoring the real laboratory environment 1:1; (b) 3D virtual laboratory bench modeling, investigating the structure of the real electrical and electronic laboratory bench, including the laboratory bench, instruments, jacks, voltage regulators, ballasts, capacitors, resistors, power supplies, indicator lights, switches, light bulbs and wires, and recording the structural data of each part according to the investigated structural characteristics, drawing a 3D layout, and using MAYA software to create a 3D virtual laboratory bench model according to the layout, restoring the real laboratory bench 1:1; Step 3: System engine development. Select UE4 as the VR development engine, which includes the following sub-steps: (a) Scene construction: Place the created 3D virtual laboratory model and 3D virtual laboratory bench model in the level scene of the UE4 engine, and arrange the virtual laboratory scene with reference to the real laboratory environment; (b) Material creation: select each 3D virtual model in the virtual laboratory scene, make and assign corresponding material balls, so that each 3D virtual model shows corresponding accurate and realistic art effects; (c) Post-processing: In the created virtual laboratory scene, add various lighting, fog effects and post-processing boxes, adjust the corresponding parameters, and create a real space atmosphere; (d) Bake lighting, select and adjust the lightmap resolution for each 3D virtual model, select production-level rendering, build lighting, and produce realistic light and shadow effects; (e) Blueprint script, using the blueprint editor / C++ programming language to create the relevant logic of the 3D virtual laboratory bench experiment, including experimental circuit connection logic, experimental effect triggering logic, experimental instrument measurement line connection logic, and experimental instrument measurement effect triggering and display logic; Step 4: System algorithm development, gesture recognition algorithm development, specifically including the following sub-steps: (a) Collection and feature calculation of hand somatosensory data. The Leap Motion sensor is used to track and collect dynamic hand somatosensory data, and the data is feature calculated and preprocessed as training samples for the gesture recognition algorithm model. The hand data information contained in the data frame collected by Leap Motion includes palm position, palm normal vector, palm direction, grasping force, fingertip position, finger length and radius of the hand-held sphere. The above collected hand data are selected to calculate the palm displacement, gesture speed, finger change characteristics, fingertip angle characteristics and fingertip height characteristics. The palm displacement represents the Euclidean distance between the palm coordinates of the target hand at two moments, which is described by formula (1). Where P represents the palm displacement, C 1 (x i ,y i , z i ) represents the spatial coordinates of the palm at time i, C 2 (x j ,y j , z j ) represents the spatial coordinates of the palm at time j; The gesture speed is the speed of the gesture movement. Here, the frame rate is taken as the gesture speed and described by formula (2): Where V k represents the rate of gesture in the kth frame, C 3 (x k ,y k , z k ) represents the spatial coordinates of the palm at the kth frame, C 4 (x k-1 ,y k-1 , z k-1 ) represents the spatial coordinates of the palm at the k-1th frame, and Δt represents the frame time difference; The finger change feature is described by the fingertip distance feature, fingertip angle feature and fingertip height feature. The fingertip distance feature refers to the distance between the fingertip and the palm, which is described by formula (3): Among them, D i Indicates fingertip distance, F i represents the coordinates of the fingertip position, i represents the finger number, C represents the coordinates of the palm position, S represents the radius of the sphere held by the hand, The fingertip angle feature refers to the angle between the palm and fingertip projection vector and the hand direction vector, which is described by formula (4): Among them, A i Indicates the fingertip angle, Indicates the angle symbol, F i represents the fingertip position coordinates, Indicates F i Projection on the hand plane, C represents the palm position coordinates, h represents the hand direction vector, and i represents the finger number; The fingertip height feature refers to the fingertip position coordinate F i The distance from the palm plane is described by formula (5), Among them, E i Indicates fingertip height, F i represents the fingertip position coordinates, Indicates F i The projection on the hand plane, n represents the normal vector of the palm plane, S represents the radius of the grip sphere, i represents the finger number, and sgn(·) represents the sign function; The gesture features calculated above are combined to create different gesture feature sets, that is, to train the corresponding gestures. After preprocessing, they are used as training samples for the gesture recognition algorithm model. Different gesture feature sets are used to train different gestures. (b) Training of gesture recognition algorithm model. The gesture recognition algorithm model adopts hidden Markov model (HMM), which is a statistical model with Markov properties. A complete HMM consists of five parameters (π, A, B, S, V), where π represents the initial distribution probability of the hidden state, A represents the hidden state transfer matrix, B represents the probability matrix of the observed state, S represents the hidden state set, and V represents the observed state set. In the process of using the HMM model, three problems need to be solved, namely learning problem, probability calculation problem and prediction problem. The model training part solves the first two problems, namely learning problem and probability calculation problem. The learning problem is manifested as the condition that the observed state sequence O is known. , through repeated training and iteration, the model parameter λ that maximizes P(O|λ) is obtained. Here, the Baum-Welch algorithm is used to solve the optimal model parameter. The model parameter λ=(π, A, B) of the model is calculated and updated through repeated iterations until the optimal model parameter λ is obtained; the probability calculation problem is expressed as calculating the output probability P(O|λ) of all models under the conditions of known parameters λ=(π, A, B, S, V) and the observation state sequence O. Here, the forward algorithm is used to solve the probability calculation problem. By extracting features from the unknown gesture observation sequence O, and then calculating the matching probability of the feature vector and each HMM, the gesture model with the largest matching probability P(O|λ) is selected as the recognition result; (c) Gesture recognition algorithm model detection. The model detection part mainly solves the prediction problem among the three problems of the HMM model. The prediction problem is to find the hidden state Q with the highest matching degree with the observed state sequence O under the condition of known parameters λ=(π, A, B, S, V) and the observed state sequence O; (d) Importing the gesture recognition algorithm model: importing the trained HMM model into the VR electrical and electronic experiment system, replacing the VR handle as a new human-computer interaction method, and combining the blueprint / C++ script to give each trained gesture a corresponding operation function, namely grabbing, sliding, zooming in, and zooming out, and triggering the corresponding gesture events to realize gesture interaction; Step 5: Multi-person online system setup, including the following sub-steps: (a) Create a multiplayer online UI interface. Create a new scene in UE4 and create a multiplayer online UI interface in it, which contains three main UI buttons, namely, Create Lab, Search Lab, and Join Lab; (b) Create multiplayer online script logic, design corresponding logic functions for corresponding UI buttons through Blueprint Editor / C++ script, click Create Lab to execute CreateSession to create a new multiplayer online virtual laboratory, that is, create a new virtual laboratory scene; click Search Lab to execute FindSessions to view the currently created multiplayer online virtual laboratory scenes; click Join Lab to execute JoinSession to select the searched multiplayer online virtual laboratory to join.

Images