Virtual interaction method and system based on muscle electrical stimulation

Abstract

The invention belongs to the technical field of gesture virtualization, and particularly discloses a virtual interaction method and system based on muscle electrical stimulation, and the method comprises the steps: S1, or a combination of S1 and S2: in a virtual reality environment, generating a virtual object, and controlling a non-habitual hand to make a gesture to simulate the virtual object by using muscle electrical stimulation; and S2, touching the virtual object with a conventional hand which touches a non-conventional hand simulating the virtual object to obtain tactile feedback of touching the virtual object. By adopting the technical scheme, the control feeling of the user on the hand is reduced by using the muscle electrical stimulation technology, so that the experience of the user on regarding the hand as a virtual object is enhanced.

Description

Technical Field

[0001] The present invention belongs to the technical field of gesture virtualization, and relates to a virtualization interaction method and system based on muscle electrical stimulation. Background Art

[0002] Electrical muscle stimulation (EMS) is a technique that uses electrical pulses to activate muscles through electrodes attached to the skin. As a force feedback and tactile actuation technology, electrical muscle stimulation has attracted widespread attention due to its compact design compared to traditional mechanical actuators (such as exoskeletons). This compactness has promoted the use of non-invasive EMS in replacing traditional mechanical force feedback systems, enabling its use in a wider range of wireless and mobile applications.

[0003] However, the current gesture-driven technology based on muscle electrical stimulation still has the following two problems:

[0004] 1. Lack of clear anatomical guidelines corresponding to muscle electrical stimulation-driven gestures;

[0005] Takahashi et al. have achieved unprecedented levels of hand-actuated dexterity by targeting the interosseous and lumbrical muscles with novel electrode placement on the back of the innovative palm. However, since there are seven interosseous muscles and four lumbrical muscles in the hand, there is still a lack of clear anatomical guidelines to specify the target muscles for each finger. This makes it difficult for researchers and users to correctly place electrodes and flex specific fingers with high selectivity through muscle electrical stimulation.

[0006] 2. The loss of control over the user's body caused by muscle electrical stimulation will affect the sense of immersion;

[0007] Forcing the user to move can cause the user to lose control of their body (i.e., a sense of control over their body), which is particularly detrimental to the user's immersive experience. One of the most common challenges with EMS is loss of control, i.e., a reduced sense of control over the stimulated body part. Various approaches have been explored to address this issue. For example, Tajima et al. developed a framework to balance EMS-driven touch with the user's sense of control. However, few people have considered turning this limitation - loss of control - into an advantage.

[0008] Currently, gesture virtualization technology still has the following three problems:

[0009] 1. Users need to learn these gestures before interacting. It is difficult for them to imagine their hands as virtual objects. They also need to enhance their sense of embodiment, that is, their hands really become virtual objects.

[0010] 2. During long - term interactions, users need extra attention to maintain a specific gesture to keep the gesture virtualized, which leads to fatigue and distraction;

[0011] 3. During the interaction, since gesture recognition is achieved through computer vision algorithms, it severely restricts users from performing interactive actions where both hands overlap visually. Summary of the Invention

[0012] The purpose of the present invention is to provide a virtualization interaction method and system based on muscle electrical stimulation, which uses muscle electrical stimulation technology to reduce the user's sense of control over the hand, thereby enhancing the user's experience of regarding the hand as a virtual object.

[0013] To achieve the above - mentioned purpose, the basic solution of the present invention is: a virtualization interaction method based on muscle electrical stimulation, including step S1, or a combination of S1 and S2:

[0014] S1, in a virtual reality environment, generate a virtual object and use muscle electrical stimulation to control the non - dominant hand to make a gesture to simulate the virtual object;

[0015] S2, use the dominant hand to touch the virtual object, and the dominant hand touches the non - dominant hand that is simulating the virtual object to obtain the tactile feedback of touching the virtual object.

[0016] The working principle and beneficial effects of this basic solution are as follows: This technical solution applies muscle electrical stimulation - driven gestures to a gesture virtualization interaction system, and uses the disadvantage of muscle electrical stimulation - driven gestures, that is, reducing the user's sense of control over the hand, to enhance the immersion of the user's gesture virtualization.

[0017] Muscle electrical stimulation enables the user to passively make a gesture simulating the virtual object and keep the gesture stable, thereby providing the user with a proprioceptive feedback of being unable to control the hand. This proprioceptive feedback is combined with the visual feedback provided by the gesture virtualization technology, that is, the user sees their own hand turn into a virtual object. As a hint, through the dual feedback of vision and proprioception, the user is more convinced that their hand is virtualized.

[0018] Further, in a virtual reality environment, the method of generating a virtual object and using muscle electrical stimulation to control the non - dominant hand to make a gesture to simulate the virtual object is as follows:

[0019] Based on anatomical electrode layout, allocate independent stimulation electrodes on each finger, palm, and back of the hand of the user's non - dominant hand and measure the threshold current;

[0020] The user wears a VR helmet, selects the object they want to obtain in the virtual reality environment, and triggers the corresponding electrical stimulation parameters;

[0021] The electrical stimulator outputs corresponding current stimulation to the electrodes on the user's hand, and the user's hand is driven by the electrical stimulation parameters to make corresponding gestures. The motion capture glove worn on the user's hand collects gesture data in real time to establish a gesture data set;

[0022] Using the gesture data set as input, train a neural network;

[0023] The neural network recognizes the gesture data collected by the motion capture glove and outputs the recognition result to the virtual reality environment, generating corresponding virtual objects and replacing the user's virtual hand for display.

[0024] Electrical muscle stimulation enables the user to passively make gestures simulating virtual objects and keep the gestures stable, thus providing the user with a proprioceptive feedback of being unable to control the hand.

[0025] Furthermore, there is also a step of generating customized virtual objects in the virtual reality environment, specifically:

[0026] Scan the real environment where the user is located, identify the main objects, and reconstruct the virtual environment according to the recognition result. Set a virtual cuboid with the same shape for each recognized object;

[0027] The user enters the virtual environment. The user creates electrical stimulation areas on the virtual cuboids of each real object through the gestures of the dominant hand to store electrical stimulation parameters;

[0028] Set up an object library. The user selects different objects from it and places them on the electrical stimulation area. The area displays the icons of the virtual objects and stores the corresponding electrical stimulation parameters;

[0029] When the user's virtual non-dominant hand touches the electrical stimulation area, the corresponding stimulation parameters will be transmitted to the electrodes tied to the user's non-dominant hand, stimulating the user to make corresponding gestures, and through gesture recognition and the motion capture glove, generating corresponding virtual objects on the non-dominant hand.

[0030] Based on the gesture virtualization technology of the motion capture glove, realize the interaction between the dominant hand and the non-dominant hand.

[0031] Furthermore, the method for establishing a gesture data set and training a neural network is:

[0032] The user wears a motion capture glove and makes 8 specific gestures. Each gesture is sampled at a rate of 1 Hz for 4 minutes. Each gesture collects 240 data samples in total, and each data sample contains the three-dimensional coordinate information of 21 joints of the hand;

[0033] Normalize the joint coordinates of each gesture to ensure the consistency of the data scale between different gestures;

[0034] The collected gesture data is divided into a training set and a test set, with 80% of the data used for training and 20% for testing;

[0035] The neural network is trained using the cross - entropy loss function. The optimizer selects the Adam optimization algorithm. During the training process, an early stopping strategy is adopted to prevent overfitting.

[0036] A gesture dataset is established, the neural network is trained, and the network performance is optimized.

[0037] Furthermore, the method for obtaining the tactile feedback of touching a virtual object by touching the virtual object with the dominant hand and having the dominant hand touch the non - dominant hand that is simulating the virtual object is as follows:

[0038] Tactile feedback electrodes are worn on the dominant hand;

[0039] The virtual hand of the dominant hand is displayed in the virtual environment;

[0040] Control the virtual hand of the dominant hand to touch the virtual object, that is, the dominant hand touches the non - dominant hand that is simulating the virtual object;

[0041] The tactile feedback electrodes on the dominant hand output tactile feedback stimulation signals, and the dominant hand obtains the tactile feedback of touching the virtual object.

[0042] In the tactile feedback, when the user touches the virtual object with the dominant hand, the dominant hand will touch the non - dominant hand that is simulating the virtual object, enhancing the gesture virtualization experience.

[0043] The present invention also provides a virtualization interaction system based on the method of the present invention, including a muscle stimulator, tactile feedback electrodes, a motion capture glove, a VR head - mounted device, and a neural network module;

[0044] The electrodes of the muscle stimulator are arranged on the non - dominant hand of the user, and are used to stimulate the non - dominant hand of the user to make corresponding gestures to simulate the movement of the virtual object;

[0045] The tactile feedback electrodes are arranged on the dominant hand and are used to output tactile feedback stimulation signals to the dominant hand, and the dominant hand obtains the tactile feedback of touching the virtual object;

[0046] The motion capture glove worn on the non - dominant hand of the user collects gesture data in real - time, establishes a gesture dataset and inputs it into the neural network module;

[0047] The neural network module recognizes the gesture data collected by the motion capture glove and outputs the recognition result to the virtual reality environment to generate the corresponding virtual object;

[0048] The VR head - mounted device is worn on the user's head and is used to display the movement of the virtual object in the virtual reality environment.

[0049] This system comprehensively enhances the experience of virtualizing non-dominant hand gestures through visual, proprioceptive, and tactile feedback.

[0050] Furthermore, the electrodes of the muscle electrical stimulator include: an electrode for stimulating the thumb, with a size of 2×3 cm; electrodes for stimulating the index finger and middle finger, with a size of 1×2 cm; an electrode for stimulating the ring finger, with a size of 1×1.5 cm; a gel electrode for stimulating the little finger, with a size of 1×3 cm; and a 5×5 ground electrode placed on the palm.

[0051] The muscle electrical stimulator has a simple structure and is easy to use.

[0052] Furthermore, the neural network module includes three convolutional neural network units, and each convolutional neural network unit contains three one-dimensional convolutional layers, a batch normalization layer, and a tanh activation layer connected in sequence.

[0053] The neural network module can effectively extract the spatial data in the hand joint data, and the tanh activation layer helps the network converge better during the training process.

[0054] Furthermore, the tactile feedback electrode is worn on the fingertips of the user's dominant hand. The tactile feedback electrode includes a central electrode and a surrounding electrode. The central electrode is a stimulating electrode, and the surrounding electrode is an inhibitory electrode. The inhibitory electrode applies a unidirectional wave opposite to the stimulating electrode, and the current amplitude is less than that of the stimulating electrode.

[0055] In this way, on the one hand, it restricts the diffusion of the stimulating current and reduces interference; on the other hand, it cancels the charge accumulation and delays the generation of numbness. Brief Description of the Drawings

[0056] Figure 1 is a schematic flow chart of the virtualization interaction method based on muscle electrical stimulation of the present invention;

[0057] Figure 2 is a schematic diagram of gesture virtualization based on a motion capture glove and a neural network of the virtualization interaction method based on muscle electrical stimulation of the present invention;

[0058] Figure 3 is a schematic structural diagram of the neural network module of the virtualization interaction system of the present invention;

[0059] Figure 4 is a flow chart of gesture virtualization based on a motion capture glove and a neural network of the virtualization interaction method based on muscle electrical stimulation of the present invention;

[0060] Figure 5 is a schematic diagram of muscle electrical stimulation-driven gestures of the virtualization interaction method based on muscle electrical stimulation of the present invention;

[0061] Figure 6It is a schematic flow diagram of the virtualization interaction system of the present invention. Detailed implementation manners

[0062] The embodiments of the present invention will be described in detail below. The examples of the embodiments are shown in the accompanying drawings, where the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by referring to the accompanying drawings are exemplary and are only used to explain the present invention and should not be construed as a limitation to the present invention.

[0063] In the description of the present invention, it should be understood that the orientation or positional relationships indicated by the terms "longitudinal", "transverse", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", etc. are based on the orientation or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and thus should not be construed as a limitation to the present invention.

[0064] In the description of the present invention, unless otherwise specified and limited, it should be noted that the terms "installation", "connection", and "connection" should be understood in a broad sense. For example, it may be a mechanical connection or an electrical connection, or it may be the communication inside two elements. It may be directly connected or indirectly connected through an intermediate medium. For those of ordinary skill in the art, the specific meanings of the above terms can be understood according to specific situations.

[0065] The present invention discloses a virtualization interaction method based on muscle electrical stimulation, which uses muscle electrical stimulation technology to reduce the sense of control of the hand (SoA) of the user, thereby enhancing the user's experience of regarding the hand as a virtual object. As Figure 1 shown, it includes step S1, or a combination of S1 and S2:

[0066] S1, in a virtual reality environment, generate a virtual object and use muscle electrical stimulation to control the non-dominant hand to make gestures to simulate the virtual object; apply the lack of the sense of control brought by muscle electrical stimulation to the field of hand virtualization, and transform the disadvantages of muscle electrical stimulation into the advantages of enhancing the user's immersion in hand virtualization.

[0067] S2, touch the virtual object with the dominant hand, and the dominant hand touches the non-dominant hand that is simulating the virtual object to obtain the tactile feedback of touching the virtual object.

[0068] Apply the muscle electrical stimulation-driven gesture to the gesture virtualization interaction system, and use the disadvantage of the muscle electrical stimulation-driven gesture, that is, reducing the user's sense of control of the hand, to enhance the user's immersion in gesture virtualization.

[0069] Specifically, muscle electrical stimulation enables the user to passively make gestures that simulate virtual objects and keep the gestures stable, thereby providing the user with a proprioceptive feedback of being unable to control the hand. This proprioceptive feedback is combined with the visual feedback provided by the gesture virtualization technology, that is, the user sees their own hand turn into a virtual object, as a cue, allowing the user to be more convinced of the virtualization of their hand through the dual feedback of vision and proprioception.

[0070] In a preferred embodiment of the present invention, as Figure 2 and Figure 4 shown, according to the anatomical functions of the hand muscles, in order to bend the fingers, each finger has a corresponding stimulating muscle. In order to achieve the interaction between the dominant hand and the non-dominant hand, the present invention adopts a gesture virtualization technology based on a motion capture glove. In a virtual reality environment, the method of generating a virtual object and using muscle electrical stimulation to control the non-dominant hand to make a gesture to simulate the virtual object is as follows:

[0071] Based on the anatomical electrode layout, independent stimulating electrodes are distributed on each finger, the palm and the back of the hand of the user's non-dominant hand and the threshold current is measured; by adopting the anatomical electrode layout and allocating independent stimulating electrodes for each finger, it is possible to highly selectively control the bending of each finger, form a variety of gestures, and provide a clear corresponding method for muscle electrical stimulation parameters and gesture driving, as Figure 5 shown.

[0072] The user wears a VR helmet, selects the object they want to obtain in the virtual reality environment, and triggers the corresponding electrical stimulation parameters; the virtual reality scene can be developed using the Unity 3D engine. Unity 3D is a real-time 3D interactive content creation and operation platform, which is widely used in the development of fields such as 3D games, AR, and VR. The programming language used by Unity 3D is C#, which is an object-oriented programming high-level programming language derived from C / C++ released by Microsoft and runs on the.NET Framework and.NET Core platforms, and can provide programming support for the Unity 3D engine.

[0073] The electrical stimulator outputs the corresponding current stimulation to the electrodes on the user's hand, and the user's hand is driven by the electrical stimulation parameters to make corresponding gestures. The motion capture glove worn on the user's hand collects gesture data in real time to establish a gesture data set; for example, if the user's non-dominant hand wears a motion capture glove, the three-dimensional coordinates of its 23 joint points are obtained in real time and input into a deep learning model based on a one-dimensional convolutional neural network to obtain a real-time recognition result. According to the recognition result of the deep learning network, the system generates a corresponding virtual object, and determines the position and rotation direction of the virtual object according to the three-dimensional coordinates of the 23 joint points of the non-dominant hand provided by the motion capture glove.

[0074] Use the gesture dataset as input to train a neural network;

[0075] The neural network recognizes the gesture data collected by the motion capture glove and outputs the recognition result into the virtual reality environment, generating corresponding virtual objects and replacing the user's virtual hands for display. By using data glove and neural network technologies, it can recognize the user's gestures in real time, enabling the user to perform visually overlapping hand interactions. Muscle electrical stimulation drives the user to make gestures and helps the user maintain the gestures. Therefore, the user does not need to learn gestures before interaction and does not need to be distracted to maintain the gestures to obtain objects during interaction.

[0076] In a preferred embodiment of the present invention, the virtualization interaction method based on muscle electrical stimulation further includes the step of generating customized virtual objects in the virtual reality environment, specifically:

[0077] Scan the real environment where the user is located, identify the main objects, and reconstruct the virtual environment according to the recognition result. Set a virtual cuboid with the same shape for each recognized object;

[0078] The user enters the virtual environment. The user creates electrical stimulation areas on the virtual cuboids of each real object through the gestures of the dominant hand to store the electrical stimulation parameters;

[0079] Set up an object library. The user selects different objects from it and places them on the electrical stimulation areas. The icons of the virtual objects are displayed on this area and the corresponding electrical stimulation parameters are stored;

[0080] When the user's virtual non-dominant hand touches the electrical stimulation area, the corresponding stimulation parameters will be transmitted to the electrodes tied to the user's non-dominant hand, stimulating the user to make the corresponding gesture, and through gesture recognition and the motion capture glove, a corresponding virtual object is generated on the non-dominant hand.

[0081] The present invention comprehensively enhances the experience of virtualizing non-dominant hand gestures through virtual objects generated by mixed reality as visual feedback, muscle electrical stimulation-driven gestures as proprioceptive feedback, and the dominant hand touching the non-dominant hand as tactile feedback.

[0082] In a preferred embodiment of the present invention, the method for establishing a gesture dataset and training a neural network is:

[0083] The user wears a motion capture glove and makes 8 specific gestures. Each gesture is sampled at a rate of 1 Hz for 4 minutes. A total of 240 data samples are collected for each gesture. Each data sample contains the three-dimensional coordinate information of 21 joints of the hand;

[0084] Normalize the joint coordinates of each gesture to ensure the consistency of the data scale between different gestures;

[0085] The collected gesture data is divided into a training set and a test set, with 80% of the data used for training and 20% for testing;

[0086] The neural network is trained using the cross - entropy loss function, and the Adam optimization algorithm is selected as the optimizer. During the training process, an early stopping strategy is adopted to prevent overfitting.

[0087] In practical applications, the neural network processes the hand joint data from the Manus motion capture glove in real - time at a frequency of 1 Hz. The processed recognition results are transmitted to the Unity3D engine through the Socket communication protocol. Unity3D generates corresponding virtual objects at the position of the user's virtual non - dominant hand according to the recognition results and replaces the virtual non - dominant hand.

[0088] This interaction method enables users to interact with objects naturally in the virtual environment, enhancing the sense of immersion.

[0089] In a preferred embodiment of the present invention, the method for obtaining the tactile feedback of touching a virtual object by touching the virtual object with the dominant hand and having the dominant hand touch the non - dominant hand that is simulating the virtual object is as follows:

[0090] Tactile feedback electrodes are worn on the dominant hand;

[0091] The virtual hand of the dominant hand is displayed in the virtual environment;

[0092] Control the virtual hand of the dominant hand to touch the virtual object, that is, the dominant hand touches the non - dominant hand that is simulating the virtual object;

[0093] The tactile feedback electrodes on the dominant hand output tactile feedback stimulation signals, and the dominant hand obtains the tactile feedback of touching the virtual object.

[0094] The non - dominant hand is driven by electromyostimulation to make a gesture to simulate a spring. When the user uses the dominant hand to interact with the virtual spring in the virtual reality, in reality, they actually interact with the gesture made by the non - dominant hand, and thus obtain the tactile feedback of the virtual spring by touching their own non - dominant hand.

[0095] The present invention also provides a virtualization interaction system based on the method described in the present invention, as Figure 6 shown, including an electromyostimulator, tactile feedback electrodes, a motion capture glove, a VR head - mounted device, and a neural network module (which can be installed on a portable computer).

[0096] The electrodes of the electromyostimulator are arranged on the non - dominant hand of the user to stimulate the non - dominant hand of the user to make corresponding gestures to simulate the movement of the virtual object.

[0097] The tactile feedback electrode is set on the dominant hand and is used to output tactile feedback stimulation signals to the dominant hand, so that the dominant hand obtains the tactile feedback of touching the virtual object. The motion capture glove worn on the user's non-dominant hand collects gesture data in real time, establishes a gesture data set and inputs it into the neural network module.

[0098] The neural network module recognizes the gesture data collected by the motion capture glove and outputs the recognition result into the virtual reality environment to generate the corresponding virtual object. The VR head-mounted device is worn on the user's head and is used to display the movement of the virtual object in the virtual reality environment.

[0099] This system comprehensively enhances the experience of non-dominant hand gesture virtualization through vision, proprioception and tactile feedback.

[0100] In a preferred embodiment of the present invention, the electrodes of the muscle stimulator include: an electrode for stimulating the thumb, with a size of 2×3 cm; an electrode for stimulating the index finger and middle finger, with a size of 1×2 cm; an electrode for stimulating the ring finger, with a size of 1×1.5 cm; a gel electrode for stimulating the little finger, with a size of 1×3 cm; and a 5×5 ground electrode placed in the palm. Each electrode is only responsible for bending one finger, and the stimulation of each electrode is independent. In addition, the size of the electrode should be slightly adjusted according to the size of the user's hand and muscles.

[0101] By selectively activating five different electrodes, such as only activating the electrodes for bending the thumb and little finger, the user will make a gesture with the thumb and little finger bent and the index finger, middle finger and ring finger straight. The electrical stimulation parameters are: the current amplitude is adjustable within 0-10 mA, the step size is 0.01 mA, the unidirectional pulse width is 220 μs; the frequency is 50 Hz. The electrical stimulation parameters for each finger need to be set separately before using the system.

[0102] Finger: Target stimulated muscle - Anatomical function

[0103] Thumb: Flexor pollicis brevis - Bend the proximal phalanx of the thumb at the metacarpophalangeal joint;

[0104] Index finger: Dorsal interosseous muscle of the first digit - Flex the index finger at the metacarpophalangeal joint;

[0105] Middle finger: Dorsal interosseous muscle of the second digit - Flex the middle finger at the metacarpophalangeal joint;

[0106] Ring finger: Dorsal interosseous muscle of the fourth digit - Flex the ring finger at the metacarpophalangeal joint;

[0107] Little finger: Flexor digiti minimi brevis - Bend the proximal phalanx of the fifth finger at the metacarpophalangeal joint.

[0108] In a preferred embodiment of the present invention, such as Figure 3As shown, the neural network module includes three convolutional neural network units, and each convolutional neural network unit contains three one-dimensional convolutional layers, a batch normalization layer (Batch Normalization), and a tanh activation layer connected in sequence. This design can effectively extract the spatial features in the hand joint data, accelerate the training process through the batch normalization layer, and at the same time, the tanh activation function can help the network converge better during the training process.

[0109] In a preferred embodiment of the present invention, the tactile feedback electrode is worn on the fingertip of the user's dominant hand. The tactile feedback electrode includes a central electrode and a surrounding electrode. The central electrode is a stimulating electrode, and the surrounding electrode is an inhibitory electrode. The inhibitory electrode applies a unidirectional wave opposite to that of the stimulating electrode, and the current amplitude is smaller than that of the stimulating electrode. Preferably, the current ratio of the inhibitory electrode to the stimulating electrode is 1:4.

[0110] On the one hand, this restricts the diffusion of the stimulating current and reduces interference; on the other hand, it cancels the charge accumulation and delays the generation of numbness.

[0111] The present invention converts the disadvantage of the reduced sense of control brought by muscle electrical stimulation into enhancing the immersive feeling of the user's hand virtualization, provides a clear electrode placement guide based on anatomy, provides an obvious contrast relationship for the electrical stimulation parameters and the target-driven gestures, and is convenient for more users and researchers to reproduce the gesture-driven effect. By driving different gestures of the user through muscle electrical stimulation, the user can obtain different objects without having to learn different gestures corresponding to different objects, reducing the learning burden of the user.

[0112] During long-term interaction, muscle electrical stimulation can maintain the gestures driven by the user, reducing the fatigue and distraction of the user for maintaining the gestures; the gesture recognition method based on the motion capture glove and the neural network enables the user to perform interactive actions with overlapping hands, allowing the user to interact with the virtual non-dominant hand with the dominant hand and feel the self-tactile feedback.

[0113] In the description of this specification, the description with reference to terms such as "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in a suitable manner in any one or more embodiments or examples.

[0114] Although embodiments of the present invention have been shown and described, those of ordinary skill in the art can understand that various changes, modifications, substitutions, and variations can be made to these embodiments without departing from the principles and spirit of the present invention. The scope of the present invention is defined by the claims and their equivalents.

Claims

1. A virtual interaction method based on muscle electrical stimulation, characterized in that: The method comprises steps S1, or a combination of S1 and S2: S1, in a virtual reality environment, generates virtual objects and uses muscle electrical stimulation to control the non-dominant hand to make gestures to simulate the virtual objects; S2, touching the virtual object with the dominant hand, the dominant hand touches the non-dominant hand that is simulating the virtual object, and obtains tactile feedback of touching the virtual object.

2. The virtual interaction method based on muscle electrical stimulation according to claim 1, characterized in that: In a virtual reality environment, a method of generating a virtual object and using muscle electrical stimulation to control a non-dominant hand to perform gestures to simulate the virtual object is as follows: Based on the anatomical electrode layout, separate stimulation electrodes are assigned on each finger, palm, and back of the user's non-dominant hand and threshold current is measured; The user wears a VR helmet and selects the object he wants to obtain in the virtual reality environment, triggering the corresponding electrical stimulation parameters; The electrical stimulator outputs the corresponding current to stimulate the electrodes on the user's hands. The user's hands are driven by the electrical stimulation parameters to make corresponding gestures. The motion capture gloves worn by the user's hands collect gesture data in real time to establish a gesture data set. Take the gesture dataset as input and train the neural network; The neural network recognizes the gesture data collected by the motion capture gloves and outputs the recognition results to the virtual reality environment, generates corresponding virtual objects and replaces the user's virtual hands for display.

3. The method for simulated interaction based on muscle electrical stimulation as claimed in claim 2, characterized in that: The method also includes the steps of generating customized virtual objects in a virtual reality environment, specifically: Scan the user's real environment, identify the main objects, and reconstruct the virtual environment based on the recognition results, setting a virtual cuboid with the same shape for each recognized object; The user enters the virtual environment, and the user creates an electrical stimulation area on a virtual cuboid of each real object through gestures of the dominant hand, which is used to store electrical stimulation parameters; Setting up an object library from which users can select different objects and place them on the electrical stimulation area, which displays icons of virtual objects and stores corresponding electrical stimulation parameters; When the user's virtual non-dominant hand touches the electrical stimulation area, the corresponding stimulation parameters will be transmitted to the electrodes tied to the user's non-dominant hand, stimulating the user to make corresponding gestures, and through gesture recognition and motion capture gloves, the corresponding virtual objects will be generated on the non-dominant hand.

4. The virtual interaction method based on muscle electrical stimulation according to claim 2, characterized in that: The method to establish a gesture data set and train a neural network is as follows: The user wears motion capture gloves and makes 8 specific gestures. Data is collected at a sampling rate of 1Hz for each gesture for 4 minutes. A total of 240 data samples are collected for each gesture. Each data sample contains the 3D coordinate information of 21 joints of the hand. Normalize the joint coordinates of each gesture to ensure that the data scales of different gestures are consistent; The collected gesture data is divided into a training set and a test set, with 80% of the data used for training and 20% for testing; The neural network is trained using the cross entropy loss function, and the optimizer selects the Adam optimization algorithm. During the training process, the early stopping strategy is adopted to prevent overfitting.

5. The virtual interaction method based on muscle electrical stimulation according to claim 1, characterized in that: The method of touching a virtual object with the dominant hand and the dominant hand touching the non-dominant hand that is simulating the virtual object to obtain tactile feedback of touching the virtual object is as follows: Wearing tactile feedback electrodes on the dominant hand; Displaying a virtual hand of the dominant hand in a virtual environment; Control the dominant virtual hand to touch the virtual object, that is, the dominant hand touches the non-dominant hand that is simulating the virtual object; The tactile feedback electrode on the dominant hand outputs a tactile feedback stimulation signal, and the dominant hand obtains tactile feedback of touching the virtual object.

6. A virtualized interactive system based on the method according to any one of claims 1 to 5, characterized in that: Including muscle electrical stimulators, tactile feedback electrodes, motion capture gloves, VR headsets, and neural network modules; The electrodes of the muscle electrical stimulator are arranged on the user's non-dominant hand, and are used to stimulate the user's non-dominant hand to make corresponding gestures, simulating the movement of the virtual object; The tactile feedback electrode is arranged on the dominant hand, and is used to output a tactile feedback stimulation signal to the dominant hand, so that the dominant hand obtains tactile feedback of touching the virtual object; The motion capture glove worn by the user's non-dominant hand collects gesture data in real time, builds a gesture data set and inputs it into the neural network module; The neural network module recognizes the gesture data collected by the motion capture gloves and outputs the recognition results to the virtual reality environment to generate corresponding virtual objects; The VR head mounted device is worn on the user's head and is used to display the movement of virtual objects in a virtual reality environment.

7. The virtualization interaction system according to claim 6, characterized in that: It also includes a tactile feedback electrode, which is arranged on the dominant hand and is used to output a tactile feedback stimulation signal to the dominant hand, so that the dominant hand obtains tactile feedback of touching the virtual object.

8. The virtualization interaction system according to claim 6, characterized in that: The electrodes of the muscle electrical stimulator include: an electrode for stimulating the thumb, with a size of 2×3 cm; an electrode for stimulating the index finger and middle finger, with a size of 1×2 cm; an electrode for stimulating the ring finger, with a size of 1×1.5 cm; a gel electrode for stimulating the little finger, with a size of 1×3 cm; and a 5×5 ground electrode placed on the palm.

9. The virtualization interaction system according to claim 6, characterized in that: The neural network module includes three convolutional neural network units, each of which includes three one-dimensional convolutional layers, a batch normalization layer and a tanh activation layer connected in sequence.

10. The virtualization interaction system according to claim 7, characterized in that: The tactile feedback electrode is worn on the user's dominant fingertip, and includes a central electrode and a surrounding electrode. The central electrode is a stimulating electrode, and the surrounding electrode is an inhibitory electrode. The inhibitory electrode applies a unidirectional wave opposite to the stimulating electrode, and the current amplitude is smaller than the stimulating electrode.

Images