Data glove interface implementation method, device and equipment for power interaction courseware

Abstract

The application discloses a data glove interface implementation method, device and equipment for power interaction courseware, and the method comprises the following steps: obtaining hand capture degree of freedom data of a VR data glove in a MakeReal3D virtual reality editor, and modeling a virtual hand model based on analysis of the hand capture degree of freedom data; developing a virtual hand real-time driving interface based on the virtual hand model, so as to provide a driving interface for user setting and operation; developing a virtual hand event instruction interaction interface, so as to realize interaction events and responses; and realizing interaction between courseware and the VR data glove based on a courseware interface and the event instruction interaction interface.

Description

Technical Field

[0001] This invention relates to the field of virtual reality technology, and in particular to a method, apparatus, and device for implementing a data glove interface for interactive courseware. Background Technology

[0002] Existing safety production training and education is usually mainly didactic, using videos and PPT presentations to deliver safety production training content. This is not only boring and tedious, but also ineffective. The existing safety production training methods have poor experiential elements. VR virtual simulation education has weak interactivity, and is basically either weakly interactive or non-interactive. After the training is completed, the education fails to deeply resonate with the trainees and play a good warning role.

[0003] Traditional virtual simulation technologies, and even VR-based products, often offer only a single sensory experience, such as vision, hearing, and touch. Virtual simulation sensory projects applied to occupational safety training and education can only provide a virtual simulation of the scene, failing to offer the tactile experience of injuries in real production environments. Traditional VR experiences often require VR controllers for interaction, which is extremely unfriendly, cumbersome, and unrealistic for the occupational safety education industry. Furthermore, traditional safety sensory education equipment is purely sensory, lacking substantial case studies and content, and cannot be integrated with actual production injury cases to create a narrative-driven, interactive experience.

[0004] In response to the above situation, we need a safe haptic production training system based on the interaction of virtual reality and haptic devices. This system can enable communication between virtual scenes and external devices through VR devices, and can judge the response action and complete the relevant experience action through the bending and movement of the hand. Summary of the Invention

[0005] To overcome the shortcomings of the prior art, the present invention provides a method, apparatus and device for implementing a data glove interface for power interactive courseware, in order to solve at least one of the above-mentioned technical problems.

[0006] According to one aspect of the present invention, a method for implementing a data glove interface for power interactive courseware is provided, comprising:

[0007] Obtain the hand capture degrees of freedom data of the VR data glove in the MakeReal3D virtual reality editor, and build a virtual hand model based on the analysis of the hand capture degrees of freedom data;

[0008] A real-time virtual hand driving interface was developed based on a virtual hand model to provide a driving interface for user settings and operations.

[0009] Develop a virtual hand event command interaction interface to implement interactive events and responses;

[0010] Based on the courseware interface and the event command interaction interface, the interaction between the courseware and the VR data glove is realized.

[0011] As a further technical solution, the method also includes: developing a communication interface through the API of the VR driver interface to transmit the hand capture degree of freedom data of the VR data glove and the data collected by the data glove to the MakeReal3D virtual reality editor in real time.

[0012] As a further technical solution, the method also includes: the MakeReal3D virtual reality editor parses the obtained data and converts the parsed data into real-time driving data in the MakeReal3D virtual reality editor.

[0013] As a further technical solution, the method also includes: binding the skeleton of the hand model according to the skeleton structure driven by the MakeReal3D virtual reality editor, and setting the skin weight of the controller of each skeleton in the skeleton according to the original data structure to realize the modeling of the virtual hand model.

[0014] As a further technical solution, the method also includes: when developing the virtual hand real-time drive interface, entering through the virtual hand real-time drive entry point, setting up connection communication, calibrating and setting the real-time hardware data of the data glove and the initial state of the virtual hand, and releasing the connection communication after setting the data.

[0015] As a further technical solution, the method also includes: in a virtual hand model and courseware scenario with real-time driven actions, driving the virtual hand to perform event and response operations in the scenario in real time based on the data glove.

[0016] As a further technical solution, the method also includes: when the virtual hand performs events and response operations in the scene in real time, it follows the process of judging the grasping gesture, interfering with the grasping hand, judging the grasping object, interfering with the grasping object, and judging the association between the grasping gesture and the grasping object.

[0017] According to one aspect of the present invention, an apparatus for implementing an interactive courseware data glove interface is provided, for implementing the method, the apparatus comprising:

[0018] The virtual hand model modeling module is used to obtain the hand capture degree of freedom data of the VR data glove in the MakeReal3D virtual reality editor, and to perform virtual hand model modeling based on the parsing of the hand capture degree of freedom data;

[0019] The virtual hand real-time drive interface module is used to develop a virtual hand real-time drive interface based on the virtual hand model, providing a drive interface for user settings and operations.

[0020] The virtual hand event command interaction interface module is used to develop virtual hand event command interaction interfaces and implement interactive events and responses.

[0021] The courseware and data glove interaction module is used to realize the interaction between the courseware and the VR data glove based on the courseware interface and the event command interaction interface.

[0022] According to one aspect of the present invention, an electronic device is provided, the electronic device including a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein when the computer program is executed by the processor, it implements the steps of the method for implementing the data glove interface for power interactive courseware.

[0023] According to one aspect of the present invention, a computer-readable storage medium is provided, on which a computer program is stored, wherein when the computer program is executed by a processor, it implements the steps of the method for implementing the data glove interface for power interactive courseware.

[0024] Compared with the prior art, the beneficial effects of the present invention are as follows: The present invention is based on the existing virtual reality editor MakeReal3D, combined with courseware data, and developed based on VR data gloves. It proposes a data glove interface for interactive courseware. This interface can support event capture and feedback response of the main hand movement commands, and can simulate the needs of hand interaction in the visible content, such as grasping, releasing, and five-finger movement commands, thereby meeting the needs of safety production training. Attached Figure Description

[0025] Figure 1 This is a flowchart illustrating a method for implementing a data glove interface for interactive courseware according to an embodiment of the present invention.

[0026] Figure 2 This is a flowchart illustrating the development process of a communication interface according to an embodiment of the present invention.

[0027] Figure 3 This is a schematic diagram illustrating the development of a real-time driving interface for a virtual hand according to an embodiment of the present invention.

[0028] Figure 4 This is a flowchart illustrating the grip interaction implementation according to an embodiment of the present invention. Detailed Implementation

[0029] The technical solutions of various embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0030] This invention provides a method for implementing a data glove interface for interactive power courseware, such as... Figure 1 As shown, the method includes:

[0031] Obtain the hand capture degrees of freedom data of the VR data glove in the MakeReal3D virtual reality editor, and build a virtual hand model based on the analysis of the hand capture degrees of freedom data;

[0032] A real-time virtual hand driving interface was developed based on a virtual hand model to provide a driving interface for user settings and operations.

[0033] Develop a virtual hand event command interaction interface to implement interactive events and responses;

[0034] Based on the courseware interface and the event command interaction interface, the interaction between the courseware and the VR data glove is realized.

[0035] The data glove used in this invention is the Noitom Hi5 VR data glove.

[0036] Specifically, the Noitom Hi5 VR hardware can provide motion capture data of the hands and fingers. Through the API of the Noitom Hi5 VR driver interface, communication interface development is carried out, enabling the acquisition of the hand capture degrees of freedom data of the Noitom Hi5 VR data glove in the MakeReal3D virtual reality editor. Based on the analysis of this hand capture degrees of freedom data, a virtual hand model matching the Noitom Hi5 VR data structure is created. Then, through interface development, real-time driving of virtual hand movements is achieved. Next, an interactive response interface is developed according to conventional interactive events and response methods, and the data glove interactive interface is implemented. Finally, through this interactive interface, the virtual hand driven by the data glove movements and the event-triggered response can be quickly provided to enable interaction between courseware and Noitom Hi5 VR.

[0037] Optionally, the method further includes: developing a communication interface through the API of the VR driver interface to transmit the hand capture degree of freedom data of the VR data glove and the data acquired by the data glove to the MakeReal3D virtual reality editor in real time.

[0038] The communication interface, based on the Noitom Hi5 VR data glove hardware driver, transmits the hand freedom data collected and processed by the Noitom Hi5 VR data glove to the MakeReal3D virtual reality editor in real time. The MakeReal3D virtual reality editor needs to parse the freedom data, identify symbols and structures, and acquire response time frames into usable 6-DOF formatted real-time data. Figure 2 As shown, the hardware data of the Noitom Hi5 VR data glove is developed through the driver's API interface to establish a communication interface in MakeReal3D, i.e., real-time communication. The raw data of the Noitom Hi5 VR data glove is obtained through the real-time communication switch, data reception, and real-time storage. Then, the obtained raw data is parsed and developed, and the parsed and developed data is transformed into real-time driver data for the MakeReal3D virtual reality editor.

[0039] Optionally, the method further includes: binding the skeleton of the hand model according to the skeleton structure driven by the MakeReal3D virtual reality editor, and setting the skin weight of the controller of each skeleton in the skeleton according to the original data structure to realize the modeling of the virtual hand model.

[0040] The development of the virtual hand model is mainly based on the data collected by Noitom Hi5 VR communication, which is parsed, formatted, and identified as the real-time data structure for driving. The real-time graphics driver of the data glove needs to be developed by combining the original data with the real-time driver graphics interface of the MakeReal3D editor.

[0041] Based on the skeleton structure of the hand model driven by the MakeReal3D virtual reality editor, the skeleton of the hand model is bound, and the skin weights of the controllers of each skeleton are set according to the original data structure to realize the development of the virtual hand model.

[0042] To illustrate this example, the virtual hand model development process may include: establishing the fingers and the finger coordinate system; setting the size matching of bones and skin; matching all joint bones and skin; matching the distance of the root bones; setting weights based on the driving structure; setting the initial calibration state; and setting the skin texture to complete the virtual hand model.

[0043] Optionally, the method further includes: when developing the virtual hand real-time drive interface, entering through the virtual hand real-time drive entry point, setting up connection communication, calibrating and setting the real-time hardware data of the data glove and the initial state of the virtual hand, and releasing the connection communication after setting the data.

[0044] The virtual hand's real-time driver interface primarily provides a driver interface for user settings and operations. This driver interface typically includes the development of settings such as data connection, data calibration, data configuration, and data shutdown. For example... Figure 3 As shown, the virtual hand real-time drive interface is accessed through the virtual hand real-time drive entry point, setting the connection (communication port), calibration settings (calibration verification is the calibration posture, and the real-time hardware data of the Noitom Hi5 VR data glove is automatically associated with the initial state of the virtual hand according to the fixed calibration posture), data settings (connection type, virtual hand left and right model selection, etc.), and closing settings (releasing connection communication).

[0045] Optionally, the interactive interface involves the virtual hand performing events and responses within the scene, such as grasping, releasing, and five-finger gesture commands. The interactive interface is crucial for enabling interaction with the courseware, and its foundation lies in the virtual hand model with real-time driven actions and the courseware scene.

[0046] The Noitom Hi5 VR data glove's grasping interaction is the process of picking up objects. To quickly implement grasping interaction, it can be broken down into the grasping object (the virtual hand grasping, the object being grasped) and the grasping event response (grasping prompts, grasping binding associations). For example... Figure 4 As shown, the implementation of the grip interaction interface function is based on the NoitomHi5VR data glove driving the hand model in real time. It needs to realize functions such as grip gesture judgment, grip hand interference, grip object judgment, grip object interference, and association and binding of grip gesture and grip object.

[0047] Specifically, grasp gesture judgment: In virtual scenes, grasp gesture judgment is the foundation of grasp interaction. Common grasp interaction algorithms judge grasp gestures by judging the distance from the finger node endpoints to the palm. For example, when the hand is straight, the fingertips are far from the palm, while when the hand is in a grasping state, the fingertips and other finger joints are closer to the palm. In addition, grasp gestures are judged in real time based on the bending angle of the finger joints and the movement of each fingertip.

[0048] Determining the object to be grasped: The object to be grasped is the object grasped by the virtual hand model. Objects in MakeReal3D have various picking attributes, but grasping objects differ from controller picking. Controller picking uses a single ray-bounding box input condition, while determining the object to be grasped is more complex. Its input contact involves multi-point contact interference from the hand and gesture input. Here, further determination is needed using the interface between grasping gestures and grasping interference. This needs to be implemented by setting whether the object can be grasped by the corresponding hand model.

[0049] Interference with the grasping object: Interference with the grasping object can provide visual judgment, i.e., the directionality of the grasping object. Determining the uniqueness and graspability of the grasping object requires prompting through interference. Specifically, this involves inputting parameters for the grasping object's interference. When the interference of the grasping object interferes with the interference of the grasping hand, a visual interference prompt is displayed to indicate that the object can be grasped, thus implementing the grasping object's interference.

[0050] Association determination between grasping gesture and grasping object: After the above grasping process is completed, the association binding of the grasping object is picked up. That is, association binding is to constrain the object to the hand model for following through constraint.

[0051] Release interaction: The release interaction process is the reverse process of the grasp interaction, and its method is the same as the grasp implementation method.

[0052] Five-finger motion command interaction: Following the interface and VR implementation method of hand operation in MakeReal3D, the interaction of five-finger motion commands is mainly based on the interaction input triggered by the fingertip interference of the hand.

[0053] Optionally, after implementing the above interactive development of the Noitom Hi5 VR data glove in MakeReal3D, continue to implement the interface for courseware interaction of the Noitom Hi5 VR data glove.

[0054] First, configure the Noitom Hi5 VR data glove driver. Calibrate the initial hand position according to the standard hand position.

[0055] Next, individual hand fingers are driven. Driving individual hand fingers is the first step in implementing the interface, which involves using the Noitom Hi5 VR data glove in the MakeReal3D virtual reality editor to obtain data and drive the virtual hand model.

[0056] The implementation method is illustrated by the following example:

[0057] In the File Explorer, under ControlSkeleton\Glove, find Glove Character and drag it into the scene;

[0058] Select the two hands model, find the Animation property of the gloves in the Property Manager, and then search for the Anim Class property parameters of LeftHand and RightHand in turn. Open LeftHand and RightHand in the Resource Manager.

[0059] In the opened animation visualization script interface for the left and right hands, set the motion capture connection respectively: select Glove Type as Hi5, uncheck the Read From Config option, check Really Connect, and click Compile and Save; (set the connection and type of the glove driver);

[0060] To set the default player for the gloves: Select the hand in the scene, enter Player in the properties window, find Auto Possess Player under Pawn, and select Player0 as the parameter to complete the Hi5 driver settings for the fingers.

[0061] Next is the setting of the data glove's interactive interface.

[0062] The data glove's interactive interface is designed to provide interactive implementation methods. Here, we take the interactive grasp-and-release action as an example to illustrate the implementation method:

[0063] To create a visual script for the grabbed object: Select the grabbed object, click Visual Script / Add Visual Script in the Properties panel, and then click Create Visual Script in the pop-up panel.

[0064] Set the properties of the object being grabbed: In the panel that pops up after creating the visual script, select StaticMeshComponent, and in the properties, set Simulate Physics (check the box) and Enable Gravity (set the gripping interference implementation);

[0065] Add a grab event to the object to be grabbed: Switch to the Event Graph in the panel, add the Grabable component, add a Set Up event graph to StaticMeshComponent under the Event Graph, and click Compile and Save to complete the setup of the data glove grabbing object (set the interactive properties of the grabbing object).

[0066] Finally, the interactive courseware was implemented. Following the operating method of the MakeReal3D virtual reality editor, project settings and global settings were set to inherit from the virtual hand, and then the courseware was run and interactive functionality was achieved through a 3D menu.

[0067] The present invention also provides a device for implementing an interactive courseware data glove interface, for implementing the method described herein, the device comprising:

[0068] The virtual hand model modeling module is used to obtain the hand capture degree of freedom data of the VR data glove in the MakeReal3D virtual reality editor, and to perform virtual hand model modeling based on the parsing of the hand capture degree of freedom data;

[0069] The virtual hand real-time drive interface module is used to develop a virtual hand real-time drive interface based on the virtual hand model, providing a drive interface for user settings and operations.

[0070] The virtual hand event command interaction interface module is used to develop virtual hand event command interaction interfaces and implement interactive events and responses.

[0071] The courseware and data glove interaction module is used to realize the interaction between the courseware and the VR data glove based on the courseware interface and the event command interaction interface.

[0072] The device is also used to develop a communication interface through the API of the VR driver interface, so as to transmit the hand capture degree of freedom data of the VR data glove and the data collected by the data glove to the MakeReal3D virtual reality editor in real time.

[0073] The device is also used to parse the obtained data using the MakeReal3D virtual reality editor and convert the parsed data into real-time driving data in the MakeReal3D virtual reality editor.

[0074] The virtual hand model modeling module is also used to bind the skeleton of the hand model according to the skeleton structure driven by the MakeReal3D virtual reality editor, and set the skin weight of the controller of each skeleton in the skeleton according to the original data structure to realize the modeling of the virtual hand model.

[0075] The virtual hand real-time drive interface module is also used to enter through the virtual hand real-time drive entry point when developing the virtual hand real-time drive interface, set up connection communication, calibrate and set the real-time hardware data of the data glove and the initial state of the virtual hand, and release the connection communication after setting the data.

[0076] The virtual hand event command interaction interface module is also used to drive the virtual hand to perform event and response operations in the scene based on the data glove in real time in a virtual hand model and courseware scene with real-time driven actions.

[0077] The courseware and data glove interaction module is also used to drive the virtual hand to perform events and response operations in the scene in real time, based on the process of grasping gesture judgment, grasping hand interference, grasping object judgment, grasping object interference, and grasping gesture and grasping object association judgment.

[0078] In one aspect, the present invention also provides an electronic device, which can be an industrial control computer, a server, or a computer terminal.

[0079] The electronic device includes a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein when the computer program is executed by the processor, it implements the steps of the method for implementing the data glove interface for power interactive courseware.

[0080] The electronic device includes a processor, memory, and network interface connected via a system bus, wherein the memory may include non-volatile storage media and internal memory.

[0081] Non-volatile storage media can store operating systems and computer programs. These computer programs include program instructions that, when executed, cause the processor to perform any implementation method for the power interactive courseware data glove interface.

[0082] The processor provides computing and control capabilities to support the operation of the entire electronic device. Internal memory provides an environment for the execution of computer programs stored on non-volatile storage media. When executed by the processor, this program allows the processor to implement any method for the data glove interface of the power interactive courseware.

[0083] This network interface is used for network communication, such as sending assigned tasks.

[0084] It should be understood that the processor can be a Central Processing Unit (CPU), but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among these, a general-purpose processor can be a microprocessor or any conventional processor.

[0085] In one embodiment, the processor is configured to run a computer program stored in memory to perform the following steps:

[0086] Obtain the hand capture degrees of freedom data of the VR data glove in the MakeReal3D virtual reality editor, and build a virtual hand model based on the analysis of the hand capture degrees of freedom data;

[0087] A real-time virtual hand driving interface was developed based on the virtual hand model to provide a driving interface for user settings and operations.

[0088] Develop a virtual hand event command interaction interface to implement interactive events and responses;

[0089] Based on the courseware interface and the event command interaction interface, the interaction between the courseware and the VR data glove is realized.

[0090] In another aspect, the present invention provides a computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, it implements the steps of the method for implementing the data glove interface for power interactive courseware.

[0091] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the technical solutions of the embodiments of the present invention.

Claims

1. A method for implementing a data glove interface for interactive power courseware, characterized in that, include: Obtain the hand capture degrees of freedom data of the VR data glove in the MakeReal3D virtual reality editor, and build a virtual hand model based on the analysis of the hand capture degrees of freedom data; A real-time virtual hand driving interface was developed based on the virtual hand model to provide a driving interface for user settings and operations. Develop a virtual hand event command interaction interface to implement interactive events and responses; Based on the courseware interface and the event command interaction interface, the interaction between the courseware and the VR data glove is realized; The method also includes: developing a communication interface through the API of the VR driver interface to transmit the hand capture degree of freedom data and the data acquired by the VR data glove to the MakeReal3D virtual reality editor in real time; The MakeReal3D virtual reality editor parses the obtained data and converts the parsed data into real-time driving data in the MakeReal3D virtual reality editor. This includes: the hardware data of the data glove is developed through the driver's API interface to establish a communication interface in MakeReal3D, i.e., real-time communication; the raw data of the data glove is obtained through the switching, data reception, and real-time storage of real-time communication; then the obtained raw data is parsed and developed, and the parsed and developed data is converted into real-time driving data for the MakeReal3D virtual reality editor.

2. The method for implementing the data glove interface for interactive power courseware according to claim 1, characterized in that, The method further includes: binding the skeleton of the hand model according to the skeleton structure driven by the MakeReal3D virtual reality editor, and setting the skin weight of the controller of each skeleton according to the original data structure to realize the modeling of the virtual hand model.

3. The method for implementing the data glove interface for interactive power courseware according to claim 1, characterized in that, The method further includes: when developing the virtual hand real-time drive interface, entering through the virtual hand real-time drive entry point, setting up connection communication, calibrating and setting the real-time hardware data of the data glove and the initial state of the virtual hand, and releasing the connection communication after setting the data.

4. The method for implementing the data glove interface for interactive power courseware according to claim 1, characterized in that, The method further includes: in a virtual hand model and courseware scenario with real-time driven actions, driving the virtual hand to perform event and response operations in the scenario in real time based on the data glove.

5. The method for implementing the data glove interface for interactive power courseware according to claim 4, characterized in that, The method further includes: when the virtual hand performs events and response operations in the scene in real time, it follows the process of judging the grasping gesture, interfering with the grasping hand, judging the grasping object, interfering with the grasping object, and judging the association between the grasping gesture and the grasping object.

6. A device for implementing an interactive courseware data glove interface, used to implement the method described in any one of claims 1-5, characterized in that, The device includes: The virtual hand model modeling module is used to obtain the hand capture degree of freedom data of the VR data glove in the MakeReal3D virtual reality editor, and to perform virtual hand model modeling based on the parsing of the hand capture degree of freedom data; The virtual hand real-time drive interface module is used to develop a virtual hand real-time drive interface based on the virtual hand model, providing a drive interface for user settings and operations. The virtual hand event command interaction interface module is used to develop virtual hand event command interaction interfaces and implement interactive events and responses. The courseware and data glove interaction module is used to realize the interaction between the courseware and the VR data glove based on the courseware interface and the event command interaction interface.

7. An electronic device, characterized in that, The electronic device includes a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein when the computer program is executed by the processor, it implements the steps of the method for implementing the data glove interface for power interactive courseware as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, it implements the steps of the method for implementing the data glove interface for power interactive courseware as described in any one of claims 1 to 5.

Images