Method, apparatus and device for implementing a data glove interface for use in interactive courseware on electrical power - Patents.com

Abstract

The present invention discloses a method, apparatus and device for realizing a data glove interface for interactive courseware related to electric power, the method includes: acquiring data of the captured degrees of freedom of a hand of a VR data glove in a MakeReal3D virtual reality editor, modeling a virtual hand model based on the analysis of the captured degrees of freedom of the hand, developing a virtual hand real-time driving interface based on the virtual hand model to provide a driving interface for user settings and operations, developing a virtual hand event-command interaction interface to realize and respond to interactions, and realizing the interaction between the courseware and the VR data glove based on the courseware interface and the event-command interaction interface.

Description

[Technical field]

[0001] The present invention relates to the technical field of virtual reality (VR), and in particular to a method, apparatus and device for implementing a data glove interface for use in interactive courseware on electrical power. [Background technology]

[0002] Traditional safety production training generally focuses on teaching methods, such as playing videos and promoting PPTs, and the training content of safety production is not only boring, but also has poor teaching effect. Traditional safety production training methods have poor experience, and the interaction of VR virtual simulation education is not strong, and it is basically weak or no interaction. After the training is completed, the training for the trainees cannot resonate with people's hearts, and does not play a good warning role.

[0003] Traditional virtual simulation technology and products based on VR devices only provide a single sense of sensation, such as vision, hearing, and touch, and the virtual simulation experience project applied to the training and education of occupational safety production can only provide a simulation in the virtual simulation scene, and cannot provide a tactile experience of injuries caused in a real production environment. Traditional virtual simulation VR experience projects require interaction through a VR operating handle, which gives a very poor, complicated, and unrealistic experience to the occupational safety education industry. Traditional safety experience education devices are simpler experiences, lack real cases and content, and cannot provide a theatrical interactive experience according to injury cases in real production processes. Summary of the Invention [Problem to be solved by the invention]

[0004] In view of the above circumstances, a safe experiential production and training system based on the interaction between virtual reality and a haptic device is needed, in which the virtual scene and an external device can communicate with each other through the VR device, and the system can determine a response action and complete the related experiential action based on the bending and movement of the hand. [Means for solving the problem]

[0005] The present invention provides a method, apparatus and device for implementing a data glove interface for use in interactive courseware on electrical power, in order to solve at least one of the above-mentioned technical problems that should overcome the above-mentioned deficiencies of the prior art.

[0006] One aspect of the present disclosure provides a method for implementing a data glove interface for use in interactive courseware relating to electrical power, the method comprising: acquiring hand capture freedom data of the VR data glove in the MakeReal3D virtual reality editor, and modeling a virtual hand model based on an analysis of the hand capture freedom data; Developing a virtual hand real-time driving interface based on the virtual hand model to provide a driving interface for user setting and operation; Developing a virtual hand event command interaction interface for implementing and responding to interactions; and realizing an interaction between the courseware and the VR data glove based on the courseware interface and the event command interaction interface.

[0007] As a further technical measure, the method further includes developing a communication interface through an API of the VR driving interface to transfer the captured degree of freedom data of the hand of the VR data glove and the collected data of the data glove to the MakeReal3D virtual reality editor in real time.

[0008] As a further technical measure, the method further includes: analyzing the data acquired by the MakeReal3D virtual reality editor, and converting the analyzed data into real-time driving data in the MakeReal3D virtual reality editor.

[0009] As a further technical means, the method further includes: driving the skeletal structure of the hand model in the MakeReal3D virtual reality editor, performing skeletal binding of the hand model, and setting the skin weight of the controller of each skeleton in the skeleton according to the present data structure, thereby realizing modeling of the virtual hand model.

[0010] As a further technical means, the method further includes, when developing a virtual hand real-time driving interface, entering from the real-time driving interface of the virtual hand to set up a connection communication, calibrating the real-time hardware data of the data glove and the initial state of the virtual hand, and releasing the connection communication after setting the data.

[0011] As a further technical means, the method further includes driving the virtual hand in real time based on the data glove in the scene of the courseware and the virtual hand model driving the action in real time, such as the virtual hand responding to events in the scene and operations.

[0012] As a further technical means, the method further includes: based on the processes of grasping gesture judgment, grasping hand interference, grasping object judgment, grasping object interference, and grasping gesture / grasping object association judgment, a virtual hand generates an event in the scene, and the virtual hand is driven in real time to respond to an operation.

[0013] According to one aspect of the present specification, there is provided an apparatus for implementing a data glove interface for use in interactive courseware implementing the above-mentioned method, the apparatus comprising: a virtual hand model modeling module for acquiring data of a captured degree of freedom of a hand of a VR data glove in a MakeReal3D virtual reality editor, and modeling a virtual hand model based on an analysis of the captured degree of freedom of the hand; A virtual hand real-time driving interface module for developing a virtual hand real-time driving interface for providing a driving interface for a user's setting and operation based on the virtual hand model; a virtual hand event command interaction interface module for developing a virtual hand real-time driving interface based on the virtual hand model to provide a driving interface for user setting and operation; The courseware includes an interaction module between the courseware and the data glove, for realizing the interaction between the courseware and the VR data glove according to the courseware interface and the event command interaction interface.

[0014] According to one aspect of the present specification, there is provided an electronic device including a processor, a memory, and a computer program stored in the memory and executable by the processor, the computer program, when executed by the processor, performing steps of a method for implementing a data glove interface for use with the interactive courseware.

[0015] According to one aspect of the present specification, a computer readable storage medium is provided, the computer readable storage medium having stored thereon a computer program which, when executed by a processor, performs method steps for implementing a data glove interface for use with the interactive courseware. Effect of the Invention

[0016] Compared with the prior art, the present invention is based on the conventional MakeReal3D virtual reality editor, and is developed based on the VR data glove in accordance with the courseware data, to provide a data glove interface for interactive courseware, which can support event capture and feedback response of the main hand action commands, and can simulate the needs of hand interaction in the viewing of content, such as grasping, releasing, and five-finger action commands, thus having the beneficial effect of meeting the needs of fostering safe production. [Brief description of the drawings]

[0017] [Figure 1] 4 is a flow chart of a method for implementing a data glove interface for use in interactive courseware on electrical power according to an embodiment of the present invention. [Diagram 2] 4 is a flowchart of a communication interface development according to an embodiment of the present invention. [Diagram 3] FIG. 2 is a schematic diagram of the development of a virtual hand real-time driving interface according to an embodiment of the present invention; [Figure 4] 1 is a flowchart for implementing a gripping interaction according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0018] The following clearly and completely describes the technical solutions of the embodiments of the present invention with reference to the drawings. Obviously, the described embodiments are only some embodiments of the present invention, and do not include all embodiments. Based on the embodiments of the present invention, all other embodiments that can be obtained by those skilled in the art without creative labor are also included in the protection scope of the present invention.

[0019] As shown in FIG. 1 , one aspect of the present invention provides a method for implementing a data glove interface for use in interactive courseware on electrical power, the method comprising: acquiring hand capture freedom data of the VR data glove in the MakeReal3D virtual reality editor, and modeling a virtual hand model based on an analysis of the hand capture freedom data; Developing a virtual hand real-time driving interface based on the virtual hand model to provide a driving interface for user setting and operation; Developing a virtual hand event command interaction interface for implementing and responding to interactions; and realizing an interaction between the courseware and the VR data glove based on the courseware interface and the event command interaction interface.

[0020] The data glove used in the present invention is the Noitom Hi5 VR data glove.

[0021] Specifically, the Noitom Hi5 VR hardware can provide finger motion capture data, and by developing a communication interface through the API of the Noitom Hi5 VR driving interface, the MakeReal3D virtual reality editor can obtain the capture freedom data of the hand of the Noitom Hi5 VR data glove, and based on the analysis of the capture freedom data of the hand, a virtual hand model is modeled that is consistent with the Noitom Hi5 VR data structure, and the interface is developed to realize the real-time driving of the virtual hand motion, and then an interaction response interface is developed based on normal interaction events and response methods to realize the interaction interface of the data glove, and finally, the interaction interface allows the motion of the data glove to drive the virtual hand, conveniently providing event response triggers, and realizing the interaction between the courseware and Noitom Hi5 VR.

[0022] Optionally, the method further includes developing a communication interface via an API of the VR drive interface to transfer the captured degree of freedom data of the VR data glove hand and the collected data of the data glove to the MakeReal3D virtual reality editor in real time.

[0023] The communication interface transfers the hand degree of freedom data collected and processed by the Noitom Hi5 VR data glove to the MakeReal3D virtual reality editor in real time based on the hardware driver of the Noitom Hi5 VR data glove, and the MakeReal3D virtual reality editor identifies, labels and structures the degree of freedom data, collects response time frames, and analyzes them into usable 6 degree of freedom format real time data. As shown in Figure 2, the hardware data of the Noitom Hi5 VR data glove establishes a communication interface, i.e., real-time communication, in MakeReal3D through the development of the driver's API interface, and acquires and stores the original data from the Noitom Hi5 VR data glove in real time according to the on / off of the real-time communication and data reception, and then converts the acquired original data into real-time driving data for the MakeReal3D virtual reality editor.

[0024] Optionally, the method further includes driving a skeletal structure of the hand model in the MakeReal3D virtual reality editor, performing skeletal binding of the hand model, and setting skin weights of the controllers of each skeleton in the skeleton according to the original data structure to realize modeling of the virtual hand model.

[0025] The development of the virtual hand model is mainly done by analyzing and formatting the data collected by Noitom Hi5 VR communication, and then identifying and developing it as a driver real-time data structure. The real-time graphic driver of the data glove needs to be developed according to this data and the real-time driver graphic interface in the MakeReal3D editor.

[0026] The skeletal structure of the hand model is driven by the MakeReal3D virtual reality editor, skeletal binding of the hand model is performed, and the skin weight of the controller of each skeleton in the skeleton is set according to this original data structure, thereby realizing the development of a virtual hand model.

[0027] For example, the development process of the virtual hand model may include completing the virtual hand model by establishing fingers and finger coordinate system, setting bone and skin size matching, matching all joint bones and skin, matching carpal bone distances, setting weights based on driver structure, setting and calibrating initial state, and setting skin texture.

[0028] Optionally, the method further includes, when developing a virtual hand real-time driving interface, entering from the virtual hand real-time driving interface to set up a connection communication, calibrating real-time hardware data for setting the data glove and the initial state of the virtual hand, and releasing the connection communication after setting the data.

[0029] The virtual hand real-time driving interface mainly provides a driving interface for user settings and operation, and the driving interface generally includes developments related to settings such as data connection, data calibration, data setting, data off, etc. As shown in Figure 3, the virtual hand real-time driving interface enters from the virtual hand real-time driving interface to set the connection (communication port), calibrate the setting (calibrate the setting is to calibrate the posture, and the real-time hardware data of the Noitom Hi5 VR data glove and the initial state of the virtual hand are automatically associated and set according to the fixed calibration posture), set the data (connection type, selection of the left and right models of the virtual hand, etc.), turn off the setting (disconnect the connection communication), etc.

[0030] As an option, the interaction interface can be implemented by the virtual hand to realize events in the scene, such as grasping, releasing, and five-finger motion commands, and respond to operations. The interaction interface is the key to realizing courseware interaction, and the basis for realization is the virtual hand model with real-time driven motion and the courseware scene.

[0031] The grasping interaction of the Noitom Hi5 VR data glove is the process of picking up an object, and when the grasping interaction needs to be realized quickly, the grasping interaction can be decomposed into the grasping object (the grasping virtual hand, the object to be grasped), the grasping event response (presentation of the grasping, binding association of the grasping). As shown in Figure 4, the grasping interaction interface function is realized based on the functions of grasping gesture judgment, grasping hand interference, grasping object judgment, grasping object interference, and association binding of the grasping gesture and the grasping object after the Noitom Hi5 VR data glove drives the hand model in real time.

[0032] Specifically, grasping gesture judgment: In virtual scenes, grasping gesture judgment is the basis of grasping interaction. A general-purpose grasping interaction algorithm judges the distance from the end point of the finger node to the center of the hand to judge the grasping gesture. For example, when the hand is straightened, the fingertip is far from the center of the hand, and when the hand is in a grasping state, the fingertip and other finger joints are close to the center of the hand. In addition, the grasping gesture is judged based on the bending angle of the finger joints and the movement of each fingertip, thereby realizing real-time judgment of the grasping gesture.

[0033] Grasping object judgment: The grasping object is the object grasped by the virtual hand model. In MakeReal3D, the object has various pick-up attributes. The grasping object is different from the pick-up by the handle, which is the input condition of a single ray and a bounding box. However, the judgment of the grasping object is relatively complicated, and the input contact is the multi-point contact interference of the hand and the gesture input. Here, it is necessary to realize the judgment at the interface of the grasping gesture and the grasping interference. Here, it is necessary to realize the set method and set whether the object can be captured by the corresponding hand model.

[0034] Interference of the grasped object: The interference of the grasped object can provide visual judgment, that is, the directionality of the grasped object. In order to judge the uniqueness of the grasped object, the grasp pick-up property, etc., it is necessary to present it by presenting the interference. As a specific realization method, it is necessary to set the input for the interference of the grasped object, and when interference occurs between the interference of the grasped object and the interference of the grasper, it can be presented that it is visually captured by the real interference and can realize the interference of the grasped object.

[0035] Association judgment between grasping gesture and grasped object: After realizing the above grasping process, the association judgment between the grasping gesture and the grasped object becomes the association binding of the picked-up grasped object, i.e., association binding, which is realized by constraining the object to follow the hand model using a constraint method.

[0036] Let-go interaction: The let-go interaction process is the reverse process of the grasp interaction, and the method is consistent with the method of achieving grasp.

[0037] Five-finger motion command interaction: Based on the interface and VR realization method of hand operation in MakeReal3D, the five-finger motion command interaction is mainly based on the trigger response event of the interference interaction of the fingertips of the hand.

[0038] As an option, after realizing the interaction development in MakeReal3D of the Noitom Hi5 VR data glove mentioned above, we will continue to realize the interface of courseware interaction by the Noitom Hi5 VR data glove.

[0039] First, we set up the driver for the Noitom Hi5 VR data glove. We calibrate the initial hand state based on the standard hand state.

[0040] Next, we drive each finger. Driving each finger is the first step in realizing the interface, which is to acquire data using the Noitom Hi5 VR data glove to drive the virtual hand model in the MakeReal3D virtual reality editor.

[0041] The implementation method will be described below with an example.

[0042] In the Resource Manager, find the GloveCharacter in ControlSkeletonContent\Glove and drag it into the scene.

[0043] Select the two-handed model, find the Animation attribute of the glove in the attribute manager, and search for the AnimClass attribute parameters of the LeftHand and RightHand in turn, and find and open the LeftHand and RightHand in the resource manager.

[0044] In the opened left and right hand animation visualization script interface, set the motion capture connection respectively, select GloveType as Hi5, uncheck the ReadFromConfig option, select ReallyConnect, and click Compile and Save (to set the glove driver connection and type).

[0045] Set the default player for the glove. Select the hand in the scene, enter Player in the attribute window, find Auto Possess Player in Pawn, and set the parameter to Player0, then you can complete the driver setup for Hi5 to drive the fingers.

[0046] Next comes the configuration of the data glove's interaction interface.

[0047] The interaction interface setting of the data glove provides a method for realizing interaction. Here, the realization method is explained as follows, taking the interaction capture and release as examples.

[0048] Create a visualization script for the object to be captured: select the object to be captured, click Visualization Script / Add Visualization Script in the Properties panel, and click the pop-up panel to create the visualization script.

[0049] Set the attributes of the object to be captured. Specifically, select StaticMeshComponent in the panel that pops up after creating the visualization script, and set the attributes to SimulatePhysics (checked) and EnableGravity (to set up the interference realization of the grasp).

[0050] Add a capture event to the object to be captured: switch to Event Graph in the panel, add a Grabable component, add a SetUp event graph to the StaticMeshComponent in the event graph, click Compile and Save to complete the configuration of the data glove capturing the object (setting the interaction attributes of the grabbed object).

[0051] The last step is to realize the interactive courseware. Based on the execution mode of the MakeReal3D virtual reality editor, we set the project settings and global settings that inherit the virtual hands, then execute the courseware and realize the interaction through the 3D menu.

[0052] The present invention further provides an apparatus for implementing a data glove interface for use in interactive courseware to implement the above-mentioned method, the apparatus comprising: a virtual hand model modeling module for acquiring hand capture degree of freedom data of the VR data glove in the MakeReal3D virtual reality editor and modeling a virtual hand model based on an analysis of the hand capture degree of freedom data; A virtual hand real-time driving interface module for developing a virtual hand real-time driving interface for providing a driving interface for a user's setting and operation based on the virtual hand model; a virtual hand event command interaction interface module for developing a virtual hand real-time driving interface based on the virtual hand model to provide a driving interface for user setting and operation; The courseware includes an interaction module between the courseware and the data glove, for realizing the interaction between the courseware and the VR data glove according to the courseware interface and the event command interaction interface.

[0053] The device is further used to develop a communication interface through the API of the VR driving interface to transfer the captured freedom degree data of the VR data glove hand and the collected data of the data glove to the MakeReal3D virtual reality editor in real time.

[0054] The device is further used for analyzing data acquired by the MakeReal3D virtual reality editor, and converting the analyzed data into real-time driving data in the MakeReal3D virtual reality editor.

[0055] The virtual hand model modeling module is further used to drive the skeletal structure of the hand model in the MakeReal3D virtual reality editor, perform skeletal binding of the hand model, and set the skin weight of the controller of each skeleton in the skeleton according to the original data structure to realize the modeling of the virtual hand model.

[0056] The virtual hand real-time driving interface module is further used for entering and setting up connection communication from the real-time driving interface of the virtual hand when developing a real-time driving interface of the virtual hand, calibrating the real-time hardware data of the data glove and the initial state of the virtual hand, and releasing the connection communication after setting the data.

[0057] The virtual hand event command interaction interface module is further used to drive the virtual hand in real time based on the data glove in the scene of the virtual hand model and courseware to drive the action in real time, such as the virtual hand responding to events in the scene and operations.

[0058] The courseware and data glove interaction module is further used to drive the virtual hand in real time to respond to events and operations in the scene based on the processes of grasping gesture judgment, grasping hand interference, grasping object judgment, grasping object interference, and grasping gesture / grasp object association judgment.

[0059] An aspect of the present invention further provides an electronic device, which may be an industrial computer, a server or a computer terminal.

[0060] The electronic device includes a processor, a memory, and a computer program stored in the memory and executable by the processor, and when the computer program is executed by the processor, steps of a method for implementing a data glove interface for use in the interactive courseware on power described above are performed.

[0061] The electronic device includes a processor, a memory, and a network interface connected by a system bus, and the memory may include a non-volatile storage medium and an internal memory.

[0062] The non-volatile storage medium can store an operating system and a computer program including program instructions that, when executed, cause the processor to execute a method for implementing a data glove interface for use with any power-related interactive courseware.

[0063] The processor is used to provide computing and control capabilities and to support the overall operation of the electronic device. The internal memory is used to provide an environment for the execution of a computer program in a non-volatile storage medium, which, when executed by the processor, can execute a method for implementing a data glove interface for use in any power-related interactive courseware.

[0064] The network interface is used to distribute network communications, such as assigned tasks.

[0065] It should be understood that the processor may be a Central Processing Unit (CPU), other general purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Also, the general purpose processor may be a microprocessor, or the processor may be any conventional processor, etc.

[0066] Also, in one embodiment, the processor is adapted to execute a computer program stored in the memory and to perform the following steps:

[0067] In the MakeReal3D virtual reality editor, data on the captured degrees of freedom of the hand of the VR data glove is acquired, and a virtual hand model is modeled based on an analysis of the captured degrees of freedom of the hand data.

[0068] A virtual hand real-time driving interface is developed based on the virtual hand model, and the driving interface is provided for user setting and operation.

[0069] A virtual hand event command interaction interface is developed to realize and respond to interactions.

[0070] Based on the courseware interface and the event command interaction interface, the interaction between the courseware and the VR data glove is realized.

[0071] An aspect of the present invention further provides a computer-readable storage medium having a computer program stored thereon, the computer program being executed by a processor to perform the steps of a method for implementing a data glove interface for use in the interactive courseware relating to electrical power described above.

[0072] Finally, it should be noted that the above embodiments are only for illustrating the technical solutions of the present invention, and are not intended to limit the same. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art can make modifications to the technical solutions described in the above embodiments, or replace some or all of the technical features with equivalents, and these modifications and replacements do not deviate from the essence of the corresponding technical solutions of the embodiments of the present invention.

Claims

1. 1. A method for implementing a data glove interface for use in interactive courseware relating to electrical power, comprising: acquiring hand capture degree of freedom data of the VR data glove in a MakeReal 3D virtual reality editor, and modeling a virtual hand model based on an analysis of the hand capture degree of freedom data; Developing a virtual hand real-time driving interface based on the virtual hand model to provide a driving interface for user setting and operation; Developing a virtual hand event command interaction interface for implementing and responding to interactions; and realizing an interaction between the courseware and the VR data glove according to the courseware interface and the event command interaction interface. A method comprising:

2. 2. The method of claim 1, further comprising developing a communication interface through an API of the VR drive interface to transfer the captured degree of freedom data of the VR data glove hand and the collected data of the data glove to the MakeReal3D virtual reality editor in real time.

3. 3. The method of claim 2, further comprising: analyzing the acquired data by the MakeReal3D virtual reality editor; and converting the analyzed data into real-time driving data in the MakeReal3D virtual reality editor.

4. 2. The method of claim 1, further comprising: driving a skeletal structure of the hand model in a MakeReal3D virtual reality editor, performing skeletal binding of the hand model, and setting skin weights of the controller of each skeleton in the skeleton according to the original data structure to realize modeling of the virtual hand model.

5. The method according to claim 1, further comprising: when developing a virtual hand real-time driving interface, entering from the real-time driving interface of the virtual hand to set up a connection communication, calibrating real-time hardware data for setting the data glove and an initial state of the virtual hand, and releasing the connection communication after setting the data.

6. 10. The method of claim 1, further comprising driving the virtual hand in real time based on the data glove in a scene of a courseware and a virtual hand model that drives actions in real time, such as the virtual hand responding to events in the scene or operations.

7. The method of claim 6, further comprising: based on the processes of grasping gesture determination, grasping hand interference, grasping object determination, grasping object interference, and grasping gesture / grasp object association determination, a virtual hand is triggered in the scene and the virtual hand is driven in real time to respond to an operation.

8. A device for implementing the method according to any one of claims 1 to 7 and for implementing a data glove interface for use in interactive courseware, comprising: The apparatus comprises: a virtual hand model modeling module for acquiring hand capture DOF data of the VR data glove in the MakeReal 3D virtual reality editor and modeling a virtual hand model based on an analysis of the hand capture DOF data; A virtual hand real-time driving interface module for developing a virtual hand real-time driving interface for providing a driving interface for a user's setting and operation based on the virtual hand model; a virtual hand event command interaction interface module for developing a virtual hand real-time driving interface based on the virtual hand model to provide a driving interface for user setting and operation; a courseware and data glove interaction module for realizing an interaction between the courseware and the VR data glove according to the courseware interface and the event command interaction interface; 1 is a block diagram showing an apparatus for implementing a data glove interface for use in interactive courseware;

9. 1. An electronic device comprising: the electronic device includes a processor, a memory, and a computer program stored in the memory and executable by the processor; When the computer program is executed by the processor, the steps of a method for implementing a data glove interface for use in interactive courseware on electrical power according to any one of claims 1 to 7 are performed.

1. An electronic device comprising:

10. 1. A computer-readable storage medium, comprising: A computer program is stored in the computer-readable storage medium; When the computer program is executed by a processor, the steps of a method for implementing a DataGlove interface for use in interactive courseware on electrical power according to any one of claims 1 to 7 are performed. A computer-readable storage medium comprising:

Images