Virtual reality pose estimation apparatus and method using haptic device

Abstract

The present invention relates to a device and method for estimating a virtual reality pose through a haptic device, wherein the device comprises: an input unit that receives input point data for estimating the pose of the haptic device through a plurality of input devices; a global feature extraction unit that inputs the input point data into a Multilayer Perceptron (MLP) model to extract point-specific features and extracts global features based on the point-specific features; and a pose estimation unit that inputs the global features into a hierarchical neural network to estimate the pose of the haptic device.

Description

Technology Field

[0001] The present invention relates to a virtual reality pose estimation technology, and more specifically, to a device and method for estimating a virtual reality pose through a haptic device that can improve the accuracy of aligning virtual reality onto a haptic device by utilizing finger tracking of a Head Mounted Display (HMD) and deep learning. Background Technology

[0003] With the recent advancement of Head Mounted Display (HMD) devices, various applications are being developed that allow interaction with virtual objects using haptic devices in Virtual Reality (VR) and Augmented Reality (AR).

[0004] If haptic devices are not accurately aligned in a VR environment, it affects the user's stiffness perception. In particular, in simulations where haptic sensation is important, such as medical training simulations, perceived stiffness distortion can reduce training effectiveness by delivering stiffness different from what was intended. For example, if an object that appears hard to the eye feels soft when touched, it can confuse the user and hinder immersion.

[0005] To solve this, several methods have been proposed to accurately align VR to a haptic device.

[0006] For example, a method has been proposed that relies on hand tracking by referencing the relative coordinates of the controller provided by the manufacturer or the HMD. This method can be used when the manufacturer provides the necessary functionality.

[0007] As another example, a method has been proposed to track an HMD and an object using QR codes, external trackers equipped with cameras, or magnetic trackers, without using the hand tracking function provided by the controller or HMD device. This method increases the bulk of the system, marker tracking may fail when noise such as lighting or occlusion occurs, and it cannot be used when markers cannot be attached to the target object for alignment. Additionally, since object detection takes precedence, this method has the problem that the estimation accuracy depends on the resolution of the input image.

[0008] Meanwhile, a method using kinematics for alignment has also been proposed, but it requires the kinematics of a haptic device and has the problem of not being applicable to haptic interfaces that include soft actuators. Prior art literature

[0010] Korean Registered Patent No. 10-1826911 (2018.02.01) The problem to be solved

[0011] One embodiment of the present invention aims to provide a device and method for estimating a virtual reality pose through a haptic device that can accurately align virtual reality onto a haptic device by robustly estimating a reference position of the haptic device using a plurality of user inputs consisting of the position of the HMD, the fingertip position obtained through hand tracking, and the end position of the haptic device without using a marker. means of solving the problem

[0013] Among the embodiments, a virtual reality pose estimation device using a haptic device includes: an input unit that receives input point data for estimating the pose of the haptic device through a plurality of input devices; a global feature extraction unit that inputs the input point data into a Multilayer Perceptron (MLP) model to extract point-specific features and extracts global features based on the point-specific features; and a pose estimation unit that inputs the global features into a hierarchical neural network to estimate the pose of the haptic device.

[0014] The above input unit can receive HMD (Head Mounted Display) position data, fingertip position data, and pen pose data as input point data based on the user's visual reference point.

[0015] The above input unit can receive the HMD position data and the fingertip position data through the world coordinate system.

[0016] The above input unit can receive the position and quaternion constituting the pen pose data through a local coordinate system.

[0017] The above global feature extraction unit can generate HMD position data, fingertip position data, and pen pose data constituting the above input point data into an N-dimensional input vector (where N is a natural number greater than or equal to 3).

[0018] The above global feature extraction unit can input the above N-dimensional input vector into the above MLP model composed of a shared weight-based MLP layer structure.

[0019] The global feature extraction unit can extract a vector of dimension M (where M is a natural number greater than N) as a global feature by performing Max Pooling and Flatten operations on the feature for each point.

[0020] The pose estimation unit inputs the global feature into the hierarchical neural network implemented as a fully connected Deep Neural Network (DNN), and the input layer and at least one hidden layer of the DNN are implemented with an activation function (ReLU), and the output layer can output a pose of L (where L is a natural number that constitutes the coordinates and quaternion of the pose).

[0021] The pose estimator above can implement the loss function of the DNN through the Mean Squared Error (MSE).

[0022] Among the embodiments, a virtual reality pose estimation method using a haptic device, in a virtual reality pose estimation method using a haptic device performed in a device for estimating a virtual reality pose using a haptic device, comprises: an input step of receiving input point data for estimating the pose of the haptic device through a plurality of input devices; a global feature extraction step of inputting the input point data into a Multilayer Perceptron (MLP) model to extract point-specific features and extracting global features based on the point-specific features; and a pose estimation step of inputting the global features into a hierarchical neural network to estimate the pose of the haptic device.

[0023] The above input step may include a step of receiving HMD (Head Mounted Display) position data, fingertip position data, and pen pose data as input point data based on the user's visual reference point.

[0024] The above global feature extraction step may include a step of generating HMD position data, fingertip position data, and pen pose data constituting the above input point data into an N-dimensional input vector (where N is a natural number greater than or equal to 3).

[0025] The above global feature extraction step may further include a step of inputting the above N-dimensional input vector into the above MLP model composed of a shared weight-based MLP layer structure.

[0026] The global feature extraction step may include a step of extracting a vector of dimension M (where M is a natural number greater than N) as the global feature by performing Max Pooling and Flatten operations on the point-specific features.

[0027] The pose estimation step includes the step of inputting the global feature into the hierarchical neural network implemented as a fully connected Deep Neural Network (DNN), wherein the input layer and at least one hidden layer of the DNN are implemented with an activation function (ReLU), and the output layer can output a pose of L (where L is a natural number that constitutes the coordinates and quaternion of the pose). Effects of the invention

[0029] The disclosed technology may have the following effects. However, this does not mean that a specific embodiment must include all of the following effects or only the following effects; therefore, the scope of the rights of the disclosed technology should not be understood as being limited by this.

[0030] A device and method for estimating a virtual reality pose using a haptic device according to one embodiment of the present invention can accurately align virtual reality onto a haptic device by robustly estimating a reference position of the haptic device using a plurality of user inputs consisting of the position of the HMD, the fingertip position obtained through hand tracking, and the end position of the haptic device, without using markers.

[0031] Accordingly, the present invention can robustly estimate the reference position of a haptic device through various user inputs regardless of input noise, and can provide the effect of improving the accuracy of the user's cognitive senses by accurately aligning virtual reality over the haptic device. Brief explanation of the drawing

[0033] FIG. 1 is a diagram illustrating a virtual reality pose estimation system using a haptic device according to the present invention. Figure 2 is a diagram illustrating the system configuration of the virtual reality pose estimation device of Figure 1. Figure 3 is a diagram illustrating the functional configuration of the virtual reality pose estimation device of Figure 1. FIG. 4 is a flowchart illustrating a method for estimating a virtual reality pose through a haptic device according to the present invention. FIG. 5 is a diagram illustrating the process of determining the position of a plane corresponding to an actual floor surface on which a haptic device according to the present invention is placed. FIG. 6 is a diagram illustrating the generation of a dataset used in an experiment regarding a method for estimating virtual reality poses through a haptic device according to the present invention. FIGS. 7 to 10 are drawings illustrating experimental results regarding a method for estimating a virtual reality pose through a haptic device according to the present invention. Specific details for implementing the invention

[0034] The description of the present invention is merely an example for structural or functional explanation, and therefore the scope of the present invention should not be interpreted as being limited by the examples described in the text. That is, since the examples are subject to various modifications and may take various forms, the scope of the present invention should be understood to include equivalents capable of realizing the technical concept. Furthermore, the objectives or effects presented in the present invention do not imply that a specific example must include all of them or only such effects; therefore, the scope of the present invention should not be understood as being limited by them.

[0035] Meanwhile, the meaning of the terms described in this application should be understood as follows.

[0036] Terms such as "first," "second," etc., are intended to distinguish one component from another, and the scope of rights shall not be limited by these terms. For example, the first component may be named the second component, and similarly, the second component may be named the first component.

[0037] When it is stated that one component is "connected" to another component, it should be understood that it may be directly connected to that other component, or that there may be other components in between. Conversely, when it is stated that one component is "directly connected" to another component, it should be understood that there are no other components in between. Meanwhile, other expressions describing the relationships between components, such as "between" and "exactly between," or "adjacent to" and "directly adjacent to," should be interpreted in the same way.

[0038] A singular expression should be understood to include a plural expression unless the context clearly indicates otherwise, and terms such as "include" or "have" are intended to specify the existence of the implemented features, numbers, steps, actions, components, parts, or combinations thereof, and should be understood not to preclude the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.

[0039] In each step, identifiers (e.g., a, b, c, etc.) are used for convenience of explanation and do not describe the order of the steps; the steps may occur differently from the specified order unless a specific order is clearly indicated in the context. That is, the steps may occur in the same order as specified, may be performed substantially simultaneously, or may be performed in the reverse order.

[0040] The present invention may be implemented as computer-readable code on a computer-readable recording medium, and the computer-readable recording medium includes all types of recording devices in which data that can be read by a computer system is stored. Examples of computer-readable recording media include ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, etc. Additionally, the computer-readable recording medium may be distributed across networked computer systems, so that computer-readable code can be stored and executed in a distributed manner.

[0041] Unless otherwise defined, all terms used herein have the same meaning as generally understood by those skilled in the art to which this invention pertains. Terms defined in commonly used dictionaries should be interpreted as having meanings consistent with the context of the relevant technology and should not be interpreted as having an ideal or overly formal meaning unless explicitly defined in this application.

[0043] FIG. 1 is a diagram illustrating a virtual reality pose estimation system using a haptic device according to the present invention.

[0044] Referring to FIG. 1, a virtual reality pose estimation system (100) through a haptic device may include an HMD terminal (110), a haptic device (130), a virtual reality pose estimation device (150), and a database (170).

[0045] The HMD terminal (110) may correspond to a Head Mounted Display (HMD) device. The HMD terminal (110) is a VR headset worn on the user's head and can reproduce a three-dimensional virtual space in a real environment by interacting with a virtual reality pose estimation device (150). The HMD terminal (110) can generate control inputs based on the user's movements and transmit them to the virtual reality pose estimation device (150). Additionally, the HMD terminal (110) may have a specific program or application installed and executed that is necessary to align a real-world haptic device (130) with the three-dimensional virtual space.

[0046] To this end, the HMD terminal (110) may be connected to the virtual reality pose estimation device (150) via a wired or wireless network, and multiple HMD terminals (110) may be connected to the virtual reality pose estimation device (150) simultaneously.

[0047] A haptic device (130) is a device that reproduces physical stimuli, such as skin sensations and proprioceptive sensations, to enhance immersion and realism in a virtual or real world, and can be implemented as various devices such as gloves, joysticks, and buttons. The haptic device (130) includes a force feedback device that transmits physical force to muscles or joints and a tactile feedback device that transmits skin stimuli such as temperature, pressure, texture, vibration, and pain through mechano receptors in the skin. Among these, the force feedback device is mainly used in virtual reality, remote control, games, and simulations, and is used to identify the geometric characteristics of an object by providing proprioceptive stimuli, while the tactile feedback device mainly uses actuators such as vibration motors or pin arrays to express textures or provide feedback on specific skin sensations.

[0048] The haptic device (130) can be positioned on an actual floor surface. Here, the floor surface may correspond to a plane in real space where mixed reality (MR) is implemented. For example, the floor surface may correspond to a desk top, in which case mixed reality (MR) can be implemented on the desk.

[0049] The virtual reality pose estimation device (150) can track the position of the user's finger at the position of the HMD terminal (110) to estimate the specific position of the pen to which the tip of the user's index finger is fixed while holding and operating the haptic device (130), and perform an alignment operation by aligning the specific position of the haptic device (130) with the position of the virtual haptic device so that they are identical. Here, the virtual haptic device can be implemented as a three-dimensional virtual object identical to the actual haptic device (130).

[0050] The virtual reality pose estimation device (150) can be implemented as a computing device or an independent server device capable of executing a visual haptic mixed reality (VHMR) environment and transmitting a virtual object to an HMD terminal (110) during the virtual reality alignment process on a haptic device (130). That is, if the virtual reality pose estimation device (150) is implemented as a cloud server, it may correspond to a cloud terminal. The virtual reality pose estimation device (150) can be linked with the HMD terminal (110) via a wireless network, and the wireless network may include Bluetooth, WiFi, LTE, etc. Additionally, the virtual reality pose estimation device (150) can receive a request from the HMD terminal (110) and execute an application installed on a remote user computing device (not shown), and can transmit a 3D image in which the execution screen of the application is integrated into a virtual space to the HMD terminal (110) based on capture data received from the user computing device. The virtual reality pose estimation device (150) can perform alignment of virtual reality with a real-world haptic device (130) and can perform a process of estimating the 6 degrees of freedom (DOF) of the haptic device (130) base using the position of the HMD terminal (110), the position of the user's fingertip, and the position of the pen of the haptic device (130) in order to accurately align virtual reality with the haptic device (130). To this end, the virtual reality pose estimation device (150) can operate in conjunction with the HMD terminal (110).

[0051] The database (170) may correspond to a storage device that stores various information required during the operation of the virtual reality pose estimation device (150). For example, the database (170) may store a Multilayer Perceptron (MLP) model for pose estimation of the haptic device (130), but is not limited thereto, and may store information collected or processed in various forms during the process in which the virtual reality pose estimation device (150) performs the virtual reality pose estimation method through the haptic device according to the present invention.

[0052] In addition, in FIG. 1, the database (170) is shown as a device independent of the virtual reality pose estimation device (150), but it is not necessarily limited thereto and can be implemented as a logical storage device included in the virtual reality pose estimation device (150).

[0054] Figure 2 is a diagram illustrating the system configuration of the virtual reality pose estimation device of Figure 1.

[0055] Referring to FIG. 2, the virtual reality pose estimation device (150) may include a processor (210), memory (230), user input / output unit (250), network input / output unit (270), and communication port unit (290).

[0056] The processor (210) can execute a virtual reality pose estimation procedure through a haptic device according to an embodiment of the present invention, manage memory (230) that is read or written during this process, and schedule the synchronization time between volatile memory and non-volatile memory in memory (230). The processor (210) can control the overall operation of the virtual reality pose estimation device (150) and can control the data flow between the memory (230), user input / output unit (250), network input / output unit (270), and communication port unit (290) by being electrically connected to them. The processor (210) can be implemented as a CPU (Central Processing Unit) or GPU (Graphics Processing Unit) of the virtual reality pose estimation device (150).

[0057] The memory (230) may include an auxiliary storage device implemented as non-volatile memory such as an SSD (Solid State Disk) or HDD (Hard Disk Drive) and used to store all data required for the virtual reality pose estimation device (150), and may include a main memory device implemented as volatile memory such as RAM (Random Access Memory). Additionally, the memory (230) may store a set of instructions that execute the virtual reality pose estimation method through the haptic device according to the present invention by being executed by an electrically connected processor (210).

[0058] The user input / output unit (250) includes an environment for receiving user input and an environment for outputting specific information to the user, and may include an input device including an adapter such as a touch pad, touch screen, virtual keyboard, or pointing device, and an output device including an adapter such as a monitor or touch screen. In one embodiment, the user input / output unit (250) may correspond to a computing device connected via remote access, and in such case, the virtual reality pose estimation device (150) may be performed as an independent server.

[0059] The network input / output unit (270) provides an environment for connecting to an external device or system through a network and may include an adapter for communication such as a LAN (Local Area Network), MAN (Metropolitan Area Network), WAN (Wide Area Network), and VAN (Value Added Network). Additionally, the network input / output unit (270) may be implemented to provide short-range communication functions such as WiFi and Bluetooth, or wireless communication functions of 4G or higher for wireless transmission of data.

[0060] The communication port section (290) is a hardware interface for connecting to external hardware, for example, the external hardware may include a printer, a mouse, and USB hardware. The communication port section (290) can detect the connection of specific USB hardware to enable it to perform the role of a virtual reality pose estimation device (150).

[0062] Figure 3 is a diagram illustrating the functional configuration of the virtual reality pose estimation device of Figure 1.

[0063] Referring to FIG. 3, a virtual reality pose estimation device (150) can perform a virtual reality pose estimation method through a haptic device according to the present invention. To this end, it may include an input unit (310), a global feature extraction unit (330), a pose estimation unit (350), and a control unit (370).

[0064] At this time, the virtual reality pose estimation device (150) according to the embodiment of the present invention does not have to include all of the above functional configurations simultaneously, and depending on each embodiment, some of the above configurations may be omitted, or some or all of the above configurations may be selectively included.

[0065] The input unit (310) can receive input point data for estimating the pose of the haptic device (130) through a plurality of input devices. Here, the plurality of input devices may include the HMD terminal (110), the pen of the haptic device (130), and the fingertip of a user holding the pen. Specifically, the input unit (310) can receive HMD (Head Mounted Display) position data, fingertip position data, and pen pose data as input point data using the user's visual reference point. The input unit (310) can receive HMD position data and fingertip position data through a world coordinate system. A world coordinate system is a fixed reference coordinate system in 3D space, where the X, Y, and Z axes are usually defined based on the origin at (0,0,0), and the position and orientation of all objects are measured based on this fixed world coordinate system. HMD position data is input through the world coordinate system to represent the absolute position of the HMD in the virtual environment. For example, if the HMD position data is in the (1, 2, 3) coordinates, this means a position 1 unit away from the origin of virtual reality, 2 units away from the X-axis, and 3 units away from the Z-axis. Since the fingertip position data is also input through the world coordinate system, it can represent the absolute position of the hand in the virtual environment. Within virtual reality, the user's fingertip position data can play an important role in interaction with virtual objects. Here, the input unit (310) receives the HMD position data and fingertip position data through the world coordinate system at the same standard, thereby easily calculating the relative position and orientation between objects in the VR environment.

[0066] Additionally, the input unit (310) can receive the position and quaternion constituting the pen pose data through a local coordinate system. A local coordinate system refers to a relative coordinate system set around a reference point of a specific device or object, such as a haptic device (130) or a pen. Here, the position is a 3D coordinate (x,y,z) representing the position of the pen, and since it is input through the local coordinate system, it represents the relative position from the reference point of the haptic device (130) or the pen. The quaternion is a 4D value representing the direction (rotation) of the pen, and since it is also input through the local coordinate system, it represents the relative rotation direction from the reference point of the pen. Here, the relative position and direction between the haptic device (130) and the pen can be accurately estimated by using the local coordinate system for the pen pose data.

[0067] The global feature extraction unit (330) can extract point-specific features by inputting input point data into a Multilayer Perceptron (MLP) model and extract global features based on the point-specific features. In one embodiment, the global feature extraction unit (330) can generate HMD position data, fingertip position data, and pen pose data constituting the input point data into an N-dimensional input vector (where N is a natural number greater than or equal to 3). The global feature extraction unit (330) can input the N-dimensional input vector into an MLP model composed of a shared weight-based MLP layer structure. Shared weights refer to a structure in which weights are shared by applying the same MLP to all input points. That is, the global feature extraction unit (330) can consistently extract features for the input point data by using the same set of weights without applying independent weights, as each input point passes through an MLP network with shared weights.

[0068] The global feature extraction unit (330) can extract a global feature as an M-dimensional vector (where M is a natural number greater than N) by performing a Max Pooling operation and a Flatten operation on the feature for each point. The global feature extraction unit (330) can select important features for each dimension by performing a Max Pooling operation on the feature for each point. Additionally, the global feature extraction unit (330) can extract global features by converting the Max Pooling result into an M-dimensional vector through a Flatten layer.

[0069] The pose estimation unit (350) can estimate the pose of the haptic device (130) by inputting global features into a hierarchical neural network. The pose estimation unit (350) can input global features into a hierarchical neural network implemented as a fully connected Deep Neural Network (DNN). At this time, the input layer and at least one hidden layer of the DNN are implemented with an activation function (ReLU), and the output layer can output a pose of L (where L is a natural number that constitutes the coordinates and quaternion of the pose). The activation function (ReLU) adds non-linearity to enable the model to learn complex patterns. The pose estimation unit (350) can improve the accuracy of position and quaternion estimation by implementing the loss function of the DNN through Mean Squared Error (MSE). MSE can train the model to minimize the difference between the predicted value and the actual value.

[0070] The control unit (370) controls the overall operation of the virtual reality pose estimation device (150) and can manage the control flow or data flow between the input unit (310), the global feature extraction unit (330), and the pose estimation unit (350).

[0072] FIG. 4 is a flowchart illustrating a method for estimating a virtual reality pose through a haptic device according to the present invention.

[0073] Referring to FIG. 4, the virtual reality pose estimation device (150) can receive input point data for estimating the pose of a haptic device (130) through a plurality of input devices via an input unit (310) (step S410). The virtual reality pose estimation device (150) can generate HMD position data, fingertip position data, and pen pose data constituting the input point data into an N-dimensional input vector (where N is a natural number greater than or equal to 3) via a global feature extraction unit (330) (step S430).

[0074] Additionally, the virtual reality pose estimation device (150) can extract point-specific features by inputting an N-dimensional input vector into an MLP model composed of a shared weight-based MLP layer structure through the global feature extraction unit (330) (step S450). The virtual reality pose estimation device (150) can extract a global feature by performing Max Pooling and Flatten operations on the point-specific features through the global feature extraction unit (330) to obtain an M-dimensional vector (where M is a natural number greater than N) (step S470).

[0075] Additionally, the virtual reality pose estimation device (150) can estimate the pose of the haptic device (130) by inputting global features into a hierarchical neural network through the pose estimation unit (350) (step S490).

[0077] FIG. 5 is a diagram illustrating a pose estimation network model of a haptic device according to the present invention.

[0078] Referring to FIG. 5, the virtual reality pose estimation device (150) can be implemented as a network model (500) that estimates the pose (position and quaternion) of a haptic device (130) using the HMD position, fingertip position and pen pose (position and quaternion).

[0079] The pose estimation network model (500) can be designed as a network structure capable of receiving at least 6 to 9 user inputs for estimation performance robust to noise, and can be implemented including an input unit (510), a global feature extraction model (530), and a pose estimation model (550).

[0080] When a user holds and operates the haptic device (130), the input unit (510) can collect vector data (1×13) containing the HMD position (1×3), fingertip position (1×3), pen position, and quaternion (1×7). The input unit (510) can set the size of the input data to 10×13 to flexibly receive a minimum of 6 to a maximum of 9 vector inputs, and can perform zero padding to fill the empty space caused by the difference in the number of vectors with a zero value.

[0081] The global feature extraction model (530) can be composed of a shared MLP and a max pooling structure to ensure that the pose estimation network model (500) is not affected by the order of the vectors. The MLP model (531) is composed of three convolution layers, and each layer is composed of a shared weight-based MLP layer structure having units of sizes 64×1×13, 128×1×13, and 256×1×13 in order. The value passing through the convolution layer is converted into a 256-dimensional vector, which becomes an N×256 matrix for N input points. By applying max pooling to the 256-dimensional feature vector of the input point data, the maximum value among the feature values ​​is selected, allowing the global features of the input vectors to be extracted regardless of the order of the input vectors. Afterward, it passes through a flatten layer and is converted into a 1×1×256 dimension, which is then used as the input to the DNN constituting the pose estimation model (550).

[0082] The pose estimation model (550) can be implemented as a DNN having a fully connected network structure and consisting of a total of 5 layers. Among the total 5 layers of the DNN, one input layer and three hidden layers have an activation function (ReLU) applied, and the last output layer outputs the position (x,y,z) and quaternion (rotation) that constitute the pose of the haptic device.

[0084] FIGS. 6 to 10 are drawings illustrating experimental results regarding a method for estimating a virtual reality pose through a haptic device according to the present invention.

[0085] In the case of Figures 7 to 10, they may correspond to experimental results using a dataset generated from 74,000 data collected through the process of Figure 6.

[0086] In the case of Fig. 6, HMD position data and fingertip position data were measured in the virtual reality world coordinate system using the spatial recognition function of HoloLens 2, and pen pose data was acquired using the function provided by the haptic device. While holding and moving the pen, the user periodically samples the pen pose relative to the base of the haptic device, the HMD position, and the position of the user's index finger in the world coordinate system. The top left of Fig. 6 shows the information included in the dataset, and the bottom shows the selected vector among the vectors; 10,000 datasets were generated by combining N (6 ≤ N ≤ 9) vectors for a single base, resulting in a total dataset of size 740,000 × N × 13. Here, the QR code tracking function of HoloLens 2 was utilized to obtain the reference value of the haptic device base through the world coordinate system. After fixing the haptic device and the QR code to a plate, the QR code was captured using HoloLens 2 to track the position of the haptic device base in the world coordinate system. Here, the haptic device was aligned with respect to the QR code coordinate system by utilizing the fixed relationship between the QR code and the haptic device. After capturing the QR code and fixing its position, data was collected while moving the pen.

[0087] When a user wears an HMD connected to Unity, the HMD position in the virtual reality world coordinate system starts at (0,0,0). The HMD position and fingertip position collected using the virtual reality world coordinate system are measured near the origin. Therefore, the estimation accuracy may decrease as the distance from the origin increases. To address this, data augmentation is performed by adding a bias to the position data collected in the virtual reality world coordinate system. Data is generated by adding a bias of 0.1m within the range of -0.2 to 0.2 on the X and Z axes, and a bias within the range of 0 to 0.2 on the Y axis.

[0088] To flexibly use the number of vectors in the input data, a range of vector counts is determined, and to solve the problem of empty spaces occurring in the input data due to differences in the number of vectors, zero-padding is applied to the number of empty vectors to create a dataset. In addition, to apply zero-padding equally to all data, the size of the input data is set to be 1 greater than the number of vectors so that zero-padding is applied even when 9 vectors are input. An input point dataset is generated from the sampled data for the pose estimation network model (500) proposed in the present invention. Six to nine points, each with a size of 1×13, are collected as a single dataset. Points with a distance of at least 3.2 cm between them are extracted to collect uniformly distributed points. A total of 10,000 datasets are constructed for each of the 74 6-DOF values ​​based on the haptic device, thereby finally generating a dataset of size 740,000×10×13.

[0089] The experiment was conducted on Windows 10 OS and an NVIDIA RTX 3090 Ti. The Hololens 2 was used as the HMD, and the Touch 3D was used as the haptic device. All acquired data, except for the pen's 6 degrees of freedom, used coordinates in the Unity virtual environment coordinate system. The pen's 6 degrees of freedom use the haptic device's local coordinate system.

[0090] Training and testing were conducted on 740,000 collected data points. As shown in Figure 7(a), the test results showed that the Euclidean distance error averaged 1.74 mm, the minimum was 0.148 mm, and the maximum was 8.22 mm. Although the maximum distance is relatively large compared to the average distance error, 50% of the data is distributed within 1.67 mm, and 99.9% of the data is distributed within 1.86 mm. Furthermore, an analysis of the distance error by axis revealed RMSE values ​​of 0.816 mm for the x-axis, 0.684 mm for the y-axis, and 1.68 mm for the z-axis. Additionally, the x-axis distance error ranged from a minimum of 2.35e^(-5) mm to a maximum of 3.00 mm, the y-axis distance error ranged from a minimum of 1.38e^(-5) mm to a maximum of 2.24 mm, and the z-axis distance error ranged from a minimum of 2.19e^(-5) mm to a maximum of 8.21 mm.

[0091] In addition, to analyze the rotational error for each axis, the output results in quaternions were converted into Euler angles for analysis. Furthermore, to compare the magnitude of the rotational error, the difference between the correct value and the predicted value was converted to a range of -180° to +180° and the absolute value was applied to analyze the magnitude of the rotational error. As shown in Fig. 7(b), the RMSE results were 0.274° for the x-axis, 0.397° for the y-axis, and 0.966° for the z-axis; the x-axis rotational error ranged from a minimum of 0.0994° to a maximum of 0.449°, the y-axis rotational error ranged from a minimum of 3.64e^(-5)° to a maximum of 1.43°, and the z-axis rotational error ranged from a minimum of 3.66e^(-5)° to a maximum of 2.35°.

[0092] Since the data used for training and testing consisted of 6 to 9 vectors per dataset, the average distance error, RMSE for distance along each axis, and RMSE for rotation error along each axis were analyzed according to the number of vectors to examine the impact of the number of vectors on the results. As a result of the analysis, as shown in Fig. 8(a), it can be seen that the RMSE for the x and y axes showed almost no change. The maximum rate of change in the x-axis RMSE according to the number of vectors was 0.00888 mm, and the maximum rate of change in the y-axis RMSE according to the number of vectors was 0.00365 mm, both showing small changes. The z-axis RMSE according to the number of vectors increased up to 8 vectors and then decreased again when the number reached 9. However, the maximum rate of change in RMSE was 0.0273 mm, showing a small change. As shown in Fig. 8(b), the RMSE for rotation along each axis showed almost no change depending on the number of vectors.

[0093] The data used for training consisted of vectors where the distance between each vector in a single data point was 3.2 cm or greater. Therefore, experiments were conducted on cases where vectors were clustered within a certain distance. Ranges of 3 cm, 6 cm, 9 cm, 10 cm, and 12 cm were set, a reference vector was randomly selected, and a dataset was generated to include 6 to 9 vectors within that range, including the reference vector, based on the selected vector. As the range containing 6 to 10 points in the dataset increased, the average distance error and the distance RMSE values ​​for each axis showed a common decreasing trend, as shown in Figures 9 (a) and (b), while the rotation RMSE values ​​for each axis showed almost no change. As the range increased, the average distance error was 1.89 mm when the range was 3 cm, but decreased as the range increased, showing an error of 1.70 mm when the range was 12 cm. In addition, when the range was 3 cm, the x-axis distance RMSE was 0.897 mm, the y-axis distance RMSE was 0.707 mm, and the z-axis distance RMSE was 2.30 mm; however, when the range was 12 cm, the x-axis, y-axis, and z-axis RMSEs were 0.821 mm, 0.696 mm, and 1.62 mm, respectively. The rotation RMSE according to the range showed almost no change. Although the y-axis and z-axis rotation RMSEs showed an increasing trend, the maximum rate of change for the y-axis and z-axis rotation RMSEs was 0.008086° and 0.00138°, respectively, indicating a very small change.

[0094] The finger tracking provided by Hololens2 used in the experiment does not always estimate the exact position of the finger. The performance of the pose estimation network model proposed in this invention was evaluated in the case where noise occurs when estimating the finger position. Data was generated by varying the number of noises and the distance from the original finger position in the acquired data, and the number of noises was set to 1 to 5, and the distance error was set to 1 cm to 3 cm.

[0095] A total of 15 experiments were conducted by combining the number of noises and distance errors. In most cases, similar values ​​were observed when there was no noise, as shown in Fig. 10. Although the average distance error was higher than in the experiment without noise in some instances, the maximum error was 0.00551 mm. Additionally, the y-axis and z-axis RMSE were also higher in some cases, but the maximum errors were 0.000602 mm and 0.0104 mm, respectively. The Z-axis rotation RMSE also showed a higher value than the results of the experiment without noise, but the difference was 6.45e^(-5)°.

[0096] As a result, it was confirmed that the pose estimation network model proposed in this invention is composed of a shared weight-based MLP layer structure, where filters are applied independently to input vectors, and global features of vectors are extracted by selecting the largest feature value through max pooling, thereby allowing features of noisy vectors to be filtered out, thus demonstrating robust performance against noise.

[0098] Although the present invention has been described above with reference to preferred embodiments, those skilled in the art will understand that various modifications and changes can be made to the invention without departing from the spirit and scope of the invention as described in the following claims. Explanation of the symbols

[0100] 100: Virtual Reality Pose Estimation System via Haptic Devices 110: HMD terminal 130: Haptic device 150: Virtual reality pose estimation device 170: Database 210: Processor 230: Memory 250: User I / O Section 270: Network I / O Section 290: Communication port section 310: Input section 330: Global feature extraction section 350: Pose Estimation Unit 370: Control Unit 500: Pose Estimation Network Model 510: Input section 530: Global feature extraction model 531: MLP (Multilayer Perceptron) Model 550: Pose Estimation Model

Claims

Claim 1 A virtual reality pose estimation device using a haptic device, comprising: an input unit receiving input point data for estimating the pose of a haptic device through a plurality of input devices; a global feature extraction unit inputting the input point data into a Multilayer Perceptron (MLP) model to extract point-specific features and extracting global features based on the point-specific features; and a pose estimation unit inputting the global features into a hierarchical neural network to estimate the pose of the haptic device. Claim 2 A virtual reality pose estimation device using a haptic device, characterized in that, in claim 1, the input unit receives HMD (Head Mounted Display) position data, fingertip position data, and pen pose data as input point data using the user's visual reference point. Claim 3 A virtual reality pose estimation device using a haptic device, characterized in that, in paragraph 2, the input unit receives the HMD position data and the fingertip position data through a world coordinate system. Claim 4 A virtual reality pose estimation device using a haptic device, characterized in that, in paragraph 2, the input unit receives the position and quaternion constituting the pen pose data through a local coordinate system. Claim 5 A virtual reality pose estimation device using a haptic device, wherein, in claim 1, the global feature extraction unit generates HMD position data, fingertip position data, and pen pose data constituting the input point data into an N-dimensional input vector (where N is a natural number greater than or equal to 3). Claim 6 A virtual reality pose estimation device using a haptic device, wherein, in claim 5, the global feature extraction unit inputs the N-dimensional input vector into the MLP model composed of a shared weight-based MLP layer structure. Claim 7 A virtual reality pose estimation device using a haptic device, wherein, in claim 5, the global feature extraction unit performs Max Pooling and Flatten operations on the point-specific features to extract an M-dimensional vector (where M is a natural number greater than N) as the global feature. Claim 8 A virtual reality pose estimation device using a haptic device according to claim 1, wherein the pose estimation unit inputs the global feature to the hierarchical neural network implemented as a fully connected Deep Neural Network (DNN), the input layer and at least one hidden layer of the DNN are implemented with an activation function (ReLU), and the output layer outputs a pose of L (where L is a natural number constituting the coordinates and quaternion of the pose). Claim 9 A virtual reality pose estimation device using a haptic device, characterized in that, in claim 8, the pose estimation unit implements the loss function of the DNN through the Mean Squared Error (MSE). Claim 10 A method for estimating a virtual reality pose through a haptic device, performed in a device for estimating a virtual reality pose through a haptic device, comprising: an input step of receiving input point data for estimating the pose of the haptic device through a plurality of input devices; a global feature extraction step of inputting the input point data into a Multilayer Perceptron (MLP) model to extract point-specific features and extracting global features based on the point-specific features; and a pose estimation step of inputting the global features into a hierarchical neural network to estimate the pose of the haptic device. Claim 11 A method for estimating a virtual reality pose through a haptic device, characterized in that, in claim 10, the input step includes receiving HMD (Head Mounted Display) position data, fingertip position data, and pen pose data as input point data using the user's visual reference point. Claim 12 A method for estimating a virtual reality pose through a haptic device, characterized in that, in claim 10, the global feature extraction step comprises the step of generating HMD position data, fingertip position data, and pen pose data constituting the input point data into an N-dimensional input vector (where N is a natural number greater than or equal to 3). Claim 13 A method for estimating a virtual reality pose through a haptic device, characterized in that, in claim 12, the global feature extraction step further includes the step of inputting the N-dimensional input vector into the MLP model composed of a shared weight-based MLP layer structure. Claim 14 A method for estimating a virtual reality pose through a haptic device, characterized in that, in claim 12, the global feature extraction step includes a step of extracting a vector of dimension M (where M is a natural number greater than N) as the global feature by performing a Max Pooling operation and a Flatten operation on the point-specific features. Claim 15 A method for estimating a virtual reality pose through a haptic device, wherein, in claim 10, the pose estimation step comprises the step of inputting the global feature into the hierarchical neural network implemented as a fully connected Deep Neural Network (DNN), wherein the input layer and at least one hidden layer of the DNN are implemented with an activation function (ReLU), and the output layer outputs a pose of L (where L is a natural number constituting the coordinates and quaternion of the pose).

Images