Method and device for predicting cybersickness in virtual reality by using multimodal and time-series data

Abstract

The method for predicting cybersickness in virtual reality by using multimodal and time-series data includes the steps in which a prediction device: receives an input of information on a subject; inputs the information on a subject to a neural network model; and predicts a cybersickness level of the subject on the basis of an output value of the neural network model. The information on a subject includes eye movement data, head movement data, and physiological signals data. The neural network model is a transformer-based model that performs multi-head attention.

Description

Method and device for predicting cybersickness in virtual reality using multimodal and time-series data

[0001] The technology described below relates to a method and device for predicting cybersickness in virtual reality.

[0002] Virtual Reality (VR) is a technology that creates a specific environment (or situation) that is not real but closely resembles it. With recent advancements in computer technology, various VR-based content has emerged. For example, technology has been developed to create driving situations within VR, allowing people to practice driving.

[0003] Users experience cybersickness when experiencing virtual reality. Cybersickness refers to the feeling of motion sickness experienced when enjoying 3D games or videos. Cybersickness is known to have various causes. Recently, various technologies are being researched and developed to reduce cybersickness.

[0004] To reduce cybersickness, it's necessary to analyze the types of content that trigger it. To address this, technologies are being developed to predict cybersickness using artificial neural network-based AI models.

[0005] [Prior Art Literature]

[0006] [Patent Document]

[0007] (Patent Document 1) Korean Registered Patent No. 10-2233099

[0008] Predicting cybersickness using AI models relies on time-series data measured from various sensors. Previous techniques for predicting cybersickness using artificial neural network-based AI models have utilized models based on convolutional neural networks (CNNs) and recurrent neural networks (RNNs). While CNN-based models are specialized for learning data features, they struggle to process time-series data. While RNN-based models are specialized for processing time-series data, they struggle to learn data features.

[0009] The technology described below aims to provide a method for predicting cybersickness in virtual reality using a neural network-based model capable of analyzing multimodal and time-series data.

[0010] A method for predicting cybersickness in virtual reality using multimodal and time-series data includes a step in which a prediction device receives information about a subject; a step in which the prediction device inputs information about the subject into a neural network model; and a step in which the prediction device predicts a level of cybersickness of the subject based on an output value of the neural network model.

[0011] The information of the subject includes eye movement data, head movement data, and physiological signals data.

[0012] The above neural network model is a Transformer-based model that performs multi-head attention.

[0013] The technology described below can be used to predict and assess the level of cybersickness experienced by users experiencing virtual reality. Specifically, the level of cybersickness can be predicted and assessed based on various types of data. Furthermore, the level of cybersickness can be predicted and assessed based on time-series data.

[0014] The technology described below can be used to create virtual reality content that reduces cybersickness.

[0015] Figure 1 shows the overall process of the prediction device (100) performing the cyber motion sickness prediction method.

[0016] Figure 2 is a flowchart (200) of one embodiment of a cyber motion sickness prediction method.

[0017] Figures 3 to 5 briefly illustrate examples of neural network models.

[0018] Figure 6 shows the process of collecting information on the subject, preprocessing it, and building a model based on it.

[0019] Figures 7 to 11 show the structure of the constructed neural network model.

[0020] Figure 12 shows the results of evaluating the model's performance.

[0021] Figures 13 to 15 show the results of analyzing the attention scores assigned to each data by the attention MS-STTN model.

[0022] Fig. 16 is a configuration of one embodiment of a prediction device (300).

[0023] The technology described below is susceptible to various modifications and embodiments. Specific embodiments of the technology described below may be illustrated in the drawings of the specification. However, these are intended to illustrate the technology described below and are not intended to limit the technology described below to any specific embodiments. Therefore, it should be understood that all modifications, equivalents, or alternatives that fall within the spirit and scope of the technology described below are encompassed by the technology described below.

[0024] Terms such as first, second, A, and B may be used to describe various components. However, these terms are only used to distinguish one component from other components and are not intended to limit the components. For example, without departing from the scope of the technology described below, the first component may be referred to as the second component, and similarly, the second component may also be referred to as the first component. The term "and / or" includes any combination of multiple related items described or any item among multiple related items described.

[0025] In the terms used hereinafter, singular expressions should be understood to include plural expressions unless the context clearly dictates otherwise, and terms such as "comprises" should be understood to mean the presence of a described feature, number, step, operation, component, part, or combination thereof, but not to exclude the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0026] Before going into a detailed description of the drawings, it should be made clear that the division of components in this specification is merely a division based on the main function of each component. That is, two or more components described below may be combined into one component, or one component may be further divided into two or more components with more detailed functions. In addition to its own main function, each component described below may additionally perform some or all of the functions of other components, and of course, some of the main functions of each component may be exclusively performed by other components.

[0027] Additionally, in performing a method or method of operation, each process constituting the method may occur in a different order than the stated order, unless the context clearly indicates a specific order. That is, each process may occur in the same order as the stated order, may be performed substantially simultaneously, or may be performed in the opposite order.

[0028] Below, we examine the overall process of the prediction device performing a cyber-motion sickness prediction method (hereinafter, “cyber-motion sickness prediction method”) in virtual reality using multimodal and time-series data.

[0029] Figure 1 shows the overall process of the prediction device (100) performing the cyber motion sickness prediction method.

[0030] The prediction device (100) can receive information about the subject. The prediction device (100) can input the subject's information into a neural network model. The prediction device (100) can predict the subject's level of cybersickness based on the output value of the neural network model. Furthermore, the prediction device (100) can preprocess the subject's information.

[0031] The subject's information may include eye movement data, head movement data, and physiological signals. The neural network model may be a Transformer-based model that performs multi-head attention.

[0032] Below, the cyber motion sickness prediction method is explained in detail.

[0033] Figure 2 is a flowchart (200) of one embodiment of a cyber motion sickness prediction method.

[0034] The prediction device can receive information about the subject (210).

[0035] The subject's information may include information necessary to predict cybersickness.

[0036] Subject information may be obtained while the subject is wearing a VR device. For example, subject information may be measured while the subject is experiencing virtual reality while wearing a VR head-mounted display (HMD).

[0037] Subject information may be measured over time. In other words, subject information may include time-series data. For example, subject information may be measured from a subject wearing a virtual reality device for 45 seconds.

[0038] Subject information may be measured at regular intervals. For example, it may be measured at a frequency of 60 measurements per second (60Hz).

[0039] The subject's information may include eye movement data, head movement data, and physiological signals data.

[0040] Eye movement data can contain 23 types of data.

[0041] The eye movement data may include at least one of the X, Y, Z coordinates of the gaze direction (x, y, z) of the left and right eyes, respectively, the X, Y, Z coordinates of the gaze origin (x, y, z) of the left and right eyes, respectively, the pupil diameter (x, y) of the left and right eyes, respectively, the X, Y coordinates of the pupil position (x, y) of the left and right eyes, respectively, the eye openness (whether the left and right eyes were open), and the X, Y, Z coordinates of the gaze direction with both eyes integrated (x, y, z)).

[0042] Head movement data can include six types of data.

[0043] Head movement data may include at least one of X, Y, Z coordinates of head position (head position (x, y, z)) and roll, pitch, yaw of head rotation (head rotation (roll, pitch, yaw)).

[0044] Biosignal data can contain three types of data.

[0045] The biosignal data may include at least one of electrodermal activity, blood volume pulse, and skin temperature.

[0046] The prediction device may also preprocess information about the subject (220).

[0047] Preprocessing may include at least one of a process of removing redundant processes from data included in the subject information, a process of normalizing the subject information, and a process of Fourier transforming the subject information.

[0048] Preprocessing may include a process for removing duplicate values. In one embodiment, to eliminate data duplication that occurs during data collection, preprocessing may include a process for continuously removing data with duplicate values.

[0049] Preprocessing may include a process of normalizing the subject's information. In one embodiment, preprocessing may include a process of normalizing the subject's information to values ​​between 0 and 1.

[0050] Preprocessing may include performing a Fourier transform on the subject's information. Furthermore, preprocessing may include performing a short-time Fourier transform (STFT) on the subject's information. In one example, preprocessing may include performing a short-time Fourier transform (STFT) on the subject's information, which is 45 seconds long, at 15-second intervals.

[0051] The prediction device can input the subject's information into the neural network model (230).

[0052] A neural network model can be a model that predicts the degree of cybersickness of a subject based on information about the subject.

[0053] The neural network model can be an artificial neural network-based model. It can also be a Transformer-based model that performs multi-head attention. Specifically, the neural network model can have the structure of a Vision Transformer.

[0054] Neural network models can embed subject information. To do this, they can utilize a linear projection layer. Alternatively, they can include a linear patch embedding layer. The linear patch embedding layer can map input data to a constant vector size. This allows all layers of the transformer to use a constant vector size D.

[0055] A neural network model can include a positional embedding layer that reflects location information in the input subject information. This layer is used to reflect the location information of each data point, as it is difficult for the transformer to determine the order of position vectors.

[0056] Before passing the position embedding layer, the neural network model can add learnable Class Tokens (CLS). After passing through the Transformer, the CLS can serve as a representation vector containing important features of the data for predicting cybersickness.

[0057] A neural network model can be a model trained to identify the characteristics of each modality (eye movement, head movement, biosignals).

[0058] A neural network model can be of three types. The first neural network model can include a temporal transformer encoder. The second neural network model can include a spatial transformer encoder and a temporal transformer encoder sequentially. The third neural network model can include a first module, a second module, and a third module, each sequentially including a spatial transformer encoder and a temporal transformer encoder.

[0059] A detailed description of the neural network model is provided below.

[0060] The prediction device can predict the level of cybersickness of the subject based on the output value of the neural network model (240).

[0061] Cybersickness levels can be categorized into multiple levels. For example, cybersickness levels could be categorized into four levels based on the severity of cybersickness (0: Not at all dizzy, 1: Slightly dizzy, 2: Somewhat dizzy, 3: Extremely dizzy).

[0062] Below, the neural network model is described in detail.

[0063] Figures 3 to 5 briefly illustrate examples of neural network models.

[0064] As shown in Fig. 3, the neural network model may include a temporal transformer encoder.

[0065] A temporal transformer encoder can perform multi-head attention based on the relationship between data at different time points (t). A temporal transformer encoder can perform multi-head attention based on the relationship between embedded values ​​of data at different time points.

[0066] Specifically, the temporal transformer encoder can analyze the relationship between values ​​embedded in data at different points in time, calculate a temporal attention score, and then assign attention to each value embedded in data at different points in time based on the temporal attention score.

[0067] The embedded data value can be a single vector value created by synthesizing eye movement data, head movement data, and biosignal data measured at the same point in time. For example, the embedded data value can be a single vector value created by passing 23 types of eye movement data, 6 types of head movement data, and 3 types of biosignals measured over 1 second through a linear patch layer.

[0068] The temporal transformer encoder can accept Class Tokens (CLS) as input. Specifically, the temporal transformer encoder can accept TCLS as input. After passing through the temporal transformer, Tcls can become a representation vector that plays a crucial role in predicting cybersickness.

[0069] As shown in Fig. 4, the neural network model may sequentially include a spatial transformer encoder and a temporal transformer encoder.

[0070] A spatial transformer encoder can perform multi-head attention based on the relationship between data contained in a single viewpoint.

[0071] A spatial transformer encoder can perform multi-head attention based on the relationship between embedded values ​​of data contained at a single point in time.

[0072] Specifically, the spatial transformer encoder can analyze the relationship between the embedded values ​​of data included in one point in time, calculate a spatial attention score, and then assign attention to each embedded value of data included in one point in time based on the spatial attention score.

[0073] A spatial transformer encoder can generate an embedding value for a single time point that reflects spatial information by combining the embedded values ​​of data included in a single time point to which spatial attention has been given.

[0074] A spatial transformer encoder can input data from multiple time points and generate data embedding values ​​that reflect spatial information for each time point. In one embodiment, the spatial transformer encoder can input data from 45 time points measured every second for 45 seconds and generate 45 data embedding values ​​that reflect spatial information for each time point.

[0075] A temporal transformer encoder can receive the output of a spatial transformer encoder and perform multi-head attention based on the relationships between data at different time points. In other words, the embedded value of the data input to the temporal transformer encoder can be an embedded value for a single time point that reflects the spatial information output by the spatial transformer encoder.

[0076] A spatial transformer encoder can also extract Class Tokens (CLS). Specifically, a spatial transformer encoder can extract Scls. The Scls extracted by the spatial transformer encoder can be matched to the dimension of Tcls input to the temporal transformer after passing through the linear layer. Therefore, Tcls can reflect both spatial and temporal characteristics.

[0077] As shown in Fig. 5, the neural network model may include a first module, a second module, and a third module, each sequentially including a spatial transformer encoder and a temporal transformer encoder. That is, the first module, the second module, and the third module may sequentially include a spatial transformer encoder and a temporal transformer encoder, respectively.

[0078] The neural network model can combine the outputs of the first, second, and third modules to calculate the information needed to predict the degree of cybersickness.

[0079] The first module can receive eye movement data from the subject. The second module can receive head movement data from the subject. The third module can receive biosignal data from the subject.

[0080] Modules 1, 2, and 3 utilize spatial and temporal transformer encoders and, respectively, analyze the spatial and temporal characteristics of eye movement data, head movement data, and biosignal data, and then generate information necessary to predict cybersickness. In other words, Modules 1, 2, and 3 can each perform the computations of the model described in Fig. 4.

[0081] Below, we examine the experimental results of actually constructing the neural network model used in the present invention and predicting cybersickness using the constructed model.

[0082] Figure 6 shows the process of collecting information on the subject, preprocessing it, and building a model based on it.

[0083] Subject information was collected from 45 subjects who viewed 20 pieces of 360 VR content.

[0084] Subject information includes motion sickness multimodal sensor data. Subject information includes eye movement data (e1, e2, …e23), head movement data (h1, h2, h3, …h6), and biosignal (p1, p2, p3) data. Eye movement data and head movement data were measured from a VR head-mounted display (HDM). Biosignal data was measured from a biosignal device (Empatica E4 wristband).

[0085] The eye movement data used in the experiment included 23 types of data. The eye movement data included the X, Y, Z coordinates of the gaze direction of the left and right eyes respectively (gaze direction (x, y, z)), the X, Y, Z coordinates of the gaze origin of the left and right eyes respectively (gaze origin (x, y, z)), the pupil diameter of the left and right eyes respectively (pupil position (x, y)), the X, Y coordinates of the pupil position of the left and right eyes respectively (eye openness), and the X, Y, Z coordinates of the gaze direction with both eyes integrated (x, y, z)).

[0086] The head movement data used in the experiment includes six types of data. The head movement data used in the experiment includes the X, Y, and Z coordinates of the head position (head position (x, y, z)), and the roll, pitch, and yaw of the head rotation (head rotation (roll, pitch, yaw)).

[0087] The biosignal data used in the experiment included three types of data: skin conductance (Electrodermal Activity), blood volume pulse, and skin temperature.

[0088] Subject information was measured at a measurement frequency of 90 times per second (90Hz).

[0089] The subjects' level of cybersickness was analyzed every 15 seconds. The level of cybersickness was measured from responses to the Fast Motion Sickness (FMS) scale. The FMS scale is used to quantitatively measure the level of cybersickness.

[0090] The measured subject information was preprocessed.

[0091] First, to address data duplication that occurs when sensor data is collected, values ​​with consecutive duplicates were removed. The measured subject information was downsampled from 90Hz to 25Hz.

[0092] Afterwards, the subject information was normalized to a value from 0 to 1. As described below, the subject information that had undergone downsampling and normalization was used to build the TTN, STTN, and MS-SSTN models.

[0093] The subject information was then converted into multimodal spectrogram data via short-time Fourier transform (SFT). As described below, the downsampled, normalized, and SFT-transformed subject information was used to build the STTS and MS-STTS models.

[0094]

[0095] Figures 7 to 11 show the structure of the constructed neural network model.

[0096] There are a total of five neural network models constructed.

[0097] As shown in Fig. 7, the first neural network model is a model that includes a temporal transformer encoder. This can be called Temporal for normalized sensor data (TTN).

[0098] As shown in Figure 8, the second neural network model is a model that includes a spatial transformer encoder. This can be called a spatial transformer encoder (STE).

[0099] As shown in Fig. 9, the third neural network model may include a first module, a second module, and a third module, each sequentially including a spatial transformer encoder and a temporal transformer encoder. This may be referred to as a modality-specific spatial-temporal transformer for normalized sensor data (MS-STTN).

[0100] As shown in Figure 10, the fourth neural network model preprocesses the subject's information using the Short-Time Fourier Transform (STTF) and then inputs it into the aforementioned TTN model. This can be called a Spatial-Temporal Transformer for Spectrogram (STTS).

[0101] As shown in Figure 11, the fifth neural network model preprocesses the subject's information using STTF and then inputs it into the aforementioned MS-STTN model. This can be called a modality-specific-spatial-temporal transformer for spectrogram (MS-STTS).

[0102] Figure 12 shows the results of evaluating the model's performance.

[0103] Experimental results confirm that MS-STTN achieves the highest performance. This demonstrates the effectiveness of using normalized data, training with independent transformer encoders for each modality, and then integrating these data to predict cybersickness.

[0104] Figures 13 to 15 show the results of analyzing the attention scores assigned to each data by the attention MS-STTN model.

[0105] Figure 13 shows the attention scores assigned to eye movement data. Figure 13(a) shows the attention scores assigned to data when both eyes are combined. Figure 13(b) shows the attention scores assigned to data from the right eye. Figure 13(c) shows the attention scores assigned to data from the left eye. Figure 13(d) shows the average of the attention scores assigned to the right and left eyes.

[0106] When both eyes are combined, the Z coordinate of the gaze direction shows the highest importance. When both eyes are combined, the Y coordinate of the gaze direction shows a significant influence in predicting cybersickness in the intermediate phase.

[0107] We can see that the X-coordinate and Y-coordinate of the starting point of the right eye's gaze direction show high importance in most phases. Conversely, we can see that the X-coordinate of the starting point of the left eye's gaze direction and the Z-coordinate of the left eye's gaze direction show high importance in most phases.

[0108] It can be seen that the Y-coordinate of the gaze direction, the diameter of the pupil, and the X-coordinate of the pupil position are affected in the intermediate phase.

[0109] We can see that the X-coordinate of the gaze direction, whether the eyes were open or not, and the X-coordinate of the gaze direction when both eyes were combined had a low influence at all stages.

[0110] Figure 14 shows the attention scores assigned to head movement data.

[0111] We can see that the X-coordinate of the head position has the greatest influence on the prediction of cybersickness in all phases. We can see that the roll of head rotation has a large influence in the middle and last phases. This may be because the roll of head rotation means rotating around the x-axis, such as nodding the head forward or tilting it backward. VR videos are often 360 degrees, and subjects show a lot of head movement to explore the surroundings while experiencing virtual reality content. Such head movement can cause postural instability and display lag, which can induce cybersickness.

[0112] Figure 15 shows the attention scores assigned to biosignal data.

[0113] Skin conductance was found to have the greatest impact on cybersickness in all phases.

[0114] The prediction device is described below through Fig. 16.

[0115] Fig. 16 is a configuration of one embodiment of a prediction device (300).

[0116] The prediction device (300) may correspond to the prediction device (100) described in Fig. 1. That is, the prediction device (300) may be a device that performs the cyber motion sickness prediction method described above.

[0117] The prediction device (300) can be physically implemented in various forms. For example, the prediction device (300) can take the form of a PC, laptop, smart device, server, or data processing-dedicated chipset.

[0118] The prediction device (300) may include an input device (310), a storage device (320), an operation device (330), an output device (340), an interface device (350), and a communication device (360).

[0119] The input device (310) may include an interface device (keyboard, mouse, touch screen, etc.) that receives a certain command or data. The input device (310) may also include a configuration that receives information through a separate storage device (USB, CD, hard disk, etc.). The input device (310) may receive the input data through a separate measuring device or a separate database. The input device (310) may also receive data through wired or wireless communication through a communication device (360). The input device (310) may receive information necessary for performing the aforementioned cybersickness prediction method. The input device (310) may receive a neural network model necessary for performing the aforementioned cybersickness prediction method.

[0120] The input device (310) can receive information about the subject. The input device (310) can receive a neural network model.

[0121] The storage device (320) may be a device that stores certain information. The storage device (320) may store information input through the input device (310). The storage device (320) may store information generated during the operation of the operation device (330). In other words, the storage device (320) may include a memory.

[0122] The storage device (320) can store information necessary for performing the aforementioned cybersickness prediction method. The storage device (320) can store the neural network model necessary for performing the aforementioned cybersickness prediction method. The storage device (320) can store information about the subject. The storage device (320) can store the neural network model.

[0123] The computing device (330) may be a device such as a processor, AP, or a chip embedded with a program that processes data and performs certain operations. The computing device (330) may generate a control signal that controls the prediction device (300). The computing device (330) may generate a control signal that controls the input device (310), storage device (320), output device (340), interface device (350), and communication device (360) included in the prediction device (300).

[0124] The computing device (330) can perform the calculations necessary to perform the aforementioned cybersickness prediction method. The computing device (330) can input the subject's information into a neural network model. Based on the output of the neural network model, the computing device (330) can predict the subject's level of cybersickness. The computing device (330) can preprocess the subject's information.

[0125] The output device (340) may be a device that outputs certain information. The output device (340) may output interfaces required for data processing, input data, analysis results, etc. The output device (340) may be physically implemented in various forms, such as a display, a device that outputs documents, a speaker, etc. The output device (340) may output information stored in the storage device (330). The output device (340) may output information generated during the process of the calculation device (330). The output device (340) may output the result of the calculation performed by the calculation device (330).

[0126] The interface device (350) may be a device that receives certain commands and data from the outside. The interface device (350) may receive a control signal for controlling the prediction device (300). The interface device (350) may output the results analyzed by the prediction device (300). The interface device (350) may receive information and a neural network model necessary for performing the aforementioned cyber motion sickness prediction method from a physically connected input device or an external storage device.

[0127] The communication device (360) may refer to a configuration that receives and transmits certain information via a wired or wireless network. The communication device (360) may perform network communication such as Wi-Fi (Wireless Fidelity), Wi-Fi Direct, Bluetooth, UWB (Ultra-Wide Band), NFC (Near Field Communication), USB (Universal Serial Bus), HDMI (High Definition Multimedia Interface), LAN (Local Area Network), etc. The communication device (360) may receive a control signal necessary to control the prediction device (300). The communication device (360) may transmit the result analyzed by the prediction device (300). The communication device (360) may receive information necessary to perform the aforementioned cybersickness prediction method. The communication device (360) may receive a neural network model necessary to perform the aforementioned cybersickness prediction method.

[0128] The aforementioned cybersickness prediction method can be implemented as a program (or application) including an executable algorithm that can be run on a computer.

[0129] The above program may be provided stored on a non-transitory computer readable medium.

[0130] The above-mentioned temporarily readable medium refers to various RAMs such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous DRAM (Synclink DRAM, SLDRAM), and direct Rambus RAM (DRRAM).

[0131] The above non-transitory readable medium refers to a medium that stores data semi-permanently and can be read by a device, rather than a medium that stores data for a short period of time, such as a register, cache, or memory. Specifically, the various applications or programs described above may be stored and provided in a non-transitory readable medium, such as a CD, DVD, hard disk, Blu-ray disk, USB, memory card, ROM (read-only memory), PROM (programmable read only memory), EPROM (Erasable PROM, EPROM), EEPROM (Electrically EPROM), or flash memory.

[0132] The present embodiment and the drawings attached to the present specification only clearly illustrate a part of the technical idea included in the above-described technology, and it will be obvious that all modified examples and specific embodiments that can be easily inferred by a person skilled in the art within the scope of the technical idea included in the specification and drawings of the above-described technology are included in the scope of the rights of the above-described technology.

Claims

1. A step in which the prediction device receives information about the subject; The step of the prediction device inputting the information of the subject into a neural network model; and A step in which the prediction device predicts the level of cybersickness of the subject based on the output value of the neural network model; The information of the subject includes eye movement data, head movement data, and physiological signals data. The above neural network model is a Transformer-based model that performs multi-head attention, and is a method for predicting cybersickness in virtual reality using multimodal and time-series data.

2. In paragraph 1, A method for predicting cybersickness in a virtual reality using multimodal and time series data, wherein the eye movement data includes at least one of the X, Y, Z coordinates of the gaze direction (x, y, z) of the left / right eyes respectively, the X, Y, Z coordinates of the gaze origin (x, y, z) of the left / right eyes respectively, the pupil diameter (x, y) of the left / right eyes respectively, the X, Y coordinates of the pupil position (x, y) of the left / right eyes respectively, the eye openness (whether the left / right eyes were open) and the X, Y, Z coordinates of the gaze direction with both eyes integrated (x, y, z)).

3. In paragraph 1, A method for predicting cybersickness in virtual reality using multimodal and time series data, wherein the head movement data includes at least one of X, Y, Z coordinates of the head position (head position (x, y, z)) and roll, pitch, yaw of the head rotation (head rotation (roll, pitch, yaw)).

4. In paragraph 1, A method for predicting cybersickness in virtual reality using multimodal and time series data, wherein the biosignal data includes at least one of electrodermal activity, blood volume pulse, and skin temperature.

5. In paragraph 1, The above prediction device further includes a step of preprocessing information of the subject, A method for predicting cybersickness in virtual reality using multimodal and time-series data, wherein the preprocessing includes at least one of a process of removing duplicate processes from data included in the subject information, a process of normalizing the subject information, and a process of Fourier transforming the subject information.

6. In paragraph 1, The above neural network model is a method for predicting cybersickness in virtual reality using multimodal and time-series data, which is a model based on Vision Transformer.

7. In paragraph 1, The above neural network model includes a temporal transformer encoder, A method for predicting cybersickness in virtual reality using multimodal and time-series data, wherein the temporal transformer encoder performs multi-head attention based on the relationship between data at different points in time.

8. In paragraph 1, The above neural network model sequentially includes a spatial transformer encoder and a temporal transformer encoder, The above spatial transformer encoder performs multi-head attention based on the relationship between data included in one point in time, A method for predicting cybersickness in virtual reality using multimodal and time series data, wherein the temporal transformer encoder receives the output value of the spatial transformer encoder and performs multi-head attention based on the relationship between data at different points in time.

9. In paragraph 1, The above neural network model includes a first module, a second module, and a third module, each sequentially including a spatial transformer encoder and a temporal transformer encoder, The above spatial transformer encoder performs multi-head attention based on the relationship between data included in one point in time, The above temporal transformer encoder receives the output value of the above spatial transformer encoder and performs multi-head attention based on the relationship between data at different points in time. The above first module receives eye movement data from the subject's information, The second module receives head movement data from the subject's information, The third module receives biosignal data from the subject's information, A method for predicting cybersickness in virtual reality using multimodal and time series data, wherein the neural network model predicts the degree of cybersickness based on the output values ​​of the first module, second module, and third module.

10. Input device for receiving information from the subject; A computing device that inputs the subject's information into a neural network model and predicts the subject's level of cybersickness based on the output value of the neural network model; and Including a storage device that stores the information of the subject and the neural network model, The information of the subject includes eye movement data, head movement data, and physiological signals data. The above neural network model is a Transformer-based model that performs multi-head attention, and is a device for predicting cybersickness in virtual reality using multimodal and time-series data.

11. In paragraph 10, The above computational device preprocesses the information of the subject, A device for predicting cyber-motion sickness in virtual reality using multimodal and time-series data, wherein the preprocessing includes at least one of a process of removing duplicate processes from data included in the subject information, a process of normalizing the subject information, and a process of Fourier transforming the subject information.

12. In paragraph 10, The above neural network model includes a temporal transformer encoder, The above temporal transformer encoder is a cyber motion sickness prediction device in virtual reality using multimodal and time series data, which performs multi-head attention based on the relationship between data at different points in time.

13. In paragraph 10, The above neural network model sequentially includes a spatial transformer encoder and a temporal transformer encoder, The above spatial transformer encoder performs multi-head attention based on the relationship between data included in one point in time, A cyber motion sickness prediction device in virtual reality using multimodal and time series data, wherein the temporal transformer encoder receives the output value of the spatial transformer encoder and performs multi-head attention based on the relationship between data at different points in time.

14. In paragraph 10, The above neural network model includes a first module, a second module, and a third module, each sequentially including a spatial transformer encoder and a temporal transformer encoder, The above spatial transformer encoder performs multi-head attention based on the relationship between data included in one point in time, The above temporal transformer encoder receives the output value of the above spatial transformer encoder and performs multi-head attention based on the relationship between data at different points in time. The above first module receives eye movement data from the subject's information, The second module receives head movement data from the subject's information, The third module receives biosignal data from the subject's information, A cybersickness prediction device in virtual reality using multimodal and time series data, wherein the neural network model predicts the degree of cybersickness based on the output values ​​of the first module, second module, and third module.

15. A computer-readable recording medium recording a program for executing a method for predicting cybersickness in virtual reality using multimodal and time-series data described in Article 1.

Images