A brain wave acquisition device measures brain waves, converts them into numerical values, replaces them with moving power, and operates a doppelgänger movement system that controls the operator's doppelgänger in a three-dimensional virtual space on the Internet without manual operation.

Abstract

To provide a system that realizes three-dimension movement control of an operator's alter ego in a VR space by eliminating devices that are operated by hand such as controllers in the three-dimension virtual space (hereinafter referred to as VR space) on the Internet.SOLUTION: In a system, a brain wave acquisition device and a VR goggles are connected by bluetooth, and brain wave information is transmitted to the VR goggles in real time, and a movement of one's own alter ego in the three-dimension virtual space is controlled with a combination of three data of concentration, line-of-sight tracking function, and a gyro sensor, and thus anyone can easily move smoothly in the three-dimension virtual space.SELECTED DRAWING: Figure 5

Description

Technical Field

[0001] The present invention relates to a system that eliminates devices operated by hand, such as a controller, within a three-dimensional virtual space (hereinafter referred to as a VR space) on the Internet and realizes three-dimensional movement control of an operator's avatar within the VR space.

Background Art

[0002] There is Second Life, where Linden Lab in the United States built the world's first virtual city on VR in the early 2000s and opened it for users around the world to use (Figure 1). It was a mechanism for creating one's own avatar, called an avatar, and communicating with users around the world. The movement was performed using the up, down, left, and right cursor keys on the personal computer, and it was only a primitive operation method. Although it was possible to move in space away from the floor in the past, it is now impossible.

[0003] Five types of brain waves (alpha wave, beta wave, gamma wave, theta wave, delta wave) can be measured in real time by a brain wave acquisition device (Figure 2), digitized, and data can be transmitted to each Bluetooth-compatible device (e.g., smartphone) using short-range wireless means (Bluetooth method). The relationship between the explanations of the five types of brain waves and the mental state will be explained in Figure 3.

[0004] By wearing a VR goggles (1 in Figure 4) on the head, a scenery similar to the real world can be expressed within the VR space. With the development of the system, a virtual city space has been constructed, and it has become possible to take a walk in the city, stop by a store, talk to a store clerk, and do shopping using a dedicated controller (2 in Figure 4), just like in reality. By constructing a school classroom environment, it is becoming possible to take classes in a form similar to taking a normal lecture.

Summary of the Invention

Problems to be Solved by the Invention

[0005] Starting from Second Life which represents a virtual city on the two-dimensional plane screen of a personal computer (Figure 1), it has become possible to realistically represent attractive content in the VR space with a VR goggle set (Figure 4), but manual operation is still essential. In sports where multiple competitors compete in an online game on the Internet, so-called e-sports, the buttons of a dedicated controller are moved up, down, left, and right, and multiple buttons are operated simultaneously to play.

[0006] As an issue, 1. When causing some action to the avatar in the VR space, it is essential to operate the keys on the personal computer or manually operate the dedicated controller attached to the VR goggles. Therefore, people with hand disabilities, bedridden patients, the elderly, and children cannot easily operate. The operation is complicated and has a high difficulty level to use without stress.

[0007] 2. Conventional e-sports mainly consist of fighting games, and the victory or defeat is determined by quickly operating the controller and buttons for the fighting characters, so the range of competitors is limited. Since the superiority or inferiority is determined by the reflex nerve, the competitors are limited to a specific young age group. Therefore, e-sports have not expanded to games with various competition formats.

Means for Solving the Problem

[0008] <Function 1> A function of connecting (pairing) an electroencephalogram acquisition device and a VR goggle and transmitting electroencephalogram measurement data in real time by Bluetooth wireless method. (Figure 5)

[0009] <Function 2> Substitute the measured values of the five types of electroencephalograms (alpha wave, beta wave, gamma wave, theta wave, delta wave) obtained in Function 1 into the following calculation formula as input data to obtain the movement speed 3D virtual space movement speed = Measured value of alpha wave × α + Measured value of beta wave × β + Measured value of gamma wave × γ + Measured value of theta wave × θ + Measured value of delta wave × Δ ※α, β, γ, θ, Δ are fixed parameter values that determine the weighting of each brain wave measurement value. An app for monitoring the brain wave state is provided by the manufacturer. The system operating within the VR goggles can be developed using the development kit provided by the manufacturer. Example of a measurement screen in a dedicated app for 5 types of brain waves (Figure 6). By weighting and summing the various brain wave measurement values, concentration optimized for the movement speed is obtained and defined as the movement speed when moving in space within the VR goggles. In particular, low beta waves form the basis of concentration. As a specific example in the real world, there is an example of a racing car that controls the throttle with brain waves (Figure 8).

[0010] <Function 3> The movement direction is determined by the difference information between the current location data information of the operator's avatar within the VR goggles and the line-of-sight marker (Figure 9).

[0011] <Function 4> The gyro sensor within the VR goggles detects the displacement of the movement of the head, and switches the video projected within the VR goggles. For example, when turning to the right, the video on the right side within the VR goggles moves to the front (Figure 15). The dedicated controller attached to the VR goggles in Figures 2 of Figure 4 and Figure 16 is not used except for operations such as turning on the power and selecting a menu.

Advantages of the Invention

[0012] When performing various activities within the VR space, it is necessary to skillfully use the buttons, triggers, and sticks using the dedicated controller in Figure 16. In the present invention, the conventional hardware-based controller operations are used only minimally (turning the power on and off, selecting operations from the menu). Regarding movement within the VR space, movement within the VR space is achieved only by the movement of the head, the movement of the eyes, and concentration. By eliminating the method of movement by hand controller operation, smooth movement similar to the real world can be realized in the present invention. The influence of physical handicaps due to healthy individuals, disabled individuals, gender, age, and nationality does not exist within the VR space. Based on the present invention, a true equal virtual reality social life becomes possible.

Brief Explanation of Drawings

[0013]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Mode for Carrying Out the Invention

Example

[0014] The example is shown in the form of a process flowchart in Figure 7. 1. By wirelessly connecting the electroencephalogram acquisition device and the VR goggles via Bluetooth, the transmission and reception of electroencephalogram measurement values are prepared. 2. Set the VR goggles and the electroencephalogram acquisition device on the operator's head. By enhancing concentration, electroencephalogram measurement values are obtained and the speed for spatial movement is obtained. 3. Measure each electroencephalogram with the electroencephalogram acquisition device and transmit it to the VR goggles in real time. 4. Input various electroencephalogram measurement values into the calculation formula program built into the VR goggles, and calculate the output value as the three-dimensional space movement speed. 5. The electroencephalogram changes at any time, and the reception~conversion~update is repeated at any time in the program. The spatial movement speed also changes accordingly at any time. 6. Determine the moving direction from the current position information (three-dimensional information of X, Y, Z) of the avatar and the line-of-sight marker information (three-dimensional information of X, Y, Z) (update at any time). 7. The avatar starts moving inside the VR goggles based on the moving direction and the moving speed information. 8. When the head is moved up, down, left, or right, the internal image inside the VR goggles also follows and switches. If the head is rotated 90 degrees to the right, the right side of the internal image is projected onto the front. 9. When the concentration falls below the specified value, the avatar inside the VR goggles falls towards the bottom surface of the three-dimensional space. 10. When the concentrated force is below the specified value, the avatar in the VR goggles comes to a standstill on the bottom surface of the three-dimensional space.

[0015] ※1 Three-dimensional space movement speed = Measured value of alpha wave × α + Measured value of beta wave × β + Measured value of gamma wave × γ + Measured value of theta wave × θ + Measured value of delta wave × Δ The parameters α, β, γ, θ, and Δ are weighting parameters. ※2 The specified value is the minimum value for flying in the VR space by overcoming virtual gravity. ※3 The three-dimensional space movement speed is determined by converting it into an absolute value such as an electrical signal (decibel) that can be compared with others, and using the gaze tracking function built into the VR goggles. ※4 The movement of the head is detected by the gyro sensor built into the VR goggles to control the projected image. If the head is turned 90 degrees to the right, the image in the VR goggles also switches to the scenery on the right side.

[0016] As shown in Figure 5, Bluetooth pairing can be performed between the two devices, and brain wave data can be transmitted to the VR goggles using the sample software attached to the device.

[0017] Using the VR goggles development kit, create a program to implement the following functions, perform operation verification, and build Prototype Version 1.

[0018] Perspectives for verification · After receiving the brain wave data from the brain wave acquisition device, the three-dimensional space movement speed can be obtained in real time by putting it into the above-mentioned calculation formula. Since the brain wave called "SMR wave (Sensory Motor Rhythm)" in the low beta wave range of 12 Hz to 15 Hz in a state where the balance between relaxation and concentration is achieved serves as the basis for concentration, it can be treated with emphasis. Other brain waves can be utilized as values that reduce the movement speed or apply brakes. The goodness of the reaction of the turning property during movement can also be expressed by optimizing each brain wave. The displacement from the constant speed movement is gently displaced by applying the law of inertia in the real world. Perform tuning of the weighting parameters to determine the optimal values. Rather than a single brainwave, an optimized solution that combines each brainwave can create an illusion as if one is freely flying in the real world in person.

[0019] Regarding Figure 9, The determination of the moving direction uses the eye-tracking function installed in the VR goggles. The avatar moves towards the eye marker. The speed at which it is attracted to the marker and the sensitivity of the reaction are determined by the brainwave weighting parameters. By moving the head, the landscape video projected inside the VR goggles also changes. If you turn around, the video rotates 180 degrees, and if you concentrate on increasing the three-dimensional space movement speed, you move in the opposite direction.

[0020] Once the individual tests are completed, create a program in which the avatars of multiple operators are mixed in a common VR space and construct the prototype version 2. The verification viewpoints are · Can one distinguish one's own avatar and others' avatars in the video inside the VR goggles? · Can one recognize the relative speed of one's own avatar and others' avatars with the function of capturing the position information (function similar to GPS) inside the VR space? · Create a circuit course inside the VR space to see if one can fly as expected and compete with others' avatars, and conduct a trial race to finely adjust the system. · Tune so that there is a relative difference in the three-dimensional space movement speed between oneself and others. Convert the calculated strength of concentration into the absolute value of the electrical signal in decibels to express the difference in relative speed.

Industrial Applicability

[0021] The construction of cities similar to the real world inside the VR space has become a global trend. Many services similar to or better than the real world will be released in the future without physical movement. Case 1 A virtual city is constructed within the VR space, allowing users to move on the ground solely by brainwaves, eye movements, and head movements without manual operation when strolling through the streets, shopping at various stores, or chatting with others, just like in the real world. Even for those with difficulty walking in the real world, bedridden patients, people with hand disabilities, or patients with complete paralysis, they can perform the same activities as healthy people within the VR space. Set the specified value in Figure 7 low for walking and adjust it so that anyone can walk on the ground.

[0022] Case No. 2 Form partnerships with various companies engaged in virtual city development and launch a service for sightseeing flights over the virtual city in the VR space as one of the various contents being developed within the city. Night flights to enjoy the night view are also possible. Follow the business model of sightseeing flights in the real world and conduct business in the form of charging a usage fee for the flight service per hour. If there are many buildings in the virtual city and it takes a long time to move on foot, using this service can save travel time (Figure 10).

[0023] Case No. 3 Propose a new sports competition in which a certain circuit course is defined within the VR space and multiple people race and fly to determine the winner. Virtual Sports, hereinafter referred to as V-Sports. In the real world, there is a drone with a camera as a motif for V-Sports. A system that captures the front image with a camera attached to the drone using a dedicated goggle while operating it is already commercially available. It gives the feeling of operating as if riding on the drone. There is no goal of improving the operation technique in solo drone flight, so it gets boring. To prevent boredom, races using drones equipped with the above-mentioned equipment are being held around the world and are very popular. Professional racers have also emerged. The operation of the drone is different from that of a radio-controlled car. It is necessary to control the movement in three-dimensional space, and when racing at high speed, advanced operation techniques are required. It has produced many competitors who seek advanced technologies by competing along established regulations. It has successfully evolved into a well-known sports competition where recognized organizations are born, a skill training system is established, and a prize system emerges. Virtual reality V-sports follow the trajectory of the above drones, and the speed of development and expansion exceeds that of real-world drone races.

[0024] When the value obtained from the three-dimensional space movement speed calculation formula can be maintained above a specified value for a certain period of time, the recognized organization certifies the person as a V-sports competitor. The weighted parameter values are determined and notified by the organizer immediately before the race so that they are not advantageous only to specific competitors. Conversely, there could also be a rule-making that allows each competitor to have parameter values optimized for their brain wave characteristics. It is similar to the concepts of the stipulated performance and free performance in figure skating competitions. It imitates the concept of the regulations of real-world F1 Grand Prix. The moving direction is determined using the gyro sensor and eye-tracking function attached to the VR goggles. When you want to apply the brakes, you can also apply the brakes by controlling the brain waves to reduce the three-dimensional space movement speed, or by turning 180 degrees in the opposite direction to increase the output. The competitor's avatar is set in the center of the image inside the VR goggles, so you can control it to fly within the specified course by moving your line of sight and head. Alternatively, implement a function that projects the perspective of the avatar itself as an image inside the VR goggles and make it a selection system. Make it possible to select whether to display or not display the line-of-sight marker.

[0025] Course setting example No. 1 A time trial race where pylons are installed in the VR space and the lap time is competed. Inherit the regulations of air race competitions in the real world (Figure 11).

[0026] Course Setting Example 2 Install obstacles in the middle of the circuit course to make the competition more story-driven. Install gravity spots that cannot be identified in the middle. Entering this area of space will increase the gravity, causing the player to deviate from the course and lower their ranking. Conversely, install warp spots. Entering this area of space allows for instant teleportation to the front of the course, enabling the player to raise their ranking (Figure 12). In this way, devices that are impossible to achieve in the real world can be realized.

[0027] Case 4 Establishment of a V-sports competition organization and expansion into a new sports business. Create a revenue structure by attaching sponsors through categorizing from amateur to professional in a pyramid structure. Imitate the mechanism of real-world sports businesses. Since victory or defeat depends on the mental condition of humans, it is difficult to predict, and V-sports races can become the subject of betting. The same concept as bicycle racing.

[0028] Regarding the value and prospects of V-sports advocacy. Since the competition method is simple and anyone can participate, the number of competitors will increase. Healthy people, people of all ages and genders, and people with disabilities can participate in V-sports from anywhere. From a hospital bed, from a remote island, from overseas, without language barriers, anyone can compete as long as they are human. At present, it is impossible to predict who can control their concentration the most. Just imagining whether it will be an e-sports active gamer, the top athlete in each field, a chess player, a high-deviation-value examinee, whether there will be an age gap, whether a certain country will be strong, or perhaps a child, etc. is already exciting. It is easy to imagine that it can activate the world. With sponsors, the race pattern is relayed by the media, fans follow professional players, multiple competing teams emerge, and when a league system is established, a revenue structure similar to that of real professional sports will be born. It is also realistically possible to provide life options for young people who cannot find their purpose, or for children from poor families to become professional athletes and earn a large amount of money. There is a rumor in the world that e-sports will become an Olympic event in the future, and V-sports may also be adopted as the same event. As an advantage, V-sports have a wide range of participants, and there is no distinction between athletes such as the able-bodied and the disabled. Therefore, there is no distinction between the Olympics and the Paralympics, and athletes are integrated. The ultimate goal advocated by the Olympic Charter is achieved.

[0029] Expansion of the interpretation of existing e-sports, improvement of social status, and professional sports. The utilization of concentration is an unprecedented field with profound depth. It can also exercise concentration, and derivative effects on work and study are expected. It becomes a role model for the full utilization of brainwaves.

[0030] There are things that can be done and things that cannot be done in the real and virtual worlds respectively (Figure 13). Things that can be done in the real world but not in the virtual world include Example: Eating, muscle building training Things that can be done in both the real and virtual worlds include Example: Meetings, classes, shopping Things that cannot be done in the real world but can be done in the virtual world include Example: Flying through space in person. The operation of the controller attached to the VR goggles is not required, and one can freely move (fly) in the virtual three-dimensional space just by thinking with brainwaves, vision, and head movements. A similar flying scene is shown in the movie "The Matrix", and it is possible to embody the scene of the virtual reality world at present (Figure 14).

Claims

【Claim 1】 An avatar movement system that moves an operator's avatar in a 3D virtual space on the Internet, comprising an electroencephalogram acquisition device and VR goggles, defining numerical data obtained from five types of electroencephalogram measurement values of alpha waves, beta waves, gamma waves, theta waves, and delta waves obtained from the electroencephalogram acquisition device as the 3D virtual space movement speed, determining the movement direction based on the difference information between the current location data of the avatar obtained from the VR goggles and the gaze tracking marker, the gyro sensor in the VR goggles detecting the displacement of the movement of the head, instantaneously changing the 360-degree projection screen video in the VR goggles based on the displacement information, An avatar movement system characterized by moving the avatar only by electroencephalogram without manual operation in a 3D virtual space, substituting the five types of electroencephalogram measurement values of the electroencephalogram acquisition device into the following calculation formula, defining the calculation result as the 3D virtual space movement speed, obtaining with a calculation formula where 3D virtual space movement speed = alpha wave measurement value × α + beta wave measurement value × β + gamma wave measurement value × γ + theta wave measurement value × θ + delta wave measurement value × Δ, An avatar movement system, wherein α, β, γ, θ, and Δ are fixed parameter values that determine the weighting of each electroencephalogram measurement value.

Images