VIRTUAL EXPERIENCE PROVIDING DEVICE, VIRTUAL EXPERIENCE PROVIDING SYSTEM, VIRTUAL EXPERIENCE PROVIDING METHOD, AND PROGRAM

Abstract

To provide a virtual experience providing device, a virtual experience providing system, a virtual experience providing method, and a program capable of enhancing the feeling of traveling together with a travel companion of an experiencer who has a virtual experience of traveling with a local companion.SOLUTION: A virtual experience providing device for providing a person with a virtual travel experience of traveling with a companion at a travel destination includes a video generation unit that generates a virtual reality video on the basis of a local video captured by an imaging device arranged around the companion, and a display control unit that displays the virtual reality video on a display device, and the video generation unit generates the virtual reality video in which the viewpoint of the experiencer and the viewpoint of the companion are different, and the companion is included in the field of vision of the experiencer.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a virtual experience providing device, a virtual experience providing system, a virtual experience providing method, and a program. [Background technology]

[0002] 2. Description of the Related Art Conventionally, there is an experience device that allows a person who is considering travel contents including a travel destination and a travel plan to share a virtual travel experience (for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2018-155887 A Summary of the Invention [Problem to be solved by the invention]

[0004] The experience device described above allows the user to share a virtual experience of traveling with a companion at a destination. However, in a virtual experience, it has been difficult to give the user a sense of actually traveling with the companion on site.

[0005] The present invention has been made in consideration of the above circumstances, and one of its objectives is to provide a virtual experience providing device, a virtual experience providing system, a virtual experience providing method, and a program that can enhance the sense of reality of a person taking a virtual travel experience traveling with a companion as if they are traveling together with the companion. [Means for solving the problem]

[0006] The virtual experience providing device, the virtual experience providing system, the virtual experience providing method, and the program according to the present invention employ the following configuration. (1): A virtual experience providing device according to one embodiment of the present invention is a virtual experience providing device for providing a second user with a virtual travel experience of traveling together with a first user, and includes an image generation unit that generates virtual reality images based on local images captured by imaging devices present around the first user at a travel destination, and a display control unit that displays the virtual reality images on a display device, wherein the image generation unit generates the virtual reality images in which the first user is included in the field of view of the second user.

[0007] (2): In the above aspect (1), an operation content receiving unit that acquires information indicating the content of an operation performed by the second user on an operation unit; The virtual reality image further includes an adjustment unit that adjusts the distance between the first user and the second user in the virtual reality image based on information acquired by the operation content receiving unit.

[0008] (3): In the above aspect (2), the image generation unit generates a virtual reality image in accordance with movement of a moving object operated by the second user, and the display control unit turns the virtual reality image into a still image when the moving object is stopped.

[0009] (4): In any of the above aspects (1) to (3), the image generation unit generates the virtual reality image including a gaze image indicating at least one of the first user's line of sight or point of gaze.

[0010] (5): A virtual experience providing system according to one embodiment of the present invention is a virtual experience providing system comprising any one of the virtual experience providing devices (1) to (4) described above, a moving body on which the second user boards during the virtual travel experience, and a display device that displays the virtual reality image.

[0011] (6): In the above aspect (5), the device further includes at least one of a first movable device that can be contacted by the first user or a second movable device that can be contacted by the second user, and a transmitting / receiving device that transmits and receives at least one of first operation information that is information corresponding to the operation of the first user and is transmitted to the second movable device, or second operation information that is information corresponding to the operation of the second user and is transmitted to the first movable device.

[0012] (7): A virtual experience providing method according to one embodiment of the present invention is a virtual experience providing method, in which a computer of a virtual experience providing device for providing a second user with a virtual travel experience of traveling together with a first user generates virtual reality images based on local images captured by an imaging device present around the first user at a travel destination and displays the virtual reality images on a display device, and the computer generates the virtual reality images in which the first user is included in the second user's field of view.

[0013] (8): A program according to one embodiment of the present invention is a program that causes a computer of a virtual experience providing device for providing a second user with a virtual travel experience of traveling with a first user to generate virtual reality images based on local images captured by an imaging device present around the first user at a travel destination and to execute a process of displaying the virtual reality images on a display device, and causes the computer to generate the virtual reality images in which the first user is included in the second user's field of view. Effect of the Invention

[0014] According to (1) to (8), it is possible to enhance the sense of reality of a person taking a virtual travel experience traveling with a companion. [Brief description of the drawings]

[0015] [Figure 1] 1 is a diagram illustrating an example of a configuration of a virtual experience providing system 1 according to a first embodiment. [Diagram 2]FIG. 2 is a perspective view showing a moving body 210. [Diagram 3] 13 is a bird's-eye view of a participant P2 using the virtual experience facility 200. FIG. [Figure 4] A figure showing an example of a first virtual reality image GA1 displayed on display device 232. [Diagram 5] 13 is a flowchart showing an example of a process of each device 100 at a travel destination. [Figure 6] 13 is a flowchart showing an example of processing of the virtual experience providing device 300. [Figure 7] 1 is a sequence diagram of the virtual experience providing system 1 relating to the first haptic 130 and the second haptic 40. FIG. [Figure 8] FIG. 11 is a diagram illustrating an example of a configuration of a virtual experience providing system 1 according to a second embodiment. [Figure 9] FIG. 2 is a diagram showing an example of a landscape image GC1. [Figure 10] A figure showing an example of a second virtual reality image GA2 generated based on a landscape image GC1. [Figure 11] 13 is a flowchart showing a part of an example of processing of the virtual experience providing device 300. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0016] Hereinafter, with reference to the drawings, an embodiment of a virtual experience providing device, a virtual experience providing system, a virtual experience providing method, and a program of the present invention will be described. The virtual experience providing system is, for example, a system that provides a virtual travel experience of traveling with a companion traveling to a travel destination by using a virtual experience facility where a virtual experience can be performed. The virtual experience providing system provides a user with a virtual travel experience traveling with a companion, such as a celebrity who is difficult to travel with in reality, a guide who is knowledgeable about the geography and history of the travel destination, or a friend who is unable to go on a trip that was supposed to accompany a friend and ends up traveling alone. The companion may be a person other than a real person, for example, a historical figure or character, etc.

[0017] <First embodiment> FIG. 1 is a diagram showing an example of the configuration of a virtual experience providing system 1 of the first embodiment. The virtual experience providing system 1 includes, for example, a travel destination device 100, a virtual experience facility 200, and a virtual experience providing device 300, which are installed at a travel destination. The travel destination device 100 includes a communication device 110, a drone 120, and a first haptic 130. At the travel destination, a companion P1 experiences a real trip, for example, on foot, and at the virtual experience facility 200, a participant P2 experiences a virtual trip in which the companion P1 travels to the travel destination together with the companion P1. The companion P1 is an example of a first user. The participant P2 is an example of a second user.

[0018] The communication device 110 is, for example, a wireless communication module for connecting to a network and directly communicating with other terminal devices, such as the virtual experience providing device 300 of the virtual experience facility 200. The communication device 110 performs wireless communication based on Wi-Fi, DSRC (Dedicated Short Range Communications), Bluetooth (registered trademark), or other communication standards. The communication device 110 is capable of communicating with the virtual experience facility 200 via the network NW.

[0019] The drone 120 is an unmanned aerial vehicle. The drone 120 flies by automatic operation. The drone 120 may also fly by remote control. The drone 120 flies around the companion P1 using a flight control device. The drone 120 is equipped with a camera 122. The camera 122 is present around the companion P1. The camera 122 captures images of the scenery of the travel destination including the companion P1 (hereinafter, local images). The camera 122 outputs the captured images of the scenery of the local area (hereinafter, local images) to the communication device 110. The communication device 110 transmits the output local images to the virtual experience facility 200. The camera 122 is an example of an imaging device.

[0020] The first haptic 130 is, for example, a movable device that is held or worn by the accompanying person P1 and can come into contact with the accompanying person P1. The first haptic 130 is a so-called tactile feedback device. The first haptic 130 detects, for example, a motion toward a person next to the person, such as tapping on the shoulder. The first haptic 130 outputs first motion information based on the detected motion to the communication device 110. The communication device 110 transmits the output first motion information to the virtual experience facility 200.

[0021] The first haptic 130 includes an actuator that reproduces the action of a person next to the person, for example, tapping the shoulder. When second action information, which will be described later, transmitted by the virtual experience facility 200 is output by the communication device 110, the first haptic 130 executes an action corresponding to the transmitted second action information on the companion P1. The first haptic 130 is an example of a first movable device. The communication device 110 is an example of a transmitting / receiving device that transmits and receives the first action information or the second action information.

[0022] The virtual experience facility 200 includes, for example, a moving object 210, a traveling space 220 in which the moving object 210 travels, a head-mounted display worn by the experiencer P2, for example, VR (Virtual Reality) goggles 230, a second haptic 240, and a virtual experience providing device 300. The experiencer P2 wears the VR goggles 230 and the second haptic 240, gets on the moving object 210, and travels within the traveling space 220. The VR goggles 230 are provided with a display device 232 that displays an image that can be viewed by the experiencer P2.

[0023] 2 is a perspective view showing the moving body 210. Hereinafter, the front-rear direction of the moving body 210 may be referred to as the X direction, the lateral direction as the Y direction, and the direction perpendicular to the X direction and the Y direction as the Z direction. The moving body 210 is a moving body that can be boarded by the experiencer P2 in the virtual experience facility 200. The moving body 210 travels within a travel space 220 in response to the operation of the experiencer P2.

[0024] The moving body 210 includes, for example, a base body 211. A control device 212 is housed inside the base body 211. The control device 212 controls the running direction, running speed, and the like of the moving body 210. In addition to the control device 212, a motor, a battery, and the like are housed inside the base body 211. Wheels 213 are attached to the left and right sides of the base body 211, respectively.

[0025] The control device 212 transmits boarding information to the virtual experience providing device 300 when the experience person P2 boards the moving body 210. The control device 212 transmits movement information according to the movement of the moving body 210 to the virtual experience providing device 300. The control device 212 transmits movement start information to the virtual experience providing device 300 when the moving body 210 starts moving.

[0026] When the moving object 210 stops, the control device 212 transmits stop information to the virtual experience providing device 300. When the stopped moving object 210 resumes moving, the control device 212 transmits movement resumption information to the virtual experience providing device 300. When the experience person P2 gets off the moving object 210, the control device 212 transmits dismount information to the virtual experience providing device 300.

[0027] A seating section 214 is provided on the upper part of the base body 211. The seating section 214 is like a cushion on which the experiencer P2 sits. The upper surface of the seating section 214 is a seating surface that supports the buttocks of the experiencer P2 from below. The seating section 214 is supported by the base body 211. The seating section 214 is supported by the base body 211, a first support section 215 arranged in front of the base body 211, and a second support section 216 arranged in the rear of the base body 211. The first support section 215 and the second support section 216 support the seating section 214 so that it can be raised and lowered.

[0028] An operation unit 217 is provided on the side of the seating section 214. The operation unit 217 includes an operator 217A that allows the participant P2 to control the operation of the moving body 210. For example, the operator 217A is an operator for controlling the traveling direction, traveling speed, braking operation, etc. of the moving body 210. The operation unit 217 transmits an electric signal according to the operation of the operator 217A to the control device 212. By operating the operator 217A, the participant P2 can make the moving body 210 travel or adjust the distance between the accompanying person P1 and the participant P2 displayed in the image.

[0029] A load sensor 218 is provided inside the seating section 214. The load sensor 218 detects the position where the experiencer P2 puts his / her weight on the seating section 214. For example, when the experiencer P2 leans his / her body forward, the load sensor 218 detects the load of the experiencer P2 in front of the seating section 214. The load sensor 218 transmits the detected load to the control device 212.

[0030] The control device 212 makes the moving body 210 run based on the direction in which the load sensor 218 detects the load of the experiencer P2. In this way, the experiencer P2 can make the moving body 210 run (move) by operating the operation element 217A or by applying a load in the desired direction. Furthermore, the control device 212 recognizes that the experiencer P2 has boarded and dismounted the moving body 210 based on the load transmitted by the load sensor 218.

[0031] The moving body 210 of the embodiment is a so-called hands-free moving body 210 that can be operated by the experiencer P2 shifting his / her weight, and can be operated by the experiencer P2 operating the operation unit 217, but the operation unit 217 may not be provided. The moving body 210 may be a saddle-type moving body on which the experiencer P2 straddles. The moving body 210 may be a moving body on which the experiencer P2 stands and rides on a step. The moving body 210 may be one that moves by walking using legs instead of wheels.

[0032] The running space 220 is a space large enough for the moving body 210 to move in, for example, a space with a base area of ​​about several meters by several meters. The running space 220 is provided in an appropriate location, such as inside a building. For example, in the case where there is a travel business (travel agency) that sold a travel product to the accompanying person P1, the running space 220 is provided in or near the facilities of the travel business (travel agency).

[0033] The VR goggles 230 display virtual reality images that are based on the on-site images transmitted by the communication device 110 and allow the participant P2 to imagine a virtual space. The VR goggles 230 include a display device 232 that displays the virtual reality images, and a wearing unit that is hooked onto the head of the participant P2 and allows the participant P2 to wear the VR goggles 230 so that the participant P2 can view the display device 232. Instead of the VR goggles 230, MR (Mixed Reality) goggles or AR (Augmented Reality) goggles may be used.

[0034] The second haptic 240 is, for example, a movable device that is held or worn by the experiencer P2 and can come into contact with the experiencer P2. The second haptic 240 is a so-called tactile feedback device, similar to the first haptic 130, and detects, for example, a motion directed toward a person next to the experiencer, such as tapping on the shoulder. The second haptic 240 transmits second motion information based on the detected motion to the virtual experience facility 200.

[0035] Since there is no person actually beside the experiencer P2, the second haptic 240 may generate the second motion information based on the operation unit 217 by the experiencer P2 instead of a motion toward the person beside him / her. For example, when the experiencer P2 tilts the operation element 217A of the operation unit 217 to the right, the second motion information may be generated similar to that generated when a motion of tapping the shoulder of the person beside him / her to the right is detected.

[0036] The second haptic 240 includes an actuator that reproduces the motion of a person riding on the moving object 210, for example, tapping on the shoulder. When first motion information described below is transmitted by the first haptic 130, the second haptic 240 executes a motion corresponding to the transmitted first motion information on the user P2. The second haptic 240 is an example of a second movable device.

[0037] The virtual experience providing device 300 displays images corresponding to the movement of the moving body 210 on the VR goggles 230 so that the participant P2 can experience a virtual trip traveling together with the companion P1. The participant P2 wearing the VR goggles 230 experiences a virtual trip accompanied by the companion P1 through the images displayed on the VR goggles 230 and the movement of the moving body 210.

[0038] The virtual experience providing device 300 includes, for example, a communication unit 310, an image generating unit 320, and a display control unit 330. The image generating unit 320 and the display control unit 330 are realized by, for example, a processor such as a central processing unit (CPU) which is hardware executing a program (software). A part or all of the image generating unit 320 and the display control unit 330 may be realized by hardware (including a circuit unit; circuitry) such as a large scale integration (LSI), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or a graphics processing unit (GPU), or may be realized by cooperation between software and hardware. The program may be stored in advance in a storage device such as a hard disk drive (HDD) or a flash memory (a storage device having a non-transient storage medium), or may be stored in a removable storage medium such as a DVD or a CD-ROM (a non-transient storage medium), and may be installed in the storage device by mounting the storage medium in a drive device. The storage device is realized by a hard disc drive (HDD), a DVD, a random access memory (RAM), a flash memory, or the like.

[0039] The communication unit 310 is, for example, a wireless communication module for connecting to a network or directly communicating with other terminal devices, such as the communication device 110 of each travel destination device 100. The communication unit 310 performs wireless communication based on Wi-Fi, DSRC, Bluetooth (registered trademark), or other communication standards.

[0040] The communication unit 310 receives the on-site image and the first operation information transmitted by the communication device 110. The communication unit 310 outputs the received on-site image to the image generation unit 320. The communication unit 310 receives the movement information, movement start information, stop information, and movement resume information transmitted by the control device 212. The communication unit 310 outputs the received movement information and movement start information to the image generation unit 320, and outputs the stop information and movement resume information to the display control unit 330. During the period from when the stop information is received until when the movement resume information is received, the moving object 210 is stopped.

[0041] When movement start information is output by the communication unit 310, the image generation unit 320 generates a virtual reality image to be displayed on the display device 232 of the VR goggles 230. The image generation unit 320 generates the virtual reality image based on the local image output by the communication unit 310 and the movement information output by the control device 212. In the first embodiment, the virtual reality image is, for example, an image generated based on the local image, and is an image obtained by adding the companion P1 to the real image.

[0042] For example, image generation unit 320 generates a virtual reality image based on the on-site image output by communication unit 310 at the timing when movement start information is transmitted by control device 212. Image generation unit 320 calculates the position of experiencer P2 in the virtual reality image based on the movement information transmitted by control device 212. For example, when a movement signal for moving body 210 to move forward is transmitted, image generation unit 320 generates a virtual reality image that experiencer P2 sees when moving forward in the virtual reality image.

[0043] The person P2 can have a virtual experience of traveling in the scenery shown in the virtual reality image by moving the moving object 210 while watching the virtual reality image. Also, the person P2 can have a virtual travel experience of traveling together with the companion P1 by moving the moving object 210 as if following the companion P1 in the virtual reality image.

[0044] For example, when the experiencer P2 is operating the moving object 210 so as to move together with the accompanying person P1, the image generating unit 320 generates a virtual reality image using the on-site image transmitted by the camera 122. When the experiencer P2 is operating the moving object 210 in such a direction that the experiencer P2 moves away from the accompanying person P1, the image generating unit 320 generates a virtual reality image using, for example, the on-site image transmitted by the camera 122 in the past.

[0045] When generating the virtual reality image, the image generating unit 320 calculates the position of the viewpoint (point of gaze) of the companion P1 based on the local image. The viewpoint of the companion P1 is calculated by estimation based on, for example, the facial direction of the companion P1 included in the local image. The viewpoint of the companion P1 may be indicated by the companion P1 using words or actions. The image generating unit 320 generates the virtual reality image by adding a point of gaze image indicating the position of the viewpoint (point of gaze) of the companion P1 to the local image. The image generating unit 320 notifies the display control unit 330 of information on the generated virtual reality image.

[0046] The display control unit 330 causes the display device 232 to display a virtual reality image based on the information of the virtual reality image notified by the image generation unit 320. The experiencer P2 views the virtual reality image displayed on the display device 232. The image captured by the camera 122 is an image including the companion P1, and is an image captured from a viewpoint (standing point) different from that of the companion P1. The virtual reality image is an image in which the viewpoint (point of gaze) of the experiencer P2 and the viewpoint (point of gaze) of the companion P1 are different, and the companion P1 is included in the field of view of the experiencer P2.

[0047] When movement start information is output by communication unit 310, display control unit 330 causes display device 232 to display a virtual reality image based on the information notified by image generation unit 320. When stop information is output by communication unit 310, display control unit 330 causes display device 232 to display the virtual reality image as a stopped image. The virtual reality stopped image is an image in which the virtual reality image based on the information notified by image generation unit 320 has been stopped. Display control unit 330 also causes display device 232 to display the stopped image when boarding information is received. While moving object 210 is stopped, display control unit 330 causes virtual reality image to be displayed as a stopped image.

[0048] Next, a procedure for using the virtual experience facility 200 will be described. In the virtual experience facility 200, the experiencer P2 puts on the VR goggles 230 before entering the traveling space 220. The experiencer P2, wearing the VR goggles 230, moves to the traveling space 220 and gets on the moving body 210. The experiencer P2 gets on the moving body 210 in the traveling space 220 and drives the moving body 210 while watching the image displayed on the VR goggles 230, thereby experiencing a virtual travel experience in which the experiencer P2 travels together with the companion P1.

[0049] 3 is a bird's-eye view of a person P2 using the virtual experience facility 200. The person P2, who is a user of the virtual experience facility 200, moves the moving body 210 by operating the control 217A of the moving body 210 or by leaning his / her body on the seat 214 in the direction he / she wants to move while wearing VR goggles 230 in the traveling space 220. In the virtual experience facility 200, the person P2 can have a virtual experience of traveling while riding on the moving body 210.

[0050] For example, when the experiencer P2 applies weight forward on the seating section 214, the moving body 210 moves straight ahead. At this time, an image of the experiencer P2 moving forward is displayed as a virtual reality image on the display device 232 of the VR goggles 230 under the control of the display control section 330 of the virtual experience providing device 300. The moving body 210 moves in the running space 220 due to the movement of the experiencer P2, but if the moving body 210 is about to collide with a wall in the running space 220, the moving body 210 will guide the moving body 210 and correct the trajectory of the moving body 210 within a range that is not noticed by the experiencer P2.

[0051] Experiencer P2 aboard moving object 210 experiences a virtual trip while watching images displayed on display device 232. Fig. 4 is a diagram showing an example of first virtual reality image GA1 displayed on display device 232. First virtual reality image GA1 displayed on display device 232 is an image in which a gaze image GB2 showing the gaze of companion P1 and a gaze point image GB3 showing the gaze point, which are generated by image generating unit 320, are added to a scenery image GB1 of the scenery at the travel destination captured by camera 122. Since the scenery image GB1 includes an image of companion P1, the first virtual reality image GA1 also includes a companion image GB4.

[0052] Next, the processing in the virtual experience providing system 1 will be described. First, the processing of each travel destination device 100 will be described. Here, as part of the processing of each travel destination device 100, processing excluding processing related to the first haptic 130 will be described. FIG. 5 is a flowchart showing an example of the processing of each travel destination device 100. The processing of each travel destination device 100 is executed, for example, when the companion P1 arrives at the travel destination and moves around the travel destination.

[0053] When the accompanying person P1 moves, the drone 120 flies around the accompanying person P1 (step S101). While the drone 120 flies around the accompanying person P1, the camera 122 captures local video including the surrounding scenery including the accompanying person P1 (step S103) and outputs it to the communication device 110. The communication device 110 transmits the local video output by the camera 122 to the virtual experience facility 200 (step S105). The drone 120 and the camera 122 continue to fly and capture images, respectively, and the communication device 110 repeatedly executes the process of transmitting the local video at regular time intervals.

[0054] Next, the processing in the virtual experience providing device 300 will be described. Here, as part of the processing of the virtual experience providing device 300, processing excluding processing related to the second haptic 240 will be described. FIG. 6 is a flowchart showing an example of the processing of the virtual experience providing device 300. The processing in the virtual experience providing device 300 is started, for example, after the experiencer P2 visits the virtual experience facility 200 and boards (gets in) the moving body 210 in the traveling space 220.

[0055] In the virtual experience facility 200, first, the moving object 210 starts moving based on the operation of the experiencer P2. When the moving object 210 starts moving, the control device 212 of the moving object 210 transmits movement start information to the virtual experience providing device 300. The virtual experience providing device 300 receives the movement start information via the communication unit 310 (step S201).

[0056] The communication unit 310, which has received the movement start information, receives the local image transmitted by the communication device 110 of each device 100 at the travel destination (step S203), and outputs it to the image generation unit 320. The image generation unit 320 calculates the viewpoint of the companion P1 based on the local image output by the communication unit 310 (step S205). Next, the image generation unit 320 generates a virtual reality image by adding a gaze image GB2 indicating the gaze of the companion P1 and a gaze point image GB3 indicating the position of the gaze point of the companion P1 to the local image (step S207). The image generation unit 320 notifies the display control unit 330 of the generated virtual reality image.

[0057] Next, the display control unit 330 displays the virtual reality image notified by the image generating unit 320 on the display device 232 of the VR goggles 230 (step S209). The experiencer P2 wearing the VR goggles 230 can experience a virtual trip while watching the scenery displayed in the virtual reality image by visually recognizing the image displayed on the display device 232. Furthermore, the virtual reality image shows the on-site companion P1. This enhances the realism of the experiencer P2 who is traveling together with the on-site companion P1. Furthermore, the virtual reality image shows the viewpoint of the companion P1. This makes it easier for the experiencer P2 to feel that he or she is experiencing something in common with the companion P1.

[0058] Next, the display control unit 330 determines whether the moving object 210 is stopped (step S211). If the stop information is transmitted by the moving object 210, the display control unit 330 determines that the moving object 210 is stopped, and if the stop information is not transmitted, the display control unit 330 determines that the moving object 210 is not stopped.

[0059] If it is determined that the moving object 210 is not stopped, the display control unit 330 returns the process to step S203. If it is determined that the moving object 210 is stopped, the display control unit 330 causes the display device 232 of the VR goggles 230 to display a still image (step S213).

[0060] The still image displayed on display device 232 may be, for example, an image obtained by cutting out a frame of a virtual reality image at the timing when communication unit 310 receives the stop information. The still image displayed on display device 232 may be another image, or may be an image obtained by cutting out a frame of a virtual reality image at a timing after a certain time has elapsed since communication unit 310 received the stop information.

[0061] Next, the display control unit 330 determines whether or not the moving body 210 has resumed moving (step S215). If it is determined that the moving body 210 has resumed moving, the display control unit 330 returns the process to step S203. If it is determined that the moving body 210 has not resumed moving, the display control unit 330 determines whether or not the experiencer P2 has dismounted from the moving body 210 (step S217). The display control unit 330 determines whether or not the experiencer P2 has dismounted from the moving body 210 based on whether or not the communication unit 310 has received dismounting information transmitted by the control device 212 of the moving body 210.

[0062] If it is determined that the experiencer P2 has not dismounted from the moving object 210, the display control unit 330 returns the process to step S213 and continues to display the still image. If the display control unit 330 determines that the experiencer P2 has dismounted from the moving object 210, the virtual experience providing device 300 ends the process shown in FIG.

[0063] Next, a description will be given of the processing related to the first haptic 130 and the second haptic 240 in each travel destination device 100 and the virtual experience providing apparatus 300 of the virtual experience providing system 1. Fig. 7 is a sequence diagram of the virtual experience providing system 1 related to the first haptic 130 and the second haptic 240. The processing related to the first haptic 130 and the second haptic 240 is executed independently of the processing shown in Figs. 5 and 6 after the experiencer P2 boards the moving body 210 until the experiencer P2 disembarks from the moving body 210.

[0064] In each device 100 at the travel destination, the first haptic 130 determines whether or not a movement of the companion P1 has been detected (step S301). If it is determined that a movement of the companion P1 has been detected, the first haptic 130 generates first movement information based on the movement of the companion P1 (step S303).

[0065] Next, the first haptic 130 transmits the generated first motion information to the second haptic 240 of the virtual experience facility 300 (step S305). If it is determined in step S301 that the motion of the companion P1 has not been detected, the first haptic 130 advances the process to step S317.

[0066] Meanwhile, in the virtual experience facility 300, the second haptic 240 determines whether or not a motion of the experiencer P2 has been detected (step S307). If it is determined that a motion of the experiencer P2 has been detected, the second haptic 240 generates second motion information based on the motion of the experiencer P2 (step S309).

[0067] Next, the second haptic 240 transmits the generated second motion information to the first haptic 130 of each device 100 at the travel destination (step S311). If it is determined in step S307 that the motion of the experiencer P2 has not been detected, the second haptic 240 advances the process to step S313.

[0068] Next, the second haptic 240 determines whether or not the first motion information transmitted by the first haptic 130 has been received (step S313). If it is determined that the first motion information has been received, the second haptic 240 executes a motion for the experiencer P2 based on the first motion information (step S315). In this way, the second haptic 240 ends the process shown in Fig. 7. If it is determined in step S313 that the first motion information has not been received, the second haptic 240 ends the process shown in Fig. 7 as it is.

[0069] Next, the first haptic 130 determines whether or not the second motion information transmitted by the second haptic 240 has been received (step S317). If it is determined that the second motion information has been received, the first haptic 130 executes an action for the accompanying person P1 based on the second motion information (step S319). In this way, the first haptic 130 ends the process shown in Fig. 7. If it is determined in step S317 that the second motion information has not been received, the first haptic 130 ends the process shown in Fig. 7 as it is.

[0070] The virtual experience providing system 1 of the first embodiment generates a virtual reality image in which the viewpoint of the participant P2 and the viewpoint of the companion P1 are different and the companion P1 is included in the field of view of the participant P2, and displays the generated image on the display device 232 of the VR goggles 230. Therefore, by wearing the VR goggles 230 and experiencing a virtual trip, the participant P2 who is experiencing the virtual trip together with the companion P1 can feel more realistic that he or she is traveling together with the companion P1.

[0071] <Second embodiment> Next, a second embodiment will be described. The virtual experience providing system 1 of the second embodiment is mainly different from the first embodiment in that when a companion is not shown in the on-site image, an image imitating the companion, such as an avatar, is displayed in the virtual reality image. Hereinafter, the virtual experience providing device, virtual experience providing system, virtual experience providing method, and program of the second embodiment will be described, focusing on the differences from the first embodiment.

[0072] 8 is a diagram showing an example of the configuration of the virtual experience providing system 1 of the second embodiment. In the virtual experience providing system 1 of the second embodiment, each travel destination device 100 includes a travel destination moving body 140 and a second camera 142 in addition to a communication device 110, a drone 120, a camera 122, and a first haptic 130. In the following description, the camera 122 mounted on the drone 120 is referred to as the first camera 122, and the image captured by the first camera 122 is referred to as the first local image.

[0073] The travel destination moving body 140 is, for example, a vehicle that the companion P1 boards when traveling to the travel destination. The companion P1 can, for example, rent the travel destination moving body 140 at the travel destination. The travel destination moving body 140 is the same as the moving body 210 provided in the virtual experience facility 200. The travel destination moving body 140 may be a different body from the moving body 210.

[0074] The second camera 142 is mounted, for example, on the tip of the travel destination moving object 140. The second camera 142 captures a local image (hereinafter, the second local image) within a range assumed to be the field of vision of the accompanying person P1. The second camera 142 outputs the captured second local image to the communication device 110. The communication device 110 transmits the output second local image to the virtual experience facility 200.

[0075] The virtual experience providing device 300 provided in the virtual experience facility 200 includes a communication unit 310, an image generating unit 320, a display control unit 330, an operation content receiving unit 340, and an adjustment unit 350. The operation content receiving unit 340 and the adjustment unit 350 are realized, for example, by a processor such as a CPU, which is hardware, executing a program together with the communication unit 310, the image generating unit 320, and the display control unit 330. The communication unit 310, the image generating unit 320, the display control unit 330, the operation content receiving unit 340, and the adjustment unit 350 may be realized by an LSI or the like shown in the first embodiment.

[0076] The control device 212 mounted on the moving body 210 generates information on the operation performed by the experiencer P2 on the operation unit 217 (hereinafter, operation information) by the experiencer P2 on the moving body 210 operating the operation button 217A of the operation unit 217. The content of the operation performed by the experiencer P2 is, for example, an instruction to adjust the distance of the companion P1 from the experiencer P2 in the virtual reality image GA. The control device 212 mounted on the moving body 210 predetermines an instruction for the experiencer P2 according to the operation of the operation button 217A. For example, when the operation button 217A is tilted to the right, the instruction is determined to be an instruction for the companion P1 to move away from the experiencer P2 in the virtual reality image GA, and when the operation button 217A is tilted to the left, the instruction is determined to be an instruction for the companion P1 to move closer to the experiencer P2 in the virtual reality image GA. The control device 212 transmits the generated operation information to the virtual experience providing device 300.

[0077] The communication unit 310 in the virtual experience providing device 300 receives operation information transmitted by the control device 212, in addition to the processes described in the first embodiment. The communication unit 310 outputs the received operation information to the operation content receiving unit 340.

[0078] The image generating unit 320 determines whether or not the companion P1 is shown in the local image (the first local image or the second local image) transmitted by each travel destination device 100. If the companion P1 is shown in the local image, the image generating unit 320 generates a virtual reality image including the viewpoint of the companion P1 in the local image.

[0079] When the accompanying person P1 is not shown in the on-site image, the image generating unit 320 generates an image that resembles the accompanying person, for example, an avatar, in the on-site image and includes it, and generates a virtual reality image including the viewpoint of the accompanying person P1. The accompanying person P1 cannot directly see his / her own appearance, such as his / her face. For this reason, the viewpoint of the virtual reality image including the avatar is different from the viewpoint of the accompanying person P1, and therefore is different from the viewpoint of the experiencer P2 who views the virtual reality image and the viewpoint of the accompanying person P1.

[0080] 9 is a diagram showing an example of a landscape image GC1. Since the second camera 142 is disposed at the tip of the travel destination moving body 140, it can capture the landscape image GC1 that is assumed to be within the field of vision of the companion P1. However, the companion P1 himself is not included in the image. Therefore, the image generating unit 320 generates a virtual reality image based on the landscape image GC1, in which the companion P1 is displayed together with the landscape image GC1.

[0081] FIG. 10 is a diagram showing an example of a second virtual reality image GA2 generated based on a landscape image GC1. The second virtual reality image GA2 is an image in which a gaze image GC2 showing the gaze of the companion P1, a gaze point image GC3 showing the gaze point of the companion P1, and an avatar GC4 imitating the companion P1, generated by the image generating unit 320, are added to the landscape image GC1 of the landscape of the travel destination captured by the second camera 142. By experiencing a virtual trip while watching a virtual reality image including the companion, the experiencer P2's sense of reality of being together with the local companion P1 can be enhanced. However, since the companion P1 is not shown in the landscape image GC1, it is difficult to enhance the sense of reality of the experiencer P2 by simply generating a virtual reality image by adding the gaze image GC2 and the gaze point image GC3 to the landscape image GC1.

[0082] Therefore, the image generator 320 generates the second virtual reality image GA2 by adding the gaze image GC2, the gaze point image GC3, and an avatar GC4 to the scenery image GC1. By displaying the avatar GC4 in the second virtual reality image GA2, the experiencer P2 can feel more real when he or she is traveling with the companion P1 at the site.

[0083] The avatar GC4 included in the second virtual reality image GA2 may be any kind of avatar. For example, the avatar GC4 may be an illustration or photograph of the companion P1, or may be a cartoon character. Alternatively, the avatar GC4 may be a character imitating a celebrity (famous person) such as an entertainer or an athlete, or may be an illustration or photograph of such a person. The same avatar GC4 may always be displayed, or the displayed avatar may change depending on the display time, the current time, etc. Furthermore, the avatar GC4 may be selectable by the experiencer P2.

[0084] The operation content receiving unit 340 receives operation information transmitted by the control device 212. The operation content receiving unit 340 receives an instruction from the experiencer P2 according to the content of the operation performed by the experiencer P2 (hereinafter, operation content) based on the received operation information. The operation content receiving unit 340 notifies the adjustment unit 350 of the received operation content. The instruction based on the operation information is, for example, an instruction to adjust the distance between the companion P1 and the experiencer P2, such as an instruction to move the avatar GC4 away from the experiencer P2 or closer to the experiencer P2 in the second virtual reality image GA2.

[0085] Adjustment unit 350 adjusts the display position of avatar GC4 in second virtual reality image GA2 based on the instruction of the adjustment information notified by operation content receiving unit 340. For example, when the instruction based on the operation information is an instruction to move avatar GC4 away from user P2 in second virtual reality image GA2, adjustment unit 350 moves the display position of avatar GC4 from the side where user P2 is assumed to be located to the opposite side.

[0086] Next, a description will be given of the processing of the virtual experience providing device 300 of the second embodiment. Fig. 11 is a flowchart showing a part of an example of the processing of the virtual experience providing device 300. After receiving an on-site image (step S203), the virtual experience providing device 300 determines whether or not the companion P1 is shown in the on-site image (step S251).

[0087] The determination of whether the accompanying person P1 is shown in the local video is performed based on whether the local video is the first local video or the second local video. When the local video is the first video, the video generating unit 320 determines that the accompanying person is shown in the local video, and when the local video is the second video, the video generating unit 320 determines that the accompanying person is not shown in the local video. The determination of whether the accompanying person P1 is shown in the local video may be performed by other means. For example, the accompanying person may be searched for by performing a video analysis on the local video.

[0088] If it is determined that the accompanying person P1 is shown in the local image, the image generating unit 320 proceeds to step S205. If it is determined that the accompanying person P1 is shown in the local image, the image generating unit 320 generates an avatar of the accompanying person P1 (step S253) and proceeds to step S205. Next, the image generating unit 320 calculates the viewpoint of the accompanying person P1 (step S205) and generates a virtual reality image (step S207), after which the display control unit 330 causes the display device 232 to display the virtual reality image.

[0089] Next, the operation content receiving unit 340 determines whether or not an instruction from the experiencer P2 based on the operation information transmitted by the control device 212 has been received (step S255). If the operation content receiving unit 340 determines that an instruction from the experiencer P2 based on the operation information transmitted by the control device 212 has been received, the display control unit 330 moves the display position of the avatar in the direction according to the instruction from the experiencer P2 (step S257). If the operation content receiving unit 340 determines that an instruction from the experiencer P2 based on the operation information transmitted by the control device 212 has not been received, the display control unit 330 proceeds directly to step S211. Thereafter, the virtual experience providing device 300 executes the same process as in the first embodiment, and ends the process shown in FIG. 11.

[0090] The virtual experience providing system 1 of the second embodiment has the same effect as the virtual experience providing system 1 of the first embodiment. The virtual experience providing system 1 of the second embodiment can further adjust the distance between the companion P1 and the experiencer P2 in the virtual reality image. Therefore, the experiencer who is experiencing a virtual travel experience traveling with the companion can feel more realistic about traveling together with the companion.

[0091] In the second embodiment, the image generating unit 320 of the virtual experience providing device 300 may generate an image with an avatar added as a virtual reality image based on the image always transmitted by the second camera 142. In the second embodiment, the virtual reality image is generated every time both the first local image (image including the companion P1) and the second local image (image not including the companion P1) are transmitted, but a condition may be set to generate the virtual reality image based on the local image only when either the first local image or the second local image is transmitted. For example, if the experiencer P2 desires to take a virtual trip with an avatar as his / her companion, the virtual reality image may be generated only from (the image not including the companion P1). The image generating unit 320 may also generate a virtual reality image in which the companion included in the first local image is replaced with an avatar.

[0092] In the above second embodiment, the first camera 122 and the second camera 142 are fixed to the drone 120 or the travel destination moving body 140 and capture images of fixed positions relative to the drone 120 or the travel destination moving body 140, but the second camera 142 may be a camera whose capture position is movable, for example, a 360° camera. In addition, when adjusting the distance between the companion P1 and the experiencer P2, the display position of the avatar is moved, but the distance between the companion P1 and the experiencer P2 may be adjusted by, for example, moving the capture position of the on-site image including the companion P1.

[0093] The above describes the form for carrying out the present invention using an embodiment, but the present invention is not limited to such an embodiment, and various modifications and substitutions can be made within the scope that does not deviate from the gist of the present invention. [Explanation of symbols]

[0094] 1. Virtual experience provision system 40…2nd Haptech 100…Destination equipment 110...Communication equipment 120…Drone 122...Camera (1st camera) 130…First Haptech 140... Travel destination mobile object 142…Second camera 210...Mobile 217...Operation unit 217A...Operator 218...Load sensor 220…Travel space 230…VR goggles 232...Display device 240…2nd Haptech 300…Virtual experience providing device 310…Communications Department 320…Image generation unit 330...Display control unit 340…Operation content reception section 350...Adjustment section GA…Virtual reality video GA1…First virtual reality video GA2: 2nd virtual reality video GB1, GC1…Landscape video GB2, GC2… Line of Sight Video GB3, GC3…focus point video GB4…Companion video GC4…Avatar NW…Network P1…Accompanying person P2: Participant

Claims

1. A virtual experience providing device for providing a second user with a virtual travel experience traveling together with a first user, comprising: an image generating unit that generates a virtual reality image based on a local image captured by an imaging device present around the first user at a travel destination; A display control unit that causes a display device to display the virtual reality image; an operation content receiving unit for acquiring information indicating the content of an operation performed by the second user on an operation unit; an adjustment unit that adjusts a distance between the first user and the second user in the virtual reality image based on information acquired by the operation content reception unit; The image generation unit generates the virtual reality image in which the first user is included in a field of view of the second user. A device providing virtual experiences.

2. The image generator generates a virtual reality image in response to a movement of a moving object operated by the second user; The display control unit sets the virtual reality image as a still image while the moving object is stopped. The virtual experience providing device according to claim 1 .

3. The image generating unit generates the virtual reality image including a gaze image indicating at least one of a gaze or a gaze point of the first user. The virtual experience providing device according to claim 1 or 2.

4. The virtual experience providing device according to any one of claims 1 to 3; An imaging device that captures the on-site image; A moving object on which the second user boards during the virtual travel experience; A display device for displaying the virtual reality image. Virtual experience provision system.

5. A virtual experience providing device; an imaging device that captures local video captured by imaging devices present around a first user at a travel destination; A moving object on which a second user boards during a virtual travel experience in which the second user travels together with the first user; a display device that displays a virtual reality image generated based on the on-site image captured by the imaging device; and At least one of a first movable device accessible to the first user and a second movable device accessible to the second user; a transmitting / receiving device for transmitting / receiving at least one of first motion information corresponding to a motion of the first user and transmitted to the second movable device, and second motion information corresponding to a motion of the second user and transmitted to the first movable device; The virtual experience providing device includes: A virtual experience providing device for providing the virtual travel experience to the second user, an image generating unit that generates the virtual reality image; A display control unit that causes the display device to display the virtual reality image, The image generation unit generates the virtual reality image in which the first user is included in a field of view of the second user. Virtual experience provision system.

6. A computer of a virtual experience providing device for providing a virtual travel experience of traveling with a first user to a second user, generating a virtual reality image based on a local image captured by an imaging device present around the first user at the travel destination; Displaying the virtual reality image on a display device; acquiring information indicating the content of an operation performed by the second user on an operation unit; adjusting a distance between the first user and the second user in the virtual reality image based on the information; The computer includes: generating the virtual reality image including the first user in a field of view of the second user; How to deliver virtual experiences.

7. A computer of a virtual experience providing device for providing a second user with a virtual travel experience traveling together with a first user, generating a virtual reality image based on a local image captured by an imaging device present around the first user at a travel destination; Executing a process to display the virtual reality image on a display device; causing an operation unit to acquire information indicating the content of an operation performed by the second user; adjusting a distance between the first user and the second user in the virtual reality image based on the information; The computer includes: generating the virtual reality image including the first user in a field of view of the second user; program.

Images