Virtual experience providing device, virtual experience providing system, virtual experience providing method, and program

Abstract

To provide virtual experience providing device, system, method, and program that use a simple configuration to generate an image from an experiencer's perspective that is different from the companion's perspective.SOLUTION: A virtual experience providing device includes an image generating unit that generates a virtual image having a viewpoint at a position different from the position of the 360-degree camera on the basis of an image captured by a 360-degree camera 110 attached to a moving object. The generated virtual image includes an avatar of the moving object. The image generating unit generates the virtual image as a real-time video using a predetermined distance from the 360-degree camera and a distance between the position of the 360-degree camera and the different position.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a virtual experience providing device, a virtual experience providing system, a virtual experience providing method, and a program. [Background technology]

[0002] Patent Document 1 discloses a virtual experience providing system that provides a user with a virtual travel experience of traveling together with a companion to a travel destination.

[0003] The above-mentioned virtual experience providing system generates a virtual reality image that includes the traveling companion in the traveling companion's field of view based on local images captured by imaging devices located around the traveling companion at the destination, and displays the image on the VR goggles worn by the traveling companion. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2022-142296 A Summary of the Invention [Problem to be solved by the invention]

[0005] In the above virtual experience provision system, the drone-mounted camera (imaging device around the accompanying person) captures images of the actual location, making it possible to generate images from the experiencer's perspective, which differs from the accompanying person's perspective. This creates problems in that the overall system configuration is complex and expensive. In addition, when considering the safety of drones, it can be said that operation is highly difficult.

[0006] The present invention has been made in consideration of the above problems, and has an object to make it possible to generate an image from an experiencer's perspective, which is different from the companion's perspective, using a simple configuration. [Means for solving the problem]

[0007] According to one aspect of the present invention, a virtual experience providing device is provided, which includes an image generation unit that generates a virtual image having a viewpoint at a position different from the position of a 360-degree camera mounted on a moving object, based on an image captured by the 360-degree camera. Effect of the Invention

[0008] According to one aspect of the present invention, a virtual image having a viewpoint different from that of the 360-degree camera is generated based on an image captured by a 360-degree camera mounted on a moving object, so there is no need to provide a camera for capturing an image having a viewpoint away from the 360-degree camera, i.e., an image from an experiencer's viewpoint different from the companion's viewpoint. Therefore, an image from an experiencer's viewpoint different from the companion's viewpoint can be generated using a simple configuration. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram illustrating a virtual experience providing device according to a first embodiment of the present invention. [Diagram 2] FIG. 2 is a diagram for explaining a virtual image having a viewpoint different from the position of the 360-degree camera. [Diagram 3] FIG. 3 is a flowchart illustrating an example of the virtual video generation process. [Figure 4] FIG. 4 shows a state in which a mesh has been created in an image captured by a 360-degree camera. [Diagram 5] FIG. 5 is a diagram for explaining the size of each mesh as viewed from the position of the 360-degree camera. [Figure 6] FIG. 6 is a diagram for explaining mesh size correction. [Figure 7] FIG. 7 is a diagram for explaining the size of each mesh after size correction. [Figure 8] FIG. 8 is a diagram for explaining the position of each mesh after the position correction. [Figure 9] FIG. 9 is a diagram for explaining adjustment of the viewing direction. [Figure 10] FIG. 10 is a diagram illustrating a virtual experience providing system according to the second embodiment of the present invention. [Figure 11] FIG. 11 is a schematic diagram showing an example of the configuration of a virtual experience providing system according to the second embodiment of the present invention. [Figure 12] FIG. 12 is a flowchart showing an example of a procedure in which the virtual experience providing system according to the second embodiment of the present invention displays a virtual image on the VR goggles. [Figure 13] FIG. 13 is a diagram illustrating a virtual experience providing system according to the third embodiment of the present invention. [Figure 14] FIG. 14 is a diagram for explaining the augmented reality processing. [Figure 15] FIG. 15 is a schematic diagram showing an example of the configuration of a virtual experience providing system according to the third embodiment of the present invention. [Figure 16] FIG. 16 is a flowchart showing an example of a procedure in which the virtual experience providing system according to the third embodiment of the present invention displays an AR image on the AR glasses. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] First Embodiment Hereinafter, a virtual experience providing device 100 according to a first embodiment of the present invention will be described with reference to the accompanying drawings.

[0011] Fig. 1 is a diagram for explaining a virtual experience providing device 100. Fig. 2 is a diagram for explaining a virtual video V having a viewpoint at a position different from the position of the 360-degree camera 110.

[0012] 1, the virtual experience providing device 100 includes an image generating unit 10. The image generating unit 10 generates a virtual image V having a viewpoint at a position different from the position of the 360-degree camera 110 based on an image R captured by a 360-degree camera 110 (see FIG. 2). In other words, the virtual experience providing device 100 can also be said to be a virtual image providing device.

[0013] 1, the virtual experience providing device 100 acquires the video R captured by the 360-degree camera 110 via a network NW. The network NW is, for example, the Internet, an intranet, a home network, etc. The virtual experience providing device 100 may also acquire the video R without going through the network NW. In this case, for example, a wired connection or a portable storage medium may be used.

[0014] The virtual experience providing device 100 may generate a virtual video V by acquiring in real time the video R being shot by the 360-degree camera 110, or may generate a virtual video V using data of a video R shot in the past. The video R may be a video or a still image. As is clear from the fact that data of a video R shot in the past can be used, the 360-degree camera 110 is not a required component of the virtual experience providing device 100.

[0015] The virtual experience providing device 100 is, for example, a computer that functions as a server. The virtual experience providing device 100 may be provided in a cloud environment. In other words, the virtual experience providing device 100 may be realized as software and a database. Therefore, the virtual experience providing device 100 may be, for example, a virtual server that can be realized by virtualization technology.

[0016] The 360-degree camera 110 may be a celestial sphere camera or a semi-celestial sphere camera. The 360-degree camera 110 is attached to a moving object such as a bicycle, an automobile, a motor cycle, a person, etc., and captures the video R. The 360-degree camera 110 may be fixed to a predetermined position instead of being attached to a moving object.

[0017] An example of a virtual image V shown in Fig. 2 is generated based on an image R captured by a 360-degree camera 110 mounted on a bicycle C. In each figure, the X-axis and Y-axis are horizontal, and the Z-axis is vertical.

[0018] In this way, it is conceivable that a 360-degree camera 110 is mounted on bicycle C to capture images, and a virtual image V with a viewpoint at a position (point B) different from the position of the 360-degree camera 110 (point A) is displayed on a display device in a remote location. This allows the experiencer who views the virtual image V displayed on the display device to experience the sensation of cycling together with the rider of bicycle C (companion P1) while being in a remote location. This also eliminates the need to use equipment such as a drone to capture images from the experiencer's viewpoint, which is different from the companion's viewpoint.

[0019] 2, the 360-degree camera 110 is mounted on the front side of the bicycle C. Therefore, the 360-degree camera 110 cannot capture images of the moving objects (bicycle C, accompanying person P1) from the side or rear thereof. Furthermore, the opposite side of the 360-degree camera 110 across the accompanying person P1 is in the blind spot of the 360-degree camera 110 as shown by the dashed dotted line, and therefore cannot be captured by the 360-degree camera 110.

[0020] The range that cannot be captured by the 360-degree camera 110 is also a range where no image exists in the virtual video V. Therefore, for example, it is conceivable to synthesize a moving avatar N1 (see FIG. 2) corresponding to the position of the 360-degree camera 110 in the virtual video V, or to supplement an image of a range that is a blind spot of the 360-degree camera 110 by using another video in which that range is captured.

[0021] Any image can be used as the avatar N1. In Fig. 2, the exterior images of a bicycle C and a companion P1 are synthesized as the avatar N1 of a moving object into the virtual image V. When the avatar N1 is an exterior image of a moving object, the experiencer can feel more realistic that he or she is moving together with the companion P1.

[0022] As the exterior image of the moving object, for example, 3D data of the moving object captured in advance can be used. The 3D data can be obtained, for example, by the accompanying person P1 capturing images of the bicycle C and his / her own face and body using a smartphone camera, and performing synthesis processing and 3D processing using application software. These processes can be realized by known technologies.

[0023] The created 3D data can be used as the avatar N1 by transmitting it in advance to the virtual experience providing device 100 and storing it in a memory or the like. Also, for example, it is possible to generate 3D data by combining an actual image of the face and body of the accompanying person P1 and an image of the bicycle prepared in advance in the virtual experience providing device 100.

[0024] As described above, the image R may be a moving image or a still image. Since a moving image is a series of still images, the virtual image V can be generated as a moving image by generating a virtual image V for each of the successive still images.

[0025] By taking video with the 360-degree camera 110, generating the virtual video V with the virtual experience providing device 100, and displaying the virtual video V on the display device via the network NW in real time, so-called live video distribution (real-time video distribution) can be realized. Live video distribution can further enhance the experiencer's sense of reality that he or she is acting together with the accompanying person P1.

[0026] 3 to 9, a virtual image generation process for generating a virtual image V having a viewpoint at a position (point B) different from the position (point A) of the 360-degree camera 110 based on the image R captured by the 360-degree camera 110 will be described. Note that the following describes the case where the point A side is viewed from point B.

[0027] FIG. 3 is a flowchart showing an example of a virtual image generation process. FIG. 4 is a diagram showing a state in which a mesh M is created in an image R captured by a 360-degree camera 110. FIG. 5 is a diagram for explaining the size of each mesh M as viewed from the position of the 360-degree camera 110. FIG. 6 is a diagram for explaining size correction of the mesh M. FIG. 7 is a diagram for explaining the size of each mesh M after size correction. FIG. 8 is a diagram for explaining the position of each mesh M after position correction. FIG. 9 is a diagram for explaining adjustment of the viewing direction. For ease of understanding, FIGS. 5 to 8 illustrate a two-dimensional case of an XZ plane, and FIG. 9 illustrates a two-dimensional case of an XY plane.

[0028] The virtual image generation process is realized by a CPU (Central Processing Unit) executing program instructions. The image generation unit 10 represents the functions of a computer as a virtual unit, and does not mean that it has a physical existence. However, a microcomputer or device corresponding to each function may physically exist.

[0029] In step S11 , the video generator 10 executes a mesh creation process for creating a mesh M in the video R captured by the 360-degree camera 110 .

[0030] 4, the image R can be expressed as a 360-degree image attached to the inner surface of a sphere. Point A corresponds to the position of the 360-degree camera 110. The image generator 10 treats the image R as a 360-degree image and creates a mesh M over the entire image.

[0031] In step S12, the image generating unit 10 executes a mesh size correction process to correct the size of each mesh M. Hereinafter, the mesh size correction process will be described with reference to Figs.

[0032] As shown in FIG. 5, when the position (point A) of the 360-degree camera 110 is taken as the viewpoint, the meshes M in all directions are images of the same size.

[0033] When the viewpoint is changed from point A to point B, the size of mesh M that is farther from point B than from point A is corrected to be smaller due to perspective, and the size of mesh M that is closer to point B than from point A is corrected to be larger. This makes it possible to properly view each mesh when point B is the viewpoint.

[0034] 5, when the viewpoint is changed from point A to point B, mesh Ma located in the negative X direction appears farther away than when the viewpoint is point A, so its size is corrected to be smaller. Conversely, mesh Mb located in the positive X direction appears closer than when the viewpoint is point A, so its size is corrected to be larger.

[0035] Next, a method for correcting the size of the meshes M will be specifically described. As shown in Fig. 6, the image generating unit 10 corrects the size of each mesh M using a distance α from point A to the mesh M and a distance β between points A and B.

[0036] If the size of mesh M located at a distance α from point A is L, the relationship with the size L' of mesh M (the size of mesh M after correction) when point B is the viewpoint is as follows: Reduction rate (enlargement rate)=L´ / L=α / (α+β) This reduction ratio (enlargement ratio) is applied to each mesh M. Note that when it comes to an enlargement ratio, β is calculated as a negative value.

[0037] The distance α from point A to mesh M is an arbitrary distance (predetermined distance α). As described above, in this embodiment, the video R is treated as a 360-degree image attached to the inner surface of a sphere. In other words, the viewpoint movement is virtually realized by attaching an image inside a limited virtual space and performing correction processing. Therefore, it is not affected by the actual distance to the object in the real space.

[0038] For example, in the case of a virtual experience in which the user acts together with a companion P1 riding a bicycle C (see FIG. 2), it is assumed that the user will maintain a distance of about 3 to 5 m (two-point distance β) from the companion P1 in order to achieve a sense of riding side by side. The predetermined distance α can be set based on experiments, simulations, etc., taking into account the two-point distance β that provides such a sense of riding side by side.

[0039] In this example, the predetermined distance α may be set to about 5 m to 10 m. The predetermined distance α may be set so as to be changeable as appropriate. This allows the user to select the predetermined distance α of their choice. Note that the distance β between the two points may be preset to a value according to the situation of the virtual experience, or, similar to the predetermined distance α, the user may be allowed to set a distance of their choice.

[0040] By performing the mesh size correction process as described above, the size of each mesh M is corrected at a reduction ratio (enlargement ratio) according to the predetermined distance α from point A to mesh M and the distance β between the two points, as shown in Figure 7.

[0041] In step S13, the image generating unit 10 executes a mesh position correction process for correcting the position of each mesh M.

[0042] When the size of each mesh M is corrected by the mesh size correction process, the size of each mesh M changes, causing gaps and overlaps to occur at the connection (boundary between meshes M and M) of adjacent meshes M. As a result, image continuity is lost at the connection between adjacent meshes M.

[0043] Therefore, the image generating unit 10 executes the mesh size correction process and adjusts the rotation angle (position in the rotation direction) of each mesh M centered on point A at the same rate as the reduction rate (magnification rate) of each mesh M, as shown in Fig. 8. The dashed lines indicate the positions of each mesh M before the positions are corrected. This maintains the continuity of images at the connection points of adjacent meshes M.

[0044] By performing the processes up to step S13, a virtual video V having point B as a viewpoint is generated. The processes of steps S14 to S16 are appropriately performed as necessary.

[0045] In step S14, the image generator 10 executes an avatar synthesis process to synthesize the avatar N1 with the virtual image V. The avatar synthesis process is not essential. However, by executing the avatar synthesis process, the experiencer who sees the virtual image V provided by the virtual experience providing device 100 can easily get the feeling that he or she is acting together with the companion P1.

[0046] As described above, the 360-degree camera 110 can only capture a portion of the moving object on which the 360-degree camera 110 is mounted. Therefore, the image generator 10 synthesizes an avatar N1 of the moving object corresponding to the position (point A) of the 360-degree camera 110 in the virtual image V generated by the processing up to step S13 (see FIG. 2).

[0047] In step S15, the image generator 10 executes a blind spot completion process to complete an image in a range that cannot be captured due to the blind spot of the 360-degree camera 110 (hereinafter referred to as a "blind spot range"). The blind spot completion process in step S15 is not essential. However, by executing the blind spot completion process, a user who sees the virtual image V provided by the virtual experience providing device 100 can easily recognize the virtual image V as an image closer to reality.

[0048] When a 360-degree camera 110 is attached to a moving object to capture video R (video), 360-degree images are captured continuously at different positions as the moving object moves. The video generator 10 extracts an image corresponding to the blind spot range of the 360-degree image being processed from a previously captured 360-degree image based on the moving speed and direction of the moving object. The extracted image is then composited into the blind spot range of the 360-degree image being processed.

[0049] The moving speed and direction of the moving object can be obtained from, for example, a smartphone carried by the accompanying person P1. It is also possible to extract images corresponding to blind spots using AI (Artificial Intelligence).

[0050] In step S16, the image generator 10 executes a viewing direction adjustment process to correct the angle of the virtual image V in response to a change in the direction (viewing direction) in which the point A side is viewed. If the viewing direction is not changed, the viewing direction adjustment process in step S16 is not necessary.

[0051] Point B is the viewpoint position of the experiencer looking toward the companion P1 (point A). Therefore, it is conceivable to make it possible for the experiencer to change point B to a position where the experiencer can view the companion P1 side from a desired direction.

[0052] Therefore, when point B moves and the viewing direction changes, the image generating unit 10 rotates the 360-degree image (virtual image V) in the opposite direction to the change in the viewing direction by the same amount of change as the viewing direction (amount of change in angle).

[0053] Figure 9 illustrates an example in which point B, which was in a position to view the companion P1 sideways, moves to a position to view the companion P1 from diagonally behind (moves in a clockwise direction). In Figure 9, the amount of change in the viewing direction is an angle θ. For ease of understanding, Figure 9 illustrates an example in which the distance β between the two points does not change.

[0054] The image generator 10 rotates the 360-degree image in the opposite direction to the change in the viewing direction by the same amount as the change in the viewing direction. Thus, in FIG. 9, the 360-degree image is rotated counterclockwise by angle θ. The dashed lines indicate the positions of each mesh M before their positions are corrected. This makes it possible to provide a virtual image V with an appropriate viewing direction even if the user's viewpoint position (point B) moves.

[0055] As described above, the virtual experience providing device 100 of the first embodiment is equipped with an image generation unit 10 that generates a virtual image V with a viewpoint at a position (point B) different from the position (point A) of the 360-degree camera 110 based on an image R captured by a 360-degree camera 110 mounted on a moving body (bicycle C, companion P1).

[0056] According to this, a virtual image V having a viewpoint different from that of the 360-degree camera 110 is generated based on an image R captured by a 360-degree camera 110 provided on a moving object, so there is no need to provide a camera for capturing an image having a viewpoint away from the 360-degree camera 110, i.e., an image from an experiencer's viewpoint different from the companion's viewpoint. Therefore, an image from an experiencer's viewpoint different from the companion's viewpoint can be generated using a simple configuration.

[0057] The virtual image V may include avatars N1 of moving objects (bicycle C, companion P1) synthesized in accordance with the position of the 360-degree camera 110 in the virtual image V.

[0058] This can enhance the experiencer's sense of reality that he or she is acting together with the companion P1.

[0059] When the image R captured by the 360-degree camera 110 is a moving image, the image generator 10 can generate the virtual image V as a real-time moving image.

[0060] This can enhance the experiencer's sense of reality that he or she is traveling together with the companion P1. In addition, the experiencer can experience traveling while communicating with the companion P1 through voice calls or the like.

[0061] The video generator 10 generates a virtual video V using a predetermined distance α from the 360-degree camera 110 and a distance β between two points between the position of the 360-degree camera 110 and a different position.

[0062] According to this, since a video from the experiencer's viewpoint different from the companion's viewpoint can be generated by the perspective method, no video is needed other than the video R captured by the 360-degree camera 110 to generate the virtual video V. In other words, no camera other than the 360-degree camera 110 is needed to generate the virtual video V.

[0063] The image generating unit 10 constructs meshes M in the image R, assuming that the image R captured by the 360-degree camera 110 is located at a predetermined distance α from the center of the 360-degree camera 110, and changes the size and position of each mesh M based on the predetermined distance α and the distance β between the two points.

[0064] According to this, meshes M are constructed in the image R, and the size and position of each mesh M are changed, so that the calculations in the generation process of the virtual image V are simplified.

[0065] <Second embodiment> Next, a virtual experience providing system 200 according to a second embodiment of the present invention will be described with reference to the accompanying drawings. In the second embodiment, the same components as those in the first embodiment are denoted by the same reference numerals, and the description thereof will be omitted as appropriate.

[0066] FIG. 10 is a diagram for explaining the virtual experience providing system 200. As shown in FIG.

[0067] As illustrated in FIG. 10, the virtual experience providing system 200 includes a virtual experience providing device 300, a 360-degree camera 110 provided on a moving object, and VR (Virtual Reality) goggles 120 as a display device worn by a user P2.

[0068] The virtual experience providing device 300 has the same functions as the virtual experience providing device 100 of the first embodiment. The virtual experience providing device 300 generates a virtual image V with a viewpoint at a position different from the position of the 360-degree camera 110 based on an image R captured by a 360-degree camera 110 provided on a moving object. The virtual experience providing device 300 then displays the generated virtual image V on the VR goggles 120. This allows the experiencer P2, who sees the virtual image V displayed on the VR goggles 120, to experience the sensation of traveling together with the companion P1.

[0069] In the example of Fig. 10, the accompanying person P1 is riding a bicycle C. The 360-degree camera 110 is attached to the bicycle C. That is, in this embodiment, the bicycle C and the accompanying person P1 constitute a moving body on which the 360-degree camera 110 is provided. For example, when the accompanying person P1 moves on foot or by running while holding the 360-degree camera 110, the moving body is the accompanying person P1.

[0070] The 360-degree camera 110 is communicably connected to a mobile terminal (hereinafter referred to as a companion terminal) 130 used by the companion P1. The 360-degree camera 110 and the companion terminal 130 may be connected by wire or wirelessly.

[0071] The companion terminal 130 is communicably connected to the virtual experience providing device 300 via the network NW. The video R captured by the 360-degree camera 110 is transmitted to the virtual experience providing device 300 via the companion terminal 130. The companion terminal 130 is a so-called mobile terminal, such as a tablet, a smartphone, a smart watch, or smart glasses.

[0072] The 360-degree camera 110 and the companion terminal 130 may be provided integrally. The 360-degree camera 110 may have some or all of the functions of the companion terminal 130, and the companion terminal 130 may have some or all of the functions of the 360-degree camera 110. The companion terminal 130 provided separately from the 360-degree camera 110 is not essential.

[0073] The VR goggles 120 are a head-mounted display that the experiencer P2 wears on his / her head. The VR goggles 120 may be equipped with a general liquid crystal display, or may be equipped with a retinal projection display or the like. The VR goggles 120 are communicatively connected to a terminal (hereinafter referred to as the experiencer terminal) 140 used by the experiencer P2. The VR goggles 120 and the experiencer terminal 140 may be connected by wire or wirelessly.

[0074] The display device may be a general stationary display that is connected to a PC (Personal Computer) or the like. When a head-mounted display such as VR goggles 120 is used as the display device, the virtual image V is displayed in the entire field of view of the participant P2, enhancing the sense of immersion of the participant P2. This can further enhance the participant P2's sense of reality that he or she is acting together with the accompanying person P1.

[0075] The experiencer terminal 140 is communicably connected to the virtual experience providing device 300 via the network NW. The virtual image V generated by the virtual experience providing device 300 is transmitted to the VR goggles 120 via the experiencer terminal 140. The experiencer terminal 140 is, for example, a tablet, a smartphone, a smart watch, a PC, or the like.

[0076] The display device and the experiencer terminal 140 may be provided integrally. The display device may have some or all of the functions of the experiencer terminal 140, and the experiencer terminal 140 may have some or all of the functions of the display device. The experiencer terminal 140 provided separately from the display device is not essential.

[0077] Next, a description will be given of the configuration of the virtual experience providing system 200. FIG 11 is a schematic diagram showing an example of the configuration of the virtual experience providing system 200.

[0078] 11, the virtual experience providing device 300, the companion terminal 130, and the experiencer terminal 140 are communicatively connected to each other via a network NW. The 360-degree camera 110 is communicatively connected to the companion terminal 130 via a wired or wireless connection. The VR goggles 120 are communicatively connected to the experiencer terminal 140 via a wired or wireless connection.

[0079] The accompanying person terminal 130 includes a communication unit 131 for wireless communication, a CPU 132 which is a processor for performing various calculation processes, a memory unit 133 which stores a database for storing various information, various programs, etc., and a display 134 which displays various information. The accompanying person terminal 130 may also include a speaker and microphone for voice communication with the experience person terminal 140, a GPS (Global Positioning System) for positioning, etc.

[0080] The storage unit 133 is a memory, a hard disk, or another known storage device. The storage unit 133 constitutes a storage medium that stores programs such as application programs.

[0081] The display 134 may be a touch panel that functions as an operation unit for operating the companion terminal 130. The companion terminal 130 may include an operation unit for operating the companion terminal 130, separate from the display 134.

[0082] The companion terminal 130 realizes functions such as transmitting video R, various information, etc. to the virtual experience providing device 300, and executing various instructions received from the virtual experience providing device 300, by the CPU 132 executing the instructions of the program stored in the memory unit 133.

[0083] If the 360-degree camera 110 and the virtual experience providing device 300 can be directly connected wirelessly, it is also possible to transmit the video R directly from the 360-degree camera 110 to the virtual experience providing device 300 without going through the companion terminal 130.

[0084] The VR goggles 120 include a display 121 and a sensor 122. The VR goggles 120 displays the virtual video V received from the virtual experience providing device 300 via the experiencer terminal 140 on the display 121.

[0085] The sensor 122 detects the movement of the head of the experiencer P2. The sensor 122 is composed of, for example, an acceleration sensor, a gyro sensor, etc. The sensor 122 may include a geomagnetic sensor. The sensor 122 is a known sensor used to realize a so-called head tracking function.

[0086] The VR goggles 120 transmit information relating to the movement of the head of the experiencer P2 detected by the sensor 122 (hereinafter referred to as experiencer movement information) to the virtual experience providing device 300 via the experiencer terminal 140.

[0087] The experiencer terminal 140 includes a communication unit 141 for wireless communication, a CPU 142 which is a processor for performing various calculation processes, a memory unit 143 for storing a database for storing various information, various programs, etc., and a display 144 for displaying various information. The experiencer terminal 140 may also include a speaker and a microphone for voice communication with the companion terminal 130.

[0088] The storage unit 143 is a memory, a hard disk, or another known storage device. The storage unit 143 constitutes a storage medium that stores programs such as application programs.

[0089] The display 144 may be a touch panel that functions as an operation unit for operating the experiencer terminal 140. The experiencer terminal 140 may include an operation unit for operating the experiencer terminal 140, separate from the display 144.

[0090] In the virtual experience providing system 200, the experiencer P2 can change the viewpoint position and viewing direction (see FIG. 2) of the virtual image V by operating an operation unit for operating the experiencer terminal 140. The viewing direction can also be changed based on experiencer movement information. Therefore, the experiencer terminal 140 transmits information input from the operation unit for operating the experiencer terminal 140 (hereinafter referred to as operation information) and experiencer movement information to the virtual experience providing device 300.

[0091] It is not essential that the experiencer P2 be able to change the viewpoint position and viewing direction of the virtual image V. The viewpoint position and viewing direction of the virtual image V may be configured to be fixed at a predetermined position and direction, or may be changed by, for example, an administrator of the virtual experience providing system 200. In other words, the operation unit operated by the experiencer P2 and the sensor 122 of the VR goggles 120 are not essential.

[0092] Furthermore, the operation unit operated by the experiencer P2 may be provided separately from the experiencer terminal 140. For example, if an operation unit modeled after a bicycle, such as an exercise machine, is provided, the operation information may include the travel speed, steering angle of the handlebars, etc. This can further enhance the experiencer P2's sense of reality that he or she is traveling by bicycle together with the accompanying person P1.

[0093] The experiencer terminal 140 realizes functions such as transmitting operation information, experiencer action information, etc. to the virtual experience providing device 300, and executing various instructions received from the virtual experience providing device 300, by the CPU 142 executing the instructions of the program stored in the memory unit 143.

[0094] If the VR goggles 120 and the virtual experience providing device 300 can be directly connected wirelessly, it is also possible to transmit experiencer movement information directly from the VR goggles 120 to the virtual experience providing device 300 without going through the experiencer terminal 140.

[0095] The virtual experience providing device 300 is, for example, a computer that functions as a server. The virtual experience providing device 300 may be provided in a cloud environment. In other words, the entity may be software and a database. Therefore, the virtual experience providing device 300 may be, for example, a virtual server that can be realized by virtualization technology.

[0096] The virtual experience providing device 300 includes a communication unit 310 for wireless communication, a CPU 320 which is a processor that performs various arithmetic processing, and a storage unit 330 which stores a database for storing various information, various programs, and the like.

[0097] The storage unit 330 is a memory, a hard disk, or another known storage device. The storage unit 330 constitutes a storage medium that stores programs such as application programs.

[0098] The image generating unit 311, the display control unit 312, the image receiving unit 313, the position calculation unit 314, and the direction calculation unit 315 are shown as virtual units of computer functions and do not represent physical existences. However, a microcomputer or device corresponding to each function may physically exist.

[0099] The various processes performed by the video generation unit 311, the display control unit 312, the video reception unit 313, the position calculation unit 314, and the direction calculation unit 315 are realized by the CPU 320 executing the instructions of the program stored in the memory unit 330.

[0100] The video receiving unit 313 executes a video receiving process for receiving the video R (video signal) transmitted from the companion terminal 130. The received video R is stored in the storage unit 330.

[0101] The position calculation section 314 executes a position calculation process to calculate the viewpoint position of the virtual image V based on the operation information transmitted from the experiencer terminal 140. The viewpoint position (position information) obtained by the calculation is stored in the storage section 330.

[0102] The direction calculation unit 315 executes a direction calculation process to calculate the viewing direction of the virtual image V based on the operation information and / or the experiencer action information transmitted from the experiencer terminal 140. The viewing direction (direction information) obtained by the calculation is stored in the storage unit 330.

[0103] The storage unit 330 stores various setting information of the virtual experience providing system 200, and a default viewpoint position (default position information) and a default viewing direction (default direction information) are also stored as setting information in the storage unit 330. The default predetermined position and the default viewing direction can be changed to any position and any direction.

[0104] The image generating unit 311 has the same functions as the image generating unit 10 of the first embodiment. The image generating unit 311 executes a virtual image generating process based on the position information, the direction information, and the image R. This generates a virtual image V having a viewpoint at a position different from the position of the 360-degree camera 110. When synthesizing a moving object avatar N1 (see FIG. 2) with the virtual image V, 3D data or the like previously stored in the storage unit 330 can be used as the moving object avatar N1. The generated virtual image V is stored in the storage unit 330.

[0105] The display control unit 312 executes a display control process for displaying the virtual video V generated by the video generation unit 311 on the VR goggles 120. Specifically, the display control unit 312 transmits the virtual video V generated by the video generation unit 311 to the experiencer terminal 140, and transmits an output instruction to the experiencer terminal 140 for outputting the virtual video V from the experiencer terminal 140 to the VR goggles 120. As a result, the virtual video V is displayed on the display 121 of the VR goggles 120.

[0106] If the VR goggles 120 and the virtual experience providing device 300 can be directly connected wirelessly, it is also possible to transmit the virtual image V directly from the virtual experience providing device 300 to the VR goggles 120 and display it on the display 121 without going through the experiencer terminal 140.

[0107] Next, a procedure by which the virtual experience providing system 200 displays the virtual video V on the VR goggles 120 will be described with reference to Fig. 12. Fig. 12 is a flowchart showing an example of a procedure by which the virtual experience providing system 200 displays the virtual video V on the VR goggles 120.

[0108] The process of the flowchart shown in Fig. 12 is repeatedly executed at a predetermined interval. The predetermined interval is, for example, several msec to several tens of msec. When performing so-called live video distribution, it is preferable to set the predetermined interval in accordance with a frame rate of about 20 fps (frames per second) to about 60 fps. The frame rate of the 360-degree camera 110 is set in a similar manner.

[0109] In step S21, the VR goggles 120 transmit experiencer movement information to the experiencer terminal 140.

[0110] In step S22, the experiencer terminal 140 receives the experiencer movement information transmitted by the VR goggles 120.

[0111] In step S23, the experiencer terminal 140 transmits the operation information and the experiencer action information received from the VR goggles 120 to the virtual experience providing apparatus 300.

[0112] In step S24, the virtual experience providing device 300 receives the operation information and the experiencer action information transmitted by the experiencer terminal 140.

[0113] In step S25, the virtual experience providing device 300 calculates position information and direction information based on the operation information and experiencer motion information received from the experiencer terminal 140.

[0114] In step S26, the 360-degree camera 110 captures the image R.

[0115] In step S27, the 360-degree camera 110 transmits the captured video R to the companion terminal 130.

[0116] In step S28, the companion terminal 130 receives the video R transmitted by the 360-degree camera 110.

[0117] In step S29, the companion terminal 130 transmits the video R received from the 360-degree camera 110 to the virtual experience providing device 300.

[0118] In step S30, the virtual experience providing device 300 receives the video R transmitted by the companion terminal 130.

[0119] The order of the processes from step S21 to step S30 can be changed as appropriate, except that the processes from step S21 to step S25 are executed in this order, and the processes from step S26 to step S30 are executed in this order.

[0120] In step S31, the virtual experience providing device 300 generates a virtual video V.

[0121] In step S32, the virtual experience providing device 300 transmits the generated virtual video V to the experiencer terminal 140.

[0122] In step S33, the experiencer terminal 140 receives the virtual video V transmitted by the virtual experience providing device 300.

[0123] In step S34, the experiencer terminal 140 transmits the virtual image V received from the virtual experience providing device 300 to the VR goggles 120.

[0124] In step S35, the VR goggles 120 receive the virtual image V transmitted by the experiencer terminal 140.

[0125] In step S36, the VR goggles 120 display the virtual image V received from the experiencer terminal 140 on the display 121.

[0126] As described above, the virtual experience providing system 200 of the second embodiment comprises a virtual experience providing device 300, a 360-degree camera 110 installed on a moving body (bicycle C, companion P1), and a display device (VR goggles 120) that displays the virtual image V generated by the image generation unit 311.

[0127] The virtual experience providing system 200 includes a virtual experience providing device 300, and based on the image R captured by the 360-degree camera 110 installed on the moving object, the virtual experience providing system 200 can generate a virtual image V with a viewpoint different from the position of the 360-degree camera 110. Therefore, it is not necessary to provide a camera for capturing an image with a viewpoint away from the 360-degree camera 110, that is, an image from the experiencer's viewpoint different from the companion's viewpoint. Therefore, an image from the experiencer's viewpoint different from the companion's viewpoint can be generated using a simple configuration. In addition, since the virtual experience providing system 200 displays the virtual image V on a display device, the experiencer P2 who views the virtual image V displayed on the display device can experience the sensation of traveling with the companion P1 even while being in a remote location.

[0128] The virtual experience providing system 200 includes an operation unit (display 144) operated by the experiencer P2, and the image generating unit 311 moves the viewpoint position of the virtual image V in response to the operation of the operation unit (display 144) by the experiencer P2.

[0129] This makes it possible to generate a virtual image V from a viewpoint position desired by the experiencer P2.

[0130] The display device can be a head-mounted display (VR goggles 120) worn by the experiencer P2.

[0131] According to this, the virtual image V is displayed in the entire field of vision of the participant P2, which enhances the sense of immersion of the participant P2. This further enhances the sense of reality of the participant P2 that he or she is acting together with the companion P1.

[0132] The display device includes a sensor 122 that detects the movement of the head of the experiencer P2, and the image generator 311 changes the direction of the field of view of the virtual image V displayed on the display device based on the detection result of the sensor 122 (experiencer movement information).

[0133] This allows the user P2 to view the virtual image V in a desired viewing direction without having to perform any troublesome operations.

[0134] <Third embodiment> Next, a virtual experience providing system 400 according to a third embodiment of the present invention will be described with reference to the accompanying drawings. In the third embodiment, the same components as those in the first and second embodiments are denoted by the same reference numerals, and the description thereof will be omitted as appropriate.

[0135] Fig. 13 is a diagram for explaining the virtual experience providing system 400. Fig. 14 is a diagram for explaining the augmented reality processing.

[0136] As illustrated in FIG. 13, the virtual experience providing system 400 includes a virtual experience providing device 500, a 360-degree camera 110 mounted on a moving object, VR goggles 120 as a display device, and AR (Augmented Reality) glasses 150 as an augmented reality head-mounted display worn by a companion P1 moving with the 360-degree camera 110.

[0137] The virtual experience providing device 500 has the same functions as the virtual experience providing device 300 of the second embodiment. The virtual experience providing device 500 generates a virtual image V with a viewpoint at a position different from the position of the 360-degree camera 110 based on an image R captured by a 360-degree camera 110 provided on a moving object. The virtual experience providing device 500 then displays the generated virtual image V on the VR goggles 120. This allows the experiencer P2, who sees the virtual image V displayed on the VR goggles 120, to experience the sensation of traveling together with the companion P1.

[0138] In the example of Fig. 13, the accompanying person P1 is riding a bicycle C. The 360-degree camera 110 is attached to the bicycle C. That is, in this embodiment, the bicycle C and the accompanying person P1 constitute a moving body on which the 360-degree camera 110 is provided. For example, when the accompanying person P1 moves on foot or by running while holding the 360-degree camera 110, the moving body is the accompanying person P1.

[0139] The companion terminal 130 is communicably connected to the virtual experience providing device 500 via the network NW. The video R captured by the 360-degree camera 110 is transmitted to the virtual experience providing device 500 via the companion terminal 130.

[0140] The experiencer terminal 140 is communicably connected to the virtual experience providing device 500 via the network NW. The virtual image V generated by the virtual experience providing device 500 is transmitted to the VR goggles 120 via the experiencer terminal 140.

[0141] The AR glasses 150 are an augmented reality head-mounted display worn by the companion P1 who moves together with the 360-degree camera 110. The AR glasses 150 may be equipped with a general transmissive liquid crystal display, or may be equipped with a projection display that projects an image onto the retina or lens portion. The AR glasses 150 are connected to the companion terminal 130 so as to be able to communicate with each other. The AR glasses 150 and the companion terminal 130 may be connected by wire or wirelessly.

[0142] The AR glasses 150 and the companion terminal 130 may be provided integrally. The AR glasses 150 may have some or all of the functions of the companion terminal 130. The companion terminal 130 provided separately from the AR glasses 150 is not essential.

[0143] The AR glasses 150 and the 360-degree camera 110 may be provided integrally. In this embodiment, the 360-degree camera 110 provided separately from the AR glasses 150 is not essential. The AR glasses 150, the 360-degree camera 110 and the companion terminal 130 may be provided integrally.

[0144] 14, the virtual experience providing system 400 can display an avatar N2 (AR image) of the experiencer P2 on the AR glasses 150 in correspondence with the viewpoint position (point B) of the experiencer P2 in the field of view of the companion P1 (a position different from the position (point A) of the 360-degree camera 110) (augmented reality processing). This makes the experiencer P2 appear to exist in the actual scene seen by the companion P1, enhancing the sense of reality of the companion P1 that he or she is acting together with the experiencer P2.

[0145] Any image can be used as the avatar N2. In Fig. 14, an external image of the experiencer P2 riding a bicycle is displayed on the AR glasses 150 as the avatar N2 of the experiencer P2. When the avatar N2 is an external image of the experiencer P2, the accompanying person P1 can feel more realistic that he or she is acting together with the experiencer P2.

[0146] As the external image of the experiencer P2, for example, 3D data of the experiencer P2 that has been photographed in advance can be used. The 3D data can be obtained, for example, by the experiencer P2 photographing his / her face, body, and bicycle using a smartphone camera, and performing synthesis processing and 3D processing using application software. These processes can be realized by known technologies.

[0147] The created 3D data can be used as avatar N2 by transmitting it to the virtual experience providing device 500 in advance and storing it in a memory or the like. Also, for example, it is possible to generate 3D data by combining an actual image of the face and body of the experiencer P2 with an image of a bicycle prepared in advance in the virtual experience providing device 500.

[0148] Next, a description will be given of the configuration of the virtual experience providing system 400. Fig. 15 is a schematic diagram showing an example of the configuration of the virtual experience providing system 400.

[0149] 15, the virtual experience providing device 500, the companion terminal 130, and the experiencer terminal 140 are communicatively connected to each other via a network NW. The AR glasses 150 are communicatively connected to the companion terminal 130 via a wired or wireless connection.

[0150] The AR glasses 150 include a display 151 and a sensor 152. The AR glasses 150 display, on the display 151, an AR image (avatar N2) received from the virtual experience providing device 500 via the companion terminal 130.

[0151] The sensor 152 detects the movement of the head of the companion P1. The sensor 152 is composed of, for example, an acceleration sensor, a gyro sensor, etc. The sensor 152 may include a geomagnetic sensor. The sensor 152 is a known sensor used to realize a so-called head tracking function.

[0152] The AR glasses 150 transmit information relating to the movement of the head of the companion P1 detected by the sensor 152 (hereinafter referred to as companion movement information) to the virtual experience providing device 500 via the companion terminal 130.

[0153] If the AR glasses 150 and the virtual experience providing device 500 can be directly connected wirelessly, it is also possible to transmit companion action information directly from the AR glasses 150 to the virtual experience providing device 500 without going through the companion terminal 130.

[0154] The virtual experience providing device 500 is, for example, a computer that functions as a server. The virtual experience providing device 500 may be provided in a cloud environment. In other words, the entity may be software and a database. Therefore, the virtual experience providing device 500 may be, for example, a virtual server that can be realized by virtualization technology.

[0155] The virtual experience providing device 500 includes a communication unit 310 for performing wireless communication, a CPU 320 which is a processor that performs various arithmetic processing, and a storage unit 330 which stores a database for storing various information, various programs, and the like.

[0156] The image generating unit 311, the display control unit 312, the image receiving unit 313, the position calculation unit 314, the direction calculation unit 315, and the augmented reality processing unit 316 are shown as virtual units of computer functions and do not represent physical existences. However, a microcomputer or device corresponding to each function may physically exist.

[0157] The various processes performed by the image generation unit 311, the display control unit 312, the image receiving unit 313, the position calculation unit 314, the direction calculation unit 315, and the augmented reality processing unit 316 are realized by the CPU 320 executing the instructions of the program stored in the memory unit 330.

[0158] The direction calculation unit 315 calculates the viewing direction of the virtual image V based on the operation information and / or experiencer movement information transmitted from the experiencer terminal 140. The viewing direction of the virtual image V (directional information of the virtual image V) obtained by the calculation is stored in the storage unit 330. The direction calculation unit 315 also calculates the viewing direction of the companion P1 based on the companion movement information transmitted from the companion terminal 130. The viewing direction of the companion P1 obtained by the calculation (directional information of the companion P1) is stored in the storage unit 330.

[0159] The augmented reality processing unit 316 executes augmented reality processing based on the position information and the direction information. This generates an AR image (avatar N2) to be displayed on the AR glasses 150 corresponding to the viewpoint position of the experiencer P2 in the field of view of the companion P1. 3D data or the like previously stored in the storage unit 330 can be used as the AR image.

[0160] The augmented reality processing unit 316 displays the generated AR image on the AR glasses 150. Specifically, the augmented reality processing unit 316 transmits the generated AR image to the companion terminal 130, and transmits an output instruction to the companion terminal 130 to output the AR image from the companion terminal 130 to the AR glasses 150. As a result, the AR image is displayed on the display 151 of the AR glasses 150. Note that the display control unit 312 may cause the AR image generated by the augmented reality processing unit 316 to be displayed on the AR glasses 150.

[0161] If the AR glasses 150 and the virtual experience providing device 500 can be directly connected wirelessly, it is also possible to transmit an AR image directly from the virtual experience providing device 500 to the AR glasses 150 and display it on the display 151 without going through the companion terminal 130.

[0162] Next, a procedure by which the virtual experience providing system 400 displays an AR image on the AR glasses 150 will be described with reference to Fig. 16. Fig. 16 is a flowchart showing an example of a procedure by which the virtual experience providing system 400 displays an AR image on the AR glasses 150.

[0163] The process of the flowchart shown in Fig. 16 is repeatedly executed at a predetermined interval. The predetermined interval is, for example, several msec to several tens of msec. When performing so-called live video distribution, it is preferable to set the predetermined interval in accordance with a frame rate of about 20 fps (frames per second) to about 60 fps.

[0164] In step S41, the AR glasses 150 transmit the companion operation information to the companion terminal 130.

[0165] In step S42, the companion terminal 130 receives the companion operation information transmitted by the AR glasses 150.

[0166] In step S43, the companion terminal 130 transmits the companion action information received from the AR glasses 150 to the virtual experience providing device 500.

[0167] In step S44, the virtual experience providing device 500 receives the companion action information transmitted by the companion terminal 130.

[0168] In step S45, the virtual experience providing device 500 calculates direction information (the viewing direction of the companion P1) based on the companion action information received from the companion terminal 130.

[0169] In step S46, the virtual experience providing device 500 generates an AR image (avatar N2) based on the position information and the direction information.

[0170] In step S47, the virtual experience providing device 500 transmits the generated AR image to the companion terminal 130.

[0171] In step S48, the companion terminal 130 receives the AR image transmitted by the virtual experience providing device 500.

[0172] In step S49, the companion terminal 130 transmits the AR image received from the virtual experience providing device 500 to the AR glasses 150.

[0173] In step S50, the AR glasses 150 receive the AR image transmitted by the companion terminal 130.

[0174] In step S51, the AR glasses 150 display the AR image received from the companion terminal 130 on the display 151.

[0175] As described above, the virtual experience providing system 400 according to the third embodiment comprises AR glasses 150 worn by a companion P1 who moves together with the 360-degree camera 110, and an augmented reality processing unit 316 that displays an avatar N2 of the experiencer P2 on the AR glasses 150 in accordance with the viewpoint position of the experiencer P2 (the viewpoint position of the virtual image V).

[0176] According to this, the experiencer P2 appears to be present in the actual scene seen by the companion P1, thereby enhancing the sense of reality of the companion P1 that he or she is acting together with the experiencer P2.

[0177] The main effects of the virtual experience providing device 100 according to the first embodiment of the present invention, the virtual experience providing system 200 according to the second embodiment, and the virtual experience providing system 400 according to the third embodiment will be collectively described below.

[0178] (1) The virtual experience providing device 100 of the first embodiment includes an image generating unit 10 that generates a virtual image V based on an image R captured by a 360-degree camera 110 mounted on a moving body (bicycle C, companion P1), with the viewpoint being a position (point B) different from the position (point A) of the 360-degree camera 110.

[0179] According to this, a virtual image V having a viewpoint different from that of the 360-degree camera 110 is generated based on an image R captured by a 360-degree camera 110 provided on a moving object, so there is no need to provide a camera for capturing an image having a viewpoint away from the 360-degree camera 110, i.e., an image from an experiencer's viewpoint different from the companion's viewpoint. Therefore, an image from an experiencer's viewpoint different from the companion's viewpoint can be generated using a simple configuration.

[0180] (2) The virtual image V may include an avatar N1 of a moving object (bicycle C, companion P1) synthesized in accordance with the position of the 360-degree camera 110 in the virtual image V.

[0181] This can enhance the experiencer's sense of reality that he or she is acting together with the companion P1.

[0182] (3) When the image R captured by the 360-degree camera 110 is a moving image, the image generator 10 can generate the virtual image V as a real-time moving image.

[0183] This can enhance the experiencer's sense of reality that he or she is traveling together with the companion P1. In addition, the experiencer can experience traveling while communicating with the companion P1 through voice calls or the like.

[0184] (4) The image generator 10 generates a virtual image V using a predetermined distance α from the 360-degree camera 110 and a distance β between two points between the position of the 360-degree camera 110 and a different position.

[0185] According to this, since a video from the experiencer's viewpoint different from the companion's viewpoint can be generated by the perspective method, no video is needed other than the video R captured by the 360-degree camera 110 to generate the virtual video V. In other words, no camera other than the 360-degree camera 110 is needed to generate the virtual video V.

[0186] (5) The image generating unit 10 constructs a mesh M in the image R, assuming that the image R captured by the 360-degree camera 110 is located at a predetermined distance α from the center of the 360-degree camera 110, and changes the size and position of each mesh M based on the predetermined distance α and the distance β between the two points.

[0187] According to this, meshes M are constructed in the image R, and the size and position of each mesh M are changed, so that the calculations in the generation process of the virtual image V are simplified.

[0188] (6) The virtual experience providing system 200 of the second embodiment comprises a virtual experience providing device 300, a 360-degree camera 110 installed on a moving body (bicycle C, companion P1), and a display device (VR goggles 120) that displays the virtual image V generated by the image generation unit 311.

[0189] The virtual experience providing system 200 includes a virtual experience providing device 300, and based on the image R captured by the 360-degree camera 110 installed on the moving object, the virtual experience providing system 200 can generate a virtual image V with a viewpoint different from the position of the 360-degree camera 110. Therefore, it is not necessary to provide a camera for capturing an image with a viewpoint away from the 360-degree camera 110, that is, an image from the experiencer's viewpoint different from the companion's viewpoint. Therefore, an image from the experiencer's viewpoint different from the companion's viewpoint can be generated using a simple configuration. In addition, since the virtual experience providing system 200 displays the virtual image V on a display device, the experiencer P2 who views the virtual image V displayed on the display device can experience the sensation of traveling with the companion P1 even while being in a remote location.

[0190] (7) The virtual experience providing system 200 includes an operation unit (display 144) operated by the experiencer P2, and the image generating unit 311 moves the viewpoint position of the virtual image V in response to the operation of the operation unit (display 144) by the experiencer P2.

[0191] This makes it possible to generate a virtual image V from a viewpoint position desired by the experiencer P2.

[0192] (8) The display device may be a head-mounted display (VR goggles 120) worn by the experiencer P2.

[0193] According to this, the virtual image V is displayed in the entire field of vision of the participant P2, which enhances the sense of immersion of the participant P2. This further enhances the sense of reality of the participant P2 that he or she is acting together with the companion P1.

[0194] (9) The display device is equipped with a sensor 122 that detects the movement of the head of the experiencer P2, and the image generation unit 311 changes the direction of the field of view of the virtual image V displayed on the display device based on the detection result (experiencer movement information) of the sensor 122.

[0195] This allows the user P2 to view the virtual image V in a desired viewing direction without having to perform any troublesome operations.

[0196] (10) The virtual experience providing system 400 according to the third embodiment includes AR glasses 150 worn by a companion P1 who moves together with the 360-degree camera 110, and an augmented reality processing unit 316 that displays an avatar N2 of the experiencer P2 on the AR glasses 150 in accordance with the viewpoint position of the experiencer P2 (the viewpoint position of the virtual image V).

[0197] According to this, the experiencer P2 appears to be present in the actual scene seen by the companion P1, thereby enhancing the sense of reality of the companion P1 that he or she is acting together with the experiencer P2.

[0198] An embodiment of the present invention has been described above, but the above embodiment merely illustrates one application example of the present invention, and is not intended to limit the technical scope of the present invention to the specific configuration of the above embodiment.

[0199] For example, the virtual experience providing device 100 of the first embodiment can have the same configuration as the virtual experience providing device 300 of the second embodiment or the virtual experience providing device 500 of the third embodiment.

[0200] Also, for example, in the second embodiment, the virtual experience providing device 300 is the entity that executes the process in which the virtual experience providing system 200 displays the virtual video V on the VR goggles 120. However, some or all of the functions of the virtual experience providing device 300 may be borne by other components in the virtual experience providing system 200. The same applies to the virtual experience providing system 400 of the third embodiment. The same applies to the entity that executes the process in which the virtual experience providing system 400 displays an AR image on the AR glasses 150.

[0201] Furthermore, for example, the position at which the 360-degree camera 110 is attached to the moving body is not limited to the positions shown in the respective embodiments. The 360-degree camera 110 can be attached to the moving body at any position.

[0202] The various programs executed by the virtual experience providing device 100, the virtual experience providing system 200 (virtual experience providing device 300), and the virtual experience providing system 400 (virtual experience providing device 500) may be stored on a non-transitory recording medium such as a CD-ROM. [Explanation of symbols]

[0203] 10. Image Generation Section 100 Virtual experience providing device 110 360 degree camera 120 VR goggles (display device, head-mounted display) 122 Sensors 144 Display (operation section) 150 AR Glasses (Augmented Reality Head Mounted Display) 200 Virtual experience provision system 300 Virtual experience providing device 311 Image Generation Unit 316 Augmented reality processing unit 400 Virtual Experience Providing System 500 Virtual experience providing device C Bicycle (mobile object) P1 Companion (mobile) P2 Experiencer R video M Mesh N1 Avatar (Companion Avatar) N2 Avatar (Participant's avatar) V Virtual Image α Predetermined distance β Distance between two points

Claims

1. An image generating unit that generates a virtual image having a viewpoint different from the viewpoint of a 360-degree camera based on an image captured by the 360-degree camera provided on the moving object, A device providing virtual experiences.

2. The virtual experience providing device according to claim 1 , The virtual image includes an avatar of the moving object synthesized in accordance with a position of the 360-degree camera in the virtual image. A device providing virtual experiences.

3. The virtual experience providing device according to claim 1 , The image captured by the 360-degree camera is a video, The image generation unit generates the virtual image as a real-time video. A device providing virtual experiences.

4. The virtual experience providing device according to claim 1 , The image generating unit generates the virtual image using a predetermined distance from the 360-degree camera and a distance between two points between a position of the 360-degree camera and the different position. A device providing virtual experiences.

5. The virtual experience providing device according to claim 4, the image generating unit generates meshes in the image, assuming that the image captured by the 360-degree camera is located at the predetermined distance from the 360-degree camera as a center, and changes the size and position of each mesh based on the predetermined distance and the distance between the two points. A device providing virtual experiences.

6. A virtual experience providing device according to any one of claims 1 to 5; The 360-degree camera provided on the moving body; a display device that displays the virtual image generated by the image generation unit; and Equipped with Virtual experience provision system.

7. The virtual experience providing system according to claim 6, It is equipped with an operating unit operated by the participant, The image generation unit moves the different position in response to an operation of the operation unit by the experiencer. Virtual experience provision system.

8. The virtual experience providing system according to claim 6, The display device is a head-mounted display worn by the experiencer. Virtual experience provision system.

9. The virtual experience providing system according to claim 8, The display device includes a sensor that detects a movement of the head of the user, The image generation unit changes a direction of a field of view of the virtual image displayed on the display device based on a detection result of the sensor. Virtual experience provision system.

10. The virtual experience providing system according to claim 7, An augmented reality head mounted display worn by a companion moving together with the 360-degree camera; an augmented reality processing unit that displays an avatar of the experiencer on the augmented reality head mounted display in correspondence with the different positions; Equipped with Virtual experience provision system.

11. A virtual experience providing method executed by a computer of a virtual experience providing system, generating a virtual image having a viewpoint different from that of the 360-degree camera based on an image captured by the 360-degree camera provided on the moving object; Displaying the generated virtual image on a display device. How to deliver virtual experiences.

12. A program executable by a computer of a virtual experience providing system, generating a virtual image having a viewpoint different from that of the 360-degree camera based on an image captured by the 360-degree camera provided on the moving object; Displaying the generated virtual image on a display device. causing the computer to execute instructions; program.

Images