Virtual-reality provision system, data distribution device, virtual-space provision apparatus, and program

Abstract

To provide a user with a brand new live broadcasting.SOLUTION: A virtual-reality provision system includes: a tracking data acquisition unit that acquires tracking data on an athlete playing in a game and a flying object thrown by the athlete in real time; a three-dimensional-display-data generation unit that uses the acquired tracking data to generate three-dimensional display data for displaying, in a virtual space, a virtual video including a virtual athlete corresponding to the athlete and a flight video of a virtual flying object corresponding to the flying object; a distribution unit which distributes distribution data including a game video of the game and three-dimensional display data temporally synchronized with the game video; and at least one display terminal. The display terminal includes: a first display control unit which displays the game video of the distribution data; and a second display control unit which displays, in the virtual space, a virtual video temporally synchronized with the game video, the virtual video including the virtual athlete viewed from a viewpoint position in the virtual space corresponding to a position / attitude of a user and the flight video of the virtual flying object.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a virtual reality providing system, a data distribution device, a virtual space providing device, and a program. [Background technology]

[0002] A head-mounted display (HMD) is known that is worn on a user's head and can present a three-dimensional virtual space image to the user using a display placed in front of the user's eyes, etc. A practice support system that uses this head-mounted display is described in Patent Document 1.

[0003] The practice support system of Patent Document 1 comprises an HMD that is worn on the head of a practitioner and has a display means that presents a three-dimensional image of a virtual space in which a virtual moving body, which is a virtual object of equipment handled by the practitioner, is placed, a head position detection means that detects the position and posture of the practitioner's head, an angular velocity detection means that is attached to the equipment, a calculation means that calculates trajectory data of the virtual moving body based on the angular velocity data acquired by the angular velocity detection means, a memory means that stores the trajectory data, and an image generation means that generates a trajectory image of the virtual moving body in the virtual space based on the trajectory data and causes the display means to present the trajectory image from a viewpoint position in the virtual space that corresponds to the position and posture of the practitioner's head detected by the head position detection means. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2016-47219 Summary of the Invention [Problem to be solved by the invention]

[0005] People all over the world are enjoying watching live sports from the comfort of their own homes on television or the internet, including domestic professional baseball and J-League broadcasts, as well as global events such as the Olympics and the World Cup.

[0006] However, live footage is for spectators only, and does not allow viewers to experience the performance of the athletes featured in the footage.

[0007] Therefore, the present invention aims to provide a virtual reality providing system that not only allows users to watch athletes' performances from video, but also allows users to have a new experience in which they themselves become the main characters. [Means for solving the problem]

[0008] One aspect of the present invention is a virtual reality provision system that includes: a tracking data acquisition unit that acquires, in real time, tracking data on athletes and flying objects obtained from sensor information of sensors that track athletes and flying objects released by the athletes during a game; a 3D display data generation unit that uses the acquired tracking data to generate 3D display data for displaying virtual images in a virtual space, including flying images of virtual athletes corresponding to the athletes and virtual flying objects corresponding to the flying objects; a distribution unit that distributes distribution data including game video of the game and 3D display data that is synchronized in time with the game video; and at least one or more virtual space provision systems that receive the distribution data and use the game video and the 3D display data included in the distribution data to display in the virtual space virtual images that include flying images of the virtual athletes and virtual flying objects from a viewpoint position in the virtual space that corresponds to the position and posture of the user, and that are synchronized in time with the game video.

[0009] One aspect of the present invention is a data distribution device that includes a 3D display data generation unit that uses tracking data on athletes and flying objects obtained in real time from sensor information of a sensor that tracks athletes and flying objects released by the athletes during a game to generate 3D display data for displaying virtual images in a virtual space, including flying images of virtual athletes corresponding to the athletes and virtual flying objects corresponding to the flying objects; and a distribution unit that distributes distribution data including game video of the game and 3D display data that is synchronized in time with the game video to at least one virtual space providing device, wherein the 3D display data includes flying images of the virtual athletes and virtual flying objects from a viewpoint position in the virtual space that corresponds to the position and posture of the user, and is data for displaying the virtual image that is synchronized in time with the game video in the virtual space of the virtual space providing device, and the game video of the game and the virtual image are displayed on the virtual space providing device.

[0010] One aspect of the present invention is a virtual space providing device that includes: receiving means for receiving distribution data that includes 3D display data for displaying virtual video including flight video of virtual athletes corresponding to athletes and virtual flying objects corresponding to flying objects, the virtual video being generated using tracking data of athletes and flying objects obtained in real time from sensor information from a sensor that tracks athletes and flying objects released by the athletes during a game; and display means for displaying in the virtual space the game video and the virtual video including flight video of virtual athletes and virtual flying objects from a viewpoint position in the virtual space that corresponds to the position and posture of the user, the virtual video being synchronized in time with the game video, using the 3D display data included in the distribution data.

[0011] One aspect of the present invention is a program that causes a computer to function as: receiving means for receiving distribution data that includes 3D display data for displaying virtual video including flying video of virtual athletes corresponding to athletes and virtual flying objects corresponding to flying objects, the virtual video being generated using tracking data of athletes and flying objects obtained in real time from sensor information from a sensor that tracks athletes and flying objects released by the athletes during a game; and display means for displaying the game video and virtual video in virtual space using the 3D display data included in the distribution data, the virtual video including flying video of virtual athletes and virtual flying objects from a viewpoint position in the virtual space that corresponds to the position and posture of the user, synchronized in time with the game video. [Effects of the Invention]

[0012] According to the present invention, it is possible to provide a new experience in which the user can not only watch the performance of athletes from video, but also become the main character himself. [Brief explanation of the drawings]

[0013] [Figure 1] FIG. 1 is a schematic configuration diagram of a virtual reality providing system according to the first embodiment. [Figure 2] FIG. 2 is a diagram showing the coordinate system of each parameter of the tracking data. [Figure 3] FIG. 3 is a diagram for explaining the amount of change. [Figure 4] FIG. 4 is a diagram for explaining the data structure of the tracking data. [Figure 5] FIG. 5 is a block diagram of the three-dimensional display data generation server 3 configured as a computer system. [Figure 6] FIG. 6 is a diagram for explaining the data structure of the three-dimensional display data. [Figure 7] FIG. 7 is a block diagram of the head-mounted display system 5. [Figure 8] FIG. 8 is a diagram showing an example of a three-dimensional image displayed on the display unit 511. As shown in FIG. [Figure 9] FIG. 9 is a sequence diagram showing the flow of processing and information from the tracking system 2 to the head-mounted display system 55 in the first embodiment. [Figure 10] FIG. 10 is a schematic configuration diagram of a virtual reality providing system in a modified example of the first embodiment. [Figure 11] FIG. 11 is a sequence diagram showing the flow of processing and information from the broadcast facility 7 to the head-mounted display system 5 in a modified example of the first embodiment. [Figure 12] FIG. 12 is a schematic configuration diagram of a virtual reality providing system in the second embodiment. [Figure 13] FIG. 13 is a block diagram of the distribution data generation server 10 configured as a computer system. [Figure 14] FIG. 14 is a block diagram of a head mounted display system 5 according to the second embodiment. [Figure 15] FIG. 15 is a sequence diagram showing the flow of processing and information from the live video generating unit 9 to the head mounted display system 5 in the second embodiment. [Figure 16] FIG. 16 is a block diagram of a head mounted display system 5 according to the third embodiment. [Figure 17] FIG. 17 is a diagram showing an example of an image when a virtual ball collides with a virtual bat. [Figure 18] FIG. 18 is a diagram showing an example of a trajectory image of the virtual ball after it collides with the virtual bat. [Figure 19] FIG. 19 is a diagram showing an example of a plurality of different virtual objects. [Figure 20] FIG. 20 is a schematic configuration diagram of a virtual reality providing system in the fourth embodiment. [Figure 21] FIG. 21 is a block diagram of the tallying server 11 configured as a computer system. [Figure 22] FIG. 22 is an example of an image of a statistical result when the virtual flying object is a ball and the virtual implement is a bat. [Figure 23] FIG. 23 is a diagram showing an example of a video in which a live video and a statistical image are combined. [Figure 24] FIG. 24 is a schematic diagram of a virtual reality providing system according to the fifth embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0014] (First embodiment) A virtual reality providing system according to a first embodiment will be described with reference to the drawings. Fig. 1 is a schematic diagram of the virtual reality providing system according to the first embodiment.

[0015] In Figure 1, 1 is a real athlete playing a game in a stadium, etc., 2 is a tracking system, 3 is a three-dimensional (3D) display data generation server, 4 is a broadcast transmission system, 5 is a head-mounted display system (virtual space provision system), 6 is a camera, 7 is broadcasting equipment, and 8 is a television.

[0016] In the following description of the embodiment, a head-mounted display system will be used as an example of a virtual space provision system. However, the virtual space provision system is not limited to a head-mounted display system, and may be any system capable of presenting a three-dimensional virtual space to a user. For example, the system may display images captured simultaneously for the left and right eyes on a screen, and provide the user with a virtual space by having the user wear glasses using a linear polarization filter, a liquid crystal shutter, or the like to view the images.

[0017] Athlete 1 is an athlete playing a game that is being broadcast live on television, such as a professional baseball game. The baseball game is filmed by camera 6, and the footage of the game is broadcast by broadcasting equipment 7 and viewed on television 8. Note that this embodiment uses a baseball game as an example, but it is not limited to baseball; other sports such as tennis, table tennis, golf, soccer, and martial arts (boxing and fencing) are also possible.

[0018] The tracking system 2 is a system that tracks and digitizes the athlete 1 and any flying object thrown by the athlete 1. Here, a flying object is typically a ball thrown or kicked by the athlete 1, but is not limited to this. It could also be a part of the athlete 1's body (for example, a punch in boxing) or equipment worn by the athlete 1 (for example, a fencing sword).

[0019] The tracking system 2 comprises a sensor 21 such as a radar or a camera, and a tracking data generator 22 that generates tracking data of the athlete 1, the ball, and the referee on the pitch from data obtained from the sensor 21. Typical examples of such a tracking system 2 include systems such as TrackMan and PITCHf / x. Note that in this embodiment, tracking data of all moving objects on the pitch (athlete 1, referee, etc.) is not required; it is sufficient for the tracking system 2 to be able to generate tracking data of at least a flying object (for example, the ball) released by the athlete 1.

[0020] When tracking a ball thrown by a pitcher, the following parameters are an example of tracking data generated by tracking data generator 22 in this embodiment. As shown in Fig. 2, the coordinate system for each parameter is a coordinate system with the origin at a predetermined position of home plate, the direction from the catcher to the pitcher as the front, the direction to the right of the catcher as the X direction (X axis), the direction from the catcher to the pitcher as the Y direction (Y axis), and the height direction as the Z direction (Z axis).

[0021] (1) The coordinates of the release point where the ball leaves the pitcher (release point coordinates) (2) Ball speed (initial velocity) (3) Coordinates of a predetermined passing point on home base (coordinates of the actual impact point) (4) The amount of change in the vertical direction (Z direction) and horizontal direction (X direction) from the vicinity of the release point to the vicinity of home base Here, the amount of change is the difference in both the vertical and horizontal directions between the coordinates of the provisional landing point on home base of a ball that has the same vector at the time of release but has no rotation and does not deviate at all, and the coordinates of the landing point of the ball that was actually thrown, as shown in Figure 3.

[0022] The tracking data may be of any type as long as it is in a format that represents the movements of the athlete 1, the ball, etc. It is also preferable that the tracking system 2 include the time at which the tracking data was acquired in the tracking data, as shown in FIG.

[0023] The three-dimensional (3D) display data generation server 3 generates three-dimensional display data (three-dimensional coordinate data) from tracking data for displaying a flying object such as a ball as a virtual flying object in a virtual space, and a real athlete 1 as a virtual moving object in a virtual space, in a head-mounted display system 5.

[0024] The 3D display data generation server 3 includes a tracking data acquisition unit 31 and a 3D display data generation unit 32. The tracking data acquisition unit 31 acquires tracking data provided from the tracking system 2. The 3D display data generation unit 32 generates 3D display data from the tracking data for displaying a virtual flying object such as a virtual ball and a virtual moving object such as a virtual athlete 1 on the head-mounted display system 5. Note that 3D display data for displaying the virtual moving object of the real athlete 1 is not necessarily required; it is sufficient if 3D display data for displaying at least a virtual flying object can be generated.

[0025] Specifically, the 3D display data generation server 3 can be realized by a computer system (information processing device) having a processor that performs various types of arithmetic processing, etc. Fig. 5 is a block diagram of the 3D display data generation server 3 configured by a computer system.

[0026] The 3D display data generation server 3 can be configured as a general-purpose computer having hardware resources such as a processor 300, memory (ROM or RAM) 301, storage device (hard disk, semiconductor disk, etc.) 302, input device (keyboard, mouse, touch panel, etc.) 303, display device 304, and communication device 305.

[0027] In the 3D display data generation server 3, a program stored in a storage device 302 is loaded into a memory 301 and executed by a processor 300, thereby realizing a tracking data acquisition process and a 3D display data generation process. The tracking data acquisition process corresponds to a tracking data acquisition unit 31, and the 3D display data generation process corresponds to a 3D display data generation unit 32.

[0028] The tracking data acquisition process is a process for acquiring tracking data provided by the tracking system 2.

[0029] The 3D display data generation process generates, from the tracking data, 3D display data for displaying a virtual flying object (a virtual ball in this example) and a virtual moving object such as a virtual athlete (a virtual athlete in this example) on the head-mounted display system 5. Specifically, when the tracking data is data related to the amount of deflection of a pitch, the angle of the ball hit, and the movements of athletes on the field, the batted ball, and the thrown ball, this process converts this data into 3D display data necessary for the head-mounted display system 5 to display the virtual ball and virtual athlete.

[0030] Here, the concept of 3D display data also includes data obtained by converting tracking data into a data format that can be processed by the head mounted display system 5. Therefore, the 3D display data generation process may be a process that simply converts tracking data into a data format that can be processed by the head mounted display system 5, or may be a process that generates trajectory data for a virtual flying object or the like from the tracking data.

[0031] Next, an example of generating 3D display data required for the head-mounted display system 5 to reproduce and display a virtual ball (virtual flying object) corresponding to a real ball from tracking data of the real ball will be described, in the case where the virtual flying object is a ball thrown by a pitcher.

[0032] In order for the head mounted display system 5 to reproduce and display the flying image of the virtual ball (virtual flying object), the flying image of the virtual ball (virtual flying object) can be reproduced and displayed if the following four elements are determined.

[0033] (1) Release point coordinates The release point coordinates can be obtained from tracking data. Note that the release point coordinates do not need to be precise coordinate values because they are aligned with the pitching motion of the virtual moving body of the pitcher displayed on the head-mounted display system 5.

[0034] (2) Acceleration (ball curvature force) added for each display frame of the head-mounted display system 5 The acceleration can be obtained by dividing the amount of change obtained from the tracking data by the display frame rate of the head-mounted display system 5.

[0035] (3) Air resistance to the ball The air resistance on the ball is set so that the final speed (the speed at the actual impact point coordinates) is 90%-92% of the initial speed.

[0036] (4) Initial velocity vector Of the initial velocity vectors, the Y-direction vector from the pitcher to home base can be calculated from the initial velocity obtained from the tracking data. However, the X- and Z-direction vectors from the pitcher's perspective cannot be obtained from the tracking data. On the other hand, even without the X- and Z-direction vectors of the initial velocity vector, it is possible to simulate a pitch with the exact same speed and amount of deflection, although the actual point of impact will be different. Therefore, by gradually changing the values of the X- and Z-direction vectors of the initial velocity vector, the ball's impact point will approach the original actual point of impact. The X- and Z-direction vectors when the ball's impact point is closest to the actual point of impact are used as the X- and Z-direction vectors.

[0037] The three-dimensional display data generation process generates these four elements as three-dimensional display data.

[0038] Furthermore, in the 3D display data generation process, for example, as shown in FIG. 6, timing information regarding the timing at which the head-mounted display system 5 displays the flying image of the virtual ball (virtual flying object) in synchronization with a predetermined video section of the video of the athlete 1's match broadcast by the broadcasting equipment 7 is included in the generated 3D display data.

[0039] The timing information may be of any type as long as it indicates the duration of the predetermined video section, such as time information or the time code of the video of athlete 1's match. The predetermined video section may be, for example, a section in which a commentator announces that viewers can experience a video of a virtual ball (virtual flying object) flying, or a section in which a notice is displayed on the broadcast video that viewers can experience a video of a virtual ball (virtual flying object) flying, but is not limited to these. The video section also includes audio only, such as in a radio broadcast. The predetermined video section may be acquired by having the 3D display data generation server 3 store the predetermined video section, or by receiving it from the broadcast equipment 7 as needed.

[0040] By using such timing information, for example, after the scene (video section) in which pitcher A throws the winning pitch, during the video section in which a notice is displayed that the winning pitch of pitcher A can be experienced, a flying video of the virtual ball (virtual flying object) of the winning pitch thrown by pitcher A can be reproduced on head-mounted display system 5. In this way, it is possible to link with television broadcasts and the like.

[0041] The timing information must identify the tracking data of the flying object actually shot by the athlete 1. This tracking data can be identified by using the acquisition time of the tracking data included in the tracking data.

[0042] 6, instead of timing information, three-dimensional display data identification information that can uniquely identify the three-dimensional display data may be included in the three-dimensional display data and transmitted to the head-mounted display system 5. Here, the three-dimensional display data identification information may be any information that can individually identify each piece of three-dimensional display data, such as the acquisition time of the tracking data described above or an identifier that uniquely identifies the three-dimensional display data.

[0043] When the three-dimensional display data includes three-dimensional display data identification information, after the three-dimensional display data including the three-dimensional display data identification information is distributed, execution information including the three-dimensional display data identification information for identifying the three-dimensional display data used to execute the virtual reality is transmitted from the three-dimensional display data generation server 3 or the broadcast transmission system 4 at the timing when the flying image of the virtual ball (virtual flying object) is displayed on the head-mounted display system 5. The head-mounted display system 5 receives the execution information and displays the flying image of the virtual ball (virtual flying object) on the head-mounted display system 5 using the three-dimensional display data specified by the three-dimensional display data identification information included therein.

[0044] By doing so, it is possible to link with television broadcasts and the like, just as in the case where timing information is included in the three-dimensional display data.

[0045] The three-dimensional display data is transmitted to at least one head-mounted display system 5 via a broadcast transmission system 4.

[0046] Furthermore, the 3D display data generation server 3 may not only generate 3D display data, but also acquire only the audio from the television footage of the game and transmit it together with the 3D display data. In this case, time information is also added to the audio data so that the acquisition time of the tracking data is synchronized with the audio data. In this way, as will be described later, when the virtual space image is displayed in the head-mounted display system 5, the live audio actually produced in the stadium is output, thereby realizing a more realistic virtual space.

[0047] The 3D display data generation server 3 may be configured with a single computer or may be configured with multiple computers using distributed computing. In order to speed up processing, some or all of the functions of the 3D display data generation server 3 may be realized using dedicated hardware (such as a GPU, FPGA, or ASIC).

[0048] The broadcast transmission system 4 is a system that broadcasts three-dimensional display data to a plurality of head-mounted display systems 5 connected to a network. An example of the broadcast transmission system 4 is a content delivery network (CDN).

[0049] The broadcasting system 4 preferably broadcasts the 3D display data in real time in synchronization with the live television broadcast of the game. One way to achieve this is to refer to the acquisition time of the tracking data included in the 3D display data and the time code on the television image, thereby broadcasting the 3D display data in a state where it tracks the television image.

[0050] A head-mounted display (HMD) system 5 has a display unit capable of displaying a three-dimensional image with depth to a user wearing the system. The system is configured to display a three-dimensional image of a virtual space in which various virtual objects, etc., generated by computational processing and image generation processing (described later), are arranged on the display unit. When the head-mounted display is worn on the user's head, the display unit is positioned to cover both of the user's eyes. The display unit displays images with parallax on the left and right image display surfaces, respectively, thereby displaying a three-dimensional image to the user. This allows a user wearing the head-mounted display on their head to experience a sensation close to reality as a three-dimensional image in a virtual space.

[0051] FIG. 7 is a block diagram of the head-mounted display system 5.

[0052] The head-mounted display system 5 includes a head-mounted display 51 and a computer 52.

[0053] The head-mounted display 51 includes a display unit 511 on which an image of a virtual space is displayed, a head position detection sensor 512, and a speaker 513.

[0054] The head position detection sensor 512 is configured to detect the position and orientation of the user's head in the absolute coordinate system of the user's head and output the detection signal to the computer 52. The head position detection sensor 512 is provided integrally with the head-mounted display 51, and therefore, when the head-mounted display 51 is worn on the user's head, it can track the movement of the user's head and detect the position and orientation. The detection result is then fed back to a three-dimensional image of the virtual space displayed by the display unit 511 of the head-mounted display 51, making it possible to present a three-dimensional image seen from a viewpoint position corresponding to the movement of the user's head.

[0055] The speaker 513 outputs audio to the user, for example, audio corresponding to the situation in the virtual space. Note that the means for transmitting audio to the user is not limited to the speaker 513, and other audio output means such as earphones may also be used.

[0056] Examples of head-mounted displays 51 having such functions include HTC Vive and Oculus Rift from Oculus VR, Inc. The head position detection sensor 512 may be provided separately from the head-mounted display 51 and configured to be worn on the user's head.

[0057] The computer 52 is used to generate images to be displayed on the display unit 511 of the head-mounted display 51 and to perform various arithmetic processing, etc. The computer 52 can be configured as a general-purpose computer having hardware resources such as a processor 520, a memory (ROM or RAM) 521, a storage device (hard disk, semiconductor disk, etc.) 522, an input device (keyboard, mouse, touch panel, etc.) 523, a display device 524, and a communication device 525.

[0058] The storage device 522 stores processing programs for generating images to be displayed on the display unit 511 of the head-mounted display 51 and for performing various calculation processes. In addition, various object data is stored that is used by the processor 520 to generate a virtual athlete (virtual moving object) in the virtual space of athlete 1, a virtual ball (virtual flying object) released by the virtual athlete, a virtual home base that serves as the background, and the like.

[0059] Then, the program, object data, and received 3D display data stored in the storage device 522 are loaded into the memory 521, and the processor 520 generates images of the virtual space to be displayed on the display unit 511 of the head-mounted display 51 based on the program.

[0060] Processor 520 performs image generation processing based on a program. The image generation processing uses well-known computer graphics technology to sequentially generate 3D images of the virtual athlete's movements and the flying image of the virtual ball using 3D display data of the virtual ball (virtual flying object) released from the virtual athlete (virtual moving object), the position and posture of the user's head detected by head position detection sensor 512, and various object data such as the virtual athlete (virtual moving object) and the virtual ball (virtual flying object), and displays these on display unit 511. At this time, if the 3D display data includes timing information, the image generation processing specifies a displayable period based on the time information and video time code included in the timing information, and displays 3D images such as the flying image of the virtual ball only during that period.

[0061] For example, during a broadcast video section displaying a notice that the user can experience pitcher A's finishing pitch, the pitching motion of the pitcher, which is a virtual moving body, is displayed on display unit 511 from object data, and a flight image of a virtual ball (virtual flying object) thrown from the pitcher, which is a virtual moving body, is displayed on display unit 511 using three-dimensional display data including release point coordinates, acceleration (the force that causes the ball to curve), air resistance to the ball, and an initial velocity vector. Fig. 8 is a diagram showing an example of a three-dimensional image displayed on display unit 511. The three-dimensional image shown in Fig. 8 displays the virtual pitcher, a flight image of the virtual ball, a virtual stadium, etc., as viewed from a viewpoint position in the virtual space corresponding to the user's position and posture.

[0062] In addition, when the three-dimensional display data includes three-dimensional display data identification information and a three-dimensional image such as a flying image of a virtual ball is displayed based on the distributed execution information, in response to receiving the execution information, a process of displaying a three-dimensional image such as a flying image of a virtual ball is started using the three-dimensional display data identified by the three-dimensional display data identification information included in the execution information.

[0063] The above-described head-mounted display system 5 is an example and is not limited to this. For example, head-mounted displays include Virtual Reality (VR) type head-mounted displays that display only a virtual space created by computer graphics or the like, and Mixed Reality (MR) type head-mounted displays that display a real space (for example, a real space displayed optically through the real space) and a virtual space in real time. The present invention can be applied to either type of head-mounted display. An example of an MR type head-mounted display is Microsoft's HoloLens.

[0064] Furthermore, in the above-described head-mounted display system 5, an example has been described in which the head-mounted display 51 and the computer 52 are configured in physically separate housings, but this is not limiting. The head-mounted display 51 and the computer 52 may be integrated to configure the head-mounted display system 5. An example of a head-mounted display system 5 in which the head-mounted display 51 and the computer 52 are integrated is the above-described Microsoft HoloLens.

[0065] Furthermore, the head-mounted display system 5 does not have to be a dedicated head-mounted display system. For example, a smartphone mounted on a goggle-type headset can achieve the same functionality as a head-mounted display. An example of such a system is the Galaxy Gear VR by Samsung.

[0066] Next, the operation of this embodiment will be described. Fig. 9 is a sequence diagram showing the flow of processing and information from the tracking system 2 to the head mounted display system 5 in the first embodiment.

[0067] First, tracking system 2 senses in real time using radar, cameras, etc., the ball thrown by athlete 1 (pitcher) on the pitch during a live broadcast of a game. Note that athlete 1, umpire, etc. on the pitch may also be sensed in real time using radar, cameras, etc. (Step 1). From the sensed data, tracking data is generated for at least the ball thrown by athlete 1 (pitcher). Note that tracking data for athlete 1, umpire, etc. on the pitch may also be generated (Step 2). At this time, time information for the tracking data is also acquired. The tracking data, including the time information, is then transmitted to 3D display data generation server 3.

[0068] The 3D display data generation server 3 acquires tracking data (Step 3). Based on the acquired tracking data, the 3D display data generation server 3 generates 3D display data for displaying a video of the virtual ball flying on the head-mounted display system 5. 3D display data for displaying a virtual athlete may also be generated (Step 4). At this time, timing information for displaying the video of the virtual ball flying is included in the 3D display data, using the time at which the tracking data was acquired. Furthermore, the time at which the tracking data was acquired may be included in the generated 3D display data. The 3D display data is then transmitted to the broadcast transmission system 4.

[0069] The broadcast transmission system 4 broadcasts the received 3D display data to multiple head-mounted display systems 5 connected to the network (Step 5). If the 3D display data includes the acquisition time of the tracking data, the 3D display data is broadcast in order of proximity to the time code information of the video broadcast on television. In this way, the 3D display data can be broadcast in sync with the television broadcast of the game, maintaining a certain degree of real-timeness.

[0070] The head-mounted display system 5 receives the three-dimensional display data (Step 6).

[0071] If the 3D display data includes timing information, head-mounted display system 5 identifies a displayable period based on the time information and the time code of the image included in the timing information, and displays a 3D image (flying image) of the virtual ball viewed from a viewpoint corresponding to the movement of the head of the user wearing the device during that period (Step 7). Furthermore, if the 3D display data includes 3D display data identification information, head-mounted display system 5 displays a 3D image (flying image) of the virtual ball viewed from a viewpoint corresponding to the movement of the head of the user wearing the device at the timing of receiving execution information including the 3D display data identification information (Step 7). At this time, the virtual pitcher (virtual athlete) throwing the virtual ball may be displayed using object data of the virtual pitcher stored in advance in head-mounted display system 5, or, if display data of a real pitcher is stored in the 3D display data, the virtual pitcher may be displayed using that data.

[0072] In the first embodiment, information on the movements of athletes, the ball, etc. is broadcast to a head-mounted display system while maintaining a certain degree of synchronization with the broadcast video of a sporting event, etc. This allows users to not only watch the athletes' performances from the video, but also experience the athletes' performances from their own viewpoint by using the head-mounted display system.

[0073] It is not necessary to generate and transmit 3D display data for all tracking data. Although the scenes that can be experienced are limited, it is also possible to generate and broadcast 3D display data for only limited athletes (for example, only the pitcher) or flying objects (for example, only the ball thrown by the pitcher). Similarly, it is not necessary to generate and transmit 3D display data for tracking data throughout the entire game. It is also possible to generate 3D display data for tracking data for a portion of the game, and generate and broadcast 3D display data only for that portion of the game.

[0074] (Modification of the first embodiment) A modified example of the first embodiment will be described. In the modified example of the first embodiment, an example will be described in which hybridcast is used as a means for transmitting 3D display data. Fig. 10 is a schematic configuration diagram of a virtual reality providing system in the modified example of the first embodiment.

[0075] If the head-mounted display system 5 and the television 8 are compatible with the Hybridcast standard, the head-mounted display system 5 is connected to the television 8 via the Hybridcast standard. If the broadcast program is a program-linked broadcast using Hybridcast, the broadcast transmission system 4 functions as a Hybridcast server. Then, prior to the timing designated by the broadcast station, the broadcast transmission system 4 transmits 3D display data such as a virtual flying object (virtual ball) acquired from the 3D display data generation server 3 to the head-mounted display system 5, including 3D display data identification information that can uniquely identify the 3D display data. Here, the 3D display data identification information may be any information that can individually identify each piece of 3D display data, such as the acquisition time of the tracking data described above, or an identifier that individually identifies the 3D display data.

[0076] The broadcasting facility 7 broadcasts an event message that stores three-dimensional display data identification information that identifies three-dimensional display data used when executing the virtual reality, at the timing when the virtual reality is executed in the head-mounted display system 5. Here, the three-dimensional display data identification information is, for example, the acquisition time of the tracking data described above or an identifier that individually identifies the three-dimensional display data.

[0077] The television 8 detects and analyzes the event message broadcast from the broadcasting facility 7, and transmits the 3D display data identification information stored in the event message to the head-mounted display system 5. The image generation processing unit 520 of the head-mounted display system 5 uses the 3D display data having the 3D display data identification information transmitted from the television 8 among the received 3D display data to sequentially generate 3D images of the movements of the virtual athlete and the flight image of the virtual ball, and displays them on the display unit 511, in the same manner as in the first embodiment described above.

[0078] Next, a description will be given of the operation of the modified example of the first embodiment. Fig. 11 is a sequence diagram showing the flow of processing and information from the broadcast facility 7 to the head mounted display system 5 in the modified example of the first embodiment.

[0079] First, the broadcasting equipment 7 broadcasts live footage of the game being broadcast (Step 1).

[0080] TV 8 receives the broadcast signal and displays the live footage (Step 2).

[0081] Meanwhile, tracking system 2 senses in real time using radar, cameras, etc., the ball thrown by athlete 1 (pitcher) on the pitch during the game being broadcast. Note that athlete 1, umpire, etc. on the pitch may also be sensed in real time using radar, cameras, etc. (Step 3). From the sensed data, tracking data is generated for at least the ball thrown by athlete 1 (pitcher). Note that tracking data for athlete 1, umpire, etc. on the pitch may also be generated (Step 4). The tracking data is then transmitted to 3D display data generation server 3.

[0082] The 3D display data generation server 3 acquires tracking data (Step 5). Based on the acquired tracking data, the 3D display data generation server 3 generates 3D display data for displaying a flight image of the virtual ball on the head-mounted display system 5 (Step 6). Note that 3D display data for displaying a virtual athlete may also be generated (Step 4). At this time, a 3D display data identifier for identifying the 3D display data is included in the generated 3D display data. The 3D display data is then transmitted to the broadcast transmission system 4. Note that the 3D display data identifier used here is a 3D display data identifier that has been shared in advance with the broadcast equipment 7.

[0083] The broadcast transmission system 4 transmits the received three-dimensional display data to the head-mounted display system 5 using hybridcast (Step 7).

[0084] The head-mounted display system 5 receives the three-dimensional display data (Step 8).

[0085] The broadcasting facility 7 broadcasts the event message storing the three-dimensional display data identifier at the timing when the virtual reality is executed in the head-mounted display system 5 (Step 9).

[0086] The television 8 detects the event message broadcast from the broadcasting facility 7, analyzes the event message, and transmits a three-dimensional display data identifier to the head-mounted display system 5 (Step 10).

[0087] Head-mounted display system 5 compares the 3D display data identifier received from television 8 with the 3D display data identifier of the received 3D display data (Step 11). Then, using the 3D display data having the 3D display data identifier received from television 8, it displays a 3D image (flying image) of the virtual ball viewed from a viewpoint position corresponding to the movement of the head of the user wearing the device (Step 12). At this time, the virtual pitcher (virtual athlete) throwing the virtual ball may be displayed using object data of the virtual pitcher stored in advance in head-mounted display system 5, or, if display data of a real pitcher is stored in the 3D display data, the virtual pitcher may be displayed using that data.

[0088] As with the first embodiment, it is not necessary to generate and transmit 3D display data for all tracking data. Although the scenes that can be experienced are limited, it is also possible to generate and broadcast 3D display data for only limited athletes (for example, only the pitcher) or flying objects (for example, only the ball thrown by the pitcher). Similarly, it is not necessary to generate and transmit 3D display data for tracking data throughout the entire game. The broadcasting station may generate 3D display data for tracking data of predetermined players and for a certain period of the game, generate and broadcast 3D display data only for those players and periods, and broadcast an event message including the corresponding 3D display data identification information.

[0089] With this configuration, the broadcasting station can specify the timing for executing virtual reality on the head-mounted display system 5 as a program-linked service, and the head-mounted display system 5 can provide virtual reality linked to the program footage of a sports game or other event being broadcast.

[0090] (Second embodiment) In the second embodiment, an example is described in which 3D display data of tracking data is transmitted together with live-streamed game footage using a content delivery network (CDN), etc. Fig. 12 is a schematic diagram of a virtual reality providing system in the second embodiment.

[0091] In the second embodiment, unlike the first embodiment, the video of the baseball game captured by the camera 6 is not broadcast on television, but is broadcast to terminals such as tablets and smartphones via a content delivery network system. The second embodiment differs from the first embodiment in that 3D display data is transmitted together with the video data of the game.

[0092] The second embodiment is provided with a live video generation unit 9 that generates video data of live video of the match, and a distribution data generation server 10 that generates distribution data by incorporating 3D display data into the video data of the live video of the match. Note that components similar to those in the first embodiment are denoted by the same reference numerals.

[0093] Specifically, the distribution data generation server 10 can be realized by a computer system having a processor that performs various types of arithmetic processing, etc. Fig. 13 is a block diagram of the distribution data generation server 10 configured by a computer system.

[0094] The distribution data generation server 10 can be configured as a general-purpose computer having hardware resources such as a processor 100, memory (ROM or RAM) 101, a storage device (hard disk, semiconductor disk, etc.) 102, an input device (keyboard, mouse, touch panel, etc.) 103, a display device 104, and a communication device 105.

[0095] In the distribution data generation server 10, a program stored in the storage device 102 is loaded into the memory 101 and executed by the processor 100, thereby realizing the distribution data generation process.

[0096] The processor 100 performs a receiving process for receiving live video data and three-dimensional display data, and a distribution data generating process for generating distribution data.

[0097] The distribution data generation process generates distribution data in a format that allows the head-mounted display system 5 to separate live video data and 3D display data. For example, an identifier that identifies the data as video data is added to the header of a video data packet, and an identifier that identifies the data as 3D display data is added to the header of a 3D display data packet, and distribution data is generated by doing so. The head-mounted display system 5 can separate the video data and 3D display data using the identifier. Furthermore, if the payload of each video data packet has a field that can contain data other than video data, the 3D display data may be inserted into that field in an identifiable format.

[0098] Furthermore, the distribution data generation process references the time code of the video data and the acquisition time of the tracking data included in the 3D display data, and generates the distribution data so that the data are synchronized with each other. For example, if the payload of each packet of video data has a field that can contain data other than video data, 3D display data having the same tracking data acquisition time as the time code of that packet is inserted into that field. This example is merely an example, and other methods are also possible.

[0099] The distribution data generation server 10 transmits the generated distribution data to the broadcast transmission system 4.

[0100] The broadcast transmission system 4 broadcasts the video to the head-mounted display system 5 and to tablets and smartphones on which the video is viewed.

[0101] FIG. 14 is a block diagram of a head mounted display system 5 according to the second embodiment.

[0102] In the head-mounted display system 5, the processor 520 performs a data separation process to separate the video data and the 3D display data using a predetermined separation method. Specifically, if an identifier that identifies the type of each data is assigned to a packet, the video data and the 3D display data are separated based on the identifier. Also, if the payload of each video data packet contains 3D display data together with an identifier, the video data and the 3D display data are separated based on the identifier.

[0103] In the image generation process, a three-dimensional image of the virtual space is displayed on the display unit, as in the first embodiment.

[0104] If the head-mounted display system 5 is a system that allows the user to view distributed video data, the user may be allowed to select either live video or a three-dimensional video in a virtual space.

[0105] Next, the operation of this embodiment will be described. Fig. 15 is a sequence diagram showing the processing and information flow from the live video generation unit 9 to the head mounted display system 5 in the second embodiment. Note that the same step numbers are assigned to operations similar to those in the first embodiment.

[0106] First, the live video generation unit 9 generates video data of the game being broadcast (Step 1). The generated video data is transmitted to the distribution data generation server 10.

[0107] The tracking system 2 senses, in real time, the ball thrown by the athlete 1 (pitcher) on the pitch during the game being broadcast using radar, cameras, etc. Note that the athlete 1, umpire, etc. on the pitch may also be sensed in real time using radar, cameras, etc. (Step 1). From the sensed data, tracking data is generated for at least the ball thrown by the athlete 1 (pitcher). Note that tracking data for the athlete 1, umpire, etc. on the pitch may also be generated (Step 2). At this time, time information for the tracking data is also acquired. The tracking data including the time information is then sent to the 3D display data generation server 3.

[0108] The 3D display data generation server 3 acquires tracking data (Step 3). Based on the acquired tracking data, the 3D display data generation server 3 generates 3D display data for displaying a flight image of the virtual ball on the head-mounted display system 5 (Step 4). Note that 3D display data for displaying a virtual athlete may also be generated. In this case, the acquisition time of the tracking data is included in the generated 3D display data. The 3D display data is then transmitted to the distribution data generation server 10.

[0109] The distribution data generation server 10 generates distribution data by incorporating the 3D display data into the video data of the live video of the match (Step 11). The generated distribution data is transmitted to the broadcast transmission system 4.

[0110] The broadcast transmission system 4 broadcasts the received distribution data to a plurality of head-mounted display systems 5, tablets, and smartphones (not shown) connected to the network (Step 5).

[0111] The head-mounted display system 5 receives the distribution data (Step 6). The head-mounted display system 5 performs a data separation process to separate the video data and the 3D display data using a predetermined separation method (Step 12). The head-mounted display system 5 then displays a 3D video (flying video) of the virtual ball as seen from a viewpoint corresponding to the movement of the head of the user wearing the head-mounted display system 5 (Step 7). At this time, the virtual pitcher (virtual athlete) throwing the virtual ball may be displayed using object data of the virtual pitcher stored in advance in the head-mounted display system 5. Alternatively, if display data of a real pitcher is stored in the 3D display data, the virtual pitcher may be displayed using that data.

[0112] Unlike television broadcasting, the second embodiment allows the data distributor to synchronize the distribution of 3D display data with the distribution of live video, allowing users to experience the performance of athletes in a time that is more synchronized with the current game.

[0113] (Modification of the second embodiment) In the second embodiment, as in the first embodiment, it is also possible to include timing information in the 3D display data. Furthermore, the 3D display data may not include the acquisition time of the tracking data, but may include only the timing information.

[0114] In this case, the distribution data generation server 10 generates distribution data in a format that allows the head-mounted display system 5 to separate the 3D display data and the video data before the timing (predetermined video section) at which the 3D video (flying video) specified by the timing information is displayed.

[0115] In this way, the second embodiment can also achieve the same effects as the first embodiment.

[0116] Furthermore, instead of timing information, the 3D display data may include 3D display data identification information that can uniquely identify the 3D display data, and may be transmitted to the head mounted display system 5.

[0117] When the three-dimensional display data includes three-dimensional display data identification information, after the three-dimensional display data including the three-dimensional display data identification information is distributed, execution information including the three-dimensional display data identification information for identifying the three-dimensional display data used when executing the virtual reality is transmitted from the distribution data generation server 10 at the timing when the flying image of the virtual ball (virtual flying object) is displayed on the head-mounted display system 5. The head-mounted display system 5 receives the execution information, and displays the flying image of the virtual ball (virtual flying object) on the head-mounted display system 5 using the three-dimensional display data specified by the three-dimensional display data identification information included therein.

[0118] In this way, the second embodiment can also achieve the same effects as the first embodiment.

[0119] (Third embodiment) The third embodiment is configured such that, in addition to the first embodiment (including a modified version of the first embodiment) or the second embodiment (including a modified version of the second embodiment), the user is provided with equipment similar to that used in a game (e.g., a bat or glove), and is able to experience hitting or catching a flying object (e.g., a ball) reproduced from tracking data obtained from a real athlete 1.

[0120] FIG. 16 is a block diagram of a head mounted display system 5 according to the third embodiment.

[0121] In the third embodiment, an equipment tracking acquisition unit 520 is provided for tracking the movement of an equipment used by a user while the user is wearing a head-mounted display 51. The equipment tracking acquisition unit 520 may use Microsoft's Kinect (registered trademark) or ASUS's Xtion Pro (registered trademark), etc. It is also possible to provide an acceleration sensor in the equipment to acquire tracking data of the equipment.

[0122] In the head-mounted display system 5, in addition to the operations of the first embodiment (including a modified example of the first embodiment) and the second embodiment (including a modified example of the second embodiment), the processor 520 of the computer 52 performs processing to generate 3D tool display data.

[0123] The equipment three-dimensional display data generation process calculates three-dimensional display data (trajectory data) of a virtual object corresponding to the equipment based on the equipment tracking data acquired by the equipment tracking acquisition unit 520. Also, based on the three-dimensional display data (trajectory data) of the virtual flying object and the three-dimensional display data (trajectory data) of the virtual flying object, it calculates three-dimensional display data (trajectory data) of a new virtual flying object resulting from a collision between the virtual flying object and the virtual flying object in the virtual space.

[0124] Specifically, when the virtual flying object is a virtual ball and the virtual implement is a virtual bat, the user swings the bat in time with the flight image of the virtual ball thrown by the virtual athlete. The equipment 3D display data generation process calculates 3D display data (trajectory data) of the virtual bat swing using tracking data of the bat. The equipment 3D display data generation process then determines whether the virtual bat and the virtual ball will collide in the virtual space (determines whether the virtual ball has hit the virtual bat) based on the 3D display data (trajectory data) of the virtual bat and the 3D display data (trajectory data) of the virtual ball. If a collision occurs, the image generation process generates 3D display data (trajectory data) of the flight trajectory of the virtual ball after the collision based on the size and restitution coefficient of the virtual bat. The image generation process then generates and displays a flight image of the virtual ball after the collision based on the 3D display data (trajectory data) of the flight trajectory of the virtual ball. FIG. 17 shows an example of an image when the virtual ball and the virtual bat collide, and FIG. 18 shows an example of an image of the virtual ball flying after the collision between the virtual ball and the virtual bat.

[0125] It is also possible to provide a plurality of sizes and restitution coefficients of virtual equipment corresponding to the equipment used by the user, and allow the user to select from them. For example, if the equipment is a bat, as shown in FIG. 19, a plurality of virtual bats of different sizes and restitution coefficients may be provided, and the user may select from them. The advantage of this is that even if the ball is not hit with a normal equipment, a large virtual bat with a large restitution coefficient makes it easier to hit the virtual ball, and the virtual ball will fly a longer distance. As a result, the gameplay is enhanced.

[0126] In the third embodiment, the user can not only watch the performance of an athlete, but also hit and catch virtual flying objects similar to balls thrown by real athletes, thereby challenging the performance of the athlete.

[0127] (Fourth embodiment) The fourth embodiment is an embodiment in which, in addition to the third embodiment, collision results of whether a virtual flying object has collided with a virtual concrete object in each head-mounted display system 5 and flight results of the virtual flying object obtained from new trajectory data of the virtual flying object are compiled, and the compiled results are reflected in the live video.

[0128] FIG. 20 is a schematic configuration diagram of a virtual reality providing system in the fourth embodiment.

[0129] The computer 52 of the head-mounted display system 5 determines whether a virtual flying object has collided with a virtual concrete object. For example, it determines whether a virtual ball has collided with a virtual bat, i.e., whether the virtual ball has been hit. If a virtual flying object has collided with a virtual concrete object, it calculates the flight result of the virtual flying object. For example, it determines whether the ball hit by the virtual ball was a hit or a home run. These collision results and flight results are then transmitted to the tallying server 11. Furthermore, the collision results and flight results of each user are managed individually.

[0130] Specifically, the tallying server 11 can be realized by a computer system having a processor that performs various types of arithmetic processing, etc. Fig. 21 is a block diagram of the tallying server 11 configured by a computer system.

[0131] The aggregation server 11 can be configured as a general-purpose computer having hardware resources such as a processor 110, memory (ROM or RAM) 111, storage device (hard disk, semiconductor disk, etc.) 112, input device (keyboard, mouse, touch panel, etc.) 113, display device 114, and communication device 115.

[0132] The tallying server 11 performs tallying processing by loading a program stored in the storage device 112 into the memory 111 and executing it with the processor 110. The tallying processing tallying the collision results and flight results transmitted from each head mounted display system 5. For example, the tallying processing tallying the number of head mounted display systems 5 that transmitted the collision results and flight results, the ratio of collision results (collided) to the number of head mounted display systems 5, or the ratio of flight results.

[0133] Furthermore, the tallying server 11 performs a tally result image generation process to generate a tally result image, which is an image of the tally result. Fig. 22 is an example of an image of the statistical result when the virtual flying object is a ball and the virtual implement is a bat.

[0134] The tallying server 11 transmits the generated statistical image to a broadcasting facility or a multicast transmission system.

[0135] In the broadcasting facility or the multicast transmission system, the live video and the generated statistical image are combined and broadcast or multicast. Fig. 23 shows an example of a video in which the live video and the statistical image are combined.

[0136] It goes without saying that the fourth embodiment can be applied not only to broadcasting live video, but also to transmitting live video through a content delivery network system.

[0137] In the fourth embodiment, the performance of the user wearing the head-mounted display is reflected in the live video, so that the user can not only watch the live broadcast, but also have a new experience in which he or she can become the main character.

[0138] (Fifth embodiment) The fifth embodiment includes the first embodiment (including variations of the first embodiment) or the second embodiment (including variations of the second embodiment), and further includes a prediction server 12 that predicts the type of ball that an athlete 1, such as a pitcher, will next throw to a batter based on past performance.

[0139] FIG. 24 is a schematic diagram of a virtual reality providing system according to the fifth embodiment.

[0140] The prediction server 12 includes a tracking data acquisition unit 121 and a prediction unit 122 .

[0141] The tracking data acquisition unit 121 acquires tracking data provided from the tracking system 2. The acquired tracking data also includes data identifying the pitcher and the opposing batter.

[0142] The prediction unit 122 performs machine learning on the types of balls that each pitcher has previously thrown to each batter, and when the acquired tracking data is input, predicts the type of ball that the pitcher will next throw to the batter. The predicted type of ball is sent to the 3D display data generation server 3 in the same format as the tracking data.

[0143] The 3D display data generation server 3 acquires tracking data from the prediction server 12. As in the above-described embodiment, the 3D display data generation unit 32 generates 3D display data for displaying a flight image of the virtual ball of the predicted pitch type on the head-mounted display system 5. The generated 3D display data is then transmitted to at least one or more head-mounted display systems 5 via the broadcast transmission system 4.

[0144] In addition to the effects of the first embodiment (including variations of the first embodiment) or the second embodiment (including variations of the second embodiment) described above, the fifth embodiment allows the user to experience the type of ball that the pitcher is predicted to throw next.

[0145] The fifth embodiment can be configured in combination with not only the first embodiment (including modifications of the first embodiment) but also the second to fourth embodiments.

[0146] Furthermore, some or all of the above-described embodiments can be described as, but are not limited to, the following supplementary notes.

[0147] (Appendix 1) At least one virtual space providing system that provides a user with a three-dimensional image of the virtual space; a tracking data acquisition means for acquiring tracking data of at least a flying object shot by a real athlete on the pitch, the tracking data being acquired from a sensor; a three-dimensional coordinate data generating means for generating three-dimensional coordinate data for displaying a virtual flying object corresponding to the flying object in the virtual space displayed in the virtual space providing system based on the acquired tracking data; a transmitting means for transmitting the three-dimensional coordinate data to the at least one virtual space providing system via a broadcasting means; A virtual reality providing system having:

[0148] (Appendix 2) the tracking data includes tracking data of real athletes' movements on a pitch; The three-dimensional coordinate data generating means generates three-dimensional coordinate data for displaying a virtual moving body corresponding to the real athlete in the virtual space displayed in the virtual space providing system based on the acquired tracking data. 10. The virtual reality delivery system of claim 1.

[0149] (Appendix 3) a distribution data generation means for generating distribution data including video data of actual video of the athlete's match and the three-dimensional coordinate data; the transmitting means transmits the distribution data to the at least one virtual space providing system via a broadcasting means; The virtual space providing system has a separation means for separating the video data and the three-dimensional coordinate data from the distribution data. 1. A virtual reality providing system according to claim 1 or 2.

[0150] (Appendix 4) The distribution data generating means generates the distribution data so that the video data of the real video and the three-dimensional coordinate data corresponding to the real video are synchronized. 4. A virtual reality delivery system as described in Appendix 3.

[0151] (Appendix 5) The virtual space providing system comprises: a storage means for storing object data of the virtual flying object and the virtual operating object, and object data of a virtual material object corresponding to an implement handled by the user; receiving means for receiving the three-dimensional coordinate data; a position and orientation detection means for detecting position data and orientation data of the user in the virtual space; an equipment tracking data acquisition means for acquiring tracking data of equipment handled by the user; a first calculation means for calculating three-dimensional coordinate data for displaying a virtual object corresponding to the tool in the virtual space displayed in the virtual space providing system based on the tracking data of the tool; a second calculation means for calculating three-dimensional coordinate data of a flight trajectory of the virtual flying object resulting from a collision between the virtual flying object and the virtual material object in the virtual space, based on the three-dimensional coordinate data of the virtual flying object and the three-dimensional coordinate data of the virtual material object; a first image generating means for generating trajectory images of the virtual flying object, the virtual operating object, and the virtual concrete object from a viewpoint position in the virtual space corresponding to the position and orientation of the user based on the three-dimensional coordinate data, the object data, and the user's position data and orientation data, and displaying the images on a display means of the virtual space providing system; a second image generating means for generating a trajectory image of the flight trajectory of the virtual flying object from a viewpoint position in the virtual space corresponding to the position and posture of the user based on three-dimensional coordinate data of the flight trajectory of the virtual flying object resulting from a collision between the virtual flying object and the virtual concrete object in the virtual space, and displaying the trajectory image on a display means of the virtual space providing system; 5. A virtual reality providing system as described in any one of claims 1 to 4.

[0152] (Appendix 6) the storage means stores flight object data relating to at least the sizes of a plurality of virtual flying objects and flight object restitution coefficient data relating to the restitution coefficients of the plurality of virtual flying objects relative to the virtual implement, and equipment data relating to at least the sizes of a plurality of virtual implements and equipment restitution coefficient data relating to the restitution coefficients of the plurality of virtual implements relative to the virtual flying objects; The second calculation means calculates new trajectory data of the virtual flying object resulting from a collision between the virtual flying object and the virtual implement by referring to the flying object data, the flying object restitution coefficient data, the implement data, and the restitution coefficient data. 6. A virtual reality delivery system as described in Appendix 5.

[0153] (Appendix 7) It has an aggregation server, The virtual space providing system comprises: a third calculation means for calculating a collision result of whether the virtual flying object and the virtual concrete object collide with each other and a flight result of the virtual flying object obtained from three-dimensional coordinate data of a flight trajectory of the virtual flying object; a transmission means for transmitting the collision result and the flight result to the aggregation server; and The aggregation server The virtual space providing system includes a means for calculating a result of aggregating the collision results and the flight results transmitted from the computer, and transmitting the result of aggregating the collision results and the flight results to a programming device for displaying the result of aggregating the collision results and the flight results on a broadcast image or a broadcast image of the real athlete. 7. A virtual reality providing system according to claim 5 or 6, comprising:

[0154] (Appendix 8) The virtual space providing system is a head-mounted display system that is worn on the user's head and includes a display means for displaying a three-dimensional image of the virtual space, and a three-dimensional image generating means for generating the three-dimensional image of the virtual space. 10. A virtual reality providing system according to any one of claims 1 to 7.

[0155] (Appendix 9) A computer program for a virtual space provision system that provides a user with a three-dimensional image of a virtual space, receiving, from a broadcast transmission means, three-dimensional coordinate data for displaying a virtual flying object corresponding to the flying object in a virtual space generated from tracking data of the flying object at least shot by the real athlete on the pitch, the tracking data being acquired from the sensor; detecting position data and orientation data of the user in the virtual space; generating a trajectory image of the virtual flying object from a viewpoint position in the virtual space corresponding to the position and orientation of the user based on the three-dimensional coordinate data of the virtual flying object, the object data of the virtual flying object, and the position data and orientation data of the user, and displaying the trajectory image on a display means of the virtual space providing system; A program that causes a computer to execute the following.

[0156] (Appendix 10) The program A process of acquiring tracking data of equipment handled by the user; calculating three-dimensional coordinate data of a virtual object corresponding to the tool based on the tracking data of the tool; a process of calculating three-dimensional coordinate data of a flight trajectory of the virtual flying object resulting from a collision between the virtual flying object and the virtual material object in the virtual space based on the three-dimensional coordinate data of the virtual flying object and the three-dimensional coordinate data of the virtual material object; generating trajectory images of the virtual flying object and the virtual material object from a viewpoint position in the virtual space corresponding to the position and orientation of the user based on the three-dimensional coordinate data, the object data, and the user's position data and orientation data, and displaying the images on a display means of the virtual space providing system; generating a trajectory image of the flight trajectory of the virtual flying object from a viewpoint position in the virtual space corresponding to the position and posture of the user based on three-dimensional coordinate data of the flight trajectory of the virtual flying object resulting from a collision between the virtual flying object and the virtual concrete object in the virtual space, and displaying the trajectory image on a display means of the virtual space providing system; 9. The program of claim 8, further comprising:

[0157] (Appendix 11) The program a process of calculating a collision result of whether the virtual flying object and the virtual concrete object collide with each other and a flight result of the virtual flying object obtained from three-dimensional coordinate data of a flight trajectory of the virtual flying object; a process of transmitting the collision results and the flight results to a server that compiles the collision results and the flight results; 12. The program according to claim 9 or 11, further comprising:

[0158] (Appendix 12) The virtual space providing system is a head-mounted display system that is worn on the user's head and includes a display means for displaying a three-dimensional image of the virtual space, and a three-dimensional image generating means for generating the three-dimensional image of the virtual space. 12. A program according to any one of appendices 9 to 11.

[0159] Although the present invention has been described above with reference to preferred embodiments, the present invention is not necessarily limited to the above-described embodiments and can be modified and implemented in various ways within the scope of its technical concept. [Explanation of symbols]

[0160] 1. Athletes (real athletes) 2. Tracking System 3. Three-dimensional (3D) display data generation server 4 Broadcast transmission system 5 Head-mounted display system 6. Camera 7 Broadcasting Equipment 8. Television 9 Live video generation unit 10 Distribution data generation server 11 Aggregation Server 12 Prediction Server

Claims

1. a tracking data acquisition unit that acquires, in real time, tracking data of the athlete and the projectile obtained from sensor information of a sensor that tracks the athlete and the projectile during a game; a three-dimensional display data generation unit that uses the acquired tracking data to generate three-dimensional display data for displaying a virtual image in a virtual space, the virtual image including a flight image of a virtual athlete corresponding to the athlete and a virtual flying object corresponding to the flying object; a distribution unit that distributes distribution data including game video of the game and the 3D display data that is synchronized in time with the game video; at least one or more display terminals including: a first display control unit that displays the game video of the distribution data; and a second display control unit that uses three-dimensional display data included in the distribution data to display in the virtual space a virtual video that includes a flying video of the virtual athlete and the virtual flying object from a viewpoint position in the virtual space that corresponds to the position and posture of the user, the virtual video being synchronized in time with the game video; A virtual reality providing system comprising:

2. a three-dimensional display data generation unit that uses tracking data of the athlete and the flying object obtained from sensor information of a sensor that tracks the athlete and the flying object released by the athlete during a game, the tracking data being acquired in real time, to generate three-dimensional display data for displaying a virtual image in a virtual space, the virtual image including a flying image of a virtual athlete corresponding to the athlete and a virtual flying object corresponding to the flying object; a distribution unit that distributes distribution data including game video of the game and the 3D display data that is temporally synchronized with the game video to at least one display terminal; Equipped with The game video can be displayed on the display terminal; the three-dimensional display data includes flying images of the virtual athlete and the virtual flying object from a viewpoint in the virtual space corresponding to the position and posture of the user, and is data for displaying the virtual images in the virtual space of the display terminal, the virtual images being synchronized in time with the game images; Data distribution device.

3. a receiving means for receiving distribution data including three-dimensional display data for displaying a virtual image including a flight image of a virtual athlete corresponding to the athlete and a virtual flying object corresponding to the flying object, the virtual image being generated using tracking data of the athlete and the flying object obtained in real time from sensor information of a sensor that tracks the athlete and the flying object released by the athlete during a game, and game video of the game; a first display control means for displaying the game video included in the distribution data; a second display control means for displaying in the virtual space a virtual image that includes a flying image of the virtual athlete and the virtual flying object from a viewpoint in the virtual space that corresponds to the position and posture of the user, the virtual image being synchronized in time with the game image, using three-dimensional display data included in the distribution data; A virtual space providing device comprising:

4. Computer, a receiving means for receiving distribution data including three-dimensional display data for displaying a virtual image including a flight image of a virtual athlete corresponding to the athlete and a virtual flying object corresponding to the flying object, the virtual image being generated using tracking data of the athlete and the flying object obtained from sensor information of a sensor that tracks the athlete and the flying object released by the athlete during a game, the tracking data being acquired in real time, and game video of the game; a first display control means for displaying the game video included in the distribution data; a second display control means for displaying in the virtual space a virtual image that includes a flying image of the virtual athlete and the virtual flying object from a viewpoint position in the virtual space that corresponds to the position and posture of the user, the virtual image being synchronized in time with the game image, using three-dimensional display data included in the distribution data; A program that functions as a

Images