Information processing device, information processing method, and program

Abstract

There is provided an information processing apparatus including circuitry configured to: receive motion data captured in an event from a first apparatus; receiving event metadata related to the event from the second device; and transmitting the motion data and the event metadata to the user device.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of Japanese Priority Patent Application JP 2023-019005 filed on February 10, 2023, the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present disclosure relates to an information processing device, an information processing method, and a program. Background Art

[0004] Recently, image analysis techniques have been improved. For example, motion capture techniques are known for extracting human skeletal positions (a plurality of points corresponding to each of various body parts such as the head, torso, and limbs) from captured images and recording changes in skeletal positions as human motion.

[0005] In recent years, such image analysis technology has been used for sporting events and the like; captured images obtained by imaging athletes are analyzed in real time to determine whether a specific event has occurred in the captured images. Furthermore, when a large number of subjects (athletes) are included in the captured images, the processing load of image analysis increases. Therefore, for example, in Patent Document 1 below, a more efficient image processing technology is implemented by selecting a portion of the captured image and analyzing an image smaller than the original image.

[0006] Reference List

[0007] Patent Literature

[0008] PTL 1: U.S. Patent Application Publication No. 2020 / 0151881 Summary of the Invention

[0009] Technical issues

[0010] Here, it is also possible to move virtual objects in a virtual space (e.g., avatars generated using 3DCG) based on motion data, which is motion information extracted and recorded from captured images. However, when implementing display control of virtual objects, it is not assumed that data other than motion data is used together.

[0011] Therefore, the present disclosure proposes an information processing device, an information processing method, and a program that are capable of obtaining and transmitting each of target motion data and event metadata during an event.

[0012] Solution to the problem

[0013] According to one aspect of the present disclosure, an information processing device is provided, comprising: a circuit system configured to: receive motion data captured in an event from a first device; receive event metadata related to the event from a second device; and send the motion data and event metadata to a user device.

[0014] In addition, according to another aspect of the present disclosure, an information processing method is provided, including: receiving motion data captured in an event from a first device; receiving event metadata related to the event from a second device; and sending the motion data and the event metadata to a user device.

[0015] In addition, according to another aspect of the present disclosure, a non-transitory computer-readable medium having a program embodied thereon is provided, wherein the program, when executed by a computer, causes the computer to perform an information processing method, the method comprising: receiving motion data captured in an event from a first device; receiving event metadata related to the event from a second device; and sending the motion data and event metadata to a user device.

[0016] In addition, according to another aspect of the present disclosure, an information processing system is provided, including: a circuit system configured to: receive motion data captured in an event from a first device; receive event metadata related to the event from a second device; initiate display of a virtual object on a display based on the motion data; and initiate display of information on the display based on the event metadata.

[0017] In addition, according to another aspect of the present disclosure, information displayed on a screen is provided, including: motion data captured in an event and event metadata related to the event displayed on the screen, wherein the motion data and event metadata are generated by receiving the motion data and event metadata from an information processing device, which receives the motion data from a first device and receives the event metadata from a second device. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] [ Figure 1 ] Figure 1 is a diagram showing the configuration of an information processing system 1A according to the first embodiment.

[0019] [ Figure 2 ] Figure 2 20A is a block diagram showing an example of the configuration of a server 20A according to the first embodiment.

[0020] [ Figure 3 ] Figure 3 1A is a sequence diagram showing the flow of operation processing in the information processing system 1A according to the first embodiment.

[0021] [ Figure 4 ] Figure 4is a diagram showing the configuration of an information processing system 1B according to the second embodiment.

[0022] [ Figure 5 ] Figure 5 20B is a block diagram showing an example of the configuration of a server 20B according to the second embodiment.

[0023] [ Figure 6 ] Figure 6 1B is a sequence diagram showing the flow of operation processing in the information processing system 1B according to the second embodiment.

[0024] [ Figure 7 ] Figure 7 is a view showing an example of a highlight metadata creation screen according to the second embodiment.

[0025] [ Figure 8 ] Figure 8 is a block diagram showing a hardware configuration example of the information processing apparatus 900 . DETAILED DESCRIPTION

[0026] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. Note that in this specification and the accompanying drawings, the same reference numerals are used to represent components having substantially the same functional configuration. Therefore, redundant descriptions are omitted.

[0027] Furthermore, the description is given in the following order.

[0028] 1. First Implementation

[0029] 1-1. System Configuration

[0030] 1-2. Server 20A Configuration

[0031] 1-3. Operation Processing

[0032] 2. Second Implementation

[0033] 2-1. System Configuration

[0034] 2-2. Server 20B Configuration

[0035] 2-3. Operation Processing

[0036] 2-4. Highlight metadata creation screen

[0037] 3. Modify the example

[0038] 4. Hardware Configuration

[0039] 5. Supplement

[0040] <1. First embodiment>

[0041] An information processing system according to a first embodiment of the present disclosure will now be described in detail with reference to the accompanying drawings.

[0042] <<1-1. Configuration of Information Processing System 1A>>

[0043] Figure 1 1A is a diagram showing a configuration of an information processing system 1A according to a first embodiment. Figure 1 As shown, the information processing system 1A includes a worker A device 12 (an example of a first device), a worker B device 14 (an example of a second device), a server 20A (an example of an information processing device), and a viewer terminal 30 (an example of a user terminal).

[0044] Worker A device 12 is a device used by Worker A. Worker A device 12 obtains motion data by tracking the movement of a target during an event. Examples of events include sports games. More specifically, embodiments of the present disclosure will be described assuming a soccer game. In this case, the tracked targets include the players and the ball. Furthermore, the tracking of the target's motion specifically involves tracking the position and height of the ball, obtained by analyzing captured images, as well as tracking the position and height of the players, as well as the position of each of the players' bones, obtained by analyzing captured images.

[0045] Worker A device 12 obtains captured images (hereinafter also referred to as game videos) from, for example, camera devices 11a, obtained by capturing images of a soccer match. Numerous camera devices 11a are arranged at the playing field, and worker A device 12 obtains game videos from various positions and angles. Worker A device 12 analyzes the game videos and obtains motion data of the players and the ball during the match.

[0046] An example of the data structure of motion data is shown in Table 1 below.

[0047] [Table 1]

[0048]

[0049]

[0050] As shown in Table 1 above, motion data includes ball data, player data and other data. In addition, among the data included in the motion data, ball position information, ball position height information from the ground, each player's three-dimensional skeleton position data, player position height information from the ground, etc. are time series data arranged in a time series with reference to the absolute time of the data. "Absolute time of data" is the elapsed time from a predetermined starting point. In the embodiment of the present disclosure, the elapsed time from January 1, 2000 using the Gregorian calendar is assumed to be an example of the reference time of such motion data. Hereinafter, this is referred to as "motion reference time."

[0051] The player position information included in the motion data is not limited to skeletal position data. For example, it can be joint position data indicating the position of the joints. The motion data can also include specific information of the game (which team's game takes place and at what time).

[0052] Note that each piece of information included in the motion data is an example. The embodiments of the present disclosure are not limited thereto. The operator can also manually correct missing or damaged parts of the motion data.

[0053] In addition, the worker A device 12 can acquire ambient sound data (such as audience cheers) from multiple microphones (microphones) 11b installed at the playing field. The worker A device 12 appropriately adjusts the ambient sound data (such as noise removal and volume adjustment) to generate sound data. The adjustment of the ambient sound data can be performed automatically by the worker A device 12 or manually by the operator.

[0054] For example, after a game, the worker A device 12 transmits the sound data and motion data of the game to the server 20A.

[0055] The worker B device 14 is a device used by worker B and is different from worker A. The worker B device 14 generates sports metadata including at least time series data related to the situation of a sports game such as soccer, etc. based on information input by the operator in the operation input device 13. Sports metadata is an example of event metadata.

[0056] The data structure of sports metadata includes information on the start date and time of a game, information on events that occurred in the game and the time at which the events occurred (time series data), and information related to players.

[0057] For example, while watching a video of a match in real time, an operator inputs an event that occurred during the match from the operation input device 13 (e.g., a PC). Examples of events that occurred during the match include scores (which team scored and how many points, which player scored the goal?), various warnings from the referee (in the case of football, yellow or red cards, fouls, etc.), free kicks, player substitutions, etc. The worker B device 14 generates sports metadata using, for example, the input time of the event as the event occurrence time. In addition, information about the event occurrence time can be input by the operator. Note that the time information used in the sports metadata according to the embodiment of the present disclosure is assumed to be the date and time in the Gregorian calendar.

[0058] In addition, information related to players is, for example, the name of the player, their jersey number, their role (goalkeeper, non-goalkeeper, referee), the team ID to which they belong, and other information. Information related to players can also be input by a worker.

[0059] The information included in the above-mentioned sports metadata is an example, and the embodiments of the present disclosure are not limited thereto.

[0060] The sports metadata generated in real time can be utilized in various ways, such as by being distributed to a predetermined distribution destination from the worker B device 14. In an embodiment of the present disclosure, after a game, the sports metadata of the game is transmitted from the worker B device 14 to the server 20A.

[0061] The server 20A acquires motion data and sound data from the worker A device 12 and sports metadata from the worker B device 14. The server 20A then transmits (distributes) the motion data, sound data, and sports metadata to the viewer terminal 30. The transmission format is not particularly limited, but from a data capacity perspective, for example, the worker A device 12 can cause the viewer terminal 30 to download the sports metadata before viewing and stream the motion data and sound data. Furthermore, the motion data and sound data can be converted by the server 20A into the distribution format described below.

[0062] The viewer terminal 30 is used by the viewer. The viewer terminal 30 is an example of a user terminal having a function of displaying a video of a virtual space. In particular, the viewer terminal 30 displays (renders) a 3DCG video in a virtual space by using the motion data (skeletal data of the players and the position of the ball, etc.) received from the server 20A and the 3D models of the players and the ball received in advance. The viewer terminal 30 can also switch the position of the virtual camera device according to the viewer operation. By virtualizing the game played in the real world, the viewer can switch and watch the game from various viewpoints at will, or operate their avatars arranged in the field to watch the game from a free viewpoint.

[0063] The 3D models of the players and the ball can be acquired in advance from the server 20A, etc. As a result, the viewer terminal 30 can perform reproduction with a load lower than that of receiving and reproducing a 3DCG video in a virtual space in real time.

[0064] In addition, the viewer terminal 30 can perform display control based on sports metadata (an example of event metadata) on a display screen on which a 3DCG of a player and a ball (an example of a virtual object) is displayed. Sports metadata, for example, includes score information, referee warning information, player replacement information, etc., and the viewer terminal 30 displays the score information, warning information, or player replacement information on the display screen according to the elapsed time from the start of the game. As a result, the viewer can recognize the state of the game while watching the 3DCG of the player and the ball. In addition, the sports metadata also includes the name of each player, and the viewer terminal 30 can perform display control so that the player's name follows near the 3DCG of the player. As a result, the viewer can easily identify the player.

[0065] As described above, in the information processing system 1A according to an embodiment of the present disclosure, a virtualization of a real-world game can be provided on the viewer terminal 30 based on motion data obtained from a game video and 3D models of players and a ball. Furthermore, when virtualizing the game, the information processing system 1A can also use sports metadata, which is data from a different acquisition source than the acquisition source of the motion data, to identify the circumstances of the game. The sports metadata may be data generated by a device different from the device that generated the motion data. Furthermore, it is assumed that the sports metadata is data of a rights holder different from the rights holder of the motion data.

[0066] <<1-2. Server 20A Configuration>>

[0067] Figure 2 2 is a block diagram showing an example of the configuration of the server 20A according to the first embodiment. Figure 2 As shown, the server 20A includes a communication unit 210 , a control unit 220A, and a storage unit 230 .

[0068] (Communication Unit 210)

[0069] The communication unit 210 includes a transmission unit that transmits data to an external device and a reception unit that receives data from the external device. The communication unit 210 according to an embodiment of the present disclosure can be communicatively connected to an external device or the Internet using, for example, a wired / wireless local area network (LAN), Wi-Fi (registered trademark), Bluetooth (registered trademark), a mobile communication network (Long Term Evolution (LTE)), a fourth generation mobile communication system (4G), or a fifth generation mobile communication system (5G).

[0070] The communication unit 210 receives the motion data and the sound data from the worker A device 12. In addition, the communication unit 210 receives the sports metadata from the worker B device 14. In addition, the communication unit 210 transmits each of the motion data, the sound data, and the sports metadata to the viewer terminal 30. Note that the motion data and the sound data may be data converted into the distribution format described below.

[0071] (Control Unit 220A)

[0072] The control unit 220A functions as an arithmetic processing unit and a control unit, and controls the overall operation of the server 20A according to various programs. For example, the control unit 220A is implemented by an electronic circuit such as a central processing unit (CPU) or a microprocessor. In addition, the control unit 220A may include a read-only memory (ROM) that stores programs to be used and operating parameters, etc., and a random access memory (RAM) that temporarily stores parameters that are changed as appropriate.

[0073] Furthermore, the control unit 220A also functions as a distribution format conversion unit 221 , a data storage control unit 222 , and a distribution control unit 223 .

[0074] The distribution format conversion unit 221 converts the motion data and sound data into a distribution format. The distribution format may be, for example, MPEG-DASH. Here, the reference time for the motion data (motion reference time) is assumed to be the elapsed time from January 1, 2000 using the Gregorian calendar as described above. However, when converted into a distribution format, the reference time used in the distribution format (hereinafter also referred to as the format reference time) is given anew. The reference time used for the distribution format is the elapsed time from the beginning of the motion data (time series data). In the case of MPEG-DASH, the format reference time is also referred to as MPEG-DASH time. Note that in the presence of sound data, it can be assumed that the start of the sound data is at the same timing as the start of the motion data. For example, in the case where the sound data and the motion data are data that start sound collection and imaging at the same timing in a competition venue, the start of the sound data and the motion data are at the same timing. In addition, the sound data and the motion data can be manually or automatically aligned at the beginning by sound matching at the worker A device 12, etc.

[0075] The data storage control unit 222 controls the storage unit 230 to store the sports metadata and the data converted into the distribution format. The sports metadata and the data converted into the distribution format are also referred to as content data. Note that the data storage control unit 222 may acquire the sports metadata of the game from the worker B device 14 based on the game specification information included in the sports data, and store the acquired sports metadata in the storage unit 230 in association with the data converted into the distribution format.

[0076] The distribution control unit 223 controls the distribution of sports metadata and data converted into a distribution format. For example, the distribution control unit 223 transmits the sports metadata in a download format to the viewer terminal 30 that has made a distribution request, or transmits the data (sports data and sound data) converted into a distribution format in a streaming format. The transmission of the sports metadata may be performed before the start of streaming.

[0077] (Storage Unit 230)

[0078] The storage unit 230 is implemented by a ROM that stores programs and operating parameters used for the processing of the control unit 220A, and a RAM that temporarily stores appropriately changed parameters, etc. For example, corresponding sports metadata and data converted into a distribution format are also stored in the storage unit 230 as content data.

[0079] The configuration of the server 20A is described in detail, but the configuration of the server 20A in the present disclosure is not limited to Figure 2 For example, the server 20A may be implemented using multiple devices.

[0080] <<1-3. Operation Process>>

[0081] Figure 3 1A is a sequence diagram showing the flow of operation processing in the information processing system 1A according to the first embodiment.

[0082] like Figure 3 As shown, the worker A device 12 transmits the motion data and sound data of the game to the server 20A (step S103).

[0083] Next, the worker B device 14 transmits the sports metadata of the game to the server 20A (step S106 ).

[0084] Next, the server 20A converts the motion data and the sound data into a distribution format (step S109).

[0085] Next, the server 20A stores the sports metadata of the game and the converted data in the storage unit 230 as content data (step S112 ).

[0086] Next, the server 20A controls the content data to be in a distributable state (step S115). The distributable state is a state in which the viewer terminal 30 is informed that the content data can be distributed and distribution can be started in response to a distribution request from the viewer terminal 30. The viewer operates the viewer terminal 30 to check the list of distributable content data and can make a distribution request for any content data (for example, a specific game).

[0087] Subsequently, the server 20A distributes the content data (step S121) in response to the distribution request (step S118) from the viewer terminal 30. Specifically, the server 20A causes the viewer terminal 30 to download sports metadata, and then distributes the converted data (motion data and sound data) to the viewer terminal 30 as a stream.

[0088] Then, the viewer terminal 30 implements display control based on the received data (step S124). In particular, the viewer terminal 30 arranges the 3D model of the athlete, the 3D model of the ball, etc. obtained in advance in a virtual space (for example, a space obtained by virtually reproducing the playing field in the real space), and implements the processing for generating the 3DCG video of the virtual space by moving the 3D model based on the motion data (the skeletal position data of each athlete, the position data of the ball, etc.). In addition, the viewer terminal 30 starts the reproduction of the sound data together with the start of the display control of the 3DCG video of the virtual space. In addition, in addition to the display control of the 3DCG video of the virtual space, the viewer terminal 30 also displays score information, etc. based on sports metadata.

[0089] Furthermore, the viewer terminal 30 may synchronize the sports metadata and the athletic data during the display control shown in step S124. As described above, it is assumed that the sports data and the sports metadata have different rights holders and use different data specifications, such as data structure and reference time. In this case, proper reproduction in the virtual space based on the sports data and the sports metadata is difficult, and time differences may occur. Therefore, the viewer terminal 30 can synchronize the sports data with the sports metadata to achieve proper reproduction in the virtual space.

[0090] For example, the viewer terminal 30 calculates the match start timing in the sports data based on the match start date and time information (date and time in the Gregorian calendar) included in the sports metadata, and displays score information, etc. on the display screen according to the elapsed time from the match start timing. More specifically, the viewer terminal 30 may convert the match start date and time information (in the Gregorian calendar) into the elapsed time from January 1, 2000, which is the reference time for the sports data (sports reference time), and determine the match start timing in the sports data based on the converted time.

[0091] The viewer terminal 30 then converts the match start timing of the sports reference time into the elapsed time from the beginning of the data, which is the reference time used in the distribution format (format reference time). Assume that the sports data is obtained from a time before the start of the game (e.g., a few minutes before kickoff). As described above, the viewer terminal 30 can synchronize sports metadata and sports data with reference to the match start timing, display a 3DCG of each player and the ball based on the game sports data, and display score information, etc., at the timing of scores during the game.

[0092] Note that the above-described synchronization method is an example; the embodiment is not limited thereto. In addition, synchronization processing is previously performed in the server 20A, and information on the game start timing (elapsed time from the start) in the sports data can be provided to the viewer terminal 30.

[0093] The first embodiment according to an aspect of the present disclosure has been described above.

[0094] <2. Second embodiment>

[0095] An information processing system according to a second embodiment of the present disclosure will now be described in detail with reference to the accompanying drawings. In the first embodiment described above, display control of the virtual space is implemented based on motion data from a full-time match, but the present disclosure is not limited thereto. For example, it is also conceivable to use information indicating the time (position on the timeline) of a highlight scene to implement highlight reproduction for executing display control of the virtual space based on the motion data corresponding to each highlight scene. A second embodiment for implementing such highlight reproduction will now be described.

[0096] <<2-1. System Configuration>>

[0097] Figure 4 1B is a diagram showing a configuration of an information processing system 1B according to the second embodiment. Figure 4 As shown, the information processing system 1B includes a worker A device 12 (an example of a first device), a worker B device 14 (an example of a second device), a server 20B (an example of an information processing device), a viewer terminal 30 (an example of a user terminal), and an operator terminal 40.

[0098] In addition to reference Figure 1In addition to the configuration of the information processing system 1A described above, the information processing system 1B further includes an operator terminal 40 used by a service operator to generate highlight metadata. The service operator is a worker different from the worker A and the worker B, and can be a worker who supplies the viewing application provided to the viewer terminal 30. In addition, the service operator can be a worker (team worker) of the team based at the stadium where the game is held. The team worker can use a dedicated application for providing the game to the team's fans to realize virtualization of the game played in the team worker's stadium (games played at the team's home stadium).

[0099] In the operator terminal 40, highlight metadata indicating the time information of the highlight scenes in the motion data is generated based on the time information of each highlight scene created when generating a highlight 2D video of the match based on the 2D video of the match (a video obtained by capturing the match with a camera device). The time information of the highlight scene is information indicating the entry point (IN point) and exit point (OUT point) of each highlight scene. The entry point and exit point are indicated by the elapsed time from the beginning of the 2D video. In addition, the time information of the highlight scene in the motion data is information indicating the scene reproduction start time and scene duration of each highlight scene in the motion data. The scene reproduction start time is indicated by the elapsed time from the beginning of the (format-converted) motion data.

[0100] An editor application (hereinafter also referred to as editor AP) for creating highlight content is downloaded to the operator terminal 40. The service operator can generate highlight metadata by operating the highlight metadata creation screen displayed by the editor AP. The operator terminal 40 transmits the highlight metadata to the server 20B. Note that the editor AP can be operated in the server 20B and the highlight metadata creation screen can be provided to the operator terminal 40 as a web browser.

[0101] The server 20B transmits the highlight metadata and content data received from the operator terminal 40 to the viewer terminal 30. Therefore, when controlling the display of the virtual space based on the sports data, the viewer terminal 30 can use the highlight metadata to implement display control for only the highlight scenes. The viewer can sequentially view only the exciting scenes (e.g., scoring scenes) during the game in the virtual space.

[0102] <<2-2. Server 20B Configuration>>

[0103] Figure 5 2 is a block diagram showing an example of the configuration of the server 20B according to the second embodiment. Figure 5 As shown, the server 20B includes a communication unit 210, a control unit 220B and a storage unit 230. The communication unit 210 and the storage unit 230 are as described above. Figure 2 As stated.

[0104] In addition, the control unit 220B also functions as a distribution format conversion unit 221, a data storage control unit 222, a distribution control unit 223, and a highlight metadata acquisition unit 224. The functions of the distribution format conversion unit 221, the data storage control unit 222, and the distribution control unit 223 are as described with reference to FIG. Figure 2 As stated.

[0105] The highlight metadata acquisition unit 224 controls the acquisition of highlight metadata. The highlight metadata acquisition unit 224 transmits data for generating highlight metadata from the communication unit 210 to the operator terminal 40, and receives highlight metadata from the operator terminal 40. The highlight metadata acquisition unit 224 transmits, for example, sports metadata and (format-converted) motion data stored in the storage unit 230 to the operator terminal 40 as data for generating highlight metadata. Furthermore, the highlight metadata acquisition unit 224 may transmit, for example, motion data, (format-converted) sound data, and sports metadata to the operator terminal 40 as data for generating highlight metadata.

[0106] The configuration of the server 20B is described in detail, but the configuration of the server 20B in the present disclosure is not limited to Figure 5 For example, the server 20B may be implemented using multiple devices.

[0107] <<2-3. Operation Process>>

[0108] Figure 6 1B is a sequence diagram showing the flow of operation processing in the information processing system 1B according to the second embodiment.

[0109] because Figure 6 The processing from step S203 to step S212 shown is the same as Figure 3 Steps S103 to S112 in are similar, so their description will be omitted.

[0110] Next, the operator terminal 40 generates a 2D video of the game's highlights based on the 2D video of the game (step S215). For example, the 2D video of the highlights can be generated by automatically or manually extracting scoring scenes from the game video. At this point, the time information (entry point information and exit point information) of the extracted scenes, i.e., the highlight scenes, is also generated.

[0111] Next, the server 20B transmits the content data (at least the motion data of the game and the sports metadata) to the operator terminal 40 (step S218 ).

[0112] Next, the operator terminal 40 displays the highlight metadata creation screen and receives the operation input of the service operator (step S221). On the highlight metadata creation screen, a display screen of the highlight 2D video (a captured image of a game played in real space), a display screen of a player 3DCG based on motion data (a virtual video), and an input screen for inputting metadata of each highlight scene are displayed. On the screen for inputting metadata of each highlight scene, at least the time information of each highlight scene (scene reproduction start time and scene duration) is input. This will be referred to later. Figure 7 Describes the details of the highlight metadata creation screen.

[0113] Next, the operator terminal 40 transmits the bright spot metadata to the server 20B (step S224 ).

[0114] Next, the server 20B controls the content data (including the highlight metadata) to be placed in a distributable state (step S227).

[0115] Next, the server 20B distributes the content data (including the highlight metadata) (step S233) in response to the distribution request (step S230) from the viewer terminal 30. Specifically, the server 20B sends the sports metadata, highlight metadata, and format-converted data (motion data and sound data) to the viewer terminal 30.

[0116] Then, the viewer terminal 30 implements display control based on the received data (step S236). Specifically, the viewer terminal 30 implements the following processing: 3D models of athletes and balls, etc., which have been previously obtained, are arranged in a virtual space (e.g., a space obtained by virtually reproducing a playing field in a real space), and a 3DCG video of the virtual space is generated by moving the 3D models based on the motion data (3D skeletal position data of each athlete, position data of the ball, etc.). At this time, the viewer terminal 30 performs display control of the virtual space based on the motion data based on the highlight metadata (including information on the scene reproduction start time and scene duration represented by the motion reference time).

[0117] Since the motion data is data that includes at least the time length of the entire game, the viewer terminal 30 can reproduce all highlight scenes by repeatedly displaying (reproducing) the virtual space based on the motion data from the scene reproduction start time included in the highlight metadata in the motion data, reproducing the scene until the end point of the scene (the timing of the scene duration from the scene reproduction start time), and then finding the reproduction point of the next scene (the scene reproduction start time) and reproducing the next scene at that reproduction point. Note that the motion reference time (an expression based on the elapsed time from the beginning of the motion data) is used for the reference time of the highlight metadata.

[0118] The configuration of the operation process according to the second embodiment has been described in detail above. Note that Figure 6 The illustrated operations are merely examples. The present disclosure is not limited thereto. For example, server 30B may sequentially stream motion data of highlight scenes within the motion data based on the highlight metadata. In this case, viewer terminal 30 may generate a 3DCG video of the highlight scene by moving pre-received 3D models of players and balls, etc., according to the received motion data.

[0119] <<2-4. Highlight metadata creation screen>>

[0120] Figure 7 1 is a diagram showing an example of a highlight metadata creation screen according to the second embodiment. Figure 7 As shown, on the highlight metadata creation screen 420, a display screen 421 for displaying the highlight 2D video, a display screen 422 for displaying the athlete 3DCG rendered based on motion data, etc., and an input screen for inputting metadata for each highlight scene (a first scene input screen 424 and a second scene input screen 425).

[0121] The operator terminal 40 displays a pre-generated 2D highlight video on the display screen 421. The operator (service operator) inputs each type of highlight metadata from the input screen while checking the display screen 422 on which a 3D graphics image of an athlete rendered based on motion data is displayed, based on each highlight scene reproduced on the display screen 421.

[0122] First, the operator of the operator terminal 40 enters the date of the target game in the date input field 431 displayed at the top of the highlight metadata creation screen 420. Since the sports metadata received by the operator terminal 40 from the server 20A includes the date of the game and information about the competing teams, the operator terminal 40 displays the information about the corresponding competing teams next to the date input field 431.

[0123] When the operator visually identifies the highlight 2D video displayed on the display screen 421 and selects the setting button 432 displayed below the search bar at the reproduction position of the scene where the highlight metadata of the motion data is desired to be created, the input screen of the highlight metadata (for example, the input screen 425 of the second scene) is displayed.

[0124] On the input screen, for example, you can enter the scene type (scoring scene, free kick scene, etc.), the target team (home or away), the jersey number of the player who took the last shot, the scene playback start time, the scene duration, the scene description sentence, and a short version of the scene description. The items described here are examples; this embodiment is not limited to this.

[0125] The scene type is information used when displaying an icon indicating the scene in the chapter list displayed during viewing. For example, in the case of a scoring scene, an icon of a goal post is displayed. In addition, a short version of the scene description can also be displayed next to the icon in the chapter list.

[0126] The target team is information indicating whether the opposing team in each scene is the home team or the away team. The home team is the team based at the playing field, and in embodiments of the present disclosure, a competition between the home and away teams is assumed. Furthermore, which team is the home team and which is the away team in the competition may be included in the motion data or sports metadata. For example, when viewing each highlight scene, an overlay is displayed indicating the logo of the target team for that scene, and the target team is used at that time. The overlay may also display the number of seconds of the scene and a description of the scene.

[0127] When setting the virtual camera position in the viewer terminal 30, the jersey number of the player who took the final shot is used. The viewer can arbitrarily select the virtual camera position in the viewer terminal 30 from the following options: Basic: Positioned on the extended line connecting the goal and the ball; Bird's-Eye View: Positioned above and behind the ball with a wide viewing angle; Goalkeeper's View: Positioned at the goalkeeper's viewpoint; Shooter's View: Positioned at the viewpoint of the player who took the shot at the end of each scene.

[0128] The scene playback start time and scene duration are time information indicating the highlight scene position in the motion data time axis. Since it is format-converted motion data, the scene playback start time is indicated by the format reference time, that is, the elapsed time from the start of the data.

[0129] In addition, the operator terminal 40 can calculate the scene reproduction start time and scene duration of the corresponding highlight scene in the motion data based on the scene time (entry point information, exit point information) of the highlight 2D video automatically set on the input screen, and allow the operator to manually adjust it. The highlight 2D video (including the video of each highlight scene) may include videos of non-athletes such as coaches or spectators. However, since the motion data is obtained by tracking the movements of athletes on the field, there may be no scene corresponding to the highlight 2D video in the motion data. Therefore, when the operator adjusts the scene reproduction start time and scene duration, a scene suitable for generating a 3DCG video based on the motion data can be obtained.

[0130] The operator terminal 40 will be described as calculating the scene reproduction start time and scene duration for the corresponding highlight scene in the motion data based on the highlight 2D video. The scene time (entry point information, exit point information) of the highlight 2D video is represented by the elapsed time from the start time of the game in the 2D video that is the source of the highlight 2D video. In contrast, the motion reference time used in the motion data is the elapsed time from January 1, 2000 in the Gregorian calendar. In addition, through format conversion, the elapsed time from the beginning of the motion data is set to 0ms. In this way, since the "highlight scene entry point information and exit point information" and the "motion data" reference time are different, the operator terminal 40 can automatically set the scene reproduction start time and scene duration of the highlight scene corresponding to the motion data by synchronizing the time.

[0131] For example, the operator terminal 40 first converts the match start time information (date and time in the Gregorian calendar) included in the sports metadata into a motion reference time (elapsed time from January 1, 2000). Furthermore, the operator terminal 40 obtains the motion reference time (elapsed time from January 1, 2000) corresponding to the time at the beginning of the format-converted motion data (0ms in the format reference time). By comparing these, the operator terminal 40 calculates the match start time (i.e., the elapsed time from the beginning of the motion data (0ms)) from the format reference time in the motion data. Assume that the motion data is acquired before the start of the game (e.g., at kickoff), and the start of the motion data is not necessarily the match start time. Consequently, based on the position of the match start time in the motion data, the operator terminal 40 can calculate where on the timeline of the motion data the "entry point information and exit point information" created by the operator when generating the highlight 2D video, indicated by the elapsed time from the start of the game, are located. The synchronization method described here is an example; implementation is not limited thereto. Furthermore, the time synchronization process can be performed by an editor AP operated on the operator terminal 40. Furthermore, the time synchronization process may be performed by the server 20B.

[0132] The second embodiment according to aspects of the present disclosure has been described above.

[0133] <3. Modification Example>

[0134] <<3-1. First Modification Example>>

[0135] When creating highlight metadata, the operator terminal 40 can automatically determine which goal is being attacked (offensive situation) based on which goal the ball is near in the last frame of each scene, and can automatically input that goal into the highlight metadata. When viewing on the viewer terminal 30, a virtual camera can be placed on an extension line connecting the center of the attacked goal and the position of the ball, displaying a scene that normally captures the attacked goal from a viewing angle. Note that the automatic input of highlight metadata based on the analysis of each scene can be implemented by the server 20B.

[0136] <<3-2. Second Modification Example>>

[0137] The sports metadata of a match includes information about events such as who scored, when the score was scored, who received a yellow card or a red card, etc. However, since it is assumed that workers mainly manually enter the sports metadata while watching the match in real-time video, the time information of the sports metadata may deviate by up to about one minute compared to the time it occurred in the actual match.

[0138] Therefore, server 20 (server 20A or server 20B) can analyze the motion data corresponding to the occurrence of the event recorded in the sports metadata and the time of the event one minute before and after, and correct it to a more accurate time. For example, in the case of scoring information, server 20 checks the position information of the ball data included in the motion data, and sets the time when the goal line was actually crossed to the time of the goal. In addition, in the case of a yellow card or a red card, server 20 sets the time when the referee's hand motion indicates the time of the player to the yellow card or the red card based on the motion data of the referee included in the motion data. Server 20 rewrites the sports metadata at the time of the analysis result and stores it in storage unit 230.

[0139] <<3-3. Third Modification Example>>

[0140] In the above embodiment, the sports metadata describing the game is generated by the worker B device 14, but the present disclosure is not limited to this. The server 20 (server 20A or server 20B) can also estimate the occurrence of an event based on the motion data obtained from the worker A device 12 (an example of an external device) and automatically generate sports metadata (an example of event metadata). Since the motion data includes time-series data of the positions of the players and the ball, the server 20 can determine (estimate) the occurrence of a specific event during the game (for example, determining whether a goal was scored by comparing the position of the ball with the goal line) and generate sports metadata.

[0141] <<3-4. Fourth Modification Example>>

[0142] Furthermore, in the above embodiment, it is assumed that the motion data and sports metadata are distributed after the game, but the present disclosure is not limited thereto. The information processing system may distribute the motion data and sports metadata acquired in real time during the game in real time.

[0143] <<3-5. Fifth Modification Example>>

[0144] In the above embodiment, the generation of highlight metadata based on input from the operator at the operator terminal 40 is described, but the present disclosure is not limited thereto. Server 20B can generate highlight metadata (specifically, scene start time and scene duration) based on the highlight 2D video. For example, server 20B calculates the scene reproduction start time and scene duration of the corresponding highlight scene in the motion data based on the scene time (entry point information and exit point information) of the highlight 2D video. The synchronization of the scene time (entry point information and exit point information) of the highlight 2D video and the reference time (format reference time) of the motion data is as described above.

[0145] <4. Hardware Configuration>

[0146] Next, we will refer to Figure 8 A hardware configuration example of the information processing apparatus 900 as an example of the server 20 ( 20A and 20B) according to an embodiment of the present disclosure is described. Figure 8 is a block diagram showing an example of the hardware configuration of the information processing apparatus 900. Note that the server 20 (20A and 20B) does not necessarily have Figure 8 All hardware configurations shown, and Figure 8 A portion of the hardware configuration shown does not necessarily exist in the server 20 (20A and 20B). Figure 8 The illustrated hardware configuration example can also be applied to the viewer terminal 30 or the operator terminal 40 .

[0147] like Figure 8 As shown, the information processing device 900 includes a central processing unit (CPU) 901, a read-only memory (ROM) 902, and a random access memory (RAM) 903. In addition, the information processing device 900 may include a host bus 907, a bridge 909, an external bus 911, an interface 913, an input device 915, an output device 917, a storage device 919, a drive 921, a connection port 923, and a communication device 925. Instead of the CPU 901 or in combination with the CPU 901, the information processing device 900 may have a processing circuit called a digital signal processor (DSP) or an application-specific integrated circuit (ASIC).

[0148] The CPU 901 functions as an arithmetic processing device and a control device, and controls the overall operation of the information processing device 900 or a portion of its operation according to various programs recorded in the ROM 902, the RAM 903, the storage device 919, or the removable recording medium 927. The ROM 902 stores programs and calculation parameters used by the CPU 901. The RAM 903 temporarily stores programs used when executed by the CPU 901 and parameters that change appropriately during execution. The CPU 901, the ROM 902, and the RAM 903 are interconnected by a host bus 907 configured from an internal bus such as a CPU bus. In addition, the host bus 907 is connected to an external bus 911 such as a peripheral component interconnect / interface (PCI) bus via a bridge 909.

[0149] The input device 915 is, for example, a device such as a button operated by a user. The input device 915 may include a mouse, a keyboard, a touch panel, a switch, a joystick, etc. In addition, the input device 915 may also include a microphone for detecting the user's voice. The input device 915 may be, for example, a remote control device using infrared light or other radio waves, or may be an external connection device 929 such as a mobile phone corresponding to the operation of the information processing device 900. The input device 915 includes an input control circuit that generates an input signal based on information input by the user and outputs the input signal to the CPU 901. By operating the input device 915, the user inputs various types of data to the information processing device 900 or gives instructions for performing processing operations. In addition, the input device 915 may include an imaging device. The imaging device captures the movement of, for example, the user's hand or the user's fingers. In this case, the pointing position can be determined based on the movement of the hand and the direction of the finger.

[0150] The output device 917 is composed of a device that can visually or auditorily notify the user of the acquired information. The output device 917 can be, for example, a display device such as a liquid crystal display (LCD) or an organic electroluminescent (EL) display, or an audio output device such as a speaker or headphones. In addition, the output device 917 can include a plasma display panel (PDP), a projector, a hologram, a printer, etc. The output device 917 outputs the results obtained by the processing of the information processing device 900 as text or a video such as an image, or outputs the results as audio in the form of voice or sound. In addition, the output device 917 can include a light, etc., to brighten the surrounding environment.

[0151] The storage device 919 is a data storage device configured as an example of a storage unit of the information processing device 900. The storage device 919 includes, for example, a magnetic storage device such as a hard disk drive (HDD), a semiconductor storage device, an optical storage device, a magneto-optical storage device, and the like. The storage device 919 stores programs executed by the CPU 901 and various data, various data acquired from the outside, and the like.

[0152] The drive 921 is a reader / writer for the removable recording medium 927 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, and is built in the information processing apparatus 900 or externally attached to the information processing apparatus 900. The drive 921 reads information recorded in the mounted removable recording medium 927 and outputs it to the RAM 903. In addition, the drive 921 writes records in the mounted removable recording medium 927.

[0153] The connection port 923 is a port for directly connecting a device to the information processing apparatus 900. The connection port 923 may be, for example, a universal serial bus (USB) port, an IEEE 1394 port, a small computer system interface (SCSI) port, or the like. Furthermore, the connection port 923 may be an RS-232C port, an optical audio terminal, a high-definition multimedia interface (HDMI (registered trademark)) port, or the like. By connecting an external connection device 929 to the connection port 923, various data can be exchanged between the information processing apparatus 900 and the external connection device 929.

[0154] The communication device 925 is, for example, a communication interface including a communication device for connecting to a network 931 or the like. The communication device 925 may be, for example, a communication card for a wired or wireless local area network (LAN), Bluetooth (registered trademark), or wireless USB (WUSB). In addition, the communication device 925 may be a router for optical communication, a router for asymmetric digital subscriber line (ADSL), or a modem for various communications. The communication device 925 transmits signals to and receives signals from the Internet and other communication devices using a predetermined protocol such as TCP / IP. In addition, the network 931 connected to the communication device 925 is a network connected by wire or wirelessly, and may be, for example, the Internet, a home LAN, infrared communication, radio wave communication, or satellite communication.

[0155] <5. Supplement>

[0156] The embodiments of the present disclosure have been described in detail above with reference to the accompanying drawings, but the present technology is not limited to such examples. Obviously, a person with ordinary knowledge in the technical field of the present disclosure can design various changes or modifications within the scope of the technical concept described in the claims, and it will be naturally understood that these examples also belong to the technical scope of the present disclosure.

[0157] For example, one or more computer programs may be created to cause hardware such as a CPU, ROM, and RAM built into the server 20A, server 20B, viewer terminal 30, or operator terminal 40 to exhibit the functions of the server 20A, server 20B, viewer terminal 30, or operator terminal 40. Furthermore, a computer-readable storage medium storing one or more computer programs is also provided.

[0158] In addition, the effects described in this specification are not limitative. That is, in addition to or instead of the above effects, the technology according to aspects of the present disclosure may exhibit other effects that are obvious to those skilled in the art based on the description of this specification.

[0159] Note that the present technology can also have the following configurations.

[0160] (1) An information processing device comprising:

[0161] A circuit system configured to:

[0162] receiving, from the first device, motion data captured during the event;

[0163] receiving event metadata associated with the event from the second device; and

[0164] The motion data and event metadata are sent to the user device.

[0165] (2) The information processing device according to (1),

[0166] Wherein, the circuit system is further configured to:

[0167] sending the athletic data to the user device in a streaming format; and

[0168] The event metadata is sent to the user device in a downloadable format.

[0169] (3) The information processing device according to any one of (1) or (2),

[0170] Therein, motion data is streamed by a user device and event metadata is downloaded by the user device.

[0171] (4) The information processing device according to any one of (1) to (3),

[0172] Therein, a virtual avatar or object generated based on the motion data is displayed by a user device together with event metadata.

[0173] (5) The information processing device according to any one of (1) to (4),

[0174] Wherein the motion data includes data of participants or objects in the event, and

[0175] The user device displays a video of a virtual space including a virtual avatar of the object or the participant by using the motion data and the 3D model of the participant or the object received from the information processing device prior to the motion data.

[0176] (6) The information processing device according to any one of (1) to (5),

[0177] Among them, motion data is synchronized with event metadata.

[0178] (7) The information processing device according to any one of (1) to (6),

[0179] The motion data and event metadata reference a reference time for the motion data for synchronization.

[0180] (8) The information processing device according to any one of (1) to (7),

[0181] The synchronization includes:

[0182] calculating a start timing of an event in the motion data based on information about a start date and time of the event included in the event metadata; and

[0183] displaying event metadata based on an elapsed time from a start timing in the motion data;

[0184] converting start date and time information of the start timing into an elapsed time from a reference time for the motion data;

[0185] determining a start timing in the motion data based on the transition time; and

[0186] The start timing of the motion data is converted into the elapsed time from the start timing in the motion data.

[0187] (9) The information processing device according to any one of (1) to (8),

[0188] Here, the synchronization is pre-implemented in the information processing device and sent to the user device.

[0189] (10) The information processing device according to any one of (1) to (9),

[0190] The event metadata includes information about the start date and time of the event, information about the occurrence of events in the event and the time when the event occurred, or information about participants in the event.

[0191] (11) The information processing device according to any one of (1) to (10),

[0192] Among them, the motion data includes data of objects in the event, data of participants in the event, data of the position of the object, data of the height of the position of the object from the ground, data of the three-dimensional skeletal position of the participant, or data of the height of the three-dimensional skeletal position of the participant from the ground.

[0193] (12) The information processing device according to any one of (1) to (11),

[0194] Wherein, the circuit system is further configured to:

[0195] Sound data captured in the event is received from the first device.

[0196] (13) The information processing device according to any one of (1) to (12),

[0197] The circuit is further configured as follows:

[0198] Highlight metadata indicating time information of the highlight scene, which is generated based on the time information of the highlight scene in the motion data, is received.

[0199] (14) The information processing device according to any one of (1) to (13),

[0200] The time information of the highlight scene includes information indicating the scene reproduction start time and scene duration of the highlight scene in the motion data.

[0201] (15) The information processing device according to any one of (1) to (14),

[0202] Here, the scene reproduction start time is indicated by the elapsed time from the start timing of the motion data.

[0203] (16) The information processing device according to any one of (1) to (15),

[0204] Wherein, the circuit system is further configured to:

[0205] The highlight metadata is sent to the user device.

[0206] (17) The information processing device according to any one of (1) to (16),

[0207] Wherein, the circuit system is further configured to:

[0208] Sending motion data and event metadata to an operator terminal; and

[0209] The highlight metadata generated based on the transmitted motion data and event metadata is received from the operator terminal.

[0210] (18) The information processing device according to any one of (1) to (17),

[0211] The virtual avatar or object generated based on the highlight scene is displayed by the user device together with the highlight metadata.

[0212] (19) The information processing device according to any one of (1) to (18),

[0213] The highlight metadata includes the scene reproduction start time and scene duration of the highlight scene.

[0214] (20) The information processing device according to any one of (1) to (19), wherein

[0215] The start date and time information of the start timing of the event is converted into the elapsed time from the reference time for the motion data, and

[0216] The time at the start of the motion data is compared with the elapsed time to calculate the elapsed time from the start of the motion data.

[0217] (21) An information processing method comprising:

[0218] receiving, from the first device, motion data captured during the event;

[0219] receiving event metadata associated with the event from the second device; and

[0220] The motion data and event metadata are sent to the user device.

[0221] (22) A non-transitory computer-readable medium having a program embodied thereon, the program causing a computer to perform an information processing method when executed by the computer, the method comprising:

[0222] receiving, from the first device, motion data captured during the event;

[0223] receiving event metadata associated with the event from the second device; and

[0224] The motion data and event metadata are sent to the user device.

[0225] (23) An information processing system comprising:

[0226] A circuit system configured to:

[0227] receiving, from the first device, motion data captured during the event;

[0228] receiving event metadata associated with the event from the second device;

[0229] initiating display of a virtual object on a display based on the motion data; and

[0230] Display of information on a display is initiated based on the event metadata.

[0231] (24) The information processing system according to (23),

[0232] The motion data and event metadata are received via the server.

[0233] (25) The information processing system according to any one of (23) or (24),

[0234] Among them, motion data is obtained by tracking objects in the event.

[0235] (26) The information processing system according to any one of (23) to (25),

[0236] Among them, the subject includes humans.

[0237] (27) The information processing system according to any one of (23) to (26),

[0238] Wherein, the circuit system is further configured to:

[0239] Synchronize motion data and event metadata.

[0240] (28) Information displayed on a screen, comprising:

[0241] Event metadata associated with the event displayed on screen and motion data captured during the event,

[0242] The motion data and the event metadata are generated by receiving the motion data and the event metadata from an information processing device, the information processing device receiving the motion data from a first device and receiving the event metadata from a second device.

[0243] (B1) An information processing device comprising:

[0244] A control unit that implements the following processing:

[0245] obtaining motion data obtained by tracking the motion of a target during an event and event metadata including at least time series data related to circumstances of the event; and

[0246] The motion data and the event metadata are transmitted to the user terminal, and the user terminal performs: control of displaying a virtual object of a target in a virtual space based on the motion data, and display control of a display screen on which the virtual object is displayed based on the event metadata,

[0247] Motion data and event metadata are generated in different devices.

[0248] (B2) The information processing device according to (B1) above, wherein the event is a competition.

[0249] (B3) The information processing device according to (B2) above, wherein

[0250] The control unit:

[0251] obtaining motion data from the first device; and

[0252] Event metadata is obtained from a second device that is different from the first device.

[0253] (B4) The information processing device according to (B2) above, wherein

[0254] The control unit:

[0255] Obtaining motion data from an external device; and

[0256] Generate event metadata based on the motion data.

[0257] (B5) The information processing device according to any one of (B2) to (B4) above, wherein

[0258] The event metadata includes information on the match start date and time and time series data indicating what happened in the match and when.

[0259] (B6) The information processing device according to (B5) above, wherein

[0260] The control unit performs a process of adjusting the deviation in the occurrence time based on the motion data.

[0261] (B7) The information processing device according to (B5) or (B6) above, wherein

[0262] Incidents include scoring and various warnings from the referee.

[0263] (B8) The information processing device according to any one of (B5) to (B7) above, wherein

[0264] An occurrence of an event is estimated based on the motion data.

[0265] (B9) The information processing device according to any one of (B2) to (B8) above, wherein

[0266] Event metadata also includes information related to the athletes.

[0267] (B10) The information processing device according to any one of (B2) to (B9) above, wherein

[0268] The motion data includes time series data of the skeletal position of each player and time series data of the position of the ball.

[0269] (B11) The information processing device according to any one of (B2) to (B10) above, wherein

[0270] The control unit implements a process of converting the motion data into a distribution format and transmitting the converted motion data together with event metadata to the user terminal.

[0271] (B12) The information processing device according to (B11) above, wherein

[0272] The reference time used in the distribution format is applied to the motion data converted into the distribution format.

[0273] (B13) The information processing device according to any one of (B2) to (B12) above, wherein

[0274] The control unit implements a process of further obtaining highlight metadata that is information related to highlight scenes of the game and transmitting the highlight metadata to the user terminal together with the sports data and the event metadata.

[0275] (B14) The information processing device according to (B13) above, wherein

[0276] The highlight metadata includes information indicating the start time and duration of each highlight scene of the motion data.

[0277] (B15) The information processing device according to (B13) or (B14) above, wherein

[0278] The reference time of the highlight metadata corresponds to the reference time used in the distribution format into which the motion data is converted.

[0279] (B16) The information processing device according to any one of (B13) to (B15) above, wherein

[0280] The highlight metadata is generated in a device different from the device that generates the motion data and the device that generates the event metadata.

[0281] (B17) The information processing device according to (B16) above, wherein

[0282] The control unit transmits data obtained by converting the motion data into a distribution format and event metadata to a device that generates highlight metadata, and receives highlight metadata from the device.

[0283] (B18) The information processing device according to any one of (B13) to (B17) above, wherein

[0284] The highlight metadata includes information indicating the offensiveness of each highlight scene as information used when determining the arrangement of virtual cameras in a virtual space at a user terminal.

[0285] (B19) An information processing method comprising:

[0286] obtaining, by a processor, motion data obtained by tracking motion of a target during an event and event metadata including at least time series data related to circumstances of the event; and

[0287] The processor performs processing to transmit the motion data and the event metadata to the user terminal, wherein the user terminal performs control of displaying a virtual object of a target in a virtual space based on the motion data and control of display of a display screen displaying the virtual object based on the event metadata.

[0288] Motion data and event metadata are generated in different devices.

[0289] (B20) A program for causing a computer to function as a control unit to implement the following processing:

[0290] obtaining motion data obtained by tracking motion of a target during an event and event metadata including at least time series data related to circumstances of the event; and

[0291] The motion data and the event metadata are transmitted to a user terminal, wherein the user terminal implements control of displaying a target virtual object in a virtual space based on the motion data and display control based on the event metadata in a display screen on which the virtual object is displayed,

[0292] Motion data and event metadata are generated in different devices.

[0293] Those skilled in the art should understand that various modifications, combinations, sub-combinations and alterations may occur depending on design requirements and other factors insofar as these modifications, combinations, sub-combinations and alterations are within the scope of the appended claims or the equivalents thereof.

[0294] Reference Signs List

[0295] 1A, 1B Information Processing Systems

[0296] 11a Camera

[0297] 11b Microphone

[0298] 12 Worker A device

[0299] 13 Operation input device

[0300] 14 Worker B device

[0301] Servers 20A and 20B

[0302] 210 Communication Unit

[0303] 220A, 220B control unit

[0304] 221 Distribution Format Conversion Unit

[0305] 222 Data storage control unit

[0306] 223 Distribution Control Unit

[0307] 224 Highlight metadata acquisition unit

[0308] 30 Viewer Terminal

[0309] 40 Operator Terminal

Claims

1. An information processing device, comprising: A circuit system configured to: receiving, from the first device, motion data captured during the event; receiving event metadata related to the event from a second device; and The motion data and the event metadata are sent to a user device.

2. The information processing device according to claim 1, in, The circuit system is further configured to: sending the athletic data to the user device in a streaming format; and The event metadata is sent to the user device in a download format.

3. The information processing device according to claim 1, in, The motion data is streamed by the user device, and the event metadata is downloaded by the user device.

4. The information processing device according to claim 1, in, A virtual avatar or object generated based on the motion data is displayed by the user device along with the event metadata.

5. The information processing device according to claim 1, in, The motion data includes data of participants or objects in the event, and The user device displays a video of a virtual space including a virtual avatar of the object or the participant by using the motion data and a 3D model of the participant or the object received from the information processing device before the motion data. The information processing device according to claim 1 , in, The motion data is synchronized with the event metadata.

7. The information processing device according to claim 6, in, The motion data and the event metadata are synchronized with reference to a reference time for the motion data.

8. The information processing device according to claim 7, in, The synchronization includes: calculating a start timing of the event in the motion data based on information about a start date and time of the event included in the event metadata; displaying the event metadata based on an elapsed time from the start timing in the motion data; converting the start date and time information of the start timing into an elapsed time from the reference time for the motion data; determining the start timing in the motion data based on a transition time; and The start timing of the motion data is converted into an elapsed time from the start timing in the motion data.

9. The information processing device according to claim 7, in, The synchronization is preliminarily implemented in the information processing device and is transmitted to the user device.

10. The information processing device according to claim 1, in, The event metadata includes information about a start date and time of the event, information about occurrences in the event and times of the occurrences, or information about participants in the event.

11. The information processing device according to claim 1, in, The motion data includes data of the object in the event, data of the participants in the event, data of the position of the object, data of the height of the position of the object from the ground, data of the three-dimensional skeletal position of the participant, or data of the height of the three-dimensional skeletal position of the participant from the ground.

12. The information processing device according to claim 1, in, The circuit system is further configured to: Sound data captured in the event is received from the first device.

13. The information processing device according to claim 1, in, The circuit system is further configured to: Highlight metadata indicating time information of the highlight scene, which is generated based on time information of the highlight scene in the motion data, is received.

14. The information processing device according to claim 13, in, The time information of the highlight scene includes information indicating a scene reproduction start time and a scene duration of the highlight scene in the motion data.

15. The information processing device according to claim 14, in, The scene reproduction start time is indicated by an elapsed time from the start timing of the motion data.

16. The information processing device according to claim 13, in, The circuit system is further configured to: The highlight metadata is sent to the user device.

17. The information processing device according to claim 13, in, The circuit system is further configured to: sending the motion data and the event metadata to an operator terminal; and The highlight metadata generated based on the transmitted motion data and event metadata is received from the operator terminal.

18. The information processing device according to claim 13, in, A virtual avatar or object generated based on the highlight scene is displayed by the user device together with the highlight metadata.

19. The information processing device according to claim 13, in, The highlight metadata includes a scene reproduction start time and a scene duration of the highlight scene.

20. The information processing apparatus according to claim 13, wherein: The start date and time information of the start timing of the event is converted into an elapsed time from a reference time for the motion data, and The time at the beginning of the motion data is compared with the elapsed time to calculate the elapsed time from the start of the motion data.

21. An information processing method, comprising: receiving, from the first device, motion data captured during the event; receiving event metadata associated with the event from a second device; as well as The motion data and the event metadata are sent to a user device.

22. A non-transitory computer-readable medium having a program embodied thereon, the program, when executed by a computer, causing the computer to perform an information processing method, the method comprising: receiving, from the first device, motion data captured during the event; receiving event metadata associated with the event from a second device; as well as The motion data and the event metadata are sent to a user device.

23. An information processing system comprising: A circuit system configured to: receiving, from the first device, motion data captured during the event; receiving event metadata associated with the event from a second device; initiating display of a virtual object on a display based on the motion data; and Display of information on the display is initiated based on the event metadata.

24. The information processing system according to claim 23, in, The motion data and the event metadata are received via a server.

25. The information processing system according to claim 23, in, The motion data is obtained by tracking the object in the event.

26. The information processing system according to claim 25, in, The subject includes a human.

27. The information processing system according to claim 23, in, The circuit system is further configured to: The motion data and the event metadata are synchronized.

28. Information displayed on a screen, comprising: event metadata associated with the event and motion data captured during the event displayed on the screen, The motion data and the event metadata are generated by receiving the motion data and the event metadata from an information processing device, wherein the information processing device receives the motion data from a first device and receives the event metadata from a second device.

Images