Information processing system, information processing device, and information processing method

Abstract

An information processing device 10 includes: an acquisition unit 112 that acquires first additional data associating a timestamp, which corresponds to the capture time of video included in first captured video data representing captured video of a sports event, with status identification data for identifying the status of the sports event; an adding unit 114 that adds metadata, including the first additional data, to video data converted from the first captured video data; and a distribution unit 116 that distributes video data for distribution, which is video data with added metadata, to an information terminal 20, wherein the information terminal 20 includes a display control unit 213 which causes a display unit to display video for distribution represented by the received video data for distribution and which controls whether the status identification data associated with the timestamp included as metadata in the video data for distribution is to be superimposed onto the video for distribution and displayed, or hidden.

Description

Information processing system, information processing device, and information processing method

[0001] The present invention relates to an information processing system, an information processing device, and an information processing method.

[0002] Conventionally, information about a competition has been distributed in real time along with images of the competition (see, for example, Patent Document 1).

[0003] Japanese Patent Application Laid-Open No. 2008-271219

[0004] When images of a competition are distributed in real time, a composite image may be generated by combining information about the competition with the images of the competition, and the generated composite image may be distributed. In this case, the information about the competition may overlap with the athletes in the composite image, making it difficult to confirm the status of the competition.

[0005] The present invention has been made in consideration of these points, and aims to make it easier to check the situation of the competition.

[0006] An information processing device according to one aspect of the present invention comprises an information processing device and an information terminal, wherein the information processing device has an acquisition unit that acquires first additional data that associates a timestamp corresponding to the shooting time of the video, which is included in first shot video data indicating a video of a competition, with status identification data for identifying the status of the competition, an addition unit that adds metadata including the first additional data acquired by the acquisition unit to video data converted from the first shot video data, and a distribution unit that distributes video data for distribution, which is the video data to which the metadata has been added by the addition unit, to the information terminal, and the information terminal has a receiving unit that receives the video data for distribution, and a display control unit that displays the video for distribution indicated by the video data for distribution received by the receiving unit on a display unit, and controls whether to superimpose the status identification data associated with the timestamp included as the metadata in the video data for distribution on the video for distribution and display or hide it.

[0007] According to the present invention, it is possible to easily check the status of the competition.

[0008] FIG. 1 is a diagram showing an example of a system configuration of an information processing system according to an embodiment; FIG. 2 is a diagram showing a hardware configuration of an information processing device; FIG. 3 is a diagram showing a hardware configuration of an information terminal; FIG. 4 is a diagram showing an example of a functional block configuration of a processor included in an information processing device; FIG. 5 is a diagram showing an example of a functional block configuration of a processor included in an information terminal; FIG. 6 is a diagram showing an example of a competition screen; and FIG. 7 is a flowchart showing a processing flow in the information processing system.

[0009] <System Configuration> Fig. 1 is a diagram showing an example of the system configuration of an information processing system 1 according to this embodiment. The information processing system 1 shown in Fig. 1 includes an information processing device 10, one or more information terminals 20, a first image capturing device 30, and a second image capturing device 40. The information processing device 10 and the information terminal 20 are communicably connected to each other via a network N1 such as the Internet, an intranet, a wireless local area network (LAN), or a mobile communication network. The information processing device 10, the first image capturing device 30, and the second image capturing device 40 are communicably connected to each other via a network N2 such as the Internet, an intranet, or a wireless local area network (LAN).

[0010] The information processing device 10 is a device that distributes video footage of a competition in which multiple competitors participate in real time. A competition is a competition in which the rankings of multiple competitors are determined, such as bicycle racing, auto racing, horse racing, boat racing, or electronic sports. A competitor is, for example, a competitor, a racing vehicle, a racehorse, or a racing team. Users can cast a predetermined value of votes for a competition. A voting target is an object for which a user casts a predetermined value of votes, and is determined by combining the competitor that the user predicts will win from among multiple candidate competitors (including competitors, racing vehicles, or animals) and a betting method (voting method).

[0011] The information processing device 10 is configured, for example, by one or more physical servers. The information processing device 10 may be configured using a virtual server that operates on a hypervisor, or may be configured using a cloud server.

[0012] The information terminal 20 is a device used by a user, such as a mobile phone (including a smartphone), a tablet terminal, or a personal computer. For example, a dedicated application program (hereinafter referred to as a "dedicated app") for providing a voting service to users is installed on the information terminal 20, and the information terminal 20 displays on its display unit videos of competitions distributed from the information processing device 10. Via the dedicated app, the user can watch the videos distributed from the information processing device 10 and cast votes for voting targets.

[0013] The first and second camera devices 30 and 40 are installed, for example, in a stadium. The first camera device 30 captures the competition from a first camera position to generate first captured video data. The first camera position is, for example, a camera position close to the competition location where multiple competition bodies compete. The second camera device 40 captures the competition from a second camera position different from the first camera position to generate second captured video data. The second camera position is, for example, a position where competition bodies are captured from a distance farther than the first camera position, and is a position where more competition bodies can be captured at once compared to when capturing images of the competition bodies from the first camera position.

[0014] In the past, when videos of a competition captured by the first image capturing device 30 and the second image capturing device 40 were distributed in real time to the information terminal 20, information about the competition, such as the ranking of the athletes, was sometimes combined with the video to generate the video and distributed. In this case, the information about the competition was sometimes superimposed on the athletes appearing in the video, making it difficult to confirm the status of the competition.

[0015] In response to this, the information processing device 10 acquires first additional data that associates a timestamp corresponding to the shooting time of the video included in the first captured video data showing the video of the competition captured by the first camera device 30 with status identification data for identifying the status of the competition, such as the ranking of multiple competing bodies participating in the competition. The information processing device 10 adds metadata including the acquired first additional data to the video data converted from the first captured video data, and distributes the video data for distribution, which is the video data with the added metadata, to the information terminal 20.

[0016] The information terminal 20 displays the video to be distributed indicated by the video to be distributed data received from the information processing device 10 on a display unit, and controls whether to superimpose or hide the status identification data associated with the timestamp included as metadata in the video to be distributed on the video to be distributed based on a user operation. Since information indicating the status of the competition, such as the ranking, is not synthesized in the video to be distributed but is provided as metadata, the user can, by their own operation, switch between hiding the status identification data superimposed on the video to be distributed to make it easier to check the status of the competition, and superimposing the status identification data on the video to be distributed to make it easier to check the status of the competition. The configurations and operations of the information processing device 10 and the information terminal 20 will be described in detail below.

[0017] 2 is a diagram showing an example of the hardware configuration of the information processing device 10. The information processing device 10 includes a processor 11, a storage unit 12, a communication IF (Interface) 13, an input device 14, and an output device 15.

[0018] The processor 11 is, for example, a central processing unit (CPU) or a graphical processing unit (GPU), and realizes various functions by executing programs stored in the storage unit 12 .

[0019] The storage unit 12 includes a storage medium such as a memory, a hard disk drive (HDD), a solid state drive (SSD), etc. The storage unit 12 stores the programs executed by the processor 11.

[0020] The communication IF 13 has a communication controller for performing wired communication or wireless communication with the information terminal 20 via the network N. The communication IF 13 notifies the processor 11 of data received from the information terminal 20, for example. The communication IF 13 also transmits data input from the processor 11 to the information terminal 20, for example.

[0021] The input device 14 accepts input operations from, for example, an administrator of the information processing device 10. The input device 14 is, for example, a keyboard, a touch panel, a mouse, or a microphone. The output device 15 outputs information to, for example, the administrator of the information processing device 10. The output device 15 is, for example, a display or a speaker. Note that, if the information processing device 10 is provided on the cloud, the information processing device 10 may not have the input device 14 or the output device 15. In this case, the information processing device 10 may receive operation information indicating an input operation from a terminal (not shown) used by the administrator of the information processing device 10 via the communication IF 13, and transmit output information to the terminal.

[0022] 3 is a diagram showing an example of the hardware configuration of the information terminal 20. The information terminal 20 includes a processor 21, a storage unit 22, a communication IF 23, an operation unit 24, and an output device 25.

[0023] The processor 21 is, for example, a CPU, and realizes various functions by executing programs stored in the storage unit 22 .

[0024] The storage unit 22 includes storage media such as a read-only memory (ROM) and a random access memory (RAM), etc. The storage unit 22 stores the programs executed by the processor 21.

[0025] The communication IF 23 has a communication controller for performing wired or wireless communication with the information processing device 10 via the network N. The communication IF 23 notifies the processor 21 of data received from the information processing device 10. The communication IF 23 also transmits data input from the processor 21 to the information processing device 10.

[0026] The operation unit 24 receives input operations from a user, for example. The operation unit 24 is, for example, a keyboard, a touch panel that receives touch operations, a mouse, or a microphone. The output device 25 is, for example, a display that outputs information such as video, and a speaker that outputs sound.

[0027] 4 is a diagram showing an example of a functional block configuration of the processor 11 included in the information processing device 10. The processor 11 includes an analysis unit 111, an acquisition unit 112, an encoding unit 113, an addition unit 114, a monitoring unit 115, and a distribution unit 116. The processor 11 executes a program stored in the storage unit 12 to function as the analysis unit 111, the acquisition unit 112, the encoding unit 113, the addition unit 114, the monitoring unit 115, and the distribution unit 116.

[0028] 5 is a diagram showing an example of a functional block configuration of the processor 21 included in the information terminal 20. The processor 21 includes an acquisition unit 211, a reception unit 212, and a display control unit 213. The processor 21 functions as the acquisition unit 211, the reception unit 212, and the display control unit 213 by executing a program stored in the storage unit 22.

[0029] <Explanation of Functions of Processor 11> Next, the functions of the processor 11 will be described. The analysis unit 111 receives the first captured video data from the first camera device 30 and identifies the state of the competition as a first state by analyzing the first video represented by the first captured video data. The state of the competition is, for example, the ranking of each of the multiple athletes competing in the competition. The analysis unit 111 generates first additional data that associates a timestamp included in the first captured video data with first state identification data that indicates the first state at the capture time corresponding to the timestamp, identified based on the first captured video data.

[0030] The first state specifying data is, for example, ranking data indicating the ranking of each of the multiple athletes competing in the competition, and is data in which the athletes' numbers, which serve as athletes' identification information for identifying the athletes appearing in the first video, are arranged in order from 1. For example, if the ranking data is (3, 1, 4), this indicates that the athletes' numbers for the first-place athletes are 3, the second-place athletes are 1, and the third-place athletes are 4.

[0031] Similarly, the analysis unit 111 receives the second filmed video data from the second filming device 40 and analyzes the second video represented by the second filmed video data to identify the state of the competition as a second state. The analysis unit 111 then generates second additional data that associates a timestamp included in the second filmed video data with state identification data that indicates the second state at the filming time corresponding to the timestamp, which is identified based on the second filmed video data.

[0032] The second condition specifying data, like the first condition specifying data, is ranking data indicating the ranking of each of the multiple athletes competing in the competition. For example, the ranking data is data in which the athletes' numbers, serving as athletes' identification information for identifying the athletes appearing in the second video, are arranged in order from 1, such as (3, 1, 4, 2, 6, 5). Because the second video is shot from a position where the athletes are photographed from a greater distance than the first video shooting position, the second condition specifying data includes data indicating the ranking of more athletes based on the second video than the first condition specifying data.

[0033] The analysis unit 111 generates the first additional data and the second additional data corresponding to all frames constituting the video, for example. The timestamp included in the first captured video data and the timestamp included in the second captured video data are synchronized, and the same timestamp included in the first captured video data and the second captured video data indicates the same capture time.

[0034] Although the analysis unit 111 is configured to analyze the captured video data and generate the additional data, this is not limiting. For example, the information processing system 1 may acquire the additional data from an external analysis device that analyzes the captured video data and generates the additional data. The analysis device may then analyze the first video indicated by the first captured video data and the second video indicated by the second captured video data, and generate the first additional data and the second additional data. The analysis device may transmit the generated first additional data and the second additional data to the information processing device 10.

[0035] Furthermore, although the analysis unit 111 generated the first additional data and the second additional data corresponding to all frames constituting the video, this is not limited to this. The analysis unit 111 may generate first additional data in which a timestamp corresponding to a frame at which a first state changes, identified based on the video represented by the first filmed video data, is associated with first state identification data for identifying the state of the game after the change in the first state corresponding to the timestamp. Similarly, the analysis unit 111 may generate second additional data in which a timestamp corresponding to a frame at which a second state changes, identified based on the video represented by the second filmed video data, is associated with second state identification data for identifying the state of the game after the change in the second state corresponding to the timestamp.

[0036] In this case, the analysis unit 111 identifies the timestamp of the frame at which the first state changed. The analysis unit 111 then generates first additional data that associates the identified timestamp of the change in the first state with first state identification data for identifying the state of the game after the change in the first state corresponding to the timestamp. Similarly, the analysis unit 111 generates second additional data that associates the timestamp of the frame at which the second state changed with second state identification data for identifying the state of the game after the change in the second state corresponding to the timestamp. In this way, additional data is generated only for frames within a limited time period from the time the state of the game changed, out of all frames. This allows the information processing device 10 to reduce the frequency at which additional data is generated.

[0037] Furthermore, the analysis unit 111 may generate first additional data including display control information for controlling the superimposition of effect data on the video to be distributed represented by the video to be distributed to the information terminal 20 by the addition unit 114. The effect data includes at least one of image data and text data indicating the start or end of the race. Furthermore, in the case of a race such as a bicycle race or horse race in which the racers race around a track, the effect data includes data indicating the number of times the racers have left to race around the track before reaching the finish line. For example, the analysis unit 111 receives, from a management terminal or the like, a shooting time at which effect data is to be displayed, from among the shooting times corresponding to the video to be distributed, and the type of effect data.

[0038] The analysis unit 111 then generates first additional data in which the timestamp corresponding to the received shooting time is associated with display control information for displaying the received type of performance data on the video for distribution.

[0039] The acquisition unit 112 acquires additional data that associates a timestamp corresponding to the shooting time of the video, which is included in the captured video data indicating the video of the game, with state identification data for identifying the state of the game. For example, the acquisition unit 112 acquires the first additional data and the second additional data generated by the analysis unit 111.

[0040] In addition, if an analysis device other than the information processing device 10 generates the first additional data and the second additional data instead of the analysis unit 111, the acquisition unit 112 may acquire the first additional data and the second additional data from the analysis device.

[0041] The encoding unit 113 encodes the first captured video data generated by the first camera device 30. For example, the encoding unit 113 converts the first captured video data into video data in a file format for streaming distribution such as HLS (HTTP Live Streaming). The video data converted from the first captured video data by the encoding unit 113 is called encoded video data.

[0042] The adding unit 114 adds metadata including at least one of the first additional data and the second additional data acquired by the acquiring unit 112 to the encoded data, which is video data converted from the first captured video data. First, the adding unit 114 acquires, for example, the encoded video data converted from the first captured video data by the encoding unit 113.

[0043] Next, the addition unit 114 determines whether or not a first relation, which is a relation between the timestamp indicated by the first additional data and the state of the competition, is consistent with a second relation, which is a relation between the timestamp indicated by the second additional data and the state of the competition. For example, the addition unit 114 determines whether or not the first relation and the second relation are consistent by determining whether or not the rankings of multiple competition objects indicated by the first state identification data included in the first additional data having the same timestamp are consistent with the rankings of multiple competition objects indicated by the second state identification data included in the second additional data.

[0044] For example, suppose the ranking data as data for specifying a first status included in the first additional data corresponding to a certain timestamp is (3, 4, 1), and the ranking data as data for specifying a second status included in the second additional data corresponding to that timestamp is (3, 4, 1, 5, 6, 2). In this case, since the first relationship and the second relationship are consistent, data (5, 6, 2) from the second additional data indicating the rankings of the competitors ranked fourth and below that cannot be identified from the first status identification data is combined with the first additional data. Then, the addition unit 114 adds metadata including the combined first additional data to the encoded video data to generate video data for distribution. Note that if the first relationship and the second relationship are consistent, the addition unit 114 may add metadata including the second additional data to the encoded video data to generate video data for distribution.

[0045] Furthermore, when the addition unit 114 determines that the first relationship and the second relationship are inconsistent, it adds metadata to the encoded video data, the metadata including first additional data corresponding to the part where the first relationship and the second relationship are inconsistent and second additional data corresponding to that part.

[0046] For example, suppose that the ranking data as the first state specifying data of the first additional data corresponding to a certain timestamp is (3, 4, 1), and the ranking data as the second state specifying data of the second additional data corresponding to that timestamp is (3, 1, 4, 5, 6, 2). In this case, the rankings of the second and third place player numbers indicated by the first state specifying data and the second and third place player numbers indicated by the second state specifying data are different, and therefore the addition unit 114 determines that the first relation and the second relation are not consistent.

[0047] In this case, the addition unit 114 prioritizes the first status identification data and combines the rankings (5, 6, 2) of the competitors ranked 4th and below that cannot be identified from the first status identification data with the first status identification data (3, 4, 1). As a result, the combined first status identification data becomes (3, 4, 1, 5, 6, 2). The addition unit 114 then adds metadata including the combined first status identification data to the encoded video data.

[0048] If the addition unit 114 determines that the first relationship and the second relationship are inconsistent, the addition unit 114 may add metadata that includes the first additional data but does not include the second additional data to the encoded video data. For example, assume that the ranking data as first status identification data in the first additional data corresponding to a certain timestamp is (3, 4, 1), and the ranking data as second status identification data in the second additional data corresponding to that timestamp is (3, 4, 5, 1, 6, 2). In this case, the number of the third-place player indicated by the first status identification data is the same as the number of the fourth-place player indicated by the second status identification data, and the addition unit 114 determines that the first relationship and the second relationship are inconsistent.

[0049] In this case, even if the second status specifying data is combined with the first status specifying data, giving priority to the order indicated by the first status specifying data, the combined first status specifying data will be (3, 4, 1, 1, 6, 2), and will contain multiple player numbers "1." In response to this, the addition unit 114 adds metadata that includes the first additional data but does not include the second additional data to the encoded video data. Note that, when multiple player numbers for the same player are included in the combined status specifying data, the addition unit 114 may add metadata that includes the first additional data before combination but does not include the second additional data before combination to the encoded video data.

[0050] If the first relationship, which is the relationship between the timestamp indicated by the first additional data and the state of the competition, is inconsistent with the second relationship, which is the relationship between the timestamp indicated by the second additional data and the state of the competition, the monitoring unit 115 may output information indicating that the first relationship and the second relationship are inconsistent to a monitoring terminal (not shown) that monitors the state of the competition.

[0051] For example, the monitoring unit 115 outputs error information indicating that the first relation and the second relation are inconsistent, including a timestamp of the first additional data or the second additional data in the case of inconsistency and information associating the first state specifying data with the second state specifying data, to the monitoring terminal. In this way, the monitor using the monitoring terminal can grasp the point in time at which the state of the competition specified from the first captured video data and the state of the competition specified from the second captured video data are inconsistent, and can take appropriate action, such as correcting the ranking data.

[0052] The distribution unit 116 distributes the video data for distribution, which is video data to which metadata has been added by the addition unit 114, to the information terminal 20. For example, the distribution unit 116 executes a dedicated application for providing a voting service, and streams the video data for distribution in real time to the information terminal 20 that is requesting the information processing device 10 to acquire the video data.

[0053] Note that a delay of several seconds to several tens of seconds may occur between the time when the first camera device generates the video data for shooting and the time when the video data for distribution corresponding to the video data for shooting is distributed due to various processes such as encoding and adding metadata. In this embodiment, the distribution of real-time video data also includes the distribution of video data with a delay of several seconds to several tens of seconds.

[0054] <Description of Functions of Processor 21> Next, a description will be given of functions of the processor 21. The processor 21 of the information terminal 20 executing a dedicated application for providing the voting service functions as an acquisition unit 211, a reception unit 212, and a display control unit 213.

[0055] Before receiving the video data for distribution related to the competition, the acquisition unit 211 acquires each of the multiple types of effect data in advance from a storage device that stores effect data to be superimposed on the video data for distribution. For example, the acquisition unit 211 accesses the storage device when the dedicated app is executed and acquires each of the multiple types of effect data from the storage device. The acquisition unit 211 stores the acquired effect data in the storage unit 22.

[0056] The receiving unit 212 receives video data for distribution from the information processing device 10. For example, the receiving unit 212 receives from the information processing device 10 video data for distribution that the information processing device 10 is streaming.

[0057] The display control unit 213 causes the display unit to display the video for distribution indicated by the video for distribution data received by the receiving unit 212. The display control unit 213 also controls whether to superimpose and display the state identification data associated with the timestamp included as metadata in the video for distribution on the video for distribution or not.

[0058] For example, first, the display control unit 213 causes the display unit to display a competition screen including a video for distribution and a switching image for switching whether or not to display the condition identification data superimposed on the video for distribution. Next, the display control unit 213 identifies the condition identification data associated with a timestamp included as metadata in the video for distribution. The display control unit 213 then displays a ranking image indicating the ranking of the athlete indicated by the identified condition identification data superimposed on a frame of the video for distribution corresponding to the timestamp.

[0059] 6A and 6B are diagrams illustrating an example of a competition screen. In the example shown in FIG. 6A, a ranking image showing the ranking of the competition object indicated by the state identification data is superimposed on the video for distribution, and a first switching image displaying "ranking not displayed" is also displayed. When the first switching image is selected in FIG. 6A, the display control unit 213 switches the ranking image displayed superimposed on the video for distribution to hidden, as shown in FIG. 6B, and displays a second switching image displaying "ranking displayed" in place of the first switching image. This allows the information terminal 20 to make it easier for the user to check the status of the competition.

[0060] Furthermore, when the second switching image is selected in Fig. 6(b), the display control unit 213 displays the ranking image superimposed on the video for distribution, as shown in Fig. 6(a), and displays the first switching image instead of the second switching image, thereby enabling the information terminal 20 to easily display the status of the competition, including competing objects not displayed in the video for distribution.

[0061] The acquisition unit 112 of the information processing device 10 may generate first additional data that associates a timestamp corresponding to a frame at which the first state changes, which is identified based on the video indicated by the first captured video data, with first state identification data for identifying the state of the competition after the first state changes corresponding to the timestamp. In this case, metadata is not attached to all shooting times of the video for distribution indicated by the video data for distribution, but metadata is attached corresponding to the shooting times at which the state of the competition changes.

[0062] In this case, when the display control unit 213 superimposes and displays the status identification data on the video for distribution, the display control unit 213 superimposes and displays the status identification data corresponding to a timestamp included in the metadata on the video for distribution from the shooting time corresponding to the timestamp to the shooting time corresponding to the timestamp stored in the metadata next to the metadata indicating the timestamp. For example, since metadata is not always added to video data for distribution, when metadata is added to the video data for distribution, the display control unit 213 superimposes and displays the status identification data included in the metadata on the video for distribution. Then, when the receiving unit 211 receives new video data for distribution to which metadata has been added, the display control unit 213 switches the status identification data to be superimposed on the video for distribution to the status identification data included in the metadata added to the video data for distribution.

[0063] Furthermore, when display control information is associated with a timestamp included as metadata in the video data for distribution, the display control unit 213 superimposes and displays the performance data previously acquired by the acquisition unit 211 and stored in the storage unit 22 on the video for distribution. For example, first, when display control information is associated with a timestamp included as metadata in the video data for distribution received by the receiving unit 212, the display control unit 213 identifies the type of performance data indicated by the display control information to be displayed on the video indicated by the video data for distribution.

[0064] Next, the display control unit 213 superimposes the specified type of effect data stored in the storage unit 22 onto the video for distribution indicated by the video for distribution data. In this way, the information terminal 20 can acquire the effect data without delay and display the effect data without delay on the video for distribution indicated by the video for distribution data.

[0065] Note that, when display control information is associated with a timestamp included as metadata in the video data for distribution, the display control unit 213 superimposes the effect data stored in the storage unit 22 on the video data for distribution. However, this is not limited to this. For example, when display control information is associated with a timestamp included as metadata in the video data for distribution received by the receiving unit 212, the acquisition unit 211 may acquire effect data from a storage device that stores effect data. Then, the display control unit 213 may superimpose the effect data acquired by the acquisition unit 211 on the video data for distribution. In this way, the information terminal 20 can reduce the amount of data stored.

[0066] <Processing Flow in Information Processing System 1> Next, a description will be given of the processing flow in the information processing system 1. Fig. 7 is a flowchart showing the processing flow in the information processing device 10. The flowchart shown in Fig. 7 is started, for example, when distribution of a video for distribution is started, and the processing shown in this flowchart is repeated.

[0067] First, the analysis unit 111 acquires the first captured video data and the second captured video data (S1). The encoding unit 113 encodes the first captured video data and converts the first captured video data into encoded video data (S2). Note that the process of S2 is executed in parallel with the processes of S3 to S8 described below.

[0068] The analysis unit 111 generates first additional data based on the first captured video data and generates second additional data based on the second captured video data (S3), and the acquisition unit 112 acquires the generated first additional data and second additional data (S4).

[0069] Next, the adding unit 114 determines whether the first additional data and the second additional data acquired in S4 are consistent (S5). If the adding unit 114 determines that the first additional data and the second additional data are consistent (YES in S5), the process proceeds to S6, where the first additional data and the second additional data are combined.

[0070] If the addition unit 114 determines that the first additional data and the second additional data are inconsistent (NO in S5), the process proceeds to S7, where the addition unit 114 calculates the reliability of the first additional data and the second additional data. For example, the addition unit 114 analyzes a frame of the video indicated by the first filmed video data corresponding to the first additional data and a frame of the video indicated by the second filmed video data corresponding to the second additional data, and identifies the size of the object as the subject and the orientation of the object captured in the frame. The addition unit 114 calculates the reliability of the first additional data and the second additional data based on the size of the object and the orientation of the object. For example, the addition unit 114 calculates the reliability so that the larger the size of the object and the closer the orientation of the object captured in the frame is to the orientation of the object captured when viewed edge-on, the higher the reliability.

[0071] Next, the adding unit 114 selects the additional data with the higher reliability from the first additional data and the second additional data (S8). Next, the adding unit 114 adds the additional data after being combined in S5 or the additional data selected in S7 to metadata including the additional data, thereby generating video data for distribution with the added metadata (S9). Next, the distribution unit 116 distributes the video data for distribution to the information terminal 20 (S10).

[0072] In the flowchart shown in FIG. 7, when the additional data is inconsistent, the addition unit 114 selects the additional data based on the reliability. However, as described above, when the additional data is inconsistent, the addition unit 114 may select the first additional data.

[0073] [Variant 1] In the above-described embodiment, the acquisition unit 112 acquires first additional data that associates a timestamp included in the first captured video data with first state identification data that indicates a first state identified based on the first captured video data, and second additional data that associates a timestamp included in the second captured video data with state identification data that indicates a second state identified based on the second captured video data, but this is not limited to this.

[0074] The acquisition unit 112 may acquire first additional data that associates a timestamp included in the first captured video data with state identification data that indicates a first state, which is the state of the competition identified based on the first captured video data, and difference data that indicates the difference between the first state and a second state, which is the state of the competition identified based on data different from the first captured video data.

[0075] For example, the first state is first ranking data indicating the ranking of each of the multiple competing bodies as a state of the competition identified based on the first filmed video data. The second state is second ranking data indicating the ranking of each of the multiple competing bodies as a state of the competition identified based on the second filmed video data. The second ranking data includes data indicating the rankings of more competing bodies than the first ranking data. The acquisition unit 112 identifies, as differential data, data indicating the rankings of the competing bodies that are included in the second ranking data but not included in the first ranking data. The addition unit 114 then adds metadata based on the first additional data and differential data acquired by the acquisition unit 112 to the video data.

[0076] By doing this, the information processing device 10 can reflect the state of the competition that cannot be identified from the first filmed video data in the metadata, just as when generating metadata based on the first additional data and the second additional data.

[0077] [Variation 2] In the above-described embodiment, the display control unit 213 controls whether or not to display the status identification data on the video for distribution in response to the user's operation of the switching image that switches whether or not to superimpose and display the status identification data on the video for distribution. However, this is not limited to this. For example, the display control unit 213 may display the status identification data at a position different from the position of the playing object displayed in the video for distribution.

[0078] In this case, the acquisition unit 112 may acquire first additional data that associates a timestamp included in the first filmed video data with position data that indicates the position of an image of a playing object in the video for distribution indicated by the first filmed video data. Then, the addition unit 114 may add metadata including the first additional data to the encoded video data to generate the video for distribution.

[0079] When the display control unit 213 of the information terminal 20 superimposes and displays the status identification data on the video for distribution indicated by the video for distribution data, the display control unit 213 may display a ranking image as status identification data at a position different from the position of the playing object displayed in the video for distribution corresponding to the timestamp, based on position data associated with the timestamp included in the metadata. In this case, the display control unit 213 may change the size of the ranking image so that the ranking image is displayed at a position different from the position of the image showing the playing object so that it does not superimpose on the image showing the playing object. In this way, the information terminal 20 can make it easier for the user to check the status of the playing object shown in the video for distribution and to check the ranking of the playing object indicated by the status identification data.

[0080] <Effects of Information Processing System 1> As described above, the information processing device 10 of the information processing system 1 acquires first additional data that associates a timestamp corresponding to the shooting time of the video, which is included in first captured video data indicating a video of a captured competition, with state identification data for identifying the state of the competition, and distributes video data for distribution, which is video data converted from the first captured video data and to which metadata including the first additional data has been added, to the information terminal 20. The information terminal 20 of the information processing system 1 displays the video for distribution indicated by the received video for distribution data on a display unit, and controls whether to display or hide the state identification data associated with the timestamp included as metadata in the video for distribution so as to be superimposed on the video for distribution. In this way, the information terminal 20 can enable the user to easily select a video they wish to watch.

[0081] <Summary> [General Problem] One of the objects of the present invention is to make it easier to check the situation of a competition.

[0082] [Problem Corresponding to Supplementary Note 1] One object of the present invention is to make it possible to switch between making it easier for a user to check a video showing a competition and making it easier for the user to check the state of the competition. [Supplementary Note 1] An information processing system including an information processing device and an information terminal, wherein the information processing device has an acquisition unit that acquires first additional data that associates a timestamp included in first captured video data showing a video of a competition, the timestamp corresponding to the shooting time of the video, with state identification data for identifying the state of the competition, an addition unit that adds metadata including the first additional data acquired by the acquisition unit to video data converted from the first captured video data, and a distribution unit that distributes video data for distribution to the information terminal, the video data to which the metadata has been added by the addition unit, and the information terminal has a receiving unit that receives the video data for distribution, and a display control unit that displays the video for distribution represented by the video data for distribution received by the receiving unit on a display unit, and controls whether the state identification data associated with the timestamp included as the metadata in the video data for distribution is superimposed on the video for distribution and displayed or not. By configuring the information processing system in this way, it is possible to switch between making it easier for the user to check the video showing the competition, and making it easier for the user to check the status of the competition.

[0083] [Problem corresponding to Supplementary Note 2] One of the objects of the present invention is to reduce communication load. [Supplementary Note 2] The information processing system described in Supplementary Note 1, wherein the acquisition unit acquires the first additional data that associates the timestamp at the time of a change in the state of the competition with the status identification data for identifying the status of the competition after the change in the state of the competition corresponding to the timestamp, and the display control unit, when superimposing and displaying the status identification data on the video for distribution, displays the status identification data corresponding to the timestamp included in the metadata superimposed on the video for distribution from the shooting time corresponding to the timestamp to the shooting time corresponding to the next timestamp after the timestamp. By configuring the information processing system in this way, it is possible to reduce communication load.

[0084] [Problem corresponding to Supplementary Note 3] One object of the present invention is to reflect in metadata a state of a competition that cannot be identified from first filmed video data. [Supplementary Note 3] The information processing system described in Supplementary Note 1, wherein the acquisition unit acquires first additional data that associates the timestamp included in the first filmed video data with the state identification data that indicates a first state, which is the state of the competition identified based on the first filmed video data, and difference data that indicates a difference between the first state and a second state, which is the state of the competition identified based on data different from the first filmed video data, and the addition unit adds to the video data the metadata based on the first additional data and the difference data acquired by the acquisition unit. This allows the information processing system to reflect in metadata a state of a competition that cannot be identified from the first filmed video data.

[0085] [Problem corresponding to Supplementary Note 4] One object of the present invention is to reflect in metadata the state of a competition that cannot be identified from first filmed video data. [Supplementary Note 4] The information processing system described in Supplementary Note 1, wherein the acquisition unit acquires second additional data that associates a timestamp corresponding to second filmed video data that was filmed from a filming position different from the filming position of the competition corresponding to the first filmed video data with state identification data for identifying the state of the competition, and the addition unit adds to the video data the metadata that includes at least one of the first additional data and the second additional data acquired by the acquisition unit. This allows the information processing system to reflect in metadata the state of a competition that cannot be identified from the first filmed video data.

[0086] [Problem corresponding to Supplementary Note 5] One of the objects of the present invention is to align status identification data included in the metadata of video data for distribution with the status of the competition shown in the video data for distribution. [Supplementary Note 5] The information processing system described in Supplementary Note 4, wherein, when a first relationship, which is a relationship between the timestamp indicated by the first additional data and the status of the competition, does not match a second relationship, which is a relationship between the timestamp indicated by the second additional data and the status of the competition, the addition unit adds to the video data the metadata including the first additional data corresponding to a portion where the first relationship and the second relationship do not match, and the second additional data corresponding to a portion different from that portion. This enables the information processing system to align the status identification data included in the metadata of the video data for distribution with the status of the competition shown in the video data for distribution.

[0087] [Problem corresponding to Supplementary Note 6] One of the objects of the present invention is to align status identification data included in the metadata of video data for distribution with the status of the competition indicated by the video data for distribution. [Supplementary Note 6] The information processing system described in Supplementary Note 4, wherein, when a first relationship, which is the relationship between the timestamp indicated by the additional data and the status of the competition, does not match a second relationship, which is the relationship between the timestamp indicated by the additional data and the status of the competition, the addition unit adds the metadata, which includes the first additional data but does not include the second additional data, to the video data. This enables the information processing system to align the status identification data included in the metadata of the video data for distribution with the status of the competition indicated by the video data for distribution.

[0088] [Problem corresponding to Supplementary Note 7] One of the objects of the present invention is to allow an observer monitoring the state of a competition to recognize that the state of the competition indicated by the first additional data and the second additional data is inconsistent. [Supplementary Note 7] The information processing system described in Supplementary Note 4, wherein the information processing device further has a monitoring unit that, when a first relationship, which is the relationship between the timestamp indicated by the first additional data and the state of the competition, and a second relationship, which is the relationship between the timestamp indicated by the second additional data and the state of the competition, are inconsistent, outputs information indicating that the first relationship and the second relationship are inconsistent to a terminal monitoring the state of the competition. This allows the information processing system to allow an observer monitoring the state of the competition to recognize that the state of the competition indicated by the first additional data and the second additional data is inconsistent.

[0089] [Problem Corresponding to Supplementary Note 8] One object of the present invention is to enable superimposing effect data on a video for distribution while reducing the amount of communication traffic during distribution of video data for distribution. [Supplementary Note 8] The information processing system described in Supplementary Note 1, wherein the information terminal further includes a storage unit that stores effect data to be superimposed on the video data for distribution, the first additional data includes display control information associated with the timestamp corresponding to the shooting time at which the effect data is to be displayed, and that causes the effect data to be displayed on a video represented by the video data for distribution, and the display control unit causes the effect data stored in the storage unit to be superimposed on the video for distribution when the display control information is associated with the timestamp included as metadata in the video data for distribution. This enables the information processing system to superimpose effect data on the video for distribution while reducing the amount of communication traffic during distribution of the video data for distribution.

[0090] [Problem Corresponding to Supplementary Note 9] One object of the present invention is to enable superimposition of effect data on a video for distribution without reducing the available storage space of a storage unit of an information terminal. [Supplementary Note 9] The information processing system described in Supplementary Note 1, wherein the first additional data includes display control information for superimposing effect data related to a competition on a video represented by the video data for distribution, the display control information being associated with a timestamp corresponding to a shooting time at which the effect data related to a competition is superimposed on the video represented by the video data for distribution, the information terminal having a second acquisition unit that acquires the effect data from a storage device that stores the effect data when the display control information is associated with the timestamp included as metadata in the video data for distribution, and the display control unit superimposes and displays the effect data acquired by the second acquisition unit on the video for distribution. This enables the information processing system to superimpose effect data on a video for distribution without reducing the available storage space of a storage unit of the information terminal.

[0091] [Problem Corresponding to Supplementary Note 10] One object of the present invention is to enable the superimposition of effect data on a video for distribution without delay. [Supplementary Note 10] The information processing system described in Supplementary Note 1, wherein the first additional data includes display control information for superimposing effect data related to a competition on a video represented by the video data for distribution, the display control information being associated with a timestamp corresponding to a shooting time at which the effect data related to a competition is superimposed and displayed on the video represented by the video data for distribution, the information terminal has a second acquisition unit that acquires the effect data from a storage device that stores the effect data before receiving the video data for distribution related to the competition, and the display control unit causes the effect data acquired by the second acquisition unit to be superimposed on the video for distribution if the display control information is associated with the timestamp included as metadata in the video data for distribution. This enables the information processing system to enable the information terminal to superimpose effect data on the video for distribution without delay.

[0092] [Problem Corresponding to Supplementary Note 11] One of the objects of the present invention is to make it easier to check the status of a player appearing in a video for distribution and to make it easier to check the status of the player indicated by the status identification data. [Supplementary Note 11] The information processing system described in Supplementary Note 1, wherein the first additional data includes position data associated with the timestamp and indicating a position in the video for distribution of the player appearing in the video for distribution corresponding to the timestamp, and the display control unit, when superimposing and displaying the status identification data on the video for distribution, displays the status identification data at a position different from the position of the player appearing in the video for distribution corresponding to the timestamp, based on the position data associated with the timestamp included in the first additional data. This makes it easier for users viewing the video for distribution to check the status of the player appearing in the video for distribution and to check the status of the player indicated by the status identification data.

[0093] [Problem corresponding to Supplementary Note 12] One of the objects of the present invention is to make it possible to switch between making it easier for a user to check a video showing a competition and making it easier for a user to check the status of the competition. [Supplementary Note 12] An information processing device having: an acquisition unit that acquires first additional data that associates a timestamp corresponding to the shooting time of the video, which is included in first captured video data showing a video of a competition, with status identification data for identifying the status of the competition; an addition unit that adds metadata including the first additional data acquired by the acquisition unit to video data converted from the first captured video data; and a distribution unit that distributes video data for distribution, which is the video data to which the metadata has been added by the addition unit. By configuring the information processing device in this way, it is possible to switch between making it easier for a user to check a video showing a competition and making it easier for a user to check the status of the competition.

[0094] [Problem corresponding to Supplementary Note 13] One object of the present invention is to make it possible to switch between making it easier for a user to check a video showing a competition and making it easier for a user to check the state of the competition. [Supplementary Note 13] An information processing method comprising the steps of: an information processing device acquiring first additional data that associates a timestamp included in first captured video data showing a video of a competition, the timestamp corresponding to the shooting time of the video, and state identification data for identifying the state of the competition; a step of the information processing device adding metadata including the acquired first additional data to video data converted from the first captured video data; a step of the information processing device delivering video data for distribution, which is the video data added with the metadata, to an information terminal; a step of the information terminal receiving the video data for distribution; and a step of the information terminal displaying the video for distribution indicated by the received video data for distribution on a display unit, and controlling whether the state identification data associated with the timestamp included as the metadata in the video data for distribution is superimposed on the video for distribution and displayed or not. By configuring the information processing method in this way, it is possible to switch between making it easier for the user to check the video showing the competition, and making it easier for the user to check the status of the competition.

[0095] While the present invention has been described above using embodiments, the technical scope of the present invention is not limited to the scope described in the above embodiments, and various modifications and alterations are possible within the spirit and scope of the present invention. For example, in the above embodiment, a voting service is provided by executing a dedicated app installed on the information terminal 20, and the display control unit 211 controls whether to superimpose and display or hide status identification data on the streaming video included in the competition screen of the dedicated app. However, this is not limited to this. For example, a voting service may be provided by a web browser installed on the information terminal 20, and the display control unit 211 may control whether to superimpose and display or hide status identification data on the streaming video included in the competition screen displayed by the browser in response to the voting service.

[0096] Furthermore, for example, all or part of the device can be configured by functionally or physically distributing or integrating in any unit. Furthermore, new embodiments resulting from any combination of multiple embodiments are also included in the embodiments of the present invention. The effects of the new embodiments resulting from the combination also have the effects of the original embodiments.

[0097] REFERENCE SIGNS LIST 1 Information processing system 10 Information processing device 11 Processor 12 Storage unit 13 Communication IF 14 Input device 15 Output device 20 Information terminal 21 Processor 22 Storage unit 23 Communication IF 24 Operation unit 25 Output device 30 First image capture device 40 Second image capture device 111 Analysis unit 112 Acquisition unit 113 Encoding unit 114 Addition unit 115 Monitoring unit 116 Distribution unit 211 Reception unit 212 Acquisition unit 213 Display control unit

Claims

1. An information processing system comprising an information processing device and an information terminal, wherein the information processing device has: an acquisition unit that acquires first additional data that associates a timestamp included in first shot video data indicating a video of a competition, the timestamp corresponding to the shooting time of the video, with status identification data for identifying the status of the competition; an addition unit that adds metadata including the first additional data acquired by the acquisition unit to video data converted from the first shot video data; and a distribution unit that distributes video data for distribution, which is the video data to which the metadata has been added by the addition unit, to the information terminal, wherein the information terminal has: a receiving unit that receives the video data for distribution; and a display control unit that displays the video for distribution indicated by the video data for distribution received by the receiving unit on a display unit, and controls whether to superimpose the status identification data associated with the timestamp included as the metadata in the video data for distribution on the video for distribution and display or hide it.

2. The information processing system of claim 1, wherein the acquisition unit acquires the first additional data that associates the timestamp at the time of a change in the state of the competition with the status identification data for identifying the status of the competition after the change in the state of the competition corresponding to the timestamp, and when the display control unit superimposes and displays the status identification data on the video for distribution, the display control unit superimposes and displays the status identification data corresponding to the timestamp included in the metadata on the video for distribution from the shooting time corresponding to the timestamp to the shooting time corresponding to the next timestamp after the timestamp.

3. The information processing system of claim 1, wherein the acquisition unit acquires first additional data that associates the timestamp included in the first filmed video data with the state identification data that indicates a first state, which is the state of the competition identified based on the first filmed video data, and difference data that indicates the difference between the first state and a second state, which is the state of the competition identified based on data different from the first filmed video data; and the addition unit adds the metadata based on the first additional data and the difference data acquired by the acquisition unit to the video data.

4. The information processing system of claim 1, wherein the acquisition unit acquires second additional data that associates a timestamp corresponding to second filmed video data filmed from a filming position different from the filming position of the competition corresponding to the first filmed video data with status identification data for identifying the status of the competition, and the addition unit adds the metadata including at least one of the first additional data and the second additional data acquired by the acquisition unit to the video data.

5. The information processing system of claim 4, wherein, when a first relationship, which is the relationship between the timestamp indicated by the first additional data and the state of the competition, is inconsistent with a second relationship, which is the relationship between the timestamp indicated by the second additional data and the state of the competition, the addition unit adds to the video data the metadata including the first additional data corresponding to a part where the first relationship and the second relationship are inconsistent and the second additional data corresponding to a part different from that part.

6. The information processing system of claim 4, wherein the addition unit adds the metadata including the first additional data but not the second additional data to the video data when a first relationship, which is the relationship between the timestamp indicated by the additional data and the state of the competition, is inconsistent with a second relationship, which is the relationship between the timestamp indicated by the additional data and the state of the competition.

7. The information processing system of claim 4, wherein the information processing device further has a monitoring unit that, when a first relationship, which is the relationship between the timestamp indicated by the first additional data and the state of the competition, is inconsistent with a second relationship, which is the relationship between the timestamp indicated by the second additional data and the state of the competition, outputs information indicating that the first relationship and the second relationship are inconsistent to a terminal that monitors the state of the competition.

8. The information processing system of claim 1, wherein the information terminal further has a memory unit that stores effect data to be superimposed on the video data for distribution, the first additional data includes display control information that causes the effect data to be displayed on the video indicated by the video data for distribution, the display control information being associated with the timestamp corresponding to the shooting time at which the effect data is to be displayed, and the display control unit causes the effect data stored in the memory unit to be superimposed on the video for distribution when the display control information is associated with the timestamp included as metadata in the video data for distribution.

9. The information processing system described in claim 1, wherein the first additional data includes display control information for superimposing and displaying performance data related to the competition on the video represented by the video data for distribution, the display control information being associated with the timestamp corresponding to the shooting time at which the performance data is superimposed and displayed on the video represented by the video data for distribution; the information terminal has a second acquisition unit that acquires the performance data from a storage device that stores the performance data when the display control information is associated with the timestamp included as metadata in the video data for distribution; and the display control unit superimposes and displays the performance data acquired by the second acquisition unit on the video for distribution.

10. The information processing system described in claim 1, wherein the first additional data includes display control information for superimposing and displaying performance data related to the competition on the video represented by the video data for distribution, the display control information being associated with the timestamp corresponding to the shooting time at which the performance data related to the competition is superimposed and displayed on the video represented by the video data for distribution; the information terminal has a second acquisition unit that acquires the performance data from a storage device that stores the performance data before receiving the video data for distribution related to the competition; and the display control unit, when the display control information is associated with the timestamp included as metadata in the video data for distribution, causes the performance data acquired by the second acquisition unit to be superimposed and displayed on the video for distribution.

11. The information processing system of claim 1, wherein the first additional data includes position data associated with the timestamp and indicating the position in the video for distribution of the playing object displayed in the video for distribution corresponding to the timestamp, and the display control unit, when superimposing and displaying the status identification data on the video for distribution, displays the status identification data at a position different from the position of the playing object displayed in the video for distribution corresponding to the timestamp, based on the position data associated with the timestamp included in the first additional data.

12. An information processing device having: an acquisition unit that acquires first additional data that associates a timestamp corresponding to the shooting time of the video, which is included in first shot video data indicating a video of a competition, with status identification data for identifying the status of the competition; an addition unit that adds metadata including the first additional data acquired by the acquisition unit to video data converted from the first shot video data; and a distribution unit that distributes video data for distribution, which is the video data to which the metadata has been added by the addition unit.

13. An information processing method comprising the steps of: an information processing device acquiring first additional data that associates a timestamp corresponding to the shooting time of the video, which is included in first shot video data indicating a video of a competition, with status identification data for identifying the status of the competition; a step by the information processing device adding metadata including the acquired first additional data to video data converted from the first shot video data; a step by the information processing device distributing video data for distribution, which is the video data with the metadata added, to an information terminal; a step by the information terminal receiving the video data for distribution; and a step by the information terminal displaying the video for distribution indicated by the received video data for distribution on a display unit, and controlling whether to superimpose the status identification data associated with the timestamp included as the metadata in the video data for distribution on the video for distribution and display or hide it.

Images