Video processing system, video processing method, and program

Abstract

To provide a video processing system, a video processing method, and a program capable of providing video of a competition in a form that is easier for viewers to understand. [Solution] A video processing system that processes video of a competition, comprising: an input unit that inputs multiple camera images of athletes performing the competition taken by multiple cameras; a first recognition unit that recognizes attributes related to the athletic movements performed by the athletes while performing the competition for each of the multiple camera images input by the input unit; a second recognition unit that integrates the multiple recognition results recognized by the first recognition unit for the attributes for each of the multiple camera images and recognizes the athletic movements based on the integrated recognition results; a generation unit that generates a second image for displaying information based on the recognition results of the athletic movements in synchronization with the timing of the athletic movements; and a synthesis unit that synthesizes the second image with the video of the competition.

Description

[Technical Field]

[0001] The present invention relates to a technique for processing video. [Background technology]

[0002] BACKGROUND ART Conventionally, video images of sports and other competitions have been provided to viewers via television broadcast waves, internet distribution, and the like (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2009-272970 Summary of the Invention [Problem to be solved by the invention]

[0004] However, conventional video delivery methods are not always easy to understand for viewers unfamiliar with the sport, making viewing difficult. For example, in boxing, scores (judging by round) are often not displayed during a match, making it difficult to tell which player is in the lead. Furthermore, a boxer's punch speed is extremely fast, at 0.1 to 0.2 seconds, making it difficult to tell with the naked eye whether a punch is effective or not.

[0005] The present invention has been made in consideration of these circumstances, and one of its objectives is to provide a video processing system, video processing method, and program that can provide competition footage in a format that is easier for viewers to understand. [Means for solving the problem]

[0006] One aspect of the present invention is a video processing system that processes video of a competition, comprising: an input unit that inputs multiple camera images of athletes performing the competition taken by multiple cameras; a first recognition unit that recognizes attributes related to the athletic movements performed by the athletes while performing the competition for each of the multiple camera images input by the input unit; a second recognition unit that integrates the multiple recognition results recognized by the first recognition unit for the attributes for each of the multiple camera images and recognizes the athletic movements based on the integrated recognition results; a generation unit that generates a second image to display information based on the recognition results of the athletic movements in synchronization with the timing of the athletic movements; and a synthesis unit that synthesizes the second image with the video of the competition.

[0007] One aspect of the present invention is the above-mentioned video processing system, wherein the first recognition unit recognizes multiple attributes for each frame of the camera image, calculates a score correlating with likelihood for multiple possible value options for each of the multiple attributes, and the second recognition unit integrates the multiple recognition results based on the score.

[0008] One aspect of the present invention is the above-mentioned video processing system, wherein the second recognition unit weights the scores of the multiple recognition results based on the priority corresponding to the corresponding camera among the multiple cameras, and integrates the multiple recognition results by summing the weighted scores.

[0009] One aspect of the present invention is the above-mentioned video processing system, wherein the first recognition unit detects the positions of multiple players from the multiple camera images, and the second recognition unit determines priorities for the multiple cameras based on the degree of overlap of the positions of the multiple players.

[0010] One aspect of the present invention is the above-mentioned video processing system, further comprising a statistical processing unit that performs statistical processing on the integrated recognition results, and the statistical processing unit smooths the integrated recognition results in a time series direction and determines the start and end timings of the athletic action based on the smoothed recognition results of the attributes.

[0011] One aspect of the present invention is the above-mentioned video processing system, wherein the statistical processing unit sums the integrated scores for each of the multiple options for the frames included in the start and end timing of the athletic action, and determines the option with the largest total value as the attribute of the athletic action.

[0012] One aspect of the present invention is the above-mentioned video processing system, wherein the statistical processing unit tally the number of times the athletic actions were performed within a period corresponding to the display purpose, and the generation unit generates as the second video an image that displays the number of times the athletic actions were performed tally by the statistical processing unit in a manner corresponding to the display purpose.

[0013] One aspect of the present invention is the video processing system described above, wherein the period is a period based on the rules of the competition.

[0014] One aspect of the present invention is the video processing system described above, wherein the display purpose is a purpose that differs depending on an attribute of the athletic action.

[0015] One aspect of the present invention is the above-mentioned video processing system, wherein the statistical processing unit determines conditions regarding the temporal continuity between multiple athletic movements based on the start timing and end timing of the multiple athletic movements, tallying up the number of times a series of athletic movements meets the conditions, and the generation unit generates an image displaying the number of times the series of athletic movements occurs as the second image.

[0016] One aspect of the present invention is a video processing method for processing video of a competition, comprising: an input step of inputting multiple camera images taken by multiple cameras of athletes performing the competition; a first recognition step of recognizing attributes related to the athletic movements performed by the athletes while performing the competition for each of the input camera images; a second recognition step of integrating multiple recognition results recognized for the attributes for each of the camera images and recognizing the athletic movements based on the integrated recognition results; a generation step of generating a second image for displaying information based on the recognition results of the athletic movements in synchronization with the timing of the athletic movements; and a synthesis step of compositing the second image with the video of the competition.

[0017] One aspect of the present invention is a program for causing a computer to execute the following steps in a video processing system for processing video of a competition: an input step of inputting multiple camera images taken by multiple cameras of athletes performing the competition; a first recognition step of recognizing attributes related to the athletic movements performed by the athletes while performing the competition for each of the input camera images; a second recognition step of integrating multiple recognition results recognized for the attributes for each of the camera images and recognizing the athletic movements based on the integrated recognition results; a generation step of generating a second image for displaying information based on the recognition results of the athletic movements in synchronization with the timing of the athletic movements; and a synthesis step of synthesizing the second image with the video of the competition. [Effects of the Invention]

[0018] According to one aspect of the present invention, it is possible to provide a video processing system, a video processing method, and a program that can provide video of a competition in a format that is easier for viewers to understand. [Brief explanation of the drawings]

[0019] [Figure 1] 1 is a diagram illustrating an example of a system configuration of a video processing system 1 according to an embodiment. [Figure 2]1 is a diagram showing an example of a processing flow in which the video processing system 1 of the embodiment processes a broadcast video. [Figure 3] FIG. 10 is a diagram (part 1) illustrating the flow of processing by the video recognition unit 122 to recognize the action attributes of players during a game. [Figure 4] FIG. 10 is a diagram (part 2) illustrating the flow of processing by the video recognition unit 122 to recognize the action attributes of players during a game. [Figure 5] FIG. 10 is a diagram illustrating an example of punch attributes. [Figure 6] FIG. 10 is a diagram illustrating a method for learning a punch determination model. [Figure 7] 10 is a diagram illustrating a process in which integrated recognition unit 123 integrates the recognition results of a plurality of punch attributes obtained by a plurality of image recognition units 122. FIG. [Figure 8] 10 is a diagram illustrating a first statistical process performed by the statistical processing unit 125. FIG. [Figure 9] FIG. 10 is a diagram showing an example of segment information that the statistical processing unit 125 explains in the second statistical processing. [Figure 10] FIG. 10 is a diagram (part 1) showing a display example of a broadcast video onto which CG created by the CG creating unit 126 has been combined. [Figure 11] FIG. 10 is a diagram (part 2) showing a display example of broadcast video combined with CG created by the CG creation unit 126. DETAILED DESCRIPTION OF THE INVENTION

[0020] [Video processing system] FIG. 1 illustrates an example of the system configuration of a video processing system 1 according to an embodiment. The video processing system 1 processes video of a competition for broadcast. The video processing system 1 may be configured with one or more information processing devices. The video processing system 1 functions as a system including the functional units shown in FIG. 1 when one or more information processing devices, each equipped with a processor such as a central processing unit (CPU) and memory, execute a program. Note that all or part of the functions of the information processing device 1 may be implemented using hardware such as an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a field-programmable gate array (FPGA). The program may be recorded on a computer-readable recording medium. Examples of computer-readable recording media include portable media such as a flexible disk, a magneto-optical disk, a read-only memory (ROM), a CD-ROM, and a semiconductor storage device (e.g., a solid-state drive (SSD)), as well as storage devices such as a hard disk or semiconductor storage device built into a computer system. The program may be transmitted via a telecommunications line.

[0021] The video processing system 1 includes one or more first video input units 111, and is connected to one or more broadcast cameras 10 via the one or more first video input units 111. The broadcast cameras 10 are cameras that capture video of a competition currently in progress as video for broadcast (broadcast video). The video processing system 1 inputs broadcast video from one or more broadcast cameras 10 via the one or more first video input units 111.

[0022] The video processing system 1 further includes a plurality of second video input units 121, and is connected to a plurality of CG cameras 20 via the plurality of second video input units 121. The CG cameras 20 are cameras that capture video for creating CG (CG video). The video processing system 1 inputs CG video from the plurality of CG cameras 20 via the plurality of second video input units 121. The video processing system 1 creates CG (Computer Graphic) video based on the CG video, and processes the video for broadcast by synthesizing the created CG video. Note that if the video for broadcast meets the requirements for CG video, the CG cameras 20 may be omitted, and a branch of the broadcast video may be used as the video for CG.

[0023] More specifically, the video processing system 1 includes, for example, a broadcast video switching unit 112 and a broadcast video synthesis unit 113 as functional units corresponding to one or more first video input units 111. The broadcast video switching unit 112 inputs a plurality of broadcast videos and switchably outputs one of the plurality of broadcast videos. When only one broadcast video is input to the video processing system 1, the broadcast video switching unit 112 may be omitted. In this embodiment, a case where a plurality of broadcast videos are input by a plurality of broadcast cameras 10 will be described.

[0024] The broadcast video synthesis unit 113 synthesizes the CG video with the broadcast video. The broadcast video synthesis unit 113 outputs the synthesized broadcast video. The output broadcast video is sent to a broadcasting system or the like, converted into broadcast waves, and transmitted wirelessly. In addition to wireless transmission via broadcast waves, the broadcast video may also be distributed via a telecommunications line such as the Internet.

[0025] On the other hand, the video processing system 1 includes, for example, a plurality of video recognition units 122, an integrated recognition unit 123, a competition information acquisition unit 124, a statistical processing unit 125, and a CG creation unit 126 as functional units corresponding to the plurality of second video input units 121. Each of the plurality of video recognition units 122 recognizes attributes (hereinafter referred to as "movement attributes") related to actions (hereinafter referred to as "competitive movements") performed by athletes during competition, based on the video for CG creation (hereinafter referred to as "CG video") input by the corresponding second video input unit 121. Hereinafter, the recognition results of the action attributes recognized by each of the plurality of video recognition units 122 will be referred to as "individual recognition results."

[0026] The integrated recognition unit 123 determines one final recognition result for the movement attributes of the athletic movement based on the individual recognition results of the movement attributes by the multiple video recognition units 122. In other words, the integrated recognition unit 123 can be said to integrate multiple recognition results for the movement attributes into one final recognition result. Hereinafter, this one final recognition result will be referred to as the "integrated recognition result."

[0027] The competition information acquisition unit 124 acquires competition information from the competition information server 30 and supplies it to the statistical processing unit 125. The competition information server 30 is a server that provides competition information. The competition information is information about the competition in which the athletes are participating. For example, the competition information may include information such as the athletes' profiles, the competition venue, and the rules of the competition. In addition, the competition information may include any information related to the competition. The video processing system 1 can communicate with the competition information server 30 via a network NW such as the Internet.

[0028] The statistical processing unit 125 performs a first statistical processing on the integrated recognition result of the action attributes acquired by the integrated recognition unit 123, thereby smoothing the integrated recognition result of the action attributes in the time direction. This evens out the frame-to-frame variations in the integrated recognition result of the action attributes, thereby improving the continuity of the integrated recognition result. Hereinafter, the integrated recognition result smoothed in the time direction by the first statistical processing is referred to as the "final recognition result" of the action attributes. By improving the continuity of the integrated recognition result by the first statistical processing, the statistical processing unit 125 can improve the processing accuracy of the subsequent second statistical processing.

[0029] The statistical processing unit 125 performs a second statistical processing on the final recognition results of the movement attributes regarding temporal continuity. By performing the second statistical processing, the statistical processing unit 125 recognizes the period during which each sport movement is performed (hereinafter referred to as a "segment"), and determines a label of the movement attribute to be assigned to the segment based on the multiple final recognition results corresponding to the segment. Since a segment corresponds to an individual sport movement, assigning a label of the movement attribute to a segment can be said to be determining the movement attribute for the sport movement performed in each segment.

[0030] The CG creation unit 126 recognizes the movement attributes of the athletic movements performed based on the labels assigned to each segment, and creates CG images for superimposing information about the recognized movement attributes on the broadcast image. The CG creation unit 126 outputs the created CG images to the broadcast image synthesis unit 113.

[0031] Note that while both broadcast footage and CG footage contain footage of athletes competing, they are images for different purposes as described above. Therefore, compared to broadcast footage, CG footage is captured in a manner (camera position, angle of view, direction, etc.) that is more suitable for analyzing athletic movements. For example, in analyzing athletic movements, it is desirable to capture images of the athletes' entire bodies with high visibility at all times, so in this embodiment, at least multiple CG cameras 20 are used to capture images of the athletes competing.

[0032] 2 is a flowchart showing an example of a process flow for processing a broadcast video by the video processing system 1 of the embodiment. First, in the video processing system 1, the plurality of first video input units 111 input a plurality of broadcast videos from the plurality of broadcast cameras 10, and the plurality of second video input units 121 input a plurality of CG videos from the plurality of CG cameras 20 (S101).

[0033] Next, each of the plurality of image recognition units 122 recognizes the action attributes of the athletic actions performed by the athlete based on the CG image input by the corresponding second image input unit 121 (S102).

[0034] Next, the integrated recognition unit 123 integrates the individual recognition results of the action attributes recognized by the multiple video recognition units 122 in S102 into one integrated recognition result (S103). Meanwhile, the competition information acquisition unit 124 acquires competition information from the competition information server 30 (S104).

[0035] Next, the statistical processing unit 125 performs the first statistical processing and the second statistical processing based on the integrated recognition result of the movement attributes obtained in S103, thereby obtaining the final recognition result for each athletic movement (S105).

[0036] Next, the CG creation unit 126 creates CG images to superimpose the final recognition results obtained in S105 and the competition information obtained in S104 on the broadcast image and outputs them to the broadcast image synthesis unit 113 (S106).

[0037] Next, the broadcast video synthesis unit 113 synthesizes the CG video input from the CG creation unit 126 in S106 with the broadcast video (S107). Here, the broadcast video onto which the CG video is synthesized is output from the broadcast video switching unit 112. The video output from the broadcast video switching unit 112 is the one selected by the broadcast video switching unit 112 from the multiple broadcast videos input in S101. The broadcast video onto which the CG video is synthesized can be switched to any of the videos shot by the multiple broadcast cameras 10 under the control of the broadcast video switching unit 112.

[0038] According to the video processing system 1 of the embodiment configured as described above, information indicating the movement attributes of the athletes' movements during the competition is added to the broadcast video, allowing viewers to more easily understand the situation of the competition. This makes it easier to convey the excitement of the competition to viewers who are not familiar with the competition. Furthermore, according to the video processing system 1 of the embodiment, it is possible to provide viewers with more information in synchronization with the timing of the competition, thereby enriching the experience of watching the competition and providing viewers with a more value-added viewing experience.

[0039] Hereinafter, the configuration of each functional unit of the video processing system 1 of the embodiment will be described in more detail, taking the case where the target sport is boxing as an example.

[0040] [Image Recognition Section] 3 and 4 are diagrams illustrating the flow of processing by the video recognition unit 122 to recognize the action attributes of players during a competition. First, the video recognition unit 122 performs a player position detection process to detect the position of a player for each frame of CG video (S201). Any object detection algorithm may be used in the player position detection process as long as it can detect a human body from an image.

[0041] FIG. 3 shows an example in which the detected positions of both fighters do not overlap, and FIG. 4 shows an example in which the detected positions of both fighters overlap. In the case of boxing, fighters move around the ring, so when viewed from a single camera, the fighters may or may not appear to overlap depending on the timing. When recognizing the movement attributes of fighters from an image, it is easier to recognize the movement attributes in an image in which the fighters do not overlap. For this reason, in this embodiment, multiple CG cameras 20 are used to capture CG images.

[0042] Next, the video recognition unit 122 performs a player image cropping process (S202) to crop an image (player image) containing the entirety of each player from the frame based on the position of each player detected in S201. While the frame may be cropped to the player's size, when motion attributes are estimated using a trained model based on machine learning (details will be described later), the player's motion attributes may be affected by the player's surroundings (e.g., the state of opposing players). Therefore, to improve the accuracy of motion attribute recognition, the player image may be cropped with the player's body in the foreground and a certain amount of white space (see, for example, Figures 3 and 4). As an example, the player image may be cropped to a size that includes a 20% margin relative to the player's size (1.2 times the player size). In this case, it is recommended that player images cropped to a size 1.2 times the player size be used during inference, as in learning. To further improve recognition accuracy, the cropped player image during inference may be enlarged or reduced as needed to match the input size of the trained model.

[0043] Next, the video recognition unit 122 performs a motion attribute recognition process (S203) to recognize motion attributes of the athletic motions of each athlete based on the athlete image of each athlete extracted in S202. In the case of boxing, a typical athletic motion is a punching motion. Here, the video recognition unit 122 recognizes attributes related to punching motions (punching attributes) as motion attributes based on the athlete image.

[0044] FIG. 5 is a diagram showing an example of punch attributes. For example, examples of punch attributes include the items shown in the recognition classes in FIG. 5. For each recognition class, subclasses indicating possible states are defined, and the video recognition unit 122 calculates a recognition score for each subclass for each frame. The recognition score is a value indicating the likelihood that a target subclass will be the recognition result of the corresponding recognition class. For example, in FIG. 5, the recognition score for the "blue corner" is 90 points, and the recognition score for the "red corner" is 5 points. This suggests that the "player who punched" is more likely to be a player in the "blue corner" than a player in the "red corner."

[0045] Here, the recognition score may be calculated based on one frame or multiple frames. For example, the video recognition unit 122 may be configured to input a time series of player images into a trained model (hereinafter referred to as a "punch determination model") that has trained the relationship between the recognition score of each sub-class and the player images (time series), and estimate the recognition score of each sub-class as its output.

[0046] Figure 6 is a diagram illustrating a method for training a punch detection model. First, to create training data, the images collected for training are converted to grayscale (S301). While grayscale conversion is not essential, it is recommended to do so because it can improve processing speed, reduce the amount of calculation, reduce noise, and improve the sensitivity of edge detection.

[0047] Next, for a frame Ft (hereinafter referred to as the "target frame") for which a recognition score is to be estimated, two reference frames Ft-n and Ft+m, which were captured before and after the target frame Ft, are set (S302). For example, Fig. 6 shows an example in which, when the target frame Ft is the t-th frame in a time-series frame group, three frames are set: the tn-th past frame Ft-n and the t+m-th future frame Ft+m. That is, n is a constant used when referencing a past frame from the target frame Ft (t-th), and m is a constant used when referencing a future frame from the target frame Ft (t-th).

[0048] Next, a label indicating the subclass recognition score is assigned to each of the frames set in S302 (S303). A punch determination model can be generated by performing machine learning using the labeled frames as training data. For example, a deep learning model such as a convolutional neural network or a machine learning algorithm such as reinforcement learning can be used to train the punch determination model.

[0049] In this way, by creating training data using a set of a target frame and the frames before and after it, it is possible to determine whether or not a punch is being made based on the flow of movements before and after the target frame, thereby improving the accuracy of punch recognition. In particular, in the case of boxing, a single punch is completed in an extremely short time, so a frame suitable for judgment is not necessarily obtained as the target frame. Therefore, in sports such as boxing, where a single athletic movement is completed in a short time, athletic movements can be recognized with high accuracy by learning a set of frames before and after the target frame. Note that n and m should be set to appropriate values ​​depending on the athletic movement being targeted.

[0050] [Integrated Recognition Section] 7 is a diagram illustrating the process in which the integrated recognition unit 123 acquires an integrated recognition result by integrating multiple individual recognition results for punch attributes acquired by multiple image recognition units 122. First, the integrated recognition unit 123 determines a weight value for the recognition score for each CG camera 20 that captured the CG images. Specifically, the integrated recognition unit 123 determines a weight value according to the degree of overlap of the player positions detected in each CG image (S401).

[0051] Figure 7 compares the detection results of each player's position in the examples of Figure 3 and Figure 4. In Figure 7, the example of Figure 3 corresponds to the first CG camera 20, and the example of Figure 4 corresponds to the second CG camera 20. In this example, the positions of each player detected from the CG video captured by the first CG camera 20 do not overlap, whereas the positions of each player detected from the CG video captured by the second CG camera 20 do overlap.

[0052] As mentioned above, it is thought that the smaller the overlap of the player positions, the higher the accuracy of punch recognition, and the greater the overlap, the lower the accuracy of punch recognition. In other words, it is thought that the reliability of punch attributes recognized from CG video is higher the smaller the overlap of the player positions, and lower the greater the overlap.

[0053] Therefore, the greater the overlap between the detected positions of both players, the higher the weighting value the integrated recognition unit 123 assigns to the corresponding CG camera 20, and the smaller the overlap, the lower the weighting value the integrated recognition unit 123 assigns to the corresponding CG camera 20.

[0054] Next, the integrated recognition unit 123 calculates a weighted average of the recognition scores by applying the weight values ​​determined for each CG camera 20 to the recognition scores of each subclass (S402). The example in FIG. 7 shows an example in which a weighted average is calculated for the subclasses "Blue Corner" and "Red Corner" of the recognition class "Punching Player." By integrating the recognition scores using such a weighted average, it is possible to prioritize highly reliable recognition results depending on the positional relationship between the players at each point in time. Therefore, with this configuration, it is possible to always obtain CG images with high recognition accuracy using one of the multiple CG cameras 20, and to prioritize highly reliable recognition results to obtain final recognition results.

[0055] [Statistical Processing Unit] FIG. 8 is a diagram illustrating the first statistical processing performed by the statistical processing unit 125. For example, the statistical processing unit 122 smooths the recognition scores in the time direction by taking a moving average in the time direction of the recognition scores integrated by the integrated recognition unit 123. FIG. 8 shows an example in which a moving average value St_ma is calculated for the target frame score St using the recognition scores of w past frames. The statistical processing unit 125 calculates the moving average value St_ma for the recognition score St of the target frame at each time point and sets the calculated moving average value St_ma as the score St of the target frame after smoothing, thereby smoothing the integrated recognition result of the punch attribute in the time direction. The statistical processing unit 125 performs a second statistical processing to make a final determination about the punch action based on the recognition scores after the first statistical processing.

[0056] FIG. 9 is a diagram illustrating the segment recognition process in the second statistical process and the label determination process that assigns a final punch attribute value (hereinafter referred to as a "label") to the recognized segment. More specifically, the second statistical process is a process that determines segments that represent individual punching motions based on the recognition score after the first statistical process. A segment is a set of frames for a given punching motion, from a start frame that represents the start of the punching motion to an end frame that represents the end of the punching motion. For example, the statistical processing unit 125 can recognize as the start frame or the end frame a frame at which the magnitude relationship between the recognition score of the recognition class and a threshold value changes.

[0057] The statistical processing unit 125 determines a label (subclass) to be assigned to each recognition class for each recognized segment. For example, the statistical processing unit 125 can add up the recognition scores from the start frame to the end frame for each subclass belonging to the same recognition class, and use the subclass with the maximum value as the label.

[0058] The example in Figure 9 shows a case where, for the punching motion represented by the segment (segment ID: S01) from the start frame "F003" to the end frame "F005," the subclass "blue corner" is determined as the label for the recognition class "punching player," the subclass "left" is determined as the label for the recognition class "punching arm," and the subclass "not hit" is determined as the label for the recognition class "whether hit or not."

[0059] Furthermore, the example in Figure 9 shows a case where, for the punching action represented by the segment (segment ID: SG02) from the start frame "F004" to the end frame "F006", the subclass "red corner" is determined as the label for the recognition class "punching player", the subclass "right" is determined as the label for the recognition class "punching arm", and the subclass "hit" is determined as the label for the recognition class "whether it hit or not".

[0060] The statistical processing unit 125 acquires, as statistical information, the number of punches performed within a period corresponding to various display purposes, based on information about each punch segment. For example, the statistical processing unit 125 may tally the number of punches performed during a round, or the number of punches performed throughout the entire match. Furthermore, the statistical processing unit 125 may tally the number of punches having a specific attribute based on information about the label determined for each segment. For example, the statistical processing unit 125 may tally the number of punches that simply landed, the number of punches that landed cleanly, the number of punches that landed effectively, or the number of punches that hit a specific location, for each round or throughout the entire match.

[0061] Furthermore, for example, the statistical processing unit 125 may tally the number of consecutive punches by analyzing the continuity of punches based on information about the segments of each punch. For example, the statistical processing unit 125 may recognize the number of consecutive punches (combos) that satisfy a predetermined condition by performing a combo determination on consecutive punches. For example, the statistical processing unit 125 may increase the combo count by 1 if a punch performed by the same fighter occurs within a predetermined time from the previous punch. The statistical processing unit 125 may acquire information about the number of combos recognized in this way as statistical information. Note that, in the combo determination, the statistical processing unit 125 may determine, as a combo, punches that satisfy a condition related to the label of the punch (segment) (for example, whether it hits) in addition to the condition of temporal continuity.

[0062] Here, a round or an entire match is an example of a period based on the rules of the sport (boxing in this case). In this way, the statistical processing unit 125 counts the number of times a sport action corresponding to various purposes in order to display different information depending on the period based on the rules of the sport and the attributes of the sport action (punching in this case).

[0063] [CG Creation Department] 10 and 11 are diagrams showing examples of displaying broadcast video onto which CG created by the CG creation unit 126 has been superimposed. In FIG. 10, CG images G11 to G16 are examples of CG video superimposed on broadcast video. CG image G11 is a CG image that displays match information such as player names, round information, and elapsed time. CG image G11 can be created based on competition information provided by the competition information server 30, for example.

[0064] Furthermore, CG image G12 represents the total number of punches thrown by each fighter. CG image G13 represents the total number of punches thrown by each fighter that hit the opponent. CG image G14 represents the total number of punches that were clean hits among the punches that hit the opponent. CG image G15 is a graphic that represents the ratio of punches that hit to the total number of punches thrown. This information can be created by counting segment information that meets specified conditions.

[0065] The CG image G16 represents the number of combos currently occurring. The CG image G16 can be created based on statistical information acquired by the statistical processing unit 125. In addition to these static CG images, the broadcast image may be synthesized with CG images that add visual effects according to the punching motions of the fighters.

[0066] FIG. 11 shows an example of CG video G20 inserted into a broadcast video. For example, the CG video G20 may be inserted during intervals between rounds. The CG video G20 includes, for example, CG video G21 showing the percentage of punches that landed, CG video G22 showing the number of punches that landed, CG video G23 showing the number of clean punches, CG video G24 displaying match information such as round information and fighter names, CG video G25 illustrating the timing of punches landing, and CG video G26 illustrating the location of patch hits. All of these CG images can be created based on segment information and statistical information generated from the CG video.

[0067] According to the video processing system 1 of the embodiment configured as described above, information indicating the movement attributes of the athletes' movements during the competition is added to the broadcast video, allowing viewers to more easily understand the situation of the competition. This makes it easier to convey the excitement of the competition to viewers who are not familiar with the competition. Furthermore, according to the video processing system 1 of the embodiment, it is possible to provide viewers with more information in synchronization with the timing of the competition, thereby enriching the experience of watching the competition and providing viewers with a more value-added viewing experience.

[0068] In the above embodiment, a case has been described in which CG video is synthesized with video of a competition currently in progress (so-called live video), but the video processing method of the embodiment is not limited to processing live video. The video processing system 1 of the embodiment may be configured to input recorded video for broadcast and recorded video for CG to the first video input unit 111 and the second video input unit 121 by replacing the broadcast camera 10 with a broadcast playback device that plays back recorded video for broadcast and the CG camera 20 with a CG playback device that plays back recorded video for CG. Even in such a case, the video processing method of the embodiment can process video in a similar manner.

[0069] Furthermore, in the above embodiment, the case where the CG image and the broadcast image are different has been described, but in the image processing method of the embodiment, the CG image and the broadcast image do not necessarily have to be different. For example, if the broadcast images captured by the multiple broadcast cameras 10 are captured at an angle of view that is also suitable for the CG image, the broadcast image can be branched and input to the second image input unit 121, and the broadcast image can be used as the CG image.

[0070] The above describes the form for carrying out the present invention using an embodiment, but the present invention is not limited to such an embodiment, and various modifications and substitutions can be made within the scope that does not deviate from the gist of the present invention. [Explanation of symbols]

[0071] 1. Video processing system 10 Broadcast Camera 12 Statistical Processing Unit 20 CG camera 30 Competition Information Server 111 First video input unit 112 Broadcast video switching unit 113 Broadcast video synthesis unit 121 Second video input section 122 Image Recognition Unit 123 Integrated Recognition Department 124 Competition Information Acquisition Department 125 Statistical Processing Unit 126 CG Creation Department

Claims

1. A video processing system for processing video of a competition, an input unit for inputting a plurality of camera images of athletes participating in the competition taken by a plurality of cameras; a first recognition unit that calculates, for each of the plurality of camera images input by the input unit, a score relating to the likelihood of a plurality of recognition results of attributes recognized from the camera images regarding attributes related to athletic movements performed by the athlete while performing the athletics; a second recognition unit that integrates the plurality of recognition results obtained by the first recognition unit for the attributes of each of the plurality of camera images based on the score, and recognizes the athletic action based on the integrated recognition result; a generation unit that generates a second image for displaying information based on the athletic action recognition result in synchronization with the timing of the athletic action; a synthesis unit that synthesizes the second image with the image of the competition; Equipped with the second recognition unit weights the scores for the camera images based on priorities of the camera images according to the degree of overlap of the positions of the players in the camera images, and integrates the recognition results by summing the weighted scores. Video processing system.

2. the first recognition unit recognizes the plurality of attributes for each frame of the camera image, and calculates a score correlating with likelihood for a plurality of possible value options for each of the plurality of attributes; the second recognition unit integrates the plurality of recognition results based on the score; The video processing system according to claim 1 .

3. the second recognition unit weights the scores of the plurality of recognition results based on priorities corresponding to the corresponding cameras among the plurality of cameras, and integrates the plurality of recognition results by summing the weighted scores. The video processing system according to claim 1 .

4. the first recognition unit detects the positions of a plurality of players from the plurality of camera images; the second recognition unit determines priorities for the cameras according to the degree of overlap of the positions of the players; The video processing system according to claim 3 .

5. a statistical processing unit that performs statistical processing on the integrated recognition results, the statistical processing unit smooths the integrated recognition results in a time series direction, and determines the start timing and end timing of the athletic action based on the smoothed attribute recognition results. The video processing system according to claim 2 .

6. the statistical processing unit sums up the integrated scores for each of the plurality of options for frames included in the start timing to the end timing of the athletic action, and determines the option with the largest total value as the attribute of the athletic action. The video processing system according to claim 5 .

7. The statistical processing unit counts the number of times the athletic movements are performed within a period according to a display purpose, the generation unit generates, as the second image, an image that displays the number of times of the athletic movements tallied by the statistical processing unit in a manner according to the display purpose. The video processing system according to claim 5 .

8. The period is based on the rules of the competition. The video processing system according to claim 7.

9. The display purpose is a purpose that differs depending on the attribute of the athletic action. The video processing system according to claim 7.

10. the statistical processing unit determines a condition regarding the temporal continuity between the plurality of athletic movements based on the start timings and end timings of the plurality of athletic movements, and counts the number of times a series of athletic movements satisfy the condition; The generation unit generates an image displaying the number of times of the series of athletic movements as the second image. The video processing system according to claim 7.

11. A video processing method for processing video of a competition, comprising: The computer an input step of inputting a plurality of camera images of athletes playing the game taken by a plurality of cameras; a first recognition step of calculating, for each of the plurality of input camera images, a score relating to the likelihood of a plurality of recognition results of attributes recognized from the camera images regarding attributes related to athletic movements performed by the athlete during the execution of the athletic competition; a second recognition step of integrating the plurality of recognition results obtained by recognizing the attributes for each of the plurality of camera images based on the score, and recognizing the athletic action based on the integrated recognition result; a generating step of generating a second image for displaying information based on the athletic action recognition result in synchronization with the timing of the athletic action; a combining step of combining the second image with the image of the competition; and in the second recognition step, weighting the scores for the camera images based on priorities of the camera images according to degrees of overlap of positions of the players in the camera images, and integrating the recognition results by summing the weighted scores; Video processing methods.

12. In a video processing system that processes video of competitions, an input step of inputting a plurality of camera images of athletes playing the game taken by a plurality of cameras; a first recognition step of calculating, for each of the plurality of input camera images, a score relating to the likelihood of a plurality of recognition results of attributes recognized from the camera images regarding attributes related to athletic movements performed by the athlete during the execution of the athletic competition; a second recognition step of integrating the plurality of recognition results obtained by recognizing the attributes for each of the plurality of camera images based on the score, and recognizing the athletic action based on the integrated recognition result; a generating step of generating a second image for displaying information based on the athletic action recognition result in synchronization with the timing of the athletic action; a combining step of combining the second image with the image of the competition; A program for causing a computer to execute the above, in the second recognition step, weighting the scores for the camera images based on priorities of the camera images according to degrees of overlap of positions of the players in the camera images, and integrating the recognition results by summing the weighted scores; program.

Images