A method for extracting shots from team sports video games based on color features

Abstract

A method for extracting game shots from team sports videos based on color features belongs to the field of video data processing. First, the video is preprocessed, and the video is automatically segmented and structured in units of shots; then the arithmetic means of the r, g, and b channels of the video frames and the shot video segments are calculated, and then the reference shot video segments are selected by K-means clustering and based on the extreme difference of the arithmetic means of the r, g, and b channels of the video frames in the shot video segments, and the team sports video is divided into game shot (long shot, medium shot) video segments and other shot (close-up shot, off-field shot) video segments by comparing the main color difference between the reference shot video segments and each shot video segment. The present invention utilizes the main color difference of the video frame to extract the long shot and medium shot video segments that occupy most of the content of the team sports video, which is conducive to reducing the adverse effects of redundant content on video analysis and processing, and lays a good structural foundation for further research.

Description

Technical Field

[0001] The invention belongs to the field of video data processing. Background Art

[0002] Team sports such as football and basketball are popular among people and have a broad mass base. Therefore, the research on team sports videos has broad application prospects. Video data is a kind of unstructured data. Converting it into a series of labeled sequences with shot type as the key attribute is the basis for event detection and summary generation. The usual practice is to preprocess the video first, automatically segment the video by shot and structure it; on this basis, the type of shot is automatically identified by pattern recognition and other methods. In team sports videos, according to the shooting angle and distance, the shots in the video can be divided into four types: long shots, medium shots, close-up shots, and off-field shots. The long shot shows the position of each player on the court and the progress of the game from a bird's-eye view, while the medium shot shows the game situation of most players on the court more clearly; the close-up shot is often a playback of the long shot and the medium shot; the off-field shots such as the audience, advertisements and logos are redundant content relative to the game itself. In summary, the long shots and medium shots, which usually occupy most of the content of the game, are the most important game shots for the game itself. Therefore, extracting game footage from sports videos can reduce the adverse effects of redundant content on video analysis and processing.

[0003] Eldib et al. determined the RGB range of field pixels through experiments. All pixels whose components fall within this range are considered field color pixels. Then, thresholds are set according to the field pixel ratio to distinguish different types of shots. Yu Junqing et al. used the field color ratio of the sub-window area in the video frame to reflect the type of shot, which also achieved certain results.

[0004] Shot classification must first be done by video segmentation. Accurately detecting shot boundaries to complete video segmentation is a prerequisite for effectively classifying shot types. In team sports videos, shot boundaries can be divided into mutation type and gradual type. In existing related work, segmented shots are still prone to false detection, especially for the detection of gradual shots. The main reason is that there are many slow-motion playbacks in team sports videos. At the beginning and end of this part of the playback, there are often LOGO scans, which can easily lead to incorrect judgments by the algorithm, which is very unfavorable for subsequent research work. The existing shot classification algorithm cannot compensate for the false detection of shot boundary segmentation. Instead, it will be affected by it, resulting in a decrease in the accuracy of shot classification.

[0005] Secondly, the algorithm proposed by Eldib et al. is based on the color of the venue, but the venue will change with factors such as the venue, weather, and light. The algorithm proposed by Yu Junqing et al. relies on the selection of the size and position of the sub-window, so the algorithm is not robust.

[0006] In summary, the recognition accuracy of various types of shots in the current shot classification method is significantly affected by the insufficient accuracy of shot boundary detection, resulting in an unsatisfactory shot recognition accuracy and weak versatility of the existing shot classification algorithm. Summary of the invention

[0007] The purpose of the present invention is to propose a method for extracting game shots from team sports videos based on color features, and to extract long shots and medium shots that occupy most of the content of team sports videos for subsequent analysis and research of the videos. Based on the large differences in the main colors of the entire video frame in different types of shots, team sports videos are divided into game shots (long shots, medium shots) video clips and other shots (close-up shots, off-field shots) video clips. Experiments have shown that the method has good adaptability, high accuracy, and is practical and efficient.

[0008] Input: Original team sports video

[0009] Output: A collection of video clips of the game footage

[0010] definition:

[0011] S={S 1 ,S 2 ,……,S S}: A collection of shot video clips after video segmentation;

[0012] F: a set of video frames of a shot video clip;

[0013] AM R : The arithmetic mean of the R channel values ​​corresponding to all pixels in the video frame;

[0014] AM G : The arithmetic mean of the G channel values ​​corresponding to all pixels in the video frame;

[0015] AM B : The arithmetic mean of the B channel values ​​corresponding to all pixels in the video frame;

[0016] R R : AM in the video clip R Very bad;

[0017] R G : AM in the video clip G Very bad;

[0018] R B : AM in the video clip B Very bad;

[0019] All video frames in the lens video clip correspond to AM R The arithmetic mean of

[0020] All video frames in the lens video clip correspond to AM G The arithmetic mean of

[0021] All video frames in the lens video clip correspond to AM B The arithmetic mean of

[0022] A RGB :Will Arrays stacked in row order;

[0023] C max : The category with the most video clips after K-means clustering;

[0024] TH BS : The threshold used to distinguish between the candidate baseline shots and all other shots;

[0025] CBS: a set of candidate benchmark shot video clips;

[0026] S FM : The video clip with the most video frames in CBS;

[0027] Shot video clip S FM Corresponding

[0028] Shot video clip S FM Corresponding

[0029] Shot video clip S FM Corresponding

[0030] and The percentage error of

[0031] and The percentage error of

[0032] and The percentage error of

[0033] TH GS : The threshold used to distinguish between game shots and non-game shots;

[0034] S': A collection of video clips of match shots.

[0035] Step 1: Split the video into video clips and save the video frame by frame in the form of pictures to obtain S

[0036] Step 2: Calculate AM for all video frames R ,AM G ,AM B

[0037] Step 3: Calculate R for all shot video clips R ,R G ,R B； and A RGB

[0038] Step 4: A corresponding to all lens video clips RGB Perform K-means clustering and find C max

[0039] Step 5: C max Perform the following operations on all the video clips in the video: If R R, R G, R B At least one of them is less than or equal to TH BS , then the video clip of the shot is stored in CBS

[0040] Step 6: Find

[0041] Step 7: Calculate the video clips of all shots And do the following for all lens video clips: If All less than or equal to TH GS , then store it in S'.

[0042] Value: TH BS =100,TH GS =15%.

[0043] Using the mean attribute in the ImageStat module of the Python image processing library PIL, the average value AM of the three channel pixel values ​​of each frame in a single-shot video clip can be calculated. R ,AM G ,AM B .

[0044] The formula is as follows:

[0045]

[0046] Where r, g, and b are the RGB color pixel values ​​corresponding to the pixels in the video frame, and n is the number of pixels.

[0047]

[0048] Where N is the number of video frames contained in a single-shot video clip.

[0049]

[0050] The present invention uses color features to extract game shots from team sports videos, and uses the main color difference of the entire video frame to eliminate close-ups and off-field shots, while retaining the main game shots in the video. In existing related work, the classification of shots often focuses on the venue itself. This method is not limited to this conventional idea and proposes a new idea. First of all, this method is not limited by venue factors and can be applied to a variety of team sports. At the same time, this method can also make up for some false detections in the shot boundary detection method: if there is an erroneous segmentation in the non-game shot, the entire shot will also be identified and removed, and will not affect the subsequent video research and analysis. Therefore, this method can extract the most important game shot part in the entire team sports video, laying a good structural foundation for subsequent semantic event detection and summary generation.

[0051] To test the effect of this method, 25 videos with a total of 112 minutes were extracted from the public Youtube-8M dataset to form the following Figure 2 The SportKF dataset shown is from four sports (basketball, rugby, American football, and hockey), containing 197,878 frames and 572 shots.

[0052] This method is applied to the above dataset, and the recall and precision of the five football videos are compared with the method proposed by Yu Junqing et al. The calculation formulas of recall and precision are as follows:

[0053] Recall rate = (actual number of inspections - number of false inspections) / number of inspections required

[0054] Precision = (actual detection number - false detection number) / actual detection number

[0055] For the extracted game footage, the recall rate of the four team sports is 100% and the precision rate is 86.67%. For football videos, the recall rate of the method proposed by Yu Junqing et al. is 93.2%, while the recall rate of this method reaches 100%; the precision rate of the method proposed by Yu Junqing et al. is 88.7%, while the recall rate of this method reaches 93.4%.

[0056] Experimental results show that this method achieves good results in the detection and recognition of competition shots. Compared with the more representative method proposed by Yu Junqing et al., it has improved both the recall rate and the precision rate. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 flow chart

[0058] Figure 2 SportKF dataset video cover

[0059] Figure 3 Telephoto

[0060] Figure 4 LOGO lens DETAILED DESCRIPTION

[0061] The present invention proposes a method for extracting team sports video game shots based on color features. The specific implementation steps of the invention are as follows:

[0062] Take a 7-minute NBA game video as an example. The frame width is 640, the frame height is 360, and the frame rate is 30 frames per second. Figure 3 As shown, the LOGO lens is as follows Figure 4 As shown (the following specific values ​​are rounded to 2 decimal places).

[0063] Step 1: Split the video into video shot segments, and obtain S = {S 1 ,S 2 ,……,S 33}. (Contains 33 shots in total)

[0064] Step 2: Calculate AM for all video frames R ,AM G ,AM B .like Figure 3 Corresponding AM R =100.50,AM G =80.34,AM B =74.88; AM corresponding to close-up shots R =93.04,AM G =76.01,AM B =71.23; Figure 4 Corresponding AM R =130.07,AM G =112.26,AM B =79.29.

[0065] Step 3: Calculate R for all shot video clips R ,R G ,R B； and A RGB

[0066] Figure 3 The lens (serial number is 17) corresponds to R R =37.22,RG =35.56,R B =40.65; A RGB =(110.05,90.39,79.10)

[0067] The R corresponding to the close-up shot (serial number 3) R =40.13,R G =28.46,R B =19.81; A RGB =(96.27,77.20,65.88)

[0068] Figure 4 The lens (serial number is 12) corresponds to R R =13.84,R G =12.63,R B =1.80; A RGB =(105.70,90.43,74.45)

[0069] Step 4: A corresponding to all lens video clips RGB Perform K-means clustering and find C max In this example, C max Contains 9 lenses: 4, 6, 9, 15, 17, 20, 21, 27, 31

[0070] Step 5: C max Perform the following operations on all the video clips in the video: If R R, R G, R B At least one of them is less than or equal to TH BS , then the video clip of the shot is stored in CBS

[0071] In this case, TH BS Set to 100, visible C max The lens with serial number 17 in the image has the corresponding R R =40.13,R G =28.46,R B =19.81 satisfies "If R R, R G, R B At least one of them is less than or equal to TH BS ", so it is stored in CBS; on the contrary, C max The lens with serial number 20 in the image has the corresponding R R =212.30,RG =216.18,R B =196.36 does not satisfy the condition "If R R, R G, R B At least one of them is less than or equal to TH BS ", so it is not stored in CBS; in this example, the result CBS of step 5 includes 8 shots, 4, 6, 9, 15, 17, 21, 27, and 31.

[0072] Step 6: Find

[0073] In this example, among the 8 shots included in CBS, the shot with the most frames is shot number 31, which contains 1895 frames.

[0074] Step 7: Calculate the video clips of all shots And do the following for all lens video clips: If All less than or equal to TH GS , then store it in S'

[0075] In this case, TH GS Set to 15%, such as Figure 3 The lens (serial number is 17), corresponding to satisfy" All less than or equal to TH GS ", so it is stored in S'; on the contrary, if the close-up shot (serial number is 3), the corresponding Not satisfied with " All less than or equal to TH GS ", so it is not stored in S'; in this example, the final result game shot video clip set S' contains 11 shots with serial numbers 2, 4, 6, 9, 13, 15, 17, 21, 25, 27, 31, all of which are game shots, and there is no missed detection.

Claims

1. A method for extracting team sports video game shots based on color features, characterized in that: S={S1,S2,……,S S }: A collection of shot video clips after video segmentation; AM R : The arithmetic mean of the R channel values ​​corresponding to all pixels in the video frame; AM G : The arithmetic mean of the G channel values ​​corresponding to all pixels in the video frame; AM B : The arithmetic mean of the B channel values ​​corresponding to all pixels in the video frame; R R : AM in the video clip R Very bad; R G : AM in the video clip G Very bad; R B : AM in the video clip B Very bad; All video frames in the lens video clip correspond to AM R The arithmetic mean of All video frames in the lens video clip correspond to AM G The arithmetic mean of All video frames in the lens video clip correspond to AM B The arithmetic mean of A RGB :Will Arrays stacked in row order; C max : The category with the most video clips after K-means clustering; TH BS : The threshold used to distinguish between the candidate baseline shots and all other shots; CBS: a set of candidate benchmark shot video clips; S FM : The video clip with the most video frames in CBS; Shot video clip S FM Corresponding Shot video clip S FM Corresponding Shot video clip S FM Corresponding and The percentage error of and The percentage error of and The percentage error of TH GS : The threshold used to distinguish between game shots and non-game shots; S': a collection of video clips of match shots; Step 1: Split the video into video footage segments and save the video frame by frame in the form of pictures to obtain S; Step 2: Calculate AM for all video frames R ,AM G ,AM B ; Step 3: Calculate R for all shot video clips R ,R G ,R B； and A RGB ; Step 4: A corresponding to all lens video clips RGB Perform K-means clustering and find C max ; Step 5: C max Perform the following operations on all the video clips in the video: If R R, R G, R B At least one of them is less than or equal to TH BS , then the video clip of the shot is stored in CBS; Step 6: Find Step 7: Calculate the video clips of all shots And do the following for all lens video clips: If All less than or equal to TH GS , then store it in S'; Value: TH BS =100,TH GS =15%.

Images