A method for athlete recognition based on multi-object tracking

Abstract

The present invention discloses an athlete recognition method based on multi-object tracking, belonging to the field of artificial intelligence. First, multi-object tracking of athletes is performed on the competition video to obtain the bounding box of each athlete and its unique tracking identifier in each frame of the picture. Then, for the bounding box of each athlete in a certain frame of the picture, the jersey number digits contained in the bounding box are recognized, sorted according to the confidence level, and the top K digits are selected and sorted according to the abscissa of the bounding box to obtain the K-digit jersey number in decimal system of the athlete, and a mapping is formed with the unique tracking identifier. At the same time, the color features contained in the bounding box of each athlete are extracted to determine the team to which each athlete belongs. Finally, according to the jersey number corresponding to the unique tracking identifier and the team to which it belongs, the uniquely determined athlete information is retrieved. Select the next frame of the picture and repeat the above process until the processing of each frame of the competition video is completed. The present invention has accurate athlete recognition, high robustness, and wide application scenarios.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence, and relates to the recognition of athletes in video images. Specifically, it is an athlete recognition method based on multi-object tracking. Background Art

[0002] In recent years, with the rapid development of artificial intelligence, various video analysis and interpretation technologies based on computer vision have become increasingly mature, and the improvement of various hardware computing powers has provided conditions for the application and implementation of these technologies. In September 2019, the "Opinions on Promoting National Fitness and Sports Consumption and Promoting the High-quality Development of the Sports Industry" further pointed out that it is necessary to promote the application of emerging technologies such as intelligent manufacturing, big data, and artificial intelligence in the field of sports manufacturing, and vigorously develop "Internet + sports". As Chinese athletes continue to achieve better results on the international stage, various sports have received unprecedented attention in China. The integration of emerging technologies and sports is the future development direction of sports, and how to make machines understand sports videos is an important part of sports video analysis.

[0003] In the field of sports video analysis, manual means are still heavily relied on to label athletes and competition events in sports videos. The manual labeling process is cumbersome and time-consuming. Therefore, how to automatically analyze sports videos has become a hot topic of current research. Since athlete recognition is one of the links connecting video images with reality, it has become the focus of research by scholars at home and abroad.

[0004] However, there are still some challenges in athlete recognition in sports videos, such as athletes wearing similar appearances, athletes blocking each other, large and unpredictable action changes, etc. Researchers at home and abroad have made many explorations in response to these challenges. Currently, the methods of athlete recognition can be mainly divided into 4 categories: face recognition, jersey number recognition, player tactical position recognition, and player re-identification. Face recognition is applicable to the video images where athletes' faces are exposed, and it is difficult to effectively solve the problem of images without seeing faces; due to the different numbers of athletes and tactics in different sports, it is difficult to apply player tactical position recognition to different sports; in sports, the clothing and appearances of athletes are highly similar, so there are also great challenges in player re-identification. Due to the increasing quality and clarity of current sports videos, jersey numbers, as a common sports identifier in sports, are widely used in athlete recognition.

[0005] In sports videos, the camera angles and athletes' body postures are very changeable, and athletes block each other. As a result, there will be a large number of situations where athletes' jersey numbers are blocked, not fully displayed, or even not visible in the video. These situations lead to a sharp decline in the accuracy and effectiveness of jersey numbers, and thus it is difficult to effectively achieve athlete recognition. Summary of the Invention

[0006] The present invention provides a method for athlete recognition by comprehensively applying multi-object tracking, stadium personnel classification, and jersey number recognition, which can effectively solve the problems of variable camera perspectives, variable athlete body postures, and incomplete or unshown jersey numbers in sports event videos, and realizes true athlete recognition in sports videos.

[0007] The athlete recognition method based on multi-object tracking is specifically as follows:

[0008] Step 1: Construct a player information database according to the publicly available game information;

[0009] The player information database includes player names, team names, jersey numbers, jersey colors, etc.

[0010] Step 2: Input the event video of the game into the multi-object tracking model, and perform multi-object tracking on the athletes in the video frame according to the player information database to obtain the bounding box of each athlete and its unique tracking identifier in each frame;

[0011] The bounding box of a certain athlete i is bbox i , and the corresponding unique tracking identifier is ID i , and the total number of athletes is m, 0 ≤ i ≤ m.

[0012] Step 3: For the bounding box of each athlete in a certain frame, respectively take the upper half of each bounding box for detection to obtain the numbers included in each athlete's bounding box.

[0013] Detect the upper half of the athlete's bounding box bbox i to obtain the bounding box Dbbox ij of each number included therein, each number Digit ij and the confidence Dconf ij of each number.

[0014] Where n represents the number of numbers in each athlete's bounding box, n ≥ 0, 0 ≤ j ≤ n, 0 ≤ Digit ij ≤ 9, 0 ≤ Dconf ij ≤ 1.

[0015] Step 4: For the bounding box of a certain athlete, select the numbers corresponding to the top K confidences, and sort the K numbers according to the abscissa of the athlete's bounding box to obtain the K-digit decimal jersey number of the athlete.

[0016] For the confidence Dconf ijSort in descending order and select the top K with the highest confidence. Then sort them by the abscissa of the athlete's bounding box to obtain the decimal K-digit jersey number JN. i 。

[0017] K is determined by the number of digits of the jersey number in different game scenarios, and K ≤ n.

[0018] Step 5: Use the unique tracking identifier of the athlete as the key and the jersey number and its confidence as the value to establish a hash table, mapping the unique tracking identifier to the actual jersey number.

[0019] The confidence of the jersey number is the average of the confidences of the K digits.

[0020] This hash table is a one-to-many mapping, that is, one ID i may correspond to multiple JNs j (0 ≤ j).

[0021] Select the jersey number corresponding to the maximum confidence of the jersey number as the ID i corresponding to the jersey number JN, thus mapping the virtual ID being tracked i to the actual jersey number JN i 。

[0022] Step 6: Meanwhile, extract the color features contained in each athlete's bounding box in each frame of the video, distinguish different teams, and determine the team to which each athlete belongs according to the jersey color.

[0023] Step 7: Retrieve from the constructed player information database according to the jersey number and the team to which the athlete corresponding to the unique tracking identifier belongs to obtain the uniquely determined athlete name and athlete information.

[0024] Step 8: Repeat Steps 3 to 7 until the processing of each frame of the said game video is completed.

[0025] The present invention has the following advantages:

[0026] (1) The present invention comprehensively applies the multi-object tracking algorithm and the jersey number recognition algorithm to establish a stable and accurate mapping between the tracking ID and the jersey number during the video processing, realizing that even when the athlete's jersey number cannot be accurately recognized due to deformation or occlusion, the athlete can still be accurately recognized through the mapping relationship.

[0027] (2) The present invention combines the jersey number recognition with the multi-object tracking algorithm, using the jersey number as the unique ID of the athlete target, and can solve the problem of discontinuous cross-shot tracking in sports videos through the mapping relationship between the tracking ID and the jersey number, realizing more robust athlete tracking.

[0028] (3) The multi - target tracking algorithm and jersey number recognition adopted by the present invention are both real - time processing algorithms, so it can achieve real - time recognition of athletes in sports videos.

[0029] (4) The present invention is a general athlete recognition method, applicable to most sports scenarios such as basketball, football, volleyball, marathon, etc. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 is a flowchart of an athlete recognition method based on multi - target tracking according to the present invention;

[0031] Figure 2 is a flowchart of the specific algorithm steps of an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0032] The present invention will be further described in detail below with reference to the drawings and embodiments.

[0033] The present invention provides an accurate and effective athlete recognition method applicable to a variety of sports scenarios, which comprehensively uses multi - target tracking, stadium personnel classification, and jersey number recognition to identify athletes, effectively solves the problems of jersey occlusion, mutilation, or even non - appearance, and realizes the real - sense athlete recognition in sports videos.

[0034] An athlete recognition method based on multi - target tracking, as Figure 1 shown, the specific steps are as follows:

[0035] Step 1: Construct a player information database according to the publicly available game information.

[0036] Since the jerseys and team information of players in different games are uncertain, it is necessary to establish a player information database in advance according to the announced game information, including information such as player names, team names, jersey numbers, and jersey colors, so as to query the recognized jersey numbers from the database in the subsequent steps.

[0037] Step 2: Input the event video of the game into the multi - target tracking model, and perform multi - target tracking on the athletes in the video frame according to the player information database, to obtain the bounding box and its tracking unique identifier of each athlete in each frame;

[0038] Perform multi - target tracking on the athletes in the input sports event video to obtain the bounding box bbox i corresponding to each of the m athletes in each frame i and the tracking unique identifier ID

[0039] Step 3: For the bounding box of each athlete in a certain frame of the video, take the upper half of the bounding box for detection to obtain the jersey number digits contained in the bounding box of each athlete.

[0040] For the m bboxes of athletes in a certain frame obtained in Step 2, traverse each bbox i , and take the upper half of the bounding box for jersey number detection. Detect the n digit bounding boxes Dbbox i in the i-th bounding box bbox ij and the digit Digit ij , as well as the confidence Dconf of each digit ij , where 0 ≤ Digit ij ≤ 9, 0 ≤ Dconf ij ≤ 1, n ≥ 0, 0 ≤ j ≤ n.

[0041] Step 4: For the bounding box of a certain athlete, select the digits corresponding to the top K confidences, and sort the K digits according to the abscissa of the athlete's bounding box to obtain the K-digit jersey number in decimal system of the athlete.

[0042] According to the n digit bounding boxes Dbbox ij obtained in Step 3, sort the confidences of each digit from high to low, and take the top K ones; K is determined according to the number of digits of the jersey number in different game scenarios, K ≤ n;

[0043] Sort the K digits according to the abscissa of the bounding box to obtain the decimal jersey number JN i .

[0044] Step 5: Use the unique tracking identifier of the athlete as the key, and the jersey number and its confidence as the value to establish a hash table to map the unique tracking identifier to the real jersey number.

[0045] Use the average value of the confidences of the K digits as the confidence of the jersey number.

[0046] Establish the mapping between the jersey number and the tracking ID i , specifically: use the ID i obtained by multi-object tracking in Step 2 as the key, and the jersey number and its confidence recognized in Step 4 as the value to establish a hash table. This hash table is a one-to-many mapping, that is, one ID i may correspond to multiple JN j (0 ≤ j), and take the one with the largest average confidence as the jersey number JN (Jersey Number) corresponding to the ID i , so as to map the virtual ID i of the tracking to the real jersey number JNi 。

[0047] Step 6: Meanwhile, extract the color features contained in the bounding box of each athlete in each frame of the video, distinguish different teams, and determine the team to which each athlete belongs according to the jersey color.

[0048] According to the bbox obtained in Step 2 i , extract the color features in the bbox, distinguish the athletes of different teams by the color features, and then determine the team T to which the player belongs according to the jersey color i 。

[0049] Step 7: According to the jersey number of the athlete corresponding to the tracking unique identifier and the team to which the athlete belongs, retrieve from the constructed player information database to obtain the uniquely determined athlete name and the athlete information.

[0050] According to the jersey number JN obtained in Step 5 i and the team T of the athlete obtained in Step 6 i , retrieve from the player information database constructed in Step 1 to obtain the uniquely determined athlete name and the corresponding other player information.

[0051] Step 8: Repeat Step 3 to Step 7 until the processing of each frame of the said game video is completed.

[0052] Figure 2 This is the flowchart of the specific algorithm steps of an embodiment of the present invention, mainly including the following steps:

[0053] In Step 201, read the sports video to be processed frame by frame through openCV to obtain the video frame Frame i ;

[0054] In Step 202, perform multi-object tracking on the video frame read in Step 201 to obtain the bbox and the tracking ID corresponding to the athlete target. The multi-object tracking algorithm adopted here is FairMOT;

[0055] In Step 203, in order to improve the processing speed of the overall algorithm for sports videos, the jersey number recognition and athlete recognition are performed every 5 frames. Therefore, in this step, it is judged whether the frame number Frame i is divisible by 5. If so, execute Step 204 and Step 205 simultaneously; otherwise, execute Step 206.

[0056] In Step 204, if the frame number is divisible by 5, perform jersey number recognition. Specifically:

[0057] For the bbox of the m athletes obtained in Step 202, traverse each bboxi Take the upper half of the bounding box for jersey number detection. Detect the i-th bounding box bbox i among the n digital bounding boxes Dbbox ij and the digit Digit ij as well as the confidence Dconf of each digit ij where 0 ≤ Digit ij ≤ 9, 0 ≤ Dconf ij ≤ 1, n ≥ 0, 0 ≤ j ≤ n.

[0058] The athlete jersey number recognition algorithm adopted here is the YoloV5 algorithm retrained with the dataset SVHN.

[0059] In step 205, if the frame number is divisible by 5, then perform athlete team recognition. According to the bbox i obtained in step 202, extract the color features of each bbox i to distinguish athletes of different teams by color features, and then determine the team T to which the player belongs according to the jersey color i .

[0060] In step 206, maintain a mapping table of the tracking ID and the jersey number. Since in some sports scenarios, there may be the same jersey number in different teams, the jersey number here has a leading digit indicating the team to which the athlete belongs. If the frame number is not divisible by 5, then directly map according to the tracking ID from the mapping table; otherwise, update the mapping table according to steps 204 and 205, and then map from the updated mapping table according to the tracking ID to obtain the jersey number.

[0061] In step 207, collect information through web crawlers and other means to establish an athlete information database, and store specific information such as the team, jersey number, and nationality of the athlete. Obtain the team and jersey number information according to the mapping table in step 206, and retrieve the corresponding real information of the athlete from the database, so as to realize athlete recognition.

[0062] The athlete recognition method combining multi-object tracking proposed by the present invention comprehensively uses deep learning methods such as multi-object tracking and jersey number recognition, and can effectively handle various complex scenarios in sports videos to achieve true athlete recognition.

Claims

1. A method for athlete recognition based on multi-object tracking, characterized in that, The specific steps are as follows: Step 1: Construct a player information database according to the publicly available game information; Step 2: Input the event video of the game into a multi-object tracking model, and perform multi-object tracking on the athletes in the video frame according to the player information database to obtain the bounding box of each athlete and its tracking unique identifier in each frame; The bounding box of an athlete i is bbox i , and the corresponding unique tracking identifier is ID i , the total number of athletes is m, 0 ≤ i ≤ m; Step 3: For the bounding box of each athlete in a certain frame, respectively take the upper half of each bounding box for detection to obtain the jersey number digits included in each athlete's bounding box; Detect the upper half of the athlete's bounding box bbox i to obtain the bounding box Dbbox of each digit contained therein ij , each digit Digit ij and the confidence Dconf of each digit ij ; where n represents the number of digits in each athlete bounding box, n≥0, 0≤j≤n, 0≤Digit ij ≤9, 0≤Dconf ij ≤1; Step 4: For the bounding box of a certain athlete, select the numbers corresponding to the top K position confidences, and sort the K numbers according to the abscissa of the athlete's bounding box to obtain the K-digit jersey number in decimal system of the athlete; The confidence Dconf of each detected digit in the athlete's bounding box ij Sorted from high to low, take the top K digits with the highest confidence, and sort them according to the abscissa of the athlete's bounding box to obtain the decimal K-digit jersey number JN i ; K is determined according to the number of digits of the jersey number in different game scenarios, and K ≤ n; Step 6: Use the tracking unique identifier of the athlete as the key, and the jersey number and its confidence as the value to establish a hash table to map the tracking unique identifier to the real jersey number; The hash table is a one-to-many mapping, i.e., one ID i may correspond to multiple JNs i ; Take the jersey number corresponding to the maximum confidence of the jersey number as the ID i The corresponding jersey number JN, so as to track the virtual ID i Map to the real jersey number JN i ; Step 7: At the same time, extract the color features included in the bounding box of each athlete in each frame, distinguish different teams, and determine the team to which each athlete belongs according to the jersey color; Step 8: Retrieve from the constructed player information database according to the jersey number and the team to which the athlete corresponding to the tracking unique identifier belongs to obtain the uniquely determined athlete name and athlete information; Step 9: Repeat steps 3 to 7 until the processing of each frame of the game video is completed.

2. The method for athlete recognition based on multi-object tracking according to claim 1, characterized in that, The player information database includes player name, team name, jersey number and jersey color.

3. The method for athlete recognition based on multi-object tracking according to claim 1, characterized in that, The confidence of the jersey number is the average value of the confidences of the K-digit numbers.

Images