A player tracking and ball holder identification method, device and storage medium

Abstract

The present invention relates to a method, device, and storage medium for player tracking and ball handler identification. The method comprises: camera calibration; acquiring video frames; player detection based on a multi-target tracking model and a human skeleton key point detection model; marking players on the court based on key points on both feet; acquiring player information from the first frame of the video; establishing a tracking queue and a Kalman filter; establishing a uniform velocity change model for tracking object position information; storing the uniform velocity change model and player information in the tracking queue; acquiring player information from the latest frame of the video as a target to be matched; determining a cost matrix; matching the cost matrix based on the Hungarian algorithm; updating the tracking queue based on the matching results to track the players on the court; determining ball handler candidates based on a score derived from the ratio of player distance to basketball radius; and determining the ball handler based on the average of the ratios. Compared with existing technologies, the present invention has the advantages of rapid detection and stable tracking.

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and in particular to a method, device and storage medium for player tracking and ball holder identification based on a multi-target tracking algorithm. Background Art

[0002] With the continuous development of national sports, basketball has become one of the most popular sports. More and more people are participating in the sport and want to record their own highlights. However, relying solely on manual video editing is not only time-consuming but also prone to missing highlights. Therefore, using algorithms to identify and track players on the basketball court and provide a tracking foundation for automated editing of highlight videos has broad application prospects.

[0003] The current mainstream method for detecting the ball carrier in video processing relies on the recognition of the player's movements. CN112001324A discloses a method for player movement recognition in basketball game videos. The method uses the player's detection frame and the feature vector of the last layer of feature map as input into a relationship extraction network to obtain a feature matrix; based on the linear feedforward operation network, dimension reduction is performed, and the player's movement is recognized according to the reduced dimension feature matrix. Although this method realizes the player's movement recognition through feature extraction and dimension reduction, and improves the recognition speed to a certain extent, it will not produce ideal results when the player is partially blocked, resulting in a decrease in detection accuracy. In addition, it is necessary to perform player identification tracking and corresponding player movement recognition detection on each frame in the video, which not only consumes a lot of time, but also has high requirements on the computing power of the equipment. Summary of the Invention

[0004] The purpose of the present invention is to provide a player tracking and ball holder identification method based on a multi-target tracking algorithm, which can improve detection efficiency and stably identify the ball holder on the court.

[0005] The purpose of the present invention can be achieved by the following technical solutions:

[0006] A method for player tracking and ball holder identification based on a multi-target tracking algorithm includes the following steps:

[0007] Calibrate the intrinsic and extrinsic parameters of the camera to obtain camera intrinsic parameters and camera extrinsic parameters, wherein the camera extrinsic parameters are obtained based on the calibration of the real coordinate system and the stadium coordinate system under the camera perspective;

[0008] Get the video frames captured by the camera;

[0009] Perform player detection on the video frame based on a multi-target tracking model and a human skeleton key point detection model to obtain player information, wherein the player information includes a player ID, a player image feature vector, a player detection frame, and a player human skeleton key point pose estimation matrix;

[0010] Determine the image coordinates of the key points of the player's feet in the key point pose estimation matrix of the human skeleton, wherein the image coordinates are the coordinates in the court coordinate system under the camera's perspective;

[0011] Based on the camera extrinsic parameters, the image coordinates of the key points of the two feet are converted to the real coordinate system, and the three-dimensional coordinates of the key points of the two feet in the real coordinate system are determined;

[0012] Based on the three-dimensional coordinates of the key points of both feet in the real coordinate system, determine whether the player is in the target field. If so, mark the player corresponding to the current key points of both feet as a player on the field. Otherwise, mark him as an off-field player and filter him out.

[0013] Get the player information of the players on the field in the first frame of the video frame;

[0014] A tracking queue and a Kalman filter are established based on player information of the on-field players in the first frame of the video frame, and initialized, wherein the tracking queue is provided with a plurality of trackers, and one tracker corresponds to one player information;

[0015] Establishing a uniform speed change model of the position information of the tracking object based on the Kalman filter, wherein the tracking object is the on-field player, and the tracking object position information is the size and position of the detection box in the player information of the on-field player;

[0016] Save the uniform speed change model and player information of the tracking object to the tracking queue;

[0017] Get the player information of the players on the field in the latest frame of the video frame as the target to be matched;

[0018] Determine the cost matrix of the target to be matched and the tracking object stored in the tracking queue based on the player information;

[0019] Match the cost matrix based on the Hungarian algorithm;

[0020] Determine whether the match is successful. If so, update the player information in the tracker and the parameters of the Kalman filter. If not, perform IOU matching.

[0021] Determine whether the IOU match is successful. If the IOU match is successful, update the player information and Kalman filter parameters in the tracker. If the IOU match fails, determine whether the target to be matched fails or the tracked object fails.

[0022] If the tracking object fails to match, it is determined whether the execution time exceeds the pre-configured time threshold. If so, the tracking object is deleted. Otherwise, the tracking object is retained and the next matching judgment based on the Hungarian algorithm is performed.

[0023] If the target to be matched fails to match, it is determined whether the confidence level is greater than a first confidence threshold. If so, the target to be matched that failed to match is added to the tracking queue as a new tracking target. If not, the target to be matched is deleted. The confidence level is determined based on the difference between the position information of the tracking target and the position information predicted by the Kalman filter. The greater the difference, the lower the confidence level.

[0024] Real-time update of tracking queue to track players on the field;

[0025] Determine a time window based on the basketball court event, and obtain video frames within the time window;

[0026] Determine the distance between the player's hand key points and the center of the basketball detection frame in the current frame's human skeleton key point pose estimation matrix, where the basketball detection frame is derived based on the basketball detection model;

[0027] Calculate the ratio of the player's distance to the basketball's radius;

[0028] Determine whether the ratio of the player's distance to the basketball's radius is less than a pre-configured distance threshold. If so, a score is counted; otherwise, no score is counted.

[0029] Traverse all video frames within the current time window, accumulate the scores of each player, determine the total score of each player, and determine the ball holder candidate based on the total score;

[0030] Calculate the average value of the ratio of the distance between the ball handler candidate and the basketball radius within the current time window, and mark the ball handler candidate with the smallest average value as the ball handler.

[0031] The camera intrinsic parameters include distortion coefficients and intrinsic parameter matrices; the camera extrinsic parameters include rotation vectors, displacement vectors, projection matrices, and homography change matrices.

[0032] There are a total of 18 key points of the player's human skeleton.

[0033] The key points of the feet of the player's human skeleton are located at the forefoots of the left and right feet, and the key points of the hands are located at the wrists of the left and right hands.

[0034] The cost matrix is ​​obtained based on the weighted average of the relative position change values ​​of the human skeleton key points in the human skeleton key point pose estimation matrix of the target to be matched and the tracked object and the difference values ​​of the image feature vector.

[0035] The cost values ​​of matching items in the cost matrix whose confidence is lower than a second confidence threshold are set to infinity.

[0036] Updating the player information in the tracker includes updating the player image feature vector in an iterative manner. The updating rule of the player image feature vector is: updated player image feature vector = 0.9*player image feature vector saved in the tracking queue + 0.1*player image feature vector in the latest frame.

[0037] The player information in the updated tracker includes an updated player ID, where the updated player ID is the player ID of the continued tracking object, and the player ID of the target to be matched is discarded.

[0038] A player tracking and ball holder identification device based on a multi-target tracking algorithm includes a memory, a processor, and a program stored in the memory. When the processor executes the program, the method described above is implemented.

[0039] A storage medium stores a program thereon, and when the program is executed, the method described above is implemented.

[0040] Compared with the prior art, the present invention has the following beneficial effects:

[0041] (1) The present invention utilizes the ratio of the basketball radius to the handball distance to realize the recognition of the person holding the ball on the court, thereby improving the recognition effect. It is no longer necessary to recognize every frame of the video, which simplifies the complexity of the recognition, shortens the time used for video detection, and improves the detection speed and efficiency.

[0042] (2) The present invention utilizes binocular vision to solve the problem of unstable tracking caused by occlusion in a monocular camera, thereby improving the continuity of tracking. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 Flow chart of the method of the present invention. DETAILED DESCRIPTION

[0044] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0045] A player tracking and ball holder identification method based on multi-target tracking algorithm, such as Figure 1 As shown, the following steps are included:

[0046] Step 1) Calibrate the intrinsic and extrinsic parameters of the camera to obtain camera intrinsic parameters and camera extrinsic parameters, wherein the camera extrinsic parameters are calibrated based on the real coordinate system and the stadium coordinate system under the camera perspective.

[0047] The camera intrinsic parameters include distortion coefficients and intrinsic parameter matrices; the camera extrinsic parameters include rotation vectors, displacement vectors, projection matrices, and homography change matrices.

[0048] Step 2) Obtain the video frame captured by the camera.

[0049] Step 3) Perform player detection on the video frame based on the multi-target tracking model MOT and the human skeleton key point detection model Skeleton to obtain player information.

[0050] Player information includes the player ID, player image feature vector, player detection bounding box, and the player's skeletal keypoint pose estimation matrix. The player's skeletal keypoint pose estimation matrix stores the position information of the player's skeletal keypoints. There are 18 keypoints in total, with foot keypoints located on the forefoot of the left and right feet, and hand keypoints located on the wrists of the left and right hands.

[0051] Step 4) Determine the image coordinates of the key points of the feet of the player in the key point pose estimation matrix of the human skeleton, wherein the image coordinates are the coordinates in the court coordinate system under the camera perspective.

[0052] Step 5) Based on the camera extrinsic parameters, the image coordinates of the key points of the two feet are converted into the real coordinate system to determine the three-dimensional coordinates of the key points of the two feet in the real coordinate system.

[0053] Step 6) Based on the three-dimensional coordinates of the key points of both feet in the real coordinate system, determine whether the player is in the target venue (i.e., basketball court). If so, mark the player corresponding to the current key points of both feet as a player on the court; otherwise, mark him as an off-court player and filter him out.

[0054] Step 7) Obtain player information of the players on the field in the first frame of the video frame.

[0055] Step 8) A tracking queue and a Kalman filter are established based on the player information of the on-field players in the first frame of the video frame, and initialized. The tracking queue is provided with multiple trackers, and one tracker corresponds to one player information.

[0056] Step 9) A uniform speed change model of the position information of the tracking object is established based on the Kalman filter, where the tracking object is the on-field player, and the tracking object position information is the size and position of the detection box in the player information of the on-field player.

[0057] Step 10) Save the uniform speed change model of the tracking object and the player information to the tracking queue.

[0058] Step 11) Obtain the player information of the player on the field in the latest frame of the video frame as the target to be matched.

[0059] Step 12) Determine the cost matrix between the target to be matched and the tracking object stored in the tracking queue based on the player information.

[0060] The cost matrix is ​​obtained based on the weighted average of the relative position change values ​​of the human skeleton key points in the human skeleton key point pose estimation matrix of the target to be matched and the tracked object and the difference values ​​of the image feature vector.

[0061] Matches in the cost matrix with confidence levels below a set lower confidence threshold are set to infinity, ensuring that the matches represented by these costs will not be successful. The confidence level is determined by the difference between the tracked object's position information and the position information predicted by the Kalman filter; the greater the difference, the lower the confidence level.

[0062] Step 13) Match the cost matrix based on the Hungarian algorithm and determine whether the match is successful. If the match is successful, update the player information in the tracker and the parameters of the Kalman filter. If the match fails, perform IOU matching, that is, the intersection over union ratio of the image area of ​​the matching object.

[0063] Step 14) Determine whether the IOU match is successful. If the IOU match is successful, update the player information and Kalman filter parameters in the tracker. If the IOU match fails, determine whether the match failed for the target to be matched or the tracked object.

[0064] If the tracking object fails to match, it is determined whether the execution time exceeds the pre-configured time threshold. If so, the tracking object is deleted. Otherwise, the tracking object is retained and the next matching judgment based on the Hungarian algorithm is performed.

[0065] If the target fails to match, it is determined whether the confidence is greater than the manually set higher confidence threshold. If so, the target that fails to match is added to the tracking queue as a new tracking object. If not, the target is deleted.

[0066] Updating the player information in the tracker in steps 13) and 14) includes:

[0067] (1) The player image feature vector is updated in an iterative manner. The updating rule of the player image feature vector is: the updated player image feature vector = 0.9 * the player image feature vector saved in the tracking queue + 0.1 * the player image feature vector in the latest frame.

[0068] (2) Update the player ID, continue the player ID of the tracking object, and discard the player ID of the target to be matched.

[0069] (3) Update the pose estimation matrix of the player's human skeleton key points.

[0070] Step 15) Real-time updating of the tracking queue to track the players on the field, wherein the tracking queue outputs the following information: ① player ID, ② player image feature vector, ③ player human skeleton key point pose estimation matrix, ④ player detection frame, including the position and size of the detection frame.

[0071] Steps 1) to 15) enable the judgment and real-time tracking of players on the field.

[0072] Step 16) Determine a time window based on the occurrence of the pitching or goal event, typically 8-10 frames, and obtain video frames within the time window.

[0073] Step 17) Determine the player distance between the hand key points in the player's human skeleton key point pose estimation matrix in the current frame and the center of the basketball detection frame. The basketball detection frame is obtained based on the basketball detection model. Since there are two hand key points, located on the left wrist and the right wrist respectively, the player distance is determined as the shorter distance between the left hand key point and the center of the basketball frame.

[0074] Step 17) Calculate the ratio of the player's distance to the basketball's radius.

[0075] Since a single camera lacks scale information, it is necessary to record the average radius of the basketball object in the current time window and replace the player distance with the ratio of the player distance to the basketball radius.

[0076] Step 18) Determine whether the ratio of the player's distance to the basketball's radius is less than a set distance threshold (generally 4-5 times the basketball's radius). If so, a score is counted; otherwise, no score is scored.

[0077] Step 19) Traverse all video frames within the current time window, accumulate the scores of each player, determine the total score of each player, sort the scores from high to low, and select the top five players as ball handler candidates.

[0078] Step 20) Calculate the average value of the ratio of the distance between the ball handler and the basketball radius of the ball handler candidate in the current time window, and mark the ball handler candidate with the smallest average value as the ball handler.

[0079] The above step numbers do not limit the order in which the steps are executed.

[0080] If the above functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0081] The above describes in detail the preferred embodiments of the present invention. It should be understood that those skilled in the art can make numerous modifications and variations based on the concepts of the present invention without inventive effort. Therefore, any technical solutions that can be derived by those skilled in the art through logical analysis, reasoning, or limited experimentation based on the concepts of the present invention and the prior art should be within the scope of protection defined by the claims.

Claims

1. A player tracking and ball holder identification method based on a multi-target tracking algorithm, characterized in that: The following steps are involved: Calibrate the intrinsic and extrinsic parameters of the camera to obtain camera intrinsic parameters and camera extrinsic parameters, wherein the camera extrinsic parameters are obtained based on the calibration of the real coordinate system and the stadium coordinate system under the camera perspective; Get the video frames captured by the camera; Perform player detection on the video frame based on a multi-target tracking model and a human skeleton key point detection model to obtain player information, wherein the player information includes a player ID, a player image feature vector, a player detection frame, and a player human skeleton key point pose estimation matrix; Determine the image coordinates of the key points of the player's feet in the key point pose estimation matrix of the human skeleton, wherein the image coordinates are the coordinates in the court coordinate system under the camera's perspective; Based on the camera extrinsic parameters, the image coordinates of the key points of the two feet are converted to the real coordinate system, and the three-dimensional coordinates of the key points of the two feet in the real coordinate system are determined; Based on the three-dimensional coordinates of the key points of both feet in the real coordinate system, determine whether the player is in the target field. If so, mark the player corresponding to the current key points of both feet as a player on the field. Otherwise, mark him as an off-field player and filter him out. Get the player information of the players on the field in the first frame of the video frame; A tracking queue and a Kalman filter are established based on player information of the on-field players in the first frame of the video frame, and initialized, wherein the tracking queue is provided with a plurality of trackers, and one tracker corresponds to one player information; Establishing a uniform speed change model of the position information of the tracking object based on the Kalman filter, wherein the tracking object is the on-field player, and the tracking object position information is the size and position of the detection box in the player information of the on-field player; Save the uniform speed change model and player information of the tracking object to the tracking queue; Get the player information of the players on the field in the latest frame of the video frame as the target to be matched; Determine the cost matrix of the target to be matched and the tracking object stored in the tracking queue based on the player information; Match the cost matrix based on the Hungarian algorithm; Determine whether the match is successful. If so, update the player information in the tracker and the parameters of the Kalman filter. If not, perform IOU matching. Determine whether the IOU match is successful. If the IOU match is successful, update the player information and Kalman filter parameters in the tracker. If the IOU match fails, determine whether the target to be matched fails or the tracked object fails. If the tracking object fails to match, it is determined whether the execution time exceeds the pre-configured time threshold. If so, the tracking object is deleted. Otherwise, the tracking object is retained and the next matching judgment based on the Hungarian algorithm is performed. If the target to be matched fails to match, it is determined whether the confidence level is greater than a first confidence threshold. If so, the target to be matched that failed to match is added to the tracking queue as a new tracking target. If not, the target to be matched is deleted. The confidence level is determined based on the difference between the position information of the tracking target and the position information predicted by the Kalman filter. The greater the difference, the lower the confidence level. Real-time update of tracking queue to track players on the field; Determine a time window based on the basketball court event, and obtain video frames within the time window; Determine the distance between the player's hand key points and the center of the basketball detection frame in the current frame's human skeleton key point pose estimation matrix, where the basketball detection frame is derived based on the basketball detection model; Calculate the ratio of the player's distance to the basketball's radius; Determine whether the ratio of the player's distance to the basketball's radius is less than a pre-configured distance threshold. If so, a score is counted; otherwise, no score is counted. Traverse all video frames within the current time window, accumulate the scores of each player, determine the total score of each player, and determine the ball holder candidate based on the total score; Calculate the average value of the ratio of the distance between the ball handler candidate and the basketball radius within the current time window, and mark the ball handler candidate with the smallest average value as the ball handler.

2. The method for player tracking and ball holder identification based on a multi-target tracking algorithm according to claim 1, characterized in that: The camera intrinsic parameters include distortion coefficients and intrinsic parameter matrices; the camera extrinsic parameters include rotation vectors, displacement vectors, projection matrices, and homography change matrices.

3. The method for player tracking and ball holder identification based on a multi-target tracking algorithm according to claim 1, characterized in that: There are a total of 18 key points of the player's human skeleton.

4. The method for player tracking and ball holder identification based on a multi-target tracking algorithm according to claim 3, characterized in that: The key points of the feet of the player's human skeleton are located at the forefoots of the left and right feet, and the key points of the hands are located at the wrists of the left and right hands.

5. The method for player tracking and ball holder identification based on a multi-target tracking algorithm according to claim 1, characterized in that: The cost matrix is ​​obtained based on the weighted average of the relative position change values ​​of the human skeleton key points in the human skeleton key point pose estimation matrix of the target to be matched and the tracked object and the difference values ​​of the image feature vector.

6. The method for player tracking and ball holder identification based on a multi-target tracking algorithm according to claim 5, characterized in that: The cost values ​​of matching items in the cost matrix whose confidence is lower than a second confidence threshold are set to infinity.

7. The method for player tracking and ball holder identification based on a multi-target tracking algorithm according to claim 1, characterized in that: Updating the player information in the tracker includes updating the player image feature vector in an iterative manner. The updating rule of the player image feature vector is: updated player image feature vector = 0.9*player image feature vector saved in the tracking queue + 0.1*player image feature vector in the latest frame.

8. The method for player tracking and ball holder identification based on a multi-target tracking algorithm according to claim 1, characterized in that: The player information in the updated tracker includes an updated player ID, where the updated player ID is the player ID of the continued tracking object, and the player ID of the target to be matched is discarded.

9. A player tracking and ball holder identification device based on a multi-target tracking algorithm, comprising a memory, a processor, and a program stored in the memory, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 8 is implemented.

10. A storage medium having a program stored thereon, characterized in that: When the program is executed, the method according to any one of claims 1 to 8 is implemented.

Images