Basketball court highlight moment recognition method, device and storage medium

Abstract

The present invention relates to a method, device, and storage medium for identifying highlight moments on a basketball court. The method comprises the following steps: camera calibration; acquiring video frames, cue ball marking information, and continuously updated cue ball tracking information and on-field player tracking information, and performing ball holder detection; determining the three-dimensional coordinates of the cue ball in a real-world coordinate system; performing shot event detection, and if a shot event occurs, further determining whether it is a goal event or a miss event; if a goal event occurs, determining whether a ball holder switch has occurred based on the shot holder determination video frame; if a ball holder switch has occurred, updating the ball holder and marking the most recent ball holder before the shot release video frame as the shot holder; and completing the goal event highlight moment identification based on the goal event and shot holder marking. Compared with existing technologies, the present invention has the advantages of convenient and rapid detection and high detection efficiency.

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and in particular to a method, device and storage medium for identifying highlight moments on a basketball court based on computer vision. Background Art

[0002] With the continuous development of internet technology, video has become the most widely disseminated information medium. Previously, most tasks related to video content analysis required manual work by professionals. However, with the massive amount of video information growing daily, relying solely on human effort is nearly impossible. Basketball, one of the sports with the largest fan base in China, is a highly valuable endeavor for both professional and amateur players, especially when it comes to post-game highlights. With the recent growth of the short video industry, more and more people are willing to share their videos on short video platforms, especially short videos related to competitive sports like basketball. However, these videos are typically shot by designated personnel on the sidelines and manually processed and edited by professionals. This requires significant manpower and effort, and can easily miss highlights, resulting in suboptimal editing results. Therefore, using algorithms to detect highlight events, stably track and identify players with the ball, and achieve automated editing has broad application prospects.

[0003] When using algorithms to edit highlight moments in basketball, there are several main problems:

[0004] (1) When the basketball stagnates on the basket, it is impossible to further distinguish the basketball event, resulting in a decrease in detection accuracy.

[0005] (2) When players compete for rebounds, it is easy to misjudge it as a shot.

[0006] (3) The method of recognizing highlight events based on the action recognition of the ball holder requires processing each frame of the video, which is labor-intensive and has low recognition efficiency. Summary of the Invention

[0007] The purpose of the present invention is to provide a computer vision-based highlight moment recognition method for basketball scenes, which can accurately identify highlight events occurring in basketball courts in real time and automatically.

[0008] The purpose of the present invention can be achieved by the following technical solutions:

[0009] A method for identifying highlight moments on a basketball court based on computer vision includes the following steps:

[0010] Calibrate the intrinsic and extrinsic parameters of the camera to obtain camera intrinsic parameters and camera extrinsic parameters, wherein the camera extrinsic parameters are obtained based on the calibration of the real coordinate system and the stadium coordinate system under the camera perspective;

[0011] Acquire multiple video frames captured by multiple cameras at the same time, where one camera corresponds to one video frame.

[0012] Obtaining cue ball marking information and continuously updated cue ball tracking information based on the video frames, wherein the cue ball tracking information includes the two-dimensional coordinates of the cue ball in the court coordinate system under the camera's perspective;

[0013] The two-dimensional coordinates are converted into a real coordinate system based on the camera extrinsic parameters to determine the three-dimensional coordinates of the cue ball in the real coordinate system;

[0014] Performing pitching event detection based on the three-dimensional coordinates of the cue ball, the pitching event detection comprising the following steps: determining two video frames before and after the cue ball passes through the basket from top to bottom based on the three-dimensional coordinates of the cue ball, and determining the intersection point of a line connecting the center of the cue ball in the two video frames with the plane of the basket;

[0015] Determining the spatial distance between the intersection point and the center point of the basket;

[0016] Determining whether the spatial distance is less than a first distance threshold and whether the center height of the cue ball in the previous frame of the two video frames is greater than the height of the basket frame, if so, determining that a pitching event has occurred; if not, determining that no pitching event has occurred, updating the three-dimensional coordinates of the cue ball and performing pitching event detection;

[0017] If a shot event occurs, determining whether the spatial distance is less than a second distance threshold; if so, determining that a goal event has occurred, and saving a video frame of the cue ball's most recent bottom-up crossing of a pre-configured height line before the goal event occurs as a shot-taken moment video frame; if not, determining that a miss event has occurred, wherein the second threshold distance is less than the first threshold distance;

[0018] Get continuously updated on-field player tracking information and perform ball carrier detection;

[0019] The video frame of the shooter's judgment is obtained by rewinding the preconfigured number of frames based on the moment of shooting;

[0020] Based on the cue ball tracking information, on-field player tracking information and ball holder detection results, all ball holder information in the pitcher judgment video frame is saved into the result matrix;

[0021] Determine whether the number of ball holder IDs in the ball holder information is greater than 1;

[0022] If the number of ball holder IDs is less than or equal to 1, no ball holder switch occurs. If the number of ball holder IDs is greater than 1, a ball holder switch occurs, the ball holder is updated, and the latest ball holder before the video frame at the time of the shot is marked as the shooter.

[0023] The highlight moment of the goal event is identified based on the goal event and the shooter mark.

[0024] The method further includes: when it is determined that a goal event occurs and the pitcher marking is completed,

[0025] Determine the image coordinates of the key points of the shooter's feet in the court coordinate system from the camera's perspective based on the video frame at the moment of shooting and the tracking information of the players on the court;

[0026] Based on the camera extrinsic parameters, the image coordinates are converted to the real coordinate system, and the three-dimensional coordinates of the key points of the two feet in the real coordinate system are determined;

[0027] Based on the spatial distance between the three-dimensional coordinates of the key points of both feet in the real coordinate system and the area of ​​the three-point line, it is determined whether the shooter is outside the three-point line. If so, the shooting event is marked as a three-point shot event; if not, the shooting event is marked as a two-point shot event;

[0028] The highlight moment of the three-point shot event is identified based on the three-point shot event and the shooter mark.

[0029] The method further includes: when it is determined that a goal event occurs and a ball holder switch occurs,

[0030] Based on the cue ball tracking information, determine whether the cue ball is blocked.

[0031] If the cue ball is not blocked, determining the distance between the cue ball and the floor based on the cue ball tracking information, and judging whether the cue ball has landed based on the distance, and if so, the pitcher has dribbled the ball; otherwise, the pitcher has not dribbled the ball;

[0032] If the cue ball is blocked, determine whether the cue ball lands and bounces in a pre-configured number of video frames based on the cue ball tracking information. If so, the shooter has dribbled the ball; if not, the shooter has not dribbled the ball.

[0033] Whether there is an assist event is determined based on whether the shooter has dribbled the ball. If the shooter has dribbled the ball, it is determined that there is no assist. If the shooter has not dribbled the ball, it is determined that there is an assist event, and the previous ball holder is marked as the assist player.

[0034] The highlight moment recognition of assist events is completed based on assist events and assist player tags.

[0035] The method further includes, when it is determined that a miss event occurs,

[0036] Advance the current video frame in time to obtain the rebound video frame;

[0037] Re-acquire continuously updated cue ball tracking information and on-field player tracking information based on the backboard video frame and perform ball holder detection;

[0038] Mark the first ball handler detected as the rebounder;

[0039] Recognize the highlight moments of rebounding events based on the rebounder markers.

[0040] The spatial distance d is:

[0041]

[0042] Among them, (x1, y1, z1) are the coordinates of the intersection of the line connecting the center of the main ball and the plane of the basket in the two video frames, and (x2, y2, z2) are the coordinates of the center point of the basket.

[0043] The preconfigured number of frames is determined based on a preconfigured time and a camera frame rate.

[0044] The continuously updated cue ball tracking information includes the size and position information of the cue ball detection frame.

[0045] The continuously updated on-field player tracking information includes player ID, player image feature vector, player human skeleton key point pose estimation matrix and player detection frame, wherein the player detection frame includes size and position information of the player detection frame.

[0046] A device for identifying highlight moments on a basketball court based on computer vision includes a memory, a processor, and a program stored in the memory. When the processor executes the program, the method described above is implemented.

[0047] A storage medium stores a program thereon, and when the program is executed, the method described above is implemented.

[0048] Compared with the prior art, the present invention has the following beneficial effects:

[0049] (1) The present invention realizes the event type judgment when the basketball stops on the basket by identifying the previous and next frames, and uses temporal logic judgment to better distinguish the types of highlight events, reduce the possibility of misjudgment, and improve recognition accuracy. For example, the rebounding event is recognized only after the shooting event occurs, avoiding the rebounding event being recognized as a shooting event.

[0050] (2) The present invention realizes the recognition of highlight moments based on computer vision, without placing sensors on the basket, making the entire recognition process more intelligent and highly automated.

[0051] (3) The present invention reversely infers the player ID based on the type of shooting event. Compared with constantly detecting and tracking the player's motion status, it does not need to perform player identification and action recognition on every video frame, which greatly simplifies the complexity of detection and tracking and improves detection efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 is a flow chart of the method of the present invention;

[0053] Figure 2 This is a flowchart for identifying the highlight moment of a three-point shot event according to the present invention;

[0054] Figure 3 This is a flowchart for identifying highlight moments of assist events in the present invention;

[0055] Figure 4 This is a flowchart for identifying highlight moments in rebounding events according to the present invention. DETAILED DESCRIPTION

[0056] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0057] A basketball court highlight moment recognition method based on computer vision, such as Figure 1 As shown, the following steps are included:

[0058] Step 1) Camera parameter calibration

[0059] The intrinsic and extrinsic parameters of the camera are calibrated to obtain camera intrinsic parameters and camera extrinsic parameters, wherein the camera extrinsic parameters are calibrated based on the real coordinate system and the stadium coordinate system under the camera perspective.

[0060] First, the camera's intrinsic parameters are calibrated. The camera's intrinsic parameters include distortion coefficients and intrinsic matrix. Based on the camera's intrinsic parameters, the camera's extrinsic parameters are calibrated. The camera's extrinsic parameters include rotation vectors, displacement vectors, projection matrix, and homography transformation matrix.

[0061] Step 2) Video frame acquisition

[0062] Acquire multiple video frames captured synchronously by multiple cameras at the same time, where one camera corresponds to one video frame.

[0063] Step 3) Basketball Tracking and Cue Ball Marking

[0064] Step 3-1) Establish multiple single-camera tracking queues, where the single-camera tracking queues are used to store the basketball object tracked by one camera. The number of the single-camera tracking queues is the same as the number of cameras.

[0065] Step 3-2) Establish single-camera basketball tracking for each basketball video frame based on the single-camera tracking queue.

[0066] Step 3-2-1) For a basketball video frame, use the basketball detection model to detect the basketball object in the latest frame of the basketball video frame, where the basketball detection model is established based on the Yolo network. After the detection is completed, the basketball object detected in the basketball video frame is marked with a detection box, and one detection box corresponds to one basketball object;

[0067] Step 3-2-2) Determine an adjacency matrix based on the distance information between the basketball object in the latest frame and the basketball object stored in the corresponding single-camera tracking queue, where the distance information includes the length and width of the detection box and position change information;

[0068] Step 3-2-3) Determine whether the elements in the adjacency matrix exceed the first distance threshold,

[0069] If so, it means that the match must be unsuccessful, and the basketball object in the latest frame is discarded.

[0070] If not, the adjacency matrix is ​​processed based on the maximum bipartite matching principle to obtain the basketball matching result of a single camera;

[0071] The maximum bisection matching principle is based on the principle of minimum matching value, matching the distance information of each row or column in the matrix. The matching result is that there is at most one distance information in each row or column, so that the sum of all successfully matched distance information in the matrix is ​​minimized.

[0072] Step 3-2-4) If the basketball matching result of the single camera is successful, the basketball object stored in the single camera tracking queue is updated to the basketball object of the latest frame;

[0073] If the basketball matching result of a single camera is a match failure, it is determined whether the match failure is a match result of not matching the basketball object stored in the tracking queue or a match result of not matching the basketball object in the latest frame.

[0074] If the matching result does not match the basketball object stored in the tracking queue, the basketball object stored in the tracking queue is deleted. If the matching result does not match the basketball object in the latest frame, the basketball object in the latest frame is added to the tracking queue as a newly identified tracking object.

[0075] Step 3-3) For a basketball object that is continuously tracked in basketball tracking by a single camera, an average value of a change in distance information of the basketball object within a preconfigured time window is calculated, wherein the continuously tracked basketball object refers to a basketball object for which a basketball matching result of the single camera is a successful match.

[0076] Step 3-4) Determine whether the average value in step 3-3) exceeds a second distance threshold. If so, mark the continuously tracked basketball object as in motion; if not, mark the continuously tracked basketball object as stationary. The second distance threshold is one-fifth of the radius of the basketball.

[0077] Step 3-5) Based on the intrinsic and extrinsic parameters of the camera, the sum of the projection errors of all basketball objects marked as in motion in different camera perspectives is determined to obtain an error matrix, where one error matrix is ​​determined for every two cameras.

[0078] The projection error is obtained based on the epipolar correction principle, specifically: the epipolar line is determined based on the intersection of the three-point plane and the corrected camera plane, wherein the three-point plane is the plane determined by the center point of the basketball and the center points of the two cameras. The correction process of the camera plane is to correct the initial camera plane through the rotation matrix so that the camera planes of the two cameras are parallel and the epipolar lines are horizontal; the distance between the center point of the i-th basketball object in the perspective of the a-th camera and the projection of the epipolar line of the b-th camera is the projection error The distance between the epipolar line of the a-th camera and the projection of the center point of the j-th basketball object in the view of the b-th camera is the projection error Then the projection error between the i-th basketball object in the a-th camera view and the j-th basketball object in the b-th camera view is

[0079] The error matrix is:

[0080]

[0081] in, is the sum of the projection errors of the i-th basketball object in the a-th camera view and the j-th basketball object in the b-th camera view, adj ab is the error matrix between the a-th camera and the b-th camera.

[0082] Step 3-6) Process the error matrix based on the maximum bisection matching principle to obtain the multi-camera basketball matching result. If the match is successful, the depth of the basketball's image coordinates in the court coordinate system under the camera's perspective is estimated based on the binocular parallax method. If the match fails, the object is discarded.

[0083] The maximum bisection matching principle is based on the principle of minimum matching value, matching the projection error sum of each row or column in the matrix. The matching result is that there is at most one projection error sum in each row or column, so that the sum of all successfully matched projection error sums in the matrix is ​​minimized.

[0084] Step 3-7) The depth estimation result is converted into a real coordinate system based on the homography transformation matrix to determine the three-dimensional coordinates of the basketball object in the real coordinate system.

[0085] Step 3-8) Determine whether the basketball object is in the target field based on the three-dimensional coordinates of the basketball object in the real coordinate system. If so, it indicates that the basketball object is in the basketball court and is in motion, and is marked as the main ball. If not, it is marked as a non-main ball.

[0086] Step 3-9) Output the three-dimensional coordinates of the main ball in the real coordinate system to obtain the main ball tracking information.

[0087] Step 4) Ball pitching event detection

[0088] Step 4-1) determining the two preceding and following video frames in which the cue ball passes through the basket hoop from top to bottom based on the three-dimensional coordinates of the cue ball, and determining the intersection of the line connecting the center of the cue ball and the plane of the basket hoop in the two video frames;

[0089] Step 4-2) Determine the spatial distance between the intersection point and the center point of the basket; the spatial distance d is:

[0090]

[0091] Where (x1, y1, z1) is the coordinate of the intersection of the line connecting the center of the cue ball and the plane of the basket in the two video frames, and (x2, y2, z2) is the coordinate of the center point of the basket;

[0092] Step 4-3) Determine whether the spatial distance d is less than a first distance threshold and whether the center height of the cue ball in the previous frame of the two video frames is greater than the height of the basket. If so, determine that a pitching event has occurred; if not, determine that no pitching event has occurred, update the three-dimensional coordinates of the cue ball, and perform pitching event detection;

[0093] The first distance threshold is generally 45 cm;

[0094] Step 4-4) If a shot event occurs, determine whether the spatial distance is less than a second distance threshold. If so, determine that a goal event has occurred, and save the video frame of the cue ball's most recent bottom-up crossing of a pre-configured height line before the goal event occurs as the shot shot moment video frame. If not, determine that a miss event has occurred. The second distance threshold is 10 cm, which is obtained by subtracting the basketball radius of 13 cm from the basket radius of 23 cm. Considering the error situation, a certain amount of redundancy can be taken. The pre-configured height line is generally set to 2 meters.

[0095] Step 5) Player tracking and pitcher marking

[0096] Step 5-1) Obtain continuously updated on-field player tracking information

[0097] Step 5-1-1) Perform player detection on the video frame based on the multi-target tracking model MOT and the human skeleton key point detection model Skeleton to obtain player information;

[0098] Player information includes the player ID, player image feature vector, player detection box, and the player's human skeleton keypoint pose estimation matrix. The player's human skeleton keypoint pose estimation matrix stores the position information of the player's human skeleton keypoints. There are 18 player skeleton keypoints in total, of which the foot keypoints are located on the forefoot of the left and right feet, and the hand keypoints are located on the wrists of the left and right hands.

[0099] Step 5-1-2) Determine the image coordinates of the key points of the feet of the player in the key point pose estimation matrix of the human skeleton, wherein the image coordinates are the coordinates in the court coordinate system under the camera perspective.

[0100] Step 5-1-3) Convert the image coordinates of the foot key points to the real coordinate system based on the camera extrinsic parameters, and determine the three-dimensional coordinates of the foot key points in the real coordinate system;

[0101] Step 5-1-4) Determine whether the player is in the target venue (i.e., basketball court) based on the 3D coordinates of the key points of both feet in the real coordinate system. If so, mark the player corresponding to the current key points of both feet as a player on the court; otherwise, mark the player as an off-court player and filter it out.

[0102] Step 5-1-5) obtaining player information of the players on the field in the first frame of the video frame;

[0103] Step 5-1-6) establishing a tracking queue and a Kalman filter based on the player information of the on-field players in the first frame of the video frame, and initializing them, wherein the tracking queue is provided with multiple trackers, one tracker corresponding to one player information;

[0104] Step 5-1-7) Establishing a uniform velocity variation model of the position information of the tracking object based on the Kalman filter, wherein the tracking object is the on-field player, and the tracking object position information is the size and position of the detection box in the player information of the on-field player;

[0105] Step 5-1-8) Save the uniform speed change model of the tracking object and the player information to the tracking queue;

[0106] Step 5-1-9) Obtain the player information of the player on the field in the latest frame of the video frame as the target to be matched;

[0107] Step 5-1-10) Determine the cost matrix between the target to be matched and the tracking object stored in the tracking queue based on the player information;

[0108] The cost matrix is ​​obtained based on the weighted average of the relative position change values ​​of the human skeleton key points in the human skeleton key point pose estimation matrix of the target to be matched and the tracked object and the difference values ​​of the image feature vector;

[0109] The cost matrix sets the cost value of any match whose confidence level is below a set lower confidence threshold to infinity, ensuring that the match represented by that cost value will not be a successful match. The confidence level is determined based on the difference between the position of the tracked object and the position predicted by the Kalman filter. The greater the difference, the lower the confidence level.

[0110] Step 5-1-11) Match the cost matrix based on the Hungarian algorithm and determine whether the match is successful. If the match is successful, update the player information in the tracker and the parameters of the Kalman filter. If the match fails, perform IOU matching, that is, the intersection over union ratio of the image area of ​​the matching object;

[0111] Step 5-1-12) Determine whether the IOU match is successful. If the IOU match is successful, update the player information and Kalman filter parameters in the tracker. If the IOU match fails, determine whether the match failed for the target to be matched or the tracked object.

[0112] If the tracking object fails to match, it is determined whether the execution time exceeds the pre-configured time threshold. If so, the tracking object is deleted. Otherwise, the tracking object is retained and the next matching judgment based on the Hungarian algorithm is performed.

[0113] If the target fails to be matched, it is determined whether the confidence is greater than the manually set higher confidence threshold. If so, the target that fails to be matched is added to the tracking queue as a new tracking object.

[0114] The player information updated in the tracker in steps 5-1-11) and 5-1-12) includes:

[0115] (1) updating the player image feature vector in an iterative manner, wherein the updating rule of the player image feature vector is: updated player image feature vector = 0.9 * player image feature vector saved in the tracking queue + 0.1 * player image feature vector in the latest frame;

[0116] (2) Update the player ID, continue the player ID of the tracking target, and discard the player ID of the target to be matched;

[0117] (3) Update the pose estimation matrix of the player's human skeleton key points;

[0118] Step 15) updating the tracking queue in real time to track the players on the field, wherein the information output by the tracking queue includes: ① player ID, ② player image feature vector, ③ player human skeleton key point pose estimation matrix, ④ player detection frame, including the position and size of the detection frame;

[0119] Step 5-2) Ball holder detection

[0120] Step 5-2-1) Determine a time window based on the occurrence of an event such as a pitch or goal, typically 8-10 frames, and obtain video frames within the time window;

[0121] Step 5-2-2) Determine the player distance between the hand key points in the player's human skeleton key point pose estimation matrix in the current frame and the center of the basketball detection frame. The basketball detection frame is obtained based on the basketball detection model. Since there are two hand key points, located at the left wrist and the right wrist respectively, the player distance is determined as the shorter distance between the left hand key point and the center of the basketball frame;

[0122] Step 5-2-3) Calculate the ratio of the player's distance to the basketball's radius;

[0123] Since a single camera lacks scale information, it is necessary to record the average radius of the basketball object in the current time window and replace the player distance with the ratio of the player distance to the basketball radius.

[0124] Step 5-2-4) Determine whether the ratio of the player's distance to the basketball's radius is less than a set distance threshold (generally 4-5 times the basketball's radius). If so, a score is counted; otherwise, no score is counted.

[0125] Step 5-2-5) Traverse all video frames within the current time window, accumulate the scores of each player, determine the total score of each player, sort the scores from high to low, and select the top five players as ball handler candidates;

[0126] Step 5-2-6) Calculate the average value of the ratio of the distance between the ball handler and the basketball radius of the ball handler candidates in the current time window, and mark the ball handler candidate with the smallest average value as the ball handler;

[0127] Step 5-3) rewinding the video frame at the moment of the shot by a certain number of frames to obtain the shooter's judgment video frame; the number of rewind frames is determined by the camera frame rate and the rewind time, which is generally set to 5-7 seconds;

[0128] Step 5-4) Based on the cue ball tracking information, the on-field player tracking information, and the ball holder detection results, save all ball holder information in the pitcher judgment video frame into a result matrix;

[0129] Step 5-5) determining whether the number of player IDs in the ball carrier information is greater than 1;

[0130] If the number of player IDs is less than or equal to 1, no ball holder switch occurs and the ball holder is marked as the shooter. If the number of player IDs is greater than 1, a ball holder switch occurs, the ball holder is updated, and the latest ball holder before the video frame at the time of the shot is marked as the shooter.

[0131] Step 6) Complete the highlight moment recognition of the goal event based on the goal event and the shooter mark.

[0132] Step 7) Identify the highlight moments of the three-point shot event

[0133] When a goal is scored and the shooter is marked,

[0134] Determine the image coordinates of the key points of the shooter's feet in the court coordinate system from the camera's perspective based on the video frame at the moment of shooting and the tracking information of the players on the court;

[0135] Based on the camera extrinsic parameters, the image coordinates are converted to the real coordinate system, and the three-dimensional coordinates of the key points of the two feet in the real coordinate system are determined;

[0136] Based on the spatial distance between the three-dimensional coordinates of the key points of both feet in the real coordinate system and the area of ​​the three-point line, it is determined whether the shooter is outside the three-point line. If so, the shooting event is marked as a three-point shot event; if not, the shooting event is marked as a two-point shot event;

[0137] The highlight moment of the three-point shot event is identified based on the three-point shot event and the shooter mark.

[0138] Step 8) Identify the highlight moments of assist events

[0139] When a goal is scored and the ball holder switches,

[0140] Based on the cue ball tracking information, determine whether the cue ball is blocked.

[0141] If the cue ball is not blocked, determining the distance between the cue ball and the floor based on the cue ball tracking information, and judging whether the cue ball has landed based on the distance, and if so, the pitcher has dribbled the ball; otherwise, the pitcher has not dribbled the ball;

[0142] If the cue ball is blocked, determine whether the cue ball lands and bounces in a pre-configured number of video frames based on the cue ball tracking information. If so, the shooter has dribbled the ball; if not, the shooter has not dribbled the ball.

[0143] Whether there is an assist event is determined based on whether the shooter has dribbled the ball. If the shooter has dribbled the ball, it is determined that there is no assist. If the shooter has not dribbled the ball, it is determined that there is an assist event, and the previous ball holder is marked as the assist player.

[0144] The highlight moment recognition of assist events is completed based on assist events and assist player tags.

[0145] Step 9) Identify the highlight moments of rebounding events

[0146] When a missed shot is determined,

[0147] Advance the current video frame in time to obtain the rebound video frame;

[0148] Re-acquire continuously updated cue ball tracking information and on-field player tracking information based on the backboard video frame and perform ball holder detection;

[0149] Mark the first ball handler detected as the rebounder;

[0150] Recognize the highlight moments of rebounding events based on the rebounder markers.

[0151] If the above functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0152] The above describes in detail the preferred embodiments of the present invention. It should be understood that those skilled in the art can make numerous modifications and variations based on the concepts of the present invention without inventive effort. Therefore, any technical solutions that can be derived by those skilled in the art through logical analysis, reasoning, or limited experimentation based on the concepts of the present invention and the prior art should be within the scope of protection defined by the claims.

Claims

1. A method for identifying highlight moments on a basketball court based on computer vision, characterized in that: The following steps are involved: Calibrate the intrinsic and extrinsic parameters of the camera to obtain camera intrinsic parameters and camera extrinsic parameters, wherein the camera extrinsic parameters are obtained based on the calibration of the real coordinate system and the stadium coordinate system under the camera perspective; Acquire multiple video frames captured by multiple cameras at the same time, where one camera corresponds to one video frame. Obtaining cue ball marking information and continuously updated cue ball tracking information based on the video frames, wherein the cue ball tracking information includes the two-dimensional coordinates of the cue ball in the court coordinate system under the camera's perspective; The two-dimensional coordinates are converted into a real coordinate system based on the camera extrinsic parameters to determine the three-dimensional coordinates of the cue ball in the real coordinate system; Performing pitching event detection based on the three-dimensional coordinates of the cue ball, the pitching event detection comprising the following steps: determining two video frames before and after the cue ball passes through the basket from top to bottom based on the three-dimensional coordinates of the cue ball, and determining the intersection point of a line connecting the center of the cue ball in the two video frames with the plane of the basket; Determining the spatial distance between the intersection point and the center point of the basket; Determining whether the spatial distance is less than a first distance threshold and whether the center height of the cue ball in the previous frame of the two video frames is greater than the height of the basket frame, if so, determining that a pitching event has occurred; if not, determining that no pitching event has occurred, updating the three-dimensional coordinates of the cue ball and performing pitching event detection; If a shot event occurs, determining whether the spatial distance is less than a second distance threshold; if so, determining that a goal event has occurred, saving a video frame of the cue ball's most recent bottom-up crossing of a pre-configured height line before the goal event occurs as a shot-taken moment video frame; if not, determining that a miss event has occurred, wherein the second distance threshold is less than the first distance threshold; Get continuously updated on-field player tracking information and perform ball carrier detection; The video frame of the shooter's judgment is obtained by rewinding the preconfigured number of frames based on the moment of shooting; Based on the cue ball tracking information, on-field player tracking information and ball holder detection results, all ball holder information in the pitcher judgment video frame is saved into the result matrix; Determine whether the number of ball holder IDs in the ball holder information is greater than 1; If the number of ball holder IDs is less than or equal to 1, no ball holder switch occurs. If the number of ball holder IDs is greater than 1, a ball holder switch occurs, the ball holder is updated, and the latest ball holder before the video frame at the time of the shot is marked as the shooter. The highlight moment of the goal event is identified based on the goal event and the shooter mark.

2. The method for identifying highlight moments on a basketball court based on computer vision according to claim 1, characterized in that: The method further includes: when it is determined that a goal event occurs and the pitcher marking is completed, Determine the image coordinates of the key points of the shooter's feet in the court coordinate system from the camera's perspective based on the video frame at the moment of shooting and the tracking information of the players on the court; Based on the camera extrinsic parameters, the image coordinates are converted to the real coordinate system, and the three-dimensional coordinates of the key points of the two feet in the real coordinate system are determined; Based on the spatial distance between the three-dimensional coordinates of the key points of both feet in the real coordinate system and the area of ​​the three-point line, it is determined whether the shooter is outside the three-point line. If so, the shooting event is marked as a three-point shot event; if not, the shooting event is marked as a two-point shot event; The highlight moment of the three-point shot event is identified based on the three-point shot event and the shooter mark.

3. The method for identifying highlight moments on a basketball court based on computer vision according to claim 1, characterized in that: The method further includes: when it is determined that a goal event occurs and a ball holder switch occurs, Based on the cue ball tracking information, determine whether the cue ball is blocked. If the cue ball is not blocked, determining the distance between the cue ball and the floor based on the cue ball tracking information, and judging whether the cue ball has landed based on the distance, and if so, the pitcher has dribbled the ball; otherwise, the pitcher has not dribbled the ball; If the cue ball is blocked, determine whether the cue ball lands and bounces in a pre-configured number of video frames based on the cue ball tracking information. If so, the shooter has dribbled the ball; if not, the shooter has not dribbled the ball. Whether there is an assist event is determined based on whether the shooter has dribbled the ball. If the shooter has dribbled the ball, it is determined that there is no assist. If the shooter has not dribbled the ball, it is determined that there is an assist event, and the previous ball holder is marked as the assist player. The highlight moment recognition of assist events is completed based on assist events and assist player tags.

4. The method for identifying highlight moments on a basketball court based on computer vision according to claim 1, characterized in that: The method further includes, when it is determined that a miss event occurs, Advance the current video frame in time to obtain the rebound video frame; Re-acquire continuously updated cue ball tracking information and on-field player tracking information based on the backboard video frame and perform ball holder detection; Mark the first ball handler detected as the rebounder; Recognize the highlight moments of rebounding events based on the rebounder markers.

5. The method for identifying highlight moments on a basketball court based on computer vision according to claim 1, characterized in that: The spatial distance d is: Among them, (x1, y1, z1) is the coordinate of the intersection of the line connecting the center of the main ball and the plane of the basket in the two video frames, and (x2, y2, z2) is the coordinate of the center point of the basket.

6. The method for identifying highlight moments on a basketball court based on computer vision according to claim 1, characterized in that: The preconfigured number of frames is determined based on a preconfigured time and a camera frame rate.

7. The method for identifying highlight moments on a basketball court based on computer vision according to claim 1, characterized in that: The continuously updated cue ball tracking information includes the size and position information of the cue ball detection frame.

8. The method for identifying highlight moments on a basketball court based on computer vision according to claim 1, characterized in that: The continuously updated on-field player tracking information includes player ID, player image feature vector, player human skeleton key point pose estimation matrix and player detection frame, wherein the player detection frame includes size and position information of the player detection frame.

9. A device for identifying highlight moments on a basketball court based on computer vision, comprising a memory, a processor, and a program stored in the memory, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 8 is implemented.

10. A storage medium having a program stored thereon, characterized in that: When the program is executed, the method according to any one of claims 1 to 8 is implemented.

Images