A basketball tracking and main ball marking method, device and storage medium

Abstract

The application relates to a basketball tracking and main ball marking method, device and storage medium, wherein the method comprises the following steps: camera calibration; obtaining a basketball video frame; establishing a single-camera tracking queue; establishing single-camera basketball tracking; calculating the average value of the distance information change of a basketball object in a time window for the continuously tracked basketball object; judging whether the average value exceeds a threshold value; if yes, marking as a motion state; determining the projection error sum of all basketball objects marked as the motion state in different camera perspectives, and obtaining an error matrix; processing the error matrix based on the maximum bipartite matching principle to obtain a multi-camera basketball matching result; if the matching is successful, performing depth estimation on the image coordinates of the basketball based on a binocular disparity method; converting the depth estimation result into a real coordinate system to obtain three-dimensional coordinates; judging whether the basketball is in a target field; and if yes, marking as a main ball. Compared with the prior art, the application has the advantages of stable detection, high detection precision and the like.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer vision, and in particular to a basketball tracking and main ball marking method, device and storage medium. BACKGROUND

[0002] With the continuous development of the whole people's movement, basketball has become one of the most popular sports. More and more people are involved in basketball, and want to record and leave their own highlight moments of shooting. But many times, there are multiple basketballs in different states on the court, and the related software cannot effectively distinguish the main ball, resulting in an undesirable state of editing effect. At this time, if traditional manual analysis and editing of basketball are used, a large amount of manpower and energy is needed, and important moments can be easily ignored due to distraction. Therefore, using algorithms to detect the main ball (i.e. the game ball in the basketball court) and stably track the main ball to realize automatic game ball differentiation has a very broad application prospect.

[0003] When using algorithms to identify and track the main ball, the following problems mainly exist:

[0004] (1) When a basketball is blocked in one camera, the other camera loses the ability to identify the main ball and stops tracking the main ball.

[0005] (2) The tracking of the main ball cannot be continuously tracked due to obstruction.

[0006] (3) When there are multiple moving basketballs in the basketball court, the main ball matching error occurs. SUMMARY

[0007] The purpose of the present application is to provide a basketball tracking and main ball marking method to realize stable and continuous basketball tracking, distinguish the movement state of the basketball in the basketball venue and mark the main ball based on the position information.

[0008] The purpose of the present application can be achieved by the following technical solutions:

[0009] A basketball tracking and main ball marking method, comprising the following steps:

[0010] Calibrate the intrinsic and extrinsic parameters of the camera to obtain camera intrinsic and extrinsic parameters, wherein the camera extrinsic parameters are calibrated based on the real coordinate system and the court coordinate system under the camera view angle;

[0011] Obtain multiple basketball video frames synchronously captured by multiple cameras at the same time, wherein one camera corresponds to one basketball video frame;

[0012] Multiple single-camera tracking queues are established. Each single-camera tracking queue is used to store basketball objects tracked by a single camera. The number of single-camera tracking queues is the same as the number of cameras.

[0013] Basketball tracking for each basketball video frame is established based on a single-camera tracking queue. This process includes the following steps: For a basketball video frame, a basketball detection model is used to detect the basketball object in the latest frame; based on the distance information between the basketball object in the latest frame and the corresponding basketball object stored in the single-camera tracking queue, an adjacency matrix is ​​determined; it is then determined whether any element in the adjacency matrix exceeds a first distance threshold. If so, the basketball object in the latest frame is discarded; otherwise, the adjacency matrix is ​​processed based on the maximum binary matching principle to obtain a single-camera tracking result. The system tracks the basketball object in the tracking queue of a single camera. If the basketball matching result of a single camera is successful, the basketball object stored in the single camera tracking queue is updated to the basketball object of the latest frame. If the basketball matching result of a single camera is unsuccessful, it is determined whether the unsuccessful match is due to the failure to find a matching result of a basketball object stored in the tracking queue or the failure to find a matching result of a basketball object in the latest frame. If the unsuccessful match is due to the failure to find a matching result of a basketball object stored in the tracking queue, the basketball object stored in the tracking queue is deleted. If the unsuccessful match is due to the failure to find a matching result of a basketball object in the latest frame, the basketball object of the latest frame is added to the tracking queue as a newly identified tracking object.

[0014] For a basketball object continuously tracked in a single-camera basketball tracking, calculate the average value of the change in distance information of the basketball object within a pre-configured time window, and determine whether the average value exceeds a second distance threshold. If it does, mark the continuously tracked basketball object as a moving state; otherwise, mark the continuously tracked basketball object as a stationary state. Here, the continuously tracked basketball object refers to a basketball object whose basketball matching result is a successful match in a single-camera basketball tracking.

[0015] Based on the intrinsic and extrinsic parameters of the camera, the projection errors of all basketball objects marked as in motion are determined in different camera views, and an error matrix is ​​obtained, wherein an error matrix is ​​determined for every two cameras.

[0016] The error matrix is ​​processed based on the maximum binary matching principle to obtain the basketball matching results from multiple cameras. If the matching is successful, the depth of the image coordinates of the basketball in the court coordinate system from the camera's perspective is estimated based on the binocular parallax method. If the matching fails, the object is discarded.

[0017] Based on the camera extrinsic parameters, the depth estimation results are transformed into the real coordinate system to determine the three-dimensional coordinates of the basketball object in the real coordinate system;

[0018] Based on the three-dimensional coordinates of the basketball object in the real coordinate system, determine whether the basketball object is in the target field. If it is, mark it as the main ball; otherwise, mark it as a non-main ball.

[0019] The camera intrinsic parameters include distortion coefficients and intrinsic parameter matrices; the camera extrinsic parameters include rotation vector, displacement vector, projection matrix, and homography matrix.

[0020] The camera extrinsic parameters are obtained based on the following steps:

[0021] Establish the real coordinate system for the basketball court;

[0022] Determine reference points based on the lines drawn on the basketball court;

[0023] Initial camera pose values ​​are determined based on the RANSAC and PnP algorithms.

[0024] Based on the nonlinear least squares method, the camera pose relative to the real coordinate system is solved using the reference point and the initial values ​​of the camera pose, and the rotation vector, displacement vector and projection matrix in the camera extrinsic parameters are obtained.

[0025] The transformation parameters for converting the image coordinates of the reference point in the stadium coordinate system from the camera's perspective to the coordinates in the real coordinate system are calculated, thus obtaining the homography transformation matrix in the camera's extrinsic parameters.

[0026] The distance information includes the length and width of the detection box and its position change information. The detection box is obtained based on the basketball detection model, and one detection box corresponds to one basketball object.

[0027] The maximum binary matching principle is based on the principle of minimum matching value. It matches each row or column element in the matrix, and the matching result is that there is at most one matching value in each row or column, so that the sum of all matching values ​​in the matrix is ​​minimized.

[0028] The second distance threshold is one-fifth of the basketball radius.

[0029] The projection error is obtained based on the epipolar correction principle, specifically: the epipolar line is determined by the intersection of a three-point plane and the corrected camera plane, where the three-point plane is the plane determined by the center point of the basketball and the center points of the two cameras. The correction process of the camera plane involves correcting the initial camera plane using a rotation matrix, making the camera planes of the two cameras parallel and the epipolar line horizontal; the distance between the projection of the center point of the i-th basketball object in the view of the a-th camera and the projection of the epipolar line of the b-th camera is the projection error. The distance between the epipolar line of camera a and the projection of the center point of the j-th basketball object in the view of camera b is the projection error. Then the projection error between the i-th basketball object in the view of camera a and the j-th basketball object in the view of camera b is...

[0030] The error matrix is:

[0031]

[0032] in, Let adj be the sum of the projection errors of the i-th basketball object in the view of camera a and the j-th basketball object in the view of camera b. ab Let be the error matrix for the a-th camera and the b-th camera.

[0033] A basketball tracking and ball marking device includes a memory, a processor, and a program stored in the memory, wherein the processor executes the program to implement the method described above.

[0034] A storage medium having a program stored thereon, which, when executed, implements the method described above.

[0035] Compared with the prior art, the present invention has the following beneficial effects:

[0036] (1) The present invention uses on-site three-dimensional reference objects and moving objects to calibrate the camera extrinsic parameters, which solves the problem of inaccurate Z-direction calibration of camera extrinsic parameters and improves the calibration effect.

[0037] (2) The present invention matches basketballs in multiple cameras based on maximum binary matching, and can still achieve stable and continuous basketball tracking even when the basketball is blocked. The detection is stable and the detection accuracy is high.

[0038] (3) Based on the movement state of the basketball and its position, the main ball is judged and marked, which solves the problem of main ball matching error when there are multiple moving basketballs on the basketball court. Attached Figure Description

[0039] Figure 1 This is a flowchart of the method of the present invention. Detailed Implementation

[0040] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. These embodiments are based on the technical solution of the present invention and provide detailed implementation methods and specific operating procedures. However, the scope of protection of the present invention is not limited to the following embodiments.

[0041] A basketball tracking and ball marking method, such as Figure 1 As shown, it includes the following steps:

[0042] Step 1) Calibrate the camera's intrinsic and extrinsic parameters to obtain the camera's intrinsic and extrinsic parameters.

[0043] First, the camera's intrinsic parameters are calibrated. These parameters include distortion coefficients and the intrinsic parameter matrix.

[0044] The camera extrinsic parameters are calibrated based on the camera intrinsic parameters. The camera extrinsic parameters include rotation vector, displacement vector, projection matrix, and homography transformation matrix.

[0045] First, a true coordinate system for the basketball court is established, with the bottom left corner of the court facing the basket as the origin, the baseline as the x-axis, and the sidelines as the y-axis (unit: cm). A standard basketball court is used as a reference, as the lines on a standard court have a certain width (5cm), providing high differentiation, and their position and size are standard values. Therefore, a certain number of corner points can be selected from the lines as reference points in the true coordinate system. The image coordinates of the reference points in the court coordinate system under different camera views are identified. Initial camera pose values ​​are determined based on the RANSAC and PnP algorithms. Using all inliers, the pose relative to the true coordinate system is optimized using the nonlinear least squares method (Gaussian-Newton method), yielding the rotation vector, translation vector, and projection matrix in the camera extrinsic parameters. The transformation parameters for converting the image coordinates of the reference points in the court coordinate system from the camera view to the coordinates in the true coordinate system are calculated, yielding the homography transformation matrix in the camera extrinsic parameters.

[0046] There are several methods for obtaining the correspondence between image coordinates in the stadium coordinate system from the camera's perspective and camera extrinsic parameters:

[0047] 1. The pairing of reference objects relies on manually selecting the reference points of the stadium and the image coordinates in the stadium coordinate system from the camera's perspective.

[0048] 2. Sharpen and perform edge detection on the images captured by the camera, detect corner points and extract the coordinates of corresponding reference points, pair them with the actual stadium markers, and use the RANSAC algorithm to solve for the camera extrinsic parameters. Each set of extrinsic parameters can be determined by every 4 reference points.

[0049] 3. Based on the fact that the markings on a basketball court are all white, color features are extracted from the images captured by the camera, and the PnL algorithm is used to match line segments and solve for camera extrinsic parameters.

[0050] To address the issue of inaccurate Z-direction calibration of camera extrinsic parameters, calibration is improved by using on-court 3D reference objects and moving objects. The 3D reference objects can be markers at the basket and backboard, and the moving objects can be single basketball targets moving in the same frame of images acquired by a synchronously captured camera. The correspondence is constructed based on the camera model, and the extrinsic parameters are solved using a graphical model.

[0051] Step 2) Obtain multiple basketball video frames captured simultaneously by multiple cameras at the same time, with one camera corresponding to one basketball video frame.

[0052] Step 3) Establish multiple single-camera tracking queues. The single-camera tracking queues are used to store basketball objects tracked by one camera. The number of single-camera tracking queues is the same as the number of cameras.

[0053] Step 4) Establish single-camera basketball tracking for each basketball video frame based on the single-camera tracking queue.

[0054] Step 4-1) For a basketball video frame, use the basketball detection model to detect the basketball object in the latest frame of the basketball video frame. The basketball detection model is based on the YOLO network. After the detection is completed, the detected basketball objects in the basketball video frame are marked with detection boxes. One detection box corresponds to one basketball object.

[0055] Step 4-2) Based on the distance information between the basketball object in the latest frame and the basketball objects stored in the corresponding single-camera tracking queue, determine the adjacency matrix, wherein the distance information includes the length and width of the detection box and position change information;

[0056] Step 4-3) Determine whether any element in the adjacency matrix exceeds the first distance threshold.

[0057] If so, it means the match will definitely fail, and the basketball object in the latest frame should be discarded.

[0058] If not, the adjacency matrix is ​​processed based on the maximum binary matching principle to obtain the basketball matching result of a single camera;

[0059] The maximum binary matching principle is based on the principle of minimum matching value. It matches the distance information in each row or column of the matrix. The matching result is that there is at most one distance information in each row or column, so that the sum of all successfully matched distance information in the matrix is ​​minimized.

[0060] Step 4-4) If the basketball matching result of a single camera is a successful match, then update the basketball object stored in the single camera tracking queue to the basketball object of the latest frame.

[0061] If the basketball matching result for a single camera is a failure, then determine whether the failure is due to not finding a matching result for a basketball object stored in the tracking queue or not finding a matching result for a basketball object in the latest frame.

[0062] If no matching result is found for a basketball object stored in the tracking queue, the basketball object stored in the tracking queue is deleted. If no matching result is found for a basketball object in the latest frame, the basketball object in the latest frame is added to the tracking queue as a newly identified tracking object.

[0063] Step 5) For basketball objects continuously tracked in single-camera basketball tracking, calculate the average value of the distance information change of the basketball object within the pre-configured time window, wherein the continuously tracked basketball object refers to the basketball object whose basketball matching result of the single camera is a successful match.

[0064] Step 6) Determine whether the average value mentioned in Step 5) exceeds the second distance threshold. If yes, mark the continuously tracked basketball object as in motion. If no, mark the continuously tracked basketball object as stationary.

[0065] The second distance threshold is one-fifth of the basketball radius.

[0066] Steps 2) to 6) are the implementation steps for basketball tracking and marking the basketball's motion state. Marking the basketball's motion state is a prerequisite step for marking the main ball; that is, the main ball marking only applies to basketballs marked as being in motion. Steps 7) to 10) are the implementation steps for marking the main ball.

[0067] Step 7) Based on the intrinsic and extrinsic parameters of the camera, determine the sum of the projection errors of all basketball objects marked as in motion from different camera perspectives, and obtain the error matrix, wherein an error matrix is ​​determined for every two cameras.

[0068] The projection error is obtained based on the epipolar correction principle, specifically: the epipolar line is determined by the intersection of a three-point plane and the corrected camera plane, where the three-point plane is the plane determined by the center point of the basketball and the center points of the two cameras. The correction process of the camera plane involves correcting the initial camera plane using a rotation matrix, making the camera planes of the two cameras parallel and the epipolar line horizontal; the distance between the projection of the center point of the i-th basketball object in the view of the a-th camera and the projection of the epipolar line of the b-th camera is the projection error. The distance between the epipolar line of camera a and the projection of the center point of the j-th basketball object in the view of camera b is the projection error. Then the projection error between the i-th basketball object in the view of camera a and the j-th basketball object in the view of camera b is...

[0069] The error matrix is:

[0070]

[0071] in, Let adj be the sum of the projection errors of the i-th basketball object in the view of camera a and the j-th basketball object in the view of camera b. ab Let be the error matrix for the a-th camera and the b-th camera.

[0072] Step 8) Process the error matrix based on the maximum binary matching principle to obtain the basketball matching results from multiple cameras. If the match is successful, perform depth estimation on the image coordinates of the basketball in the court coordinate system from the camera's perspective using the binocular parallax method. If the match fails, discard the object.

[0073] The maximum binary matching principle is based on the principle of minimizing the matching value. It matches the sum of projection errors in each row or column of the matrix. The matching result is that there is at most one sum of projection errors in each row or column, so that the sum of all successfully matched sums of projection errors in the matrix is ​​minimized.

[0074] Step 9) Based on the camera extrinsic parameters, transform the depth estimation results into the real coordinate system to determine the three-dimensional coordinates of the basketball object in the real coordinate system.

[0075] Step 10) Determine whether the basketball object is in the target field based on its three-dimensional coordinates in the real coordinate system. If yes, it indicates that the basketball object is in the basketball court and is in motion, and is marked as the main ball. If no, it is marked as a non-main ball.

[0076] The step numbers above do not restrict the order in which the steps are executed.

[0077] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0078] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.

Claims

1. A method for tracking and marking the ball in a basketball, characterized in that, Includes the following steps: The camera's intrinsic and extrinsic parameters are calibrated to obtain the camera's intrinsic parameters and extrinsic parameters, wherein the camera's extrinsic parameters are calibrated based on the real coordinate system and the stadium coordinate system under the camera's viewpoint; Acquire multiple basketball video frames captured simultaneously by multiple cameras at the same time, with one camera corresponding to one basketball video frame; Multiple single-camera tracking queues are established. Each single-camera tracking queue is used to store basketball objects tracked by a single camera. The number of single-camera tracking queues is the same as the number of cameras. Basketball tracking for each basketball video frame is established based on a single-camera tracking queue. This process includes the following steps: For a basketball video frame, a basketball detection model is used to detect the basketball object in the latest frame; based on the distance information between the basketball object in the latest frame and the corresponding basketball object stored in the single-camera tracking queue, an adjacency matrix is ​​determined; it is then determined whether any element in the adjacency matrix exceeds a first distance threshold. If so, the basketball object in the latest frame is discarded; otherwise, the adjacency matrix is ​​processed based on the maximum binary matching principle to obtain a single-camera tracking result. The system tracks the basketball object in the tracking queue of a single camera. If the basketball matching result of a single camera is successful, the basketball object stored in the single camera tracking queue is updated to the basketball object of the latest frame. If the basketball matching result of a single camera is unsuccessful, it is determined whether the unsuccessful match is due to the failure to find a matching result of a basketball object stored in the tracking queue or the failure to find a matching result of a basketball object in the latest frame. If the unsuccessful match is due to the failure to find a matching result of a basketball object stored in the tracking queue, the basketball object stored in the tracking queue is deleted. If the unsuccessful match is due to the failure to find a matching result of a basketball object in the latest frame, the basketball object of the latest frame is added to the tracking queue as a newly identified tracking object. For a basketball object continuously tracked in a single-camera basketball tracking, calculate the average value of the change in distance information of the basketball object within a pre-configured time window, and determine whether the average value exceeds a second distance threshold. If it does, mark the continuously tracked basketball object as a moving state; otherwise, mark the continuously tracked basketball object as a stationary state. Here, the continuously tracked basketball object refers to a basketball object whose basketball matching result is a successful match in a single-camera basketball tracking. Based on the intrinsic and extrinsic parameters of the camera, the projection errors of all basketball objects marked as in motion are determined in different camera views, and an error matrix is ​​obtained, wherein an error matrix is ​​determined for every two cameras. The error matrix is ​​processed based on the maximum binary matching principle to obtain the basketball matching results from multiple cameras. If the matching is successful, the depth of the image coordinates of the basketball in the court coordinate system from the camera's perspective is estimated based on the binocular parallax method. If the matching fails, the object is discarded. Based on the camera extrinsic parameters, the depth estimation results are transformed into the real coordinate system to determine the three-dimensional coordinates of the basketball object in the real coordinate system; Based on the three-dimensional coordinates of the basketball object in the real coordinate system, determine whether the basketball object is in the target field. If it is, mark it as the main ball; otherwise, mark it as a non-main ball.

2. The basketball tracking and ball marking method according to claim 1, characterized in that, The camera intrinsic parameters include distortion coefficients and intrinsic parameter matrices; the camera extrinsic parameters include rotation vector, displacement vector, projection matrix, and homography matrix.

3. The basketball tracking and ball marking method according to claim 2, characterized in that, The camera extrinsic parameters are obtained based on the following steps: Establish the real coordinate system for the basketball court; Determine reference points based on the lines drawn on the basketball court; Initial camera pose values ​​are determined based on the RANSAC and PnP algorithms. Based on the nonlinear least squares method, the camera pose relative to the real coordinate system is solved using the reference point and the initial values ​​of the camera pose, and the rotation vector, displacement vector and projection matrix in the camera extrinsic parameters are obtained. The transformation parameters for converting the image coordinates of the reference point in the stadium coordinate system from the camera's perspective to the coordinates in the real coordinate system are calculated, thus obtaining the homography transformation matrix in the camera's extrinsic parameters.

4. The basketball tracking and ball marking method according to claim 1, characterized in that, The distance information includes the length and width of the detection box and its position change information. The detection box is obtained based on the basketball detection model, and one detection box corresponds to one basketball object.

5. The basketball tracking and ball marking method according to claim 1, characterized in that, The maximum binary matching principle is based on the principle of minimum matching value. It matches each row or column element in the matrix, and the matching result is that there is at most one matching value in each row or column, so that the sum of all matching values ​​in the matrix is ​​minimized.

6. The basketball tracking and ball marking method according to claim 1, characterized in that, The second distance threshold is one-fifth of the basketball radius.

7. The basketball tracking and ball marking method according to claim 1, characterized in that, The projection error is obtained based on the epipolar correction principle, specifically: the epipolar line is determined by the intersection of a three-point plane and the corrected camera plane, where the three-point plane is the plane determined by the center point of the basketball and the center points of the two cameras. The correction process of the camera plane involves correcting the initial camera plane using a rotation matrix, making the camera planes of the two cameras parallel and the epipolar line horizontal; the distance between the projection of the center point of the i-th basketball object in the view of the a-th camera and the projection of the epipolar line of the b-th camera is the projection error. The distance between the epipolar line of camera a and the projection of the center point of the j-th basketball object in the view of camera b is the projection error. Then the projection error between the i-th basketball object in the view of camera a and the j-th basketball object in the view of camera b is...

8. The basketball tracking and ball marking method according to claim 7, characterized in that, The error matrix is: in, Let adj be the sum of the projection errors of the i-th basketball object in the view of camera a and the j-th basketball object in the view of camera b. ab Let be the error matrix for the a-th camera and the b-th camera.

9. A basketball tracking and ball marking device, comprising a memory, a processor, and a program stored in the memory, characterized in that, When the processor executes the program, it implements the method as described in any one of claims 1-8.

10. A storage medium having a program stored thereon, characterized in that, When the program is executed, it implements the method as described in any one of claims 1-8.

Images