A Multi-Target Tracking Method and System for Badminton Matches Based on Visual Sensing Technology

Abstract

This invention provides a method and system for multi-target tracking in badminton matches based on visual sensing technology, belonging to the technical field of computer vision. The method includes dynamic camera calibration to acquire and correct the internal and external parameters of each camera; graph-based global optimization to determine the optimal camera parameters for multiple cameras based on the acquired internal and external parameters of each camera; data acquisition based on the optimal camera parameters of the multiple cameras; joint spatial-temporal optimization based on the data acquired from the multiple cameras to reconstruct the 3D position of dynamic objects; and target identification and multi-target tracking based on the 3D position of the target dynamic object reconstructed by the spatial-temporal optimization. This invention achieves real-time and accurate tracking of multiple targets in badminton matches through dynamic camera calibration, graph-based global optimization, and joint spatial-temporal optimization.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a method and system for multi-target tracking in badminton matches based on visual sensing technology. Background Technology

[0002] With the development of video capture and image processing technologies, target tracking systems, such as the Hawk-Eye system, have been widely used in existing ball sports (such as tennis, volleyball, and football) to capture and analyze the movement trajectories and positions of players, the ball, and other related targets during the game. However, the application of current target tracking systems in badminton still faces many challenges. Badminton is a typical high-speed and highly dynamic sport, characterized by extremely fast shuttlecock speed and rapid movement of multiple players. This scenario in badminton poses a significant challenge to traditional target tracking systems.

[0003] In badminton matches, the shuttlecock can travel at speeds exceeding 200 km / h. The distance the shuttlecock travels between consecutive video frames may exceed the camera's capture range. This high-speed movement can cause traditional video tracking systems to fail to accurately capture the shuttlecock's position and trajectory. Furthermore, the rapid movement and frequent position changes of players on the court increase the complexity of tracking. Players' mutual occlusion and rapid reactions can render conventional tracking algorithms ineffective, especially without pre-set tracking markers. In addition, the environmental factors of badminton matches also pose challenges to the tracking system: matches are typically held indoors, where lighting conditions may vary or be uneven, affecting the performance of visual sensors.

[0004] Therefore, there is an urgent need for a multi-target tracking system capable of handling high-speed moving objects, occlusion, and adapting to changes in lighting conditions, in order to meet the need for real-time and accurate identification and tracking of dynamic targets on the court during badminton matches. Summary of the Invention

[0005] To address the above problems and cope with the complex and dynamic environment of badminton matches, this invention provides a method and system for multi-target tracking in badminton matches using visual sensing technology, enabling real-time and accurate tracking of multiple targets during the match.

[0006] On one hand, the present invention provides a method for multi-target tracking in badminton matches using visual sensing technology, comprising:

[0007] Dynamic camera calibration is performed to acquire and correct the internal and external parameters of each camera.

[0008] Based on graph-based global optimization, the optimal camera parameters for the multi-camera system are determined according to the obtained internal and external parameters of each camera.

[0009] Data is acquired based on the optimal camera parameters of multiple cameras. Then, joint space-time optimization is performed based on the data acquired by the multiple cameras to reconstruct the three-dimensional position of the dynamic object.

[0010] Identify moving targets and perform multi-target tracking based on the 3D position of the target dynamic object reconstructed by the space-time optimization.

[0011] As a further improvement of the present invention, the dynamic camera calibration, which acquires and corrects the internal and external parameters of each camera, includes constructing a calibration optimization model and performing dynamic calibration using the calibration optimization model.

[0012] As a further improvement of the present invention, the calibration optimization model is as follows:

[0013]

[0014] in,

[0015] K is the intrinsic parameter matrix, which defines the optical characteristics of the camera;

[0016] R t and T t These are the rotation matrix and translation vector, respectively, describing the camera's rotation and orientation for each frame;

[0017] x ti These are image coordinates, representing the positions of feature points on the image plane;

[0018] X i These are world coordinates, representing the position of the feature point within the world coordinate system.

[0019] Π is the projection function, responsible for mapping three-dimensional spatial points to a two-dimensional image plane;

[0020] ω ti The weights are dynamically adjusted based on the visibility of feature points and tracking quality.

[0021] Ω is a regularization term used to ensure the stability of the solution and prevent overfitting.

[0022] As a further improvement of the present invention, the graph-based global optimization, which determines the optimal camera parameters of the multi-camera system based on the obtained internal and external parameters of each camera, includes constructing a graph optimization model to minimize the sum of errors between the observations and expected projections of all cameras.

[0023] As a further improvement to the present invention, the graph optimization model is as follows:

[0024]

[0025] in,

[0026] {K c},{R c},{T c} is the set of parameters for all cameras;

[0027] ε includes all edges formed by the same feature point observed by different cameras;

[0028] The Ψ term ensures the smoothness of global parameter updates to avoid system-level jitter caused by rapid changes in local optimizations.

[0029] As a further improvement of the present invention, the step of acquiring optimal camera parameters based on multiple cameras, performing joint space-time optimization based on the data acquired by multiple cameras, and reconstructing the three-dimensional position of dynamic objects includes constructing a space-time optimization model and using the space-time optimization model for three-dimensional reconstruction.

[0030] As a further improvement of the present invention, the space-time optimization model is as follows:

[0031]

[0032] in,

[0033] X i,t This represents the 3D point observed by the i-th camera at time t;

[0034] f(X i,t-1 V i,t ) is a prediction function based on a physics model, taking into account the speed V of the badminton shuttlecock. i,t and previous position X i,t-1 To estimate the current position;

[0035] This represents the set of matching pairs at time t, containing the same 3D point observed by different cameras;

[0036] α and β are weighting coefficients that control motion smoothness and spatial consistency, respectively.

[0037] As a further improvement to this invention, the Levenberg-Marquardt algorithm is used for iterative optimization, and X is adjusted in each iteration. i,t The value is used to reduce the overall residual.

[0038] As a further improvement of the present invention, the step of acquiring optimal camera parameters based on multiple cameras, and performing joint space-time optimization based on the data acquired by the multiple cameras to reconstruct the three-dimensional position of the dynamic object, further includes calculating the residual term in the space-time optimization model relative to X. i,tThe gradient vector and Jacobian matrix are used to guide the search direction of the optimization algorithm, where ∥∥X i,t -f(X i,t-1 V i,t )∥∥ 2 and ∥∥X i,t -X j,t ∥∥ 2 This is the residual term.

[0039] As a further improvement of the present invention, the calculation of each residual term in the space-time optimization model relative to X i,t The gradient vector and Jacobian matrix include:

[0040] For X i,t Calculate the partial derivative for each component.

[0041] in,

[0042] For motion smoothness terms:

[0043]

[0044] For spatial consistency:

[0045]

[0046] As a further improvement to the present invention, X is updated using the following algorithm. i,t :

[0047]

[0048] in,

[0049] λ is the damping coefficient;

[0050] I is the identity matrix;

[0051] E is the error vector;

[0052] J is the Jacobian matrix;

[0053] J T It is the transpose of the Jacobian matrix.

[0054] On the other hand, the present invention provides a badminton match multi-target tracking system using visual sensing technology, comprising multiple cameras, a memory, multiple processors, and a computer program stored in the memory, wherein the cameras are used to acquire images of multiple targets on the court in real time, and the processors execute the computer program to implement the following method:

[0055] Dynamic camera calibration: acquiring and correcting the internal and external parameters of each camera;

[0056] Optimal camera parameters for multiple cameras are determined based on graph-based global optimization.

[0057] Based on multi-camera data, combined space-time optimization is used to reconstruct the 3D position of dynamic objects;

[0058] Identify moving targets and perform multi-target tracking based on the aforementioned space-time optimization.

[0059] This invention provides a method and system for multi-target tracking in badminton using visual sensing technology. Through dynamic camera calibration and global image optimization, the system adjusts the camera's internal and external parameters in real time. It also utilizes joint space-time optimization to minimize the difference between the predicted target position error and the actual detection result. This allows the system to adapt to the rapid movement and changes of the badminton shuttlecock and athletes on the court, improving the accuracy and stability of the vision system in capturing moving objects, enhancing the accuracy of 3D reconstruction of moving objects, and increasing the precision of multi-target tracking in badminton matches. Attached Figure Description

[0060] Figure 1 This is a flowchart illustrating a multi-target tracking method for badminton matches based on visual sensing technology, according to an embodiment of the present invention.

[0061] Figure 2 This is a schematic diagram of a badminton match multi-target tracking method based on visual sensing technology, according to an embodiment of the present invention. Detailed Implementation

[0062] The following describes specific embodiments and appendices. Figure 1-2 The invention is described in detail so that those skilled in the art can more fully understand its purpose, features and effects.

[0063] Unless otherwise specified, all technical and scientific terms used in this invention have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. In the event of any discrepancy between the definitions of terms in this invention and their commonly understood meaning by one of ordinary skill in the art to which this invention pertains, the definitions set forth herein shall prevail.

[0064] This invention provides a method and system for multi-target tracking in badminton matches based on visual sensing technology, overcoming the difficulties in target tracking caused by the high-speed movement of the shuttlecock and the rapid movement and position changes of athletes during badminton matches.

[0065] Example 1

[0066] As a specific embodiment of the present invention, this embodiment provides a multi-target tracking method for badminton matches based on visual sensing technology, referring to... Figure 1 , Figure 2 The specific steps are as follows:

[0067] S10. Dynamic camera calibration: Acquire and correct the camera's internal and external parameters.

[0068] Dynamic calibration is performed on multiple cameras used for multi-target tracking to obtain their intrinsic and extrinsic parameters. The intrinsic parameters are those related to the camera's inherent characteristics, including focal length, principal point, and distortion coefficients. The extrinsic parameters are those in the world coordinate system, including position and rotation. These intrinsic and extrinsic parameters are crucial for accurately reconstructing 3D scene information from 2D images. The multiple targets include a badminton shuttlecock and an athlete.

[0069] Due to the extremely high speed of the shuttlecock during a badminton match, and the rapid movement and position changes of players on the court, traditional static calibration cannot effectively handle the rapid motion changes that occur between consecutive video frames. Dynamic camera calibration, on the other hand, allows for real-time adjustment of both internal and external camera parameters to adapt to the changes that may occur during a badminton match.

[0070] Furthermore, it also includes constructing a calibration optimization model and using the calibration optimization model for dynamic calibration to minimize projection error.

[0071] Specifically, the calibration optimization model is as follows:

[0072]

[0073] in,

[0074] K is the intrinsic parameter matrix, which defines the optical characteristics of the camera;

[0075] R t and T t These are the rotation matrix and translation vector, respectively, describing the camera's rotation and orientation for each frame;

[0076] x ti These are image coordinates, representing the positions of feature points on the image plane;

[0077] X i These are world coordinates, representing the position of the feature point within the world coordinate system.

[0078] Π is the projection function, responsible for mapping three-dimensional spatial points to a two-dimensional image plane;

[0079] ω ti The weights are dynamically adjusted based on the visibility of feature points and tracking quality.

[0080] Ω is a regularization term used to ensure the stability of the solution and prevent overfitting.

[0081] The parameters of the calibration optimization model, while describing the camera's optical characteristics and spatial position in each frame, determine the transformation of the camera's viewpoint and the correct projection of the target onto the image plane. Using this calibration optimization model, the system can dynamically adjust the camera position based on the error between the actual observed target position and the expected projected position.

[0082] By dynamically calibrating the camera and continuously adjusting parameters to minimize projection errors, the accuracy and stability of capturing badminton shuttlecocks and athletes during matches are improved.

[0083] The calibration optimization model takes into account the feature point positions x on the image plane. ti and the corresponding world coordinates X i In one instance, feature points are detected using image processing methods, such as edge detection or feature matching algorithms.

[0084] S20. Determine the optimal camera parameters based on graph-based global optimization.

[0085] After updating the parameters of all cameras in the system, the parameters of the entire system are optimized using the data from all cameras. Since multiple cameras are required for multi-target tracking in badminton matches, in order to further improve the accuracy of calibration, global optimization is performed after each camera is dynamically calibrated separately, thereby determining the optimal camera parameters, including internal and external parameters, so that all observations are as consistent as possible.

[0086] Specifically, in this embodiment, a graph-based global optimization method is employed to represent the overall calibration problem as a graph optimization problem. In the graph, each node represents a point in the world coordinate system, and each edge represents an observation. Preferably, the following graph optimization model is used to minimize the sum of errors between all camera observations and the expected projections.

[0087]

[0088] in,

[0089] {K c},{R c},{T c} is the set of parameters for all cameras;

[0090] ε includes all edges formed by the same feature point observed by different cameras;

[0091] The Ψ term ensures the smoothness of global parameter updates to avoid system-level jitter caused by rapid changes in local optimizations.

[0092] The optimizer uses Levenberg-Marquardt, Gauss-Newton, or Conjugate Gradient. During optimization, the optimizer iteratively updates the node parameters until convergence is reached or a preset number of iterations are completed. After optimization, the error of the camera parameters is evaluated, and the results are analyzed.

[0093] Furthermore, to ensure high consistency and accuracy in the calibration process of the entire system, weights are assigned and errors are calculated for each node and edge in the graph. The weights are determined based on the reliability of the observations and the geometric consistency between them; for example, feature points closer to the camera or in the center of the field of view are assigned higher weights.

[0094] S30. Joint space-time optimization to reconstruct the 3D position of dynamic objects.

[0095] Based on multi-view fusion analysis, the three-dimensional position of dynamic objects in the scene is reconstructed using camera data from different angles. In this embodiment, the three-dimensional position of multiple targets, such as badminton shuttlecock and athlete, is reconstructed.

[0096] Furthermore, a space-time optimization model is constructed and used for 3D reconstruction. Specifically, the space-time optimization model is a cost function, and the space-time optimization model is as follows:

[0097]

[0098] in,

[0099] X i,t This represents the 3D point observed by the i-th camera at time t;

[0100] f(X i,t-1 V i,t ) is a prediction function based on a physics model, taking into account the speed V of the badminton shuttlecock. i,t and previous position X i,t-1 To estimate the current position;

[0101] This represents the set of matching pairs at time t, containing the same 3D point observed by different cameras;

[0102] α and β are weighting coefficients that control motion smoothness and spatial consistency, respectively.

[0103] Since objects move across consecutive video frames, a single spatial analysis would overlook the continuity of their motion. This paper employs a joint space-time optimization model that considers the continuity of object motion, incorporating the object's motion across consecutive video frames. This improves the estimation of the 3D position of moving objects through 3D reconstruction, thereby enhancing the accuracy of the 3D reconstruction.

[0104] Furthermore, for each time point t, the three-dimensional point position X i,t An initial estimate is made. Preferably, the prediction is made using the position information from the previous moment obtained through a stereo vision-based 3D reconstruction method.

[0105] Furthermore, the cost function is transformed into a nonlinear least squares problem, where each term ∥∥X i,t -f(X i,t-1 V i,t )∥∥ 2 and ∥∥X i,t -X j,t ∥∥ 2 All are treated as a single residual term. Iterative optimization is performed using the Levenberg-Marquardt algorithm. In each iteration, the algorithm adjusts X... i,t The value is used to reduce the overall residual.

[0106] Furthermore, the cost function is calculated relative to X for each residual term. i,t The gradient vector and Jacobian matrix are used to guide the search direction of the optimization algorithm, enabling the cost function to quickly reach its optimum. The gradient is the cost function with respect to X. i,t The partial derivatives of .

[0107] Specifically, for X i,t Calculate the partial derivative for each component.

[0108] in,

[0109] For motion smoothness terms:

[0110]

[0111] For spatial consistency:

[0112]

[0113] The Jacobian matrix is ​​a matrix formed by the partial derivatives of a function vector with respect to multiple variables. In this embodiment, the Jacobian matrix represents the error term with respect to each position parameter X. i,t The partial derivatives of .

[0114] Assume E i,tLet J represent the error term observed by the i-th camera at time t. Then, an element J of the Jacobian matrix J is... k,l Represented as:

[0115]

[0116] in,

[0117] k represents the index of the error term;

[0118] l represents the index of the position parameter.

[0119] Due to location estimation X i,t The parameters are updated after each iteration, therefore the Jacobian matrix and gradient need to be recalculated at each iteration. Specifically, the Levenberg-Marquardt algorithm updates the parameters using the following method:

[0120]

[0121] in,

[0122] λ is the damping coefficient;

[0123] I is the identity matrix;

[0124] E is the error vector;

[0125] J T It is the transpose of the Jacobian matrix.

[0126] Using the above method, the algorithm can effectively iteratively update variable X along the direction of the fastest descent of the cost function. i,t This gradually reduces errors and optimizes the overall 3D reconstruction results.

[0127] S40. Identify moving targets and perform multi-target tracking.

[0128] After completing the camera dynamic calibration, the target is automatically identified from the video frames of multiple dynamic cameras using deep learning methods. Preferably, a convolutional neural network (CNN) is used to identify and locate the ball and players in a badminton match.

[0129] Key features, such as edges, corners, textures, or other visual markers, are extracted from the detected targets. Using position and potential velocity information from the previous video frame, a physical or statistical model is applied to predict the target's likely position in the next frame. For example, simple kinematic equations or a learning-based motion model can be used to predict the target's movement. In the new video frame, the target's actual position is updated based on the predicted position. The spatial-temporal optimization model in S30 minimizes the difference between the predicted target position error and the actual detection result, thus achieving accurate multi-target tracking.

[0130] The present invention provides a multi-target tracking method for badminton matches based on visual sensing technology. This method adjusts the camera's internal and external parameters in real time through dynamic camera and global optimization calibration of the image, adapting to the rapid movement and changes of the shuttlecock and athletes on the court. This ensures that the camera parameters always match the current observation conditions, thereby improving the accuracy and stability of the vision system in capturing moving objects. Furthermore, by utilizing joint space-time optimization, the method fully considers the motion patterns of objects in consecutive frames, improving the estimation of the three-dimensional position of moving objects and enhancing the accuracy of 3D reconstruction. This enables effective identification and tracking of multiple targets in badminton matches.

[0131] Example 2

[0132] As a specific embodiment of the present invention, this embodiment of the present invention provides a badminton match multi-target tracking system based on visual sensing technology, including multiple cameras, a memory, multiple processors and a computer program stored in the memory, wherein the multiple cameras are used to acquire images of multiple targets on the court in real time, and the processors execute the computer program to implement the method described in Embodiment 1.

[0133] The present invention relates to a badminton match multi-target tracking system based on visual sensing technology, which continuously acquires image information of multiple targets on the court, including badminton shuttlecocks and athletes, through multiple cameras in real time. The processor processes the image information to realize the identification and tracking of multiple targets.

[0134] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any other way. Any modifications or equivalent changes made based on the technical essence of the present invention shall still fall within the scope of protection claimed by the present invention.

Claims

1. A multi-target tracking method for badminton matches based on visual sensing technology, characterized in that, The method includes: Dynamic camera calibration is performed to acquire and correct the internal and external parameters of each camera. Based on graph-based global optimization, the optimal camera parameters for the multi-camera system are determined according to the obtained internal and external parameters of each camera. Data is acquired based on the optimal camera parameters of multiple cameras. Then, joint space-time optimization is performed based on the data acquired by the multiple cameras to reconstruct the three-dimensional position of the dynamic object. Identify moving targets and perform multi-target tracking based on the 3D position of the target dynamic object reconstructed by the space-time optimization. in, The process involves acquiring optimal camera parameters based on multiple cameras, performing joint space-time optimization based on the data obtained from the multiple cameras, and reconstructing the 3D position of the dynamic object. This includes constructing a space-time optimization model and using the space-time optimization model for 3D reconstruction. The space-time optimization model is as follows: in, Indicates time By the Three-dimensional points observed by a camera; It is a prediction function based on a physics model, taking into account the speed of the badminton shuttlecock. and previous position To estimate the current position; Indicates time The set of matching pairs contains the same 3D point observed by different cameras; and These are weighting coefficients, which respectively control motion smoothness and spatial consistency.

2. The multi-target tracking method for badminton matches based on visual sensing technology according to claim 1, characterized in that, The dynamic camera calibration, which acquires and corrects the internal and external parameters of each camera, includes constructing a calibration optimization model and performing dynamic calibration using the calibration optimization model.

3. The multi-target tracking method for badminton matches based on visual sensing technology according to claim 2, characterized in that, The calibration optimization model is as follows: in, K is the intrinsic parameter matrix, which defines the optical characteristics of the camera; and These are the rotation matrix and translation vector, respectively, describing the camera's rotation and orientation for each frame; These are image coordinates, representing the positions of feature points on the image plane; These are world coordinates, representing the position of the feature point within the world coordinate system. This is a projection function, responsible for mapping points in three-dimensional space to a two-dimensional image plane; The weights are dynamically adjusted based on the visibility of feature points and tracking quality. This is a regularization term used to ensure the stability of the solution and prevent overfitting.

4. The multi-target tracking method for badminton matches based on visual sensing technology according to claim 1, characterized in that, The graph-based global optimization, which determines the optimal camera parameters for the multi-camera system based on the obtained internal and external parameters of each camera, includes constructing a graph optimization model to minimize the sum of errors between the observations and expected projections of all cameras.

5. The multi-target tracking method for badminton matches based on visual sensing technology according to claim 4, characterized in that, The graph optimization model is as follows: in, It is a set of parameters for all cameras; It includes all edges formed by the same feature point observed by different cameras; This ensures the smoothness of global parameter updates, thus avoiding system-level jitter caused by rapid changes in local optimizations.

6. The multi-target tracking method for badminton matches based on visual sensing technology according to claim 1, characterized in that, The process of acquiring optimal camera parameters based on multiple cameras, and then performing joint space-time optimization to reconstruct the 3D position of the dynamic object, further includes calculating the residual term in the space-time optimization model relative to the target object. The gradient vector and Jacobian matrix are used to guide the search direction of the optimization algorithm, where and This is the residual term.

7. The multi-target tracking method for badminton matches based on visual sensing technology according to claim 6, characterized in that, The calculation of each residual term in the space-time optimization model relative to... The gradient vector and Jacobian matrix include: right Calculate the partial derivative for each component. in, For motion smoothness terms: For spatial consistency: 。 8. A multi-target tracking system for badminton matches based on visual sensing technology, characterized in that, The device includes multiple cameras, a memory, multiple processors, and a computer program stored in the memory, wherein the cameras are used to acquire images of multiple targets in a field in real time, and the processors execute the computer program to implement the method of any one of claims 1-7.

Images