Detection tracking system and method used in sports scene analysis

By combining YOLOv7 model and deep learning technology in sports scene analysis, the appearance and position characteristics of athletes are extracted, and the high-score and low-score detection boxes are matched, and the Kalman filtering and sparse optical flow method are combined to solve high-speed motion and frequent change. The problem of following and error matching in the scene achieves more accurate and stable athlete tracking.

CN118941592APending Publication Date: 2024-11-12NINGBO ARTIFICIAL INTELLIGENCE RES INST OF SHANGHAI JIAOTONG UNIV
View PDF 0 Cites 2 Cited by

Patent Information

Application Number
CN202411016906.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-26
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

In the high-speed motion and frequent movement scenarios, the prior art is difficult to achieve accurate detection and tracking, and it is prone to problems of loss and error matching, mainly due to the difficulty in feature extraction caused by the fuzzy detection box and the lack of dependence on appearance information.

Method used

The system adopts a system that includes camera module, object detection module, appearance feature extraction module, prediction module, motion camera compensation module, high score matching module, position feature extraction module, low score matching module, update module and trajectory maintenance module, through the YOLOv7 model, Convolutional neural network, Transformer, sparse optical flow method and SplineCNN technologies extract the appearance and position characteristics of athletes, perform greedy matching and similarity calculations, and combine Kalman filtering to update the trajectory to improve tracking stability.

Benefits of technology

Effectively overcome the problems of loss and mismatch in traditional models when athletes move quickly and block, improve the accuracy and stability of athletes' tracking, reduce the dependence on appearance characteristics, enhance the utilization of timing information, and reduce the Computational complexity.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118941592A_ABST
    Figure CN118941592A_ABST
Patent Text Reader

Abstract

The invention discloses a detecting and tracking system and method used in sports scene analysis, and relates to the field of multi-target detecting and tracking. The system comprises a camera module, a target detection module, an appearance feature extraction module, a prediction module, a motion camera compensation module, a high-score matching module, a position feature extraction module, a low-score matching module, an updating module and a track maintenance module. The method comprises the following steps: step 1, capturing video image data; 2, generating a prediction frame; 3, position information of the athletes is recognized, and a high score detection frame and a low score detection frame are generated; 4, extracting appearance characteristics of the athletes; 5, calculating the similarity between the detection frame and the track, and carrying out greedy matching; step 6, extracting position features of the low-score detection frame; 7, calculating the similarity between the detection frame and the track, and performing matching; step 8, updating Kalman filtering parameters of the track; and 9, updating the appearance characteristics of the track.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of multi-target detection and tracking, and in particular to a detection and tracking system and method used in sports scene analysis. Background Art

[0002] In the analysis of modern sports scenes, athlete detection and tracking technology plays a key role. Traditional athlete detection and tracking systems first use detectors to detect targets in each frame to generate detection boxes. Then, the system extracts information about the detected athletes, including position information and appearance information, and evaluates the similarity of this information with the information of the previous trajectory to achieve tracking association. Common matching features include the intersection over union (IOU) of the trajectory prediction box and the detection box and the athlete re-identification feature (REID). The prediction box predicts the position of the athlete at time t based on the athlete's position at time t-1, and the Kalman filter is usually used to model the athlete's motion trajectory. The Kalman filter assumes that the time interval between the previous and next frames is short and the athlete's motion conforms to the linear assumption, so as to predict the position of the next frame of the trajectory, and then use the IOU as the cost function to achieve greedy matching of the prediction box and the detection box through the Hungarian algorithm. The athlete's appearance information, as a key feature to improve tracking continuity, is particularly important when the target frequently leaves the field of view, because when the target reappears, the appearance information can help the target match back to the original trajectory and reduce the number of identity identifier switches (IDS).

[0003] However, the movement of athletes is usually irregular and nonlinear, which means that athletes may experience large speed changes in a short period of time, which makes it easy to lose track when using Kalman filtering. In addition, in competitive sports, frequent physical contact and occlusion further increase the difficulty of tracking. OC-SORT calculates virtual trajectories during occlusions to correct the accumulated errors of Kalman filter noise and velocity estimation to prevent the re-matched trajectory from being lost again. Deep-EIoU directly abandons the use of Kalman filters and proposes iteratively extended intersection over union (EIoU) to solve this problem. EIoU expands the size of the detection box to handle the large movement changes of athletes between adjacent frames, and combines REID features to perform multi-level matching of detections and trajectories to improve the stability and accuracy of athlete tracking.

[0004] However, the effectiveness of the Deep-EIoU model depends largely on the effective extraction of REID features, which requires clear, high-quality image input, which is particularly difficult in real sports scenes. In competitive sports such as football, basketball, or track and field, the high-speed movement and frequent changes of direction of athletes can easily lead to blurred images, thus affecting the accuracy of appearance information. In addition, physical contact and occlusion between athletes, as well as camera shake and zoom, further increase the difficulty of feature extraction.

[0005] Therefore, technicians in this field are committed to developing a detection and tracking system and method that is more suitable for sports scenes to solve the problems of lost tracking and mismatching that are prone to occur in the prior art. Summary of the invention

[0006] In view of the above-mentioned defects of the prior art, the technical problem to be solved by the present invention is how to accurately detect and track in motion scenes with high-speed motion and frequent changes of direction, and solve the problems of lost tracking and wrong matching caused by blurred detection frames.

[0007] To achieve the above-mentioned object, the present invention provides a detection and tracking system for use in sports scene analysis, comprising a camera module, a target detection module, an appearance feature extraction module, a prediction module, a motion camera compensation module, a high-score matching module, a position feature extraction module, a low-score matching module, an update module, and a track maintenance module; The camera module is the main sensor module for the entire system to obtain external environment information, and provides real-time athlete position and movement information by capturing video image data; The target detection module is connected to the camera module, processes the acquired video image data, and generates a detection frame; The appearance feature extraction module is connected to the target detection module, performs high-score detection on the detection frames with confidence greater than 0.6, i.e., high-score detection frames, extracts the appearance of the athlete, and then uses a convolutional neural network to extract preliminary features, and inputs the extracted preliminary features into the Transformer module to capture the dependency relationship between different parts of the athlete's body; The prediction module is connected to the camera module, receives the video image data and the trajectory information of the t-1 frame, and generates a trajectory estimate of the tth frame; The motion camera compensation module is connected to the prediction module to repair the inconsistency problem between the video image data and the trajectory information between two frames; The high-score matching module is connected to the appearance feature extraction module and the motion camera compensation module, calculates the appearance similarity and spatial similarity between the appearances obtained by the high-score detection and the trajectory estimation, and performs Hungarian matching to obtain a final allocation result; for the high-score detection frame that has not been successfully matched, it is directly initialized as a new trajectory, and the appearance features of the new trajectory are initialized using the appearance features of the high-score detection frame; The position feature extraction module is connected to the target detection module and the high score matching module, performs low score detection on the detection frame with a confidence less than 0.6 and greater than 0.2, i.e., the low score detection frame, generates a preliminary position feature embedding using the position information and shape of the detection frame, and updates the position feature embedding in combination with the relative position between the athlete and the surrounding athletes; The low-score matching module is connected to the position feature extraction module, calculates the similarity between the low-score detection frame and the remaining tracks in the high-score matching module, and performs Hungarian matching to obtain the final matching result. The low-score detection frame that is not successfully matched is directly discarded; The updating module is connected to the high-score matching module and the low-score matching module, and updates the Kalman filter parameters of the trajectory after successful matching, so as to facilitate a more accurate position estimation later; The trajectory maintenance module is connected to the update module. For the trajectory matched in the high-score matching module, the appearance features of the corresponding high-score detection are used to update the appearance features of the trajectory; for the trajectory matched in the low-score matching module, the current appearance features are used; for the new trajectory, it is determined whether the new trajectory matches the detection in the next frame. If it matches, the new trajectory is converted into a valid trajectory. If it does not match, the new trajectory is directly discarded.

[0008] Furthermore, the video detection model adopted by the target detection module is YOLOv7, and the prediction box generated by the prediction module is used to generate a region of interest in the video image data, thereby giving the region of interest a greater attention weight during the target detection process.

[0009] Furthermore, the convolutional neural network used in the appearance feature extraction module is a pre-trained model similar to ResNet.

[0010] Furthermore, the preliminary position feature embedding in the position feature extraction module is recorded as , can be done by directly applying Enter the position and shape of the detection box in the formula as follows: When two athletes and Euclidean distance in a graph Less than the set threshold When , the two are considered adjacent, where the adjacency matrix is ​​defined as follows: .

[0011] Furthermore, in the position feature extraction module, the relative position between the athlete and surrounding athletes is obtained by calculating the relative distance and angle between adjacent detection frames to further enrich the preliminary position feature embedding; Use SplineCNN to complete the relative position information fusion and generate the final position feature embedding , the formula is as follows: .

[0012] Furthermore, the prediction module and the update module use a modified Kalman filter model for prediction and update.

[0013] Furthermore, the motion camera compensation algorithm adopted by the motion camera compensation module is a sparse optical flow method.

[0014] Furthermore, in the trajectory maintenance module, for the valid trajectory that is not matched in the current frame, the number of consecutive misses of the valid trajectory will be increased. If the next frame matches the detection frame, the number of consecutive misses will be cleared. If the number of consecutive misses exceeds 60, the valid trajectory will be directly discarded.

[0015] The present invention also provides a detection and tracking method for sports scene analysis, comprising the following steps: Step 1: The camera module captures video image data and provides real-time athlete position and movement information; Step 2: The prediction module receives the trajectory obtained from the previous round of matching, estimates the position of the trajectory in the current frame, generates a prediction box, and sends the position information of the prediction box to the target detection module and the high-score matching module; Step 3, the target detection module receives the video image data and the position information of the prediction frame, processes the current frame image information, identifies the position information of the athlete, generates a detection frame, and divides the detection frame into a high-score detection frame and a low-score detection frame according to the detection confidence, and sends them to the appearance feature extraction module and the position feature extraction module respectively; Step 4: After receiving the high-score detection frame, the appearance feature extraction module extracts the appearance features of the athlete and sends them together with the athlete's position information to the high-score matching module; Step 5, the high-score matching module receives the appearance features and position information of the athlete and the appearance features of the trajectory and the position information of the prediction box to calculate the similarity between the detection box and the trajectory, and uses the Hungarian algorithm for greedy matching. The matched trajectory is sent to the update module; the unmatched detection is initialized as a new trajectory and directly sent to the trajectory maintenance module, and the unmatched trajectory is sent to the position feature extraction module; Step 6: The position feature extraction module receives the low-score detection frame and the unmatched trajectory, extracts the position, shape and surrounding information of the low-score detection frame, generates position features, and sends them to the low-score detection module; Step 7, the low-score detection module receives the position features and original position information of the low-score detection frame and the unmatched trajectory to calculate the similarity between the detection frame and the trajectory and perform matching. The matched trajectory is sent to the update module, the unmatched detection is directly discarded, and the unmatched trajectory is directly sent to the trajectory maintenance module; Step 8, the updating module receives the matched trajectory, updates the Kalman filter parameters of the trajectory, and sends the updated trajectory to the trajectory maintenance module; Step 9: The trajectory maintenance module receives all trajectory information, updates the appearance features of the trajectory, discards useless trajectories, and sends the trajectory information to the prediction module.

[0016] Furthermore, the step 1 includes the following sub-steps: Step 1.1, select the central axis position (i.e. the position with the largest detection angle) in the front, back, left, right and four directions of the sports venue, and install the sports pan / tilt and camera; Step 1.2, wait for the game to start, power on the sports pan / tilt and the camera, start collecting the video image data, provide real-time player position and action information, and send it to the target detection module.

[0017] The detection and tracking system and method provided by the present invention for use in sports scene analysis have at least the following technical effects: 1. The technical solution provided by the present invention proposes YOLOv7-V, which uses the prediction module to predict the position of the current frame based on the trajectory generated by the previous frame, and uses the prediction box of the trajectory to generate a region of interest in the graph, so as to help the backbone network of YOLOv7 to better extract the features of the athletes, thereby overcoming the problem that the traditional target detection model of sports scenes is often not stable enough when dealing with the situation of rapid movement and frequent occlusion of the target, lacks the use of temporal information, is easily disturbed by background changes, has high computational complexity, and may miss the detection of athletes or mistakenly detect spectators as athletes.

[0018] 2. For low-score detection frames, the technical solution provided by the present invention proposes a position feature extraction module. When the appearance features are unreliable, the final match is completed by calculating the similarity of the position, shape, confidence and surrounding information between the detection frame and the prediction frame, thereby effectively overcoming the problem that the traditional tracking model relies too much on the appearance information of the athlete in sports scenes and cannot obtain effective appearance features when the athlete's posture and direction change rapidly, is occluded or motion blurred. For low-score detection caused by occlusion or motion blur, the traditional tracking model only measures the similarity by calculating the intersection-over-union ratio between the low-score detection frame and the trajectory, which is prone to loss or wrong matching.

[0019] The concept, specific structure and technical effects of the present invention will be further described below in conjunction with the accompanying drawings to fully understand the purpose, characteristics and effects of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 A structural flow chart of a preferred embodiment of the present invention for performing detection and tracking tasks in a sports scene; Figure 2 for Figure 1 The structural flow chart of YOLOv7-V of the illustrated embodiment; Figure 3 for Figure 1 A structural flow chart of the position feature extraction module of the illustrated embodiment. DETAILED DESCRIPTION

[0021] The following describes several preferred embodiments of the present invention with reference to the drawings in the specification, so that the technical content is clearer and easier to understand. The present invention can be embodied in many different forms of embodiments, and the protection scope of the present invention is not limited to the embodiments mentioned in the text.

[0022] Accurately detect and track in sports scenes with high-speed motion and frequent changes of direction, and solve the problems of lost tracking and wrong matching caused by blurred detection frames. To achieve the above purpose, an embodiment of the present invention provides a detection and tracking system for use in sports scene analysis, which first detects sports videos, and then divides the obtained detection frames into high-score and low-score detections according to the confidence of the detection frames, so as to adopt different processing strategies for detection frames of different qualities. For high-score clear detection, an embodiment of the present invention uses the extracted appearance features of athletes to perform greedy matching of detection frames and trajectory prediction frames; for low-score blurred detection, an embodiment of the present invention uses a contrastive learning method to generate feature embeddings of the extracted surrounding information of athletes and the position shape of the detection frame, so as to achieve more stable matching. Combined with the motion camera compensation algorithm, the appearance features and position features of athletes, the problem of athlete tracking in the case of occlusion and motion blur is effectively solved.

[0023] Example 1 like Figure 1 As shown, an embodiment of the present invention provides a detection and tracking system for sports scene analysis, including a camera module, a target detection module, an appearance feature extraction module, a prediction module, a motion camera compensation module, a high-score matching module, a position feature extraction module, a low-score matching module, an update module, and a track maintenance module; Among them, the camera module is the main sensor module for the entire system to obtain external environmental information. By capturing video image data, it provides real-time athlete position and movement information. Usually, a camera with high resolution, fast frame rate and low latency is required to ensure that the subtle movements and rapid changes of athletes can be captured. At night or in indoor venues, the camera usually needs to have good low-light performance to ensure that it can still provide clear images when the light is insufficient. In order to cover a larger sports scene, the camera often needs to be able to rotate in order to fully monitor the entire venue. The embodiment of the present invention uses a sports gimbal equipped with a camera as a camera module.

[0024] The target detection module is connected to the camera module, processes the acquired video image data, and generates a detection frame. Its detection accuracy directly determines the level of subsequent tracking performance. In order to ensure efficient and accurate target detection, the embodiment of the present invention uses YOLOv7 as the main framework for improvement and proposes a model YOLOv7-V that is more suitable for video detection. While maintaining a faster reasoning speed, it enhances its stability in video detection, such as Figure 2 As shown in the figure. YOLOv7 is one of the latest versions of the YOLO series of target detection algorithms. The core idea of ​​the YOLO series is to transform the target detection problem into a classification problem of whether there is an object and a position regression problem. Compared with the previous two-stage method of first selecting candidate boxes from the generated anchor boxes, the YOLO series only achieves target positioning and classification through a single forward propagation, which greatly improves the detection speed. Many training techniques that only increase training time but not reasoning time are used, so that its performance can match some complex models, but the reasoning time is shorter. However, YOLOv7 is mainly used for image detection. When processing videos, it detects frame by frame. It lacks the use of temporal information, which makes it unstable when dealing with fast-moving and frequently occluded targets. To solve this problem, YOLOv7-V uses the tracking results of the previous frame to generate regions of interest in the image according to the prediction boxes generated by the prediction module, and gives these regions greater attention weights during the detection process.

[0025] The appearance feature extraction module is connected to the target detection module, and performs high-score detection on the detection frame with a confidence greater than 0.6, i.e., the high-score detection frame, to extract the appearance of the athlete. In order to quickly extract effective appearance features, the embodiment of the present invention first uses a convolutional neural network such as a pre-trained model such as ResNet to perform preliminary feature extraction on the athlete, and inputs the extracted features into the Transformer module to capture the dependencies between different parts of the athlete's body. This method can extract more stable and accurate features when dealing with complex backgrounds, multi-view changes, and fast-moving athletes.

[0026] The prediction module is connected to the camera module, receives the video image data and the trajectory information of the t-1 frame, and generates a trajectory estimate of the tth frame to facilitate subsequent matching with the detection frame.

[0027] The motion camera compensation module is connected to the prediction module to repair the inconsistency problem between the video image data and the trajectory information between the two frames; motion camera compensation is a technology used to reduce or eliminate the inconsistency problem between the two frames caused by camera motion. Common motion camera compensation algorithms include image registration algorithms based on enhanced correlation coefficients and optical flow methods. The image registration algorithm based on enhanced correlation coefficients calculates the correlation between the two images, determines the motion coefficient, and corrects the image to achieve motion compensation. It has high accuracy but complex calculations and is suitable for scenes that require high-precision registration; optical flow is divided into dense optical flow method and sparse optical flow method. The dense optical flow method needs to calculate the motion vector of each pixel, which has a large amount of calculation, while the sparse optical flow method only estimates the motion of the feature points in the image. These feature points are usually corner points or points on the edge line in the image, and the calculation efficiency is high. In sports scenes, since real-time is crucial, the embodiment of the present invention adopts the sparse optical flow method.

[0028] The high-score matching module connects the appearance feature extraction module and the motion camera compensation module. It is mainly used to handle the tracking and matching problems of the high-score detection frame. When the confidence of the detection frame is greater than 0.6, it can be considered that the detection is accurate and belongs to the high-score detection. Since the imaging quality of these detection frames is high, they can provide rich and reliable appearance features. By calculating the cosine similarity or Euclidean distance between the appearance features of the high-score detection and the trajectory, it can be determined whether the two are similar. Then, the IOU value of the two can be calculated to determine whether the two are similar in space. Subsequently, Hungarian matching is performed by combining appearance similarity and spatial similarity to obtain the final allocation result. For the high-score detection frame that has not been successfully matched in this matching stage, it is directly initialized as a new trajectory, and the appearance features of the detection frame are used to initialize the appearance features of the new trajectory.

[0029] The position feature extraction module connects the target detection module and the high-score matching module, performs low-score detection on the detection frames with confidence less than 0.6 and greater than 0.2, i.e., low-score detection frames, and generates preliminary position feature embedding using the position information and shape of the detection frames. The position embedding is further updated by combining the relative position between the athlete and the surrounding athletes, such as Figure 3 shown.

[0030] The low-score matching module is connected to the position feature extraction module, and is mainly used to handle the tracking and matching problem of the low-score detection frame. When the confidence of the detection frame is less than 0.6 and greater than 0.2, it can be considered that the detection is unreliable and belongs to a low-score detection. Low-score detection in motion scenes is often due to occlusion or motion blur due to movement, resulting in low detection confidence. This module measures whether the low-score detection frame and the remaining trajectory in the high-score matching stage are similar by calculating the Euclidean distance between the position features, and combines the IOU scores of the two to perform Hungarian matching to obtain the final matching result. The low-score detection frame that is not successfully matched in this matching stage is directly discarded.

[0031] The update module connects the high-score matching module and the low-score matching module to update the Kalman filter parameters of the trajectory after successful matching, so as to facilitate more accurate position estimation in the future.

[0032] The track maintenance module is connected to the update module. For tracks matched in the high-score matching module, the appearance features of the corresponding high-score detection are used to update the appearance features of the track. For tracks matched in the low-score matching module, the current appearance features are used. For new tracks, it is determined whether the new track matches the detection in the next frame. If it matches, the new track is converted into a valid track. If it does not match, the new track is directly discarded. In the track maintenance module, for valid tracks that are not matched in the current frame, the number of consecutive misses of the valid track will be increased. If the next frame matches the detection frame, the number of consecutive misses will be cleared. If the number of consecutive misses exceeds 60, the valid track will be directly discarded.

[0033] Example 2 Based on Example 1, the preliminary position feature embedding in the position feature extraction module is recorded as , can be done by directly applying The position and shape of the input detection box are generated by the following formula: In order to better extract the surrounding information of the athletes, when two athletes and Euclidean distance in a graph Less than the set threshold When , the two are considered adjacent, where the adjacency matrix is ​​defined as follows: .

[0034] In the position feature extraction module, the relative distance and angle between adjacent detection frames are calculated to obtain the relative position between the athlete and the surrounding athletes to further enrich the preliminary position feature embedding; Relative position information can be used to further enrich position feature embedding by calculating the relative distance and angle between adjacent detection boxes. This module uses SplineCNN to complete the relative position information fusion and generate the final position feature embedding. , the formula is as follows: .

[0035] Example 3 On the basis of embodiments 1 and 2, the update module updates the Kalman filter parameters of the trajectory after successful matching, so as to facilitate more accurate position estimation later. The embodiment of the present invention uses a modified Kalman filter model for prediction and updating.

[0036] Traditional Kalman filters usually assume that system noise and observation noise are Gaussian distributed, but in practical applications, this assumption is often not true. The embodiment of the present invention introduces a more flexible noise modeling method, so that the Kalman filter can more accurately estimate the target state when processing nonlinear moving targets.

[0037] Secondly, the embodiment of the present invention optimizes the update process of the Kalman filter. The traditional Kalman filter adjusts the state estimate according to the difference between the predicted value and the observed value each time the state is updated, but this process does not respond promptly when facing a rapidly changing moving target. The embodiment of the present invention introduces a dynamic adjustment mechanism, which enables the filter to adapt to the changes of the target more quickly, thereby improving the real-time performance and accuracy of tracking.

[0038] Example 4 The embodiment of the present invention provides a detection and tracking method for sports scene analysis, which is as follows: First, camera modules are arranged in the sports venue to record the game in real time, and the trajectories obtained from the previous round of matching are sent to the prediction module to generate prediction frames. During the recording process, the target detection module is used to combine the prediction frames of the previous round of trajectories for real-time detection to obtain the position and confidence of the athlete's detection frame. According to the detection confidence, it is divided into high-score detection frames and low-score detection frames. The high-score detection frames are sent to the appearance feature extraction module. After obtaining the appearance features, they are sent to the high-score matching module together with all trajectories for matching. Then the low-score detection frames and the trajectories that were not matched in the previous step are sent to the position feature extraction module. After obtaining the position features, they are sent to the low-score matching module for matching. Finally, the matching results of the two modules are sent to the trajectory maintenance module to maintain the trajectory information, and then sent to the update module to update the trajectory status.

[0039] The specific steps include: Step 1: The camera module captures video image data and provides real-time athlete position and movement information; Step 2: The prediction module receives the trajectory obtained from the previous round of matching, estimates the position of the trajectory in the current frame, generates a prediction box, and sends the position information of the prediction box to the target detection module and the high-score matching module; Step 3: The target detection module receives the video image data and the position information of the prediction box, processes the current frame image information, identifies the position information of the athlete, generates a detection box, and divides the detection box into a high-score detection box and a low-score detection box according to the detection confidence, and sends them to the appearance feature extraction module and the position feature extraction module respectively; Step 4: After receiving the high-score detection frame, the appearance feature extraction module extracts the appearance features of the athlete and sends them together with the athlete's position information to the high-score matching module; Step 5: The high-score matching module receives the appearance features and position information of the athlete, the appearance features of the trajectory and the position information of the prediction box, calculates the similarity between the detection box and the trajectory, and uses the Hungarian algorithm for greedy matching. The matched trajectories are sent to the update module; the unmatched detections are initialized as new trajectories and directly sent to the trajectory maintenance module, and the unmatched trajectories are sent to the position feature extraction module; Step 6: The position feature extraction module receives the low-score detection frame and the unmatched trajectory, extracts the position, shape and surrounding information of the low-score detection frame, generates position features, and sends them to the low-score detection module; Step 7: The low-score detection module receives the position features and original position information of the low-score detection frame and the unmatched trajectory to calculate the similarity between the detection frame and the trajectory and perform matching. The matched trajectory is sent to the update module, the unmatched detection is directly discarded, and the unmatched trajectory is directly sent to the trajectory maintenance module; Step 8: The update module receives the matched trajectory, updates the Kalman filter parameters of the trajectory, and sends the updated trajectory to the trajectory maintenance module; Step 9: The trajectory maintenance module receives all trajectory information, updates the appearance features of the trajectory, discards useless trajectories, and sends the trajectory information to the prediction module.

[0040] In particular, step 1 includes the following sub-steps: Step 1.1, select the central axis position (i.e. the position with the largest detection angle) in the front, back, left, right and four directions of the sports venue, and install the sports pan / tilt and camera; Step 1.2: Wait for the game to start, power on the motion gimbal and camera, start collecting video image data, provide real-time athlete position and movement information, and send it to the target detection module.

[0041] The preferred specific embodiments of the present invention are described in detail above. It should be understood that ordinary technicians in the field can make many modifications and changes based on the concept of the present invention without creative work. Therefore, all technical solutions that can be obtained by technicians in the technical field based on the concept of the present invention through logical analysis, reasoning or limited experiments on the basis of the prior art should be within the scope of protection determined by the claims.

Claims

1. A detection and tracking system for use in sports scene analysis, characterized in that: The system includes a camera module, a target detection module, an appearance feature extraction module, a prediction module, a motion camera compensation module, a high-score matching module, a position feature extraction module, a low-score matching module, an update module, and a trajectory maintenance module; The camera module is the main sensor module for the entire system to obtain external environment information, and provides real-time athlete position and movement information by capturing video image data; The target detection module is connected to the camera module, processes the acquired video image data, and generates a detection frame; The appearance feature extraction module is connected to the target detection module, performs high-score detection on the detection frames with confidence greater than 0.6, i.e., high-score detection frames, extracts the appearance of the athlete, and then uses a convolutional neural network to extract preliminary features, and inputs the extracted preliminary features into the Transformer module to capture the dependency relationship between different parts of the athlete's body; The prediction module is connected to the camera module, receives the video image data and the trajectory information of the t-1 frame, and generates a trajectory estimate of the tth frame; The motion camera compensation module is connected to the prediction module to repair the inconsistency problem between the video image data and the trajectory information between two frames; The high-score matching module is connected to the appearance feature extraction module and the motion camera compensation module, calculates the appearance similarity and spatial similarity between the appearances obtained by the high-score detection and the trajectory estimation, and performs Hungarian matching to obtain a final allocation result; for the high-score detection frame that has not been successfully matched, it is directly initialized as a new trajectory, and the appearance features of the new trajectory are initialized using the appearance features of the high-score detection frame; The position feature extraction module is connected to the target detection module and the high score matching module, performs low score detection on the detection frame with a confidence less than 0.6 and greater than 0.2, i.e., the low score detection frame, generates a preliminary position feature embedding using the position information and shape of the detection frame, and updates the position feature embedding in combination with the relative position between the athlete and the surrounding athletes; The low-score matching module is connected to the position feature extraction module, calculates the similarity between the low-score detection frame and the remaining tracks in the high-score matching module, and performs Hungarian matching to obtain the final matching result. The low-score detection frame that is not successfully matched is directly discarded; The updating module is connected to the high-score matching module and the low-score matching module, and updates the Kalman filter parameters of the trajectory after successful matching, so as to facilitate a more accurate position estimation later; The trajectory maintenance module is connected to the update module. For the trajectory matched in the high-score matching module, the appearance features of the corresponding high-score detection are used to update the appearance features of the trajectory; for the trajectory matched in the low-score matching module, the current appearance features are used; for the new trajectory, it is determined whether the new trajectory matches the detection in the next frame. If it matches, the new trajectory is converted into a valid trajectory. If it does not match, the new trajectory is directly discarded.

2. The detection and tracking system for sports scene analysis as claimed in claim 1, characterized in that: The video detection model adopted by the target detection module is YOLOv7, and the prediction box generated by the prediction module is used to generate a region of interest in the video image data, thereby giving the region of interest a greater attention weight during the target detection process.

3. The detection and tracking system for use in sports scene analysis as claimed in claim 1, characterized in that: The convolutional neural network used in the appearance feature extraction module is a pre-trained model similar to ResNet.

4. The detection and tracking system for use in sports scene analysis as claimed in claim 1, characterized in that: The preliminary position feature embedding in the position feature extraction module is denoted as , can be done by directly applying Enter the position and shape of the detection box in the formula as follows: When two athletes and Euclidean distance in graph Less than the set threshold When , the two are considered adjacent, where the adjacency matrix is ​​defined as follows: 。 5. The detection and tracking system for use in sports scene analysis as claimed in claim 4, characterized in that: In the position feature extraction module, the relative position between the athlete and surrounding athletes is obtained by calculating the relative distance and angle between adjacent detection frames to further enrich the preliminary position feature embedding; Use SplineCNN to complete the relative position information fusion and generate the final position feature embedding , the formula is as follows: 。 6. The detection and tracking system for use in sports scene analysis as claimed in claim 1, characterized in that: The prediction module and the update module use a modified Kalman filter model to perform prediction and update.

7. The detection and tracking system for use in sports scene analysis as claimed in claim 1, characterized in that: The motion camera compensation algorithm adopted by the motion camera compensation module is a sparse optical flow method.

8. The detection and tracking system for use in sports scene analysis as claimed in claim 1, characterized in that: In the trajectory maintenance module, for the valid trajectory that is not matched in the current frame, the number of consecutive misses of the valid trajectory will be increased. If the next frame matches the detection frame, the number of consecutive misses will be cleared. If the number of consecutive misses exceeds 60, the valid trajectory will be directly discarded.

9. A detection and tracking method for sports scene analysis, characterized in that: The method comprises the following steps: Step 1: The camera module captures video image data and provides real-time athlete position and movement information; Step 2: The prediction module receives the trajectory obtained from the previous round of matching, estimates the position of the trajectory in the current frame, generates a prediction box, and sends the position information of the prediction box to the target detection module and the high-score matching module; Step 3, the target detection module receives the video image data and the position information of the prediction frame, processes the current frame image information, identifies the position information of the athlete, generates a detection frame, and divides the detection frame into a high-score detection frame and a low-score detection frame according to the detection confidence, and sends them to the appearance feature extraction module and the position feature extraction module respectively; Step 4: After receiving the high-score detection frame, the appearance feature extraction module extracts the appearance features of the athlete and sends them together with the athlete's position information to the high-score matching module; Step 5, the high-score matching module receives the appearance features and position information of the athlete and the appearance features of the trajectory and the position information of the prediction box to calculate the similarity between the detection box and the trajectory, and uses the Hungarian algorithm for greedy matching. The matched trajectory is sent to the update module; the unmatched detection is initialized as a new trajectory and directly sent to the trajectory maintenance module, and the unmatched trajectory is sent to the position feature extraction module; Step 6: The position feature extraction module receives the low-score detection frame and the unmatched trajectory, extracts the position, shape and surrounding information of the low-score detection frame, generates position features, and sends them to the low-score detection module; Step 7, the low-score detection module receives the position features and original position information of the low-score detection frame and the unmatched trajectory to calculate the similarity between the detection frame and the trajectory and perform matching. The matched trajectory is sent to the update module, the unmatched detection is directly discarded, and the unmatched trajectory is directly sent to the trajectory maintenance module; Step 8, the updating module receives the matched trajectory, updates the Kalman filter parameters of the trajectory, and sends the updated trajectory to the trajectory maintenance module; Step 9: The trajectory maintenance module receives all trajectory information, updates the appearance features of the trajectory, discards useless trajectories, and sends the trajectory information to the prediction module.

10. The detection and tracking method for sports scene analysis as claimed in claim 9, characterized in that: The step 1 includes the following sub-steps: Step 1.1, select the central axis position (i.e. the position with the largest detection angle) in the four directions of the front, back, left, and right of the sports venue, and install the sports pan / tilt and camera; Step 1.2, wait for the game to start, power on the sports pan / tilt and the camera, start collecting the video image data, provide real-time player position and action information, and send it to the target detection module.

Citation Information

Cited By

  • Target tracking method based on camera motion compensation

    CN120726094A

  • Football match video processing analysis method and system based on target detection model

    CN122157075A