A multi-target tracking matching method in sports video based on target nomination replacement

Abstract

The present application relates to a kind of multi-target tracking matching method in sports video based on target nomination replacement, belong to target detection technical field, solve the problem of repeated detection and target loss in sports video in the multi-target tracking method in prior art.The present application uses Hungarian algorithm to construct matching loss matrix multiple times, constantly updates the target matching quality of adjacent frame, finally obtains the optimal matching scheme, can reduce the problem of repeated detection, in the case where the appearance of player in video is similar, single target without occlusion or coincidence will not be judged as appearing occlusion or coincidence, multiple bounding boxes will not be output for the single target.Also can reduce the target loss caused by mutual occlusion and fast movement of player in sports video.

Description

Technical Field

[0001] This invention belongs to the field of target detection technology, specifically relating to a multi-target tracking and matching method in sports videos based on target nomination permutation. Background Technology

[0002] Compared to pedestrian tracking in general surveillance videos, multi-target (e.g., athlete) tracking in sports videos presents some unique challenges, such as... Figure 1-2 As shown: First, the uniformity of athletes' uniforms can be considered as multiple targets having extremely similar appearances, leading to similar distributions of visual features among different targets, which can cause duplicate detection problems. Second, multiple athletes frequently run, resulting in frequent occlusion phenomena, and athletes move faster than pedestrians, exacerbating the target loss problem. If both of these situations occur simultaneously, existing multi-target tracking methods will face significant challenges. Multi-target tracking in sports videos can obtain athletes' movement information. For competitors, this can effectively assist in analyzing the competitive habits of athletes of interest, thereby developing targeted strategies; for viewers, it can provide a multi-faceted view of the competition dynamics and athlete statistics.

[0003] Existing multi-object tracking methods mostly address the problem of tracking multiple pedestrians in surveillance videos. In this scenario, although the number of targets is large, the camera's perspective is fixed, the targets have certain differences in appearance, and their movement speed is relatively slow, allowing multi-object tracking methods to achieve good tracking results. However, they still have certain limitations. Currently, there is very little research specifically on multi-object tracking for sports videos. Even if existing methods can train models based on sports videos to address the aforementioned challenges, they lack a reasonable design for the sports video scenario, so existing multi-object tracking methods are generally not directly applicable to sports video scenarios. Summary of the Invention

[0004] In view of the above problems, the present invention provides a multi-object tracking and matching method in sports videos based on target nomination permutation, which solves the problems of repeated detection and target loss in existing multi-object tracking methods in sports videos.

[0005] This invention provides a multi-target tracking and matching method in sports videos based on target nomination permutation, the specific steps of which are as follows:

[0006] S1, perform target detection on two adjacent frames of the sports video to obtain the tracking result of the previous frame and the detection result of the next frame; the detection result of the next frame includes the initial matching target nomination set and the nomination substitute set;

[0007] S2, construct the first matching loss matrix based on the bounding box coordinates of the first matching target nomination set and the target nominations in the tracking results of the previous frame;

[0008] S3, based on the first-match loss matrix, obtain multiple matching values ​​of the target nominations in the first-match target nomination set between two adjacent frames in the sports video;

[0009] S4, determine a matching value obtained in step S3. If the matching value is a bad match, the target nomination in the first matching target nomination set corresponding to the bad match is a low-matching-quality target nomination. Select a substitute nomination from the nomination substitute set and replace the low-matching-quality target nomination corresponding to the bad match with the selected substitute nomination to obtain the first generation matching result of the next frame and proceed to step S5. If all matching values ​​obtained in step S3 are good matches, the target nomination in the first matching target nomination set corresponding to each good match is a high-matching-quality target nomination. Obtain the high-matching-quality target nomination set and end the multi-target tracking matching of the images of the two adjacent frames in the sports video to obtain the tracking matching result of the next frame.

[0010] S5. Construct the nth generation matching loss matrix based on the nth generation matching result of the next frame and the bounding box coordinates of the target nomination in the tracking result of the previous frame, n=1;

[0011] S6, based on the nth generation matching loss matrix, obtain multiple matching values ​​of the target nomination in the nth generation matching result of the next frame in the sports video between two adjacent frames;

[0012] S7. Determine a matching value obtained in step S6. If the matching value is a bad match, the target nomination in the nth generation matching result of the next frame corresponding to the bad match is a low-match quality target nomination. Select a substitute nomination from the nomination substitute set and replace the low-match quality target nomination corresponding to the bad match with the selected substitute nomination to obtain the (n+1)th generation matching result of the next frame. Let n = n+1 and return to step S5. If all the matching values ​​are good matches, the target nomination in the nth generation matching result of the next frame corresponding to each good match is a high-match quality target nomination. Obtain the high-match quality target nomination set and end the multi-target tracking matching of the images of the two adjacent frames in the sports video to obtain the tracking matching result of the next frame.

[0013] Optionally, a multi-target tracking network model can be used to obtain the tracking result of the previous frame and the detection result of the next frame in step S1;

[0014] Specifically, the training set of the historical sports video database is used to label and track multiple targets in the historical sports videos to obtain the tracking label for each target; based on all the tracking labels, a multi-target tracking model is trained using deep learning to obtain a multi-target tracking network model.

[0015] Optionally, the multi-target tracking model includes a multilayer perceptron; the learning steps of the multilayer perceptron are as follows:

[0016] The expression for obtaining a multilayer perceptron is:

[0017] Y = W MLP (X);

[0018] Where Y is the output target bounding box; X is the input single-frame image; W MLP These are learnable parameters;

[0019] The deviation between the target bounding box Y output by the multilayer perceptron and the target's tracking label is used to pass the learnable parameters W. MLP The trainable parameter W MLP This makes the output target bounding box Y approximate the target's tracking label.

[0020] Optionally, the expression for the Track_Box of the previous frame obtained in step S1 is:

[0021] rack_Box = {P1, ..., P} M};

[0022] Among them, P m Let m represent the m-th target nomination in the previous frame of the sports video, where m = 1, 2, ..., M, and M represents the total number of target nominations in the previous frame of the sports video.

[0023] Optionally, the detection results of the next frame obtained in step S1 are sorted in descending order of target confidence; the top N target nominations for the next frame that exceed the target confidence threshold are extracted, and the expression for each target nomination in the next frame is:

[0024] P i ={Score i ,x 1,i ,y 1,i ,x 2,i ,y 2,i};

[0025] Among them, P i Score represents the i-th target nomination in the next frame, where i = 1, 2, ..., N, and N is the total number of target nominations extracted in the next frame; i x represents the target confidence score of the target nomination in the i-th subsequent frame; 1,i and y 1,i x represents the coordinates of the top-left corner of the bounding box of the target nomination in the i-th subsequent frame. 2,i and y 2,i This represents the coordinates of the bottom right corner of the bounding box of the target nomination in the i-th subsequent frame;

[0026] Take the first K target nominations from the N target nominations in the next frame as the first matching target nominations, and obtain the first matching target nomination set Lineup_Det_Box={P1,…,P K}; The other NK target nominations in the next frame are used as substitute nominations, resulting in a nomination substitute set Substitution = {P} K+1 ,…,P N}

[0027] Optionally, the expression for the initial matching loss matrix Match_Cost in step S2 is:

[0028] Match_Cost=1–GIoU(Lineup_Det_Box,Track_Box)

[0029] Wherein, GIoU represents the general crossover ratio;

[0030] The expression for the general intersection-union ratio (CIU) of GIoU is:

[0031]

[0032] Where L represents the rectangular region of the first matching target nomination set in the next frame; T represents the rectangular region of the tracking result in the previous frame; C represents the smallest rectangular region L∪T that can cover the rectangular region of the first matching target nomination set in the next frame and the rectangular region of the tracking result in the previous frame; \ represents the difference; IoU represents the intersection-union ratio.

[0033] Optionally, in step S3 and / or step S6, the Hungarian matching algorithm is used to obtain multiple matching values ​​of the target nomination's target box coordinates in two adjacent frames of the sports video.

[0034] Optionally, in steps S4 and / or S7, if the matching value is too large or two matching values ​​are very close, the matching value is considered an outlier.

[0035] Optionally, in step S4 and / or step S7, if the matching value exceeds the matching value threshold, the matching value is considered an outlier.

[0036] Optionally, the following steps are also included:

[0037] SS1. Use the aforementioned method to obtain the tracking and matching result of the w-th frame, where w = 2;

[0038] SS2. Obtain the detection result of frame w+1;

[0039] SS3. Use the Hungarian algorithm to match the tracking matching result of frame w with the detection result of frame w+1 to obtain the tracking matching result of frame w+1.

[0040] SS4. If w < W, where W is the total number of frames of the sports video and W ≥ 3, let w = w + 1, and return to step SS2; if w ≥ W, use the tracking and matching result of the (w + 1)-th frame as the total target tracking and matching result of the sports video.

[0041] Compared with the prior art, the present invention has at least the following beneficial effects:

[0042] (1) The method of the present invention can reduce the problem of repeated detection. In the case where the appearances of athletes in the video are similar, it will not determine an unoccluded or non-overlapped single target as occluded or overlapped, and will not output multiple bounding boxes for this single target.

[0043] (2) The method of the present invention can reduce the situation of target loss in sports videos caused by mutual occlusion and rapid movement of athletes.

[0044] (3) The method of the present invention utilizes the upper bound of the number of athletes in the sports video and only retains the number of target nominations below the upper bound, effectively filtering out interfering nominations.

[0045] (4) The method of the present invention simulates the rule of replacing athletes during the competition process in the sports video, replaces the target nominations with low matching quality (such as athletes in a slump) with nominations of high matching quality (such as athletes in high spirits), and improves the multi-target tracking quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The drawings are only for the purpose of illustrating specific embodiments and are not considered to be a limitation of the present invention.

[0047] Figure 1 It is a schematic diagram of multi-target tracking in a sports video;

[0048] Figure 2 It is a schematic diagram of the problems of repeated detection and target loss during multi-target tracking in a sports video;

[0049] Figure 3 It is a flowchart of the multi-target tracking and matching method of the present invention;

[0050] FIG. 4(a) and (b) are schematic diagrams of two adjacent frames and a matching loss matrix in the sports video of the present invention;

[0051] FIG. 5(a) and (b) are schematic diagrams of the tracking result after replacement and substitution and a new matching loss matrix of the present invention;

[0052] Figure 6 It is an effect diagram showing that when taking the top 12 target nominations by using the method of the present invention, the redundant nominations (left) are filtered out (right);

[0053] Figure 7To nominate replacements using the method of this invention, ID_35 and ID_44 are repeatedly detected (left), and one of them is replaced with substitute ID_1 (right).

[0054] Figure 8 This is a diagram illustrating the adaptability of the present invention in basketball videos.

[0055] Figure 9 This is a diagram illustrating the adaptive effect of the present invention in football videos. Detailed Implementation

[0056] To better understand the above-described objectives, features, and advantages of the present invention, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments of the present invention and the features thereof can be combined with each other. Furthermore, the present invention can be implemented in other ways different from those described herein; therefore, the scope of protection of the present invention is not limited to the specific embodiments disclosed below.

[0057] A specific embodiment of the present invention, such as Figure 1-9 This paper discloses a multi-object tracking and matching method in sports videos based on target nomination permutation. The specific steps are as follows:

[0058] S1, perform target detection on two adjacent frames of the sports video to obtain the tracking result of the previous frame and the detection result of the next frame; the detection result of the next frame includes the initial matching target nomination set and the nomination substitute set;

[0059] Optionally, the multi-target tracking network model TransTrack is used to obtain the tracking results of the previous frame and the detection results of the next frame in step S1;

[0060] Specifically, the training set of the historical sports video database RallyTrack is used to label and track multiple targets in historical sports videos to obtain the tracking label for each target; based on all tracking labels, a multi-target tracking model is trained using deep learning to obtain the multi-target tracking network model TransTrack.

[0061] Optionally, the multi-target tracking model includes an encoder, a decoder, and a multilayer perceptron (MLP).

[0062] Optionally, the learning steps of a multilayer perceptron (MLP) are as follows:

[0063] The expression for obtaining a multilayer perceptron (MLP) is:

[0064] Y = W MLP (X);

[0065] Where Y is the output target bounding box; X is the input single-frame image; W MLP These are learnable parameters;

[0066] Furthermore, the learnable parameter W MLP In order to continuously adjust its own values ​​through backpropagation to fit the parameters of the true probability distribution.

[0067] The backpropagation mechanism is used to pass the deviation between the target bounding box Y output by the multilayer perceptron and the target's tracking label to the learnable parameters W. MLP This makes the learnable parameter W MLP It is continuously trained until the output Y eventually approximates the target's tracking label.

[0068] Optionally, the training method for other parameters such as the encoder and decoder is the same as that for the multilayer perceptron (MLP); after the multi-object tracking model is trained, the multi-object tracking network model TransTrack is obtained.

[0069] Specifically, the expression for Track_Box, which uses the multi-object tracking network model TransTrack to obtain the tracking result of the previous frame between two adjacent frames in a sports video, is as follows:

[0070] rack_Box = {P1, ..., P} M};

[0071] Among them, P m Let m represent the m-th target nomination in the previous frame of the sports video, where m = 1, 2, ..., M, and M represents the total number of target nominations in the previous frame of the sports video.

[0072] Specifically, in a sports video, the m-th target nomination P in the previous frame... m This includes target confidence, target bounding box coordinates, and target ID.

[0073] It is understandable that the target ID is the target's identity number.

[0074] The tracking result from the previous frame, Track_Box = {P1,…,P} M} as the basis for tracking and matching;

[0075] Specifically, the detection results of the next frame are sorted in descending order of target confidence; the top N target nominations for the next frame that exceed the target confidence threshold are extracted, and the expression for each target nomination in the next frame is as follows:

[0076] P i ={Score i ,x 1,i ,y 1,i,x 2,i ,y 2,i};

[0077] Among them, P i Score represents the i-th target nomination in the next frame, where i = 1, 2, ..., N, and N is the total number of target nominations extracted in the next frame; i x represents the target confidence score of the target nomination in the i-th subsequent frame; 1,i and y 1,i x represents the coordinates of the top-left corner of the bounding box of the target nomination in the i-th subsequent frame. 2,i and y 2,i This represents the coordinates of the bottom right corner of the bounding box of the target nomination in the i-th subsequent frame.

[0078] Take the first K target nominations from the N target nominations in the next frame as the first matching target nominations, and obtain the first matching target nomination set Lineup_Det_Box={P1,…,P K}; The other NK target nominations in the next frame are used as substitute nominations, resulting in a nomination substitute set Substitution = {P} K+1 ,…,P N}, used for replacement of low-match quality target nominations;

[0079] Optionally, K ≤ the upper bound of the number of targets in the sports video.

[0080] Understandably, the detection result of the next frame includes multiple target bounding boxes in the image of that frame, as well as the target nominations corresponding to each target bounding box; see [link to relevant documentation]. Figure 1-2 The target bounding box is a rectangular box that surrounds the target, and one target bounding box contains one target. The target nomination includes the target bounding box coordinates and the target confidence score. The target nomination is automatically generated by the multi-target tracking network model TransTrack. The target confidence score is the probability value that the target within the target bounding box is an athlete. The higher the target confidence score, the higher the quality of the target nomination corresponding to that target confidence score.

[0081] This invention sets a target confidence threshold to ensure the quality of the selected target nominations. At the same time, it uses the number of targets in sports videos to limit the upper bound, retaining only target nominations with a confidence greater than the target confidence threshold and below the limited upper bound as the first matching target nominations. This ensures that the tracked targets are locked to athletes and excludes non-on-field targets to be tracked.

[0082] Furthermore, the Hungarian matching algorithm is used to perform quality matching on target nominations in two adjacent frames of the sports video to obtain matching values. If the matching value is an outlier, the corresponding target nomination is a low-matching quality target nomination (i.e., a bad match). A target nomination from the next frame is selected from the nomination replacement set to replace the low-matching quality target nomination. The Hungarian algorithm is repeated until all target nominations are high-matching quality target nominations.

[0083] S2, construct the first matching loss matrix Match_Cost based on the bounding box coordinates of the target nominations in the Lineup_Det_Box of the first matching target nomination set in the next frame and the Track_Box of the tracking result in the previous frame;

[0084] The expression for the initial matching loss matrix Match_Cost is:

[0085] Match_Cost=1–GIoU(Lineup_Det_Box,Track_Box)

[0086] Wherein, GIoU represents the general crossover ratio;

[0087] The expression for the general intersection-union ratio (CIU) of GIoU is:

[0088]

[0089] Where L represents the rectangular region of the Lineup_Det_Box that is first matched with the target nomination set in the next frame; T represents the rectangular region of the Track_Box that is the tracking result in the previous frame; C represents the smallest rectangular region L∪T that can cover the rectangular regions of the Lineup_Det_Box that is first matched with the target nomination set in the next frame and the rectangular regions of the Track_Box that is the tracking result in the previous frame; \ represents the difference; IoU represents the intersection-union ratio.

[0090] It is understandable that the rectangular region is a rectangular region composed of the bounding boxes of multiple target nominations; the elements in the first matching loss matrix Match_Cost are the matching values ​​of the same target nomination in two adjacent frames in the sports video, and the smaller the value, the higher the matching degree.

[0091] S3, based on the first matching loss matrix Match_Cost, obtain multiple matching values ​​of the target nominations in the first matching target nomination set Lineup_Det_Box of the next frame in the sports video between two adjacent frames;

[0092] Optionally, based on the initial matching loss matrix Match_Cost, the Hungarian matching algorithm is used to obtain multiple matching values ​​of the target nominee's box coordinates in two adjacent frames of the sports video, as expressed by:

[0093] Match_Pair=Hungarian(Match_Cost).

[0094] S4, determine a matching value obtained in step S3. If the matching value is a bad match, the target nomination in the first matching target nomination set corresponding to the bad match is a low-matching-quality target nomination. Select a substitute nomination from the nomination substitute set and replace the low-matching-quality target nomination corresponding to the bad match with the selected substitute nomination to obtain the first generation matching result of the next frame and proceed to step S5. If all matching values ​​obtained in step S3 are good matches, the target nomination in the first matching target nomination set corresponding to each good match is a high-matching-quality target nomination. Obtain the high-matching-quality target nomination set and end the multi-target tracking matching of the images of the two adjacent frames in the sports video to obtain the tracking matching result of the next frame.

[0095] Specifically, if a matching value is an outlier, the target nomination corresponding to that matching value is a bad match, i.e., a low-quality target nomination. A substitute nomination is selected from the nomination substitute set Substitution to replace the first-match target nomination in the first-match target nomination set Lineup_Det_Box that corresponds to the bad match. Thus, the low-quality target nomination is replaced with a high-quality target nomination. This process is repeated for all matching values ​​to obtain the first generation of matching results. Figure 3 Matching results in;

[0096] Referring to Figure 4(b), the column numbers of the matching loss matrix are the element numbers of the tracking results in frame (t-1) (the previous frame), and the row numbers are the element numbers of the first matching target nomination set Lineup_Dex_Box in frame t (the next frame). The elements (matching values) corresponding to each row and column constitute the matching loss matrix. According to the results of the Hungarian algorithm, (k,j) indicates that the k-th target nomination in frame t and the j-th target nomination in frame (t-1) of the matching loss matrix Match_Pair are considered to belong to the same target nomination (the shaded area in Figure 4(b)). In addition, if a matching value is too large (significantly larger than the matching values ​​of other shaded areas in the matching loss matrix, such as 1.7011 being much larger than 0.0473) or if two matching values ​​are very close, such as 0.0401 and 0.0419, then the matching value is an outlier.

[0097] Optionally, in step S4, if the matching value obtained in step S3 exceeds the matching value threshold, the matching value is considered an abnormal value.

[0098] Optionally, when selecting a substitute nominee from the nominee set Substitution, the Hungarian algorithm is used to iterate through all substitute nominees and select the optimal substitute nominee.

[0099] S5. Construct the nth generation matching loss matrix based on the bounding box coordinates of the target nomination in Track_Box of the previous frame tracking result, n=1; Figure 3 The new matching loss matrix in;

[0100] S6, based on the nth generation matching loss matrix, obtain multiple matching values ​​of the target nomination in the nth generation matching result of the next frame in the sports video between two adjacent frames;

[0101] S7. Determine a matching value obtained in step S6. If the matching value is a bad match, the target nomination in the nth generation matching result of the next frame corresponding to the bad match is a low-match quality target nomination. Select a substitute nomination from the nomination substitute set. Replace the low-match quality target nomination corresponding to the bad match with the selected substitute nomination. Thus, the low-match quality target nomination is replaced with a high-match quality nomination, obtaining the matching result of the (n+1)th generation. Figure 3 The new matching result is returned to step S3, and n = n + 1 is set. If all matching results in the nth generation are good matches, i.e. high-quality target nominations, the high-quality target nomination set is obtained, and the multi-target tracking matching of any two adjacent frames in the sports video is ended to obtain the tracking matching result of the next frame.

[0102] In the matching loss matrix of Figure 4, the shaded area represents the index of the target nomination in the tracking result of the (t-1)th frame (the previous frame) that best matches the target nomination in frame t, i.e., Match_Pair = {(0,0),(1,5),(2,1),(3,2),(4,3),(5,4)}. The matching value at (5,4) in the 5th row and 4th column is 1.7011, which is significantly larger than the matching values ​​of other shaded areas in the matching loss matrix. This matching value is an outlier, and this matching value will be analyzed. The specific steps of the analysis are as follows: The matching value of 1.7011 is the best match according to the characteristics of the Hungarian algorithm, that is, the matching value is already the minimum among all unmatched columns in the row where the matching value is located, and the target corresponding to the matching value is not a newly appearing target in the current space (the current sports field). If there is a smaller matching value in the matched column of the row, and the smaller value has not been matched, it means that there are other better matching rows in the column where the smaller value is located. If the matching value of 1.7011 is the minimum value of the row, it indicates that the bounding box quality of the target is poor. At the same time, it is found that all matching values ​​of the row are extremely close to those of the first row. Therefore, it is considered that the target nomination represented by the row and the target nomination represented by the first row have detected the same target, that is, duplicate detection. Therefore, the target nomination represented by the row is replaced by the substitute nomination from the nomination substitute set.

[0103] Referring to FIG. 5, which shows an example after permutation substitution. It can be seen in the figure that only one of the target nominations detected repeatedly in FIG. 4 is retained, and the lost target is found again. Obtain the new matching loss matching matrix. Although the same matching result as before is obtained after the new matching loss matching matrix passes through the Hungarian algorithm again, as Figure 3 indicated by the solid arrow in, the loss value is significantly reduced (from 1.7011 to 0.0138). If the loss value is still abnormal, repeat this step 5 until the loss is reduced or the substitute nominations are exhausted, as Figure 3 shown by the dashed arrow direction in.

[0104] On the other hand, the present invention also provides a multi-target tracking and matching method for complete sports videos, and the specific steps are as follows:

[0105] SS1. Use the foregoing method to obtain the tracking and matching result of the w-th frame, where w = 2;

[0106] SS2. Obtain the detection result of the (w + 1)-th frame;

[0107] SS3. Use the Hungarian algorithm to match the tracking and matching result of the w-th frame and the detection result of the (w + 1)-th frame to obtain the tracking and matching result of the (w + 1)-th frame;

[0108] SS4. If w < W, where W is the total number of frames of the sports video and W ≥ 3, let w = w + 1, and return to step SS2; if w ≥ W, use the tracking and matching result of the (w + 1)-th frame as the total target tracking and matching result of the sports video.

[0109] Optionally, use the multi-target tracking network model TransTrack to obtain the detection result of the (w + 1)-th frame.

[0110] It can be understood that the adjacent two frames of images in the sports video are used as the first frame image and the second frame image to obtain the tracking and matching result of the second frame.

[0111] The present invention can truncate redundant detection targets (i.e., repeatedly detected targets) and filter out interfering nominations (audience or other targets other than athletes on the sports field); in addition, when performing the first matching, there must be target nominations with high matching quality and low matching quality for repeatedly detected targets. Replacing the nominations with low matching quality with the target nominations in the substitute nomination set can not only eliminate the phenomenon of repeated detection, but also retrieve the lost targets.

[0112] Furthermore, in order to illustrate the effectiveness of the method proposed by the present invention, ablation experiments are carried out on multiple parameters obtained by using the foregoing multi-target tracking and matching method, and Table 1 is obtained. The specific indicators are as follows:

[0113] IDF1: Identification F-Score refers to the F-value of the target ID identified in each target bounding box. The target ID is the identification number assigned to each target during tracking.

[0114] IDP: Identification Precision refers to the accuracy of target ID identification within each target bounding box.

[0115] IDR: Identification Recall refers to the recall rate of target ID identification within each target bounding box.

[0116] MT: Mostly Tracked trajectories, which is the number of true trajectories that are successfully tracked in more than 80% of the total frames.

[0117] FP: False Positive refers to a negative sample that is predicted as positive by the model, also known as a false alarm.

[0118] FN: False Negative refers to a positive sample that is predicted as negative by the model, also known as a missed detection.

[0119] IDS: ID switches refers to the number of times the target ID changes instantaneously in the tracking trajectory, which usually reflects the stability of the tracking.

[0120] MOTA: Multiple Object Tracking Accuracy is an indicator that measures the accuracy of multi-object tracking by a single camera based on FP, FN, and IDS.

[0121] MOTP: Multiple Object Tracking Precision is a metric that measures the positional error of a single camera in tracking multiple objects.

[0122] Table 1 shows the comparison of the performance indicators of this invention with the baseline (i.e., the original model) and different parameter settings, under the premise of confidence score > 0.4, taking the top N = 12 nominations, the maximum number of frames to retain the target ID is 32 or 80, and whether or not nomination permutation is used. In the table, upward arrows (↑) indicate that the higher the indicator, the better, and downward arrows (↓) indicate that the lower the indicator, the better. It can be seen that the second row only takes the top 12 nominations compared to the baseline in the first row. The main impact is a decrease in FP, that is, a reduction in false alarms. It can be considered that taking the top 12 nominations eliminates some interfering nominations, such as... Figure 6As shown, the top three indicators related to the target ID also decreased, suggesting that selecting the top 12 nominations also excluded some correct nominations. Compared to the second row, the third row extends the maximum ID retention frame count to 80 frames, and the top three indicators related to the target ID rebound, indicating that a longer ID trajectory was tracked. The fourth row uses the nomination replacement method proposed in this invention, and it can be found that all indicators are improved compared to the third row or the baseline. It can be concluded that the nomination replacement not only tracks a longer target ID trajectory, but also significantly improves the overall tracking quality MOTA. Figure 7 As shown.

[0123] Table 1. Impact of Nomination Replacement on Various Indicators

[0124]

[0125]

[0126] Furthermore, to illustrate the effectiveness of the proposed method, comparative experiments were conducted on different nomination replacement strategies. It was assumed that the initial matching target nomination Lineup_Det_Box contained p low-matching quality target nominations (to be replaced), and the Substitution nomination contained q items to be replaced, already sorted in descending order of Score (i.e., detection quality from high to low). The parameter settings are shown in Table 2, where:

[0127] Sub1: Delete p low-matching quality target nominations and replace them with the highest-ranked candidate from the substitutes;

[0128] Sub2: Delete p low-matching-quality target nominations and replace the first item in the substitutes that is less than the matching loss threshold.

[0129] Sub3: Delete p low-matching quality target nominations and take the smaller of p and q as substitutes;

[0130] Sub4: Delete the first low-match quality target nomination and replace it with the highest-ranked item from the substitutes;

[0131] Sub5: Delete the first low-matching-quality target nomination and replace it with the first replacement item in the substitutes that is less than the matching loss threshold.

[0132] Table 2 Different Nomination Replacement Strategies

[0133]

[0134] The experimental results are shown in Table 3:

[0135] Table 3. Impact of different nomination replacement strategies on various indicators

[0136]

[0137]

[0138] Table 3 shows that Sub4 and Sub5 significantly improve upon Sub1, Sub2, and Sub3 in all metrics. Sub4, by replacing the highest-scoring candidate among the substitutes, achieves higher scores in the top three ID-related metrics. Sub5, by replacing the first candidate among the substitutes whose score is below the matching loss threshold, exhibits better performance in the tracking metric MOTA. This section's experiments demonstrate the effectiveness of the nomination replacement method; regardless of the replacement strategy, it improves metrics by varying degrees compared to the baseline.

[0139] Furthermore, to demonstrate the effectiveness of the method proposed in this invention, an adaptability test was conducted on various types of sports videos using the multi-target tracking and matching method of this invention;

[0140] like Figure 8 As shown, after taking the first N=10 nominations and following the Sub4 permutation strategy in the previous section, this invention is also applicable to multi-target tracking tasks in basketball videos.

[0141] like Figure 9 As shown, since the football field is large and not all players are usually visible in a football match, the first N=15 nominees are selected. Following the Sub4 permutation strategy in the previous section, this invention is also applicable to multi-target tracking tasks in football videos.

[0142] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. A multi-target tracking and matching method in sports videos based on target nomination permutation, characterized in that, The specific steps are as follows: S1, perform target detection on two adjacent frames of the sports video to obtain the tracking result of the previous frame and the detection result of the next frame; the detection result of the next frame includes the initial matching target nomination set and the nomination substitute set; S2, construct the first-match loss matrix based on the bounding box coordinates of the target nominations in the first-match target nomination set and the tracking results of the previous frame. The expression is: in, Indicates the general intersection-union ratio; The general expression for the intersection-union ratio is: in, L This indicates the rectangular region that first matches the target nomination set in the next frame; T The rectangular region representing the tracking result of the previous frame; This represents the smallest rectangular region that can cover the rectangular region of the first matching target nomination set in the subsequent frame and the rectangular region of the tracking result in the previous frame. ; Indicates the difference; Indicates intersection, union, and ratio; S3, based on the first-match loss matrix, obtain multiple matching values ​​of the target nominations in the first-match target nomination set between two adjacent frames in the sports video; S4, determine a matching value obtained in step S3. If the matching value is a bad match, the target nomination in the first matching target nomination set corresponding to the bad match is a low-matching-quality target nomination. Select a substitute nomination from the nomination substitute set and replace the low-matching-quality target nomination corresponding to the bad match with the selected substitute nomination to obtain the first generation matching result of the next frame and proceed to step S5. If all matching values ​​obtained in step S3 are good matches, the target nomination in the first matching target nomination set corresponding to each good match is a high-matching-quality target nomination. Obtain the high-matching-quality target nomination set and end the multi-target tracking matching of the images of the two adjacent frames in the sports video to obtain the tracking matching result of the next frame. S5. Construct the nth generation matching loss matrix based on the nth generation matching result of the next frame and the bounding box coordinates of the target nomination in the tracking result of the previous frame, n=1. S6, based on the nth generation matching loss matrix, obtain multiple matching values ​​of the target nomination in the nth generation matching result of the next frame in the sports video between two adjacent frames; S7. Determine a matching value obtained in step S6. If the matching value is a bad match, the target nomination in the nth generation matching result of the next frame corresponding to the bad match is a low-match quality target nomination. Select a substitute nomination from the nomination substitute set and replace the low-match quality target nomination corresponding to the bad match with the selected substitute nomination to obtain the (n+1)th generation matching result of the next frame. Let n=n+1 and return to step S5. If all the matching values ​​are good matches, the target nomination in the nth generation matching result of the next frame corresponding to each good match is a high-match quality target nomination. Obtain the high-match quality target nomination set and end the multi-target tracking matching of the images of the two adjacent frames in the sports video to obtain the tracking matching result of the next frame.

2. The multi-target tracking and matching method in sports videos according to claim 1, characterized in that, Use a multi-object tracking network model to obtain the tracking result of the previous frame and the detection result of the next frame in step S1; Specifically, the training set of the historical sports video database is used to label and track multiple targets in the historical sports videos to obtain the tracking label for each target; based on all the tracking labels, a multi-target tracking model is trained using deep learning to obtain a multi-target tracking network model.

3. The multi-target tracking and matching method in sports videos according to claim 2, characterized in that, Multi-target tracking models include multilayer perceptrons; the learning steps of a multilayer perceptron are as follows: The expression for obtaining a multilayer perceptron is: ； in, The target bounding box for output; The input is a single-frame image; These are learnable parameters; The target bounding box output by the multilayer perceptron is obtained using the backpropagation mechanism. The deviation from the target's tracking label is passed to the learnable parameters. Trainable parameters Make the output target bounding box Y Tracking tags that are approaching the target.

4. The multi-target tracking and matching method in sports videos according to any one of claims 1-3, characterized in that, The tracking result of the previous frame obtained in step S1 The expression is: ； in, This indicates the first frame of the previous frame in a sports video. m One target nomination, m= 1,2,…, M , M This indicates the total number of target nominations in the previous frame of a sports video.

5. The multi-target tracking and matching method in sports videos according to any one of claims 1-3, characterized in that, The detection results of the next frame obtained in step S1 are sorted in descending order of target image confidence; the preceding frames exceeding the target confidence threshold are extracted. Each subsequent frame's target nomination has an expression as follows: ； in, Indicates the first i The next frame's target nomination, i =1,2,…, N , N The total number of target nominations extracted for the next frame; Score i Indicates the first i The target confidence level of the target nomination in the next frame; Indicates the first i The coordinates of the top-left corner of the target bounding box in the next frame. Indicates the first i The coordinates of the bottom right corner of the target bounding box of the target nomination in the next frame; Will The previous frame in the target nomination The target nomination of the next frame is used as the first matching target nomination to obtain the first matching target nomination set. ;other The next frame's target nomination is used as a substitute nomination, thus obtaining the nomination substitute set. .

6. The multi-target tracking and matching method in sports videos according to claim 1, characterized in that, In step S3 and / or step S6, the Hungarian matching algorithm is used to obtain multiple matching values ​​of the target nomination's target box coordinates in two adjacent frames of the sports video.

7. The multi-target tracking and matching method in sports videos according to claim 1 or 6, characterized in that, In step S4 and / or step S7, if the matching value is too large or two matching values ​​are very close, the matching value is considered an outlier.

8. The multi-target tracking and matching method in sports videos according to claim 1 or 7, characterized in that, In step S4 and / or step S7, if the matching value exceeds the matching value threshold, the matching value is considered an abnormal value.

9. The multi-target tracking and matching method in sports videos according to claim 1, characterized in that, It also includes the following steps: SS1. Obtain the tracking and matching results of the w-th frame using the aforementioned method, where w = 2; SS2. Obtain the detection results of the (w + 1)-th frame; SS3. Use the Hungarian algorithm to match the tracking and matching results of the w-th frame and the detection results of the (w + 1)-th frame to obtain the tracking and matching results of the (w + 1)-th frame; SS4. If w < W, where W is the total number of frames of the sports video and W ≥ 3, set w = w + 1 and return to step SS2; If w ≥ W, use the tracking and matching results of the (w + 1)-th frame as the total target tracking and matching results of the sports video.

Images