A method and system for evaluating target tracking performance

Abstract

The application provides a target tracking effect evaluation method and system, and belongs to the technical field of target tracking. The evaluation method comprises the following steps: acquiring video data; extracting a true value track and a predicted track of the video data; acquiring a true value identification sequence corresponding to the predicted track; comparing true value identifications of adjacent two frames in the true value identification sequence from the beginning, and determining whether positive matching or false matching occurs; acquiring a predicted identification sequence corresponding to a frame with a corresponding predicted value in the true value track; comparing predicted identifications of adjacent two frames in the predicted identification sequence from the beginning, and determining whether missing matching occurs; and evaluating the target tracking effect according to the occurrence of positive matching, false matching and missing matching. The application can provide reliable evaluation indexes for target tracking matching, is simple to calculate, has clear physical meaning, improves the target tracking effect, and improves the comprehensive performance of the tracking algorithm.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of target tracking, in particular to a target tracking effect evaluation method and system. BACKGROUND

[0002] Multi-target tracking refers to tracking multiple targets in a continuous video, and its main application scenarios are security monitoring and automatic driving.

[0003] The existing multi-target tracking evaluation index binds the detection and tracking matching effect together for evaluation, thereby obtaining a comprehensive score.

[0004] Such a score is often complex to calculate, and the calculation method is relatively abstract, lacking a clear physical meaning. Even if the score is low, it is not possible to determine whether the problem is in the detection link or the tracking link, and the score value can only evaluate the advantages and disadvantages of the entire detection-tracking system, but lacks guiding significance for improving the tracking effect.

[0005] In the target detection field, MAP (Mean Average Precision, average precision mean) has become a common standard in the target detection direction. However, a simple evaluation index for target tracking matching has not yet appeared, which cannot provide effective guidance for optimizing the target tracking matching process, resulting in the inability to further improve the target tracking effect. SUMMARY

[0006] The main purpose of the present application is to provide a target tracking effect evaluation method and system to solve the problem that a simple evaluation index for target tracking matching has not yet appeared in the prior art, which cannot provide effective guidance for optimizing the target tracking matching process, resulting in the inability to further improve the target tracking effect.

[0007] In order to achieve the above purpose, according to one aspect of the present application, a target tracking effect evaluation method is provided, comprising:

[0008] S1: obtaining video data;

[0009] S2: extracting the ground truth trajectory and the predicted trajectory of the video data;

[0010] S3: obtaining the ground truth label sequence corresponding to the predicted trajectory;

[0011] S4: comparing the ground truth labels of adjacent two frames in the ground truth label sequence from the beginning in turn to determine whether positive matching or false matching occurs;

[0012] S5: obtaining the predicted label sequence corresponding to the frame with the corresponding predicted value in the ground truth trajectory;

[0013] S6: sequentially compare the prediction identifiers of two adjacent frames in the prediction identifier sequence from the beginning, and determine whether missing assignment occurs;

[0014] S7: evaluate the target tracking effect according to the occurrence of correct assignment, incorrect assignment and missing assignment.

[0015] Further, the S2 specifically comprises:

[0016] detecting the targets in each frame of the ground truth image in the video data, extracting the ground truth bounding box in each frame of the ground truth image, concatenating the ground truth bounding boxes of the same target in each frame of the ground truth image to form the ground truth trajectory;

[0017] predicting the target position according to the video data, extracting the prediction bounding box in each frame of the prediction image, and concatenating the prediction bounding boxes of the same target in each frame of the prediction image to form the prediction trajectory.

[0018] Further, the S3 specifically comprises:

[0019] determining whether the prediction bounding box in the prediction trajectory matches the ground truth bounding box in the ground truth trajectory frame by frame; if yes, extracting the number of the matched ground truth bounding box as the ground truth identifier; otherwise, setting the ground truth identifier as -1; and forming the ground truth identifier sequence.

[0020] Further, the S4 specifically comprises:

[0021] when the ground truth identifiers of two adjacent frames are the same and not -1, it is determined that there is one correct assignment, and the number of correct assignments is increased by 1;

[0022] when the ground truth identifiers of two adjacent frames are different, it is determined that there is one incorrect assignment, and the number of incorrect assignments is increased by 1.

[0023] Further, the S6 specifically comprises:

[0024] when the prediction identifiers of two adjacent frames are different, it is determined that there is one missing assignment, and the number of missing assignments is increased by 1.

[0025] Further, the S7 specifically comprises:

[0026] evaluate the target tracking effect by the accuracy, and the calculation method of the accuracy is specifically:

[0027]

[0028] wherein, P represents the accuracy, TP represents the number of correct assignments, and FP represents the number of incorrect assignments.

[0029] Further, the S7 specifically comprises:

[0030] The target tracking effect is evaluated by a recall rate, and the calculation method of the recall rate is specifically as follows:

[0031]

[0032] wherein R represents the recall rate, and FN represents the number of missed matches.

[0033] Further, the S7 specifically comprises:

[0034] The target tracking effect is evaluated by an F value, and the calculation method of the F value is specifically as follows:

[0035]

[0036] wherein F1 represents the F value.

[0037] Further, the target tracking effect evaluation method further comprises:

[0038] S8: When a mis-match or a missed match occurs, record error information.

[0039] According to one aspect of the present application, a target tracking effect evaluation system is provided, comprising:

[0040] a processor;

[0041] a memory, wherein the memory has computer readable instructions stored thereon, and the computer readable instructions are executed by the processor to implement the target tracking effect evaluation method.

[0042] The technical scheme of the present application can evaluate the target tracking effect according to the occurrence of correct matches, mis-matches and missed matches, provide reliable evaluation indexes for target tracking matching, is simple to calculate, has clear physical meaning, and can provide effective guidance for optimizing the target tracking matching process, improve the target tracking effect, and improve the comprehensive performance of the tracking algorithm.

[0043] In addition to the purposes, features and advantages described above, the present application has other purposes, features and advantages. The present application will be further described in detail below with reference to the drawings. BRIEF DESCRIPTION OF DRAWINGS

[0044] The drawings accompanying the specification of the present application form a part thereof, and serve to provide further understanding of the present application, the illustrative embodiments of the present application and its description serve to explain the present application, and do not constitute improper limitations on the present application. In the drawings:

[0045] Figure 1 a flowchart of a target tracking effect evaluation method provided by an embodiment of the present application is shown;

[0046] Figure 2A structure diagram of a true trajectory and a predicted trajectory is shown.

[0047] Figure 3 A structure diagram of an evaluation system of target tracking effect is shown. DETAILED DESCRIPTION

[0048] It should be noted that the embodiments and features of the embodiments in the present application can be combined with each other without conflict. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0049] In order for those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below in combination with the drawings of the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor should belong to the scope of protection of the present application.

[0050] It should be noted that the terms "first", "second" and the like in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily describe a specific order or sequence. It should be understood that the terms used in this way can be interchanged under appropriate circumstances, so as to describe the embodiments of the present application described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not necessarily limit to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0051] It should be noted that the terms used herein are only for the purpose of describing specific embodiments, and are not intended to limit the exemplary embodiments according to the present application. As used herein, the singular form is intended to include the plural form unless the context clearly indicates otherwise, and it should also be understood that when the terms "comprise" and / or "include" are used in the specification, there is a feature, step, operation, device, component and / or combination thereof.

[0052] Reference is made to the accompanying drawings Figure 1 A flowchart of a target tracking effect evaluation method is shown.

[0053] The embodiment of the present application provides a target tracking effect evaluation method, comprising:

[0054] S1: acquiring video data.

[0055] S2: extracting a ground truth trajectory and a predicted trajectory of the video data.

[0056] It should be noted that the tracking process needs to match and concatenate the same target at adjacent time points into a trajectory, during which the following situations may occur:

[0057] (1) When a matching relationship that should exist is found, it can be called a positive match.

[0058] (2) When a matching relationship that should exist is not found, it can be called a missed match.

[0059] (3) When a matching relationship that should not exist is found, it can be called a false match.

[0060] Further, under the premise that the detection result is determined, the topology of the tracking link is determined by the matching relationship between the bounding boxes of adjacent frames.

[0061] In one possible implementation, S2 specifically includes:

[0062] detecting targets in each frame of the ground truth image in the video data, extracting ground truth bounding boxes in each frame of the ground truth image, concatenating the ground truth bounding boxes of the same target in each frame of the ground truth image to form a ground truth trajectory.

[0063] According to the video data, the target position is predicted, the predicted bounding box in each frame of the predicted image is extracted, and the predicted bounding boxes of the same target in each frame of the predicted image are concatenated to form a predicted trajectory.

[0064] In the present application, by extracting the ground truth bounding boxes and the predicted bounding boxes of each frame and concatenating them to form a trajectory, not only can the motion trajectory of the target be more clearly displayed, but also strong support can be provided for the evaluation, comparison and optimization of the target tracking algorithm. This method effectively improves the detailed degree and guidance of evaluation, and helps to continuously improve the comprehensive performance of the tracking algorithm.

[0065] S3: obtaining a ground truth identification sequence corresponding to the predicted trajectory.

[0066] Optionally, S3 specifically includes:

[0067] determining whether the predicted bounding box in the predicted trajectory matches the ground truth bounding box in the ground truth trajectory frame by frame; if so, extracting the number of the matched ground truth bounding box as a ground truth identification; otherwise, setting the ground truth identification as -1; and forming the ground truth identification sequence.

[0068] S4: sequentially comparing the ground truth identifications of adjacent two frames in the ground truth identification sequence from the beginning to determine whether a positive match or a false match occurs.

[0069] It should be noted that the matching between the predicted trajectory and the true trajectory is checked frame by frame, so that the accuracy of target tracking can be evaluated in a fine-grained manner. Through frame-by-frame comparison, it can be identified that matching or mismatching occurs on which frames, providing a basis for in-depth analysis.

[0070] In a possible implementation, S4 specifically comprises:

[0071] When the true value identifiers of the two adjacent frames are the same and are not -1, it is determined that there is one positive match, and the number of positive matches is increased by 1.

[0072] When the true value identifiers of the two adjacent frames are different, it is determined that there is one false match, and the number of false matches is increased by 1.

[0073] It should be noted that the matching condition is simplified to a binary identifier (1 represents matching, and -1 represents mismatching), so that the calculation process is more concise and clear, the complex calculation steps are reduced, and the evaluation result can still be provided effectively.

[0074] In the present application, by obtaining the true value identifier sequence of the predicted trajectory, and comparing the true value identifiers of the two adjacent frames to determine the positive match and the false match, this method provides a fine-grained, simple and clear, real-time and continuous evaluation method. It not only helps to clearly define and calculate the positive match and the false match, but also provides strong support for error analysis, algorithm optimization and comprehensive evaluation, thereby improving the target tracking effect and the comprehensive performance of the tracking algorithm.

[0075] Referring to the accompanying drawings Figure 2 , a structure diagram of a true value trajectory and a predicted trajectory provided by an embodiment of the present application is shown.

[0076] For example, Figure 2 In the figure, the circular trajectory is the true value trajectory, the square trajectory is the predicted trajectory, the gray scale of the same trajectory is the same, and the gray scale of different trajectories is different. The true value identifier sequence corresponding to the predicted trajectory pd1 is (1, -1, 1). According to the above calculation method, 1! = -1 = > FP, -1! = 1 = > FP, then the number of false matches is 2, and the number of positive matches is 0.

[0077] S5: Obtain a predicted identifier sequence corresponding to the frame having a corresponding predicted value in the true value trajectory.

[0078] Optionally, the frame having a corresponding predicted value in the true value detection box in the true value trajectory is selected, the number of the predicted detection box in the corresponding frame in the predicted trajectory is obtained, and a predicted identifier sequence is formed.

[0079] S6: Compare the predicted identifiers of the two adjacent frames in the predicted identifier sequence in sequence from the beginning, and determine whether there is a false match.

[0080] In a possible implementation, S6 specifically comprises:

[0081] When the prediction identifiers of two adjacent frames are different, it is determined that there is one missing match, and the missing match number is increased by 1.

[0082] In the present application, by acquiring the prediction identifier sequence and comparing the prediction identifiers of two adjacent frames frame by frame, the missing match situation is determined, and this method provides an accurate, comprehensive, simple and clear physical meaning evaluation method. It not only accurately locates the missing match, but also provides strong support for algorithm improvement, complex scene adaptability improvement and real-time monitoring, thereby improving the target tracking effect and the comprehensive performance of the algorithm.

[0083] For example, Figure 2 In the present application, the true value identifier sequence corresponding to the prediction trajectory pd1 is (1, -1, 1), and according to the above calculation method, 2! = 1 = > FN, 1 == 1, so the missing match number is 1.

[0084] Optionally, the correct match number, the false match number and the missing match number can be stored in a three-element array for dimension-by-dimension accumulation to obtain a three-dimensional array, and then the correct match number, the false match number and the missing match number recorded in the three-dimensional array are used for index calculation.

[0085] S7: According to the occurrence of correct match, false match and missing match, the target tracking effect is evaluated.

[0086] It should be noted that, according to the occurrence of correct match, false match and missing match, the target tracking effect is evaluated, which can reflect the consistency of the topological structure of the output trajectory in the tracking matching process with the true value.

[0087] In one possible implementation, S7 specifically includes:

[0088] The accuracy rate is used to evaluate the target tracking effect, and the calculation method of the accuracy rate is specifically:

[0089]

[0090] Wherein, P represents the accuracy rate, TP represents the correct match number, and FP represents the false match number.

[0091] In one possible implementation, S7 specifically includes:

[0092] The recall rate is used to evaluate the target tracking effect, and the calculation method of the recall rate is specifically:

[0093]

[0094] Wherein, R represents the recall rate, and FN represents the missing match number.

[0095] In one possible implementation, S7 specifically includes:

[0096] The target tracking effect is evaluated by F value, and the calculation method of F value is as follows:

[0097]

[0098] Wherein, F1 represents F value.

[0099] It should be noted that the use of accuracy, recall and F value and other quantitative indicators makes the evaluation standard more clear and quantitative. The calculation method and result of accuracy, recall and F value have clear physical meaning, which is convenient for understanding and explanation. The accuracy reflects the accuracy of the prediction result, the recall reflects the ability of the model to capture the target, and the F value provides a comprehensive trade-off index. Through these indicators, problems existing in the tracking system can be easily identified. If the accuracy is low, there may be more misassignment problems; if the recall is low, there may be more missing assignment problems. In this way, specific problems can be improved.

[0100] In the present application, by using accuracy, recall and F value and other indicators for evaluation according to the cases of correct assignment, misassignment and missing assignment, the performance of the target tracking system can be comprehensively and quantitatively reflected. This evaluation method has clear physical meaning, which is helpful to identify and solve problems in the system, guide system optimization, facilitate algorithm comparison, simplify the evaluation process, and thus improve the comprehensive performance of the target tracking effect and the tracking algorithm.

[0101] In one possible implementation, the evaluation method of the target tracking effect further comprises:

[0102] S8: When misassignment or missing assignment occurs, record error information.

[0103] Specifically, the error information can be recorded in the log, and the error information includes error type, frame number, true value, predicted value and other information, so as to determine the problem by playback.

[0104] In the present application, by recording the detailed error information of misassignment and missing assignment and recording these information in the log, not only detailed error analysis and problem reproduction support can be provided, but also the pertinence of algorithm improvement and the transparency of debugging can be improved. A solid foundation is provided for performance evaluation and team cooperation, which is helpful for feedback processing and system robustness improvement, thereby comprehensively improving the comprehensive performance of the target tracking system.

[0105] From the above description, it can be seen that the evaluation method of the target tracking effect provided by the present application can evaluate the target tracking effect according to the occurrence of correct assignment, misassignment and missing assignment, provide reliable evaluation indicators for target tracking matching, the calculation is simple, the physical meaning is clear, and effective guidance is provided for the optimization of the target tracking matching process, the target tracking effect is improved, and the comprehensive performance of the tracking algorithm is improved.

[0106] With reference to the accompanying drawings Figure 3 , a structure schematic diagram of a target tracking effect evaluation system provided by an embodiment of the present application is shown.

[0107] An embodiment of the present application provides a target tracking effect evaluation system 20, which comprises:

[0108] A processor 201;

[0109] A memory 202, wherein the memory 202 stores computer readable instructions, and the computer readable instructions are executed by the processor 201 to realize the target tracking effect evaluation method.

[0110] From the above description, it can be seen that the target tracking effect evaluation system provided by the present application can evaluate the target tracking effect according to the occurrence of positive matching, false matching and missing matching, and provide reliable evaluation indexes for target tracking matching, and the calculation is simple, the physical meaning is clear, and the target tracking matching process is effectively guided to optimize the target tracking effect and improve the comprehensive performance of the tracking algorithm.

[0111] The relative arrangement, numerical expressions and values of the components and steps set forth in these embodiments do not limit the scope of the present application, unless otherwise specified. Meanwhile, it should be understood that the sizes of the various parts shown in the drawings are not drawn in accordance with the actual proportional relationship for the convenience of description. The known technologies, methods and devices of the related field may not be discussed in detail, but should be considered as part of the authorized description under appropriate circumstances. In all examples shown and discussed herein, any specific value should be interpreted as merely exemplary, and not as a limitation. Therefore, other examples of the exemplary embodiments can have different values. It should be noted that similar reference numerals and letters represent similar items in the following drawings, and therefore, once an item is defined in one drawing, it does not need to be further discussed in subsequent drawings.

[0112] For ease of description, spatial relative terms such as "above," "on top of," "on the upper surface of," "above," etc., are used herein to describe the spatial positional relationship of a device or feature as shown in the figures to other devices or features. It should be understood that spatial relative terms are intended to encompass different orientations in use or operation beyond the orientation of the device as described in the figures. For example, if the device in the figures were inverted, a device described as "above" or "on top of" other devices or structures would subsequently be positioned as "below" or "under" other devices or structures. Thus, the exemplary term "above" can include both "above" and "below." The device may also be positioned in other different ways (rotated 90 degrees or in other orientations), and the spatial relative descriptions used herein will be interpreted accordingly.

[0113] In the description of this invention, it should be understood that the orientation or positional relationship indicated by directional terms such as "front, back, up, down, left, right", "horizontal, vertical, horizontal" and "top, bottom" is generally based on the orientation or positional relationship shown in the accompanying drawings, and is only for the convenience of describing this invention and simplifying the description. Unless otherwise stated, these directional terms do not indicate or imply that the device or element referred to must have a specific orientation or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation on the scope of protection of this invention; the directional terms "inner" and "outer" refer to the inner and outer contours relative to the outline of each component itself.

[0114] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method of evaluating the effectiveness of target tracking, characterized by, The method comprises: S1: obtaining video data; S2: extracting a ground truth trajectory and a predicted trajectory of the video data; S3: obtaining a ground truth identification sequence corresponding to the predicted trajectory; determining whether a predicted bounding box in the predicted trajectory matches a ground truth bounding box in the ground truth trajectory by frame; if yes, extracting the number of the matched ground truth bounding box as a ground truth identification; otherwise, setting the ground truth identification as -1; and forming the ground truth identification sequence; S4: sequentially comparing the ground truth identifications of adjacent two frames in the ground truth identification sequence from the beginning to determine whether positive matching or mis-matching occurs; when the ground truth identifications of the adjacent two frames are the same and not -1, it is determined that there is one positive matching, and the number of positive matching is increased by 1; when the ground truth identifications of the adjacent two frames are different, it is determined that there is one mis-matching, and the number of mis-matching is increased by 1; S5: obtaining a predicted identification sequence corresponding to the frames with corresponding predicted values in the ground truth trajectory; S6: sequentially comparing the predicted identifications of adjacent two frames in the predicted identification sequence from the beginning to determine whether missing matching occurs; when the predicted identifications of the adjacent two frames are different, it is determined that there is one missing matching, and the number of missing matching is increased by 1; S7: evaluating the target tracking effect according to the occurrence of positive matching, mis-matching and missing matching. The S2 specifically comprises:

2. The method of claim 1, wherein, performing target detection on each frame of ground truth image in the video data, extracting a ground truth bounding box in each frame of ground truth image, and concatenating the ground truth bounding boxes of the same target in each frame of ground truth image to form the ground truth trajectory; predicting the target position according to the video data, extracting a predicted bounding box in each frame of predicted image, and concatenating the predicted bounding boxes of the same target in each frame of predicted image to form the predicted trajectory. The S7 specifically comprises:

3. The method of claim 1, wherein evaluating the target tracking effect by accuracy, and the calculation method of the accuracy specifically comprises: The S7 specifically comprises: where P denotes the accuracy, TP denotes the number of true positives, and FP denotes the number of false positives.

4. The method of claim 1, wherein evaluating the target tracking effect by recall rate, and the calculation method of the recall rate specifically comprises: The S7 specifically comprises: where R represents recall rate and FN represents the number of missed prescriptions.

5. The method of claim 1, wherein, evaluating the target tracking effect by F value, and the calculation method of the F value specifically comprises: The method further comprises: wherein F1 represents the F-value.

6. The method of claim 1, wherein S8: when mis-matching or missing matching occurs, recording error information. The method comprises:

7. An evaluation system of a target tracking effect, characterized by, a processor; a memory having computer readable instructions stored thereon, wherein the computer readable instructions, when executed by the processor, implement the method for evaluating the target tracking effect according to any one of claims 1 to 6. ​

Images