Target tracking algorithm evaluation method and device, electronic equipment and storage medium

Abstract

The invention provides a target tracking algorithm evaluation method and device, electronic equipment and a storage medium, and the method comprises the steps: determining a scene set for evaluating a target tracking algorithm; for each scene in the scene set, obtaining test data corresponding to the scene, the test data comprising multiple frames of video images corresponding to the scene; inputting the test data into a model corresponding to the target tracking algorithm to obtain a detection file; marking a tracking target in the test data to obtain a truth value file; and analyzing the test value file and the truth value file to obtain an evaluation index for representing the tracking effect of the target tracking algorithm. According to the method, the test data is determined based on the scene set instead of directly adopting the existing data set, so that the test data can cover the special scene, and the robustness of the target tracking algorithm is more favorably tested.

Description

Technical Field

[0001] The present application relates to the technical field of algorithm evaluation, and in particular to a target tracking algorithm evaluation method, device, electronic device and storage medium. Background Art

[0002] Multi-Object Tracking (MOT) refers to the tracking of multiple pedestrians, vehicles and other targets in a video, and is mainly used in security monitoring and autonomous driving. However, in the tracking process, it is inevitable that tracking errors will occur due to certain special scenarios, which will have a significant impact on the subsequent decision-making applications and even pose a safety hazard. In the application scenario of autonomous driving, when the target vehicle (the vehicle in the field of view of the ego vehicle camera) is temporarily blocked, temporarily disappears from the field of view of the ego vehicle camera, or the distance between the ego vehicle and the target vehicle is close or far, there will be problems of unstable target tracking and ID jump, which is likely to cause sudden changes in the attributes of the tracked target, such as the horizontal and vertical distance, horizontal and vertical speed, and target orientation. Any sudden change in any attribute may cause the autonomous emergency braking system (AEB) to make an incorrect decision, thereby increasing the risk of collision. Therefore, it is necessary to conduct a multi-scenario, flexible and comprehensive evaluation of the multi-target tracking algorithm. Summary of the invention

[0003] The embodiments of the present application provide a target tracking algorithm evaluation method, device, electronic device and storage medium to solve the problem of how to evaluate the target tracking algorithm.

[0004] In a first aspect, an embodiment of the present application provides a target tracking algorithm evaluation method, the method comprising:

[0005] Determine the set of scenarios for evaluating the object tracking algorithm;

[0006] For each scene in the scene set, acquiring test data corresponding to the scene, the test data comprising a plurality of frames of video images corresponding to the scene;

[0007] Inputting the test data into the model corresponding to the target tracking algorithm to obtain a detection file;

[0008] Annotating the tracking target in the test data to obtain a true value file;

[0009] The test value file and the true value file are analyzed to obtain an evaluation index for characterizing the tracking effect of the target tracking algorithm.

[0010] In a second aspect, an embodiment of the present application further provides a target tracking algorithm evaluation device, the device comprising:

[0011] A determination module, used to determine a set of scenes for evaluating the target tracking algorithm;

[0012] An acquisition module, configured to acquire, for each scene in the scene set, test data corresponding to the scene, wherein the test data includes a plurality of frames of video images corresponding to the scene;

[0013] A first processing module, used for inputting the test data into a model corresponding to the target tracking algorithm to obtain a detection file;

[0014] A second processing module is used to mark the tracking target in the test data to obtain a true value file;

[0015] The third processing module is used to analyze the test value file and the true value file to obtain an evaluation index for characterizing the tracking effect of the target tracking algorithm.

[0016] In a third aspect, an embodiment of the present application provides an electronic device, the electronic device comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0017] Memory, used to store computer programs;

[0018] The processor is used to implement the steps in the above-mentioned target tracking algorithm evaluation method when executing the program stored in the memory.

[0019] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps in the above-mentioned target tracking algorithm evaluation method are implemented.

[0020] The embodiments of the present application include at least the following technical effects:

[0021] The technical solution of the embodiment of the present application is to determine test data based on a scene set, input the test data into a model corresponding to the target tracking algorithm to obtain a detection file, mark the tracking targets in the test data to obtain a true value file, and then analyze the detection file and the true value file to obtain a tracking effect evaluation index of the target tracking algorithm. Since the test data is determined based on a scene set rather than directly using an existing data set, the test data can cover special scenarios, which is more conducive to testing the robustness of the target tracking algorithm. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art are briefly introduced below.

[0023] Figure 1 It is a flowchart of a target tracking algorithm evaluation method provided in an embodiment of the present application;

[0024] Figure 2 It is a structural schematic diagram of a target tracking algorithm evaluation device provided in an embodiment of the present application;

[0025] Figure 3 A block diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0026] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0027] It should be understood that the references to "one embodiment" or "an embodiment" throughout the specification mean that the specific features, structures, or characteristics associated with the embodiment are included in at least one embodiment of the present application. Therefore, the references to "in one embodiment" or "in an embodiment" appearing throughout the specification do not necessarily refer to the same embodiment. In addition, these specific features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0028] In the various embodiments of the present application, it should be understood that the size of the serial numbers of the following processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0029] This application embodiment provides a target tracking algorithm evaluation method, such as Figure 1 As shown, the method includes:

[0030] Step 101, determining a scene set for evaluating a target tracking algorithm.

[0031] Specifically, the target tracking algorithm evaluation method provided in the embodiment of the present application is applied to the target tracking algorithm evaluation system, which can determine the type of scene used to evaluate the target tracking algorithm based on the application scenario of the target tracking algorithm. For example, when the application scenario of the target tracking algorithm is an autonomous driving scenario, the corresponding scene type is a traffic road scene type. Then, various scenes corresponding to the traffic road scene type are obtained as a scene set for evaluating the target tracking algorithm.

[0032] Step 102: For each scene in the scene set, obtain test data corresponding to the scene, wherein the test data includes a plurality of frames of video images corresponding to the scene.

[0033] After determining the scene set, for each scene in the scene set, corresponding test data is obtained, that is, multiple frames of video images obtained based on the scene, wherein the video images include at least one target to be tracked.

[0034] Step 103: input the test data into the model corresponding to the target tracking algorithm to obtain a detection file.

[0035] After obtaining the test data corresponding to each scene in the scene set, the test data is input into the corresponding model of the target tracking algorithm to obtain a detection file.

[0036] Step 104: annotate the tracking targets in the test data to obtain a true value file.

[0037] After obtaining the test data corresponding to each scene in the scene set, it is also necessary to annotate the tracking targets in the test data to obtain a true value file.

[0038] Step 105: Analyze the test value file and the true value file to obtain an evaluation index for characterizing the tracking effect of the target tracking algorithm.

[0039] After the detection file and the true value file are determined, an evaluation index for characterizing the tracking effect of the target tracking algorithm can be obtained by analyzing the detection file and the true value file.

[0040] The embodiment of the present application determines test data based on a scene set, inputs the test data into a model corresponding to a target tracking algorithm to obtain a detection file, labels the tracking targets in the test data to obtain a true value file, and then obtains a tracking effect evaluation index of the target tracking algorithm by analyzing the detection file and the true value file. Since the test data is determined based on a scene set rather than directly using an existing data set, the test data can cover special scenarios, which is more conducive to testing the robustness of the target tracking algorithm.

[0041] In an optional embodiment of the present application, determining a scene set for evaluating the target tracking algorithm includes:

[0042] Determine, based on a correspondence between at least one perception problem and at least one occurrence scenario corresponding to each perception problem, a plurality of sub-scene sets, wherein there is no intersection between the plurality of sub-scene sets, and each of the sub-scene sets includes a plurality of scene elements;

[0043] The scene elements respectively included in the multiple sub-scene sets are combined to obtain the scene set.

[0044] When determining a scene set, multiple sub-scene sets may be determined based on the correspondence between at least one perception problem and at least one occurrence scene corresponding to each perception problem. Here, the perception problem and the occurrence scene corresponding to the perception problem may be obtained based on experience, etc., wherein there may be multiple perception problems and multiple corresponding occurrence scenes. Then, the scene elements included in the sub-scene set are combined to obtain a scene set.

[0045] The following is an introduction to the vehicle as the tracking object of the target tracking algorithm. According to experience, three perception problems can be obtained, namely: missed recognition, misidentification and ID jump. The corresponding scenarios of the perception problem of missed recognition include: target crossing, being blocked, existing at the edge of the field of view (near, far, and both sides of the field of view), insufficient light, complex light, light brightness changes, bad weather (rain, snow, fog), etc. The corresponding scenarios of the perception problem of misidentification include: ground water, being blocked, complex light, strong light, bad weather, etc. The corresponding scenarios of the perception problem of ID jump include: being blocked, re-inspection after a short disappearance at the boundary of the field of view, re-inspection after missed recognition, crossing FOV (for example: the target moves from the field of view of the periscope camera to the front view), the target vehicle is too far or too close to the vehicle, resulting in unstable recognition, target crossing, complex light, bad weather, etc. The above is the corresponding relationship between at least one perception problem and at least one occurrence scenario corresponding to each perception problem.

[0046] Specifically, you can use {A m} represents the m types of scenes corresponding to missed recognition, and {B n} means except {A m}, and use {C p} means except {A m} and {B n}, {D q} is a collection of other change elements, including changing, adding or removing static elements (such as buildings, trees, guardrails, etc.), changing the trajectory of the ego vehicle, changing the trajectory of the target vehicle, adding vehicle targets, etc. m}、{B n}、{C p} and {D q} is a collection of multiple sub-scenes.

[0047] Furthermore, when the scene elements included in the sub-scene set are combined to obtain the scene set, {A m}、{Bn}、{C p} and {D q}set to achieve reasonable scene generalization and obtain the generalized scene set, namely {X i}. The scenario generalization formula is as follows:

[0048] {X i}={A m}*{B n}+{B n}*{C p}+{C p}*{D q}+{A m}*{C p}+{A m}*{D q}+{B n}*{D q}+{A m}*{B n}*{C p}+{A m}*{B n}*{D q}+{A m}*{C p}*{D q}+{B n}*{C p}*{D q}+{A m}*{B n}*{C p}*{D q}.

[0049] The above implementation scheme of the present application determines multiple sub-scenario sets based on the correspondence between perception problems and the occurrence scenarios corresponding to the perception problems, and combines the scene elements respectively included in the multiple sub-scenario sets to obtain a scene set. Through scene generalization, the scenarios used to evaluate the target tracking algorithm can be effectively expanded to cover more special scenarios, thereby ensuring the accurate completion of the robustness verification of the target tracking algorithm.

[0050] In an optional embodiment of the present application, for each scene in the scene set, obtaining test data corresponding to the scene includes:

[0051] For each scenario, obtain a test case corresponding to the scenario;

[0052] According to the test case, a virtual scene corresponding to the test case is built on the simulation module;

[0053] Adding a virtual image acquisition device to a host vehicle corresponding to the virtual scene;

[0054] The multiple frames of video images returned by the virtual image acquisition device are determined as the test data.

[0055] After the scene set is determined, further test data corresponding to each scene in the scene set is required.

[0056] Specifically, for each scene in the scene set, first obtain the test case corresponding to the scene, and then build the corresponding virtual scene based on the test case. For example, when the vehicle is used as the tracking object, the process of building the scene includes: selecting the ground environment, selecting roads, buildings and other infrastructure, setting the vehicle driving path, selecting the vehicle and other road parameters, and placing it on the response path, setting the speed of each vehicle in each section to achieve the interaction of the target vehicle, etc. After the virtual scene is built, add a virtual image acquisition device, i.e., a camera, to the main vehicle to record the scene data. Here, the installation position, pitch angle and FOV (Field of View) of the camera can also be adjusted as needed. In addition, the pixel and frame rate of the camera need to be set and recorded. The purpose of the scene building in this application is to verify the quality of the target tracking algorithm, so the image data of 1920*1080 or higher pixel values ​​can be saved, and the frame rate can be set to 25 frames / s or higher frame rates. Finally, the multi-frame video image returned by the virtual image acquisition device is determined as test data.

[0057] The above implementation scheme of the present application, by building a virtual scene for each scene in the scene set, and then obtaining corresponding test data, is convenient for obtaining a test file and a true value file based on the test data, and then evaluating the target tracking algorithm.

[0058] In an optional embodiment of the present application, the test data is input into a model corresponding to the target tracking algorithm to obtain a detection file, including:

[0059] Input the test data into the model corresponding to the target tracking algorithm to obtain an output file, wherein the output file includes a frame sequence, coordinates, annotation box height and annotation box width corresponding to each tracked target;

[0060] The detection file is determined according to the frame sequence, coordinates, height of the annotation box and width of the annotation box corresponding to each tracked target.

[0061] Specifically, after determining the test data, the test data is input into the model corresponding to the target tracking algorithm, and an output file including the frame sequence, coordinates, annotation box height and annotation box width corresponding to each tracked target can be obtained. Then, based on the frame sequence, coordinates, annotation box height and annotation box width corresponding to each tracked target in the output file, a detection file can be obtained.

[0062] It should be noted that usually, after the target tracking algorithm is run, only a video image result with a target detection frame and a tracking ID is generated. The embodiment of the present application further prints and saves the coordinates of the upper left corner vertex of the target detection frame (pixel value), the width and height (pixel length) of the detection frame, the tracking ID corresponding to each target detection frame, and the number of video frames in which it exists in the same detection file. The information of each line in the detection file is a description of a tracking target, and the content is the number of video frames in which the tracking target exists, the tracking ID of the target detection, the horizontal coordinate of the upper left corner vertex of the target detection frame, the vertical coordinate of the upper left corner vertex of the target detection frame, the width value of the target detection frame, and the height value of the target detection frame.

[0063] The above implementation scheme of the present application, by taking the test data as input, obtains a detection file which includes the frame sequence, coordinates, annotation box height and annotation box width corresponding to each tracked target, thereby facilitating the evaluation of the tracking effect of the target tracking algorithm based on the detection file.

[0064] In an optional embodiment of the present application, the tracking target in the test data is labeled to obtain a true value file, including:

[0065] Annotating the tracking target in each frame of the video image in the test data based on a preset annotation rule;

[0066] Obtaining the frame sequence, coordinates, annotation box height and annotation box width corresponding to the tracking target;

[0067] The true value file is determined according to the frame sequence, coordinates, annotation box height and annotation box width corresponding to each tracked target.

[0068] Specifically, after obtaining the test data, the detection file can be obtained through the model corresponding to the target tracking algorithm. In order to determine the tracking effect of the tracked target in the detection file, it is also necessary to determine the true value file corresponding to the test data. The true value file here includes the actual frame sequence, coordinates, annotation box height and annotation box width of each tracked target, so that the accuracy of target tracking in the detection file can be determined based on the true value file.

[0069] The embodiment of the present application can use the DarkLabel annotation tool to annotate the data of continuous multi-frame images and save the true value, thereby obtaining a true value file. The specific steps of tracking target annotation are as follows: Step 1, open the darklabel.yml file. Modify the format of the truth value file and the included items of the file content. The content of the truth value file is the number of video frames where the actual target exists, the tracking ID of the target, the horizontal coordinate of the vertex in the upper left corner of the target truth value box, the vertical coordinate of the vertex in the upper left corner of the target truth value box, the width value of the target truth value box, and the height value of the target truth value box. Modify the type of truth value annotation according to the needs of the project. The types of this technical solution include trucks, pedestrians, bicycles, cars, motorcycles, and buses. Step 2, open the DarkLabel software and load the continuous multi-frame images output by the scene generation module. Step 3, select the file annotation format customized in the first step. Step 4, you can manually annotate the first frame data, and then automatically annotate the subsequent frame targets. Step 5, determine whether the target should be annotated according to the annotation rules, and delete the annotation boxes that do not meet the annotation rules. Step 6, correct the positions of the remaining annotation boxes; Step 7, click the GT Load button to generate a true value file of the corresponding format.

[0070] The above implementation scheme of the present application obtains the frame sequence, coordinates, annotation box height and annotation box width corresponding to each tracked target by annotating the test data according to preset annotation rules, thereby facilitating the evaluation of the accuracy of the tracked target in the detection file obtained using the target tracking algorithm based on the true value file.

[0071] In an optional embodiment of the present application, the evaluation index includes at least one of the following: HOTA index, detection recall rate, detection precision rate, association recall rate, association precision rate, positioning accuracy, multi-target tracking accuracy rate, and multi-target tracking precision rate.

[0072] Specifically, according to the values ​​in the detection file and the true value file, the number of target detection frames and the number of target true value frames in each video image can be counted. According to the information of the detection frame and the true value frame combined with the Hungarian matching algorithm, the number of effective associations between the largest detection frame and the true value frame in each frame can be obtained, which is recorded as TP, the unmatched detection frame is recorded as FP, and the unmatched true value frame is recorded as FN.

[0073] For the evaluation of the quality of the multi-target tracking algorithm, the evaluation index HOTA of detection error, association error and positioning error can be comprehensively considered. The embodiment of this application will mainly use the HOTA index as the primary index for verifying the multi-target tracking algorithm in the self-built simulation scenario. The calculation formula of the HOTA index is as follows:

[0074]

[0075]

[0076]

[0077] Where c is the TP of Interest, i.e., the location where the TP occurs, and A(c) is the association score, which is the threshold for the score of the detection box and the ground truth box in Hungarian matching. In addition, TPA, FNA, and FPA are explained as follows.

[0078] TPA indicates the number of the following situations: The same target exists in the image of c in the kth frame and the most recent frame (but it should be ensured that it does not exceed the set matching time threshold), the true value boxes in both frames exist and have the same ID, and the detection boxes exist and the tracking ID has not changed.

[0079] FNA indicates the number of the following situations. Situation 1: The same target exists in image c in the kth frame and the most recent frame (but it should not exceed the set matching time threshold), the true value boxes exist in both frames and have the same ID, the detection boxes exist but the tracking ID jumps (actually the same target). Situation 2: The same target exists in image c in the kth frame and the most recent frame (but it should not exceed the set matching time threshold), the true value boxes exist in both frames and have the same ID, but the detection box in the kth frame does not exist (the target is not detected, missed detection).

[0080] FPA indicates the number of the following situations. Case 1: The same target exists in image c in the kth frame and the most recent frame (but it should be ensured that it does not exceed the set matching time threshold), the detection frames exist in both frames and the tracking ID is the same, but the IDs of the two corresponding truth frames are not the same ID (actually different targets). Case 2: The same target exists in image c in the kth frame and the most recent frame (but it should be ensured that it does not exceed the set matching time threshold), the detection frames exist in both frames and the tracking ID is the same, but the truth frame of the kth frame does not exist (false target detected, false detection) or the ID of this truth frame has changed from the corresponding truth frame ID of the previous frame (inaccurate detection and positioning, false matching occurred).

[0081] In addition, there are DetRe detection recall rate, DetPr detection precision rate, AssRe association recall rate, AssPr association precision rate, LocA positioning accuracy, MOTA multi-target tracking accuracy rate, MOTP multi-target tracking precision rate, MODA multi-target tracking accuracy rate excluding IDSW, the number of MT trajectories that are mostly tracked, the number of PT trajectories that are partially tracked, the number of ML trajectories that are mostly not tracked, and other indicators together constitute a complete multi-target tracking algorithm evaluation index system.

[0082] The above implementation scheme of the present application can evaluate the target tracking algorithm from multiple dimensions by setting the evaluation index to at least one of the following: HOTA index, detection recall rate, detection precision rate, association recall rate, association precision rate, positioning accuracy, multi-target tracking accuracy rate, and multi-target tracking precision rate.

[0083] The target tracking algorithm evaluation method provided in the embodiment of the present application is introduced above. The target tracking algorithm evaluation device provided in the embodiment of the present application will be introduced below in conjunction with the accompanying drawings.

[0084] like Figure 2 As shown, an embodiment of the present invention further provides a target tracking algorithm evaluation device, including a determination module 201, which is used to determine a scene set for evaluating the target tracking algorithm;

[0085] An acquisition module 202 is used to acquire, for each scene in the scene set, test data corresponding to the scene, wherein the test data includes a plurality of frames of video images corresponding to the scene;

[0086] The first processing module 203 is used to input the test data into the model corresponding to the target tracking algorithm to obtain a detection file;

[0087] The second processing module 204 is used to mark the tracking target in the test data to obtain a true value file;

[0088] The third processing module 205 is used to analyze the test value file and the true value file to obtain an evaluation index for characterizing the tracking effect of the target tracking algorithm.

[0089] Optionally, the determining module includes:

[0090] A first determination submodule is configured to determine a plurality of sub-scene sets based on a correspondence between at least one perception problem and at least one occurrence scenario corresponding to each of the perception problems, wherein there is no intersection between the plurality of sub-scene sets, and each of the sub-scene sets includes a plurality of scene elements;

[0091] The second determining submodule is used to combine the scene elements respectively included in the multiple sub-scene sets to obtain the scene set.

[0092] Optionally, get the module:

[0093] A first acquisition submodule is used to acquire, for each scenario, a test case corresponding to the scenario;

[0094] A building submodule is used to build a virtual scene corresponding to the test case on the simulation module according to the test case;

[0095] An adding submodule, used for adding a virtual image acquisition device to a host vehicle corresponding to the virtual scene;

[0096] The third determining submodule is used to determine the multiple frames of video images returned by the virtual image acquisition device as the test data.

[0097] Optionally, the first processing module includes:

[0098] A second acquisition submodule is used to input the test data into the model corresponding to the target tracking algorithm to obtain an output file, wherein the output file includes a frame sequence, coordinates, annotation box height and annotation box width corresponding to each tracked target;

[0099] The fourth determination submodule is used to determine the detection file according to the frame sequence, coordinates, annotation box height and annotation box width corresponding to each tracked target.

[0100] Optionally, the second processing module includes:

[0101] A labeling submodule, used for labeling the tracking target in each frame of the video image in the test data based on a preset labeling rule;

[0102] The third acquisition submodule is used to acquire the frame sequence, coordinates, annotation box height and annotation box width corresponding to the tracked target;

[0103] The fifth determination submodule is used to determine the true value file according to the frame sequence, coordinates, annotation box height and annotation box width corresponding to each tracked target.

[0104] Optionally, the evaluation index includes at least one of the following: HOTA index, detection recall rate, detection precision rate, association recall rate, association precision rate, positioning accuracy, multi-target tracking accuracy rate, and multi-target tracking precision rate.

[0105] The target tracking algorithm evaluation device provided in the present application determines test data based on a scene set, inputs the test data into a model corresponding to the target tracking algorithm to obtain a detection file, labels the tracking targets in the test data to obtain a true value file, and then obtains the tracking effect evaluation index of the target tracking algorithm by analyzing the detection file and the true value file. Since the test data is determined based on a scene set rather than directly using an existing data set, the test data can cover special scenarios, which is more conducive to testing the robustness of the target tracking algorithm.

[0106] The present application also provides an electronic device, such as Figure 3As shown, the electronic device may include: a processor 310, a communication interface 320, a memory 330 and a communication bus 340, wherein the processor 310, the communication interface 320 and the memory 330 communicate with each other through the communication bus 340. The processor 310 may call the logic instructions in the memory 330, and the processor 310 is used to perform the following steps: determine a scene set for evaluating the target tracking algorithm; for each scene in the scene set, obtain the test data corresponding to the scene, and the test data includes multiple frames of video images corresponding to the scene; input the test data into the model corresponding to the target tracking algorithm to obtain a detection file; mark the tracking target in the test data to obtain a true value file; analyze the test value file and the true value file to obtain an evaluation index for characterizing the tracking effect of the target tracking algorithm.

[0107] In addition, the logic instructions in the above-mentioned memory 330 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art, and the computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk.

[0108] An embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the various processes of the above-mentioned target tracking algorithm evaluation method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0109] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive Solid State Disk (SSD)), etc.

[0110] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0111] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0112] The above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application are included in the protection scope of the present application.

Claims

1. A target tracking algorithm evaluation method, characterized in that: The method comprises: Determine the set of scenarios for evaluating the object tracking algorithm; For each scene in the scene set, acquiring test data corresponding to the scene, the test data comprising a plurality of frames of video images corresponding to the scene; Inputting the test data into the model corresponding to the target tracking algorithm to obtain a detection file; Annotating the tracking target in the test data to obtain a true value file; The test value file and the true value file are analyzed to obtain an evaluation index for characterizing the tracking effect of the target tracking algorithm.

2. The target tracking algorithm evaluation method according to claim 1, characterized in that: The scene set determined for evaluating the target tracking algorithm includes: Determine, based on a correspondence between at least one perception problem and at least one occurrence scenario corresponding to each perception problem, a plurality of sub-scene sets, wherein there is no intersection between the plurality of sub-scene sets, and each of the sub-scene sets includes a plurality of scene elements; The scene elements respectively included in the multiple sub-scene sets are combined to obtain the scene set.

3. The target tracking algorithm evaluation method according to claim 1, characterized in that: For each scene in the scene set, obtaining test data corresponding to the scene includes: For each scenario, obtain a test case corresponding to the scenario; According to the test case, a virtual scene corresponding to the test case is built on the simulation module; Adding a virtual image acquisition device to a host vehicle corresponding to the virtual scene; The multiple frames of video images returned by the virtual image acquisition device are determined as the test data.

4. The target tracking algorithm evaluation method according to claim 1, characterized in that: Input the test data into the model corresponding to the target tracking algorithm to obtain a detection file, including: Input the test data into the model corresponding to the target tracking algorithm to obtain an output file, wherein the output file includes a frame sequence, coordinates, annotation box height and annotation box width corresponding to each tracked target; The detection file is determined according to the frame sequence, coordinates, height of the annotation box and width of the annotation box corresponding to each tracked target.

5. The target tracking algorithm evaluation method according to claim 1, characterized in that: Annotating the tracking targets in the test data to obtain a true value file, including: Annotating the tracking target in each frame of the video image in the test data based on a preset annotation rule; Obtaining the frame sequence, coordinates, annotation box height and annotation box width corresponding to the tracking target; The true value file is determined according to the frame sequence, coordinates, annotation box height and annotation box width corresponding to each tracked target.

6. The target tracking algorithm evaluation method according to claim 1, characterized in that: The evaluation index includes at least one of the following: HOTA index, detection recall rate, detection precision rate, association recall rate, association precision rate, positioning accuracy, multi-target tracking accuracy rate, and multi-target tracking precision rate.

7. A target tracking algorithm evaluation device, characterized in that: include: A determination module, used to determine a set of scenes for evaluating the target tracking algorithm; An acquisition module, configured to acquire, for each scene in the scene set, test data corresponding to the scene, wherein the test data includes a plurality of frames of video images corresponding to the scene; A first processing module, used for inputting the test data into a model corresponding to the target tracking algorithm to obtain a detection file; A second processing module is used to mark the tracking target in the test data to obtain a true value file; The third processing module is used to analyze the test value file and the true value file to obtain an evaluation index for characterizing the tracking effect of the target tracking algorithm.

8. The target tracking algorithm evaluation device according to claim 7, characterized in that: The determination module comprises: A first determination submodule is configured to determine a plurality of sub-scene sets based on a correspondence between at least one perception problem and at least one occurrence scenario corresponding to each of the perception problems, wherein there is no intersection between the plurality of sub-scene sets, and each of the sub-scene sets includes a plurality of scene elements; The second determining submodule is used to combine the scene elements respectively included in the multiple sub-scene sets to obtain the scene set.

9. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor is used to implement the steps in the target tracking algorithm evaluation method described in any one of claims 1 to 6 when executing the program stored in the memory.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the target tracking algorithm evaluation method according to any one of claims 1 to 6 are implemented.

Images