Method for evaluating a target detection model, method and device for obtaining a test set

Abstract

The application discloses a method and device for evaluating a target detection model in a track tracking system and a method and device for obtaining a test set. The evaluation process comprises the following steps: obtaining a test set comprising a labeled result with a track identifier based on the method for obtaining a test set provided by the application, obtaining a test result according to a plurality of pictures in the test set and a target detection model of the track tracking system; and calculating a value of a track evaluation index of the target detection model according to the test result and the labeled result, wherein the track evaluation index is used to reflect the working condition of the target detection model in the track tracking system. It can be seen that the working condition of the target detection model in the track tracking system can be reasonably evaluated through the newly defined track evaluation index of the target detection model, and the obtaining of the test set also does not need to consume a large amount of labor cost.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, in particular to a method and device for evaluating a target detection model in a trajectory tracking system, and a method and device for obtaining a test set. BACKGROUND

[0002] The target detection model can detect targets (such as people, vehicles, etc.) in a picture. The trajectory tracking system obtains the moving trajectory of the target in space in the video by means of the target detection model and trajectory generation logic. The accuracy of the target detection model in the trajectory tracking system is crucial to the trajectory tracking system. Currently, the parameters of the target detection model are usually evaluated in the granularity of the detection box of the target detected by the target detection model, which reflects the good or bad of the target detection model, but cannot reflect whether the target detection model meets the use requirements of the trajectory tracking system.

[0003] Therefore, there is an urgent need to provide a technical solution that can reflect the working condition of the target detection model in the trajectory tracking system, for evaluating the good or bad of the target detection model in the trajectory tracking system, so that high-quality trajectory tracking systems can be used by users and provide better services for users. SUMMARY

[0004] The embodiments of the present application provide a method and device for evaluating a target detection model in a trajectory tracking system, which can reasonably evaluate the working condition of the target detection model in the trajectory tracking system by a newly defined index, so that it is possible to provide high-quality trajectory tracking systems for users.

[0005] In addition, the embodiments of the present application also provide a method and device for obtaining a test set, which can obtain a labeled result with a trajectory identifier by inputting multiple pictures into an existing trajectory tracking system and correcting the result output by the trajectory tracking system. The multiple pictures and the corresponding labeled result are used as a test set for the method of evaluating the target detection model in the trajectory tracking system, which saves the human cost consumed by the current manual labeling method for generating a test set and improves the efficiency of generating a test set and evaluating the target detection model in the trajectory tracking system.

[0006] In a first aspect, the embodiments of the present application provide a method for evaluating a target detection model in a trajectory tracking system, comprising:

[0007] obtaining a test result based on multiple pictures in a test set and a target detection model of a trajectory tracking system, wherein the test set comprises labeled results of the multiple pictures, the test result and the labeled result both comprise number and detection box position information of a picture in a video frame, and the labeled result further comprises a trajectory identifier;

[0008] According to the test result and the annotation result, a value of a trajectory evaluation index of the target detection model is obtained, the trajectory evaluation index being used to reflect a working condition of the target detection model in the trajectory tracking system.

[0009] Optionally, the trajectory evaluation index comprises a trajectory recall rate and / or a trajectory breaking rate, the trajectory recall rate being a ratio of a number of recalled trajectories to a total number of trajectories, and the trajectory breaking rate being a ratio of a total number of times of breaking of all trajectories to the number of recalled trajectories, a target being recorded as a recalled trajectory if the target is detected in N consecutive frames, the N being an integer greater than or equal to 1.

[0010] Optionally, the target detection model of the trajectory tracking system is used to obtain a test result based on a plurality of pictures in a test set.

[0011] The plurality of pictures in the test set are input into the target detection model of the trajectory tracking system to obtain a first intermediate result.

[0012] Based on a first threshold, invalid detection boxes in the first intermediate result are removed to obtain the test result.

[0013] Optionally, the test result is obtained based on the first threshold and the removal of the invalid detection boxes in the first intermediate result, and the obtaining of the test result comprises:

[0014] Based on the first threshold, invalid detection boxes in the first intermediate result are removed to obtain a second intermediate result.

[0015] A target trajectory identifier is added to each detection box in the second intermediate result to obtain the test result.

[0016] Optionally, the value of the trajectory evaluation index of the target detection model is calculated based on the test result and the annotation result, and the calculating comprises:

[0017] An evaluation result is obtained based on the test result and the annotation result, the evaluation result comprising a corresponding relationship between each trajectory identifier, a number of times of appearance of the trajectory in the test result, and a number of times of appearance of the trajectory in the annotation result, the evaluation result being used to represent the appearance of each trajectory in the test result and the annotation result.

[0018] The value of the trajectory evaluation index of the target detection model is calculated based on the evaluation result.

[0019] Optionally, the test set comprises a first picture, the test result comprises a first test box of the first picture, and the annotation result comprises a first annotation box of the first picture, and the evaluation result is obtained based on the test result and the annotation result, and the obtaining comprises:

[0020] calculate a first Intersection over Union (IoU) of the first test bounding box and the first annotation bounding box;

[0021] determine that the first IoU meets a preset condition, set a track identifier of the first test bounding box as a first track identifier of the first annotation bounding box, the preset condition including that a maximum value in the IoU is greater than a second threshold.

[0022] Optionally, the test set includes a first picture, the test result includes a first test bounding box and a second test bounding box of the first picture, the annotation result includes a first annotation bounding box and a second annotation bounding box of the first picture, and the evaluation result obtained according to the test result and the annotation result for the first picture includes:

[0023] calculate a first IoU of the first test bounding box and the first annotation bounding box, a second IoU of the first test bounding box and the second annotation bounding box, a third IoU of the second test bounding box and the first annotation bounding box, and a fourth IoU of the second test bounding box and the second annotation bounding box;

[0024] select a first IoU meeting a preset condition from the first IoU and the second IoU, set a track identifier of the first test bounding box as a first track identifier of the first annotation bounding box, and select a fourth IoU meeting a preset condition from the third IoU and the fourth IoU, set a track identifier of the second test bounding box as a second track identifier of the second annotation bounding box, and the evaluation result includes the first track identifier of the first test bounding box and the second track identifier of the second test bounding box. Optionally, the preset condition includes that a maximum value in the IoU is greater than a second threshold. Then, the first IoU meeting the preset condition can mean that the value of the first IoU is greater than the value of the second IoU, and the value of the first IoU is greater than the second threshold; the fourth IoU meeting the preset condition can mean that the value of the fourth IoU is greater than the value of the third IoU, and the value of the fourth IoU is greater than the second threshold.

[0025] Optionally, the first picture further includes a third test bounding box, and the evaluation result obtained according to the test result and the annotation result for the first picture further includes:

[0026] calculate a fifth IoU of the third test bounding box and the first annotation bounding box and a sixth IoU of the third test bounding box and the second annotation bounding box;

[0027] If the IoU in the fifth IoU and the sixth IoU does not meet the preset condition, the trajectory identification of the third test frame is set as a target trajectory identification, and the evaluation result further includes the target trajectory identification of the third test frame. Assuming that the value of the fifth IoU is greater than the value of the sixth IoU, the determination that the IoU in the fifth IoU and the sixth IoU does not meet the preset condition can mean that the value of the fifth IoU is not greater than a second threshold. Assuming that the value of the fifth IoU is less than the value of the sixth IoU, the determination that the IoU in the fifth IoU and the sixth IoU does not meet the preset condition can mean that the value of the sixth IoU is not greater than the second threshold.

[0028] Optionally, the test set is obtained by the following steps:

[0029] The plurality of pictures are input into the target detection module of the trajectory tracking system to obtain information of an initial annotation frame of each picture in the plurality of pictures, the information of the initial annotation frame including a number of the picture in a video frame, detection frame position information, and trajectory identification;

[0030] According to a preset detection frame supplement model and the information of the initial annotation frame of each picture in the plurality of pictures, annotation frames of pictures in the plurality of pictures that have a missing annotation frame phenomenon are supplemented to obtain information of an intermediate annotation frame of each picture in the plurality of pictures, the information of the intermediate annotation frame including the information of the initial annotation frame and further including information of a supplemented frame;

[0031] A test set is obtained according to the plurality of pictures and the corresponding information of the intermediate annotation frame.

[0032] Optionally, in one case, the annotation result includes the information of the intermediate annotation frame of each picture in the plurality of pictures; in another case, the obtaining of the test set according to the information of the intermediate annotation frame includes: correcting the information of the intermediate annotation frame to obtain information of a final annotation frame; and obtaining the test set according to the information of the final annotation frame, the annotation result in the test set including the information of the final annotation frame of each picture in the plurality of pictures.

[0033] In a second aspect, an embodiment of the present application further provides a device for evaluating a target detection model in a trajectory tracking system, comprising:

[0034] A first obtaining unit is configured to obtain a test result based on a plurality of pictures in a test set and a target detection model of a trajectory tracking system, the test set including annotation results of the plurality of pictures, the test result and the annotation results both including a number of a picture in a video frame and detection frame position information, and the annotation results further including trajectory identification;

[0035] The second obtaining unit is configured to obtain a value of a track evaluation index of the target detection model according to the test result and the annotation result, the track evaluation index being used to reflect a working condition of the target detection model in the track tracking system.

[0036] Optionally, the track evaluation index comprises a track recall rate and / or a track breaking rate, the track recall rate being a ratio of a number of recalled tracks to a total number of tracks, and the track breaking rate being a ratio of a total number of times of breaking of all tracks to the number of recalled tracks, a target being recorded as a recalled track if the target is detected in N consecutive frames, the N being an integer greater than or equal to 1.

[0037] Optionally, the first obtaining unit comprises:

[0038] The first obtaining sub-unit is configured to input a plurality of pictures in the test set into the target detection model of the track tracking system to obtain a first intermediate result.

[0039] The screening sub-unit is configured to remove invalid detection boxes in the first intermediate result based on a first threshold to obtain the test result.

[0040] Optionally, the screening sub-unit is specifically configured to:

[0041] remove invalid detection boxes in the first intermediate result based on the first threshold to obtain a second intermediate result;

[0042] add a target track identifier to each detection box in the second intermediate result to obtain the test result.

[0043] Optionally, the second obtaining unit comprises:

[0044] The second obtaining sub-unit is configured to obtain an evaluation result according to the test result and the annotation result, the evaluation result comprising a corresponding relationship between each track identifier, a number of times of appearance of the track in the test result, and a number of times of appearance of the track in the annotation result, the evaluation result being used to represent appearance of each track in the test result and the annotation result.

[0045] The calculating sub-unit is configured to calculate the value of the track evaluation index of the target detection model according to the evaluation result.

[0046] Optionally, the test set comprises a first picture, the test result comprises a first test box of the first picture, the annotation result comprises a first annotation box of the first picture, and for the first picture, the second obtaining sub-unit is specifically configured to:

[0047] calculate a first IoU of the first test box and the first annotation box.

[0048] determining that the first IoU meets a preset condition, setting a track identifier of the first test box as a first track identifier of the first labeled box, the preset condition including: a maximum value in the IoU is greater than a second threshold.

[0049] Optionally, the test set includes a first picture, the test results include a first test box and a second test box of the first picture, the labeled results include a first labeled box and a second labeled box of the first picture, and for the first picture, the second obtaining subunit is specifically configured to:

[0050] calculate a first overlap IoU of the first test box and the first labeled box, a second IoU of the first test box and the second labeled box, a third IoU of the second test box and the first labeled box, and a fourth IoU of the second test box and the second labeled box;

[0051] select the first IoU that meets a preset condition from the first IoU and the second IoU, set a track identifier of the first test box as a first track identifier of the first labeled box, and select the fourth IoU that meets the preset condition from the third IoU and the fourth IoU, set a track identifier of the second test box as a second track identifier of the second labeled box, and the evaluation result includes the first track identifier of the first test box and the second track identifier of the second test box. Optionally, the preset condition includes: a maximum value in the IoU is greater than a second threshold.

[0052] Optionally, the first picture further includes a third test box, and for the first picture, the second obtaining subunit is further configured to:

[0053] calculate a fifth IoU of the third test box and the first labeled box and a sixth IoU of the third test box and the second labeled box;

[0054] determine that neither of the fifth IoU and the sixth IoU meets the preset condition, set a track identifier of the third test box as a target track identifier, and the evaluation result further includes the target track identifier of the third test box.

[0055] Optionally, the device further includes a third obtaining unit configured to obtain the test set, wherein the third obtaining unit includes:

[0056] The third obtaining subunit is used to input the multiple images into the target detection module of the trajectory tracking system to obtain the information of the initial annotation box of each image in the multiple images. The information of the initial annotation box includes the image number in the video frame, the detection box position information and the trajectory identifier.

[0057] The fourth obtaining subunit is used to supplement the annotation boxes of the images with missing annotation boxes in the multiple images according to the preset detection box supplementation model and the information of the initial annotation box of each image in the multiple images, and to obtain the information of the intermediate annotation box of each image in the multiple images. The information of the intermediate annotation box includes the information of the initial annotation box, and the annotation result includes the information of the intermediate annotation box of each image in the multiple images. The information of the intermediate annotation box also includes the information of the supplemented box.

[0058] The fifth subunit is used to obtain a test set based on the information of the multiple images and their corresponding intermediate annotation boxes.

[0059] Optionally, the annotation result includes information about the middle annotation box of each of the plurality of images;

[0060] or,

[0061] The fifth obtaining subunit is specifically used for:

[0062] Correct the information of the intermediate annotation boxes to obtain the information of the final annotation boxes;

[0063] The test set is obtained based on the information of the final annotation boxes, and the annotation results include the information of the final annotation boxes of each of the multiple images.

[0064] Thirdly, embodiments of this application also provide a method for obtaining a test set, which may include, for example:

[0065] Multiple images are input into the trajectory tracking system to obtain the initial annotation box information for each image. The initial annotation box information includes the image number in the video frame, the detection box position information, and the trajectory identifier.

[0066] Based on the preset detection box supplementation model and the information of the initial annotation box of each image in the multiple images, the annotation boxes of images with missing annotation boxes are supplemented to obtain the information of the intermediate annotation box of each image in the multiple images. The information of the intermediate annotation box includes the information of the initial annotation box and the information of the supplemented box.

[0067] A test set is obtained based on the multiple images and the information of their corresponding intermediate annotation boxes.

[0068] Optionally, the test set is used in the method provided in the first aspect above.

[0069] Optionally, in one case, the annotation result includes information about the intermediate annotation boxes of each of the plurality of images. Alternatively, in another case, obtaining the test set based on the intermediate annotation box information includes: correcting the intermediate annotation box information to obtain final annotation box information; obtaining the test set based on the final annotation box information, wherein the annotation result in the test set includes information about the final annotation boxes of each of the plurality of images.

[0070] Fourthly, embodiments of this application also provide an apparatus for acquiring a test set, which may include:

[0071] The first obtaining unit is used to input multiple images into the trajectory tracking system and obtain the information of the initial annotation box of each image in the multiple images. The information of the initial annotation box includes the image number in the video frame, the detection box position information and the trajectory identifier.

[0072] The second obtaining unit is used to supplement the annotation boxes of the images with missing annotation boxes in the multiple images according to the preset detection box supplementation model and the information of the initial annotation box of each image in the multiple images, and obtain the information of the intermediate annotation box of each image in the multiple images. The information of the intermediate annotation box includes the information of the initial annotation box and the information of the supplemented box.

[0073] The third obtaining unit is used to obtain a test set based on the information of the multiple images and their corresponding intermediate annotation boxes. The test set includes the multiple images and the annotation result of each image in the multiple images.

[0074] Optionally, the test set is used in the method provided in the first aspect above.

[0075] Optionally,

[0076] The annotation results include information about the middle annotation box of each of the multiple images;

[0077] or,

[0078] The third obtaining unit includes:

[0079] The correction subunit is used to correct the information of the intermediate annotation box to obtain the information of the final annotation box.

[0080] A subunit is obtained for obtaining the test set based on the information of the final annotation box, wherein the annotation result in the test set includes the information of the final annotation box of each of the multiple images.

[0081] Fifthly, embodiments of this application also provide an electronic device, the electronic device including a processor and a memory:

[0082] The memory is used to store computer programs;

[0083] The processor is configured to execute the method provided in the first or third aspect above, according to the computer program.

[0084] In a sixth aspect, embodiments of this application also provide a computer-readable storage medium for storing a computer program for performing the methods provided in the first or third aspect above.

[0085] Therefore, the embodiments of this application have the following beneficial effects:

[0086] This application provides a method for evaluating a target detection model in a trajectory tracking system, comprising: obtaining test results based on multiple images in a test set and the target detection model of the trajectory tracking system; wherein the test set includes the annotation results of the multiple images, and both the test results and the annotation results include the image's number in a video frame, trajectory identifier, and detection box position information; then, obtaining the value of a trajectory evaluation index of the target detection model based on the test results and the annotation results; the trajectory evaluation index is used to reflect the performance of the target detection model in the trajectory tracking system. It can be seen that, through this method, a newly defined trajectory evaluation index for the target detection model (such as trajectory recall and / or trajectory disconnection rate) is used to evaluate the quality of the target detection model in the trajectory tracking system. That is, the trajectory evaluation index not only reflects the state of the target detection model but also reflects the state of trajectory tracking including the target detection model. Thus, with the help of this newly defined trajectory evaluation index, the performance of the target detection model in the trajectory tracking system can be reasonably evaluated, thereby making it possible to provide users with a high-quality trajectory tracking system.

[0087] Furthermore, the present application provides a method and apparatus for obtaining a test set. By inputting multiple images into an existing trajectory tracking system and correcting the output of the trajectory tracking system, a labeled result with trajectory identifiers can be obtained. These multiple images and the corresponding labeled result serve as the test set used to evaluate the target detection model in the trajectory tracking system. This saves the manpower cost of generating the test set using the current manual annotation method and improves the efficiency of generating the test set and evaluating the target detection model in the trajectory tracking system. Attached Figure Description

[0088] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments recorded in the present invention. For those skilled in the art, other drawings can be obtained based on these drawings.

[0089] Figure 1 This is a diagram illustrating the meaning of IoU;

[0090] Figure 2 A flowchart illustrating a method for evaluating a target detection model in a trajectory tracking system, provided as an embodiment of this application;

[0091] Figure 3 A schematic diagram of the structure of a device for evaluating a target detection model in a trajectory tracking system, provided in an embodiment of this application;

[0092] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0093] To make the above-mentioned objectives, features, and advantages of this application more apparent and understandable, the embodiments of this application will be further described in detail below with reference to the accompanying drawings and specific implementation methods. It should be understood that the specific embodiments described herein are merely for explaining this application and are not intended to limit it. Furthermore, it should be noted that, for ease of description, only the parts relevant to this application are shown in the accompanying drawings, not the entire structure.

[0094] An object detection model is used to detect the location of objects in an image. The object can be all or part of a human body, vehicle, etc. For example, a human detection model can detect human bodies in an image, a face detection model can detect faces, a vehicle detection model can detect vehicles, and a license plate detection model can detect license plates. The input of an object detection model is an image, and the output is the location information of the objects in the image. Specifically, it can be an image with bounding boxes that include all or part of the object.

[0095] A trajectory tracking system is used to display the movement trajectory of a target in space. The system includes at least one target detection model and a trajectory generation module. The trajectory generation module integrates logic to generate a trajectory based on the target's position in various images, using the output of the target detection model to generate the trajectory. It should be noted that the target detection model in the trajectory tracking system can be designed for a single target (such as a human body) to achieve trajectory tracking for that target; or it can be designed for multiple targets (such as human bodies, vehicles, etc.) to achieve trajectory tracking for multiple targets, depending on the specific requirements.

[0096] Currently, the performance of object detection models in trajectory tracking systems is typically evaluated using independent object detection model metrics, such as accuracy, precision, recall rate, PR curve, average precision (AP), and mean average precision (mAP). In evaluating object detection models, images are first randomly selected as a test set. Annotators mark the locations of all targets in these images; these marked locations are called bounding boxes. Next, the images from the test set are input into the object detection model to be evaluated to obtain the test boxes. Then, by comparing the bounding boxes and test boxes of all images in the test set, the IoU (Intersection over Union) between the corresponding bounding boxes and test boxes is calculated, yielding the values ​​of metrics such as mAP. Here, IoU can be understood as the intersection-union ratio, such as... Figure 1 As shown, assuming the labeled box is 1 and the test box is 2, then the intersection of 1 and 2 is A, and the union of 1 and 2 is B. Therefore, the IoU between 1 and 2 can be equal to A divided by B. Both the labeled box and the test box can be considered as detection boxes of the target in the image, and their range includes all or most of the target in the image.

[0097] However, the evaluation of the target detection model obtained through the current method is not related to the trajectory to which the detection box belongs. It can only reflect the target detection performance when the target detection model works alone, but cannot reflect the performance of the target detection model in the trajectory tracking system. That is, it cannot reflect whether the target detection model can meet the usage requirements of the trajectory tracking system to which it belongs.

[0098] For better description, the following text uses a pedestrian trajectory tracking system as an example, and the target detection model corresponds to a human body detection model. Based on research on current trajectory tracking systems and target detection models within them, the inventors have found that the evaluation of the target detection module in a trajectory tracking system should consider the following two issues: First, whether the trajectory tracking system can track all people within a certain period of time; second, whether the trajectory tracking system can track the location of a person at any time within a certain period of time, that is, the completeness of tracking the trajectories of all people who appear.

[0099] Based on this, embodiments of this application provide a method for evaluating a target detection model in a trajectory tracking system, comprising: obtaining test results based on multiple images in a test set and the target detection model of the trajectory tracking system, wherein the test set includes the annotation results of the multiple images, and both the test results and the annotation results include the image number in the video frame and the detection box position information, and the annotation results also include trajectory identifiers; then, obtaining the value of the trajectory evaluation index of the target detection model according to the test results and the annotation results, wherein the trajectory evaluation index is used to reflect the working performance of the target detection model in the trajectory tracking system.

[0100] As can be seen, through this method, a new trajectory evaluation index for the target detection model (such as trajectory recall and / or trajectory disconnection rate) is defined. This trajectory evaluation index is used to evaluate the quality of the target detection model in the trajectory tracking system. That is, this trajectory evaluation index not only reflects the state of the target detection model, but also reflects the state of trajectory tracking including the target detection model. In this way, with the help of this newly defined trajectory evaluation index, the working performance of the target detection model in the trajectory tracking system can be reasonably evaluated, thus making it possible to provide users with a high-quality trajectory tracking system.

[0101] To facilitate understanding of the specific implementation of the method for evaluating the target detection model in the trajectory tracking system provided in the embodiments of this application, the following description will be provided in conjunction with the accompanying drawings.

[0102] It should be noted that the main body implementing the method for evaluating the target detection model in the trajectory tracking system can be the device for evaluating the target detection model in the trajectory tracking system provided in the embodiments of this application. This device can be carried in an electronic device or a functional module of an electronic device. The electronic device in the embodiments of this application can be any device capable of implementing the method for evaluating the target detection model in the trajectory tracking system in the embodiments of this application, such as an IoT device.

[0103] Figure 2 This is a flowchart illustrating a method for evaluating a target detection model in a trajectory tracking system, provided as an embodiment of this application. This method can be applied to, for example... Figure 3 The device 300 shown is used to evaluate the target detection model in the trajectory tracking system, or it can also be applied to... Figure 4 In the electronic device 400 shown.

[0104] like Figure 2 As shown, the method may include the following steps S201 to S202:

[0105] S201, Based on multiple images in the test set and the target detection model of the trajectory tracking system, test results are obtained. The test set includes the annotation results of the multiple images. Both the test results and the annotation results include the image number in the video frame and the detection box position information. The annotation results also include trajectory identifiers.

[0106] The test set includes at least one video segment, and multiple images within that video segment are labeled with annotations, including bounding boxes and annotation information. Each bounding box covers all or most of a target. For example, if the target is a human body, each image in the test set includes a bounding box, and the area of ​​each bounding box covers all or most of a person's body (e.g., the upper body). The annotation information includes the image's frame number in the video, the bounding box (or detection box) location information, and trajectory identifiers.

[0107] In one embodiment of this application, the test set can be obtained by labeling and manually marking trajectory identifiers based on the current test set acquisition method; in another embodiment, in order to reduce the workload of obtaining the test set, this application also provides a new method for obtaining the test set.

[0108] As an example, obtaining a test set could involve sampling the video to obtain multiple images, and then labeling each sampled image with target information to obtain the test set. For instance, for a 10-minute video with a frame rate of 25 frames per second, the video would have a total of (25 * 60 * 10 = 15000) frames. 1000 frames could be extracted from these 15000 frames, and target information and trajectory markers could be added to these 1000 extracted frames to obtain the test set. However, the test set obtained in this way is incomplete and cannot fully cover all target information within the corresponding time period of the video. Alternatively, obtaining a test set could also involve not sampling the video, but labeling all images included in the video with target information and trajectory markers to obtain the test set. However, in current methods for obtaining test sets, image labeling is usually done manually, which results in a massive workload for labeling. Assuming an annotator can annotate 250 images per hour, sampling 1000 images would require an annotator to work continuously for 4 hours to complete the annotation; while full annotation without sampling would require an annotator to work continuously for 60 hours. Furthermore, although full annotation of all videos in the test set without sampling can enrich the information carried by the test set samples, the test set obtained in this way can only calculate the overall precision, recall, and mAP of the detection boxes, and cannot be further correlated with the performance of the trajectory tracking system, nor can it provide details on how many trajectories were detected.

[0109] Furthermore, to ensure precision in identifying the number of detected trajectories, each image in the test set requires additional annotation of the trajectory identifier (also known as trajectory ID) for each detection box. This trajectory identifier uniquely identifies the trajectory, but this undoubtedly represents a significant annotation cost, especially for scenarios with numerous targets, such as large job fairs, airports, and train stations—places with high population density. Performing human detection and trajectory tracking on videos in these scenarios incurs substantial annotation costs. Therefore, this application's embodiments design a test set construction method with low annotation costs.

[0110] As another example, the method for obtaining a test set provided in this application embodiment may include, for example, the following steps: S11, inputting the multiple images into the target detection module of the trajectory tracking system to obtain the initial bounding box information for each image in the multiple images, wherein the initial bounding box information includes the image number in the video frame, the detection box position information, and the trajectory identifier; S12, supplementing the bounding boxes for images with missing bounding boxes in the multiple images according to a preset detection box supplementation model and the initial bounding box information for each image in the multiple images to obtain the intermediate bounding box information for each image in the multiple images, wherein the intermediate bounding box information includes the initial bounding box information and the supplemented box information, wherein the supplemented box may refer to the detection box that was not identified after S11 but was identified and marked by the detection box supplementation model; S13, obtaining a test set based on the intermediate bounding box information, wherein the test set includes the multiple images and the annotation results for each image in the multiple images. The annotation result includes information about the intermediate annotation box of each of the multiple images; optionally, it may also include S13, correcting the information of the final annotation box based on the information of the intermediate annotation box, wherein the annotation result includes information about the final annotation box of each of the multiple images, and the information of the final annotation box is corrected based on the information of the intermediate annotation box.

[0111] For S11, in one scenario, the pedestrian trajectory tracking function can be implemented based on the existing detection model and the Person Re-identification (ReID) model, marking detection boxes and trajectory identifiers for each pedestrian in the video, and obtaining the initial annotation box information for each image in the test set; or, in another scenario, the trajectory tracking system to be evaluated can be used to implement the pedestrian trajectory tracking function, marking detection boxes and trajectory identifiers for each pedestrian in the video, and obtaining the initial annotation box information for each image in the test set. The initial annotation box information may include: the image number in the video frame, the coordinate position of the annotation box in the image, and the trajectory identifier. For example, the initial annotation box information may include: {"1", [[1445,121,105,134],9],[[1698,202,112,134],10]}, where "1" represents that the image is the first frame of the video, [[1445,121,105,134],9] represents that the coordinate position of the pedestrian with trajectory identifier 9 in the image is [1445,121,105,134], and similarly, [[1698,202,112,134],10] represents that the coordinate position of the pedestrian with trajectory identifier 10 in the image is [1698,202,112,134]. The coordinates are typically an array of [x, y, w, h], where x and y represent the top-left corner coordinates of the person's bounding box in the image, and w and h represent the width and height of the bounding box, respectively. There is a one-to-one correspondence between the trajectory and the target. If there are three pedestrians in the image, the initial bounding box information for that image can include three bounding boxes, each corresponding to a trajectory identifier.

[0112] For S12, the preset detection box supplementation model can be a program module designed for the output of S11. It is used to check whether each image in the output of S11 is missing a detection box. If there is a missing detection box, the program module will automatically supplement the detection boxes for the images with missing detection boxes. The input of the detection box supplementation model is the output of S11, and the output is the information of the intermediate annotation boxes obtained after automatic detection box supplementation. The automatic detection box supplementation strategy designed in the detection box supplementation model may include, but is not limited to: Method 1, selecting the detection box of image 2 that is closest to image 1 with a missing detection box as the detection box 2 in image 1, so that image 1 includes detection box 2, and the information of detection box 2 is the information of detection box 1, and the detection box 2 of image 1 is the supplemented box; Method 2, calculating the average of the detection boxes of multiple images (such as image 2 and image 3) that are closest to image 1 with a missing detection box as the detection box 2 of image 1. For example, calculating the average of detection box 2 and detection box 3 of images 2 and 3 that are closest to image 1 with a missing detection box as detection box 1', so that image 1 includes the information of detection box 1', and the information of detection box 1' is the average of the information of detection box 2 and the information of detection box 3, and the detection box 1' of image 1 is the supplemented box.

[0113] In one scenario, a test set can be obtained based on S11 to S13. In this case, each image in the test set obtained in S13 includes information about the initial detection box obtained in S11 and information about the intermediate detection box obtained in S12.

[0114] In another scenario, to ensure greater accuracy of the samples in the test set, S13 may also include manual verification and correction of the results output by S12. S13 may include, for example,: readjusting the positions of detection boxes that are particularly inaccurate in the S12 output; re-labeling detection boxes that are obviously missing in the S12 output; correcting obvious errors such as trajectory labels in the S12 output; and supplementing detection boxes for individuals who have no trajectory in S12 and generating corresponding trajectory labels.

[0115] It should be noted that the detection boxes in the test set generated based on S11 to S13 are not necessarily accurate detection boxes, but can be understood as relatively accurate detection boxes. When the sample size of the test set is very large, the disadvantage that the detection boxes are not completely accurate can be ignored and they can be used as completely accurate detection boxes. That is, the test set obtained in this way can be used to evaluate the target detection model in the trajectory tracking system.

[0116] It should be noted that after S11, the information of the initial annotation box can be improved based solely on the preset detection box supplementation model, and no further verification and supplementation will be performed in S13, directly generating the test set; or, after S11, the information of the initial annotation box can be improved without relying on the preset detection box supplementation model, and the information of the initial annotation box can be corrected only in S13 to generate the test set; or, after S11, the information of the initial annotation box can be improved based on the preset detection box supplementation model, and verification and supplementation can also be performed on the information of the initial annotation box in S13, and the order of improving the information of the initial annotation box based on the preset detection box supplementation model and correcting the information of the initial annotation box in S13 is not limited.

[0117] The following describes in detail the method for obtaining the test set provided in the embodiments of this application:

[0118] The first step is data acquisition: extract each image from each video in the test set. For example, you can use the VideoCapture() function of OpenCV2, taking the video as input and outputting each image from the video.

[0119] The second step involves initial bounding box information for each image: Each image from the video is input sequentially into the trajectory tracking system, which outputs the trajectory of all people in each image. The output can be stored as a dictionary in a JSON file, for example, named gt.json. The dictionary's key is the frame number of each image in the video segment, and the value corresponding to each key is the position coordinates of the bounding box and its trajectory identifier for each person included in that image. For example, the output gt.json can be: {"1":[[[1445,121,105,134],9],[[1698,202,112,134],10]],"2":[[[1673,203,109,137],10]],"3":[[[1403,141,107,128],9],[[1640,199,110,139],11]]}, which means that in the first frame of the video, there are two people, and their trajectory identifiers are 9 and 10 respectively; in the second frame, there is one person, and the trajectory identifier is 10; in the third frame, there are two people, and their trajectory identifiers are 9 and 11 respectively.

[0120] The third step is manual annotation. For example, merging and splitting the trajectory markers of one person's disconnection. Assuming gt.json {"1":[[[1445,121,105,134],9],[[1698,202,112,134],10]],"2":[[[1673,203,109,137],10]],"3":[[[1403,141,107,128],9],[[1640,199,110,139],11]]}, if 10 and 11 actually identify the trajectory of one person, then trajectory markers 10 and 11 can be merged into 10. That is, the content of gt.json can be modified to: {"1 ":[[[1445,121,105,134],9],[[1698,202,112,134],10]],"2":[[[1673,203,109,137],10]],"3":[[[1403,141,107,128],9],[[1640,199,110,139],10]]};If 9 and 10 overlap in the second image, actually representing the trajectories of two people, then the content of gt.json can be modified to: {"1":[[[1445,121,105,134],9],[[1698,202,112,134],10]],"2":[ [[1673,203,109,137],9] ,[[1673,203,109,137],10]],"3":[[[1403,141,107,128],9],[[1640,199,110,139],10]]}.

[0121] For example, if the detection boxes are particularly inaccurate, their positions are readjusted; if the detection boxes are missing or have obviously incorrect trajectory markers, they are relabeled. For those without any trajectory, the detection boxes are relabeled, and new trajectory markers are generated. For example, based on the above examples, the content of the revised gt.json could be: {"1":[[[1445,121,105,134],9],[[1698,202,112,134],10]],"2":[ [[1425,131,106,130],9] ,[[1673,203,109,137],10]],"3":[[[1403,141,107,128],9],[[1640,199,110,139],10]]}.

[0122] The fourth step is to write a script (i.e., the program module or pre-defined detection box supplementation model mentioned above): Read gt.json and check the continuity of each trajectory marker. For example, if a pedestrian with trajectory marker 9 is found to exist in images 1 and 3, but not in image 2, then supplement the detection box for the pedestrian with trajectory marker 9 in image 2. It should be noted that with a sufficiently large test set, even if detection boxes are missing or cannot be supplemented in individual images, their impact on the final metric results is negligible.

[0123] The method for obtaining the test set provided in this application can greatly reduce the amount of annotation work in the process of obtaining the test set. Taking a 10-minute video as an example, the time for an annotator to perform full annotation can be shortened from 60 hours to 8 hours, and the cost is reduced by about 7.5 times.

[0124] It should be noted that the method for obtaining a test set provided in this application embodiment can be implemented as a standalone technical solution. By inputting samples into an existing model and correcting the model's output, a more accurate test set can be obtained. Compared to the current method of generating test sets through manual annotation, this significantly reduces the cost of manual annotation and improves the efficiency of test set generation. Moreover, applying this test set to the method for evaluating the target detection model in the trajectory tracking system provided in this application embodiment improves the efficiency of evaluating the accuracy of the target detection model in the trajectory tracking system due to the rich sample size and rapid acquisition of the test set.

[0125] Specifically, S201 may include: S2011, inputting multiple images from the test set into the target detection model of the trajectory tracking system to obtain a first intermediate result; S2012, based on a first threshold, removing invalid detection boxes from the first intermediate result to obtain the test result. S2012 may include, for example: S201a, based on the first threshold, removing invalid detection boxes from the first intermediate result to obtain a second intermediate result; S201b, adding a target trajectory identifier to each detection box in the second intermediate result to obtain the test result. The first threshold may, for example, be a threshold set for valid detection boxes in the target detection model of a normally used trajectory tracking system.

[0126] For S2011, for example, each image in the test set is input into OpenCV's DNN module to call the model, obtaining the target detection results of the trajectory tracking system being evaluated on the images, which is recorded as the first intermediate result. Furthermore, the first intermediate result can be saved to a txt file, with a format such as: image name + confidence score + category + coordinates, e.g., img / ch08003_20200108153031_15261.jpg 0.919154 1 1449 124 103 134 0.889048 1 1693 200 112 130 0.8733 1 0 806 126 160 0.006487 1 1290 170 73 84. The following explains the meaning of each number in order of appearance: `img / ch08003_20200108153031_15261.jpg` is the image name; `0.919154` is the confidence score of the first detection box, i.e., the probability that the first detection box belongs to this category, where 1 represents the category of the first detection box; `1449 124 103 134` represents the position coordinates of the first detection box in the image; `0.889048` is the confidence score of the second detection box, i.e., the probability that the second detection box belongs to this category, where 1 represents the category of the second detection box; `1693 200 112 130` represents the position coordinates of the second detection box in the image; `0.8733` is the confidence score of the third detection box, i.e., the probability that the third detection box belongs to this category, where 1 represents the category of the third detection box; `0 806 126` represents the position coordinates of the third detection box. 160 represents the position coordinates of the third detection box in the image; 0.006487 is the confidence score of the fourth detection box, that is, the probability that the fourth detection box belongs to this category, 1 represents the category of the fourth detection box, and 1290 170 7384 represents the position coordinates of the fourth detection box in the image.

[0127] For S2012, the invalid detection boxes detected in S1011 can be filtered out using the first threshold. Corresponding to S201a, the second intermediate result can be recorded as the test result. For example, the test result can be recorded as dt.json, and dt.json can be {"1":[[1449,124,103,134],,[1693,200,112,130],,[0,806,126,160]]}; Alternatively, to prepare for subsequent processes, S201b can be executed, marking each detection box obtained in S201a with a target trajectory identifier. For example, if the target trajectory identifier is -1, the test result dt.json can be {"1":[[[1449,124,103,134],-1],[[1693,200,112,130],-1],[[0,806,126,160],-1]]}. It should be noted that for ease of subsequent calculations, the format of the test results and the labeled results in the test set can be kept the same. Therefore, the target trajectory identifier added in S201b does not indicate the actual trajectory.

[0128] S202, Based on the test results and the annotation results, obtain the value of the trajectory evaluation index of the target detection model. The trajectory evaluation index is used to reflect the working performance of the target detection model in the trajectory tracking system.

[0129] In a specific implementation, S202 may include: S2021, obtaining an evaluation result based on the test result and the annotation result, the evaluation result including: the correspondence between each trajectory identifier, the number of the trajectory appearing in the test result, and the number of the trajectory appearing in the annotation result, the evaluation result being used to characterize the occurrence of each trajectory in the test result and the annotation result; S2022, calculating the value of the trajectory evaluation index of the target detection model based on the evaluation result.

[0130] For S2021, for example, it can include: For each image in the test set, calculate the IoU between each bounding box in the image's dt.json and all bounding boxes in the image's gt.json. Assuming the image has n bounding boxes in dt.json and m bounding boxes in gt.json, then n*m IoUs can be obtained. These n*m ​​IoUs can be arranged into an n x m array, where the row number represents the bounding box number in dt.json and the column number represents the bounding box number in gt.json. Then, iterate through the n*m ​​array, select IoUs that meet preset conditions from each row, and assign the trajectory identifier of the bounding box corresponding to the column number of the IoU that meets the preset conditions to the bounding box corresponding to the row number of that IoU. If the IoUs in a row do not meet the preset conditions, then assign the target trajectory identifier to the bounding box in that row. In this way, all bounding boxes in dt.json have trajectory identifiers.

[0131] The preset conditions may include: the maximum value of IoU is greater than a second threshold (e.g., 0.5). The above "selecting IoUs that meet the preset conditions from each row" may refer to determining whether the maximum value of IoU in each row of an n*m array is greater than the second threshold.

[0132] As an example, taking a test set including a first image, a test result including a first test bounding box of the first image, and a annotation result including a first annotation bounding box of the first image as an example, for the first image, S2021 may include: calculating the first IoU between the first test bounding box and the first annotation bounding box; determining that the first IoU meets a preset condition, then setting the trajectory identifier of the first test bounding box as the first trajectory identifier of the first annotation bounding box. Wherein, the first IoU meeting the preset condition may include: the value in the first IoU is greater than a second threshold.

[0133] As another example, taking a test set including a first image, a test result including a first test box and a second test box of the first image, and an annotation result including a first annotation box and a second annotation box of the first image, for the first image, S2021 may include: S20211, calculating the first IoU between the first test box and the first annotation box, the second IoU between the first test box and the second annotation box, the third IoU between the second test box and the first annotation box, and the fourth IoU between the second test box and the second annotation box; S20212, selecting the first IoU that meets a preset condition from the first IoU and the second IoU, and setting the trajectory identifier of the first test box as the first trajectory identifier of the first annotation box; and selecting the fourth IoU that meets a preset condition from the third IoU and the fourth IoU, and setting the trajectory identifier of the second test box as the second trajectory identifier of the second annotation box, wherein the evaluation result includes the first trajectory identifier of the first test box and the second trajectory identifier of the second test box. The preset condition may include: the IoU value is greater than a second threshold.

[0134] As another example, if the first image also includes a third test box, then, for the first image, S2021, in addition to S20211 and S20212 described above, may also include: S20213, calculating the fifth IoU between the third test box and the first labeled box, and the sixth IoU between the third test box and the second labeled box; S20214, determining that the IoU in both the fifth and sixth IoU do not satisfy the preset condition, then setting the trajectory identifier of the third test box to a target trajectory identifier (e.g., -1), and the evaluation result also includes the target trajectory identifier of the third test box. This target trajectory identifier can have the same value as the aforementioned target trajectory identifier, or it can have a different value.

[0135] Taking the dt.json obtained after S201 as {"1":[[[1449,124,103,134],-1],[[1693,200,112,130],-1],[[0,806,126,160],-1]]} as an example, the pt.json obtained after S202 could be: {"1":[[[1449,124,103,134],9],[[1693,200,112,130],10],[[0,806,126,9],9],[[1693,200,112,130],10],[[0,806,126,9],9],[[1693,200,112,130],10],[[0,806,126,9],9],[[1449,124,103,134 ...9],[[0,806,126,9],9],[[1449,124,103,134],9],[[1693,200,112,130],9],[[1693,200 The IoU between the detection box with coordinates [1449,124,103,134] and the detection box with trajectory identifier 9 in the image annotation results satisfies the preset condition; the IoU between the detection box with coordinates [1693,200,112,130] and the detection box with trajectory identifier 10 in the image annotation results satisfies the preset condition; the IoU between the detection box with coordinates [0,806,126,160] and any detection box in the image annotation results does not satisfy the preset condition.

[0136] For S2022, as an example, S2022 may include: determining relevant information for each trajectory based on the evaluation results, wherein the relevant information for each trajectory includes the trajectory identifier, the image number of the trajectory appearing in the test results, and the image number of the trajectory appearing in the annotation results; and calculating the value of the trajectory evaluation index based on the relevant information for each trajectory.

[0137] The trajectory evaluation metrics include trajectory recall rate and / or trajectory disconnection rate. The trajectory recall rate is the ratio of the number of recalled trajectories to the total number of trajectories. The trajectory disconnection rate is the ratio of the total number of disconnections of all trajectories to the number of recalled trajectories. If a target is detected in N consecutive frames, the target is recorded as a recalled trajectory, where N is an integer greater than or equal to 1.

[0138] Before introducing trajectory recall, this application first defines trajectory recall in its embodiments. Trajectory recall refers to the determination of a recalled trajectory if, within a trajectory, there are N consecutive frames where the target is detected correctly. The trajectory recall count is then incremented by one. Depending on design requirements, N can be set to any value greater than or equal to 1; for example, N can be 1, 5, or 10. Trajectory recall rate = number of recalled trajectories / total number of trajectories. The trajectory recall rate is greater than or equal to 0 and less than or equal to 1. A higher trajectory recall rate indicates a more stable target detection model in the trajectory tracking system being evaluated, and better completeness of the output trajectory. For example, "detecting the target correctly" can mean that the IoU between the bounding box of the target in the image annotation results and the test box of the target in the test results is less than a third threshold (e.g., 0.6).

[0139] For trajectory breakage rate, the trajectory breakage rate = the number of breaks in all trajectories in the video / the total number of recalled trajectories in the video. For example, in a video with trajectories of 10 people, 9 people are recalled, and the trajectories of 3 people are discontinuous. Among the discontinuous trajectories, one person's trajectory is broken into 3 segments, and the other two people's trajectory is broken into 2 segments. That is, the 9 trajectory segments are broken into (6+3+2+2) = 13 segments. Therefore, the trajectory breakage rate = 13 / 9. The higher the trajectory breakage rate, the worse the stability of the target detection model of the trajectory tracking system being evaluated, and the worse the completeness of the output trajectory. If a target is detected in M ​​consecutive frames, it is determined that the trajectory is not broken. M is greater than or equal to 0 and less than or equal to 1. The value of M can be set according to actual needs, for example, M can be set to 1, 2, or 4. For example, suppose the test set contains only one trajectory with 10 frames. The frames in the video containing the detected target are numbered 1, 2, 3, 5, 7, and 10. If trajectory recall occurs, and M is set to 1, the trajectory is broken into 4 segments: 1, 2, and 3 form one segment, and 5, 7, and 10 form another segment, resulting in a trajectory breakage rate of (4 / 1) = 4. If M is set to 2, the trajectory is broken into 2 segments: 1, 2, 3, 5, and 7 form one segment, and 10 forms another segment, resulting in a trajectory breakage rate of (2 / 1) = 2. If M is set to 4, the trajectory is broken into 1 segment: 1, 2, 3, 5, 7, and 10 form one segment, resulting in a trajectory breakage rate of (1 / 1) = 1.

[0140] For example, for S2022, the evaluation results obtained in S2021 can be organized first. For instance, using trajectory identifiers as units, the annotation results gt.json and the evaluation results pt.json are compared to generate a value with the trajectory identifier as the key and the key in gt.json and pt.json as the value. The key in gt.json and pt.json can refer to the image's number in the video frame. If the trajectory identifier does not appear in pt.json, then the value should be filled with 0.

[0141] Assuming the pt.json obtained after S2021 is {"1":[[[1449,124,103,134],9],[[1693,200,112,130],10],[[0,806,126,160],-1]]}, and the gt.json obtained after S201 is {"1":[[[1445,121,105,134],9],[[1698,202,112,134],10]],"2":[[[1445,121,105,134],9],[[0,20, 100,100],11],[[1698,203,109,137],10]],"3":[[[1403,141,107,128],9],[[1640,199,110,139],10]]},Then,the relevant information of each trajectory obtained after sorting can be:{9:{"1":"1","2":0,"3":0},10:{"1":“1”,"2":0,"3":0},11:{"2":0},-1:{"1":[0,806,126,160]}}. Wherein, key is the trajectory identifier, and value is the number (or frame number) of the trajectory indicated by the trajectory identifier in the annotation results: the number (or frame number) of the trajectory indicated by the trajectory identifier in the evaluation results, and the trajectory with key is the target trajectory identifier (such as -1), which appears in the evaluation results pt.json but not in the test results gt.json, is most likely a false positive. Therefore, it directly indicates the frame number and position coordinates of the trajectory in the evaluation results pt.json.

[0142] In this example, there are three trajectories with identifiers 9, 10, and 11. Assuming N = 1, trajectories with identifiers 9 and 10 are recalled, so the recall rate is 2 / 3. Assuming M = 1, the trajectory corresponding to identifier 9 was interrupted twice, the trajectory corresponding to identifier 10 was interrupted twice, and the trajectory corresponding to identifier 11 was not recalled. Therefore, the number of recalled trajectories is 1, and the trajectory interruption rate is (2+2) / 2 = 2.

[0143] It should be noted that, in this application embodiment, the target detection model in the trajectory tracking system can also be determined based on the trajectory recall rate and / or trajectory disconnection rate to determine whether the target detection model in the trajectory tracking system meets the usage requirements of the trajectory tracking system. If it does not meet the requirements, the target detection model will be further trained and optimized so that the trajectory tracking system including the target detection model can provide users with better services.

[0144] As can be seen, through the method provided in this application embodiment, a new trajectory evaluation index (such as trajectory recall rate and / or trajectory disconnection rate) for the target detection model is defined. This trajectory evaluation index is used to evaluate the quality of the target detection model in the trajectory tracking system. That is, the trajectory evaluation index not only reflects the state of the target detection model, but also reflects the state of trajectory tracking including the target detection model. Thus, with the help of this newly defined trajectory evaluation index, the working condition of the target detection model in the trajectory tracking system can be reasonably evaluated, thereby making it possible to provide users with a high-quality trajectory tracking system.

[0145] Accordingly, embodiments of this application also provide an apparatus 300 for evaluating a target detection model in a trajectory tracking system, see below. Figure 3 As shown. The device 300 includes:

[0146] The first obtaining unit 301 is used to obtain test results based on multiple images in the test set and the target detection model of the trajectory tracking system. The test set includes the annotation results of the multiple images. Both the test results and the annotation results include the image number in the video frame and the detection box position information. The annotation results also include trajectory identifiers.

[0147] The second obtaining unit 302 is used to obtain the value of the trajectory evaluation index of the target detection model based on the test results and the annotation results. The trajectory evaluation index is used to reflect the working performance of the target detection model in the trajectory tracking system.

[0148] Optionally, the trajectory evaluation metrics include trajectory recall rate and / or trajectory disconnection rate. The trajectory recall rate is the ratio of the number of recalled trajectories to the total number of trajectories. The trajectory disconnection rate is the ratio of the total number of disconnections of all trajectories to the number of recalled trajectories. If a target is detected in N consecutive frames, the target is recorded as a recalled trajectory, where N is an integer greater than or equal to 1.

[0149] Optionally, the first obtaining unit 301 includes:

[0150] The first obtaining subunit is used to input multiple images from the test set into the target detection model of the trajectory tracking system to obtain a first intermediate result;

[0151] A filtering subunit is used to remove invalid detection boxes from the first intermediate result based on a first threshold, thereby obtaining the test result.

[0152] Optionally, the filtering subunit is specifically used for:

[0153] Based on the first threshold, invalid detection boxes in the first intermediate result are removed to obtain the second intermediate result;

[0154] Add a target trajectory identifier to each detection box in the second intermediate result to obtain the test result.

[0155] Optionally, the second obtaining unit 302 includes:

[0156] The second obtaining subunit is used to obtain an evaluation result based on the test result and the annotation result. The evaluation result includes the correspondence between each trajectory identifier, the number of the trajectory appearing in the test result, and the number of the trajectory appearing in the annotation result. The evaluation result is used to characterize the occurrence of each trajectory in the test result and the annotation result.

[0157] The calculation subunit is used to calculate the value of the trajectory evaluation index of the target detection model based on the evaluation results.

[0158] Optionally, the test set includes a first image, the test result includes a first test box of the first image, and the annotation result includes a first annotation box of the first image. For the first image, the second obtaining sub-unit is specifically used for:

[0159] Calculate the first IoU between the first test box and the first annotation box;

[0160] If the first IoU selection meets the preset conditions, the trajectory identifier of the first test box is set as the first trajectory identifier of the first annotation box. The preset conditions include: the maximum value in the IoU is greater than the second threshold.

[0161] Optionally, the test set includes a first image, the test results include a first test box and a second test box of the first image, and the annotation results include a first annotation box and a second annotation box of the first image. For the first image, the second obtaining sub-unit is specifically used for:

[0162] Calculate the first overlap IoU between the first test box and the first annotation box, the second IoU between the first test box and the second annotation box, the third IoU between the second test box and the first annotation box, and the fourth IoU between the second test box and the second annotation box;

[0163] The first IoU that satisfies a preset condition is selected from the first IoU and the second IoU, and the trajectory identifier of the first test box is set as the first trajectory identifier of the first annotation box; and the fourth IoU that satisfies the preset condition is selected from the third IoU and the fourth IoU, and the trajectory identifier of the second test box is set as the second trajectory identifier of the second annotation box. The evaluation result includes the first trajectory identifier of the first test box and the second trajectory identifier of the second test box. Optionally, the preset condition includes: the maximum value of the IoU is greater than a second threshold.

[0164] Optionally, the first image further includes a third test box, and the second obtaining sub-unit is further configured to:

[0165] Calculate the fifth IoU between the third test box and the first annotation box, and the sixth IoU between the third test box and the second annotation box;

[0166] If it is determined that neither the fifth IoU nor the sixth IoU satisfies the preset condition, then the trajectory identifier of the third test box is set as the target trajectory identifier, and the evaluation result also includes the target trajectory identifier of the third test box.

[0167] Optionally, the device 300 further includes a third obtaining unit for obtaining the test set, wherein the third obtaining unit includes:

[0168] The third obtaining subunit is used to input the multiple images into the target detection module of the trajectory tracking system to obtain the information of the initial annotation box of each image in the multiple images. The information of the initial annotation box includes the image number in the video frame, the detection box position information and the trajectory identifier.

[0169] The fourth obtaining subunit is used to supplement the annotation boxes of the images with missing annotation boxes in the multiple images according to the preset detection box supplementation model and the information of the initial annotation box of each image in the multiple images, and to obtain the information of the intermediate annotation box of each image in the multiple images. The information of the intermediate annotation box includes the information of the initial annotation box and the information of the supplemented box.

[0170] The fifth subunit is used to obtain a test set based on the information of the multiple images and their corresponding intermediate annotation boxes.

[0171] Optionally, the annotation result includes information about the middle annotation box of each of the plurality of images;

[0172] or,

[0173] The fifth obtaining subunit is specifically used for:

[0174] Correct the information of the intermediate annotation boxes to obtain the information of the final annotation boxes;

[0175] The test set is obtained based on the information of the final annotation boxes. The annotation results in the test set include the information of the final annotation boxes of each of the multiple images.

[0176] It should be noted that the portion of the device 300 that obtains the test set can be embodied as a separate device; that is, the embodiments of this application also provide a device for obtaining the test set, which may include:

[0177] The first obtaining unit is used to input multiple images into the trajectory tracking system and obtain the information of the initial annotation box of each image in the multiple images. The information of the initial annotation box includes the image number in the video frame, the detection box position information and the trajectory identifier.

[0178] The second obtaining unit is used to supplement the annotation boxes of the images with missing annotation boxes in the multiple images according to the preset detection box supplementation model and the information of the initial annotation box of each image in the multiple images, and obtain the information of the intermediate annotation box of each image in the multiple images. The information of the intermediate annotation box includes the information of the initial annotation box and the information of the supplemented box.

[0179] The third obtaining unit is used to obtain a test set based on the information of the multiple images and their corresponding intermediate annotation boxes.

[0180] Optionally, the test set is used in the method for evaluating target detection models in a trajectory tracking system provided in the embodiments of this application.

[0181] in,

[0182] The annotation results include information about the middle annotation box of each of the multiple images;

[0183] or,

[0184] The third obtaining unit includes:

[0185] The correction subunit is used to correct the information of the intermediate annotation box to obtain the information of the final annotation box.

[0186] A subunit is obtained for obtaining the test set based on the information of the final annotation box, wherein the annotation result in the test set includes the information of the final annotation box of each of the multiple images.

[0187] Furthermore, embodiments of this application also provide an electronic device 400, such as... Figure 4 As shown, the electronic device 400 includes a processor 401 and a memory 402:

[0188] The memory 402 is used to store computer programs;

[0189] The processor 401 is used to execute the method provided in the embodiments of this application according to the computer program.

[0190] Furthermore, embodiments of this application also provide a computer-readable storage medium for storing a computer program for executing the method provided in embodiments of this application.

[0191] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that all or part of the steps in the methods of the above embodiments can be implemented by means of software plus a general-purpose hardware platform. Based on this understanding, the technical solution of this application can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as a read-only memory (ROM) / RAM, magnetic disk, optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, a server, or a network communication device such as a router) to execute the methods described in various embodiments or some parts of the embodiments of this application.

[0192] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system and device embodiments are basically similar to the method embodiments, so the descriptions are relatively simple; relevant parts can be referred to the descriptions in the method embodiments. The device and system embodiments described above are merely illustrative. Modules described as separate components may or may not be physically separate, and components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0193] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application. It should be noted that those skilled in the art can make various improvements and modifications without departing from this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A method for evaluating a target detection model in a trajectory tracking system, characterized in that, include: Based on multiple images in the test set and the target detection model of the trajectory tracking system, test results are obtained. The test set includes the annotation results of the multiple images. Both the test results and the annotation results include the image number in the video frame and the detection box position information. The annotation results also include trajectory identifiers. Based on the test results and the annotation results, the value of the trajectory evaluation index of the target detection model is obtained. The trajectory evaluation index is used to reflect the working performance of the target detection model in the trajectory tracking system. The trajectory evaluation metrics include trajectory recall rate and / or trajectory disconnection rate. The trajectory recall rate is the ratio of the number of recalled trajectories to the total number of trajectories. The trajectory disconnection rate is the ratio of the total number of disconnections of all trajectories to the number of recalled trajectories. If a target is detected in N consecutive frames, the target is recorded as a recalled trajectory, where N is an integer greater than or equal to 1.

2. The method according to claim 1, wherein calculating the value of the trajectory evaluation index of the target detection model based on the test results and the annotation results includes: An evaluation result is obtained based on the test results and the annotation results. The evaluation result includes the correspondence between each trajectory identifier, the number of the trajectory appearing in the test results, and the number of the trajectory appearing in the annotation results. The evaluation result is used to characterize the occurrence of each trajectory in the test results and the annotation results. The trajectory evaluation index of the target detection model is calculated based on the evaluation results.

3. The method according to claim 2, characterized in that, The test set includes a first image, the test results include a first test box of the first image, and the annotation results include a first annotation box of the first image. For the first image, obtaining an evaluation result based on the test results and the annotation results includes: Calculate the first overlap (IoU) between the first test box and the first annotation box; If the first IoU selection meets the preset conditions, the trajectory identifier of the first test box is set as the first trajectory identifier of the first annotation box. The preset conditions include: the maximum value in the IoU is greater than the second threshold.

4. The method according to claim 2, characterized in that, The test set includes a first image, the test results include a first test box and a second test box of the first image, and the annotation results include a first annotation box and a second annotation box of the first image. For the first image, obtaining an evaluation result based on the test results and the annotation results includes: Calculate the first overlap IoU between the first test box and the first annotation box, the second IoU between the first test box and the second annotation box, the third IoU between the second test box and the first annotation box, and the fourth IoU between the second test box and the second annotation box; Select the first IoU that satisfies the preset conditions from the first IoU and the second IoU, and set the trajectory identifier of the first test box as the first trajectory identifier of the first annotation box; and select the fourth IoU that satisfies the preset conditions from the third IoU and the fourth IoU, and set the trajectory identifier of the second test box as the second trajectory identifier of the second annotation box. The evaluation result includes the first trajectory identifier of the first test box and the second trajectory identifier of the second test box. The preset conditions include: the maximum value among the IoUs and the maximum value is greater than a second threshold.

5. The method according to claim 4, characterized in that, The first image also includes a third test box. For the first image, obtaining the evaluation result based on the test results and the annotation results further includes: Calculate the fifth IoU between the third test box and the first annotation box, and the sixth IoU between the third test box and the second annotation box; If it is determined that neither the fifth IoU nor the sixth IoU satisfies the preset condition, then the trajectory identifier of the third test box is set as the target trajectory identifier, and the evaluation result also includes the target trajectory identifier of the third test box.

6. The method according to claim 1, characterized in that, The target detection model based on multiple images and trajectory tracking systems in the test set obtains test results, including: Multiple images from the test set are input into the target detection model of the trajectory tracking system to obtain a first intermediate result; Based on the first threshold, invalid detection boxes in the first intermediate result are removed to obtain the test result.

7. A method for obtaining a test set, characterized in that, The method includes: Multiple images are input into the trajectory tracking system to obtain the initial annotation box information for each image. The initial annotation box information includes the image number in the video frame, the detection box position information, and the trajectory identifier. Based on the preset detection box supplementation model and the information of the initial annotation box of each image in the multiple images, the annotation boxes of images with missing annotation boxes are supplemented to obtain the information of the intermediate annotation box of each image in the multiple images. The information of the intermediate annotation box includes the information of the initial annotation box and the information of the supplemented box. A test set is obtained based on the multiple images and the information of their corresponding intermediate annotation boxes; Check whether any image in the initial bounding box information of the multiple images has a missing detection box. If there is a missing detection box, the preset detection box supplementation model automatically supplements the detection boxes for the images with missing detection boxes.

8. The method according to claim 7, characterized in that, The step of obtaining the test set based on the information in the intermediate annotation box includes: Correct the information of the intermediate annotation boxes to obtain the information of the final annotation boxes; The test set is obtained based on the information of the final annotation boxes, and the annotation results in the test set include the information of the final annotation boxes of each of the multiple images.

9. An apparatus for evaluating a target detection model in a trajectory tracking system, characterized in that, include: The first obtaining unit is used to obtain test results based on multiple images in the test set and the target detection model of the trajectory tracking system. The test set includes the annotation results of the multiple images. Both the test results and the annotation results include the image number in the video frame and the detection box position information. The annotation results also include trajectory identifiers. The second obtaining unit is used to obtain the value of the trajectory evaluation index of the target detection model based on the test results and the annotation results. The trajectory evaluation index is used to reflect the working performance of the target detection model in the trajectory tracking system. The trajectory evaluation metrics include trajectory recall rate and / or trajectory disconnection rate. The trajectory recall rate is the ratio of the number of recalled trajectories to the total number of trajectories. The trajectory disconnection rate is the ratio of the total number of disconnections of all trajectories to the number of recalled trajectories. If a target is detected in N consecutive frames, the target is recorded as a recalled trajectory, where N is an integer greater than or equal to 1.

10. An apparatus for acquiring a test set, characterized in that, include: The first obtaining unit is used to input multiple images into the trajectory tracking system and obtain the information of the initial annotation box of each image in the multiple images. The information of the initial annotation box includes the image number in the video frame, the detection box position information and the trajectory identifier. The second obtaining unit is used to supplement the annotation boxes of the images with missing annotation boxes in the multiple images according to the preset detection box supplementation model and the information of the initial annotation box of each image in the multiple images, and obtain the information of the intermediate annotation box of each image in the multiple images. The information of the intermediate annotation box includes the information of the initial annotation box and the information of the supplemented box. The third obtaining unit is used to obtain a test set based on the information of the multiple images and their corresponding intermediate annotation boxes; Check whether any image in the initial bounding box information of the multiple images has a missing detection box. If there is a missing detection box, the preset detection box supplementation model automatically supplements the detection boxes for the images with missing detection boxes.

11. An electronic device, characterized in that, The electronic device includes a processor and a memory: The memory is used to store computer programs; The processor is configured to perform the method according to any one of claims 1-8 according to the computer program.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store a computer program for performing the method according to any one of claims 1-8.

Images