A method and system for producing a single-camera multi-target tracking data set

Abstract

The present invention discloses a single-camera multi-target tracking data set production method and system, which manages the collected original video or image sequence to be annotated as total material data and divides it into first material data and second material data; pre-annotates the first material data using a traditional multi-target tracking algorithm, and performs frame extraction preprocessing on the first material data and the corresponding pre-annotation to obtain pre-processed material data and pre-annotated tracking results respectively; calls a staged correction and calibration pre-annotation result process, performs correction and calibration processing on the pre-annotated tracking results, and obtains a corrected annotation file; uses the corrected annotation file and the corresponding pre-processed material data to train and optimize the multi-target tracking algorithm deep neural network model; calls the trained multi-target tracking algorithm deep neural network model for forward reasoning to obtain pre-annotation, and uses the corrected pre-annotation result process to process the second material data. The present invention has high accuracy, simple and efficient operation.

Description

Technical Field

[0001] The present invention relates to the field of computer vision and artificial intelligence deep learning technology, and in particular discloses a method and system for producing a single-camera multi-target tracking data set. Background Art

[0002] In order to successfully apply the theoretical research results of multi-target tracking algorithms based on deep neural networks to actual industrial production, it is necessary to provide a large amount of material data and its precise annotation data for the specific application scenarios of the algorithm. The material data used for multi-target tracking includes videos and image sequences extracted from videos containing targets of interest. The scale and richness of material data are important guarantees for the applicability of the model to a wide range of scenarios, while the accuracy of the annotation data of the material data is a necessary condition for reliable training and accurate reasoning of the model. Setting up a certain number of surveillance cameras at a specific location can obtain the video material data required for multi-target tracking model training; however, it is not easy to obtain the corresponding precise annotation data to construct a multi-target tracking dataset.

[0003] There are usually three data labeling methods for building the data set required for deep learning: machine algorithm labeling, pure manual labeling, and machine pre-labeling plus correction and calibration. Machine algorithm labeling refers to the use of existing specific algorithms to process the material data to directly obtain the labeling results; pure manual labeling refers to the method of completely manually labeling all material data one by one according to specific labeling requirements; machine pre-labeling plus correction and calibration is based on machine algorithm labeling, adding human subjective judgment, and manually adding, deleting, and modifying the obtained labeling results to obtain accurate labeling results. However, for data labeling for single-camera multi-target tracking, the above three labeling methods are all insufficient. Because the existing multi-target tracking algorithm cannot ensure the tracking consistency of the tracking target, and will be accompanied by the tracking target ID (identity number) jump error, especially for multi-category multi-target tracking involving mixed traffic of people and vehicles at urban street scene intersections, the algorithm labeling error is very serious, resulting in the unusable labeled data. Among them, labeling errors can be divided into two categories: the first type of labeling error is that the tracklets of multiple target objects are mistakenly labeled as the complete tracklets of one target object; the second type of labeling error is that the tracklet of one target object is mistakenly labeled as the tracklets of two or more target objects. If the machine labeling method based on the existing multi-target tracking algorithm is adopted, although it is efficient and low-cost, it cannot avoid the frequent occurrence of the above two types of labeling errors; if the completely manual labeling method is adopted, it is time-consuming, labor-intensive, inefficient, and too costly; if the relevant data set production methods of other deep learning tasks are borrowed, and the algorithm pre-labeling plus correction and calibration method is simply adopted, then in the process of correcting and calibrating the above two types of labeling errors pre-labeled by the multi-target tracking algorithm, it is inevitable that it will be difficult to quickly locate and correct the relevant labeling errors, especially the correction and calibration operation workload is large and cumbersome, and the implementation difficulty is even no less than the complete manual labeling method. Therefore, the labeling efficiency of this method is obviously greatly reduced compared to the data labeling efficiency of other deep learning tasks.

[0004] In patent CN202110647385.0, the tracking information of the target object in the video image is annotated by manually annotating the first M frames of images, superpixel segmentation of infrared images, clustering, calculating positive and negative sample sets, calculating the internal position structure and class structure constraints of the sample set, and fusing confidence maps. The patent involves a large amount of calculation and high calculation cost, and it is difficult to use GPU hardware computing resources to accelerate the relevant calculations, and the accuracy of the selection of the tracking target object is far inferior to the target detection algorithm based on deep neural network. In addition, the patent is only effective for the data annotation of single target object tracking in infrared image sequences, and it is difficult to apply it to the data annotation of multi-target object tracking, and it is even more unsuitable for the annotation of multi-category multi-target object tracking at urban street intersections where people and vehicles are mixed, because in addition to being unable to accurately and comprehensively select the tracking target object of interest, the solution cannot avoid two types of annotation errors: two or more tracking target trajectory segments are mistakenly connected in series as one trajectory, and the same tracking target trajectory is mistakenly split into two or more target trajectory segments.

[0005] In patent CN202110674075.8, a human-machine combined data annotation method and platform is provided. The patent first uses a part of the annotated image data to train the machine learning model. When the model reaches a certain accuracy, it uses it to perform inference prediction on the remaining unlabeled image set to achieve pre-annotation, and displays the pre-annotation results on the annotation interface, and finally uses the pre-annotation results verified and confirmed by the annotator as the annotation results of the image data. Although the patent only requires the annotator to accept or correct the pre-annotation results of the machine algorithm, which reduces the manual workload to a certain extent, it can only achieve the purpose of improving the annotation efficiency when the annotation task is target detection or image classification, and it cannot be applied to the verification and correction of multi-target tracking data. Because it is difficult to quickly locate and correct the two types of multi-target tracking data pre-annotated by the algorithm, the annotation errors and the correction and calibration operations are large and cumbersome, so the patent cannot efficiently produce a multi-target tracking data set that meets the accuracy requirements.

[0006] In patent CN202011438924.1, a method and system for constructing a complex scene target detection dataset is disclosed. Although the dataset constructed by the invention is used for target detection tasks, its construction method can be analogized to produce a target tracking dataset. For example, in a specific scene image sequence containing a specified category of tracking target objects, the trained generator of the generative adversarial network is used to construct more tracking target objects, and some simulation parameters such as motion trajectory, occlusion, and scaling changes are added to construct the tracking data of the newly generated target object; for another example, for a single scene image sequence that has completed target tracking annotation, a generative adversarial network is used to generate a richer scene image sequence or video. Although these analogy schemes can enrich data samples to a certain extent, due to the essential difference between the generated data samples and the actual application scenarios, and the image sequences or videos of the actual application scenarios for multi-target tracking tasks are easier to collect, what is lacking is the production of the corresponding annotation data involved, so the dataset construction scheme based on this analogy cannot fundamentally promote the implementation of multi-target tracking algorithms in industrial production, and it is even more impossible to complete efficient and accurate target object tracking and annotation tasks for a given video or image sequence, and produce corresponding multi-target tracking datasets.

[0007] Therefore, the existing dataset construction methods are unable to complete the production of multi-target tracking datasets in a given video or image sequence, which is a technical problem that needs to be solved urgently. Summary of the invention

[0008] The present invention provides a single-camera multi-target tracking dataset production method and system, aiming to solve the technical problem that the existing dataset construction method cannot complete the production of a multi-target tracking dataset in a given video or image sequence.

[0009] One aspect of the present invention relates to a method for preparing a single-camera multi-target tracking data set, comprising the following steps:

[0010] Collecting original videos or image sequences to be annotated, and managing the collected original videos or image sequences to be annotated as total material data, and dividing them into first material data and second material data;

[0011] Pre-labeling the first material data by using a conventional multi-target tracking algorithm, and performing frame extraction pre-processing on the first material data and the corresponding pre-labeling to obtain pre-processed material data and pre-labeling tracking results respectively;

[0012] Call the staged correction and calibration pre-marking result process to perform correction and calibration on the pre-marking tracking results to obtain a corrected marking file;

[0013] Use the corrected annotation files and the corresponding pre-processed material data to train and optimize the deep neural network model of the multi-target tracking algorithm;

[0014] Calling the trained multi-target tracking algorithm deep neural network model for forward reasoning to obtain pre-annotation, and using the revised pre-annotation result process to process the second material data to obtain more revised annotation files and corresponding pre-processed material data;

[0015] Collect and summarize the corrected annotation files and the corresponding pre-processed material data to form a single-camera multi-target tracking dataset.

[0016] Furthermore, the steps of pre-marking the first material data by using a conventional multi-target tracking algorithm, and pre-processing the first material data and the corresponding pre-marks by extracting frames to obtain the pre-processed material data and the pre-marked tracking results respectively include:

[0017] Using an existing deep learning multi-target detection algorithm to perform target detection frame and category annotation on the first material data frame by frame and after correction and confirmation, the position, size and target category attribute information of the circumscribed rectangular frame of the target object of interest are obtained as the target object detection result;

[0018] The target object detection results are pre-labeled using a traditional multi-target tracking algorithm. Each line of multi-target tracking labeling data corresponds to each frame in the first material data one by one. Each line of labeling data includes a frame number, a target ID number, the coordinate values ​​of the upper left corner and the lower right corner of the target frame, and a target category label number, to obtain a pre-labeling file for tracking of the target object of interest.

[0019] Extract frames of the first material data and save them as video files to form basic pre-processed material data, which together with corresponding annotation files form pre-annotated data.

[0020] Furthermore, the steps of training and optimizing the deep neural network model of the multi-target tracking algorithm using the corrected annotation file and the corresponding pre-processed material data include:

[0021] Collect the expanded and corrected annotation files and form a temporary tracking dataset together with the corresponding pre-processed material data, and then divide the temporary tracking dataset into a training dataset and a verification dataset according to the set ratio;

[0022] Use the training data set to train and optimize the deep neural network in the multi-target tracking algorithm to obtain the deep neural network model of the multi-target tracking algorithm;

[0023] Perform inference tests on the validation dataset using a multi-target tracking algorithm deep neural network model;

[0024] If the inference accuracy meets expectations, the multi-target tracking algorithm deep neural network model is saved.

[0025] Further, calling the trained multi-target tracking algorithm deep neural network model for forward reasoning to obtain pre-annotation, and using the revised pre-annotation result process to process the second material data to obtain more revised annotation files and corresponding pre-processed material data includes the following steps:

[0026] removing a copy of the second material data as the current material data;

[0027] Use the deep neural network model of the multi-target tracking algorithm with the required inference accuracy to perform forward inference on the material data frame by frame to obtain the pre-labeled tracking file of the target object of interest;

[0028] In the frame-by-frame processing of the multi-target tracking algorithm, the sampling frequency is weighed according to the displacement size of the target object of interest in the previous and next frames, and the current material data is extracted and saved to form this pre-processed material data, and the multi-target tracking pre-labeling results of the corresponding frames are saved in a specific format to form this optimized pre-labeling file;

[0029] Waiting for the process of correcting and calibrating the pre-annotation tracking results in stages, correcting and calibrating the optimized pre-annotated file in stages, and obtaining the submitted corrected and annotated file corresponding to the pre-processed material data;

[0030] Call training and optimize the deep neural network model process of multi-target tracking algorithm;

[0031] It is determined whether the second material data is empty, and if so, the multi-target tracking pre-labeling correction calibration is terminated.

[0032] Further, the step of waiting for the phased correction and calibration pre-annotation tracking result process, correcting and calibrating the optimized pre-annotated file in phases, and obtaining the submitted corrected annotation file corresponding to the pre-processed material data includes:

[0033] Obtain pre-processed material data and corresponding multi-target tracking annotation files to form pre-annotated data;

[0034] Crop out independent target object images by ID and store them in groups with the ID number as the folder name;

[0035] Corrected the first type of labeling errors;

[0036] Generate a tracking target object image result record file after correcting the first type of labeling errors;

[0037] Convert the record file into a multi-target tracking annotation file in which each line of multi-target tracking annotation data corresponds to each frame in the pre-processed material data in sequence, and each line of annotation data includes the frame number, target ID number, coordinate values ​​of the upper left corner and lower right corner of the target frame, and target category label;

[0038] Load the annotations in the annotation file and the corresponding pre-processed material data and display them for correction and adjustment;

[0039] Correct the second type of labeling errors and add or adjust the target box;

[0040] The adjusted and corrected multi-target tracking annotation file is saved as a corrected annotation file.

[0041] Another aspect of the present invention relates to a single-camera multi-target tracking data set production system, comprising:

[0042] A first collection module is used to collect original videos or image sequences to be annotated, and manage the collected original videos or image sequences to be annotated as total material data, and divide them into first material data and second material data;

[0043] A labeling module, used to pre-label the first material data by using a traditional multi-target tracking algorithm, and to perform frame extraction preprocessing on the first material data and the corresponding pre-labeling, to obtain pre-processed material data and pre-labeling tracking results respectively;

[0044] The first calling module is used to call the staged correction and calibration pre-marking result process, perform correction and calibration processing on the pre-marking tracking result, and obtain a corrected marking file;

[0045] A training module, used to train and optimize a deep neural network model of a multi-target tracking algorithm using the corrected annotation files and the corresponding pre-processed material data;

[0046] The second calling module is used to call the trained multi-target tracking algorithm deep neural network model for forward reasoning to obtain pre-labeling, and use the correction pre-labeling result process to process the second material data to obtain more corrected labeling files and corresponding pre-processed material data;

[0047] The second collection module is used to collect and summarize the corrected annotation files and the corresponding pre-processed material data to form a single-camera multi-target tracking data set.

[0048] Furthermore, the annotation module includes:

[0049] A first acquisition unit is used to use an existing deep learning multi-target detection algorithm to perform target detection frame and category annotation on the first material data frame by frame and obtain the position, size and target category attribute information of the circumscribed rectangular frame of the target object of interest as the target object detection result after correction and confirmation;

[0050] The second acquisition unit is used to pre-label the target object detection results by using a traditional multi-target tracking algorithm, each line of multi-target tracking labeling data corresponds to each frame in the first material data one by one, and each line of labeling data includes a frame number, a target ID number, coordinate values ​​of the upper left corner and the lower right corner of the target frame, and a target category label number, so as to obtain a pre-labeling file for tracking the target object of interest;

[0051] The first composition unit is used to extract frames of the first material data and save them as video files to form basic pre-processed material data, which together with the corresponding annotation files form pre-annotated data.

[0052] Furthermore, the training module includes:

[0053] A collecting unit, used for collecting the expanded and corrected annotation files, and forming a temporary tracking data set together with the corresponding pre-processed material data, and then dividing the temporary tracking data set into a training data set and a verification data set according to a set ratio;

[0054] A third acquisition unit is used to train and optimize the deep neural network in the multi-target tracking algorithm using the training data set to obtain a deep neural network model of the multi-target tracking algorithm;

[0055] The testing unit is used to perform inference testing on the validation dataset using the multi-target tracking algorithm deep neural network model;

[0056] The saving unit is used to save the deep neural network model of the multi-target tracking algorithm if the inference accuracy reaches the expectation.

[0057] Furthermore, the second calling module includes:

[0058] a moving-out unit, used for moving out a copy of the second material data as the current material data;

[0059] A fourth acquisition unit is used to perform forward reasoning on the material data frame by frame using a multi-target tracking algorithm deep neural network model with satisfactory reasoning accuracy to obtain a pre-labeled file for tracking a target object of interest;

[0060] The second component unit is used to weigh the sampling frequency according to the displacement size of the target object of interest in the previous and next frames during the frame-by-frame processing of the multi-target tracking algorithm, extract frames of the material data and save them to form the pre-processed material data, and save the multi-target tracking pre-labeling results of the corresponding frames in a specific format to form the optimized pre-labeling file;

[0061] A fifth acquisition unit is used to wait for the staged correction and calibration pre-annotation tracking result process, perform correction and calibration on the optimized pre-annotation file in stages, and obtain the submitted correction and annotation file corresponding to the pre-processed material data;

[0062] A calling unit is used to call the training and optimize the deep neural network model process of the multi-target tracking algorithm;

[0063] The judging unit is used to judge whether the second material data is empty, and if so, end the multi-target tracking pre-labeling correction calibration.

[0064] Furthermore, the second collection module includes:

[0065] The acquisition subunit is used to acquire pre-processed material data and corresponding multi-target tracking annotation files to form pre-annotated data;

[0066] The storage subunit is used to cut out independent target object images according to ID and store them in groups with the ID number as the folder name;

[0067] A first correction subunit, used to correct the first type of labeling errors;

[0068] A generating subunit, used for generating a tracking target object image result record file after correcting the first type of labeling errors;

[0069] A conversion subunit, used to convert the record file into a multi-target tracking annotation file in which each line of multi-target tracking annotation data corresponds to each frame in the pre-processed material data in a one-to-one order, and each line of annotation data includes a frame number, a target ID number, coordinate values ​​of the upper left corner and the lower right corner of the target frame, and a target category label;

[0070] The loading subunit is used to load the annotations in the annotation file and the corresponding pre-processed material data and then display them for correction and adjustment;

[0071] The correction subunit is used to correct the second type of annotation errors and add or adjust the target box;

[0072] The saving subunit is used to save the adjusted and corrected multi-target tracking annotation file as a corrected annotation file.

[0073] The present invention provides a single-camera multi-target tracking data set production method and system, which collects original videos or image sequences to be labeled, manages the collected original videos or image sequences to be labeled as total material data, and divides them into first material data and second material data; adopts a traditional multi-target tracking algorithm to pre-label the first material data, and performs frame extraction pre-processing on the first material data and the corresponding pre-labeling to obtain pre-processed material data and pre-labeling tracking results respectively; calls a staged correction and calibration pre-labeling result process, performs correction and calibration processing on the pre-labeling tracking result, and obtains a corrected annotation file; uses the corrected annotation file and the corresponding pre-processed material data to train and optimize a multi-target tracking algorithm deep neural network model; calls the trained multi-target tracking algorithm deep neural network model for forward reasoning to obtain pre-labeling, and uses the corrected pre-labeling result process to process the second material data to obtain more corrected annotation files and corresponding pre-processed material data; collects and summarizes the corrected annotation files and the corresponding pre-processed material data to form a single-camera multi-target tracking data set.

[0074] The beneficial effects achieved by the present invention are:

[0075] First, since the traditional multi-target tracking algorithm that does not require supervised training is used for initial pre-labeling, only part of the video data is manually corrected and calibrated in stages for the initial pre-labeling, and then the manually corrected and calibrated initial pre-labeling file is used to optimize the deep neural network model of the multi-target tracking algorithm based on deep learning, thereby obtaining a relatively accurate optimized pre-labeling result. After continuously collecting and expanding the manually corrected labeling files and iteratively training and optimizing the deep neural network model of the multi-target tracking algorithm, manually correcting, reviewing and confirming the optimized pre-labeling results one by one, the final multi-target tracking labeling file is obtained. This not only utilizes the efficiency and tirelessness of the machine for pre-labeling and reduces the workload of manual labeling, but also effectively solves the problem of labeling errors in the target tracking algorithm, ensuring the accuracy of the labeling results of the single-camera multi-target tracking data.

[0076] Secondly, a new multi-target tracking annotation organization format is adopted. One annotation file covers the target detection information and target tracking information of the corresponding video segment or image sequence. Compared with using two files to represent the detection and tracking annotation information respectively, the redundancy and fragmentation of the annotation data are reduced, and the storage and management efficiency of the annotation data can be improved. Since the multi-target tracking annotation organization format represents the target objects with different IDs appearing in the same frame with a row of data, and the video frame numbers are arranged in order from small to large, the efficiency can be greatly improved when cropping the target object graph, and there is no need to repeatedly crop the target objects one by one across multiple frames. In addition, the start and end video frame numbers or tracking ID ranges when cropping the target object graph can be flexibly specified.

[0077] Third, since the tracking target object of each frame in the video or image sequence is cut out into independent target object images according to the corresponding ID, and the target object images are stored in groups with the ID number as the folder name, it is convenient to quickly locate the first type of labeling errors in target tracking manually, that is, the trajectory segments (Tracklet) of multiple target objects are mistakenly labeled as the complete trajectory (Trajectory) of a target object. Moreover, when manually correcting the first type of labeling errors, it is only necessary to move some of the target object images in a certain ID folder to another newly created ID folder or the ID folder of the corresponding target object. In addition, the operation of deleting irrelevant targets or background images in a certain ID folder is actually equivalent to "purifying" the complete trajectory of the target object. The operation is simple and efficient, highly feasible, and saves data labeling costs.

[0078] Fourth, the video or image sequence is sampled, saving data storage space; there is no need to centrally crop independent images of the tracking target object in each frame of the video or image sequence on the server, reducing the frequency of random read and write of the server memory; the independent images cropped from the tracking target object in each frame of the video or image sequence are only completed on each client, the server only sends pre-labeled files and corresponding pre-processed materials, and the client only uploads manually corrected labeling files, thereby saving data transmission bandwidth between the server and the client, and the correction and calibration process of the pre-labeled files can be completed offline on each client. BRIEF DESCRIPTION OF THE DRAWINGS

[0079] Figure 1 A schematic diagram of a flow chart of an embodiment of a method for producing a single-camera multi-target tracking data set provided by the present invention;

[0080] Figure 2 for Figure 1 As shown in the figure, a detailed flow chart of the steps of pre-marking the first material data using a traditional multi-target tracking algorithm, and performing frame extraction pre-processing on the first material data and the corresponding pre-marking to obtain the pre-processed material data and the pre-marked tracking result respectively;

[0081] Figure 3 for Figure 1 A detailed flow chart of the steps of training and optimizing a deep neural network model for a multi-target tracking algorithm using a corrected annotation file and corresponding pre-processed material data is shown in;

[0082] Figure 4 for Figure 1 As shown in the figure, calling the trained multi-target tracking algorithm deep neural network model for forward reasoning to obtain pre-annotation, and using the modified pre-annotation result process to process the second material data to obtain more modified annotation files and corresponding pre-processed material data is a detailed flow chart of the first embodiment of the step;

[0083] Figure 5 for Figure 4 A detailed flow chart of the first embodiment of the step of waiting for the staged correction calibration pre-annotation tracking result process shown in , correcting and calibrating the optimized pre-annotated file in stages, and obtaining the submitted correction annotation file corresponding to the pre-processed material data;

[0084] Figure 6 A functional block diagram of an embodiment of a single-camera multi-target tracking data set production system provided by the present invention;

[0085] Figure 7 for Figure 6 A functional module diagram of an embodiment of a marking module shown in FIG.

[0086] Figure 8 for Figure 6 The functional module diagram of the training module 1 embodiment shown in FIG.

[0087] Fig. 9 for Figure 6 The functional module diagram of the second calling module 1 embodiment shown in FIG.

[0088] Fig.10 for Fig. 9 A functional module diagram of an embodiment of a fifth acquisition unit is shown in FIG.

[0089] Description of Figure Numbers:

[0090] 10. First collection module; 20. Labeling module; 30. First calling module; 40. Training module; 50. Second calling module; 60. Second collection module; 21. First acquisition unit; 22. Second acquisition unit; 23. First composition unit; 41. Collection unit; 42. Third acquisition unit; 43. Testing unit; 44. Storage unit; 51. Removal unit; 52. Fourth acquisition unit; 53. Second composition unit; 54. Fifth acquisition unit; 55. Calling unit; 56. Judgment unit; 541. Acquisition subunit; 542. Storage subunit; 543. First correction subunit; 544. Generation subunit; 545. Conversion subunit; 546. Loading subunit; 547. Correction subunit; 548. Storage subunit. DETAILED DESCRIPTION

[0091] In order to better understand the above technical solution, the above technical solution will be described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0092] like Figures 1 to 4 As shown, an embodiment of the present invention proposes a method for preparing a single-camera multi-target tracking data set, comprising the following steps:

[0093] Step S100: collect original videos or image sequences to be labeled, and manage the collected original videos or image sequences to be labeled as total material data, and divide them into first material data and second material data.

[0094] The original video or image sequence to be labeled is collected to form the total material data, and is divided into 100 parts, from which 3 parts are randomly selected to form the first material data, and the remaining 97 parts constitute the second material data.

[0095] Note: In order to reduce the difficulty of labeling, three representative simple scene data with relatively sparse distribution of objects of interest in the picture can be selected from the total material data as the first material data.

[0096] Step S200: pre-label the first material data using a conventional multi-target tracking algorithm, and perform frame extraction pre-processing on the first material data and the corresponding pre-labeling to obtain pre-processed material data and pre-labeling tracking results, respectively.

[0097] Please see Figure 2 , Figure 2 for Figure 1 In the detailed flow chart of step S200 shown in FIG. 1 , in this embodiment, step S200 includes:

[0098] Step S210: Use the existing deep learning multi-target detection algorithm to perform target detection frame and category annotation on the first material data frame by frame, and after correction and confirmation, obtain the position, size and target category attribute information of the circumscribed rectangular frame of the target object of interest as the target object detection result.

[0099] The existing deep learning multi-target detection algorithm is used to perform target detection frame and category annotation on the first material data frame by frame and after manual correction and confirmation, the position, size and target category attribute information of the circumscribed rectangular frame of the target object of interest are obtained as the target detection annotation.

[0100] Step S220: Use a traditional multi-target tracking algorithm to pre-label the target object detection results. Each line of multi-target tracking labeling data corresponds to each frame in the first material data one by one. Each line of labeling data includes the frame number, target ID number, coordinate values ​​of the upper left and lower right corners of the target box, and the target category label number, to obtain a pre-labeling file for tracking the target object of interest.

[0101] The first material data and the target detection annotations corresponding to each frame are taken as input, and a traditional multi-target tracking algorithm (different from the currently popular multi-target tracking algorithm based on deep learning) is used to track the target detection annotations of the target objects of interest in each frame of the first material data in series with the previous and next frames to obtain the multi-target tracking result.

[0102] Step S230: extract frames of the first material data and save them as video files to form basic pre-processed material data, which together with the corresponding annotation files form pre-annotated data.

[0103] In the process of the multi-target tracking algorithm tracking the target objects frame by frame, the sampling frequency is weighed according to the displacement size of the target object of interest in the previous and next frames (usually the sampling frequency for a video with a frame rate of 25 frames per second is 2 to 5 times), the first material data is sampled and saved as a video file to form the basic pre-processed material data, and the multi-target tracking results of the corresponding frames are saved as an initial pre-labeled file in a specific format.

[0104] Note: (1) The traditional multi-target tracking algorithm processes each frame of the first material data. In addition, the distribution of the target of interest in the picture is relatively sparse and the scene is simple, which can make the initial pre-labeling more accurate and reduce the pre-labeling errors of the algorithm. (2) The first material data is sampled and saved as the basic pre-processed material data, so that on the basis of providing algorithm pre-labeling, the workload of manual correction and calibration verification is further reduced. (3) The first material data is a video or image sequence, but the output basic pre-processed material data is a video, which is to reduce fragmented file management and improve data distribution management efficiency. (4) Traditional multi-target tracking algorithms include but are not limited to SORT (Simple Online and Realtime Tracking) target tracking algorithm. Any traditional multi-target tracking algorithm based on detection results that does not require supervised training information can be used in this step. (5) Target detection and labeling, frame the target object of interest in the basic pre-processed material data in the form of a minimum horizontal circumscribed rectangular frame, including the position, size and target category attribute information of the frame. These target detection frames come from the inference results of existing deep learning multi-target detection algorithms that meet the accuracy requirements and are manually corrected, reviewed and confirmed.

[0105] It should be noted that the specific format and convention are as follows: each line of multi-target tracking annotation data corresponds to each frame in the basic preprocessing material data one by one in sequence, and each line of annotation data includes: frame number, target ID number, coordinate values ​​of the upper left and lower right corners of the target box, and target category label number, which are organized in the following format: frame_id, [obj_id, x1, y1, x2, y2, cls_id], where "[]" indicates that all targets of interest in the picture with frame number frame_id are enumerated in this form; the target ID number obj_id is provided by the multi-target tracking algorithm; the coordinate values ​​of the upper left and lower right corners of the target box x1, y1, x2, y2, and the target category label number cls_id are all from the target object detection results.

[0106] Step S300: calling the staged correction and calibration pre-marking result process, performing correction and calibration processing on the pre-marking tracking result, and obtaining a corrected marking file.

[0107] Step S300 includes:

[0108] Step S310: Use the basic pre-processed material data and the corresponding initial pre-labeled file as input to the target object cropping program, crop independent target object images from the video screen according to the ID of the tracked target object, and store them in groups with the ID number as the folder name.

[0109] The basic preprocessed material data and the corresponding initial pre-labeled files are used as the input of the target object cropping program. According to the ID of the tracked target object, independent target object images are cropped from the video screen and stored in folders with the ID number as the name. The naming rule of the images is: video frame number_category label_coordinate values ​​of the upper left and lower right corners of the target frame.

[0110] Step S320, check each folder one by one whether there is one and only one picture of the target object of interest under the folder with the ID number: if there is more than one picture of the target object of interest, move them to the newly added ID folder accordingly.

[0111] Manually check each folder one by one whether there is one and only one picture of the target object of interest under the ID folder: for pictures with more than one target object of interest, move them to the newly added ID folder. Specifically, if there are N (N≥2) pictures of the target object of interest under the ID folder, add N-1 ID folders, and keep the newly added ID folders and the output ID folder directory structure of the cropping program at the same level, so as to correct the first type of labeling errors; if the cropped picture is not the target object of interest, delete it directly; if there is no picture of the target object of interest under a certain ID folder, delete the ID folder.

[0112] Step S330: Generate a record file of the storage paths of all target object images under the condition that all ID folders have one and only one image of the target object of interest.

[0113] Under the condition that all ID folders contain only one image of the target object of interest, a record file of the storage path of all target object images is generated using the Shell command.

[0114] Step S340: extract key information corresponding to each line in the record file.

[0115] Extract the key information corresponding to each line in the record file: target ID number, video frame number, coordinate values ​​of the upper left corner and lower right corner of the target box, target category label, and sort them in ascending order according to the frame number, reallocate the target ID number to a continuous number from small to large, reorganize the annotation information, and convert it into an annotation file in the specific format described in step S230.

[0116] Step S350: load the basic pre-processed material data and the corresponding initial pre-annotation file into the annotation display and adjustment software, use different colors to represent the detection boxes of different target objects of interest, and display the category label name and the ID information of the tracking trajectory / trajectory segment.

[0117] The basic pre-processed material data and the corresponding initial pre-annotated files are loaded into the annotation display and adjustment software, and different colors are used to represent the detection boxes of different target objects of interest. At the same time, the category label name and the ID information of the tracking trajectory / trajectory segment are displayed.

[0118] Step S360: Correct and proofread the second type of labeling errors one by one on the tracking trajectory IDs to ensure that the same target object has only one tracking trajectory ID and the tracking trajectory is continuous.

[0119] Manually correct and proofread the second type of labeling errors one by one by tracking track ID to ensure that the same target object has only one tracking track ID and that the tracking track is coherent: If a target object has multiple different tracking track segment IDs, you only need to select the subsequent tracking track segment ID in the labeling display and adjustment software and modify it to the tracking track segment ID when the target first appears, so that the tracking ID number of the wrong target object in the continuous frame can be changed; if the tracking track omits the target box in some frames, you need to add a target box at the corresponding position to maintain the continuity of the tracking track. In addition, if the size and / or position of the target box does not accurately surround the target object of interest due to inaccurate target detection annotation, it can also be corrected in this step.

[0120] Step S370, save the corrected and calibrated annotation results, and automatically export the annotation file in a specific format as a manually corrected annotation file by the annotation display and adjustment software, and submit it to the multi-target tracking algorithm based on deep learning for the training of the deep neural network model.

[0121] The corrected and calibrated annotation results are saved, and the annotation display and adjustment software automatically exports the annotation file in a specific format as a manually corrected annotation file, which is then handed over to the multi-target tracking algorithm based on deep learning for the training of the deep neural network model.

[0122] Note: (1) The image cropping program will automatically remove those ID folders that contain less than 3 target object images, reducing the workload of manual verification. (2) The naming of the cropped target object images needs to comply with the naming rules described in step S310, otherwise the record file cannot be correctly converted into a specific format annotation file. (3) The Shell command described in step S330 will vary depending on the operating system. For example, the command in Windows system is dir / s / b *.jpg > log.txt, where "jpg" is the extension of the image file. (4) The reason why the target ID number needs to be reassigned to a continuous number from small to large in step S340 is that the target ID number may be discontinuous, and the factors that cause the target ID number to be discontinuous include: the target tracking algorithm ID allocation mechanism, and the operation of discarding ID folders. (5) By reassigning a continuous number from small to large to the target ID number, positioning can be accelerated and the second type of annotation errors can be corrected. It should be noted that the tracking trajectory may miss the target frame in some frames. The factors that cause this situation are: the target detection annotation itself has omission problems; the multi-target tracking algorithm has the first type of annotation errors, and because the tracking results are sampled and saved in step S230, and the cropping program screens out suspected erroneous tracking situations where the tracking trajectory is less than 3 frames. However, the situation of missing the target frame does not occur frequently, so the workload of manually adding the target frame is very small.

[0123] Step S400: Use the corrected annotation file and the corresponding pre-processed material data to train and optimize the deep neural network model of the multi-target tracking algorithm.

[0124] Please see Figure 3 , Figure 3 for Figure 1 In the detailed flow chart of step S400 shown in FIG. 1 , in this embodiment, step S400 includes:

[0125] Step S410: collect the expanded and corrected annotation files, and form a temporary tracking data set together with the corresponding pre-processed material data, and then divide the temporary tracking data set into a training data set and a verification data set according to a set ratio.

[0126] Collect all the manually corrected annotation files of the completed manual correction and calibration work, and form a temporary tracking dataset together with the corresponding basic preprocessed material data. Randomly extract 1 / 10 of the basic preprocessed material data and the corresponding manually corrected annotation files from the temporary tracking dataset as the verification dataset, and the rest of the temporary tracking dataset as the training dataset.

[0127] Step S420: Use the training data set to train and optimize the deep neural network in the multi-target tracking algorithm to obtain a deep neural network model of the multi-target tracking algorithm.

[0128] A deep neural network model of a multi-target tracking algorithm based on deep learning is selected, and after modifying the hyperparameters of the deep neural network model of the multi-target tracking algorithm according to a specific training data set, the model training process is started with the training data set as input.

[0129] Step S430: Use the multi-target tracking algorithm deep neural network model to perform inference testing on the verification data set.

[0130] In the process of training the deep neural network model of the multi-target tracking algorithm, the reasoning accuracy of the trained model on the verification data set is periodically verified.

[0131] Step S440: If the reasoning accuracy reaches the expected level, the multi-target tracking algorithm deep neural network model is saved.

[0132] When the reasoning accuracy of the multi-target tracking algorithm deep neural network model reaches the expected level, the multi-target tracking algorithm deep neural network model is saved for forward reasoning of the second material data piece by piece to achieve algorithm pre-labeling.

[0133] Step S500: call the trained multi-target tracking algorithm deep neural network model to perform forward reasoning to obtain pre-labeling, and use the revised pre-labeling result process to process the second material data to obtain more revised labeling files and corresponding pre-processed material data.

[0134] Further, see Figure 4 , Figure 4 for Figure 1 In the detailed flow chart of step S500 shown in FIG. 1 , in this embodiment, step S500 includes:

[0135] Step S510: remove a copy of the second material data as the current material data.

[0136] A copy of the second material data is removed as the current material data.

[0137] Step S520: Use a multi-target tracking algorithm deep neural network model with satisfactory reasoning accuracy to perform forward reasoning on the material data frame by frame to obtain a pre-labeled file for tracking the target object of interest.

[0138] The material data is used as input, and the trained multi-target tracking algorithm deep neural network model is used to perform forward reasoning operations. The target detection results of the target objects of interest in each frame of the material data are tracked in the previous and next frames in series to obtain the multi-target tracking pre-labeling results.

[0139] Step S530: During the frame-by-frame processing of the multi-target tracking algorithm, the sampling frequency is weighed according to the displacement size of the target object of interest in the previous and next frames, and the current material data is sampled and saved to form this portion of pre-processed material data. The multi-target tracking pre-labeling results of the corresponding frames are saved in a specific format to form this portion of optimized pre-labeling files.

[0140] In the frame-by-frame processing of the multi-target tracking algorithm, the sampling frequency is weighed according to the displacement size of the target object of interest in the previous and next frames, and the current material data is sampled and saved to form this part of pre-processed material data. The multi-target tracking pre-labeling results of the corresponding frames are saved in a specific format to form this part of optimized pre-labeling files. The pre-processed material data and the optimized pre-labeling files together constitute the pre-labeling data.

[0141] Step S540, waiting for the staged correction and calibration pre-annotation tracking result process, correcting and calibrating the optimized pre-annotation file in stages, and obtaining the submitted correction and annotation file corresponding to the pre-processed material data.

[0142] The basic preprocessed material data and the corresponding initial pre-labeled file are replaced with the current preprocessed material data and the current optimized pre-labeled file respectively, and manual correction and calibration work is performed (the same as step S300), and the manually corrected labeling file is obtained for learning and training of the deep neural network model of the multi-target tracking algorithm.

[0143] See also Figure 5 , Figure 5 for Figure 4 The detailed flow chart of step S540 of an embodiment shown in FIG. 1 is a schematic diagram of a detailed flow chart of an embodiment. In this embodiment, step S540 includes:

[0144] Step S541: obtain pre-processed material data and corresponding multi-target tracking annotation files to form pre-annotated data.

[0145] Step S542: Cut out independent target object images according to ID, and store them in groups with the ID number as the folder name.

[0146] Step S543: correct the first type of labeling errors.

[0147] Step S544: Generate a target object tracking image result record file after correcting the first type of labeling errors.

[0148] Step S545, convert the record file into a multi-target tracking annotation file in which each line of multi-target tracking annotation data corresponds to each frame in the pre-processed material data in sequence, and each line of annotation data includes the frame number, target ID number, coordinate values ​​of the upper left and lower right corners of the target box, and the target category label.

[0149] Step S546: Load the annotations in the annotation file and the corresponding pre-processed material data and display them for correction and adjustment.

[0150] Step S547: correct the second type of labeling errors and add or adjust the target box.

[0151] Step S548: Save the adjusted and corrected multi-target tracking annotation file as a corrected annotation file.

[0152] Step S550, calling training and optimizing the multi-target tracking algorithm deep neural network model process.

[0153] Collect newly submitted manually corrected annotation files and form the expanded manually corrected annotation files together with the collected manually corrected annotation files. The corresponding preprocessed material data will form an expanded temporary tracking data set. Randomly extract 1 / 10 of the preprocessed material data and the corresponding manually corrected annotation files from the expanded temporary tracking data set as a new verification data set. Use the rest of the expanded temporary tracking data set as a new training data set. Then call the multi-target tracking algorithm deep neural network model training and optimize the multi-target tracking algorithm deep neural network model process.

[0154] Step S560: determine whether the second material data is empty, and if so, end the multi-target tracking pre-labeling correction calibration.

[0155] Steps S510 to S550 are executed in a loop until the second material data is empty, and the long-term learning process is stopped to end the multi-target tracking pre-labeling correction calibration process.

[0156] Note: (1) Since the manually corrected annotation files are continuously collected and expanded, the expanded temporary tracking data set, the new verification data set and the new training data set are continuously expanded. (2) Since the volume of training data for the multi-target tracking algorithm deep neural network model gradually increases, the reflected scenes, weather and light and shadow changes are becoming more and more abundant, and the included features are becoming more and more comprehensive. The multi-target tracking algorithm deep neural network model can be better optimized, which can gradually improve the accuracy of the algorithm pre-annotation, and the workload of manually correcting and reviewing the algorithm annotation errors is gradually reduced. (3) Since a certain amount of target detection and labeling results are obtained through steps S200 and S300, the target detection results in this step can be directly derived from the forward reasoning results of the deep neural network model of the multi-target tracking algorithm that are saved, thereby eliminating the need for manual correction and review of the target detection and labeling in advance. In step S540, the target detection and target tracking labeling information are corrected and reviewed together. Therefore, the multi-target tracking algorithm based on deep learning in the present invention is no longer limited to the multi-target tracking algorithm based on detection results (such as DeepSORT (Simple Online and Realtime Tracking with a Deep Association Metric) multi-target tracking algorithm). Any efficient and high-precision deep learning multi-target tracking algorithm can be used in this step, including but not limited to: FairMOT (A Simple Baseline for Multi-Object Tracking) and JDE (Towards Real-Time Multi-Object Tracking) algorithms.

[0157] Step S600: collect and summarize the corrected annotation files and the corresponding pre-processed material data to form a single-camera multi-target tracking data set.

[0158] Collect and summarize manually corrected annotation files and corresponding pre-processed material data to form a single-camera multi-target tracking dataset.

[0159] Please see Figures 6 to 10The present invention relates to a single-camera multi-target tracking data set production system, which includes a first collection module 10, a labeling module 20, a first calling module 30, a training module 40, a second calling module 50 and a second collection module 60, wherein the first collection module 10 is used to collect original videos or image sequences to be labeled, and manage the collected original videos or image sequences to be labeled as total material data, and divide them into first material data and second material data; the labeling module 20 is used to pre-label the first material data using a traditional multi-target tracking algorithm, and perform frame extraction preprocessing on the first material data and the corresponding pre-labeling to obtain pre-processed material data and pre-labeled tracking data respectively. The first calling module 30 is used to call the staged correction and calibration pre-labeling result process, perform correction and calibration processing on the pre-labeling tracking results, and obtain the correction labeling file; the training module 40 is used to train and optimize the multi-target tracking algorithm deep neural network model using the correction labeling file and the corresponding pre-processed material data; the second calling module 50 is used to call the trained multi-target tracking algorithm deep neural network model for forward reasoning, obtain the pre-labeling, and use the correction pre-labeling result process to process the second material data to obtain more correction labeling files and corresponding pre-processed material data; the second collection module 60 is used to collect and summarize the correction labeling files and the corresponding pre-processed material data to form a single-camera multi-target tracking data set. The first collection module 10 collects the original video or image sequence to be labeled to form the total material data, and divides it into 100 parts, randomly extracts 3 parts to form the first material data, and the remaining 97 parts constitute the second material data.

[0160] Further, see Figure 7 , Figure 7 for Figure 6 , in this embodiment, the labeling module 20 includes a first acquisition unit 21, a second acquisition unit 22 and a first component unit 23, wherein the first acquisition unit 21 is used to use the existing deep learning multi-target detection algorithm to perform target detection frame and category labeling on the first material data frame by frame, and after correction and confirmation, obtain the position, size and target category attribute information of the circumscribed rectangular frame of the target object of interest as the target object detection result; the second acquisition unit 22 is used to use the traditional multi-target tracking algorithm to pre-label the target object detection result, each line of multi-target tracking labeling data corresponds to each frame in the first material data one by one in sequence, each line of labeling data includes a frame number, a target ID number, the coordinate values ​​of the upper left corner and the lower right corner of the target frame and the target category label number, and obtain the pre-labeling file for tracking the target object of interest; the first component unit 23 is used to extract frames of the first material data and save them as video files to form basic pre-processed material data, which together with the corresponding labeling file constitute the pre-labeling data.

[0161] The first acquisition unit 21 uses an existing deep learning multi-target detection algorithm to perform target detection frame and category annotation on the first material data frame by frame, and after manual correction and confirmation, obtains the position, size and target category attribute information of the circumscribed rectangular frame of the target object of interest as the target detection annotation.

[0162] The second acquisition unit 22 takes the first material data and the target detection annotations corresponding to each frame as input, and uses a traditional multi-target tracking algorithm (different from the currently popular multi-target tracking algorithm based on deep learning) to track the target detection annotations of the target objects of interest in each frame of the first material data in series with the previous and next frames to obtain a multi-target tracking result.

[0163] During the process of performing target object tracking serially frame by frame in the multi-target tracking algorithm, the first component unit 23 weighs the sampling frequency according to the displacement size of the target object of interest in the previous and next frames (usually the sampling frequency for a video with a frame rate of 25 frames per second is 2 to 5 times), extracts frames of the first material data and saves them as a video file to form basic pre-processed material data, and saves the multi-target tracking results of the corresponding frames as an initial pre-labeled file in a specific format.

[0164] Preferably, see Figure 8 , Figure 8 for Figure 6 , in which the training module 40 includes a collecting unit 41, a third acquiring unit 42, a testing unit 43 and a saving unit 44, wherein the collecting unit 41 is used to collect, expand and correct the annotation files, and together with the corresponding pre-processed material data, form a temporary tracking data set, and then divide the temporary tracking data set into a training data set and a verification data set according to a set ratio; the third acquiring unit 42 is used to train and optimize the deep neural network in the multi-target tracking algorithm using the training data set to obtain a deep neural network model of the multi-target tracking algorithm; the testing unit 43 is used to use the deep neural network model of the multi-target tracking algorithm to perform inference test on the verification data set; the saving unit 44 is used to save the deep neural network model of the multi-target tracking algorithm if the inference accuracy reaches the expected level.

[0165] The collecting unit 41 collects all the manually corrected annotation files of the completed manual correction and calibration work, and together with the corresponding basic preprocessed material data, forms a temporary tracking data set, randomly extracts 1 / 10 of the basic preprocessed material data and the corresponding manually corrected annotation files from the temporary tracking data set as the verification data set, and the rest of the temporary tracking data set is used as the training data set.

[0166] The third acquisition unit 42 selects a deep neural network model of a multi-target tracking algorithm based on deep learning, modifies the hyperparameters of the deep neural network model of the multi-target tracking algorithm according to a specific training data set, and then starts a model training process with the training data set as input.

[0167] During the process of training the deep neural network model of the multi-target tracking algorithm, the testing unit 43 periodically verifies the reasoning accuracy of the trained model on the verification data set.

[0168] When the reasoning accuracy of the multi-target tracking algorithm deep neural network model reaches the expected level, the saving unit 44 saves the multi-target tracking algorithm deep neural network model for forward reasoning of the second material data piece by piece to achieve algorithm pre-labeling.

[0169] Furthermore, if Fig. 9 As shown, Fig. 9 for Figure 6 , in this embodiment, the second calling module 50 includes a removal unit 51, a fourth acquisition unit 52, a second composition unit 53, a fifth acquisition unit 54, a calling unit 55 and a judgment unit 56, wherein the removal unit 51 is used to remove a copy from the second material data as the current material data; the fourth acquisition unit 52 is used to use the multi-target tracking algorithm deep neural network model with the inference accuracy reaching the standard to perform forward inference on the current material data frame by frame to obtain the tracking pre-labeled file of the target object of interest; the second composition unit 53 is used to, during the frame-by-frame processing of the multi-target tracking algorithm, determine the pre-labeled file according to the location of the target object of interest in the previous and next frames. The sampling frequency is weighed according to the displacement size, and the frame of the material data is extracted and saved to form the pre-processed material data, and the multi-target tracking pre-labeling results of the corresponding frames are saved in a specific format to form an optimized pre-labeling file; the fifth acquisition unit 54 is used to wait for the stage-by-stage correction and calibration pre-labeling tracking result process, correct and calibrate the present optimized pre-labeling file in stages, and obtain the submitted correction and labeling file corresponding to the present pre-processed material data; the calling unit 55 is used to call the training and optimization of the deep neural network model process of the multi-target tracking algorithm; the judgment unit 56 is used to judge whether the second material data is empty, and if so, end the multi-target tracking pre-labeling correction and calibration.

[0170] The removal unit 51 removes a copy of the second material data as the current copy of the material data.

[0171] The fourth acquisition unit 52 uses a multi-target tracking algorithm deep neural network model with satisfactory reasoning accuracy to perform forward reasoning on the material data frame by frame to obtain a pre-labeled file for tracking the target object of interest.

[0172] During the frame-by-frame processing of the multi-target tracking algorithm, the second component unit 53 weighs the sampling frequency according to the displacement size of the target object of interest in the previous and next frames, extracts frames from the current material data and saves them to form this portion of pre-processed material data, and saves the multi-target tracking pre-labeling results of the corresponding frames in a specific format to form this portion of optimized pre-labeling file.

[0173] The fifth acquisition unit 54 waits for the process of correcting and calibrating the pre-annotated tracking results in stages, corrects and calibrates the optimized pre-annotated file in stages, and obtains the submitted corrected annotation file corresponding to the pre-processed material data. Collect the newly submitted manual correction annotation file, and together with the collected manual correction annotation file, form the expanded manual correction annotation file, and form the corresponding pre-processed material data together to form an expanded temporary tracking data set, randomly extract 1 / 10 of the pre-processed material data and the corresponding manual correction annotation file from the expanded temporary tracking data set as a new verification data set, and expand the rest of the temporary tracking data set as a new training data set, in preparation for calling the multi-target tracking algorithm deep neural network model training and optimizing the multi-target tracking algorithm deep neural network model process.

[0174] The calling unit 55 calls the training and optimization process of the deep neural network model of the multi-target tracking algorithm.

[0175] The judgment unit 56 judges whether the second material data is empty. If not, a new copy is removed from the second material data as the current material data until the second material data is empty, stopping the long-term learning process and ending the multi-target tracking pre-labeling correction calibration process.

[0176] Further, see Fig.10 , Fig.10 for Fig. 9, in this embodiment, the fifth acquisition unit 54 includes an acquisition subunit 541, a storage subunit 542, a first correction subunit 543, a generation subunit 544, a conversion subunit 545, a loading subunit 546, a correction subunit 547 and a storage subunit 548, wherein the acquisition subunit 541 is used to acquire pre-processed material data and corresponding multi-target tracking annotation files to form pre-annotated data; the storage subunit 542 is used to crop independent target object images according to IDs, and store them in groups with ID numbers as folder names; the first correction subunit 543 is used to correct the first type of annotation errors; the generation subunit 544 is used to generate A result record file of the tracking target object image after correcting the first type of labeling errors; a conversion subunit 545, used to convert the record file into a multi-target tracking labeling file in which each line of multi-target tracking labeling data corresponds to each frame in the pre-processed material data in a one-to-one sequence, and each line of labeling data includes a frame number, a target ID number, the coordinate values ​​of the upper left corner and the lower right corner of the target box, and a target category label; a loading subunit 546, used to load the labels in the labeling file and the corresponding pre-processed material data and display them for correction and adjustment; a correction subunit 547, used to correct the second type of labeling errors, and add or adjust the target box; a saving subunit 548, used to save the adjusted and corrected multi-target tracking labeling file as a corrected labeling file.

[0177] Compared with the prior art, the method and system for producing a single-camera multi-target tracking data set provided in this embodiment collects the original video or image sequence to be labeled, and manages the collected original video or image sequence to be labeled as the total material data, which is divided into the first material data and the second material data; the first material data is pre-labeled using the traditional multi-target tracking algorithm, and the first material data and the corresponding pre-labeling are pre-processed by extracting frames to obtain the pre-processed material data and the pre-labeled tracking results respectively; the pre-labeling result process of staged correction and calibration is called to correct and calibrate the pre-labeled tracking results to obtain a corrected annotation file; the multi-target tracking algorithm deep neural network model is trained and optimized using the corrected annotation file and the corresponding pre-processed material data; the trained multi-target tracking algorithm deep neural network model is called for forward reasoning to obtain the pre-labeling, and the second material data is processed using the corrected pre-labeling result process to obtain more corrected annotation files and the corresponding pre-processed material data; the corrected annotation files and the corresponding pre-processed material data are collected and summarized to form a single-camera multi-target tracking data set. The beneficial effects achieved by this embodiment are as follows:

[0178] First, since the traditional multi-target tracking algorithm that does not require supervised training is used for initial pre-labeling, only part of the video data is manually corrected and calibrated in stages for the initial pre-labeling, and then the manually corrected and calibrated initial pre-labeling file is used to optimize the deep neural network model of the multi-target tracking algorithm based on deep learning, thereby obtaining a relatively accurate optimized pre-labeling result. After continuously collecting and expanding the manually corrected labeling files and iteratively training and optimizing the deep neural network model of the multi-target tracking algorithm, manually correcting, reviewing and confirming the optimized pre-labeling results one by one, the final multi-target tracking labeling file is obtained. This not only utilizes the efficiency and tirelessness of the machine for pre-labeling and reduces the workload of manual labeling, but also effectively solves the problem of labeling errors in the target tracking algorithm, ensuring the accuracy of the labeling results of the single-camera multi-target tracking data.

[0179] Secondly, a new multi-target tracking annotation organization format is adopted. One annotation file covers the target detection information and target tracking information of the corresponding video segment or image sequence. Compared with using two files to represent the detection and tracking annotation information respectively, the redundancy and fragmentation of the annotation data are reduced, and the storage and management efficiency of the annotation data can be improved. Since the multi-target tracking annotation organization format represents the target objects with different IDs appearing in the same frame with a row of data, and the video frame numbers are arranged in order from small to large, the efficiency can be greatly improved when cropping the target object graph, and there is no need to repeatedly crop the target objects one by one across multiple frames. In addition, the start and end video frame numbers or tracking ID ranges when cropping the target object graph can be flexibly specified.

[0180] Third, since the tracking target object of each frame in the video or image sequence is cut out into independent target object images according to the corresponding ID, and the target object images are stored in groups with the ID number as the folder name, it is convenient to quickly locate the first type of labeling errors in target tracking manually, that is, the trajectory segments (Tracklet) of multiple target objects are mistakenly labeled as the complete trajectory (Trajectory) of a target object. Moreover, when manually correcting the first type of labeling errors, it is only necessary to move some of the target object images in a certain ID folder to another newly created ID folder or the ID folder of the corresponding target object. In addition, the operation of deleting irrelevant targets or background images in a certain ID folder is actually equivalent to "purifying" the complete trajectory of the target object. The operation is simple and efficient, highly feasible, and saves data labeling costs.

[0181] Fourth, the video or image sequence is sampled, saving data storage space; there is no need to centrally crop independent images of the tracking target object in each frame of the video or image sequence on the server, reducing the frequency of random read and write of the server memory; the independent images cropped from the tracking target object in each frame of the video or image sequence are only completed on each client, the server only sends pre-labeled files and corresponding pre-processed materials, and the client only uploads manually corrected labeling files, thereby saving data transmission bandwidth between the server and the client, and the correction and calibration process of the pre-labeled files can be completed offline on each client.

[0182] Although preferred embodiments of the present invention have been described, additional changes and modifications may be made to these embodiments by those skilled in the art once the basic inventive concepts are known. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention. Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A method for preparing a single-camera multi-target tracking dataset, characterized in that: The following steps are involved: Collecting original videos or image sequences to be annotated, and managing the collected original videos or image sequences to be annotated as total material data, and dividing them into first material data and second material data; Pre-labeling the first material data using a conventional multi-target tracking algorithm, and performing frame extraction pre-processing on the first material data and the corresponding pre-labeling to obtain pre-processed material data and pre-labeling tracking results respectively; Call the staged correction and calibration pre-marking result process to perform correction and calibration on the pre-marking tracking results to obtain a corrected marking file; Using the corrected annotation file and the corresponding pre-processed material data to train and optimize a deep neural network model for a multi-target tracking algorithm; Calling the trained multi-target tracking algorithm deep neural network model to perform forward reasoning to obtain pre-annotation, and using the revised pre-annotation result process to process the second material data to obtain more revised annotation files and corresponding pre-processed material data; The corrected annotation files and the corresponding pre-processed material data are collected and summarized to form a single-camera multi-target tracking data set.

2. The method for preparing a single-camera multi-target tracking data set according to claim 1, characterized in that: The steps of pre-marking the first material data using a conventional multi-target tracking algorithm, and performing frame extraction pre-processing on the first material data and the corresponding pre-marking to obtain the pre-processed material data and the pre-marked tracking result respectively include: Using an existing deep learning multi-target detection algorithm to perform target detection frame and category annotation on the first material data frame by frame and after correction and confirmation, the position, size and target category attribute information of the circumscribed rectangular frame of the target object of interest are obtained as the target object detection result; The target object detection results are pre-labeled using a traditional multi-target tracking algorithm, each line of multi-target tracking labeling data corresponds to each frame in the first material data in a one-to-one order, and each line of labeling data includes a frame number, a target ID number, coordinate values ​​of the upper left corner and the lower right corner of the target frame, and a target category label number, to obtain a pre-labeling file for tracking of the target object of interest; Extract frames of the first material data and save them as video files to form basic pre-processed material data, which together with corresponding annotation files form pre-annotated data.

3. The single-camera multi-target tracking data set production method according to claim 1, characterized in that: The step of using the corrected annotation file and the corresponding pre-processed material data to train and optimize the multi-target tracking algorithm deep neural network model includes: Collecting, expanding and correcting the annotation files, and forming a temporary tracking data set together with the corresponding pre-processed material data, and then dividing the temporary tracking data set into a training data set and a verification data set according to a set ratio; Using the training data set to train and optimize the deep neural network in the multi-target tracking algorithm, to obtain a deep neural network model of the multi-target tracking algorithm; Performing inference testing on the validation data set using a multi-target tracking algorithm deep neural network model; If the reasoning accuracy reaches the expectation, the multi-target tracking algorithm deep neural network model is saved.

4. The single-camera multi-target tracking data set production method according to claim 1, characterized in that: The steps of calling the trained multi-target tracking algorithm deep neural network model for forward reasoning to obtain pre-annotation, and processing the second material data using a corrected pre-annotation result process to obtain more corrected annotation files and corresponding pre-processed material data include: removing a copy of the second material data as the current material data; Use the deep neural network model of the multi-target tracking algorithm with the required inference accuracy to perform forward inference on the material data frame by frame to obtain the pre-labeled tracking file of the target object of interest; In the frame-by-frame processing of the multi-target tracking algorithm, the sampling frequency is weighed according to the displacement size of the target object of interest in the previous and next frames, and the current material data is extracted and saved to form this pre-processed material data, and the multi-target tracking pre-labeling results of the corresponding frames are saved in a specific format to form this optimized pre-labeling file; Waiting for the process of correcting and calibrating the pre-annotation tracking results in stages, correcting and calibrating the optimized pre-annotated file in stages, and obtaining the submitted corrected and annotated file corresponding to the pre-processed material data; Call training and optimize the deep neural network model process of multi-target tracking algorithm; It is determined whether the second material data is empty, and if so, the multi-target tracking pre-labeling correction calibration is terminated.

5. The method for producing a single-camera multi-target tracking data set as claimed in claim 4, characterized in that: The process of waiting for the results of the pre-marking and tracking in stages, correcting and calibrating the optimized pre-marking file in stages, and obtaining the submitted corrected and marked file corresponding to the pre-processed material data includes: Obtain pre-processed material data and corresponding multi-target tracking annotation files to form pre-annotated data; Crop out independent target object images by ID and store them in groups with the ID number as the folder name; Corrected the first type of labeling errors; Generate a tracking target object image result record file after correcting the first type of labeling errors; Convert the record file into a multi-target tracking annotation file in which each line of multi-target tracking annotation data corresponds to each frame in the pre-processed material data in sequence, and each line of annotation data includes the frame number, target ID number, coordinate values ​​of the upper left corner and lower right corner of the target frame, and target category label; Load the annotations in the annotation file and the corresponding pre-processed material data and display them for correction and adjustment; Correct the second type of labeling errors and add or adjust the target box; The adjusted and corrected multi-target tracking annotation file is saved as a corrected annotation file.

6. A single-camera multi-target tracking data set production system, characterized in that: include: A first collection module (10) is used to collect original videos or image sequences to be annotated, and manage the collected original videos or image sequences to be annotated as total material data, dividing them into first material data and second material data; A labeling module (20) is used to pre-label the first material data using a conventional multi-target tracking algorithm, and to perform frame extraction pre-processing on the first material data and the corresponding pre-labeling, to obtain pre-processed material data and pre-labeling tracking results respectively; A first calling module (30) is used to call a staged correction and calibration pre-marking result process, perform correction and calibration processing on the pre-marking tracking result, and obtain a corrected marking file; A training module (40), used to train and optimize a deep neural network model of a multi-target tracking algorithm using the corrected annotation file and corresponding pre-processed material data; A second calling module (50) is used to call the trained multi-target tracking algorithm deep neural network model to perform forward reasoning, obtain pre-annotation, and use a corrected pre-annotation result process to process the second material data to obtain more corrected annotation files and corresponding pre-processed material data; The second collection module (60) is used to collect and aggregate the corrected annotation files and the corresponding pre-processed material data to form a single-camera multi-target tracking data set.

7. The single-camera multi-target tracking data set production system according to claim 6, characterized in that: The marking module (20) comprises: A first acquisition unit (21) is used to use an existing deep learning multi-target detection algorithm to perform target detection frame and category labeling on the first material data frame by frame and obtain the position, size and target category attribute information of the circumscribed rectangular frame of the target object of interest as the target object detection result after correction and confirmation; A second acquisition unit (22) is used to pre-label the target object detection result using a traditional multi-target tracking algorithm, each line of multi-target tracking label data corresponds to each frame in the first material data in a one-to-one order, each line of label data includes a frame number, a target ID number, coordinate values ​​of the upper left corner and the lower right corner of the target frame, and a target category label number, to obtain a pre-label file for tracking the target object of interest; The first composition unit (23) is used to extract frames from the first material data and save them as video files to form basic pre-processed material data, which together with the corresponding annotation files form pre-annotated data.

8. The single-camera multi-target tracking data set production system according to claim 6, characterized in that: The training module (40) comprises: A collecting unit (41) is used to collect the expanded and corrected annotation files and form a temporary tracking data set together with the corresponding pre-processed material data, and then divide the temporary tracking data set into a training data set and a verification data set according to a set ratio; A third acquisition unit (42) is used to train and optimize the deep neural network in the multi-target tracking algorithm using the training data set to obtain a deep neural network model of the multi-target tracking algorithm; A testing unit (43), used to perform reasoning testing on the verification data set using a multi-target tracking algorithm deep neural network model; A saving unit (44) is used to save the multi-target tracking algorithm deep neural network model if the reasoning accuracy reaches the expected level.

9. The single-camera multi-target tracking data set production system according to claim 6, characterized in that: The second calling module (50) comprises: A removal unit (51), used for removing a copy of the second material data as the current material data; A fourth acquisition unit (52) is used to use a multi-target tracking algorithm deep neural network model with satisfactory reasoning accuracy to perform forward reasoning on the material data frame by frame to obtain a pre-labeled file for tracking a target object of interest; The second component unit (53) is used to weigh the sampling frequency according to the displacement size of the target object of interest in the previous and next frames during the frame-by-frame processing of the multi-target tracking algorithm, extract frames of the material data and save them to form the pre-processed material data, and save the multi-target tracking pre-labeling results of the corresponding frames in a specific format to form the optimized pre-labeling file; A fifth acquisition unit (54) is used to wait for the process of correcting and calibrating the pre-annotation tracking results in stages, correcting and calibrating the optimized pre-annotated file in stages, and obtaining the submitted corrected and annotated file corresponding to the pre-processed material data; A calling unit (55) is used to call a training and optimization process of a deep neural network model of a multi-target tracking algorithm; The judging unit (56) is used to judge whether the second material data is empty, and if so, terminate the multi-target tracking pre-labeling correction calibration.

10. The single-camera multi-target tracking data set production system according to claim 9, characterized in that: The fifth acquisition unit (54) comprises: An acquisition subunit (541) is used to acquire pre-processed material data and corresponding multi-target tracking annotation files to form pre-annotated data; The storage subunit (542) is used to cut out independent target object images according to IDs, and store them in groups with the ID numbers as folder names; A first correction subunit (543), used to correct the first type of labeling errors; A generating subunit (544), used to generate a tracking target object image result record file after correcting the first type of labeling errors; A conversion subunit (545) is used to convert the record file into a multi-target tracking annotation file in which each line of multi-target tracking annotation data corresponds to each frame in the pre-processed material data in a one-to-one order, and each line of annotation data includes a frame number, a target ID number, coordinate values ​​of the upper left corner and the lower right corner of the target frame, and a target category label; A loading subunit (546) is used to load the annotations in the annotation file and the corresponding pre-processed material data and then display them for correction and adjustment; A correction subunit (547), used to correct the second type of labeling errors, and to add or adjust the target box; The saving subunit (548) is used to save the adjusted and corrected multi-target tracking annotation file as a corrected annotation file.

Images