An algorithm closed-loop assisted bidirectional multi-model data labeling method and device

Abstract

The application provides a kind of algorithm closed loop auxiliary bidirectional multi-model data labeling method and device, comprising: initializing bidirectional multi-model labeling workflow;Based on workflow, the image data to be labeled is labeled to generate labeled data;According to the labeled data and workflow, generate data set;Wherein, the labeling includes one or more of the following: bidirectional single model labeling, bidirectional multi-model simultaneous correction labeling and multi-target bidirectional multi-model data correction labeling;The application accurately locates the target pixel coordinates by bidirectional single model labeling, bidirectional multi-model simultaneous correction labeling and / or multi-target bidirectional multi-model data correction labeling on the image data to be labeled, greatly saves the labor cost of labeling, and improves the speed of data labeling.

Description

Technical Field

[0001] This invention belongs to the field of image data processing technology, specifically relating to a method and apparatus for algorithm-closed-loop assisted bidirectional multi-model data annotation. Background Technology

[0002] Target tracking is one of the fundamental tasks in the field of computer vision. It is a technique that uses contextual information from a video or image sequence to model the appearance information of a target, thereby predicting the target's motion state and locating the target. It has wide applications in video surveillance, visual navigation, and intelligent human-computer interaction.

[0003] Current tracking algorithms handle simple scenarios well, but they face significant challenges in locating targets due to uncertainties in target motion and complex environments such as lighting changes and occlusion. Designing tracking algorithms with high accuracy and robustness remains a major hurdle. The advantage of deep learning lies in its ability to learn from vast amounts of data, ensuring that the tracking algorithm network can achieve stable tracking and accurate target location.

[0004] Currently, high-performing deep learning models require massive amounts of data for training, and this data is almost always labeled manually, incurring significant human costs and varying labeling quality. In visual object tracking algorithms, images need to be labeled frame by frame, and adjacent images often have high similarity. Relying solely on manual labeling leads to low labeling efficiency, and in some deep learning application scenarios, real-time data labeling is crucial. Summary of the Invention

[0005] To overcome the shortcomings of the existing technology, this invention proposes a method for algorithmic closed-loop assisted bidirectional multi-model data annotation, comprising:

[0006] Initialize the bidirectional multi-model annotation workflow;

[0007] Based on the workflow, the image data to be labeled is labeled to generate labeled data;

[0008] Generate a dataset based on the labeled data and workflow;

[0009] The annotation includes one or more of the following:

[0010] Two-way single-model annotation, two-way multi-model simultaneous calibration annotation, and multi-objective two-way multi-model data calibration annotation.

[0011] Preferably, the initialization of the bidirectional multi-model annotation workflow includes:

[0012] Select an auxiliary algorithm for annotation from the preset algorithm model library;

[0013] Set a confidence threshold for the auxiliary algorithm and set the initial target position coordinates;

[0014] Configure the dataset type, format, and dataset generation parameters;

[0015] The dataset type includes at least one or more of the following: object detection dataset and object tracking dataset; the auxiliary algorithm includes at least one or more of the following: initial bounding box recommendation algorithm, object detection algorithm, automatic bounding box correction and annotation algorithm, and single object tracking algorithm; the dataset generation configuration parameters include at least one or more of the following: similarity, time interval, overlap rate, center point offset, and total number of frames.

[0016] Preferably, based on the workflow, bidirectional single-model annotation is performed on the image data to be annotated to generate annotation data, including:

[0017] Based on a selected auxiliary algorithm, forward continuous image annotation is performed on the image data to be annotated to obtain the visual location result of the target position in the image data to be annotated;

[0018] If the error of the visualization positioning result is less than the confidence threshold, then the annotation data is generated and the process ends; otherwise, the continuous annotation of the forward image is stopped and the target position coordinates are reinitialized.

[0019] Based on the re-initialized target location coordinates, the visualized positioning results of the target location are continuously annotated and corrected in reverse order until a preset termination condition is met and annotation data is generated.

[0020] Preferably, based on the workflow, multi-objective bidirectional multi-model data correction annotation is performed on the image data to be annotated to generate annotated data, including:

[0021] Based on the auxiliary algorithm corresponding to each target in the labeled image data, bidirectional single-model annotation is performed on each target in the image data to be annotated to generate labeled data.

[0022] Preferably, based on the workflow, bidirectional multi-model simultaneous correction annotation is performed on the image data to be annotated to generate annotated data, including:

[0023] Based on the selected auxiliary algorithms, forward continuous image annotation is performed on the image data to be annotated, and the visualization localization results corresponding to each auxiliary algorithm of the target location in the image data to be annotated are obtained.

[0024] If the error of the visualization positioning results corresponding to all auxiliary algorithms is less than the confidence threshold, then the labeled data is generated and the process ends.

[0025] If there is an auxiliary algorithm whose visualization positioning result error is less than the confidence threshold and an auxiliary algorithm whose visualization positioning result error is greater than or equal to the confidence threshold, then the continuous annotation of the forward image is stopped. The auxiliary algorithm whose visualization positioning result error is less than the confidence threshold is used to correct the auxiliary algorithm whose error is greater than or equal to the confidence threshold. Then, the continuous annotation of the forward image data to be annotated is continued using each auxiliary algorithm until the error of the visualization positioning result corresponding to all auxiliary algorithms is less than the confidence threshold. Then, the annotation data is generated and the process ends.

[0026] If the error of the visualization positioning result corresponding to all auxiliary algorithms is greater than or equal to the confidence threshold, the forward image continuous annotation is stopped, and the target position coordinates are reinitialized. Based on the reinitialized target position coordinates, the selected auxiliary algorithms are used to perform reverse continuous annotation correction on the visualization positioning result of the target position until the preset termination condition is met, and the annotation data is generated and the process ends.

[0027] Preferably, generating the dataset based on the labeled data and workflow includes:

[0028] Based on the dataset type set in the workflow, select the target detection dataset and / or target tracking dataset that need to be generated;

[0029] When it is necessary to generate an object detection dataset, the labeled data with the lowest similarity to the same object are extracted based on the similarity algorithm to create the object detection dataset;

[0030] When it is necessary to generate a target tracking dataset, continuous images are extracted from the labeled data to form the target tracking dataset according to the set time interval, overlap rate, center point deviation and total number of frames.

[0031] Preferably, after generating the dataset, the method further includes:

[0032] Based on the object detection dataset, script training and / or hyperparameter configuration of the object detection algorithm are performed to train the detection dataset and generate the object detection algorithm model.

[0033] Based on the target tracking dataset, script training and / or hyperparameter configuration of the single target tracking algorithm are performed to train the tracking dataset and generate a single target tracking algorithm model.

[0034] Based on the same technical concept, the present invention also provides an apparatus for algorithm closed-loop assisted bidirectional multi-model data annotation, comprising: a human-computer interactive annotation interface, an annotation module, and an annotation data generation module;

[0035] The human-computer interaction annotation interface is used to initialize the bidirectional multi-model annotation workflow;

[0036] The annotation module is used to annotate the image data to be annotated based on the workflow and generate annotation data;

[0037] The labeled data generation module is used to generate a dataset based on the labeled data and workflow;

[0038] The annotation includes one or more of the following:

[0039] Two-way single-model annotation, two-way multi-model simultaneous calibration annotation, and multi-objective two-way multi-model data calibration annotation.

[0040] Preferably, the human-computer interaction annotation interface is specifically used for:

[0041] Select an auxiliary algorithm for annotation from the preset algorithm model library;

[0042] Set a confidence threshold for the auxiliary algorithm and set the initial target position coordinates;

[0043] Configure the dataset type, format, and dataset generation parameters;

[0044] The dataset type includes at least one or more of the following: object detection dataset and object tracking dataset; the auxiliary algorithm includes at least one or more of the following: initial bounding box recommendation algorithm, object detection algorithm, automatic bounding box correction and annotation algorithm, and single object tracking algorithm; the dataset generation configuration parameters include at least one or more of the following: similarity, time interval, overlap rate, center point offset, and total number of frames.

[0045] Preferably, the annotation module performs bidirectional single-model annotation on the image data to be annotated based on the workflow, generating annotation data, including:

[0046] Based on a selected auxiliary algorithm, forward continuous image annotation is performed on the image data to be annotated to obtain the visual location result of the target position in the image data to be annotated;

[0047] If the error of the visualization positioning result is less than the confidence threshold, then the annotation data is generated and the process ends; otherwise, the continuous annotation of the forward image is stopped and the target position coordinates are reinitialized.

[0048] Based on the re-initialized target location coordinates, the visualized positioning results of the target location are continuously annotated and corrected in reverse order until a preset termination condition is met and annotation data is generated.

[0049] Preferably, the annotation module, based on the workflow, performs multi-objective, bidirectional, multi-model data correction annotation on the image data to be annotated, generating annotated data, including:

[0050] Based on the auxiliary algorithm corresponding to each target in the labeled image data, bidirectional single-model annotation is performed on each target in the image data to be annotated to generate labeled data.

[0051] Preferably, the annotation module, based on the workflow, performs bidirectional multi-model simultaneous correction annotation on the image data to be annotated, generating annotated data, including:

[0052] Based on the selected auxiliary algorithms, forward continuous image annotation is performed on the image data to be annotated, and the visualization localization results corresponding to each auxiliary algorithm of the target location in the image data to be annotated are obtained.

[0053] If the error of the visualization positioning results corresponding to all auxiliary algorithms is less than the confidence threshold, then the labeled data is generated and the process ends.

[0054] If there is an auxiliary algorithm whose visualization positioning result error is less than the confidence threshold and an auxiliary algorithm whose visualization positioning result error is greater than or equal to the confidence threshold, then the continuous annotation of the forward image is stopped. The auxiliary algorithm whose visualization positioning result error is less than the confidence threshold is used to correct the auxiliary algorithm whose error is greater than or equal to the confidence threshold. Then, the continuous annotation of the forward image data to be annotated is continued using each auxiliary algorithm until the error of the visualization positioning result corresponding to all auxiliary algorithms is less than the confidence threshold. Then, the annotation data is generated and the process ends.

[0055] If the error of the visualization positioning result corresponding to all auxiliary algorithms is greater than or equal to the confidence threshold, the forward image continuous annotation is stopped, and the target position coordinates are reinitialized. Based on the reinitialized target position coordinates, the selected auxiliary algorithms are used to perform reverse continuous annotation correction on the visualization positioning result of the target position until the preset termination condition is met, and the annotation data is generated and the process ends.

[0056] Preferably, the annotation data generation module is specifically used for:

[0057] Based on the dataset type set in the workflow, select the target detection dataset and / or target tracking dataset that need to be generated;

[0058] When it is necessary to generate an object detection dataset, the labeled data with the lowest similarity to the same object are extracted based on the similarity algorithm to create the object detection dataset;

[0059] When it is necessary to generate a target tracking dataset, continuous images are extracted from the labeled data to form the target tracking dataset according to the set time interval, overlap rate, center point deviation and total number of frames.

[0060] Preferably, the device further includes an online model training module; the online model training module is specifically used for:

[0061] Based on the object detection dataset, script training and / or hyperparameter configuration of the object detection algorithm are performed to train the detection dataset and generate the object detection algorithm model.

[0062] Based on the target tracking dataset, script training and / or hyperparameter configuration of the single target tracking algorithm are performed to train the tracking dataset and generate a single target tracking algorithm model.

[0063] Preferably, it also includes an online model training module; the online model training module is specifically used for:

[0064] Based on the object detection dataset, script training and / or hyperparameter configuration of the object detection algorithm are performed to train the detection dataset and generate the object detection algorithm model.

[0065] Based on the target tracking dataset, script training and / or hyperparameter configuration of the single target tracking algorithm are performed to train the tracking dataset and generate a single target tracking algorithm model.

[0066] Compared with the closest existing technology, the present invention has the following beneficial effects:

[0067] This invention provides a method and apparatus for algorithm-closed-loop assisted bidirectional multi-model data annotation, comprising: initializing a bidirectional multi-model annotation workflow; annotating image data to be annotated based on the workflow to generate annotated data; and generating a dataset based on the annotated data and the workflow. The annotation includes one or more of the following: bidirectional single-model annotation, bidirectional multi-model simultaneous correction annotation, and multi-target bidirectional multi-model data correction annotation. This invention accurately locates target pixel coordinates by performing bidirectional single-model annotation, bidirectional multi-model simultaneous correction annotation, and / or multi-target bidirectional multi-model data correction annotation on the image data to be annotated, significantly saving manual annotation costs and improving the speed of data annotation. Attached Figure Description

[0068] Figure 1 This is a schematic diagram of a method for algorithm-assisted bidirectional multi-model data annotation provided by the present invention.

[0069] Figure 2 This is a schematic diagram of the bidirectional single-model annotation process in an algorithm-closed-loop assisted bidirectional multi-model data annotation method provided by the present invention;

[0070] Figure 3 This is a schematic diagram of the bidirectional multi-model simultaneous correction and annotation process in an algorithm-closed-loop assisted bidirectional multi-model data annotation method provided by the present invention;

[0071] Figure 4 A schematic diagram of the structure of a device for algorithm-closed-loop assisted bidirectional multi-model data annotation provided by the present invention;

[0072] Figure 5 A schematic diagram illustrating the structure of a specific example of an algorithm-closed-loop assisted bidirectional multi-model data annotation device provided by the present invention;

[0073] Figure 6 A schematic diagram illustrating the structure of a specific example of an algorithm-closed-loop assisted bidirectional multi-model data annotation device provided by the present invention;

[0074] Figure 7 A schematic diagram of the server structure for the algorithm model library provided for the invention. Detailed Implementation

[0075] Various exemplary embodiments of the present application will now be described in detail with reference to the accompanying drawings. It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of the present application.

[0076] At the same time, it should be understood that, for ease of description, the dimensions of the various parts shown in the accompanying drawings are not drawn according to actual scale.

[0077] The following description of at least one exemplary embodiment is merely illustrative and is not intended to limit the scope of this application or its application or use.

[0078] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0079] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0080] Furthermore, the technical solutions of the various embodiments of this application can be combined with each other, but only if they are based on the ability of those skilled in the art to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such combination of technical solutions does not exist and is not within the scope of protection claimed by this application.

[0081] It should be noted that all directional indicators (such as up, down, left, right, front, back, etc.) in the embodiments of this application are only used to explain the relative positional relationship and movement of each component in a certain specific posture (as shown in the figure). If the specific posture changes, the directional indicator will also change accordingly.

[0082] The specific embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0083] Example 1:

[0084] A flowchart illustrating the algorithm-assisted closed-loop bidirectional multi-model data annotation method provided by this invention is shown below. Figure 1 As shown, it includes:

[0085] Step 1: Initialize the bidirectional multi-model annotation workflow;

[0086] Step 2: Based on the workflow, annotate the image data to be annotated to generate annotated data;

[0087] Step 3: Generate a dataset based on the labeled data and workflow;

[0088] The annotations include one or more of the following:

[0089] Two-way single-model annotation, two-way multi-model simultaneous calibration annotation, and multi-objective two-way multi-model data calibration annotation.

[0090] Step 1 specifically includes: selecting an auxiliary algorithm (i.e., an auxiliary model) for annotation from a preset algorithm model library; setting a confidence threshold for the auxiliary algorithm and setting the initial target location coordinates; setting the dataset type, format, and dataset generation configuration parameters;

[0091] The dataset types include at least one or more of the following: object detection datasets and object tracking datasets; the auxiliary algorithms can be algorithms already integrated in the algorithm model library server, or support custom algorithms, including at least one or more of the following: initial target box recommendation algorithm, object detection algorithm, target box automatic correction and annotation algorithm, and single target tracking algorithm; the dataset generation configuration parameters include at least one or more of the following: similarity, time interval, overlap rate, center point offset, and total number of frames.

[0092] Step 1 also includes an image display module, which is the corresponding bidirectional multi-model annotation workflow strategy, and the number of image display modules is the number of targets to be annotated simultaneously.

[0093] Between step 1 and step 2, there is also the following: acquisition of the forward video stream and reverse reading of image data, wherein the reverse order of image data is the reverse of the labeled images;

[0094] Specifically, the forward video stream source is the video file to be labeled or the image sequence, and the reverse image data refers to the images whose target positions have been labeled with large errors.

[0095] Furthermore, the frame rate of the forward and reverse image data can be controlled by parameters.

[0096] The annotation in step 2 includes at least one or more of the following: bidirectional single-model annotation, bidirectional multi-model simultaneous calibration annotation, and multi-objective bidirectional multi-model data calibration annotation. Among them, multi-objective bidirectional multi-model data calibration annotation is bidirectional single-model annotation of multiple different target optimization algorithm models, and the annotation result is the annotation information of multiple targets.

[0097] During the annotation process, the target location information and category are identified by using a deep learning target tracking algorithm. This target tracking algorithm is a set of single target tracking algorithms based on convolutional neural networks, and can optionally integrate mainstream anchor-based and anchor-free single target tracking algorithms.

[0098] During the annotation process, deep learning object detection algorithms can be used to recommend the initial location information and category of the target to be annotated.

[0099] Furthermore, referring to Figure 2 As shown, the bidirectional single-model annotation strategy is as follows:

[0100] Continuous annotation of forward images, automatically calibrating the target location and storing the annotation results for the input image to be annotated;

[0101] When the visualization positioning error is large (i.e., the error is greater than or equal to the confidence threshold), the continuous annotation of the forward image is stopped;

[0102] The target coordinates are reinitialized, and the target coordinates are corrected by labeling them in reverse order. The reinitialized target coordinates can be the coordinates obtained based on the target detection algorithm model, or the target position coordinates selected manually.

[0103] Repeat the above process until the preset termination condition is met, at which point the target image annotation will terminate.

[0104] Furthermore, referring to Figure 3 As shown, the specific bidirectional multi-model simultaneous calibration annotation strategy is as follows:

[0105] Based on multiple selected auxiliary algorithms, continuous forward image annotation is performed, and the target position of the input image to be annotated is automatically calibrated and the annotation results are stored.

[0106] If the visualization error of some model location is large (i.e. the error is greater than or equal to the confidence threshold), or if the warning evaluation index exceeds the corresponding threshold, the positive annotation will be stopped.

[0107] The accurate location-based auxiliary algorithm corrects other auxiliary algorithms and continues positive labeling;

[0108] If the location visualization error of all auxiliary algorithms is large (i.e., the error is greater than or equal to the confidence threshold), or if the warning evaluation index exceeds the corresponding threshold, the positive annotation shall be stopped.

[0109] The target coordinates are reinitialized, and the target coordinates are corrected by labeling them in reverse order. The reinitialized target coordinates can be the coordinates obtained based on the target detection algorithm model, or the target position coordinates selected manually.

[0110] Repeat the above process until image annotation is terminated.

[0111] The early warning evaluation indicators are the overlap score (OS) and / or the location error threshold (LET). The overlap rate between the bounding box (denoted as a) obtained by auxiliary algorithm A and the box (denoted as b) obtained by auxiliary algorithm B is defined as: OS = |a∩b| / |a∪b|, where |·| represents the number of pixels in the region, and the location error is defined as: LET = distance(a,b).

[0112] The annotation result of bidirectional multi-model simultaneous calibration annotation is the weighted value of the annotation positions of multiple models, denoted as box_final=k1*a+k2*b, where k1 and k2 are the weights.

[0113] Furthermore, the multi-objective, bidirectional, multi-model data correction and annotation strategy performs the following for each objective: Figure 2 The bidirectional single-model annotation strategy is shown.

[0114] Step 3 specifically includes: generating an object detection dataset and / or an object tracking dataset based on the set dataset type, format, and dataset generation configuration parameters;

[0115] The target detection dataset is created based on the least relevant target detection dataset for the same target selected by the algorithm. The algorithm is a similarity matching algorithm, which can be an unsupervised machine learning algorithm or a deep learning algorithm model.

[0116] The target tracking dataset is created based on custom target tracking datasets with time intervals, overlap rates, center point deviations, and total number of frames.

[0117] Following step 3, the process also includes: closed-loop labeling, specifically including:

[0118] Based on the object detection dataset, script training and / or hyperparameter configuration of the object detection algorithm are performed to train the detection dataset and generate the object detection algorithm model.

[0119] Based on the target tracking dataset, script training and / or hyperparameter configuration of the single target tracking algorithm are performed to train the tracking dataset and generate a single target tracking algorithm model.

[0120] This invention accurately locates the target pixel coordinates by performing bidirectional single-model annotation, bidirectional multi-model simultaneous correction annotation, and / or multi-target bidirectional multi-model data correction annotation on the image data to be annotated, which greatly saves the cost of manual annotation and improves the speed of data annotation.

[0121] This invention also communicates the following: based on an object detection dataset, script training and / or hyperparameter configuration of an object detection algorithm are performed; the detection dataset is trained to generate an object detection algorithm model; based on an object tracking dataset, script training and / or hyperparameter configuration of a single object tracking algorithm are performed; the tracking dataset is trained to generate a single object tracking algorithm model; and online closed-loop iterative updates of the object detection algorithm and / or object tracking algorithm model parameters are achieved.

[0122] Example 2:

[0123] The following is a specific example of an algorithmic closed-loop assisted bidirectional multi-model data annotation method. This example performs bidirectional single-model data annotation, including:

[0124] Step A1: Set up the workflow for bidirectional single-model data annotation software, and set up one image visualization window accordingly;

[0125] Further, set the working paths for the video and image sequences to be labeled;

[0126] Furthermore, the software-integrated single-target tracking algorithm daSiamRPN model was selected, with a tracking confidence level set to 0.8. The algorithm model was obtained from the model algorithm library server.

[0127] Furthermore, the software-integrated object detection algorithm YOLOv4 model was selected, with a detection confidence level set to 0.5. The algorithm model was obtained from the model algorithm library server.

[0128] Furthermore, the target tracking and target detection datasets are saved using the Pascal-VOC label format;

[0129] Furthermore, the annotation software is initialized and run.

[0130] Step A2: Refer to Figure 2 Following the established bidirectional single-model annotation strategy, image sequence annotation is completed;

[0131] Furthermore, target initial coordinates are selected, which are manually fine-tuned after being recommended by the target detection algorithm;

[0132] Furthermore, the algorithm automatically tracks and labels, and stops continuous labeling of the forward image when the positioning error of the automatic label is large;

[0133] Furthermore, the target coordinates are reinitialized, and the target coordinates are corrected by continuously labeling them in reverse order;

[0134] Repeat the above process until the target image annotation is terminated.

[0135] Step A3: Dataset generation, including object detection and object tracking datasets;

[0136] The software-integrated similarity matching KNN algorithm was selected to extract the target detection dataset, and 60 labeled results were extracted for the same target.

[0137] The time interval is set to be ≥30ms, the overlap rate is ≥0.2, the OS is ≥0.1, the tracking sequence is 1500 frames, and the expected number of consecutive target sequences is 10.

[0138] Furthermore, the labeled data is transmitted to the model library algorithm server.

[0139] Step A4: Label the closed loop, train in real time, and update the model to the model library.

[0140] Furthermore, the labeled data is split into training and validation datasets in a 5:5 ratio and transmitted to the server training dataset;

[0141] Furthermore, when the overlap rate OS is set to be greater than 0.95, the tracking accuracy reaches 95% and the detection accuracy reaches 95%, which is the recommended standard of the model.

[0142] Furthermore, model training in the model library algorithm server triggers the annotation software to update the model;

[0143] Step A5: Repeat the above process until the annotation is complete.

[0144] Example 3:

[0145] The following is a specific example of a closed-loop algorithm-assisted bidirectional multi-model data annotation method. This example performs simultaneous bidirectional multi-model calibration, including:

[0146] Step B1: Set up the workflow for bidirectional multi-model simultaneous calibration and annotation software, and set up one image visualization window accordingly;

[0147] Further, set the working paths for the video and image sequences to be labeled;

[0148] Furthermore, the software-integrated single-target tracking algorithms daSiamRPN and siamfcplusplus were selected, with a tracking confidence level set to 0.8. The algorithm models were obtained from the model algorithm library server.

[0149] Furthermore, the software-integrated single-target detection algorithm YOLOv4 model was selected, with a detection confidence level set to 0.5. The algorithm model was obtained from the model algorithm library server.

[0150] Furthermore, the target tracking and target detection datasets are saved using the Pascal-VOC label format;

[0151] Furthermore, the annotation software initializes and runs;

[0152] Step B2: Refer to Figure 3 Following the established bidirectional multi-model annotation strategy (i.e., bidirectional multi-model simultaneous correction annotation), image sequence annotation is completed.

[0153] Furthermore, target initial coordinates are selected, which are manually fine-tuned after being recommended by the target detection algorithm;

[0154] Furthermore, the algorithm automatically tracks and labels, and stops forward labeling when the location of some models is significantly affected or the visualization error is large; and,

[0155] Models with accurate positioning are corrected, while other models continue to be labeled positively.

[0156] Optionally, forward annotation can be stopped when all model location warnings are large or visualization errors are large.

[0157] Reinitialize the target coordinates and correct them by labeling them in reverse order.

[0158] Repeat the above process until image annotation is terminated;

[0159] Step B3: Dataset generation, including object detection and object tracking datasets;

[0160] The software-integrated similarity matching KNN algorithm was selected to extract the target detection dataset, with 60 results extracted for the same target.

[0161] The time interval is set to be ≥30ms, the overlap rate is ≥0.2, the OS is ≥0.1, the tracking sequence is 1500 frames, and the expected number of consecutive target sequences is 10.

[0162] Furthermore, the labeled data is transmitted to the model library algorithm server.

[0163] Step B4: Label the closed loop, train in real time, and update the model to the model library.

[0164] Furthermore, the labeled data is split into training and validation datasets in a 5:5 ratio and transmitted to the server training dataset;

[0165] Furthermore, when the overlap rate OS is set to be greater than 0.95, the tracking accuracy reaches 95% and the detection accuracy reaches 95%.

[0166] Furthermore, model updates in the model library algorithm server trigger bidirectional multi-model annotation software to update the model;

[0167] Step B5: Repeat the above process until the annotation is complete.

[0168] Example 4:

[0169] The following is a specific example of an algorithmic closed-loop assisted bidirectional multi-model data annotation method. This example performs multi-objective bidirectional multi-model data correction and annotation, including:

[0170] Step C1: Set up the workflow for the bidirectional four-model data annotation software, and set up four image visualization windows accordingly;

[0171] Further, set the working paths for the video and image sequences to be labeled;

[0172] Furthermore, software-integrated single-target tracking algorithms daSiamRPN, daSiamRPN, siamfcplusplus, and Ocean models were selected for the four targets, with the tracking confidence level set to 0.8. The algorithm models were obtained from the model algorithm library server.

[0173] Furthermore, the software-integrated single-object detection algorithm YOLOv4 model was selected, with a detection confidence level set to 0.5. The algorithm model was obtained from the model algorithm library server.

[0174] Furthermore, the target tracking and target detection datasets are saved using the Pascal-VOC label format;

[0175] Furthermore, the annotation software initializes and runs;

[0176] Step C2: Refer to Figure 2 Following the established bidirectional single-model annotation strategy, image sequence annotation is completed;

[0177] Specifically, a bidirectional single-model annotation strategy is executed separately for each target;

[0178] Furthermore, the target initial coordinates and annotation results are selected and stored, where the target initial coordinates are manually fine-tuned after being recommended by the target detection algorithm; and,

[0179] Furthermore, the algorithm automatically tracks and labels the image; if the positioning error of the labeling position is large, the continuous labeling of the forward image will be stopped.

[0180] Furthermore, the target coordinates are reinitialized, and the target coordinates are corrected by continuously labeling them in reverse order;

[0181] Repeat the above process until the target image annotation is terminated.

[0182] Step C3: Dataset generation, creation of object detection and object tracking datasets;

[0183] The software-integrated similarity matching KNN algorithm was selected to extract the target detection dataset, and 60 labeled results were extracted for the same target.

[0184] Furthermore, the time interval is set to be ≥30ms, the overlap rate is ≥0.2, the tracking sequence is 1500 frames, and the expected number of consecutive target sequences is 10.

[0185] Furthermore, the labeled data is transmitted to the model library algorithm server.

[0186] Step C4: Label the closed loop, train in real time, and update the model to the model library.

[0187] Furthermore, the labeled data is split into training and validation datasets in a 5:5 ratio and transmitted to the server training dataset;

[0188] Furthermore, when the overlap rate is set to be greater than 0.95, the tracking accuracy reaches 95% and the detection accuracy reaches 95%.

[0189] Furthermore, model updates in the model library algorithm server trigger bidirectional multi-model annotation software to update the model;

[0190] Step C5: Repeat the above process until the annotation is completed.

[0191] Example 5:

[0192] Based on the same inventive concept, this invention also provides an apparatus for algorithmic closed-loop assisted bidirectional multi-model data annotation, referring to... Figure 4 ,include:

[0193] Human-computer interaction annotation interface, annotation module, and annotation data generation module;

[0194] The human-computer interaction annotation interface is used to initialize the bidirectional multi-model annotation workflow;

[0195] The annotation module is used to annotate the image data to be annotated based on the workflow and generate annotated data;

[0196] The annotation data generation module is used to generate datasets based on annotation data and workflow.

[0197] The annotations include one or more of the following:

[0198] Two-way single-model annotation, two-way multi-model simultaneous calibration annotation, and multi-objective two-way multi-model data calibration annotation.

[0199] The human-computer interaction annotation interface is specifically used for:

[0200] Select an auxiliary algorithm (i.e., an auxiliary model) from the preset algorithm model library for annotation; set the confidence threshold for the auxiliary algorithm and set the initial target location coordinates; set the dataset type, format, and dataset generation configuration parameters;

[0201] The dataset types include at least one or more of the following: object detection datasets and object tracking datasets; the auxiliary algorithms can be algorithms already integrated in the algorithm model library server, or support custom algorithms, including at least one or more of the following: initial target box recommendation algorithm, object detection algorithm, target box automatic correction and annotation algorithm, and single target tracking algorithm; the dataset generation configuration parameters include at least one or more of the following: similarity, time interval, overlap rate, center point offset, and total number of frames.

[0202] The human-computer interaction annotation interface is also used to display image display modules. Each image display module corresponds to annotation workflow strategies, and the number of image display modules corresponds to the number of targets being annotated simultaneously.

[0203] The annotation in the annotation module includes at least one or more of the following: bidirectional single-model annotation, bidirectional multi-model simultaneous calibration annotation, and multi-objective bidirectional multi-model data calibration annotation. Among them, multi-objective bidirectional multi-model data calibration annotation is bidirectional single-model annotation of multiple different target optimization algorithm models, and the annotation result is the annotation information of multiple targets.

[0204] During the annotation process, the target location information and category are identified by using a deep learning target tracking algorithm. This target tracking algorithm is a set of single target tracking algorithms based on convolutional neural networks, and can optionally integrate mainstream anchor-based and anchor-free single target tracking algorithms.

[0205] During the annotation process, deep learning object detection algorithms can be used to recommend the initial location information and category of the target to be annotated.

[0206] The bidirectional single-model annotation strategy in the annotation module is as follows: Figure 2 As shown, specifically:

[0207] Continuous annotation of forward images, automatically calibrating the target location and storing the annotation results for the input image to be annotated;

[0208] When the visualization positioning error is large (i.e., the error is greater than or equal to the confidence threshold), the continuous annotation of the forward image is stopped.

[0209] Reinitialize the target coordinates, reverse the order and continuously label the corrected target coordinates, and reinitialize the target coordinates. The target coordinates can be the coordinates obtained based on the target detection algorithm model, or the target position coordinates selected manually.

[0210] Repeat the above process until the preset termination condition is met, at which point the target image annotation will terminate.

[0211] The bidirectional multi-model simultaneous calibration annotation strategy in the annotation module is as follows: Figure 3 As shown, specifically:

[0212] Based on multiple selected auxiliary algorithms, continuous forward image annotation is performed, and the target position of the input image to be annotated is automatically calibrated and the annotation results are stored.

[0213] If the visualization error of some model location is large (i.e. the error is greater than or equal to the confidence threshold), or if the warning evaluation index exceeds the corresponding threshold, the positive annotation will be stopped.

[0214] The accurate location-based auxiliary algorithm corrects other auxiliary algorithms and continues positive labeling;

[0215] If the location visualization error of all auxiliary algorithms is large (i.e., the error is greater than or equal to the confidence threshold), or if the warning evaluation index exceeds the corresponding threshold, the positive annotation shall be stopped.

[0216] Reinitialize the target coordinates, reverse the order and continuously label the corrected target coordinates, and reinitialize the target coordinates. The target coordinates can be the coordinates obtained based on the target detection algorithm model, or the target position coordinates selected manually.

[0217] Repeat the above process until image annotation is terminated.

[0218] The early warning evaluation indicators are the overlap score (OS) and / or the location error threshold (LET). The overlap rate between the bounding box (denoted as a) obtained by auxiliary algorithm A and the box (denoted as b) obtained by auxiliary algorithm B is defined as: OS = |a∩b| / |a∪b|, where |·| represents the number of pixels in the region, and the location error is defined as: LET = distance(a,b).

[0219] The annotation result of bidirectional multi-model simultaneous calibration annotation is the weighted value of the annotation positions of multiple models, denoted as box_final=k1*a+k2*b, where k1 and k2 are the weights.

[0220] Furthermore, the multi-objective, bidirectional, multi-model data correction and annotation strategy performs the following for each objective: Figure 2 The bidirectional single-model annotation strategy is shown.

[0221] Specifically, the annotation module is used for:

[0222] Generate object detection datasets and / or object tracking datasets based on the set dataset type, format, and dataset generation configuration parameters;

[0223] The object detection dataset is created based on the algorithm-selected dataset of the least relevant objects for the same target. The algorithm is a similarity matching algorithm, which can be an unsupervised machine learning algorithm or a deep learning algorithm model.

[0224] The target tracking dataset is created based on custom target tracking datasets with time intervals, overlap rates, center point deviations, and total number of frames.

[0225] The device for algorithm-closed-loop assisted bidirectional multi-model data annotation also includes an image data stream module, which is used for acquiring forward video streams and reading image data in reverse order. Reversing the image data order is the reverse of the already annotated images.

[0226] Specifically, the forward video stream source can be either a video file to be labeled or an image sequence, while the reverse image data refers to images whose target positions have been labeled but have large calibration errors.

[0227] Furthermore, the frame rate of the forward and reverse image data can be controlled by parameters.

[0228] The device for algorithm closed-loop assisted bidirectional multi-model data annotation also includes an online model training module, which is used to train the target detection algorithm script and / or configure hyperparameters based on the target detection dataset, train the detection dataset, and generate the target detection algorithm model.

[0229] Based on a target tracking dataset, script training and / or hyperparameter configuration are performed for a single-target tracking algorithm. The tracking dataset is then trained to generate a single-target tracking algorithm model. A specific example structure of a device containing all modules for algorithm closed-loop assisted bidirectional multi-model data annotation is shown below. Figure 5 As shown.

[0230] Example 6:

[0231] Below is an example of a device for algorithmic closed-loop assisted bidirectional multi-model data labeling, which is as follows: Figure 6 As shown, it includes:

[0232] The bidirectional multi-model annotation subsystem is used to automatically correct and annotate images to be annotated, store them, generate target tracking datasets and target detection datasets, and provide training data to the algorithm model library server.

[0233] Algorithm model library server, used to assist in the configuration of algorithm training scripts / hyperparameters, online model training and evaluation, and trigger updates of labeled algorithm models to achieve a closed-loop labeling process;

[0234] The bidirectional multi-model annotation subsystem integrates, for example, Figure 5 The functional modules of the device for algorithm-closed-loop assisted bidirectional multi-model data annotation shown will not be described in detail.

[0235] Algorithm model library server such as Figure 7 As shown, it specifically includes:

[0236] Dataset partitioning is used to divide the generated labeled data into training and validation sets.

[0237] Training scripts / hyperparameter configurations: The training scripts correspond to the labeled algorithm models and can be either integrated algorithm scripts or user-defined training scripts.

[0238] The model recommendation evaluation uses the overlap rate score (OS) as the performance metric.

[0239] Algorithm model library, an integrated model library to assist in the annotation of target detection algorithms and target tracking algorithms.

[0240] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0241] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0242] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0243] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0244] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit its scope of protection. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that after reading the present invention, they can still make various changes, modifications or equivalent substitutions to the specific implementation methods of the application, but these changes, modifications or equivalent substitutions are all within the scope of protection of the claims pending approval.

Claims

1. A method for algorithmic closed-loop assisted bidirectional multi-model data annotation, characterized in that, include: Initialize the bidirectional multi-model annotation workflow; Based on the workflow, the image data to be labeled is labeled to generate labeled data; Generate a dataset based on the labeled data and workflow; The annotation includes one or more of the following: Two-way single-model annotation, two-way multi-model simultaneous calibration annotation, and multi-objective two-way multi-model data calibration annotation; The initialization of the bidirectional multi-model annotation workflow includes: Select an auxiliary algorithm for annotation from the preset algorithm model library; Set a confidence threshold for the auxiliary algorithm and set the initial target position coordinates; Configure the dataset type, format, and dataset generation parameters; The dataset type includes at least one or more of the following: object detection dataset and object tracking dataset; the auxiliary algorithm includes at least one or more of the following: initial bounding box recommendation algorithm, object detection algorithm, automatic bounding box correction and annotation algorithm, and single object tracking algorithm; the dataset generation configuration parameters include at least one or more of the following: similarity, time interval, overlap rate, center point offset, and total number of frames. Based on the aforementioned workflow, bidirectional multi-model simultaneous calibration annotation is performed on the image data to be annotated to generate annotated data, including: Based on the selected auxiliary algorithms, forward continuous image annotation is performed on the image data to be annotated, and the visualization localization results corresponding to each auxiliary algorithm of the target location in the image data to be annotated are obtained. If the error of the visualization positioning results corresponding to all auxiliary algorithms is less than the confidence threshold, then the labeled data is generated and the process ends. If there is an auxiliary algorithm whose visualization positioning result error is less than the confidence threshold and an auxiliary algorithm whose visualization positioning result error is greater than or equal to the confidence threshold, then the continuous annotation of the forward image is stopped. The auxiliary algorithm whose visualization positioning result error is less than the confidence threshold is used to correct the auxiliary algorithm whose error is greater than or equal to the confidence threshold. Then, the continuous annotation of the forward image data to be annotated is continued using each auxiliary algorithm until the error of the visualization positioning result corresponding to all auxiliary algorithms is less than the confidence threshold. Then, the annotation data is generated and the process ends. If the error of the visualization positioning result corresponding to all auxiliary algorithms is greater than or equal to the confidence threshold, the forward image continuous annotation is stopped, and the target position coordinates are reinitialized. Based on the reinitialized target position coordinates, the selected auxiliary algorithms are used to perform reverse continuous annotation correction on the visualization positioning result of the target position until the preset termination condition is met, and the annotation data is generated and the process ends.

2. The method as described in claim 1, characterized in that, Based on the aforementioned workflow, bidirectional single-model annotation is performed on the image data to be annotated to generate annotated data, including: Based on a selected auxiliary algorithm, forward continuous image annotation is performed on the image data to be annotated to obtain the visual location result of the target position in the image data to be annotated; If the error of the visualization positioning result is less than the confidence threshold, then the annotation data is generated and the process ends; otherwise, the continuous annotation of the forward image is stopped and the target position coordinates are reinitialized. Based on the re-initialized target location coordinates, the visualized positioning results of the target location are continuously annotated and corrected in reverse order until a preset termination condition is met and annotation data is generated.

3. The method as described in claim 2, characterized in that, Based on the aforementioned workflow, multi-objective, bidirectional, multi-model data correction and annotation are performed on the image data to be annotated to generate labeled data, including: Based on the auxiliary algorithm corresponding to each target in the labeled image data, bidirectional single-model annotation is performed on each target in the image data to be annotated to generate labeled data.

4. The method as described in claim 1, characterized in that, The step of generating a dataset based on the labeled data and workflow includes: Based on the dataset type set in the workflow, select the target detection dataset and / or target tracking dataset that need to be generated; When it is necessary to generate an object detection dataset, the labeled data with the lowest similarity to the same object are extracted based on the similarity algorithm to create the object detection dataset; When it is necessary to generate a target tracking dataset, continuous images are extracted from the labeled data to form the target tracking dataset according to the set time interval, overlap rate, center point deviation and total number of frames.

5. The method as described in claim 1, characterized in that, Following the generation of the dataset, the following is also included: Based on the object detection dataset, script training and / or hyperparameter configuration of the object detection algorithm are performed to train the detection dataset and generate the object detection algorithm model. Based on the target tracking dataset, script training and / or hyperparameter configuration of the single target tracking algorithm are performed to train the tracking dataset and generate a single target tracking algorithm model.

6. A device for algorithm-closed-loop assisted bidirectional multi-model data annotation, characterized in that, include: Human-computer interaction annotation interface, annotation module, and annotation data generation module; The human-computer interaction annotation interface is used to initialize the bidirectional multi-model annotation workflow; The annotation module is used to annotate the image data to be annotated based on the workflow and generate annotation data; The labeled data generation module is used to generate a dataset based on the labeled data and workflow; The annotation includes one or more of the following: Two-way single-model annotation, two-way multi-model simultaneous calibration annotation, and multi-objective two-way multi-model data calibration annotation; The human-computer interaction annotation interface is specifically used for: Select an auxiliary algorithm for annotation from the preset algorithm model library; Set a confidence threshold for the auxiliary algorithm and set the initial target position coordinates; Configure the dataset type, format, and dataset generation parameters; The dataset type includes at least one or more of the following: object detection dataset and object tracking dataset; the auxiliary algorithm includes at least one or more of the following: initial bounding box recommendation algorithm, object detection algorithm, automatic bounding box correction and annotation algorithm, and single object tracking algorithm; the dataset generation configuration parameters include at least one or more of the following: similarity, time interval, overlap rate, center point offset, and total number of frames. Based on the workflow, the annotation module performs bidirectional multi-model simultaneous correction annotation on the image data to be annotated, generating annotated data, including: Based on the selected auxiliary algorithms, forward continuous image annotation is performed on the image data to be annotated, and the visualization localization results corresponding to each auxiliary algorithm of the target location in the image data to be annotated are obtained. If the error of the visualization positioning results corresponding to all auxiliary algorithms is less than the confidence threshold, then the labeled data is generated and the process ends. If there is an auxiliary algorithm whose visualization positioning result error is less than the confidence threshold and an auxiliary algorithm whose visualization positioning result error is greater than or equal to the confidence threshold, then the continuous annotation of the forward image is stopped. The auxiliary algorithm whose visualization positioning result error is less than the confidence threshold is used to correct the auxiliary algorithm whose error is greater than or equal to the confidence threshold. Then, the continuous annotation of the forward image data to be annotated is continued using each auxiliary algorithm until the error of the visualization positioning result corresponding to all auxiliary algorithms is less than the confidence threshold. Then, the annotation data is generated and the process ends. If the error of the visualization positioning result corresponding to all auxiliary algorithms is greater than or equal to the confidence threshold, the forward image continuous annotation is stopped, and the target position coordinates are reinitialized. Based on the reinitialized target position coordinates, the selected auxiliary algorithms are used to perform reverse continuous annotation correction on the visualization positioning result of the target position until the preset termination condition is met, and the annotation data is generated and the process ends.

7. The apparatus as claimed in claim 6, characterized in that, The annotation module, based on the workflow, performs bidirectional single-model annotation on the image data to be annotated, generating annotated data, including: Based on a selected auxiliary algorithm, forward continuous image annotation is performed on the image data to be annotated to obtain the visual location result of the target position in the image data to be annotated; If the error of the visualization positioning result is less than the confidence threshold, then the annotation data is generated and the process ends; otherwise, the continuous annotation of the forward image is stopped and the target position coordinates are reinitialized. Based on the re-initialized target location coordinates, the visualized positioning results of the target location are continuously annotated and corrected in reverse order until a preset termination condition is met and annotation data is generated.

8. The apparatus as claimed in claim 6, characterized in that, The annotation module, based on the workflow, performs multi-objective, bidirectional, multi-model data correction annotation on the image data to be annotated, generating annotated data, including: Based on the auxiliary algorithm corresponding to each target in the labeled image data, bidirectional single-model annotation is performed on each target in the image data to be annotated to generate labeled data.

9. The apparatus as claimed in claim 6, characterized in that, The labeled data generation module is specifically used for: Based on the dataset type set in the workflow, select the target detection dataset and / or target tracking dataset that need to be generated; When it is necessary to generate an object detection dataset, the labeled data with the lowest similarity to the same object are extracted based on the similarity algorithm to create the object detection dataset; When it is necessary to generate a target tracking dataset, continuous images are extracted from the labeled data to form the target tracking dataset according to the set time interval, overlap rate, center point deviation and total number of frames.

10. The apparatus as claimed in claim 6, characterized in that, It also includes an online model training module; the online model training module is specifically used for: Based on the object detection dataset, script training and / or hyperparameter configuration of the object detection algorithm are performed to train the detection dataset and generate the object detection algorithm model. Based on the target tracking dataset, script training and / or hyperparameter configuration of the single target tracking algorithm are performed to train the tracking dataset and generate a single target tracking algorithm model.

Images