Target labeling method and device based on inter-frame association
Through the object detection module of inter-frame association and multi-dimensional feature fusion, the problem of inter-frame association information being ignored in the prior art is solved, the labeling accuracy is improved and the cost is reduced, and efficient labeling is achieved under low computing power.
Patent Information
- Application Number
- CN202411254867.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-09
- Publication Date
- 2025-01-10
AI Technical Summary
The existing automatic labeling method only labels single-frame images, ignoring the correlation information between video frames, resulting in low labeling accuracy and high computing power demand leading to higher costs.
Using the target annotation method based on inter-frame association, the object detection module of spatial pyramid pooling and multi-dimensional feature fusion is constructed to extract and annotate continuous video data, and to associate inter-frame object association using the predicted set and associated trajectory, based on confidence Divide the labeling box and optimize the low-computing feature extraction network.
It improves the labeling accuracy and the ability to extract small target features, reduces the missed scale rate, reduces the computing power requirement, does not rely on high-performance hardware platforms, and realizes the engineering and convenience of labeling tasks.
Smart Images
Figure CN119295988A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of image processing technology, and in particular to a target labeling method and device based on inter-frame association. Background Art
[0002] With the rapid development of artificial intelligence, high-quality labeled data has become a precious resource, but a large amount of image data has not been effectively utilized due to lack of labeling, resulting in great waste. Existing labeling methods often use manual labeling methods. Massive data requires a lot of human resources and non-professionals cannot complete the task accurately and efficiently. Automatic labeling methods can greatly improve labeling efficiency and reduce labor costs.
[0003] Existing automatic annotation methods only annotate single-frame images, while the data to be annotated is mostly continuous image sequences of captured videos. Single-frame annotation will ignore the correlation information between frames, and the accuracy needs to be improved. In addition, existing annotation methods often use high-computing detection modules, which require high computing power on the hardware platform. High-computing platforms mean higher costs and poor economic efficiency. Summary of the invention
[0004] In view of this, an embodiment of the present application provides a method and device for target labeling based on inter-frame association, which at least partially solves the problem in the prior art that the single-frame labeling method ignores the association information between frames, resulting in low accuracy.
[0005] In a first aspect, an embodiment of the present application provides a target labeling method based on inter-frame association, including:
[0006] Construct an object detection module based on spatial pyramid pooling and multi-dimensional feature fusion;
[0007] Extract frames from continuous video data to obtain a set of images to be labeled;
[0008] Inputting the set of images to be annotated into the target detection module for target annotating, and obtaining a set of annotations for each frame of the image, wherein the set of annotations includes the coordinates, category and confidence of the annotation result of each target annotation box;
[0009] Based on the annotation result confidence, the annotation set of each frame of image is divided into a high confidence set and a low confidence set according to a preset annotation result confidence;
[0010] Initialize the annotation set corresponding to the first frame image as the associated track, predict the coordinates and size of each target annotation box of the associated track in the next frame image, and obtain a predicted set;
[0011] Based on the prediction set, the high confidence set of the next frame image is associated with the associated trajectory with the same target to obtain an associated trajectory, an associated high confidence annotation box, an unassociated trajectory and an unassociated high confidence annotation box, and the associated trajectory is updated using the associated high confidence annotation box;
[0012] Associating the unassociated track with the low confidence set of the next frame image to the same target, obtaining an associated track, an associated low confidence annotation box, an unassociated track and an unassociated low confidence annotation box, and updating the associated track using the associated low confidence annotation box;
[0013] Initializing the unassociated high-confidence annotation box as the associated track for associating the next frame of unassociated images;
[0014] Continue to associate each frame of unassociated images to obtain the labeling results.
[0015] According to a specific implementation method of an embodiment of the present application, the target detection module adopts the low-computing power network architecture Darknet53 as the feature extraction network.
[0016] According to a specific implementation of an embodiment of the present application, the construction of an object detection module based on spatial pyramid pooling and multi-dimensional feature fusion includes:
[0017] A downsampling factor is set for the shallow features of the feature extraction network, downsampling is performed through a 1×1 convolution kernel, and the shallow features are fused with the deep features to achieve the multi-dimensional feature fusion.
[0018] According to a specific implementation of the embodiment of the present application, the downsampling factor is set to 2.
[0019] According to a specific implementation method of an embodiment of the present application, the CIOU loss function is used for target labeling in the target detection module.
[0020] According to a specific implementation method of an embodiment of the present application, the calculation formula of the CIOU loss function is:
[0021]
[0022] Among them, IOU is the intersection-over-union ratio between the predicted box and the real box, ρ 2 (b,b gt ) is the Euclidean distance between the center point of the predicted box and the center point of the real box, c is the diagonal distance of the minimum closed area that can contain both the predicted box and the real box, α and v are correction factors, and the calculation formulas of α and v are:
[0023]
[0024] Among them, w gt is the width of the real frame, h gt is the height of the real box, w is the width of the predicted box, and h is the height of the predicted box.
[0025] According to a specific implementation of the embodiment of the present application, based on the annotation result confidence, the annotation set of each frame image is divided into a high confidence set and a low confidence set according to a preset annotation result confidence, including:
[0026] Setting a first preset annotation result confidence level and a second preset annotation result confidence level, wherein the second preset annotation result confidence level is greater than the first preset annotation result confidence level;
[0027] When the annotation result confidence is greater than the second preset annotation result confidence, the target annotation box corresponding to the annotation result confidence is divided into the high confidence set;
[0028] When the annotation result confidence is between the first preset annotation result confidence and the second preset annotation result confidence, the target annotation box corresponding to the annotation result confidence is divided into the low confidence set;
[0029] When the annotation result confidence is less than the first preset annotation result confidence, the target annotation box corresponding to the annotation result confidence is deleted.
[0030] According to a specific implementation of the embodiment of the present application, the associating the high confidence set of the next frame image with the associated trajectory with the same target based on the prediction set includes:
[0031] Calculate the IOU value between each target annotation box in the prediction set and each target annotation box in the high confidence set;
[0032] Based on the IOU value, construct a relationship loss matrix between the prediction set and the high confidence set;
[0033] According to the relationship loss matrix, the Hungarian algorithm is used to associate the high confidence set of the next frame image with the associated trajectory as the same target.
[0034] According to a specific implementation of the embodiment of the present application, the method further includes:
[0035] Recording the number of unassociated frames for each of the unassociated trajectories;
[0036] When the number of unassociated frames exceeds a preset threshold, the unassociated track is deleted.
[0037] In a second aspect, an embodiment of the present application further provides a target labeling device based on inter-frame association, which is used to implement the target labeling method based on inter-frame association of any embodiment of the first aspect, and the device includes:
[0038] A construction module is used to construct an object detection module based on spatial pyramid pooling and multi-dimensional feature fusion;
[0039] The module for acquiring a set of images to be labeled is used to extract frames from continuous video data to obtain a set of images to be labeled;
[0040] A labeling set acquisition module, used to input the image set to be labeled into the target detection module for target labeling, and obtain a labeling set for each frame of image, wherein the labeling set includes the coordinates, category and labeling result confidence of each target labeling box;
[0041] A division module, configured to divide the annotation set of each frame of image into a high confidence set and a low confidence set according to a preset annotation result confidence based on the annotation result confidence;
[0042] A prediction module, used to initialize the annotation set corresponding to the first frame image as an associated track, predict the coordinates and size of each target annotation box of the associated track in the next frame image, and obtain a prediction set;
[0043] A first associating module is used to associate the high confidence set of the next frame image with the associated trajectory with the same target based on the prediction set, obtain an associated trajectory, an associated high confidence annotation box, an unassociated trajectory and an unassociated high confidence annotation box, and update the associated trajectory using the associated high confidence annotation box;
[0044] A second associating module is used to associate the unassociated track with the low confidence set of the next frame image to the same target, obtain an associated track, an associated low confidence annotation box, an unassociated track and an unassociated low confidence annotation box, and update the associated track using the associated low confidence annotation box;
[0045] An initialization module, used for initializing the unassociated high-confidence annotation box as the associated track for associating the next frame of unassociated image;
[0046] The result acquisition module is used to continuously associate each frame of unassociated images to obtain the annotation results.
[0047] Beneficial effects:
[0048] The target labeling method and device based on inter-frame association in the embodiment of the present application optimizes the target detection module based on spatial pyramid pooling and multi-dimensional feature fusion, optimizes the low computing power requirement feature extraction network, improves the feature extraction ability of the feature extraction network for small targets and the adaptability to scale changes, and improves the labeling accuracy; by dividing the labeling boxes into high-confidence sets and low-confidence sets according to the confidence level, firstly associate the labeling boxes in the high-confidence set with the target trajectory, and update the unsuccessfully associated labeling boxes to new trajectories, and then associate the labeling boxes in the low-confidence set with the unmatched trajectory objects, and retain the successfully associated low-confidence labeling boxes, so as to retain all the labeling results as much as possible and reduce the missed label rate. The present application has low computing power requirements and does not rely on high-performance hardware platforms. Low-computing power computers can carry out labeling work, which is of great significance for realizing the engineering and convenience of labeling tasks. BRIEF DESCRIPTION OF THE DRAWINGS
[0049] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0050] Figure 1 is a flow chart of a target labeling method based on inter-frame association according to an embodiment of the present invention;
[0051] Figure 2 is a flowchart of associating inter-frame information according to an embodiment of the present invention;
[0052] Figure 3 A Darket53 network framework according to an embodiment of the present invention;
[0053] Figure 4 An improved Darket53 network framework according to an embodiment of the present invention;
[0054] Figure 5 A schematic diagram of CIOU calculation according to an embodiment of the present invention;
[0055] Figure 6 FIG. 4 is a schematic diagram of spatial pyramid pooling according to an embodiment of the present invention. DETAILED DESCRIPTION
[0056] The embodiments of the present application are described in detail below with reference to the accompanying drawings.
[0057] The following describes the implementation methods of the present application through specific examples, and those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The present application can also be implemented or applied through other different specific implementation methods, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present application. It should be noted that, in the absence of conflict, the following embodiments and the features in the embodiments can be combined with each other. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work belong to the scope of protection of the present application.
[0058] It should be noted that various aspects of the embodiments within the scope of the appended claims are described below. It should be apparent that the aspects described herein can be embodied in a wide variety of forms, and any specific structure and / or function described herein is merely illustrative. Based on the present application, it should be understood by those skilled in the art that an aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number of aspects described herein can be used to implement the device and / or practice the method. In addition, other structures and / or functionalities other than one or more of the aspects described herein can be used to implement this device and / or practice this method.
[0059] It should also be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present application. The illustrations only show the components related to the present application rather than being drawn according to the number, shape and size of the components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed at will, and the component layout may also be more complicated.
[0060] Additionally, in the following description, specific details are provided to facilitate a thorough understanding of the examples. However, it will be understood by those skilled in the art that the aspects described may be practiced without these specific details.
[0061] In the first aspect, the embodiment of the present application provides a target labeling method based on inter-frame association, as shown below: Figures 1 to 6 Describe in detail.
[0062] Reference Figure 1 The target labeling method based on inter-frame association of this embodiment includes:
[0063] Step 1: Construct an object detection module based on spatial pyramid pooling and multi-dimensional feature fusion;
[0064] Step 2: Extract frames from continuous video data to obtain a set of images to be labeled;
[0065] Step 3: input the image set to be annotated into the target detection module for target annotation, and obtain a annotation set for each frame of the image, wherein the annotation set includes the coordinates, category and annotation result confidence of each target annotation box;
[0066] Step 4: Based on the annotation result confidence, the annotation set of each frame of image is divided into a high confidence set and a low confidence set according to a preset annotation result confidence;
[0067] Step 5: Initialize the annotation set corresponding to the first frame image as the associated trajectory, predict the coordinates and size of each target annotation box of the associated trajectory in the next frame image, and obtain a predicted set;
[0068] Step 6: Based on the prediction set, the high confidence set of the next frame image is associated with the associated trajectory to obtain an associated trajectory, an associated high confidence annotation box, an unassociated trajectory and an unassociated high confidence annotation box, and the associated trajectory is updated using the associated high confidence annotation box;
[0069] Step 7: Associating the unassociated track with the low confidence set of the next frame image with the same target, obtaining an associated track, an associated low confidence annotation box, an unassociated track and an unassociated low confidence annotation box, and updating the associated track using the associated low confidence annotation box;
[0070] Step 8: Initialize the unassociated high-confidence annotation box as the associated track for associating the next frame of unassociated image;
[0071] Step 9: Continue to associate each frame of unassociated images to obtain the labeling results.
[0072] The target labeling method in this embodiment optimizes the target detection module based on spatial pyramid pooling and multi-dimensional feature fusion, optimizes the low computing power requirement feature extraction network, improves the feature extraction ability of the feature extraction network for small targets and its adaptability to scale changes, and improves the labeling accuracy; by dividing the labeling boxes into high-confidence sets and low-confidence sets according to the confidence level, firstly associate the labeling boxes in the high-confidence set with the target trajectory, and update the unsuccessfully associated labeling boxes to new trajectories, and then associate the labeling boxes in the low-confidence set with the unmatched trajectory objects, and retain the successfully associated low-confidence labeling boxes, so as to retain all the labeling results as much as possible and reduce the missed labeling rate. This application has low computing power requirements and does not rely on high-performance hardware platforms. Low-computing power computers can carry out labeling work, which is of great significance for realizing the engineering and convenience of labeling tasks.
[0073] In specific implementation, the method of the present application is realized through two modules, one is a target detection module, and the other is an association module. The target detection module is responsible for obtaining relevant information of the target of interest, and the association module is responsible for associating the same target between previous and next frames.
[0074] In one embodiment, the target detection module uses the low-computing network architecture Darknet53 as the feature extraction network, and its framework structure refers to Figure 3 .
[0075] In one embodiment, the construction of the target detection module based on spatial pyramid pooling and multi-dimensional feature fusion includes:
[0076] A downsampling factor is set for the shallow features of the feature extraction network, downsampling is performed through a 1×1 convolution kernel, and the shallow features are fused with the deep features to achieve the multi-dimensional feature fusion.
[0077] In one embodiment, the downsampling factor is set to 2.
[0078] When implementing it, refer to Figure 4 In order to ensure that the features of the original data to be labeled are not destroyed due to scale changes, the feature maps are migrated and fused in the feature extraction backbone network to achieve feature reuse and improve the ability of the convolutional neural network to extract features; with multiple convolutions, the deep features have more abstract expression capabilities. Based on the ability of deep features, this embodiment sets the downsampling factor of the shallow features to 2, downsamples them through a 1×1 convolution kernel, and then fuses them with the deep features; the method of adding scale migration and fusion to the feature layers with reference feature scales of 208×208, 104×104, 52×52, 13×13, etc. in the original feature extraction network, such as Figure 4 Shown is the improved Darknet structure.
[0079] For example, for shallow features, the downsampling factor of feature vectors such as 208×208, 104×104, 52×52 is set to 2, downsampled by a 1×1 convolution kernel, and then fused with 104×104, 52×52, 26×26, 13×13, thereby realizing the multi-dimensional feature fusion.
[0080] In one embodiment, the target detection module uses CIOU loss function for target labeling, and the calculation formula of the CIOU loss function is:
[0081]
[0082] Among them, IOU is the intersection-over-union ratio between the predicted box and the real box, ρ 2(b,b gt ) is the Euclidean distance d between the center of the predicted box and the center of the real box, c is the diagonal distance of the minimum closed area that can contain both the predicted box and the real box, refer to Figure 5 , α and v are correction factors, and the calculation formulas of α and v are:
[0083]
[0084] Among them, w gt is the width of the real frame, h gt is the height of the real box, w is the width of the predicted box, and h is the height of the predicted box.
[0085] In specific implementation, the target detection module uses the confidence of the bounding box to judge the effect of the prediction box and guide regression, that is, to calculate the IOU (Intersection over union) of the prediction box and the Ground Truth Box:
[0086]
[0087] The product of the confidence and the predicted target category probability for each cell is the score of the bounding box in each category:
[0088]
[0089] in, is the confidence score of the category, is the confidence score of the object in the prediction box, and Pr(Classi|objecl) is the confidence score of the correct prediction of the target category.
[0090] Since IOU is a concept of ratio, it is not sensitive to the size of the target object and cannot express the distance between the predicted box and the real box in the absence of overlap. In order to solve this problem, this embodiment uses CIOU for optimization. CIOU can better consider the shape information and position information of the target box, so that the detection module can comprehensively learn the characteristics of the target box, and then realize the constraints on the distance, overlap rate and penalty items between the predicted box and the real box, thereby improving the single-frame annotation accuracy.
[0091] Furthermore, since the fully connected layer of the target detection module requires a fixed number of nodes, it is necessary to ensure that the number of features when the calculation tensor enters the fully connected layer is consistent. Usually, the image is scaled normalized when the original image is input; simple normalization processing such as zooming in, zooming out, cropping, etc. may cause the image features to disappear or introduce noise. Therefore, this embodiment introduces SPP (spatial pyramid pooling), which can skip the image scale normalization process to prevent the disappearance of feature information; SPP can perform pooling of different scales on the original image of any scale, and then merge the pooling results obtained at each scale into a new feature map, and then perform a full connection operation, such as Figure 6 Shown is the structural diagram of SPP.
[0092] In specific implementation, after completing the modification of the network architecture of the target detection module, model training is performed to obtain the optimized weight parameters of the target detection module, and the optimized weight parameters are used to process the image to be labeled.
[0093] In one embodiment, based on the annotation result confidence, the annotation set of each frame of image is divided into a high confidence set and a low confidence set according to a preset annotation result confidence, including:
[0094] Setting a first preset annotation result confidence level and a second preset annotation result confidence level, wherein the second preset annotation result confidence level is greater than the first preset annotation result confidence level;
[0095] When the annotation result confidence is greater than the second preset annotation result confidence, the target annotation box corresponding to the annotation result confidence is divided into the high confidence set;
[0096] When the annotation result confidence is between the first preset annotation result confidence and the second preset annotation result confidence, the target annotation box corresponding to the annotation result confidence is divided into the low confidence set;
[0097] When the annotation result confidence is less than the first preset annotation result confidence, the target annotation box corresponding to the annotation result confidence is deleted.
[0098] When implementing it, refer to Figure 2 , send the image set to be labeled into the target detection module, and obtain the labeling result set A, A = {a1, a2, ..., a n}, where a n Corresponding to the nth annotation result output by the target detection module for the current frame image, a={x min ,y min , x max ,y max, class, score}, x min is the x-axis coordinate of the upper left corner of the annotation result; y min is the y-axis coordinate of the upper left corner of the annotation result; max is the x-axis coordinate of the lower right corner of the annotation result; y max is the y-axis coordinate of the lower right corner of the annotation result; class is the target category; score is the confidence score of the annotation result.
[0099] After processing the first frame of the image, all results are initialized as associated trajectories. The associated trajectory can be understood as the movement direction of the target, which is used to connect the annotation results of the same target predicted between frames in series and to associate the annotation results between frames. The associated trajectory is expressed as Γ = {Γ1, Γ2, ..., Γ n}, Γ n is the associated trajectory corresponding to the nth annotation result.
[0100] Start the first inter-frame association; use Kalman filtering to predict the possible position and size of the next frame target annotation box of all associated trajectories of the current frame, and obtain the prediction set A predicted When processing the next frame of image, the annotation result set B = {b1, b2, ..., b n}, put the annotation results of the next frame image with a confidence greater than 0.6 into the high confidence set B high , put the annotation results with confidence less than 0.6 and greater than 0.1 into the low confidence set B low , discard the annotation results with confidence less than 0.1.
[0101] In one embodiment, the first preset annotation result confidence is set to 0.1, and the second preset annotation result confidence is set to 0.6.
[0102] In one embodiment, the associating the high confidence set of the next frame image with the associated trajectory as the same target based on the prediction set includes:
[0103] Calculate the IOU value between each target annotation box in the prediction set and each target annotation box in the high confidence set;
[0104] Based on the IOU value, construct a relationship loss matrix between the prediction set and the high confidence set;
[0105] According to the relationship loss matrix, the Hungarian algorithm is used to associate the high confidence set of the next frame image with the associated trajectory as the same target.
[0106] In specific implementation, the prediction set A is calculated predictedThe annotation box and all high confidence sets B high The IOU of all the annotation boxes in , based on the IOU value, build the prediction set A predicted With high confidence set B high According to the relationship loss matrix, the Hungarian algorithm is used to assign all trajectory objects and high confidence set annotation boxes, and the associated trajectory Γ is obtained. related , associated high confidence annotation box B related , unassociated trajectory Γ un-related , unassociated high confidence annotation box B un-related , update the associated trajectory Γ with the already associated annotation box.
[0107] Start the second frame association and associate the unassociated trajectory Γ in the first association un-related Calculation and low confidence set B low The IOU values of all the annotation boxes in the , construct the relationship loss matrix, and use the Hungarian algorithm to convert the unrelated trajectories Γ un-related and low confidence set B low The label boxes are assigned and the associated trajectory Γ is updated with the successfully associated label boxes.
[0108] In one embodiment, the method further comprises:
[0109] Recording the number of unassociated frames for each of the unassociated trajectories;
[0110] When the number of unassociated frames exceeds a preset threshold, the unassociated track is deleted.
[0111] In the specific implementation, the unassociated trajectories are judged. If they have exceeded a preset threshold, for example, 15 frames, the target is considered to have disappeared and no annotation is required. The trajectory is deleted, otherwise it is stored; the annotation box in the high confidence set that is still unassociated is stored as a new trajectory to Γ, and the annotation result is output.
[0112] The method of the present application is described below with a specific embodiment, which specifically includes the following steps:
[0113] (1) Constructing a detection module;
[0114] (2) First, the network features and network parameters are trained, which only needs to be trained once;
[0115] (3) Darknet53 is used as the feature extraction backbone network;
[0116] (4) By modifying the cfg network structure configuration file, the feature vectors such as 208×208, 104×104, 52×52 are set to a downsampling factor of 2, downsampled by a 1×1 convolution kernel, and then fused with 104×104, 52×52, 26×26, and 13×13;
[0117] (5) CIOU is used for optimization. The calculation method of CIOU is:
[0118]
[0119] Among them, ρ 2 (b,b gt ) represents the Euclidean distance between the center point of the predicted box and the true box, that is, Figure 5 d in;
[0120] c represents the diagonal distance of the minimum enclosed area that can contain both the predicted box and the true box. α and v are calculated as follows:
[0121]
[0122] Among them, w gt is the width of the real frame, h gt is the height of the real box, w is the width of the predicted box, h is the height of the predicted box, v and α are correction factors, considering the shape and direction of the target box;
[0123] (6) When inputting the original image, the image is scaled normalized to 416*416;
[0124] (7) The introduction of the SPP module can skip image scale normalization and prevent the disappearance of feature information;
[0125] (8) After completing the modification of the target detection module network architecture, perform model training to obtain the weight parameters of the target detection module;
[0126] (9) During the training of the target detection module, the number of iterations was set to 50,000. Since the training samples had images with large resolution, in order to prevent memory overflow, the batch size was set to 32, the subdivision size was set to 16, and the initial learning rate was set to 0.25×10 -4 In order to obtain better training results, the learning rate is set to decay to 0.25×10 after 40000 and 45000 iterations respectively. -5 , 0.25×10 -6 ;
[0127] (10) Load the trained model parameters into the annotation project;
[0128] (11) The CPU of the marked environment is Intel i5-10400f, the memory is 16GB, and there is no GPU;
[0129] (12) Segment a 30Hz 5s video frame by frame to obtain a total of 150 frames of image data
[0130] IMG={img1, img2,...,img 150};
[0131] (13) First, send img1 to the detection module and obtain 4 annotation results, A img1 ={a1, a2, ..., a n},in:
[0132] a1={100, 100, 120, 120, 0, 0.8},
[0133] a2={200,200,220,220,0,0.5},
[0134] a3={50, 50, 55, 55, 0, 0.05},
[0135] a4={300, 280, 350, 330, 0, 0.7}};
[0136] (14) Keep the annotation results with score>0.1, initialize a1, a2, a4 as trajectories, Γ img1 ={a1, a2, a4}, and remove a3, and output {a1, a4} with score>0.6 as the annotation result of img1;
[0137] (15) Use Kalman filtering to predict all Γ img1 The possible position and size of the target annotation box in the next frame, and obtain the prediction set A piredicted ={pre1, pre2, pre3}, where:
[0138] pre1={105, 100, 125, 120, 0},
[0139] pre2={205, 200, 225, 220, 0},
[0140] pre3={305, 280, 355, 330, 0}
[0141] (16) Send image img2 to the target detection module for processing and obtain multiple annotation results, B img2 ={b1, b2, ..., b4}, where:
[0142] b1={107,100,127,120,0,0.7},
[0143] b2={207, 200, 227, 220, 0, 0.45}.
[0144] b3={90, 50, 105, 55, 0, 0.07}.
[0145] b4={300, 400, 310, 410,0,0.9};
[0146] (17) B img2 The annotation results with medium confidence greater than 0.6 (score>0.6) are placed in the high confidence set B. high ={b1,b4};
[0147] (18) B img2 The annotation results with medium confidence less than 0.6 and greater than 0.1, that is, 0.6>score>0.1, are placed in the low confidence set B low ={b2};
[0148] (19) B img2 The annotation results with confidence less than 0.1, i.e. score < 0.1, are discarded, i.e. b3 is removed;
[0149] (20) Calculate the prediction set A predicted Annotated box and all high confidence sets B high The IOU of all the annotation boxes in the , we get the following results, refer to Table 1:
[0150] IOU(pre1-b1)=0.82,
[0151] IOU(pre2-b1)=0,
[0152] IOU(pre3-b1)=0
[0153] IOU(pre1-b4)=0,
[0154] IOU(pre2-b4)=0,
[0155] IOU(pre3-b4)=0;
[0156] Among them, when I OU < 0.4, it is considered that there is no association;
[0157] Table 1. A based on IOU predicted With B high Relationship Matrix
[0158] <![CDATA[pre1]]> <![CDATA[pre2]]> <![CDATA[pre3]]> <![CDATA[b1]]> 0.82 0 0 <![CDATA[b4]]> 0 0 0
[0159] (21) According to the relation loss matrix, the Hungarian algorithm is used to assign all trajectory objects and high confidence set annotation boxes, and the associated trajectory Γ is obtained. related , associated annotation box A related , unassociated trajectory Γ un-related , Unassociated dimension box A un-related ,,in:
[0160] Γ related ={pre1},
[0161] A related ={b1},
[0162] Γ un-related ={pre2},
[0163] A un-related ={b4},
[0164] That is, a1 in image img1 can be associated with b1 in img2 as the same target;
[0165] (22) The unassociated trajectory Γ in the first association un-related The prediction result calculation and low confidence set B l For all the annotation boxes iou in ow, we construct the relation loss matrix and get:
[0166] IOU(pre2-b2)=0.82;
[0167] (23) Use the Hungarian algorithm to convert the unconnected trajectory Γ un-related and low confidence set B low The annotation boxes are assigned, and a2 in image img1 can be associated with b2 in img2;
[0168] (24) At this time Γ img2 ={b1, b2, a4}, if a4 is still not associated, then determine whether a4 has not been associated with a new result for 15 consecutive frames. If so, delete it from the track, otherwise keep it;
[0169] (25)B high = {b1, b4}, b4 is still not associated, it is judged as a new target, and the trajectory Γ is updated img2 ={b1, b2, a4} is Γ img2 ={b1, b2, a4, b4}, and obtain the prediction set A predicted-img2 ={pre1, pre2, pre3, pre4}, where:
[0170] pre1={113, 100, 133, 120, 0},
[0171] pre2={213, 200, 233, 220, 0}.
[0172] pre3={313,280,343,330,0}
[0173] pre4 = {303,400,313,410,0}
[0174] (26) Γ img2 ={b1, b2, b4} is stored as the annotation result of img2;
[0175] (27) Repeat step 16 until IMG = {img1, img2, ..., img 150}All processed.
[0176] In a second aspect, an embodiment of the present application further provides a target labeling device based on inter-frame association, which is used to implement the target labeling method based on inter-frame association of any embodiment of the first aspect, and the device includes:
[0177] A construction module is used to construct an object detection module based on spatial pyramid pooling and multi-dimensional feature fusion;
[0178] The module for acquiring a set of images to be labeled is used to extract frames from continuous video data to obtain a set of images to be labeled;
[0179] A labeling set acquisition module, used to input the image set to be labeled into the target detection module for target labeling, and obtain a labeling set for each frame of image, wherein the labeling set includes the coordinates, category and labeling result confidence of each target labeling box;
[0180] A division module, configured to divide the annotation set of each frame of image into a high confidence set and a low confidence set according to a preset annotation result confidence based on the annotation result confidence;
[0181] A prediction module, used to initialize the annotation set corresponding to the first frame image as an associated track, predict the coordinates and size of each target annotation box of the associated track in the next frame image, and obtain a prediction set;
[0182] A first associating module is used to associate the high confidence set of the next frame image with the associated trajectory with the same target based on the prediction set, obtain an associated trajectory, an associated high confidence annotation box, an unassociated trajectory and an unassociated high confidence annotation box, and update the associated trajectory using the associated high confidence annotation box;
[0183] A second associating module is used to associate the unassociated track with the low confidence set of the next frame image to the same target, obtain an associated track, an associated low confidence annotation box, an unassociated track and an unassociated low confidence annotation box, and update the associated track using the associated low confidence annotation box;
[0184] An initialization module, used for initializing the unassociated high-confidence annotation box as the associated track for associating the next frame of unassociated image;
[0185] The result acquisition module is used to continuously associate each frame of unassociated images to obtain the annotation results.
[0186] The embodiment provided by the present invention optimizes the target detection module based on spatial pyramid pooling and multi-dimensional feature fusion, optimizes the low computing power requirement feature extraction network, improves the feature extraction ability of the feature extraction network for small targets and the adaptability to scale changes, and improves the labeling accuracy; by dividing the labeling boxes into high confidence sets and low confidence sets according to the confidence level, firstly associate the labeling boxes in the high confidence set with the target trajectory, update the unsuccessfully associated labeling boxes to the new trajectory, and then associate the labeling boxes in the low confidence set with the unmatched trajectory objects, retain the successfully associated low confidence labeling boxes, thereby retaining all the labeling results as much as possible and reducing the missed label rate. The present application has low computing power requirements and does not rely on high-performance hardware platforms. Low computing power computers can carry out labeling work, which is of great significance for realizing the engineering and convenience of labeling tasks.
[0187] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.
Claims
1. A target labeling method based on inter-frame association, characterized in that: include: Construct an object detection module based on spatial pyramid pooling and multi-dimensional feature fusion; Extract frames from continuous video data to obtain a set of images to be labeled; Inputting the set of images to be annotated into the target detection module for target annotating, and obtaining a set of annotations for each frame of the image, wherein the set of annotations includes the coordinates, category and confidence of the annotation result of each target annotation box; Based on the annotation result confidence, the annotation set of each frame of image is divided into a high confidence set and a low confidence set according to a preset annotation result confidence; Initialize the annotation set corresponding to the first frame image as the associated track, predict the coordinates and size of each target annotation box of the associated track in the next frame image, and obtain a predicted set; Based on the prediction set, the high confidence set of the next frame image is associated with the associated trajectory with the same target to obtain an associated trajectory, an associated high confidence annotation box, an unassociated trajectory and an unassociated high confidence annotation box, and the associated trajectory is updated using the associated high confidence annotation box; Associating the unassociated track with the low confidence set of the next frame image to the same target, obtaining an associated track, an associated low confidence annotation box, an unassociated track and an unassociated low confidence annotation box, and updating the associated track using the associated low confidence annotation box; Initializing the unassociated high-confidence annotation box as the associated track for associating the next frame of unassociated images; Continue to associate each frame of unassociated images to obtain the labeling results.
2. The target labeling method based on inter-frame association according to claim 1 is characterized in that: The target detection module uses the low-computing power network architecture Darknet53 as the feature extraction network.
3. The target labeling method based on inter-frame association according to claim 2 is characterized in that: The object detection module based on spatial pyramid pooling and multi-dimensional feature fusion is constructed, including: A downsampling factor is set for the shallow features of the feature extraction network, downsampling is performed through a 1×1 convolution kernel, and the shallow features are fused with the deep features to achieve the multi-dimensional feature fusion.
4. The target labeling method based on inter-frame association according to claim 3 is characterized in that: The downsampling factor is set to 2.
5. The target labeling method based on inter-frame association according to claim 1, characterized in that: The CIOU loss function is used for target labeling in the target detection module.
6. The target labeling method based on inter-frame association according to claim 5, characterized in that: The calculation formula of the CIOU loss function is: Among them, IOU is the intersection-over-union ratio between the predicted box and the real box, ρ 2 (b,b gt ) is the Euclidean distance between the center point of the predicted box and the center point of the real box, c is the diagonal distance of the minimum closed area that can contain both the predicted box and the real box, α and v are correction factors, and the calculation formulas of α and v are: Among them, w gt is the width of the real frame, h gt is the height of the real box, w is the width of the predicted box, and h is the height of the predicted box.
7. The target labeling method based on inter-frame association according to claim 1, characterized in that: Based on the annotation result confidence, the annotation set of each frame image is divided into a high confidence set and a low confidence set according to a preset annotation result confidence, including: Setting a first preset annotation result confidence level and a second preset annotation result confidence level, wherein the second preset annotation result confidence level is greater than the first preset annotation result confidence level; When the annotation result confidence is greater than the second preset annotation result confidence, the target annotation box corresponding to the annotation result confidence is divided into the high confidence set; When the annotation result confidence is between the first preset annotation result confidence and the second preset annotation result confidence, the target annotation box corresponding to the annotation result confidence is divided into the low confidence set; When the annotation result confidence is less than the first preset annotation result confidence, the target annotation box corresponding to the annotation result confidence is deleted.
8. The target labeling method based on inter-frame association according to claim 1, characterized in that: The associating the high confidence set of the next frame image and the associated trajectory with the same target based on the prediction set includes: Calculate the IOU value between each target annotation box in the prediction set and each target annotation box in the high confidence set; Based on the IOU value, construct a relationship loss matrix between the prediction set and the high confidence set; According to the relationship loss matrix, the Hungarian algorithm is used to associate the high confidence set of the next frame image with the associated trajectory as the same target.
9. The target labeling method based on inter-frame association according to claim 1, characterized in that: The method further comprises: Recording the number of unassociated frames for each of the unassociated trajectories; When the number of unassociated frames exceeds a preset threshold, the unassociated track is deleted.
10. A target labeling device based on inter-frame association, used to implement the target labeling method based on inter-frame association according to any one of claims 1 to 9, the device comprising: A construction module is used to construct an object detection module based on spatial pyramid pooling and multi-dimensional feature fusion; The module for acquiring a set of images to be labeled is used to extract frames from continuous video data to obtain a set of images to be labeled; A labeling set acquisition module, used to input the image set to be labeled into the target detection module for target labeling, and obtain a labeling set for each frame of image, wherein the labeling set includes the coordinates, category and labeling result confidence of each target labeling box; A division module, configured to divide the annotation set of each frame of image into a high confidence set and a low confidence set according to a preset annotation result confidence based on the annotation result confidence; A prediction module, used to initialize the annotation set corresponding to the first frame image as an associated track, predict the coordinates and size of each target annotation box of the associated track in the next frame image, and obtain a prediction set; A first associating module is used to associate the high confidence set of the next frame image with the associated trajectory with the same target based on the prediction set, obtain an associated trajectory, an associated high confidence annotation box, an unassociated trajectory and an unassociated high confidence annotation box, and update the associated trajectory using the associated high confidence annotation box; A second associating module is used to associate the unassociated track with the low confidence set of the next frame image to the same target, obtain an associated track, an associated low confidence annotation box, an unassociated track and an unassociated low confidence annotation box, and update the associated track using the associated low confidence annotation box; An initialization module, used for initializing the unassociated high-confidence annotation box as the associated track for associating the next frame of unassociated image; The result acquisition module is used to continuously associate each frame of unassociated images to obtain the annotation results.