A method for automatically labeling moving targets in surveillance videos

Through the combination of deep convolutional neural network and YoloV5 detection, the problem of insufficient time and space-time correlation of motion target annotation in the existing technology is solved, and automatic labeling with high accuracy is achieved, adapting to target occlusion, and improving labeling efficiency is improved. .

CN114693742BActive Publication Date: 2024-11-22WISCOM SYSTEM CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210419230.6
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-04-20
Publication Date
2024-11-22
Estimated Expiration
2042-04-20

AI Technical Summary

Technical Problem

The existing automatic labeling method fails to effectively utilize the spatial and temporal correlation characteristics when moving target labeling in video, and cannot achieve fully automatic labeling. It can only be limited to pre-labeled source pictures, ignoring other pedestrian targets.

Method used

Deep convolutional neural network and YoloV5 are used to detect video frames, and automatically annotate the moving targets by tracking queues and feature vector matching. The adaptive weight matching algorithm is used to calculate the adaptive weight matching algorithm in continuous video frames. Identification and annotation of the same moving target.

Benefits of technology

It realizes automatic labeling of moving targets in the true sense, improves the accuracy and efficiency of labeling, can effectively utilize the spatial and temporal correlation characteristics, adapt to target occlusion, and improves the success rate of target matching.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114693742B_ABST
    Figure CN114693742B_ABST
Patent Text Reader

Abstract

The present invention discloses an automatic labeling method for moving targets in surveillance videos, which can help algorithm researchers to label moving targets based on videos and quickly generate image sample data sets; the labeling process of the present invention is unsupervised, and is a real automatic labeling of moving targets; the labeling is designed based on a multi-target tracking algorithm, which automatically detects the same moving target appearing in multiple consecutive videos and assigns the same identification ID, effectively utilizing the spatiotemporal correlation characteristics of the moving target between consecutive video frames with high accuracy; an adaptive weighted target matching algorithm is designed for the case of target occlusion, which effectively improves the target matching success rate and accuracy; a pre-trained convolutional neural network can be used to perform target detection on video frames, and the similarity of feature vectors between the detected target and the tracked target can be calculated, so as to collect different postures of the same moving target.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent video image analysis, and in particular to an automatic marking method for moving targets in a monitoring video. Background Art

[0002] Person re-identification refers to the use of computer vision technology to determine whether a specific pedestrian exists in an image or video sequence. It has extremely great practical value in the fields of public security, criminal investigation, etc. Person re-identification requires that the sample data set be classified according to each specific person, that is, each person has a unique ID, each person has multiple pictures, and multiple pictures of the same person reflect the differences in shooting angle, light, or the person's posture, clothing, etc.

[0003] There are two common ways to obtain person re-identification sample datasets: downloading from the Internet or researchers forming their own sample datasets through image annotation. For the latter, video-based pedestrian annotation is more convenient than discrete image-based pedestrian annotation because it is easy to obtain multiple images of the same person in a video.

[0004] There are two common methods for pedestrian labeling in videos: manual labeling and automatic labeling. Manual labeling refers to selecting certain video frames from the video as discrete picture sets, and manually labeling pedestrians on the discrete picture sets. Automatic labeling also selects certain video frames from the video as discrete picture sets, and selects one or more of them as source pictures, and manually labels the pedestrians of interest on the source pictures, then extracts features from the labeled pedestrians, inputs the feature parameters into the pre-trained neural network, and detects and classifies other pictures except the source pictures, thereby completing pedestrian labeling.

[0005] The existing automatic labeling methods are more efficient than manual labeling methods, but they still have several shortcomings. First, they treat multiple video frames as discrete image sets, failing to effectively utilize the spatiotemporal correlation characteristics of moving targets between consecutive video frames; second, before automatic labeling, one or more source images need to be selected for manual labeling, and then the next step of automatic labeling is performed, failing to achieve truly fully automatic labeling; in addition, the pedestrians that can be automatically labeled can only be limited to those that have been manually labeled in the source images in advance, while other pedestrians will be ignored.

[0006] The image annotation discussed above is mainly for pedestrian targets. Without loss of generality, this annotation method is also applicable to image annotation of other targets (including moving targets and stationary targets), such as vehicles, animals, tables and chairs, computers and other targets. Summary of the invention

[0007] The purpose of the present invention is to provide a method for automatically labeling moving objects in surveillance videos to solve the problems raised in the above-mentioned background technology.

[0008] In order to solve the above technical problems, the present invention provides the following technical solutions: a method for automatically marking moving targets in a surveillance video, comprising the following steps:

[0009] S1, input the first video frame and record the timestamp t1 of the video frame;

[0010] S2, use the pre-trained deep convolutional neural network to detect objects in video frames;

[0011] S3, put the detected target into the tracking queue Q, the tracking queue Q is initially empty, and each target in the tracking queue Q is called a tracking target;

[0012] S4, marking the target in the current video frame;

[0013] S5, input the next video frame, and record the timestamp t2 of the video frame;

[0014] S6, using a pre-trained convolutional neural network to detect objects in the video frame, and the detected object is called a detected object;

[0015] S7, pairwise matching of each detected target in the video frame at time t2 with each tracked target in the tracking queue Q;

[0016] S8. Update the tracking queue according to the matching result;

[0017] S9, marking the target in the current video frame;

[0018] S10, check whether the video has ended, if not, go to step S5, otherwise the process ends.

[0019] Furthermore, in step S2, the YoloV5 convolutional neural network is used to perform target detection on the input video frame, and the system sets the following six attributes for each detected target: target ID, target category (pedestrian), target initial time, target update time, position coordinates of the target enclosing rectangular box, and target feature vector; wherein the target initial time is the timestamp of the video frame where the target first appears, the target update time is the timestamp of the video frame where the target appears again, the position coordinates of the target enclosing rectangular box are the offset values ​​of the upper left and lower right corners of the rectangular box relative to the pixels in the upper left corner of the screen in the row and column directions, and the target feature vector is an image feature that describes the target, and the feature vectors of different targets are different.

[0020] For each target, its attributes are assigned as follows:

[0021] The target ID is set to empty;

[0022] The target initial time is set to the timestamp t1 of the current video frame;

[0023] The target update time is set to the timestamp t1 of the current video frame;

[0024] The position coordinates of the target enclosing rectangular box are set to (left, top, right, bottom), where (left, top) is the position coordinates of the upper left corner of the enclosing rectangular box, and (right, bottom) is the position coordinates of the lower right corner of the enclosing rectangular box;

[0025] The target feature vector is set to a normalized 128-dimensional feature vector obtained by applying another pre-trained convolutional neural network model to extract features of the target in the rectangular bounding box.

[0026] Furthermore, in step S4, the operations performed are:

[0027] 1) Set a globally unique target ID for each target in the tracking queue Q;

[0028] 2) Save the current video frame image and all tracking targets in the tracking queue Q. Obviously, all targets in the current video frame are already in the tracking queue, and the attributes of each target in the tracking queue Q: target ID and the position coordinates of the target enclosing rectangular box, that is, a target label for the current video frame.

[0029] Furthermore, in step S6, for each detected target, its six attributes are assigned as follows:

[0030] The target ID is set to empty;

[0031] The target initial time is set to the timestamp t2 of the current video frame;

[0032] The target update time is set to the timestamp t2 of the current video frame;

[0033] The position coordinates of the target enclosing rectangular box are set to (left, top, right, bottom), where (left, top) is the position coordinates of the upper left corner of the enclosing rectangular box, and (right, bottom) is the position coordinates of the lower right corner of the enclosing rectangular box;

[0034] The target feature vector is set to a normalized 128-dimensional feature vector obtained by applying another pre-trained convolutional neural network model to extract features of the target in the rectangular bounding box.

[0035] Furthermore, in step S7, the matching operation is to calculate the similarity of the feature vectors between the detection target and the tracking target, and the cosine distance is used to characterize the similarity; assuming that the tracking target and the detection target are represented by ID1 and ID2 respectively, the cosine distance of the feature vectors of the two is:

[0036]

[0037] In formula (1), Dist(ID2, ID1) represents the cosine distance between the target feature vectors of ID2 and ID1, r ID1 The normalized 128-dimensional feature vector obtained by applying the convolutional neural network to track target ID1, The vector obtained by transposing the normalized 128-dimensional feature vector obtained by applying the convolutional neural network to detect target ID2.

[0038] The solution sets a feature vector distance threshold Dist_Threshold, and Dist_Threshold satisfies: 0≤Dist_Threshold≤2; the default value of Dist_Threshold in the present invention is 0.1, and the threshold can be changed as needed during actual use to adjust the accuracy of target matching.

[0039] For the detection target ID2 and the tracking target ID1, the judgment rules are:

[0040] 1) If Dist(ID2, ID1)≤Dist_Threshold, it is considered that the detected target ID2 matches the tracked target ID1 successfully;

[0041] 2) If Dist(ID2, ID1)>Dist_Threshold, it is considered that the detection target ID2 fails to match the tracking target ID1.

[0042] If a detection target can be successfully matched with multiple tracking targets, the tracking target with the smallest cosine distance is selected as the best match; the successfully matched detection target is put into the matching success queue Q1, and it is associated with the target ID of the best matching successful tracking target.

[0043] If a detection target fails to match multiple tracking targets, the tracking target with the smallest cosine distance is selected as the best failed matching target; the failed matching detection target is placed in the failed matching queue Q2 and associated with the target ID of the best failed matching tracking target.

[0044] Furthermore, for detection targets that fail to match, it usually falls into one of the following two situations:

[0045] Case 1: the target is a moving target that newly enters the screen at time t2.

[0046] In the second case, due to image problems, such as the target being blocked or changes in light or target posture, the same moving target that appears in the image at both time t1 and time t2 fails to be matched.

[0047] Furthermore, for detection targets that fail to match, the solution will apply the following rules:

[0048] An entry area is set on each side of the monitoring screen. If the position of the detection target that fails to match is in the entry area, it belongs to case one, and the detection target is classified as a new moving target; if the position of the detection target is in the non-entry area, it belongs to case two.

[0049] For the detection target belonging to case 2, we calculate the Mahalanobis distance Dist_mah(ID1, ID2) between the detection target ID2 and the best matching failure target ID1.

[0050]

[0051] In formula (2), d ID1 is the rectangular bounding box position of the best matching failed target ID1, y ID2 To detect the predicted position of target ID2, S represents the covariance matrix between the predicted position of target ID2 and the position of the best matching failed target ID1.

[0052] Formula (3) is used to calculate the weighted distance C(ID2, ID1) between targets ID1 and ID2.

[0053] C(ID2, ID1)=λDist(ID2, ID1)+(1-λ)Dist_mah(ID2, ID1) (3)

[0054] In formula (3), the cosine distance Dist(ID2, ID1) reflects the similarity of the appearance features of targets ID1 and ID2, and Dist_mah(ID2, ID1) reflects the position deviation relationship between the predicted position of target ID1 and the detected target ID2, that is, the motion information of target ID2 between adjacent video frames.

[0055] The weight λ satisfies 0≤λ≤1. The larger the λ, the more the matching depends on the similarity of appearance features. Conversely, the smaller the λ, the more the matching is biased towards motion information.

[0056] In the present invention, we adaptively adjust the size of λ;

[0057] When the system finds that there are many moving targets detected in the current picture, the occlusion probability between the targets increases. At this time, the value of λ is reduced to match the motion association that is more biased towards the target;

[0058] When the system finds that there are fewer moving targets detected in the picture, the occlusion probability between the targets decreases. At this time, the value of λ is increased, and the matching is more biased towards the similarity of the target's appearance features.

[0059] Furthermore, in step S8, the specific operation is divided into two steps:

[0060] The first step is to traverse the matching success queue Q1, find the associated tracking target ID1 for each detection target ID2 in the queue, and perform the following updates on the six attributes of the tracking target ID1:

[0061] The target ID remains unchanged;

[0062] The target initial time remains unchanged;

[0063] The target update time is set to the timestamp t2 of the current video frame;

[0064] The position coordinates of the target enclosing rectangular frame are set to the position coordinates of the target enclosing rectangular frame of the detection target ID2;

[0065] The target feature vector is set to the target feature vector of the detection target ID2.

[0066] The second step is to traverse the failed matching queue Q2 and add all the detection targets that failed to match to the tracking queue Q. Note that the detection targets that failed to match are all moving targets that have newly entered the screen.

[0067] Furthermore, as video frames are continuously input, the tracking targets in the tracking queue Q will accumulate more and more, resulting in an excessively long queue or memory overflow. In actual situations, moving targets will also leave the monitoring screen after a certain period of time. Therefore, it is necessary to delete expired moving targets in the tracking queue Q. To this end, the following two steps are performed:

[0068] The first step is to traverse the tracking queue Q and calculate the survival time of each tracking target IDi using formula (4):

[0069] Life_Time IDi =Now_Time–Update_time IDi (4)

[0070] Among them, Life_Time in formula (4) IDi To track the survival time of the target IDi, Now_Time is the current time, Update_time IDiTarget update time for tracking target IDi.

[0071] The second step is to set the tracking target expiration threshold Track_Expiration_Threshold in advance. If Life_Time IDi >Track_Expiration_Threshold, the target IDi is deleted from the tracking queue Q; otherwise, the target IDi is retained.

[0072] Furthermore, in step S9, the following actions are performed:

[0073] 1) Setting a globally unique target ID value for each target whose target ID value is empty in the tracking queue Q. The object of this operation is the target that failed to match added in step S8;

[0074] 2) Save the current video frame image and all tracking targets in the tracking queue Q; obviously, all targets in the current video frame are already in the tracking queue, and the attributes of each target in the tracking queue Q: target ID and the position coordinates of the target enclosing rectangular box, that is, a target annotation for the current video frame.

[0075] Compared with the prior art, the beneficial effects achieved by the present invention are:

[0076] 1. The present invention can help algorithm researchers to label moving targets based on videos and quickly generate image sample data sets; the labeling process of the present invention is unsupervised and is truly automatic labeling of moving targets; the labeling is designed based on a multi-target tracking algorithm, which automatically detects the same moving target appearing in multiple consecutive videos and assigns the same identification ID, effectively utilizing the spatiotemporal correlation characteristics of the moving target between consecutive video frames with high accuracy; an adaptive weighted target matching algorithm is designed for the case of target occlusion, which effectively improves the target matching success rate and accuracy;

[0077] 2. The present invention can use a pre-trained convolutional neural network to detect targets in video frames, and can calculate the similarity of the feature vectors between the detected target and the tracked target. As the video frames are continuously input, the tracked targets in the tracking queue Q will accumulate more and more, resulting in an overlong queue or memory overflow. In actual situations, the moving target will also leave the monitoring screen after a certain period of time; therefore, it is necessary to delete the expired moving targets in the tracking queue Q, and use the cosine distance to characterize the similarity. The closer the cosine distance value is to 0, the greater the similarity of the two feature vectors. Conversely, the larger the cosine distance value is, the lower the similarity of the two feature vectors is. This can effectively improve the matching success rate when the object is obscured. For generating an image sample data set, it is usually necessary to perform annotation across multiple frames in order to collect different postures of the same moving target. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0079] Figure 1 It is a schematic diagram of the overall working principle of the present invention;

[0080] Figure 2 is a schematic diagram of target detection in the present invention;

[0081] Figure 3 is a schematic diagram of the video frame annotation results of t1 and t2 in the present invention;

[0082] Figure 4 It is a schematic diagram of the λ adjustment principle in the present invention;

[0083] Figure 5 is a schematic diagram of area setting in the present invention;

[0084] Figure 6 It is a schematic diagram of the result of automatic labeling of video frames at multiple moments in the present invention. DETAILED DESCRIPTION

[0085] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0086] like Figure 1-Figure 6 The method for automatically labeling a moving target in a surveillance video includes the following steps:

[0087] S1, input the first video frame and record the timestamp t1 of the video frame;

[0088] S2. Use a pre-trained deep convolutional neural network to detect targets in video frames. Use the YoloV5 convolutional neural network to detect targets in input video frames. The system sets the following six attributes for each detected target: target ID, target category (pedestrian), target initial time, target update time, position coordinates of the target enclosing rectangular box, and target feature vector. The target initial time is the timestamp of the video frame where the target first appears, the target update time is the timestamp of the video frame where the target appears again, the position coordinates of the target enclosing rectangular box are the offset values ​​of the upper left and lower right corners of the rectangular box relative to the pixels in the upper left corner of the screen in the row and column directions, and the target feature vector is the image feature that describes the target. The feature vectors of different targets are different.

[0089] For each target, its attributes are assigned as follows:

[0090] The target ID is set to empty;

[0091] The target initial time is set to the timestamp t1 of the current video frame;

[0092] The target update time is set to the timestamp t1 of the current video frame;

[0093] The position coordinates of the target enclosing rectangular box are set to (left, top, right, bottom), where (left, top) is the position coordinates of the upper left corner of the enclosing rectangular box, and (right, bottom) is the position coordinates of the lower right corner of the enclosing rectangular box;

[0094] The target feature vector is set to a normalized 128-dimensional feature vector obtained by applying another pre-trained convolutional neural network model to extract features of the target in the rectangular bounding box.

[0095] Figure 2 This is a schematic diagram of target detection. The system detects three targets in this video frame, and each target is surrounded by a rectangular box;

[0096] S3, put the detected target into the tracking queue Q, the tracking queue Q is initially empty, and each target in the tracking queue Q is called a tracking target;

[0097] S4. Label the target in the current video frame. The operations performed are:

[0098] 1) Set a globally unique target ID for each target in the tracking queue Q;

[0099] 2) Save the current video frame image and all tracking targets in the tracking queue Q. Obviously, all targets in the current video frame are already in the tracking queue, and the attributes of each target in the tracking queue Q: target ID and the position coordinates of the target enclosing rectangle, that is, a target label for the current video frame;

[0100] Figure 3 (a) shows the labeling result of a video frame of a surveillance video at time t1;

[0101] S5, input the next video frame, and record the timestamp t2 of the video frame;

[0102] S6. Use the pre-trained convolutional neural network to detect objects in the video frame. The detected objects are called detected objects. For each detected object, its six attributes are assigned as follows:

[0103] The target ID is set to empty;

[0104] The target initial time is set to the timestamp t2 of the current video frame;

[0105] The target update time is set to the timestamp t2 of the current video frame;

[0106] The position coordinates of the target enclosing rectangular box are set to (left, top, right, bottom), where (left, top) is the position coordinates of the upper left corner of the enclosing rectangular box, and (right, bottom) is the position coordinates of the lower right corner of the enclosing rectangular box;

[0107] The target feature vector is set to a normalized 128-dimensional feature vector obtained by applying another pre-trained convolutional neural network model to extract features of the target in the rectangular bounding box;

[0108] S7, each detected target in the video frame at time t2 is matched with each tracked target in the tracking queue Q; the matching operation is to calculate the similarity of the feature vectors between the detected target and the tracked target, and the cosine distance is used to represent the similarity; assuming that the tracked target and the detected target are represented by ID1 and ID2 respectively, the cosine distance of the feature vectors of the two is:

[0109]

[0110] In formula (1), Dist(ID2, ID1) represents the cosine distance between the target feature vectors of ID2 and ID1, r ID1 The normalized 128-dimensional feature vector obtained by applying the convolutional neural network to track target ID1, The vector obtained by transposing the normalized 128-dimensional feature vector obtained by applying the convolutional neural network to detect target ID2.

[0111] Obviously, the value range of Dist(ID2, ID1) is between 0 and 2. The closer the value of Dist(ID2, ID1) is to 0, the greater the similarity between the two feature vectors. Conversely, the larger the value of Dist(ID2, ID1), the lower the similarity between the two feature vectors.

[0112] The solution sets a feature vector distance threshold Dist_Threshold, and Dist_Threshold satisfies: 0≤Dist_Threshold≤2; the default value of Dist_Threshold in the present invention is 0.1, and the threshold can be changed as needed during actual use to adjust the accuracy of target matching.

[0113] For the detection target ID2 and the tracking target ID1, the judgment rules are:

[0114] 1) If Dist(ID2, ID1)≤Dist_Threshold, it is considered that the detected target ID2 matches the tracked target ID1 successfully;

[0115] 2) If Dist(ID2, ID1)>Dist_Threshold, it is considered that the detection target ID2 fails to match the tracking target ID1.

[0116] If a detection target can be successfully matched with multiple tracking targets, the tracking target with the smallest cosine distance is selected as the best match; the successfully matched detection target is put into the matching success queue Q1, and it is associated with the target ID of the best matching successful tracking target.

[0117] If a detection target fails to match multiple tracking targets, the tracking target with the smallest cosine distance is selected as the best failed matching target; the failed matching detection target is placed in the failed matching queue Q2 and associated with the target ID of the best failed matching tracking target.

[0118] For detection targets that fail to match, it usually falls into one of the following two situations:

[0119] Case 1: the target is a moving target that newly enters the screen at time t2.

[0120] In the second case, due to image problems, such as the target being blocked or changes in light or target posture, the same moving target that appears in the image at both time t1 and time t2 fails to be matched.

[0121] For detection targets that fail to match, the solution will apply the following rules:

[0122] like Figure 5As shown, an entry area is set on each side of the monitoring screen. If the position of the detection target that fails to match is in the entry area, it belongs to case one, and the detection target is classified as a newly added moving target; if the position of the detection target is in the non-entry area, it belongs to case two.

[0123] For the detection target belonging to case 2, we calculate the Mahalanobis distance Dist_mah(ID1, ID2) between the detection target ID2 and the best matching failure target ID1.

[0124]

[0125] In formula (2), d ID1 is the rectangular bounding box position of the best matching failed target ID1, y ID2 To detect the predicted position of target ID2, S represents the covariance matrix between the predicted position of target ID2 and the position of the best matching failed target ID1.

[0126] Formula (3) is used to calculate the weighted distance C(ID2, ID1) between targets ID1 and ID2.

[0127] C(ID2, ID1)=λDist(ID2, ID1)+(1-λ)Dist_mah(ID2, ID1) (3)

[0128] In formula (3), the cosine distance Dist(ID2, ID1) reflects the similarity of the appearance features of targets ID1 and ID2, and Dist_mah(ID2, ID1) reflects the position deviation relationship between the predicted position of target ID1 and the detected target ID2, that is, the motion information of target ID2 between adjacent video frames.

[0129] The weight λ satisfies 0≤λ≤1. The larger the λ is, the more the matching depends on the similarity of appearance features. On the contrary, the smaller the λ is, the more the matching is biased towards motion information. Therefore, the rationality of the λ value setting has a great influence on the accuracy of matching.

[0130] In this invention, we adaptively adjust the size of λ; the basic idea is as follows Figure 4 As shown:

[0131] When the system finds that there are many moving targets detected in the current picture, the occlusion probability between the targets increases. At this time, the value of λ is reduced to match the motion association that is more biased towards the target;

[0132] When the system finds that there are fewer moving targets detected in the picture, the occlusion probability between the targets decreases. At this time, the value of λ is increased, and the matching is more biased towards the similarity of the target's appearance features.

[0133] Figure 4Where TL and TH are preset thresholds, TL is a smaller threshold and TH is a larger threshold. When the number of targets detected in the picture is between TL and TH, the system maintains λ unchanged.

[0134] In this way, we can effectively improve the matching success rate when the object is blocked; if the weighted distance C(ID2, ID1) calculated by formula (3) is still greater than the preset threshold, it means that the matching fails. At this time, we will directly delete the detection target ID1 from the matching failure queue Q2, that is, give up marking the target;

[0135] S8. Update the tracking queue according to the matching results. The specific operation is divided into two steps:

[0136] The first step is to traverse the matching success queue Q1, find the associated tracking target ID1 for each detection target ID2 in the queue, and perform the following updates on the six attributes of the tracking target ID1:

[0137] The target ID remains unchanged;

[0138] The target initial time remains unchanged;

[0139] The target update time is set to the timestamp t2 of the current video frame;

[0140] The position coordinates of the target enclosing rectangular frame are set to the position coordinates of the target enclosing rectangular frame of the detection target ID2;

[0141] The target feature vector is set to the target feature vector of the detection target ID2.

[0142] The second step is to traverse the failed matching queue Q2 and add all the detection targets that failed to match to the tracking queue Q. Note that the detection targets that failed to match are all moving targets that have newly entered the screen.

[0143] In addition, as video frames are continuously input, the tracking targets in the tracking queue Q will accumulate more and more, causing the queue to be too long or memory overflow. In actual situations, moving targets will also leave the monitoring screen after a certain period of time. Therefore, it is necessary to delete expired moving targets in the tracking queue Q. To do this, perform the following two steps:

[0144] The first step is to traverse the tracking queue Q and calculate the survival time of each tracking target IDi using formula (4):

[0145] Life_Time IDi =Now_Time–Update_time IDi (4)

[0146] Among them, Life_Time in formula (4) IDiTo track the survival time of the target IDi, Now_Time is the current time, Update_time IDi Target update time for tracking target IDi.

[0147] The second step is to set the tracking target expiration threshold Track_Expiration_Threshold in advance. If Life_Time IDi >Track_Expiration_Threshold, then delete the target IDi from the tracking queue Q; otherwise, keep the target IDi;

[0148] S9, marking the target in the current video frame; the execution actions are as follows:

[0149] 1) Setting a globally unique target ID value for each target whose target ID value is empty in the tracking queue Q. The object of this operation is the target that failed to match added in step S8;

[0150] 2) Save the current video frame image and all tracking targets in the tracking queue Q; obviously, all targets in the current video frame are already in the tracking queue, and the attributes of each target in the tracking queue Q: target ID and the position coordinates of the target enclosing rectangular box, that is, a target annotation for the current video frame.

[0151] Figure 3 (b) shows the annotation result of a video frame of a surveillance video at time t2, relative to Figure 3 At time t1 shown in (a), the target with target ID 4 is a moving target that has newly entered the monitoring screen.

[0152] Figure 6 The result of automatic annotation of video frames at multiple moments. In actual situations, to generate image sample data sets, it is usually necessary to perform annotation across multiple frames in order to collect different postures of the same moving target. In this technical solution, the time interval between two adjacent annotations can be set by configuring the parameter FN. For example, if FN is set to 20, the system will automatically annotate every 20 video frames.

[0153] S10, check whether the video has ended, if not, go to step S5, otherwise the process ends.

[0154] The beneficial effects of the present invention are:

[0155] The present invention can help image algorithm researchers to perform automatic image annotation and generate required sample data sets; compared with existing image annotation methods, the present invention does not require manual participation in the automatic annotation process, greatly improving the efficiency of image annotation for moving targets; automatically detects the same moving target appearing in multiple consecutive videos and assigns the same identification ID; can help algorithm researchers to perform video-based moving target annotation and quickly generate image sample data sets; its advantages include: the annotation process is unsupervised and is truly automatic annotation of moving targets; the annotation is designed based on a multi-target tracking algorithm, which effectively utilizes the spatiotemporal correlation characteristics of moving targets between consecutive video frames and has a high accuracy rate; an adaptive weight target matching algorithm is designed for the case of target occlusion, which effectively improves the target matching success rate and accuracy.

[0156] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for automatically labeling moving targets in surveillance videos, characterized in that: The following steps are involved: S1, input the first video frame and record the timestamp t1 of the video frame; S2, use the pre-trained deep convolutional neural network to detect objects in video frames; In step S2, the YoloV5 convolutional neural network is used to detect the target of the input video frame. The system sets the following six attributes for each detected target: target ID, target category pedestrian, target initial time, target update time, position coordinates of the target enclosing rectangle, and target feature vector; The target initial time is the timestamp of the video frame where the target appears for the first time, the target update time is the timestamp of the video frame where the target appears again, the position coordinates of the target enclosing rectangular box are the offset values ​​of the upper left corner and lower right corner of the rectangular box relative to the pixels in the upper left corner of the screen in the row and column directions, and the target feature vector is the image feature describing the target, and the feature vectors of different targets are different; For each target, its attributes are assigned as follows: The target ID is set to empty; The target initial time is set to the timestamp t1 of the current video frame; The target update time is set to the timestamp t1 of the current video frame; The position coordinates of the target enclosing rectangle are set to (left, top, right, bottom), where (left, top) is the position coordinates of the upper left corner of the enclosing rectangle, and (right, bottom) is the position coordinates of the lower right corner of the enclosing rectangle; The target feature vector is set to a normalized 128-dimensional feature vector obtained by applying another pre-trained convolutional neural network model to extract features of the target in the rectangular bounding box; S3, put the detected target into the tracking queue Q, the tracking queue Q is initially empty, and each target in the tracking queue Q is called a tracking target; S4, marking the target in the current video frame; S5, input the next video frame, and record the timestamp t2 of the video frame; S6, using a pre-trained convolutional neural network to detect objects in the video frame, and the detected object is called a detected object; In step S6, for each detected target, its six attributes are assigned as follows: The target ID is set to empty; The target initial time is set to the timestamp t2 of the current video frame; The target update time is set to the timestamp t2 of the current video frame; The position coordinates of the target enclosing rectangle are set to (left, top, right, bottom), where (left, top) is the position coordinates of the upper left corner of the enclosing rectangle, and (right, bottom) is the position coordinates of the lower right corner of the enclosing rectangle; The target feature vector is set to a normalized 128-dimensional feature vector obtained by applying another pre-trained convolutional neural network model to extract features of the target in the rectangular bounding box; S7, pairwise matching of each detected target in the video frame at time t2 with each tracked target in the tracking queue Q; S8. Update the tracking queue according to the matching result; S9, marking the target in the current video frame; S10, check whether the video has ended, if not, go to step S5, otherwise the process ends.

2. The method for automatically labeling moving objects in surveillance videos according to claim 1, characterized in that: In step S4, the operations performed are: Set a globally unique target ID for each target in the tracking queue Q; Save the current video frame image and all tracking targets in the tracking queue Q; obviously, all targets in the current video frame are already in the tracking queue, and the attributes of each target in the tracking queue Q: target ID and the position coordinates of the target enclosing rectangular box, that is, a target annotation for the current video frame.

3. The method for automatically labeling moving objects in surveillance videos according to claim 1, characterized in that: In step S7, the matching operation is to calculate the similarity of the feature vectors between the detection target and the tracking target, and the cosine distance is used to characterize the similarity; assuming that the tracking target and the detection target are represented by ID1 and ID2 respectively, the cosine distance of the feature vectors of the two is: Dist(ID2,ID1)= (1) In formula (1), Dist(ID2, ID1) represents the cosine distance between the target feature vectors of ID2 and ID1. The normalized 128-dimensional feature vector obtained by applying the convolutional neural network to track target ID1, A vector obtained by transposing the normalized 128-dimensional feature vector obtained by applying a convolutional neural network to detect target ID2; The scheme sets the feature vector distance threshold Dist_Threshold, and Dist_Threshold satisfies: 0≤Dist_Threshold≤2; the default value of Dist_Threshold is 0.

1. In actual use, this threshold can be changed as needed to adjust the accuracy of target matching; For the detection target ID2 and the tracking target ID1, the judgment rules are: If Dist(ID2, ID1) ≤ Dist_Threshold, it is considered that the detected target ID2 matches the tracked target ID1 successfully; If Dist(ID2, ID1)>Dist_Threshold, it is considered that the detection target ID2 fails to match the tracking target ID1; If a detection target can be successfully matched with multiple tracking targets, the tracking target with the smallest cosine distance is selected as the best match; the successfully matched detection target is put into the matching success queue Q1, and it is associated with the target ID of the best matching successful tracking target; If a detection target fails to match multiple tracking targets, the tracking target with the smallest cosine distance is selected as the best failed matching target; the failed matching detection target is placed in the failed matching queue Q2 and associated with the target ID of the best failed matching tracking target.

4. The method for automatically labeling moving objects in surveillance videos according to claim 1, characterized in that: For detection targets that fail to match, it belongs to one of the following two situations: Case 1: the target is a moving target that enters the screen at time t2; In the second case, due to the image, the target is blocked or the light or the target posture changes, resulting in the failure of matching the same moving target that appears in the image at both time t1 and time t2.

5. The method for automatically labeling moving objects in surveillance videos according to claim 4, characterized in that: For detection targets that fail to match, the solution will apply the following rules: An entry area is set on each side of the monitoring screen. If the position of the detection target that fails to match is in the entry area, it belongs to case 1, and the detection target is classified as a newly added moving target; if the position of the detection target is in the non-entry area, it belongs to case 2; For the detection target belonging to case 2, we calculate the Mahalanobis distance Dist_mah(ID1, ID2) between the detection target ID2 and the best matching failure target ID1; Dist_mah(ID2,ID1)= (2) In formula (2), is the best matching failed target ID1's rectangular bounding box position, To detect the predicted position of target ID2, S represents the covariance matrix between the predicted position of target ID2 and the position of the best matching failed target ID1; Formula (3) is used to calculate the weighted distance C(ID2, ID1) between targets ID1 and ID2. C(ID2,ID1)=λDist(ID2,ID1)+(1-λ)Dist_mah(ID2,ID1) (3) In formula (3), the cosine distance Dist(ID2, ID1) reflects the similarity of the appearance features of targets ID1 and ID2, and Dist_mah(ID2, ID1) reflects the position deviation relationship between the predicted position of target ID1 and the detected target ID2, that is, the motion information of target ID2 between adjacent video frames; The weight λ satisfies 0≤λ≤1. The larger the λ, the more the matching depends on the similarity of appearance features. Conversely, the smaller the λ, the more the matching is biased towards motion information. Adaptively adjust the size of λ; When the system finds that there are many moving targets detected in the current picture, the occlusion probability between the targets increases. At this time, the value of λ is reduced to match the motion association that is more biased towards the target; When the system finds that there are fewer moving targets detected in the picture, the occlusion probability between the targets decreases. At this time, the value of λ is increased, and the matching is more biased towards the similarity of the target's appearance features.

6. The method for automatically labeling moving objects in surveillance videos according to claim 1, characterized in that: In step S8, the specific operation is divided into two steps: The first step is to traverse the matching success queue Q1, find the associated tracking target ID1 for each detection target ID2 in the queue, and perform the following updates on the six attributes of the tracking target ID1: The target ID remains unchanged; The target initial time remains unchanged; The target update time is set to the timestamp t2 of the current video frame; The position coordinates of the target enclosing rectangular frame are set to the position coordinates of the target enclosing rectangular frame of the detection target ID2; The target feature vector is set to the target feature vector of the detection target ID2; The second step is to traverse the failed matching queue Q2 and add all the detection targets that failed to match to the tracking queue Q. Note that the detection targets that failed to match are all moving targets that have newly entered the screen.

7. The method for automatically labeling moving objects in surveillance videos according to claim 6, characterized in that: In addition, as video frames are continuously input, the tracking targets in the tracking queue Q will accumulate more and more, causing the queue to be too long or memory overflow. In actual situations, moving targets will also leave the monitoring screen after a certain period of time. Therefore, it is necessary to delete expired moving targets in the tracking queue Q. To do this, perform the following two steps: The first step is to traverse the tracking queue Q and calculate the survival time of each tracking target IDi using formula (4): =Now_Time– (4) Among them, in formula (4) To track the survival time of the target IDi, Now_Time is the current time, The target update time for tracking target IDi; The second step is to set the tracking target expiration threshold Track_Expiration_Threshold in advance. >Track_Expiration_Threshold, the target IDi is deleted from the tracking queue Q; otherwise, the target IDi is retained.

8. The method for automatically labeling moving objects in surveillance videos according to claim 1, characterized in that: In step S9, the following actions are performed: Set a globally unique target ID value for each target whose target ID value is empty in the tracking queue Q. The object of this operation is the target that failed to match added in step S8; Save the current video frame image and all tracking targets in the tracking queue Q; obviously, all targets in the current video frame are already in the tracking queue, and the attributes of each target in the tracking queue Q: target ID and the position coordinates of the target enclosing rectangular box, that is, a target annotation for the current video frame.