An unsupervised object dense tracking method based on video colorization
By combining convolutional neural networks and SRM modules, feature extraction and label propagation are optimized. By utilizing long short-term reference frames and foreground/background segmentation mechanisms, the shortcomings of unsupervised target tracking methods in prediction accuracy and tracking quality are addressed, achieving high-precision dense target tracking.
Patent Information
- Application Number
- CN202111609449.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-12-24
- Publication Date
- 2024-11-05
- Estimated Expiration
- 2041-12-24
AI Technical Summary
Existing unsupervised target tracking methods still lag significantly behind supervised methods in terms of prediction accuracy and tracking quality. In particular, video colorization-based methods can only achieve around 40% on the F&J-mean metric, and perform poorly in scenarios with rapidly changing targets.
We design an unsupervised dense target tracking method based on video colorization. Combining convolutional neural networks and SRM modules, we optimize feature extraction and label propagation by using a feature extraction network, a dynamic adjustment module, and a target prediction module, along with long short-term reference frame groups and foreground/background segmentation mechanisms, thereby improving tracking accuracy.
It significantly improves the accuracy and quality of target tracking in various scenarios, approaching the performance of supervised methods, achieving an F&J-mean of 75% to 80%, and is better able to adapt to scenarios with drastic changes in targets.
Smart Images

Figure CN114399531B_ABST
Abstract
Description
Technical Field
[0001] The invention relates to a target intensive tracking method, and in particular to an unsupervised target intensive tracking method based on video colorization. Background Art
[0002] The target tracking problem is a widely studied problem in computer vision. However, existing target tracking often requires a large number of image labels and the target categories to be tracked are limited. Therefore, the unsupervised target tracking problem has attracted extensive attention and in-depth research in the academic community. There are four main types of solutions to the existing unsupervised target tracking problem, namely: methods based on corresponding flows, methods based on temporal cycle consistency, methods based on multi-grid prediction filters, and methods based on video colorization. The present invention is based on the method of video colorization. The general process of this type of method is: given a frame in a video, traverse each pixel point p in the frame. i , find the point set most similar to the point in a specific area of the reference frame or reference frame set {q j}, with the similarity between the points in the point set and this point as the weight w ij , weighted sum of the label values of the points in the point set That is, the predicted label value of each pixel in the frame is obtained The prediction accuracy of this type of method is generally no more than 40% (F&J-mean index), but it has been improved to about 65% by MAST, and the tracking effect has been qualitatively improved, reaching the highest level of similar algorithms, but there is still a certain gap from the supervised target tracking method, such as PReMVOS, which can reach 75% to 80%. Therefore, there is still room for improvement in unsupervised target tracking methods. Summary of the invention
[0003] The purpose of the present invention is to provide an unsupervised target intensive tracking method based on video colorization, which can achieve unsupervised target intensive tracking with high prediction accuracy and high tracking quality for a preset target object.
[0004] To achieve the above functions, the present invention designs an unsupervised target dense tracking method based on video colorization, and obtains a target tracking model according to the following steps S1 to S5, and then applies the target tracking model to complete the tracking of the preset target object;
[0005] S1. Obtaining video sample frames arranged in time order and respectively containing preset target objects;
[0006] S2. Based on the convolutional neural network and the SRM module, a feature extraction network is constructed with the video sample frame as input and the feature map corresponding to the video sample frame as output, wherein the first cycle of the feature extraction network outputting the feature map is T1;
[0007] S3. Based on the feature extraction network outputting feature graphs in real time, the feature graphs output in real time by the feature extraction network are used as real-time input, and a feature graph group consisting of a preset number of feature graphs in all feature graphs obtained in the second period is used as output to construct a dynamic adjustment module, wherein the second period of the dynamic adjustment module outputting the feature graph group is T2, and T2>T1;
[0008] S4. Based on the feature map output in real time by the feature extraction network and the feature map group output by the dynamic adjustment module, a reference frame group is constructed, and a target prediction module is constructed with the reference frame group as input and the position parameters and area parameters of the preset target object in the next frame feature map of the feature map output in real time by the feature extraction network as output;
[0009] S5. Based on the feature extraction network, dynamic adjustment module and target prediction module, the target tracking model is constructed by taking the video sample frame as input and the position parameters and area parameters of the preset target object in the next frame feature map of the feature extraction network output in real time as output.
[0010] As a preferred technical solution of the present invention: the feature extraction network, the dynamic adjustment module, and the target prediction module are connected in series in sequence, the feature extraction network outputs feature maps to the dynamic adjustment module in real time with a first period T1, the dynamic adjustment module receives feature maps in real time with a first period T1, and outputs a feature map group consisting of a preset number of feature maps from all the obtained feature maps to the target prediction module with a second period T2, a reference frame group is constructed with the feature map group output by the dynamic adjustment module, and it is used as the input of the target prediction module, wherein the reference frame group includes the latest frame feature map output by the feature extraction network in real time and received by the dynamic adjustment module.
[0011] As a preferred technical solution of the present invention: the feature extraction network, the dynamic adjustment module, and the target prediction module are connected to each other in pairs, the feature extraction network outputs feature maps to the dynamic adjustment module and the target prediction module in real time at a first period T1, the dynamic adjustment module receives feature maps in real time at T1, and outputs a feature map group consisting of a preset number of feature maps from all the obtained feature maps to the target prediction module at a second period T2, the feature map of the latest frame output by the feature extraction network to the target prediction module in real time at the first period T1, and the feature map group output by the dynamic adjustment module to the target prediction module at the second period T2 are jointly constructed into a reference frame group as the input of the target prediction module.
[0012] As a preferred technical solution of the present invention: in step S2, the SRM module is combined with each residual block of the convolutional neural network to adjust the weights of each channel of the feature map output by the feature extraction network, and the specific steps are as follows:
[0013] S21: The convolutional neural network takes the video sample frame as input and takes the feature map of dimension C×H×W as output, where C is the number of channels of the feature map, H is the length of the feature map, and W is the width of the feature map. The standard deviation, mean value, maximum value, and entropy of each channel of the feature map are calculated to obtain a matrix of dimension C×4.
[0014] S22: Based on the matrix with a dimension of C×4, perform a 1×1 convolution on it to obtain a C×1 weight vector, adjust the weights of each channel of the feature map according to the weight vector, and use the adjusted feature map as the output of the feature extraction network.
[0015] As a preferred technical solution of the present invention: the reference frame group includes a long-term reference frame group and a short-term reference frame group, and step S4 specifically includes the following steps:
[0016] S41: The next frame feature map of the feature map output by the feature extraction network in real time is used as the query frame, and the frame number is the t-th frame. Each pixel point belonging to the preset target object in each feature map corresponds to a label one by one. The number of labels in the feature map output by the feature extraction network in real time is compared with the number of labels in the feature map of the previous frame. If the change rate is greater than or equal to the preset value, the t-1, t-2, and t-3 frame feature maps are selected to construct a short-term reference frame group. If the change rate of the number of labels is less than the preset value, the t-1, t-3, and t-5 frame feature maps are selected to construct a short-term reference frame group.
[0017] When the short-term reference frame group is the feature maps of the t-1st, t-2nd, and t-3rd frames, along the historical time direction, a preset number of feature maps before the feature map of the t-3rd frame are used as candidate long-term reference frames;
[0018] When the short-term reference frame group is the feature graphs of the t-1st, t-3rd, and t-5th frames, along the historical time direction, a preset number of feature graphs before the feature graph of the t-5th frame are used as candidate long-term reference frames;
[0019] S42: taking a pixel at a preset position in the query frame as a query pixel, searching for a pixel belonging to a preset target object among candidate pixels in each candidate long-term reference frame, and marking pixels with the same label among the query pixels as pixels belonging to the preset target object according to their corresponding labels. The result obtained by this process is a long-term memory;
[0020] Among them, the candidate pixel points are constructed based on the expansion rate dil, and the expansion rate dil is as follows:
[0021]
[0022] In the formula, C t-1 is the centroid of the feature map of the previous frame of the query frame, C ris the centroid of the reference frame of the frame, H is the height of the feature map, W is the width of the feature map, and the pixels within the range corresponding to the expansion rate are candidate pixels;
[0023] S43: taking the coordinates of each query pixel in the query frame corresponding to the coordinates in each short-term reference frame as the center and 15 pixels as the side length, respectively dividing square areas, searching for pixels belonging to the preset target object in each square area, and marking the pixels with the same label among the query pixels as pixels belonging to the preset target object according to their corresponding labels. The result obtained by this process is the short-term memory;
[0024] S44: Based on the overlap degree parameters IoU and ratio of the long-term memory and the short-term memory, a reference frame group for inputting into the target prediction module is selected, where IoU and ratio are as follows:
[0025]
[0026]
[0027] Where l_m is the pixel point marked as belonging to the preset target object in the long-term memory, and s_m is the pixel point marked as belonging to the preset target object in the short-term memory;
[0028] The candidate long-term reference frames whose ratio values are greater than 0.9 and less than 1.05 are used as the long-term reference frame group, and the reference frame group is constructed as follows:
[0029] (1) When IoU>0.9, the reference frame group is constructed using the long-term reference frame group and the feature map of the t-3th frame;
[0030] (2) When 0.6≤IoU≤0.9, the reference frame group is constructed using the long-term reference frame group and the short-term reference frame group;
[0031] (3) When IoU is less than 0.6, a long-term reference frame group is reselected according to a preset rule, and a reference frame group is constructed using the reselected long-term reference frame group and the short-term reference frame group.
[0032] As a preferred technical solution of the present invention: the preset value of the tag quantity change rate is 10%.
[0033] As a preferred technical solution of the present invention: after reselecting the long-term reference frame group in step S44, based on the grabcut method, foreground and background segmentation is performed on each long-term reference frame according to long-term memory and short-term memory.
[0034] Beneficial effects: Compared with the prior art, the advantages of the present invention include:
[0035] The present invention designs an unsupervised target dense tracking method based on video colorization, integrates the SRM module into the residual module of the feature extraction network, and retrains the feature extraction network to enhance the network's ability to extract features; then combines the dynamic adjustment reference frame mechanism and the foreground and background segmentation mechanism, selects the appropriate reference frame according to the results of relevant parameters, propagates labels in the query frame, and is more adaptable to tracking scenarios with drastic changes in targets, and reduces the situation where labels are scattered on the background; overall, the model can improve the accuracy of target tracking in various scenarios. BRIEF DESCRIPTION OF THE DRAWINGS
[0036] Figure 1 is a schematic diagram of an SRM module provided according to an embodiment of the invention;
[0037] Figure 2 is a schematic diagram of a residual module combined with an SRM module provided according to an embodiment of the invention;
[0038] Figure 3 is a flow chart of a mechanism for dynamically adjusting reference frames provided according to an embodiment of the invention;
[0039] Figure 4 It is a schematic diagram of a foreground and background segmentation mechanism based on a grabcut method provided according to an embodiment of the invention. DETAILED DESCRIPTION
[0040] The present invention will be further described below in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and cannot be used to limit the protection scope of the present invention.
[0041] The present invention provides an unsupervised target intensive tracking method based on video colorization, which obtains a target tracking model according to the following steps S1 to S5, and then applies the target tracking model to complete tracking of a preset target object;
[0042] S1. Obtaining video sample frames arranged in time order and respectively containing preset target objects;
[0043] S2. Based on the convolutional neural network and the SRM module, a feature extraction network is constructed with the video sample frame as input and the feature map corresponding to the video sample frame as output, wherein the first cycle of the feature extraction network outputting the feature map is T1;
[0044] refer to Figure 1 , Figure 2 , combine the SRM module with each residual block of the convolutional neural network to adjust the weights of each channel of the feature extraction network output feature map, where the convolutional neural network can use ResNet-18 and ResNet-50. The specific steps are as follows:
[0045] S21: The original convolution part of the convolutional neural network remains unchanged. The convolutional neural network takes the video sample frame as input and the feature map of dimension C×H×W as output, where C is the number of channels of the feature map, H is the length of the feature map, and W is the width of the feature map. The standard deviation, mean, maximum value, and entropy of each channel of the feature map are calculated to obtain a matrix of dimension C×4, which is called the style matrix. For the feature map based on its channel, the entropy value is calculated according to the following formula:
[0046]
[0047] Where χ is an interval set, consisting of 20 small intervals ranging from -1 to 1 with a step size of 0.1. p(x) is the probability that the eigenvalue in X falls in the interval x, where 0<p(x)<1. The functions of standard deviation, mean and maximum value are differentiable, while the calculation function of entropy is not differentiable, so the derivative of entropy needs to be customized:
[0048]
[0049] Since p(x) is a discrete function that cannot be differentiated, we heuristically set it to be when the eigenvector X is uniformly distributed. The value of is 1, then p(x) = 0.05, p′(x) = 0.025, and the derivative of the entropy solution function is as follows:
[0050]
[0051] S22: Based on the matrix with a dimension of C×4, perform a 1×1 convolution on it to obtain a C×1 weight vector, adjust the weights of each channel of the feature map according to the weight vector, and use the adjusted feature map as the output of the feature extraction network.
[0052] S3. Based on the feature extraction network outputting feature graphs in real time, the feature graphs output in real time by the feature extraction network are used as real-time input, and a feature graph group consisting of a preset number of feature graphs in all feature graphs obtained in the second period is used as output to construct a dynamic adjustment module, wherein the second period of the dynamic adjustment module outputting the feature graph group is T2, and T2>T1;
[0053] In one embodiment, a feature extraction network, a dynamic adjustment module, and a target prediction module are connected in series in sequence. The feature extraction network outputs feature maps to the dynamic adjustment module in real time with a first period T1. The dynamic adjustment module receives feature maps in real time with a first period T1, and outputs a feature map group consisting of a preset number of feature maps from all the obtained feature maps to the target prediction module with a second period T2. A reference frame group is constructed with the feature map group output by the dynamic adjustment module and used as an input to the target prediction module, wherein the reference frame group includes the latest frame of feature maps output in real time by the feature extraction network and received by the dynamic adjustment module.
[0054] In one embodiment, the feature extraction network, the dynamic adjustment module, and the target prediction module are interconnected in pairs. The feature extraction network outputs feature maps to the dynamic adjustment module and the target prediction module in real time at a first period T1. The dynamic adjustment module receives feature maps in real time at T1, and outputs a feature map group consisting of a preset number of feature maps from all the obtained feature maps to the target prediction module at a second period T2. The latest frame feature map output by the feature extraction network to the target prediction module in real time at the first period T1 and the feature map group output by the dynamic adjustment module to the target prediction module at the second period T2 are jointly constructed into a reference frame group as the input of the target prediction module.
[0055] S4. Based on the feature map output in real time by the feature extraction network and the feature map group output by the dynamic adjustment module, a reference frame group is constructed, and a target prediction module is constructed with the reference frame group as input and the position parameters and area parameters of the preset target object in the next frame feature map of the feature map output in real time by the feature extraction network as output;
[0056] The reference frame group includes a long-term reference frame group and a short-term reference frame group. Step S4 specifically includes the following steps:
[0057] S41: The next frame feature map of the feature map output by the feature extraction network in real time is used as the query frame, and the frame number is the t-th frame. Each pixel point belonging to the preset target object in each feature map corresponds to a label one by one. The number of labels in the feature map output by the feature extraction network in real time is compared with the number of labels in the feature map of the previous frame. If the change rate is greater than or equal to the preset value, the t-1, t-2, and t-3 frame feature maps are selected to construct a short-term reference frame group. If the change rate of the number of labels is less than the preset value, the t-1, t-3, and t-5 frame feature maps are selected to construct a short-term reference frame group.
[0058] When the short-term reference frame group is the feature maps of the t-1st, t-2nd, and t-3rd frames, along the historical time direction, a preset number of feature maps before the feature map of the t-3rd frame are used as candidate long-term reference frames;
[0059] When the short-term reference frame group is the feature graphs of the t-1st, t-3rd, and t-5th frames, along the historical time direction, a preset number of feature graphs before the feature graph of the t-5th frame are used as candidate long-term reference frames;
[0060] S42: taking a pixel at a preset position in the query frame as a query pixel, searching for a pixel belonging to a preset target object among candidate pixels in each candidate long-term reference frame, and marking pixels with the same label among the query pixels as pixels belonging to the preset target object according to their corresponding labels. The result obtained by this process is a long-term memory;
[0061] Among them, the centroid is used to represent the position of the preset target object, which mainly affects the expansion rate in the long-term memory. The expansion rate represents the spacing between the horizontal and vertical coordinates of adjacent candidate pixel points. The larger the spacing, the larger the receptive field. When the value is 4, it basically covers the entire reference frame. When the position of the preset target object in the query frame deviates farther from the position of the preset target object in the query frame, the greater the expansion rate. The candidate pixel points are constructed based on the expansion rate dil. The expansion rate dil is as follows:
[0062]
[0063] In the formula, C t-1 is the centroid of the feature map of the preset target object in the frame before the query frame, C r is the centroid of the preset target object in the reference frame, H is the height of the feature map, W is the width of the feature map, and the pixels within the range corresponding to the expansion rate are candidate pixels;
[0064] S43: taking the coordinates of each query pixel in the query frame corresponding to the coordinates in each short-term reference frame as the center and 15 pixels as the side length, respectively dividing square areas, searching for pixels belonging to the preset target object in each square area, and marking the pixels with the same label among the query pixels as pixels belonging to the preset target object according to their corresponding labels. The result obtained by this process is the short-term memory;
[0065] During the experiment, it was found that according to the long-term memory prediction results, C t-1 and C r It is closer to the true center of mass because the target is closer to the center of mass than the actual center of mass. r Move to C t-1 When the label is scattered on the background, if dil=0, it is no different from the prediction process of short-term memory. Therefore, the minimum value of dil is 1. Considering the problem of centroid offset, in one embodiment, 2≤dil≤4 is set. The specific value is determined by the ratio of the centroid offset to the maximum offset of the image.
[0066] S44: The similarity between the long-term reference frame and the query frame is measured by the overlap between the long-term memory and the short-term memory. Ideally, the prediction results of the long-term and short-term memory should be completely consistent, but due to the displacement and deformation of the preset target object, the results of the long-term memory are often quite different from those of the short-term memory. Therefore, selecting a suitable long-term reference frame is beneficial to improving the tracking quality. Based on the overlap parameters IoU and ratio of the long-term and short-term memory, select the reference frame group for input into the target prediction module, where IoU and ratio are as follows:
[0067]
[0068]
[0069] Where l_m is the pixel point marked as belonging to the preset target object in the long-term memory, and s_m is the pixel point marked as belonging to the preset target object in the short-term memory;
[0070] The candidate long-term reference frames with a ratio value greater than 0.9 and less than 1.05 are used as the long-term reference frame group. Figure 3 , construct the reference frame group as follows:
[0071] (1) When IoU>0.9, the reference frame group is constructed using the long-term reference frame group and the feature map of the t-3th frame;
[0072] (2) When 0.6≤IoU≤0.9, the reference frame group is constructed using the long-term reference frame group and the short-term reference frame group;
[0073] (3) When IoU is less than 0.6, the long-term reference frame group is reselected according to the preset rules, and the reference frame group is constructed with the reselected long-term reference frame group and the short-term reference frame group. In one embodiment, if IoU is less than 0.6, the traversal starts from the 0th frame to find a reference frame with a higher IoU value. At the same time, it is necessary to ensure that IoU and ratio are closer. Ideally, the values of the two should be the same and infinitely close to 1.0. If a reference frame that meets the conditions is found, the long-term reference frame is reset and kept unchanged in the next prediction. The short-term reference frame is added for the final prediction. If no reference frame that meets the conditions is found, it means that the target has changed significantly, and frames as close as possible should be used, and finally determined as t-1 frame and t-3 frame.
[0074] After reselecting the long-term reference frame group, the foreground and background of each long-term reference frame are segmented based on the grabcut method according to the long-term memory and short-term memory.
[0075] The initial long-term reference frame, frame 0, does not have the problem of error accumulation. When the dynamic adjustment mechanism reselects the long-term reference frame, the selected result may have the accumulated errors in the previous stage prediction process, resulting in a decrease in prediction quality, or even misjudging the background that obviously does not belong to the preset target object as the preset target object. Therefore, when the dynamic adjustment mechanism reselects the long-term reference frame, it is necessary to perform foreground and background segmentation to remove labels that obviously do not belong to the preset target object. At the same time, the prediction result can also be smoothed to make the edge of the prediction result more compatible with the preset target object.
[0076] The present invention adopts the grabcut method to perform foreground and background segmentation. The grabcut method needs to set initial labels, which are background, foreground, possible background and possible foreground, which are represented by 0, 1, 2 and 3. During the operation of the method, the color mode of the foreground and background is generated according to the initially marked foreground and background, and then the color mode is used to determine whether the pixel marked as possible foreground or possible background is foreground or background.
[0077] refer to Figure 4 ,The label initialization method is as follows: mark the overlapping part in long-term memory and short-term memory as foreground; mark the overlapping background part in long and short-term memory as background; mark the objects that are foreground in long-term memory and background in short-term memory as possible background; mark the part that is foreground in short-term memory and background in long-term memory as possible background.
[0078] Considering the deformation and displacement of the preset target object, the pixels near the foreground and possible foreground, although marked as background, may still belong to the preset target object. Therefore, the pixels marked as foreground and possible foreground are grown, and the background and possible background pixels in a square area with a side length of 15 and with it as the center point are marked as possible foreground. After completing the label initialization based on the grabcut method, run the grabcut method and directly replace the original prediction result with the result.
[0079] S5. Based on the feature extraction network, the dynamic adjustment module, and the target prediction module, the video sample frame is used as input, and the position parameter and area parameter of the preset target object in the next frame feature map of the feature extraction network output in real time are used as output to construct a target tracking model. In one embodiment, the area of the preset target object is represented by the number of tags belonging to the preset target object in the query frame, and the position of the preset target object is represented by the center of mass of the preset target object in the query frame.
[0080] In one embodiment, the preset value of the tag quantity change rate is 10%.
[0081] The embodiments of the present invention are described in detail above with reference to the accompanying drawings, but the present invention is not limited to the above embodiments, and various changes can be made within the knowledge scope of ordinary technicians in this field without departing from the purpose of the present invention.
Claims
1. An unsupervised target dense tracking method based on video colorization, characterized in that: According to the following steps S1 to S5, a target tracking model is obtained, and then the target tracking model is applied to complete tracking of a preset target object; S1. Obtaining video sample frames arranged in time order and respectively containing preset target objects; S2. Based on the convolutional neural network and the SRM module, a feature extraction network is constructed with the video sample frame as input and the feature map corresponding to the video sample frame as output, wherein the first cycle of the feature extraction network outputting the feature map is T1; In step S2, the SRM module is combined with each residual block of the convolutional neural network to adjust the weights of each channel of the feature map output by the feature extraction network. The specific steps are as follows: S21: The convolutional neural network takes the video sample frame as input and takes a feature map of dimension C×H×W as output, where C is the number of channels of the feature map, H is the length of the feature map, and W is the width of the feature map. The standard deviation, mean value, maximum value, and entropy of each channel of the feature map are calculated to obtain a matrix of dimension C×4; S22: Based on the matrix with a dimension of C×4, perform a 1×1 convolution on it to obtain a C×1 weight vector, adjust the weights of each channel of the feature map according to the weight vector, and use the adjusted feature map as the output of the feature extraction network; S3. Based on the feature extraction network outputting feature graphs in real time, the feature graphs output in real time by the feature extraction network are used as real-time input, and the feature graph group consisting of a preset number of feature graphs in all feature graphs obtained in the second period is used as output to construct a dynamic adjustment module, wherein the second period of the dynamic adjustment module outputting the feature graph group is T2, and T2>T1; The feature extraction network, the dynamic adjustment module, and the target prediction module are sequentially connected in series, the feature extraction network outputs a feature map to the dynamic adjustment module in real time at a first period T1, the dynamic adjustment module receives the feature map in real time at a first period T1, and outputs a feature map group consisting of a preset number of feature maps from all the obtained feature maps to the target prediction module at a second period T2, a reference frame group is constructed with the feature map group output by the dynamic adjustment module, and the reference frame group is used as an input to the target prediction module, wherein the reference frame group includes the latest frame of feature map output by the feature extraction network in real time and received by the dynamic adjustment module; The feature extraction network, the dynamic adjustment module, and the target prediction module are connected to each other in pairs. The feature extraction network outputs feature maps to the dynamic adjustment module and the target prediction module in real time at a first period T1. The dynamic adjustment module receives feature maps in real time at T1, and outputs a feature map group consisting of a preset number of feature maps from all the obtained feature maps to the target prediction module at a second period T2. The feature map of the latest frame output by the feature extraction network to the target prediction module in real time at the first period T1 and the feature map group output by the dynamic adjustment module to the target prediction module at the second period T2 jointly construct a reference frame group as an input of the target prediction module. S4. Based on the feature map output in real time by the feature extraction network and the feature map group output by the dynamic adjustment module, a reference frame group is constructed, and a target prediction module is constructed with the reference frame group as input and the position parameters and area parameters of the preset target object in the next frame feature map of the feature map output in real time by the feature extraction network as output; S5. Based on the feature extraction network, dynamic adjustment module and target prediction module, the target tracking model is constructed by taking the video sample frame as input and the position parameters and area parameters of the preset target object in the next frame feature map of the feature extraction network output in real time as output.
2. The unsupervised target intensive tracking method based on video colorization as claimed in claim 1, characterized in that: The reference frame group includes a long-term reference frame group and a short-term reference frame group. Step S4 specifically includes the following steps: S41: Take the next frame feature map of the feature map output by the feature extraction network in real time as the query frame, whose frame number is the t-th frame, correspond each pixel point belonging to the preset target object in each feature map to a label, compare the number of labels in the feature map output by the feature extraction network in real time with the number of labels in the previous frame feature map, and if the change rate is greater than or equal to the preset value, then select the t-1, t-2, and t-3 frame feature maps to construct a short-term reference frame group, and if the change rate of the number of labels is less than the preset value, then select the t-1, t-3, and t-5 frame feature maps to construct a short-term reference frame group; When the short-term reference frame group is the feature maps of the t-1st, t-2nd, and t-3rd frames, along the historical time direction, a preset number of feature maps before the feature map of the t-3rd frame are used as candidate long-term reference frames; When the short-term reference frame group is the feature graphs of the t-1st, t-3rd, and t-5th frames, along the historical time direction, a preset number of feature graphs before the feature graph of the t-5th frame are used as candidate long-term reference frames; S42: taking a pixel at a preset position in the query frame as a query pixel, searching for a pixel belonging to a preset target object among the candidate pixels in each candidate long-term reference frame, and marking the pixel with the same label among the query pixel as a pixel belonging to the preset target object according to its corresponding label. The result obtained by this process is a long-term memory; Among them, the candidate pixel points are constructed based on the expansion rate dil, and the expansion rate dil is as follows: In the formula, C t-1 is the centroid of the feature map of the previous frame of the query frame, C r is the centroid of the reference frame of the frame, H is the height of the feature map, W is the width of the feature map, and the pixels within the range corresponding to the expansion rate are candidate pixels; S43: taking the coordinates of each query pixel in the query frame corresponding to the coordinates in each short-term reference frame as the center and 15 pixels as the side length, respectively divide the square area, search for pixels belonging to the preset target object in each square area, and mark the pixels with the same label among the query pixels as pixels belonging to the preset target object according to their corresponding labels. The result obtained by this process is the short-term memory; S44: Based on the overlap parameters IoU and ratio of the long-term memory and the short-term memory, a reference frame group for inputting into the target prediction module is selected, where IoU and ratio are as follows: Where l_m is the pixel point marked as belonging to the preset target object in the long-term memory, and s_m is the pixel point marked as belonging to the preset target object in the short-term memory; The candidate long-term reference frames whose ratio values are greater than 0.9 and less than 1.05 are used as the long-term reference frame group, and the reference frame group is constructed as follows: (1) When IoU>0.9, the reference frame group is constructed using the long-term reference frame group and the t-3th frame feature map; (2) When 0.6≤IoU≤0.9, the reference frame group is constructed using the long-term reference frame group and the short-term reference frame group; (3) When IoU<0.6, the long-term reference frame group is reselected according to the preset rule, and the reference frame group is constructed with the reselected long-term reference frame group and the short-term reference frame group.
3. The unsupervised target intensive tracking method based on video colorization as claimed in claim 2, characterized in that: The preset value of the tag quantity change rate is 10%.
4. The unsupervised target intensive tracking method based on video colorization as claimed in claim 2, characterized in that: After reselecting the long-term reference frame group in step S44, based on the grabcut method, foreground and background segmentation is performed on each long-term reference frame according to long-term memory and short-term memory.
