A flying target tracking method and system based on contrastive learning

Abstract

The present invention provides a flight target tracking method and system based on contrastive learning, which belongs to the field of computer vision technology, including: obtaining a flight target image sequence under different climate and illumination conditions; obtaining a negative sample image set, a positive sample image set and an anchor point sample; constructing a triple twin network; and adjusting the parameters of the triple twin network to obtain a flight target depth feature extractor; after obtaining the flight target depth features corresponding to the flight target template image and the adjacent background area image, classifier training is performed; according to the remaining flight target images to be detected, the positions of the flight target template images, the flight target depth feature extractor and the online classifier, the flight target is classified online to obtain the motion trajectory of the flight target to be detected. The present invention can ensure the accuracy of tracking and positioning of the flight target without the need to label massive flight target image data.

Description

Technical Field

[0001] The invention belongs to the technical field of computer vision, and in particular relates to a flying target tracking method and system based on contrastive learning. Background Art

[0002] As a basic research content in the field of computer vision, target tracking technology has been greatly developed in practical application fields such as video surveillance, robotics and line of sight tracking. In order to achieve long-term target tracking with sky background images, the target tracking algorithm needs to solve the problems of target motion blur, occlusion and background interference, as well as the complex background changes caused by various factors such as meteorological and lighting changes. Since it is difficult to accurately detect moving targets in extremely severe rainy and snowy weather, traditional Bayesian posterior estimation methods represented by Kalman filters are not competent for long-term target tracking.

[0003] Existing methods such as KCF, Struck and TLD directly use the image representation of the target in the current video sequence for tracking, but it is difficult to handle relatively complex background changes. End-to-end tracking models based on deep learning use deep network training to extract deep features of the target from massive video sequences to achieve the goal. Deep trackers represented by Siamese CNN use supervised learning methods to train and model massive labeled video sequences in sequence. However, the above supervised learning methods require manual sample annotation of massive data sets, which is time-consuming and labor-intensive. Tracking methods based on semi-supervised learning achieve target tracking based on the measurement of image similarity or consistent and significant features of target motion, while collecting a small number of partial target samples. However, there are specific restrictions on the quality of the video and the category of moving targets in the video content. Summary of the invention

[0004] One of the purposes of the present invention is to provide a flight target tracking method based on contrastive learning, which can accurately distinguish between moving targets and sky backgrounds, ensure the accuracy of tracking and positioning of flight targets, and does not require annotation processing of flight target data. It is not limited by image quality and flight target category and has strong applicability.

[0005] A second object of the present invention is to provide a flying target tracking system based on contrastive learning.

[0006] In order to achieve one of the above purposes, the present invention adopts the following technical solutions:

[0007] A flying target tracking method based on contrastive learning, the flying target tracking method comprising:

[0008] Step S1, acquiring a flying target image sequence under different climate and lighting conditions;

[0009] Step S2, extracting the effective moving target image and background image of each flying target image from the flying target image sequence; and using the background image set corresponding to each effective moving target image as a negative sample image set;

[0010] Step S3, performing data enhancement processing on each valid moving target image as a positive sample image set; and using each valid moving target image as an anchor point sample;

[0011] Step S4, using three backbone networks with the same structure to construct a triple twin network; inputting the anchor point samples, the corresponding positive sample image set and the negative sample image set into the triple twin network for parameter adjustment, and using one of the backbone networks after parameter adjustment as a flight target deep feature extractor;

[0012] Step S5, obtaining a sequence of flying target images to be detected; and performing target image and background image recognition on an initial flying target image to be detected in the sequence of flying target images to be detected, so as to obtain a flying target template image in the initial flying target image to be detected and its position and an adjacent background area image;

[0013] Step S6, inputting the flight target template image and the adjacent background area image into the flight target depth feature extractor to obtain the flight target depth features corresponding to the flight target template image and the adjacent background area image respectively;

[0014] Step S7, inputting the flight target depth features corresponding to the flight target template image and the adjacent background area image into an online classifier for classification training;

[0015] Step S8, performing online classification of the flying targets according to the remaining flying target images to be detected in the flying target image sequence to be detected, the position of the flying target template image, the flying target depth feature extractor and the trained online classifier to obtain the motion trajectory of the flying target to be detected.

[0016] Furthermore, in step S2, the specific process of the extraction includes:

[0017] Step S21, performing motion pixel point detection and scale-invariant position feature point detection on adjacent flight target images in the initial stage of the flight target image sequence, and then performing position matching to obtain a position feature point pair set;

[0018] Step S22, calculating the Euclidean distance of the pixel positions of each pair of position feature point pairs in the position feature point pair set to determine the position feature points belonging to the target motion area;

[0019] Step S23, determining a valid motion region template of the flying target in the flying target image according to the position feature points in the target motion region;

[0020] Step S24, according to the motion region template of the flying target in the flying target image, the subsequent flying target images are sequentially subjected to motion region template matching and scale transformation processing to obtain a valid motion target image in each flying target image;

[0021] Step S25: randomly select a non-effective moving target area from each flying target image and perform a scale transformation process to obtain a background image in each flying target image.

[0022] Furthermore, in step S3, the data enhancement includes one or a combination of color transformation, affine transformation, scale transformation, rotation transformation, sharpening, blurring and morphological processing; the specific process of the morphological processing includes:

[0023] Step S31, after processing each valid moving target image into a black and white image, performing horizontal convolution operation and vertical convolution operation respectively, to obtain the contour gradient of the flying target in each valid moving target image;

[0024] Step S32, calculating the maximum circumscribed rectangle of the contour gradient of the flying target in each of the valid moving target images;

[0025] Step S33, judging whether all the pixel points in each maximum circumscribed rectangle belong to the contour of the flying target, so as to obtain each circumscribed matrix;

[0026] Step S34, performing singular decomposition on each circumscribed matrix to obtain the tangent vector direction and the normal vector direction corresponding to the maximum eigenvalue of each circumscribed matrix;

[0027] Step S35, determining the tangent vector direction and the normal vector direction corresponding to the maximum eigenvalue of each circumscribed matrix to segment the contours of each flying target;

[0028] Step S36: perform corrosion / dilation and edge convolution processing on each flying target contour in turn.

[0029] Furthermore, the triple twin network includes three backbone networks and a ternary contrast loss module connected in parallel; the input end of the ternary contrast loss module is connected to the output ends of the three backbone networks;

[0030] The three backbone networks are used to extract image depth features from the images received respectively and then input them into the ternary contrast loss module;

[0031] The ternary contrast loss module is used to use a ternary loss function to compare the similarity of the image depth features output by the three backbone networks; and adjust the backbone network parameters of the triple twin network based on the similarity comparison structure.

[0032] Furthermore, in step S8, the specific implementation process of the online classification of the flight target includes:

[0033] Step S81, sampling the remaining flying target images to be detected in a circulant matrix manner according to the position of the flying target template image, and inputting the samples into the flying target depth feature extractor to obtain the flying target depth features of each sampling area;

[0034] Step S82, inputting the flight target depth features of each sampling area into the online classifier to obtain a similarity response map of the flight target template image in the corresponding remaining flight target images to be detected;

[0035] Step S83, performing noise suppression processing on each similarity response spectrum;

[0036] Step S84, selecting the position of the maximum peak from each similarity response spectrum after the noise suppression process as the tracking position of the flying target in the corresponding flying target image to be detected;

[0037] Step S85, judging whether all the remaining flight target images to be detected have been classified, if so, outputting the tracking positions in all the remaining flight target images to be detected, and ending; if not, taking the area where the tracking position is located as a new flight target template image, and proceeding to step S86;

[0038] Step S86, sampling the remaining flying target images to be detected in a circulant matrix manner according to the position of the flying target template image, and inputting the samples into the flying target depth feature extractor to obtain the flying target depth features of each sampling area;

[0039] Step S87, input the flight target depth features of each sampling area into the online classifier for updating, and return to step S82.

[0040] In order to achieve the second objective above, the present invention adopts the following technical solution:

[0041] A flying target tracking system based on contrastive learning, the flying target tracking system comprising:

[0042] The first acquisition module is used to acquire a sequence of flying target images under different climate and lighting conditions;

[0043] An extraction module is used to extract the effective moving target image and background image of each flying target image from the flying target image sequence; and use the background image set corresponding to each effective moving target image as a negative sample image set;

[0044] A data enhancement processing module is used to perform data enhancement processing on each valid moving target image as a positive sample image set; and each valid moving target image is used as an anchor point sample;

[0045] A parameter adjustment module is used to construct a triple twin network using three backbone networks with the same structure; and input the anchor point samples, the corresponding positive sample image set and the negative sample image set into the triple twin network for parameter adjustment, and use one of the backbone networks after parameter adjustment as a flight target deep feature extractor;

[0046] The recognition module is used to obtain a sequence of flying target images to be detected; and to perform target image and background image recognition on an initial flying target image to be detected in the sequence of flying target images to be detected, so as to obtain a flying target template image in the initial flying target image to be detected, its position, and an adjacent background area image;

[0047] A second acquisition module is used to input the flight target template image and the adjacent background area image into the flight target depth feature extractor to obtain the flight target depth features corresponding to the flight target template image and the adjacent background area image respectively;

[0048] A classification training module, used for inputting the flight target depth features corresponding to the flight target template image and the adjacent background area image into an online classifier for classification training;

[0049] The flying target online classification module is used to perform online classification of flying targets based on the remaining flying target images to be detected in the flying target image sequence to be detected, the position of the flying target template image, the flying target deep feature extractor and the trained online classifier to obtain the motion trajectory of the flying target to be detected.

[0050] Furthermore, the extraction module includes:

[0051] A position matching submodule is used to perform motion pixel point detection and scale-invariant position feature point detection on adjacent flight target images in the initial stage of the flight target image sequence, and then perform position matching to obtain a set of position feature point pairs;

[0052] A first calculation submodule is used to calculate the Euclidean distance of the pixel positions of each pair of position feature point pairs in the position feature point pair set to determine the position feature points belonging to the target motion area;

[0053] A determination submodule, used to determine a valid motion region template of the flying target in the flying target image according to position feature points belonging to the target motion region;

[0054] A first scale transformation processing submodule is used to perform motion region template matching and scale transformation processing on subsequent flight target images in sequence according to the motion region template of the flight target in the flight target image, so as to obtain a valid motion target image in each flight target image;

[0055] The second scale transformation processing submodule is used to randomly select a non-effective moving target area from each flying target image and perform scale transformation processing to obtain a background image in each flying target image.

[0056] Furthermore, the data enhancement processing module includes:

[0057] The first convolution operation submodule is used to process each valid moving target image into a black and white image and then perform horizontal convolution operation and vertical convolution operation respectively to obtain the contour gradient of the flying target in each valid moving target image;

[0058] The second calculation submodule is used to calculate the maximum circumscribed rectangle of the contour gradient of the flying target in each of the valid moving target images;

[0059] A judgment submodule is used to judge whether all the pixel points in each maximum circumscribed rectangle belong to the contour of the flying target, so as to obtain each circumscribed matrix;

[0060] A singular decomposition submodule is used to perform singular decomposition on each external matrix to obtain the tangent vector direction and the normal vector direction corresponding to the maximum eigenvalue of each external matrix;

[0061] A segmentation submodule, used to determine the tangent vector direction and the normal vector direction corresponding to the maximum eigenvalue of each circumscribed matrix, so as to segment the contour of each flying target;

[0062] The edge convolution processing submodule is used to perform corrosion / dilation and edge convolution processing on each flying target contour in turn.

[0063] Furthermore, the flight target online classification module includes:

[0064] A first sampling submodule is used to sample the remaining flying target images to be detected according to the position of the flying target template image in a circulant matrix manner, and input the samples into the flying target depth feature extractor to obtain the flying target depth features of each sampling area;

[0065] An acquisition submodule, used for inputting the flight target depth features of each sampling area into the online classifier to obtain a similarity response map of the flight target template image in the corresponding remaining flight target images to be detected;

[0066] A noise suppression processing submodule, used for performing noise suppression processing on each similarity response map;

[0067] A selection submodule is used to select the position of the maximum peak from each similarity response spectrum after noise suppression processing as the tracking position of the flying target in the corresponding flying target image to be detected;

[0068] A judgment submodule is used to judge whether all the remaining flight target images to be detected have completed classification. If so, the tracking positions in all the remaining flight target images to be detected are output and the process ends; if not, the area where the tracking position is located is used as a new flight target template image and transmitted to the second sampling submodule;

[0069] A second sampling submodule is used to sample the remaining flying target images to be detected according to the position of the flying target template image in a circulant matrix manner, and input the samples into the flying target depth feature extractor to obtain the flying target depth features of each sampling area;

[0070] The updating submodule is used to input the flight target depth features of each sampling area into the online classifier for updating, and transmit the updated online classifier to the acquiring submodule.

[0071] In summary, the solution proposed by the present invention has the following technical effects:

[0072] The present invention constructs a triple twin network, and forms anchor samples, positive samples and negative sample sets with the effective moving target images corresponding to each flying target image, the effective moving target images after data enhancement processing and the background image set; through the anchor samples, positive samples and negative sample sets, the parameter training of the triple twin network is realized, and then the flying target deep feature extractor is obtained, which ensures that the flying target deep feature extractor can accurately distinguish the moving target and the sky background, realizes the maximization of the spatial distance between the moving target and the sky background under the specified criterion, fully reduces the similarity between the moving target and the sky background, and has generalization; for the case of small size of the moving target and motion blur, the sample data is made more diverse and the network model is made more robust through sample enhancement; the flying target template image in the initial flying target image to be detected, the adjacent background area image and the remaining flying target images to be detected are used to realize the training of the classifier, which ensures the accuracy of the classification result of the online classifier, and the accuracy of the tracking and positioning of the flying target, and does not require massive data, is not affected by image quality and flying target category, and has strong applicability. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0074] Figure 1 It is a flow chart of the flight target tracking method based on contrastive learning of the present invention;

[0075] Figure 2 It is a schematic diagram of the triple twin network structure of the present invention;

[0076] Figure 3 It is a schematic diagram of the backbone network structure of the present invention. DETAILED DESCRIPTION

[0077] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0078] This embodiment provides a flying target tracking method based on contrastive learning, such as Figure 1 As shown, the flying target tracking method includes:

[0079] S1. Obtain a sequence of flying target images under different climate and lighting conditions.

[0080] In order to adapt to the complexity of the background in which the moving target is located, this embodiment collects a large number of flying target image sequences under different climate and lighting conditions.

[0081] S2. Extracting a valid moving target image and a background image of each flying target image from the flying target image sequence; and using a background image set corresponding to each valid moving target image as a negative sample image set.

[0082] In order to overcome the interference caused by camera shaking and illumination changes, the specific implementation process of obtaining the effective moving target image and background image in each flying target image in this embodiment includes:

[0083] Step S21, performing motion pixel point detection and scale-invariant position feature point detection on adjacent flight target images in the initial stage of the flight target image sequence, and then performing position matching to obtain a set of position feature point pairs.

[0084] This embodiment uses a dense optical flow method to detect moving pixels, and uses a scale-invariant feature point method (SURF, Speeded Up Robust Features.) to detect scale-invariant position feature points on moving pixels in adjacent flying target images to obtain two sets of position feature point sets. The random sampling consensus method (RANSAC RandomSample Consensus) is used to match the position feature points of the two sets of position feature point sets to obtain a position feature point pair set.

[0085] Step S22: Calculate the Euclidean distance between the pixel positions of each pair of position feature points in the position feature point pair set to determine the position feature points belonging to the target motion area.

[0086] When the Euclidean distance is greater than the threshold, the position feature point pair corresponding to the Euclidean distance greater than the threshold belongs to the target motion area; otherwise, the position feature point pair corresponding to the Euclidean distance less than the threshold does not belong to the target motion area, and the pixel position between the position feature point pair corresponding to the Euclidean distance less than the threshold is caused by camera shake.

[0087] Step S23, determining a valid motion region template of the flying target in the flying target image according to the position feature points in the target motion region;

[0088] Step S24, according to the motion region template of the flying target in the flying target image, the subsequent flying target images are sequentially subjected to motion region template matching and scale transformation processing to obtain a valid motion target image in each flying target image;

[0089] Step S25: randomly select a non-effective moving target area from each flying target image and perform a scale transformation process to obtain a background image in each flying target image.

[0090] S3. Perform data enhancement processing on each valid moving target image as a positive sample image set; and use each valid moving target image as an anchor point sample.

[0091] This embodiment performs data enhancement processing on the effective moving target image to ensure that while maintaining the basic image feature attributes of the moving target, appropriate mutation operations are performed on the image feature attributes of the moving target, thereby improving the robustness of the deep features of the moving target and achieving the effect that the deep features of the moving target and its depth features after image transformation can be consistent or approximate.

[0092] The data enhancement of this embodiment includes single or combined processing of color transformation, affine transformation, scale transformation, rotation transformation, sharpening, blur processing and morphological processing. The purpose of morphological processing is to simulate the motion blur of the flying target image. The specific process of morphological processing includes:

[0093] Step S31, after processing each valid moving target image into a black and white image, perform horizontal convolution operation and vertical convolution operation respectively to obtain the contour gradient of the flying target in each valid moving target image.

[0094] Perform adaptive threshold processing on the flying target image A to convert it into black and white image A binary Then, two 3*3 gradient convolution kernel s function operators are used to perform convolution operations on the black and white image in the horizontal and vertical directions respectively to obtain the contour gradient of the flying target.

[0095] A binary =255if A>threshold or A binary =0A <threshold;

[0096]

[0097] Step S32, calculating the maximum circumscribed rectangle of the contour gradient of the flying target in each of the valid moving target images.

[0098] The maximum circumscribed rectangle of this embodiment is calculated according to the following formula:

[0099] R A =convex(S*A binary ); (2)

[0100] Among them, R A is the maximum enclosing rectangle.

[0101] Step S33: determine whether all pixel points in each maximum circumscribed rectangle belong to the contour of the flying target to obtain each circumscribed matrix.

[0102] Traverse all pixels in the maximum circumscribed rectangle, set the pixel value belonging to the contour of the flying target to 1, otherwise set it to 0, and obtain the circumscribed matrix.

[0103] Step S34: performing singular decomposition on each circumscribed matrix to obtain the tangent vector direction and the normal vector direction corresponding to the maximum eigenvalue of each circumscribed matrix.

[0104] Step S35, determining the tangent vector direction and the normal vector direction corresponding to the maximum eigenvalue of each circumscribed matrix to segment the contours of each flying target.

[0105] The flight target contour is segmented according to the direction corresponding to the tangent vector of the vector corresponding to the maximum eigenvalue (i.e., the tangent vector direction) and the direction corresponding to the normal vector of the vector (the normal vector direction). Of course, the flight target contour can also be segmented according to the center position of the circumscribed rectangle of the flight target contour.

[0106] Step S36: perform corrosion / dilation and edge convolution processing on each flying target contour in turn.

[0107] The erosion / dilation operation of the morphological operation is used to process the contour to change the thickness of the contour. Then, a 3*3 sharpening convolution kernel or a 3*3 Gaussian convolution kernel is used on both sides of the contour segmentation line to convolve the edge of the flying target to achieve the motion blur effect of simulating the moving target. The sharpening convolution kernel and the Gaussian convolution kernel are:

[0108]

[0109] Finally, it is randomly combined with commonly used data operations such as cropping, color transformation, affine transformation and rotation transformation to ensure that the deep features of the flight target obtained by the deep feature extractor can adapt to situations such as affine, scale, color transformation and motion blur.

[0110] S4. Use three backbone networks with the same structure to build a triple twin network; input the anchor point samples, the corresponding positive sample image set and the negative sample image set into the triple twin network for parameter adjustment, and use one of the backbone networks after parameter adjustment as a flight target deep feature extractor.

[0111] The purpose of constructing the triple twin network is to generate a flying target deep feature extractor that can better distinguish the flying target area and the background area. The triple twin network consists of three convolutional neural network branches with the same structure and shared parameters. The three convolutional neural network branches use ResNet-50 as the backbone network (this network structure can effectively solve problems such as gradient explosion and network degradation), and the output result of the average pooling layer is used as the deep image information of the flight target image (i.e., image depth feature), which is expressed in vector form. Figure 2 The triple twin network includes three parallel backbone networks and a ternary contrast loss module; the input end of the ternary contrast loss module is connected to the output end of the three backbone networks. The three backbone networks are used to extract image depth features from the images received respectively and then input them into the ternary contrast loss module. The backbone network of this embodiment adopts a 50-layer residual network, and its structure is referenced Figure 3The ternary contrast loss module is used to use a ternary loss function to perform similarity comparison on the image depth features output by the three backbone networks; and to adjust the backbone network parameters of the triple twin network according to the similarity comparison structure.

[0112] For the triple twin network, the inputs of its three branches are the effective moving target image u, the enhanced effective moving target image v, and the + and background image v - , the output is the deep features of these three images represented by vectors. The training process of the triple twin network includes:

[0113] 1. Initialize the parameters of the backbone network, select a valid moving target image as an anchor point, select the data augmented image corresponding to the valid moving target image as a positive sample, and then select several background images as negative samples. The three together constitute a training batch data, and each image in the training batch is demeaned and then fed into the backbone network.

[0114] 2. In the 0th stage of the backbone network, the input image is scaled: the resolution of all image samples (including anchor points, positive samples and negative samples) is processed to 224 pixels in width and height. The normalized results are input into 64 different convolution channels in the convolution layer. Different 7*7 convolution kernels are used to perform convolution processing on the image with a stride of 2, and the convolution results are processed according to the RELU activation function. In the maximum pooling layer, a 3*3 sliding window is used, and the activation calculation results are slid with a stride of 2. The maximum element in each sliding window is selected, and the output depth information is in the first matrix form of 64*56*56.

[0115] 3. In the first stage of the backbone network, the depth information x of the dimension (64, 56, 56) output in the 0th stage is input into the convolution unit 1. The working process of the convolution unit 1 includes:

[0116] According to the input data channel C, input data height / width W, convolution channel C in the convolution layer 1 and convolution stride S, determine two sets of convolution sequences; the convolution operation sequence on the left, the calculation of the input data is expressed as F(x), the input data undergoes three convolution operations with kernel sizes of 1*1, 3*3 and 1*1, channel length of 64, and the corresponding Batch Normalization and RELU activation operations. After the last convolution, due to the convolution stride of 4*C 1, the number of channels of the output depth information is expanded by 4 times, and the width and height are kept unchanged; at the same time, in order to keep the dimension of the input depth information consistent with the dimension of the output depth information, so that the input information can be constantly mapped to the output information, in the right convolution operation, the number of channels of the input depth information is stretched to be consistent with the left convolution sequence. The calculation process is represented by the function G(x); then F(x) is input into the subsequent three convolution units 2 with the RELU structure; finally, the result of the convolution unit is used as the input data of stage 2.

[0117] 4. In the 2nd to 4th stages of the backbone network, in order to form a constant connection mapping with the output data, the input data first passes through the convolution stage 1, and then passes through several convolution units 2 again. Finally, in the 4th stage, the output depth information dimension (2048*7*7) is output, and then the average pooling operation with a sliding window size of 7*7 is performed to generate an input image depth feature vector with a dimension of 2048.

[0118] 5. Take the flight target image u and positive sample v respectively + And negative samples v - The image depth features obtained by the backbone network are substituted into the ternary loss function formula (4) to calculate the difference between the three. In formula (4), w represents the network parameter, the hyperparameter m represents the similarity judgment margin, and the hyperparameter λ represents the weight attenuation efficiency; the cosine distance D is used to determine the similarity of two sets of depth feature vectors. The closer the input vectors u and v are, the closer the value of D is to 0. On the contrary, the greater the difference between u and v, the closer the value of D is to 1. In this way, not only in the N samples of a training batch, not only when the depth feature distance corresponding to the flight target image and the corresponding data enhancement result is less than the depth feature distance corresponding to the flight target and the background image, that is, D(u,v + ) <D(u,v - ) when the backbone network needs to be optimized, and D(u,v + ) <D(u,v - ) <D(u,v + )+m, the backbone network parameters will still be optimized until D(u,v + )+m <D(u,v - ), the backbone network is trained to convergence, at which point the difference in depth features between the flying target and the background image is much greater than the difference in depth features between the flying target and its corresponding data enhancement.

[0119]

[0120] In order to ensure that more complex background images participate in the training process and have a greater impact on the training and updating process of the model, refer to the Hard Negative Mining method, select k (k = 6) background sample image depth feature vectors with the largest difference from the flight target area image and its data enhancement result in each training batch, and average these 6 loss functions as the loss function of the training batch data. In the process of minimizing the loss function, the optimization algorithm combined with LARS and Ranger is used to optimize the loss function based on the current network parameters as the initial value, and back propagate the network parameters when the loss function converges to the minimum value to update the network parameters.

[0121] S5, obtaining a sequence of flying target images to be detected; and performing target image and background image recognition on an initial flying target image to be detected in the sequence of flying target images to be detected, so as to obtain a flying target template image in the initial flying target image to be detected, its position, and an adjacent background area image.

[0122] S6. Input the flight target template image and the adjacent background area image into the flight target depth feature extractor to obtain the flight target depth features corresponding to the flight target template image and the adjacent background area image respectively.

[0123] S7, inputting the flight target depth features corresponding to the flight target template image and the adjacent background area image into an online classifier for classification training.

[0124] S8. Perform online classification of flying targets according to the remaining flying target images to be detected in the flying target image sequence to be detected, the position of the flying target template image, the flying target depth feature extractor and the trained online classifier to obtain the motion trajectory of the flying target to be detected.

[0125] Extract network based on flight target x After processing the flight target template and the image to be detected I, two deep features are obtained according to the output results of the fourth stage of the network, with dimensions of w*h*C and W*H*C respectively; then the deep features corresponding to the flight target template (i.e., the flight target template image) are converted into As the convolution kernel for the deep features of the image to be detected Convolution is performed to obtain a vector with a dimension of (W-w+1)*(H-h+1)*1 and element values ​​between 0 and 1. The calculation result is upsampled multiple times to the size of the image to be detected I according to the cubic spline interpolation method, and the similarity response map of the flying target template in the image is obtained, as shown in Formula 5:

[0126]

[0127] Among them, * indicates that both the input image and the flight target template have been The convolution operation of , b represents the intercept, and the response spectrum m is a two-dimensional matrix, and the position with the highest weight corresponds to the tracking position of the moving target.

[0128] This embodiment uses a support vector machine (SVM Support Vector Machine) as a binary classifier f(·) to determine the position of the flight target template image in the flight target image to be detected, and uses a circulant matrix operation method to search the flight target image to be detected. According to the circulant matrix method, the flight target template image is translated and smoothed to improve the operation efficiency. In this way, the binary classifier f(·) can obtain the same effect as the deep vector convolution on the one hand, and on the other hand, the parameters of the classifier can be adjusted and adapted according to the tracking process, specifically including:

[0129] 1. In the initial flight target image I to be detected 0 In , random positive and negative sampling (i.e., flying target sub-image and background image) is performed based on the flying target template image x and its neighborhood. The positive sample image is the flying target sub-image (including the complete or most flying target), and the negative sample is the background image that does not contain or contains a small part of the flying target. The image depth features of the two groups of samples are calculated, and the depth features of the two groups of positive and negative samples are further mapped to a higher-dimensional linearly separable space using nonlinear operations such as Gaussian kernel function and polynomial kernel function. The corresponding labels are used as training samples of SVM, and an online classifier f(·) is obtained after training.

[0130] 2. The image of the flying target to be detected at the subsequent time t I t In the figure I, the flying target to be detected is t The flight target template image is input into the online classifier f(·) to obtain the flight target template image in the flight target image to be detected I t The similarity response map m in the image is selected, and the position of the maximum value is selected as the position of the flying target in the flying target map I to be detected. t location.

[0131] The background area around the flying target area is resampled, and the depth features of two groups of targets and backgrounds are calculated to update the online classifier f(·).

[0132] 3. After all subsequent sequences of flight target images to be detected are processed online in sequence, the motion trajectory of the flight target can be obtained.

[0133] Exception handling is the error correction process when the online classification method encounters target tracking errors or tracking drift. In order to ensure the accuracy of long-term tracking of moving targets and eliminate the influence of long-term tracking of flying targets due to factors such as changes in the shape of the flying target, background interference and illumination changes, the method of modeling motion noise is adopted to improve the judgment probability of the similarity response spectrum m, thereby improving the tracking robustness of the flying target. The actual estimated position of the moving target is expressed as:

[0134]

[0135] Among them, y is the real image position of the flying target. When the random tracking error ε obeys the standard normal distribution, the conditional probability density function p(m|ε) of the similarity response map m corresponding to the flying target template image x is described by Bernoulli distribution, which is:

[0136] p(m|ε)=∏f(x)m(x)(1-f(x))(1-m(x)); (7)

[0137] The value range of the online classifier f(·) is between 0 and 1. Taking the logarithm of the conditional probability density p(m|ε), we get:

[0138] g(ε)=log(p(m|ε)); (8)

[0139] According to the conditional probability formula, the tracking error ε is integrated to obtain the probability density of the similarity response spectrum p(m):

[0140]

[0141] Since this integral has no closed interval form, we can use Taylor's formula to calculate e g(ε) Performing a second-order Taylor expansion yields The probability density of the above similarity response map is approximately expressed as:

[0142]

[0143] In the process of calculating g(ε) and its expectation, for convenience, we set m 1 (x) = m(x), m 0 (x) = 1 - m (x), h 1 (x) = log(f(x)) and h 0 (x) = log(1-f(x)), then:

[0144]

[0145] Among them, the mean (first-order moment) of the similarity response spectrum is expressed as:

[0146]

[0147] The corresponding second-order expectation (second-order moment) in the probability density approximation expression of the similarity response spectrum can be expressed as:

[0148] E ε g(ε) 2 ]=∑h b (x)V b,b' (x,x')h b' (x'),V b,b' (x,x')=E[m b (x)m b' (x')]; (13)

[0149] In calculation and V b,b' In the process, the values ​​of both can be obtained by multiple random samplings according to the distribution of ε, and then Eε[g(ε)] and Eε[g(ε) 2 ] is brought into formula (10), the logarithmic approximate expression of the similarity response spectrum and the logarithmic factor is eliminated according to the monotonically increasing property of the logarithmic function in this interval, then the probability distribution p(m) of the actual moving target in the similarity response spectrum can be maximized and corrected, and the optimization goal of this process is expressed as:

[0150]

[0151] When the loss function reaches the minimum, the maximum probability p(m) of the flight target response spectrum can be obtained, which can ensure that the motion noise is effectively suppressed during the tracking process.

[0152] In summary, the specific implementation process of online classification of flight targets in this embodiment includes:

[0153] Step S81, sampling the remaining flying target images to be detected in a circulant matrix manner according to the position of the flying target template image, and inputting the samples into the flying target depth feature extractor to obtain the flying target depth features of each sampling area;

[0154] Step S82, inputting the flight target depth features of each sampling area into the online classifier to obtain a similarity response map of the flight target template image in the corresponding remaining flight target images to be detected;

[0155] Step S83, performing noise suppression processing on each similarity response spectrum;

[0156] Step S84, selecting the position of the maximum peak from each similarity response spectrum after the noise suppression process as the tracking position of the flying target in the corresponding flying target image to be detected;

[0157] Step S85, judging whether all the remaining flight target images to be detected have been classified, if so, outputting the tracking positions in all the remaining flight target images to be detected, and ending; if not, taking the area where the tracking position is located as a new flight target template image, and proceeding to step S86;

[0158] Step S86, sampling the remaining flying target images to be detected in a circulant matrix manner according to the position of the flying target template image, and inputting the samples into the flying target depth feature extractor to obtain the flying target depth features of each sampling area;

[0159] Step S87, input the flight target depth features of each sampling area into the online classifier for updating, and return to step S82.

[0160] This embodiment constructs a triple twin network, and forms anchor samples, positive samples and negative sample sets from the valid moving target images corresponding to each flying target image, the valid moving target images after data enhancement processing and the background image set; through the anchor samples, positive samples and negative sample sets, the parameter training of the triple twin network is realized, and then the flying target deep feature extractor is obtained, which ensures that the flying target deep feature extractor can accurately distinguish the moving target from the sky background, maximizes the distance between the moving target and the sky background in the specified criterion space, fully reduces the similarity between the moving target and the sky background, and has generalization; for the case of small size of the moving target and motion blur, sample enhancement is used to make the sample data more diverse and the network model more robust; through the flying target template image in the initial flying target image to be detected, the adjacent background area image and the remaining flying target images to be detected, the classifier training is realized, the accuracy of the classification result of the online classifier is guaranteed, and the tracking and positioning of the flying target is realized.

[0161] The above embodiment can be implemented by using the flight target tracking system based on contrastive learning given in the following embodiment:

[0162] Another embodiment provides a flight target tracking system based on contrastive learning, the flight target tracking system comprising:

[0163] The first acquisition module is used to acquire a sequence of flying target images under different climate and lighting conditions;

[0164] An extraction module is used to extract the effective moving target image and background image of each flying target image from the flying target image sequence; and use the background image set corresponding to each effective moving target image as a negative sample image set;

[0165] A data enhancement processing module is used to perform data enhancement processing on each valid moving target image as a positive sample image set; and each valid moving target image is used as an anchor point sample;

[0166] A parameter adjustment module is used to construct a triple twin network using three backbone networks with the same structure; and input the anchor point samples, the corresponding positive sample image set and the negative sample image set into the triple twin network for parameter adjustment, and use one of the backbone networks after parameter adjustment as a flight target deep feature extractor;

[0167] The recognition module is used to obtain a sequence of flying target images to be detected; and to perform target image and background image recognition on an initial flying target image to be detected in the sequence of flying target images to be detected, so as to obtain a flying target template image in the initial flying target image to be detected, its position, and an adjacent background area image;

[0168] A second acquisition module is used to input the flight target template image and the adjacent background area image into the flight target depth feature extractor to obtain the flight target depth features corresponding to the flight target template image and the adjacent background area image respectively;

[0169] A classification training module, used for inputting the flight target depth features corresponding to the flight target template image and the adjacent background area image into an online classifier for classification training;

[0170] The third acquisition module is used to perform online classification of flying targets based on the remaining flying target images to be detected in the flying target image sequence to be detected, the position of the flying target template image, the flying target depth feature extractor and the trained online classifier to obtain the motion trajectory of the flying target to be detected.

[0171] Furthermore, the extraction module includes:

[0172] A position matching submodule is used to perform motion pixel point detection and scale-invariant position feature point detection on adjacent flight target images in the initial stage of the flight target image sequence, and then perform position matching to obtain a set of position feature point pairs;

[0173] A first calculation submodule is used to calculate the Euclidean distance of the pixel positions of each pair of position feature point pairs in the position feature point pair set to determine the position feature points belonging to the target motion area;

[0174] A determination submodule, used to determine a valid motion region template of the flying target in the flying target image according to position feature points belonging to the target motion region;

[0175] A first scale transformation processing submodule is used to perform motion region template matching and scale transformation processing on subsequent flight target images in sequence according to the motion region template of the flight target in the flight target image, so as to obtain a valid motion target image in each flight target image;

[0176] The second scale transformation processing submodule is used to randomly select a non-effective moving target area from each flying target image and perform scale transformation processing to obtain a background image in each flying target image.

[0177] Furthermore, the data enhancement processing module includes:

[0178] The first convolution operation submodule is used to process each valid moving target image into a black and white image and then perform horizontal convolution operation and vertical convolution operation respectively to obtain the contour gradient of the flying target in each valid moving target image;

[0179] The second calculation submodule is used to calculate the maximum circumscribed rectangle of the contour gradient of the flying target in each of the valid moving target images;

[0180] A judgment submodule is used to judge whether all the pixel points in each maximum circumscribed rectangle belong to the contour of the flying target, so as to obtain each circumscribed matrix;

[0181] A singular decomposition submodule is used to perform singular decomposition on each external matrix to obtain the tangent vector direction and the normal vector direction corresponding to the maximum eigenvalue of each external matrix;

[0182] A segmentation submodule, used to determine the tangent vector direction and the normal vector direction corresponding to the maximum eigenvalue of each circumscribed matrix, so as to segment the contour of each flying target;

[0183] The edge convolution processing submodule is used to perform corrosion / dilation and edge convolution processing on each flying target contour in turn.

[0184] Furthermore, the flight target online classification module includes:

[0185] A first sampling submodule is used to sample the remaining flying target images to be detected according to the position of the flying target template image in a circulant matrix manner, and input the samples into the flying target depth feature extractor to obtain the flying target depth features of each sampling area;

[0186] An acquisition submodule, used for inputting the flight target depth features of each sampling area into the online classifier to obtain a similarity response map of the flight target template image in the corresponding remaining flight target images to be detected;

[0187] A noise suppression processing submodule, used for performing noise suppression processing on each similarity response map;

[0188] A selection submodule is used to select the position of the maximum peak from each similarity response spectrum after noise suppression processing as the tracking position of the flying target in the corresponding flying target image to be detected;

[0189] A judgment submodule is used to judge whether all the remaining flight target images to be detected have completed classification. If so, the tracking positions in all the remaining flight target images to be detected are output and the process ends; if not, the area where the tracking position is located is used as a new flight target template image and transmitted to the second sampling submodule;

[0190] A second sampling submodule is used to sample the remaining flying target images to be detected according to the position of the flying target template image in a circulant matrix manner, and input the samples into the flying target depth feature extractor to obtain the flying target depth features of each sampling area;

[0191] The updating submodule is used to input the flight target depth features of each sampling area into the online classifier for updating, and transmit the updated online classifier to the acquisition submodule. The principles, formulas and parameter definitions involved in the above embodiments are all applicable and will not be repeated here.

[0192] Please note that the technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, all possible combinations of the technical features in the above embodiments are not described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification. The above embodiments only express several implementation methods of the present application, and their descriptions are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that for ordinary technicians in this field, without departing from the concept of the present application, several variations and improvements can be made, which all belong to the scope of protection of the present application. Therefore, the scope of protection of the patent in this application shall be based on the attached claims.

Claims

1. A flying target tracking method based on contrastive learning, characterized in that: The flying target tracking method comprises: Step S1, acquiring a flying target image sequence under different climate and lighting conditions; Step S2, extracting the effective moving target image and background image of each flying target image from the flying target image sequence; and using the background image set corresponding to each effective moving target image as a negative sample image set; Step S3, performing data enhancement processing on each valid moving target image as a positive sample image set; and using each valid moving target image as an anchor point sample; Step S4, using three backbone networks with the same structure to construct a triple twin network; inputting the anchor point samples, the corresponding positive sample image set and the negative sample image set into the triple twin network for parameter adjustment, and using one of the backbone networks after parameter adjustment as a flight target deep feature extractor; Step S5, obtaining a sequence of flying target images to be detected; and performing target image and background image recognition on an initial flying target image to be detected in the sequence of flying target images to be detected, so as to obtain a flying target template image in the initial flying target image to be detected and its position and an adjacent background area image; Step S6, inputting the flight target template image and the adjacent background area image into the flight target depth feature extractor to obtain the flight target depth features corresponding to the flight target template image and the adjacent background area image respectively; Step S7, inputting the flight target depth features corresponding to the flight target template image and the adjacent background area image into an online classifier for classification training; Step S8, performing online classification of the flying targets according to the remaining flying target images to be detected in the flying target image sequence to be detected, the position of the flying target template image, the flying target depth feature extractor and the trained online classifier to obtain the motion trajectory of the flying target to be detected.

2. The flying target tracking method according to claim 1, characterized in that: In step S2, the specific process of the extraction includes: Step S21, performing motion pixel point detection and scale-invariant position feature point detection on adjacent flight target images in the initial stage of the flight target image sequence, and then performing position matching to obtain a position feature point pair set; Step S22, calculating the Euclidean distance of the pixel positions of each pair of position feature point pairs in the position feature point pair set to determine the position feature points belonging to the target motion area; Step S23, determining a valid motion region template of the flying target in the flying target image according to the position feature points in the target motion region; Step S24, according to the motion region template of the flying target in the flying target image, the subsequent flying target images are sequentially subjected to motion region template matching and scale transformation processing to obtain a valid motion target image in each flying target image; Step S25: randomly select a non-effective moving target area from each flying target image and perform a scale transformation process to obtain a background image in each flying target image.

3. The flying target tracking method according to claim 2, characterized in that: In step S3, the data enhancement includes one or a combination of color transformation, affine transformation, scale transformation, rotation transformation, sharpening, blurring and morphological processing; the specific process of the morphological processing includes: Step S31, after processing each valid moving target image into a black and white image, performing horizontal convolution operation and vertical convolution operation respectively, to obtain the contour gradient of the flying target in each valid moving target image; Step S32, calculating the maximum circumscribed rectangle of the contour gradient of the flying target in each of the valid moving target images; Step S33, judging whether all the pixel points in each maximum circumscribed rectangle belong to the contour of the flying target, so as to obtain each circumscribed matrix; Step S34, performing singular decomposition on each circumscribed matrix to obtain the tangent vector direction and the normal vector direction corresponding to the maximum eigenvalue of each circumscribed matrix; Step S35, determining the tangent vector direction and the normal vector direction corresponding to the maximum eigenvalue of each circumscribed matrix to segment the contours of each flying target; Step S36: perform corrosion / dilation and edge convolution processing on each flying target contour in turn.

4. The flying target tracking method according to claim 3, characterized in that: The triple twin network includes three backbone networks and a ternary contrast loss module connected in parallel; the input end of the ternary contrast loss module is connected to the output ends of the three backbone networks; The three backbone networks are used to extract image depth features from the images received respectively and then input them into the ternary contrast loss module; The ternary contrast loss module is used to use a ternary loss function to perform similarity comparison on the image depth features output by the three backbone networks; And according to the similarity comparison structure, the backbone network parameters of the triple twin network are adjusted.

5. The flying target tracking method according to claim 4, characterized in that: In step S8, the specific implementation process of the online classification of the flight target includes: Step S81, sampling the remaining flying target images to be detected in a circulant matrix manner according to the position of the flying target template image, and inputting the samples into the flying target depth feature extractor to obtain the flying target depth features of each sampling area; Step S82, inputting the flight target depth features of each sampling area into the online classifier to obtain a similarity response map of the flight target template image in the corresponding remaining flight target images to be detected; Step S83, performing noise suppression processing on each similarity response spectrum; Step S84, selecting the position of the maximum peak from each similarity response spectrum after the noise suppression process as the tracking position of the flying target in the corresponding flying target image to be detected; Step S85, judging whether all the remaining flight target images to be detected have been classified, if so, outputting the tracking positions in all the remaining flight target images to be detected, and ending; if not, taking the area where the tracking position is located as a new flight target template image, and proceeding to step S86; Step S86, sampling the remaining flying target images to be detected in a circulant matrix manner according to the position of the flying target template image, and inputting the samples into the flying target depth feature extractor to obtain the flying target depth features of each sampling area; Step S87, input the flight target depth features of each sampling area into the online classifier for updating, and return to step S82.

6. A flight target tracking system based on contrastive learning, characterized in that: The flying target tracking system comprises: The first acquisition module is used to acquire a sequence of flying target images under different climate and lighting conditions; An extraction module is used to extract the effective moving target image and background image of each flying target image from the flying target image sequence; and use the background image set corresponding to each effective moving target image as a negative sample image set; A data enhancement processing module is used to perform data enhancement processing on each valid moving target image as a positive sample image set; and each valid moving target image is used as an anchor point sample; A parameter adjustment module is used to construct a triple twin network using three backbone networks with the same structure; and input the anchor point samples, the corresponding positive sample image set and the negative sample image set into the triple twin network for parameter adjustment, and use one of the backbone networks after parameter adjustment as a flight target deep feature extractor; The recognition module is used to obtain a sequence of flying target images to be detected; and to perform target image and background image recognition on an initial flying target image to be detected in the sequence of flying target images to be detected, so as to obtain a flying target template image in the initial flying target image to be detected, its position, and an adjacent background area image; A second acquisition module is used to input the flight target template image and the adjacent background area image into the flight target depth feature extractor to obtain the flight target depth features corresponding to the flight target template image and the adjacent background area image respectively; A classification training module, used for inputting the flight target depth features corresponding to the flight target template image and the adjacent background area image into an online classifier for classification training; The flying target online classification module is used to perform online classification of flying targets based on the remaining flying target images to be detected in the flying target image sequence to be detected, the position of the flying target template image, the flying target deep feature extractor and the trained online classifier to obtain the motion trajectory of the flying target to be detected.

7. The flying target tracking system according to claim 6, characterized in that: The extraction module comprises: A position matching submodule is used to perform motion pixel point detection and scale-invariant position feature point detection on adjacent flight target images in the initial stage of the flight target image sequence, and then perform position matching to obtain a set of position feature point pairs; A first calculation submodule is used to calculate the Euclidean distance of the pixel positions of each pair of position feature point pairs in the position feature point pair set to determine the position feature points belonging to the target motion area; A determination submodule, used to determine a valid motion region template of the flying target in the flying target image according to position feature points belonging to the target motion region; A first scale transformation processing submodule is used to perform motion region template matching and scale transformation processing on subsequent flight target images in sequence according to the motion region template of the flight target in the flight target image, so as to obtain a valid motion target image in each flight target image; The second scale transformation processing submodule is used to randomly select a non-effective moving target area from each flying target image and perform scale transformation processing to obtain a background image in each flying target image.

8. The flying target tracking system according to claim 7, characterized in that: The data enhancement processing module includes: The first convolution operation submodule is used to process each valid moving target image into a black and white image and then perform horizontal convolution operation and vertical convolution operation respectively to obtain the contour gradient of the flying target in each valid moving target image; The second calculation submodule is used to calculate the maximum circumscribed rectangle of the contour gradient of the flying target in each valid moving target image; A judgment submodule is used to judge whether all the pixel points in each maximum circumscribed rectangle belong to the contour of the flying target, so as to obtain each circumscribed matrix; A singular decomposition submodule is used to perform singular decomposition on each external matrix to obtain the tangent vector direction and the normal vector direction corresponding to the maximum eigenvalue of each external matrix; A segmentation submodule, used to determine the tangent vector direction and the normal vector direction corresponding to the maximum eigenvalue of each circumscribed matrix, so as to segment the contour of each flying target; The edge convolution processing submodule is used to perform corrosion / dilation and edge convolution processing on each flying target contour in turn.

9. The flying target tracking system according to claim 8, characterized in that: The flight target online classification module includes: A first sampling submodule is used to sample the remaining flying target images to be detected according to the position of the flying target template image in a circulant matrix manner, and input the samples into the flying target depth feature extractor to obtain the flying target depth features of each sampling area; An acquisition submodule, used for inputting the flight target depth features of each sampling area into the online classifier to obtain a similarity response map of the flight target template image in the corresponding remaining flight target images to be detected; A noise suppression processing submodule, used for performing noise suppression processing on each similarity response map; A selection submodule is used to select the position of the maximum peak from each similarity response spectrum after noise suppression processing as the tracking position of the flying target in the corresponding flying target image to be detected; A judgment submodule is used to judge whether all the remaining flight target images to be detected have completed classification. If so, the tracking positions in all the remaining flight target images to be detected are output and the process ends; if not, the area where the tracking position is located is used as a new flight target template image and transmitted to the second sampling submodule; A second sampling submodule is used to sample the remaining flying target images to be detected according to the position of the flying target template image in a circulant matrix manner, and input the samples into the flying target depth feature extractor to obtain the flying target depth features of each sampling area; The updating submodule is used to input the flight target depth features of each sampling area into the online classifier for updating, and transmit the updated online classifier to the acquiring submodule.

Images