A time-sensitive target discovery method based on spatiotemporal interactive reasoning

Abstract

The present invention discloses a time-sensitive target discovery method based on time-space interactive reasoning. First, a network model based on dual-path Unified Transformer detection and tracking integration is constructed. Then, a knowledge distillation method based on a diffusion model and a target track generation method based on time-space interactive reasoning are used to realize open set target detection and recognition and output stable target tracks with time-space consistency, thereby completing the discovery of time-sensitive targets. The method of the present invention adopts a dual-path Unified Transformer integrated network architecture, comprehensively utilizes multimodal image information to improve the time-sensitive target discovery capability in complex scenarios, can simultaneously process image target detection and target tracking tasks, enhance target discovery capabilities under few-sample and zero-sample conditions, provide intelligent processing means for multi-sensor collaborative detection of time-sensitive targets, and realize robust detection and stable tracking of time-sensitive targets in continuous imaging reconnaissance and surveillance applications.

Description

Technical Field

[0001] The present invention belongs to the technical field of automatic detection and tracking of time-sensitive targets, and in particular relates to a time-sensitive target discovery method based on spatiotemporal interactive reasoning. Background Art

[0002] Time-sensitive targets are objects with uncertain attributes and fast movement speed in the surveillance scene. Time-sensitive target discovery requires not only target detection to determine the target type, but also the output of the target track by temporal correlation.

[0003] At present, there are two mainstream algorithms for image target detection: anchor-based target detection algorithm and anchor-free target detection algorithm. The first type of anchor-based target detection algorithm is to find feature points from the feature map, and use this to set the prior box for screening and adjustment to obtain the final prediction box. The literature "R. Girshick, J. Donahue, T. Darrell, and J. Malik. Rich feature hierarchies for accurate object detection and semantic segmentation, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Jun. 23-28, 2014, pp. 580-587, Columbus, USA" proposed the R-CNN algorithm, which selectively searches for candidate regions and extracts features using a pre-trained model, and finally completes target prediction through a linear classifier; the literature "S. Ren, K. He, R. Girshick, and J. Sun. Faster R-CNN: Towards real-time object detection with region proposal networks, IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI), vol.39, no.6, pp.1137~1149, 2016” proposed the Faster-RCNN algorithm, in which the region proposal network can slide the window on the feature map to obtain the category and regression parameters of the candidate box, in order to solve the problem of slow candidate region generation. The second type of anchor-free target detection algorithm regards the target detection problem as a single-stage regression problem, and directly regresses the predicted box and its type in the output layer. The literature “H.Law, J.Deng.CornerNet: Detecting objects as paired keypoints, in: Proceedings of the European Conference on Computer Vision (ECCV), Sep.8-14, 2018, pp.734~750, Munich, Germany” proposed the CornerNet algorithm, which predicts a pair of key points and regresses the predicted box based on this to achieve target positioning.In addition to the above two types of target detection algorithms; the literature "N.Carion, F.Massa, G.Synnaeve, N.Usunier, A.Kirillov and S.Zagoruyko.End-to-end object detection with transformers, in: Proceedings of the European Conference on Computer Vision (ECCV), Aug.23-28, 2020, pp.213~229, Virtual" proposed DETR based on Transformer, which regards the target detection task as a set prediction problem, without relying on traditional anchor point generation and non-maximum suppression methods, and realizes end-to-end target detection.

[0004] On the other hand, there are two main types of track association methods for target discovery and tracking: track association method based on similarity fusion and track association method based on deep learning. The existing track association method based on similarity fusion, the document "Yi Xiao, Zhou Wei. Track gray association algorithm based on regional coverage. Firepower and Command and Control, 2020, 45(5): 45-50" first applied the gray association theory to track association, and judged whether the track sequence was associated by analyzing the similarity or difference of the track sequence curve. This type of method relies on manually designed weights and thresholds, and has a high error rate in uncertain scenarios with dense tracks. The track association method based on deep learning has powerful nonlinear fitting and mapping capabilities. Related research gradually introduces neural networks to assist track association. The document "R. Berndt. Aircraft micro-Doppler feature extraction from high rangeresolution profile, in: Proceedings of the IEEE Radar Conference, Oct. 27-30, 2015, pp. 457-462, Johannesburg, South Africa" ​​introduces a three-dimensional BP model to solve the three-dimensional allocation problem in track association. The paper "K. Kim. Focusing of high range resolution profiles of moving targets using stepped frequency waveforms, IET Radar, Sonar & Navigation, vol. 4, no. 4, pp. 564-575, 2010" uses convolutional neural networks to capture important features in the local sequence of tracks for track association, but the convolution kernel receptive field is limited and cannot have global features. On this basis, the paper "Huang Hongwei, Liu Yujiao, Shen Zhuokai, Zhang Shaowei, Chen Zhimin, Gao Yang. End-to-end track association based on deep learning network model. Computer Science, 2020, 47(3): 200-205" combines the long short-term memory network with the CNN model, improves the adaptive adjustment ability of the neural network model, and outputs more accurate track association results.

[0005] The above methods do not have time-sensitive target discovery under the integrated framework of collaborative association processing of image target detection and temporal position detection target tracking. Although their target detection and recognition methods have high accuracy, it is difficult to achieve few-sample or zero-sample time-sensitive target discovery; and their track association methods cannot comprehensively utilize the detection results in the previous and next time to achieve time-sensitive target track output. Summary of the invention

[0006] To solve the above technical problems, the present invention provides a time-sensitive target discovery method based on spatiotemporal interactive reasoning, which is based on a dual-path Unified Transformer detection and tracking integrated network architecture, uses two information processing paths, and respectively utilizes a knowledge distillation method based on a diffusion model and a target track generation method based on spatiotemporal interactive reasoning to realize open set target detection and recognition and spatiotemporal consistent target tracking.

[0007] The technical solution adopted by the present invention is: a time-sensitive target discovery method of spatiotemporal interactive reasoning, and the specific steps are as follows:

[0008] S1. Build a network model based on dual-path Unified Transformer integrated detection and tracking;

[0009] The network model based on dual-path Unified Transformer detection and tracking integration includes: convolutional neural network, Transformer network model, Unified Transformer network model, pre-trained CLIP model (multimodal pre-training model), diffusion model, conditional score network and adaptive fusion network.

[0010] The network model based on the integration of dual-path Unified Transformer detection and tracking takes a multimodal fusion feature set as input, and then uses two information processing paths, namely, a knowledge distillation method based on a diffusion model and a target track generation method based on spatiotemporal interactive reasoning, to achieve open set target detection and recognition and output a stable target track with spatiotemporal consistency.

[0011] The multimodal fusion feature set is obtained by using convolutional neural networks and Transformer network models. First, time-sensitive target images of different modalities are collected in advance as training sets. Then, convolutional neural networks are used to extract features of images of different modalities in the training sets. After obtaining the corresponding feature vectors, the multimodal feature vectors are uniformly input into the Transformer network model to fuse the multimodal features and obtain the multimodal fusion feature set F. fuse .

[0012] S2, based on the model constructed in step S1, using the knowledge distillation method based on the diffusion model to realize open set target detection and recognition;

[0013] S3, based on the model constructed in step S1, a target track generation method based on time-space interactive reasoning outputs a stable target track with time-space consistency;

[0014] S4. Based on steps S1-S3, open set target detection and recognition are realized and stable target tracks with spatiotemporal consistency are output to complete the discovery of time-sensitive targets.

[0015] Furthermore, the step S2 is specifically as follows:

[0016] S21, input the multimodal fusion feature set, add the position code, and use the Unified Transformer network model to process the randomly initialized target feature set to form a captured target feature set;

[0017] Multimodal fusion feature set F fuse As input, the feature set is divided into 8×8 blocks of the same size, and the sinusoidal encoding rule is used to add position encoding PE to the features of different blocks. Then, the target feature set {q} is randomly initialized to convert the prediction target into a learnable query embedding.

[0018] Then the input features are passed into the Unified Transformer network model, and the self-attention transformation mechanism in the model is used to extract key semantic structure information from the input features. The cross-attention mechanism is used to drive the initialized query embedding set to capture different target structure information to form a captured target feature set {f}. A decoder that only focuses on the position information in the feature set is used to regress the prediction box.

[0019] S22, based on step S21, train the pre-trained CLIP model and proceed to step S24;

[0020] Firstly, the decoder in the Unified Transformer network model is trained with self-supervision of the prediction box using pre-collected time-sensitive target images of different modalities.

[0021] During the training process, the predicted set y is compared with the target true value Match associations and deconstruct them into a bipartite graph matching task so that one query embedding in the set corresponds to one target, and the number of query embeddings Q should be much larger than the number of targets.

[0022] S23, based on step S21, semantically independent decoding is performed to obtain a prediction frame, and a Hungarian matching function is used to achieve a one-to-one matching between multiple prediction targets and true values;

[0023] The matching function uses the Hungarian algorithm, and the expression is as follows:

[0024]

[0025] Where N represents the total number of targets in the prediction set or the true set. Represents the category similarity loss between the predicted box and the true box, c n L represents an element in the cost matrix, and each element of the cost matrix represents the cost or price of assigning the i-th task to the j-th person.box Represents the similarity loss between the regressed predicted box and the true box.

[0026] By minimizing the Hungarian matching loss, a one-to-one matching of multiple predicted targets and true values ​​is achieved.

[0027] S24, based on steps S21 and S22, target recognition is performed under the condition of few samples or zero samples, that is, pre-training prior knowledge is transferred by using the diffusion model on the basis of capturing the target feature set, and then entering step S25;

[0028] Firstly, noise is added layer by layer based on the diffusion model, and the KL divergence is used to constrain the open set feature x so that its feature distribution tends to the Gaussian distribution N(μ,I). The open set feature x is integrated into the capture target feature set, and the open set knowledge is incorporated while the original features remain unchanged.

[0029] Then, based on the diffusion model, layer-by-layer denoising is performed, that is, inverse denoising. The pre-trained CLIP model is trained to learn the conditional distribution probability to remove the Gaussian noise with a distribution function of N(0,I). The training loss expression is as follows:

[0030]

[0031] Among them, μ represents the mean, I represents the standard deviation, t1 represents the current number of diffusion layers, and x t represents the input image of the current diffusion layer, ε represents the standard normal distribution, ε θ Represents the noise distribution removed in the back propagation of the diffusion model.

[0032] S25. For few-shot and zero-shot target recognition, a prototype network is proposed. The sample data in each category is mapped into a space using metric learning, and the “mean” is extracted as the prototype of the category.

[0033] Perform open set recognition task training, set the number of features in the multimodal fusion feature set to N, and each feature is F n , the corresponding real text feature is By minimizing the cosine distance, the Unified Transformer network model embedded with the semantic structure of the pre-trained CLIP model maps the features of the query feature set and the target semantic features extracted by the CLIP text encoder into the same feature space to achieve alignment.

[0034] In the few-shot classification task, S = {(x1,u1),...,(x N ,u N )} represents a small-scale support dataset. x represents the vectorized representation of the original data in D dimensions, u represents its corresponding category, S kRepresents a data set with category k.

[0035] The prototype network computes a prototype representation v for each class k , through an embedding function f Φ R D →R M Map the sample data of dimension D to the space of dimension M. The embedding function uses a simple linear mapping, f Φ =Wx, W represents a parameter matrix, the purpose is to transform the dimension of x, R represents the letter representation of the dimensional space. Support all the vectorized sample data in the collection to get the prototype representation v of the category k , the expression is as follows:

[0036]

[0037] The prototype network uses softmax to measure the query vector point to v k The distance is expressed as follows:

[0038]

[0039] Among them, p Φ (·) represents the probability function, d(·) represents the distance from the query vector to v k distance, k represents the true category label of the training sample, and k′ represents the set of all k values.

[0040] The training process minimizes the objective function through stochastic gradient descent: J(Φ) = -logp Φ (y=k|x).

[0041] S26: Based on steps S22-S25, enter the prediction stage to complete the positioning of anchor frames and identification of types for few-sample and zero-sample targets;

[0042] Based on steps S22-S25, the training phase is completed and the prediction phase is entered. First, the category text to be recognized is preprocessed with a template prompt, and the template uses "a photo of {}".

[0043] Among them, {} is any open class that needs to be identified.

[0044] The category text is then sent to the pre-trained CLIP text encoder to extract text features, and a non-parametric similarity calculation is performed between it and the query feature set to find the text feature that is most similar to each feature in the feature set. The corresponding word is the category to be identified.

[0045] Finally, knowledge transfer from visible classes to invisible classes is achieved, and anchor box R is localized for few-shot and zero-shot targets. box and identification type Rcls .

[0046] Furthermore, the step S3 is specifically as follows:

[0047] S31, input the multimodal fusion feature set, use the conditional score network CSN to estimate the weight of the multimodal fusion feature and use it as a guide to selectively fuse the different modal features through the adaptive fusion network;

[0048] The multimodal fusion feature set F fuse As input, the weights of multimodal fusion features are first estimated based on the conditional score network (CSN). The conditional score network models the data probability distribution gradient in different scenarios as a score function, and uses the scoring mechanism to obtain the matching score.

[0049] The multimodal fusion weights estimated by the conditional score network are used as a guide to selectively fuse different modal features through an adaptive fusion network.

[0050] S32, based on step S31, using the Unified Transformer network model to jointly process the randomly initialized target feature set and the target single-frame associated fusion features to obtain a captured target feature set, and perform feature time series expansion;

[0051] The Unified Transformer network model is used to jointly process the randomly initialized target feature set {q} and the target single-frame associated fusion features to obtain the captured target feature set {f}, and then expand it to multiple frames of data in the time dimension to form a multi-frame captured target feature set {f} T .

[0052] Among them, the randomly initialized target feature set of the current frame will replace the captured target feature set of the previous frame {f target} t-1 ,t∈T is taken into initialization as input, and T represents the overall time series.

[0053] S33, based on step S32, designing a spatiotemporal transformer to automatically enhance feature relationships and infer target tracks, and outputting target tracks with spatiotemporal consistency;

[0054] The space-time transformer includes: a space domain encoder and a time domain decoder, which are respectively composed of a UnifiedTransformer network model encoder and a decoder.

[0055] The spatial domain encoder is used to extract spatial context and infer visual relations within a frame, while the temporal domain decoder takes the output of the spatial domain encoder as input, captures temporal dependencies between frames and infers dynamic relations between objects.

[0056] The spatiotemporal transformer continuously mines the associations between targets in the multi-frame embedded feature set in the spatial and temporal domains to output a target track R with spatiotemporal consistency. track .

[0057] In the process of training the spatiotemporal interactive reasoning track association and enhancement based on the Unified Transformer network model, the loss function consists of the per-frame positioning recognition loss and the tracking query embedding loss, and the expression is as follows:

[0058]

[0059] Among them, N q represents the number of target features, B track Indicates the number of targets successfully detected in the previous frame, L object Represents the loss function for each target feature during training.

[0060] Beneficial effects of the present invention: The method of the present invention first constructs a network model based on dual-path Unified Transformer detection and tracking integration, and then respectively uses a knowledge distillation method based on a diffusion model and a target track generation method based on spatiotemporal interactive reasoning to realize open set target detection and recognition and output a stable target track with spatiotemporal consistency, thereby completing the discovery of time-sensitive targets. The method of the present invention adopts a dual-path Unified Transformer integrated network architecture, and comprehensively utilizes multimodal image information to improve the ability to discover time-sensitive targets in complex scenarios. It can simultaneously process image target detection and target tracking tasks, enhance target discovery capabilities under few-sample and zero-sample conditions, provide intelligent processing means for multi-sensor collaborative detection of time-sensitive targets, and realize robust detection and stable tracking of time-sensitive targets in continuous imaging reconnaissance and surveillance applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 The flowchart of a time-sensitive target discovery method of spatiotemporal interactive reasoning of the present invention. DETAILED DESCRIPTION

[0062] The method of the present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0063] like Figure 1 As shown, a flow chart of a time-sensitive target discovery method of spatiotemporal interactive reasoning of the present invention is shown, and the specific steps are as follows:

[0064] S1. Build a network model based on dual-path Unified Transformer integrated detection and tracking;

[0065] The network model based on dual-path Unified Transformer detection and tracking integration includes: convolutional neural network, Transformer network model, Unified Transformer network model, pre-trained CLIP model (multimodal pre-training model), diffusion model, conditional score network and adaptive fusion network.

[0066] The network model based on the integration of dual-path Unified Transformer detection and tracking takes a multimodal fusion feature set as input, and then uses two information processing paths, namely, a knowledge distillation method based on a diffusion model and a target track generation method based on spatiotemporal interactive reasoning, to achieve open set target detection and recognition and output a stable target track with spatiotemporal consistency.

[0067] The multimodal fusion feature set is obtained by using convolutional neural networks and Transformer network models. First, time-sensitive target images of different modalities are collected in advance as training sets. Then, convolutional neural networks are used to extract features of images of different modalities in the training sets. After obtaining the corresponding feature vectors, the multimodal feature vectors are uniformly input into the Transformer network model to fuse the multimodal features and obtain the multimodal fusion feature set F. fuse .

[0068] S2, based on the model constructed in step S1, using the knowledge distillation method based on the diffusion model to realize open set target detection and recognition;

[0069] S3, based on the model constructed in step S1, a target track generation method based on time-space interactive reasoning outputs a stable target track with time-space consistency;

[0070] S4. Based on steps S1-S3, open set target detection and recognition are realized and stable target tracks with spatiotemporal consistency are output to complete the discovery of time-sensitive targets.

[0071] In this embodiment, step S2 is specifically as follows:

[0072] S21, input the multimodal fusion feature set, add the position code, and use the Unified Transformer network model to process the randomly initialized target feature set to form a captured target feature set;

[0073] Multimodal fusion feature set F fuse As input, the feature set is divided into 8×8 blocks of the same size, and the sinusoidal encoding rule is used to add position encoding PE to the features of different blocks. Then, the target feature set {q} is randomly initialized to convert the prediction target into a learnable query embedding.

[0074] Then the input features are passed into the Unified Transformer network model, and the self-attention transformation mechanism in the model is used to extract key semantic structure information from the input features. The cross-attention mechanism is used to drive the initialized query embedding set to capture different target structure information to form a captured target feature set {f}. A decoder that only focuses on the position information in the feature set is used to regress the prediction box.

[0075] S22, based on step S21, train the pre-trained CLIP model and proceed to step S24;

[0076] In the knowledge distillation method based on the diffusion model, it is necessary to use the pre-collected time-sensitive target images of different modalities (training set) to perform self-supervised training of the prediction box on the decoder in the Unified Transformer network model to improve the positioning accuracy of the prediction box.

[0077] During the training process, the predicted set y is compared with the target true value Match associations and deconstruct them into bipartite graph matching tasks, so that one query embedding in the set corresponds to one target, and the number of query embeddings Q should be much larger than the number of targets to ensure that each target has a unique embedding feature to match it.

[0078] S23, based on step S21, semantically independent decoding is performed to obtain a prediction frame, and a Hungarian matching function is used to achieve a one-to-one matching between multiple prediction targets and true values;

[0079] Semantic-Agnostic Decoding regression prediction box means that in object detection, the Unified Transformer network model decoder does not consider the semantic category information of the extracted features, but only focuses on the position and size of the regression target. Then the Hungarian matching function is used to match the predicted box with the real box.

[0080] The matching function uses the Hungarian algorithm, and the expression is as follows:

[0081]

[0082] Where N represents the total number of targets in the prediction set or the true set. Represents the category similarity loss between the predicted box and the true box, c n L represents an element in the cost matrix, which is obtained by quantifying the cost or price of assigning tasks or matching objects. Each element of the cost matrix represents the cost or price of assigning the i-th task to the j-th person. box Represents the similarity loss between the regressed predicted box and the true box.

[0083] By minimizing the Hungarian matching loss, a one-to-one matching of multiple predicted targets and true values ​​is achieved.

[0084] S24, based on steps S21 and S22, target recognition is performed under the condition of few samples or zero samples, that is, pre-training prior knowledge is transferred by using the diffusion model on the basis of capturing the target feature set, and then entering step S25;

[0085] Firstly, noise is added layer by layer based on the diffusion model, and the KL divergence is used to constrain the open set feature x so that its feature distribution tends to the Gaussian distribution N(μ,I). The open set feature x is integrated into the capture target feature set, and the open set knowledge is incorporated while the original features remain unchanged.

[0086] Then, based on the diffusion model, layer-by-layer denoising is performed, that is, inverse denoising. The pre-trained CLIP model is trained to learn the conditional distribution probability to remove the Gaussian noise with a distribution function of N(0,I). The training loss expression is as follows:

[0087]

[0088] Among them, μ represents the mean, I represents the standard deviation, t1 represents the current number of diffusion layers, and x t represents the input image of the current diffusion layer, ε represents the standard normal distribution, ε θ Represents the noise distribution removed in the back propagation of the diffusion model.

[0089] By minimizing the loss function shown in formula (2), the diffusion model can gradually remove the standard Gaussian noise in order to better preserve the open set features. After the standard Gaussian noise is accurately removed by the inverse denoising process of the diffusion model, the semantic structure information extracted by the pre-trained CLIP model is effectively integrated into the capture target feature set.

[0090] S25. For few-shot and zero-shot target recognition, a prototype network is proposed. The sample data in each category is mapped into a space using metric learning, and the “mean” is extracted as the prototype of the category.

[0091] When training for open set recognition tasks, since the number of categories is not fixed, a non-parametric metric method will be used for classification. Since CLIP image features are embedded in the open set multimodal fusion feature set, and the image features extracted by the pre-trained CLIP model are naturally aligned with the text features, open set object recognition can be achieved by fine-tuning the text encoder in the pre-trained CLIP model.

[0092] Perform open set recognition task training, set the number of features in the multimodal fusion feature set to N, and each feature is F n , the corresponding real text feature is By minimizing the cosine distance, the Unified Transformer network model embedded with the semantic structure of the pre-trained CLIP model can effectively map the features of the query feature set and the target semantic features extracted by the CLIP text encoder into the same feature space to achieve alignment.

[0093] In the few-shot classification task, S = {(x1,u1),...,(x N ,u N )} represents a small-scale support dataset. x represents the vectorized representation of the original data in D dimensions, u represents its corresponding category, S k Represents a data set with category k.

[0094] The prototype network computes a prototype representation v for each class k , through an embedding function f Φ R D →R M Map the sample data of dimension D to the space of dimension M. The embedding function uses a simple linear mapping, f Φ =Wx, W represents a parameter matrix, the purpose is to transform the dimension of x, R represents the letter representation of the dimensional space. Supports the average value of all vectorized sample data in the collection to obtain the prototype representation b of the category k , the expression is as follows:

[0095]

[0096] The prototype network uses softmax to measure the query vector point to v k The distance is expressed as follows:

[0097]

[0098] Among them, p Φ (·) represents the probability function, d(·) represents the distance from the query vector to v k distance, k represents the true category label of the training sample, and k′ represents the set of all k values.

[0099] The training process minimizes the objective function through stochastic gradient descent: J(Φ) = -logp Φ (y=k|x).

[0100] S26: Based on steps S22-S25, enter the prediction stage to complete the positioning of anchor frames and identification of types for few-sample and zero-sample targets;

[0101] Based on steps S22-S25, the training phase is completed and the prediction phase is entered. First, the category text to be recognized is preprocessed with a template prompt, and the template uses "a photo of {}".

[0102] Among them, {} is any open class that needs to be identified.

[0103] The category text is then sent to the pre-trained CLIP text encoder to extract text features, and a non-parametric similarity calculation is performed between it and the query feature set to find the text feature that is most similar to each feature in the feature set. The corresponding word is the category to be identified.

[0104] Finally, knowledge transfer from visible classes to invisible classes is achieved, and anchor box R is localized for few-shot and zero-shot targets. box and identification type R cls .

[0105] In this embodiment, step S3 is specifically as follows:

[0106] S31, input the multimodal fusion feature set, use the conditional score network CSN to estimate the weight of the multimodal fusion feature and use it as a guide to selectively fuse the different modal features through the adaptive fusion network;

[0107] The multimodal fusion feature set F fuse As input, the weights of multimodal fusion features are first estimated based on the conditional score network (CSN). The conditional score network models the data probability distribution gradient in different scenarios as a score function, and uses the scoring mechanism to obtain the matching score.

[0108] The multimodal fusion weights estimated by the conditional score network are used as a guide to selectively fuse different modal features through an adaptive fusion network.

[0109] This process can drive the efficient complementation of complete modal track features to incomplete track features according to the fusion weight. Specifically, the fusion network uses the weighted self-attention modeling method to take the fusion weight as the importance prior, enhances the focus on key features and suppresses irrelevant features in the fusion process, so as to effectively control the information flow and obtain the multi-modal correlation fusion feature F. a .

[0110] S32, based on step S31, using the Unified Transformer network model to jointly process the randomly initialized target feature set and the target single-frame associated fusion features to obtain a captured target feature set, and perform feature time series expansion;

[0111] In order to decode the target spatiotemporal track from the multimodal fusion features, the Unified Transformer network model is used to jointly process the randomly initialized target feature set {q} and the target single-frame associated fusion features to obtain the captured target feature set {f}, and then expand it to multi-frame data in the time dimension to form a multi-frame captured target feature set {f} T .

[0112] Among them, the randomly initialized target feature set of the current frame will replace the captured target feature set of the previous frame {f target} t-1 ,t∈T is taken into the initialization as input, so that it can have the ability to perceive the changes of the previous target, making it easier to track fast-moving targets. T represents the overall time series.

[0113] S33, based on step S32, designing a spatiotemporal transformer to automatically enhance feature relationships and infer target tracks, and outputting target tracks with spatiotemporal consistency;

[0114] Different from the complicated feature similarity calculation during association in the existing algorithms, the spatiotemporal transformer designed in this embodiment automatically enhances the feature relationship and infers the target track.

[0115] The space-time transformer includes: a space domain encoder and a time domain decoder, which are respectively composed of a UnifiedTransformer network model encoder and a decoder.

[0116] The spatial domain encoder is used to extract spatial context and infer visual relations within a frame, while the temporal domain decoder takes the output of the spatial domain encoder as input, captures temporal dependencies between frames and infers dynamic relations between objects.

[0117] The spatiotemporal transformer continuously mines the associations between targets in the multi-frame embedded feature set in the spatial and temporal domains to output a target track R with spatiotemporal consistency. track .

[0118] In the process of training the spatiotemporal interactive reasoning track association and enhancement based on the Unified Transformer network model, the loss function consists of the per-frame positioning recognition loss and the tracking query embedding loss, and the expression is as follows:

[0119]

[0120] Among them, N q Indicates the number of target features, N track Indicates the number of targets successfully detected in the previous frame, L object Represents the loss function for each target feature during training.

[0121] By minimizing the loss function of formula (5), the model can implicitly capture the temporal and spatial relationship, construct a multi-target temporal association graph and model the multi-frame target position to regress and obtain a stable target track R with temporal and spatial consistency. track .

[0122] In summary, the method of the present invention introduces the conditional score network CSN in the track association output algorithm based on spatiotemporal interactive reasoning, and constructs a multimodal feature fusion weight calculation model based on the conditional score network. In the track association output algorithm based on spatiotemporal interactive reasoning, the Unified Transformer network model is introduced to construct a time expansion model for multimodal features. In the few-sample and zero-sample recognition method based on the diffusion model, the attention mechanism is introduced to obtain the key information of the known class to obtain the captured target feature set. In the few-sample and zero-sample recognition method based on the diffusion model, the diffusion model is used to migrate the pre-trained knowledge in the visual-language model CLIP, align and fuse the open set category features. The method of the present invention adopts a dual-path Unified Transformer integrated network architecture, comprehensively utilizes multimodal image information to improve the time-sensitive target detection capability in complex scenes, can simultaneously process image target detection and target tracking tasks, enhance the target detection capability under few-sample and zero-sample conditions, provide intelligent processing means for multi-sensor collaborative detection of time-sensitive targets, and realize robust detection and stable tracking of time-sensitive targets in continuous imaging reconnaissance and surveillance applications.

[0123] Those skilled in the art will appreciate that the above embodiments are intended to help readers understand the principles of the present invention, and should be understood that the scope of protection of the present invention is not limited to such specific statements and embodiments. For those skilled in the art, the present invention may have various changes and variations. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of the claims of the present invention.

Claims

1. A time-sensitive target discovery method based on spatiotemporal interactive reasoning, the specific steps are as follows: S1. Build a network model based on dual-path Unified Transformer integrated detection and tracking; The network model based on dual-path Unified Transformer detection and tracking integration includes: Convolutional neural network, Transformer network model, Unified Transformer network model, pre-trained CLIP model, diffusion model, conditional score network and adaptive fusion network; The network model based on dual-path Unified Transformer integrated detection and tracking takes a multimodal fusion feature set as input, and then uses two information processing paths, namely, a knowledge distillation method based on a diffusion model and a target track generation method based on spatiotemporal interactive reasoning to achieve open set target detection and recognition and output a stable target track with spatiotemporal consistency; The multimodal fusion feature set is obtained by using convolutional neural networks and Transformer network models. First, time-sensitive target images of different modalities are collected in advance as training sets. Then, convolutional neural networks are used to extract features of images of different modalities in the training sets. After obtaining the corresponding feature vectors, the multimodal feature vectors are uniformly input into the Transformer network model to fuse the multimodal features and obtain the multimodal fusion feature set F. fuse ; S2, based on the model constructed in step S1, using the knowledge distillation method based on the diffusion model to realize open set target detection and recognition; S3, based on the model constructed in step S1, a target track generation method based on time-space interactive reasoning outputs a stable target track with time-space consistency; S4, based on steps S1-S3, realize open set target detection and recognition and output stable target tracks with temporal and spatial consistency, and complete the discovery of time-sensitive targets; The step S2 is specifically as follows: S21, input the multimodal fusion feature set, add the position code, and use the Unified Transformer network model to process the randomly initialized target feature set to form a captured target feature set; Multimodal fusion feature set F fuse As input, the feature set is divided into 8×8 blocks of the same size, and the sinusoidal encoding rule is used to add position encoding PE to the features of different blocks. Then, the target feature set {q} is randomly initialized to convert the prediction target into a learnable query embedding; Then, the input features are passed into the Unified Transformer network model, and the self-attention transformation mechanism in the model is used to extract key semantic structure information from the input features. The cross-attention mechanism is used to drive the initialized query embedding set to capture different target structure information to form a captured target feature set {f}. A decoder that only focuses on the position information in the feature set is used to regress the prediction box. S22, based on step S21, train the pre-trained CLIP model and proceed to step S24; First, the decoder in the Unified Transformer network model is trained for self-supervision of prediction boxes using pre-collected time-sensitive target images of different modalities. During the training process, the predicted set y is compared with the target true value Match associations and deconstruct them into bipartite graph matching tasks, so that one query embedding in the set corresponds to one target, and the number of query embeddings Q should be much larger than the number of targets; S23, based on step S21, semantically independent decoding is performed to obtain a prediction frame, and a Hungarian matching function is used to achieve a one-to-one matching between multiple prediction targets and true values; The matching function uses the Hungarian algorithm, and the expression is as follows: Where N represents the total number of targets in the prediction set or the true set. Represents the category similarity loss between the predicted box and the true box, c n represents an element in the cost matrix, each element of the cost matrix represents the cost or price of assigning the i-th task to the j-th person; L box Represents the similarity loss between the regressed prediction box and the true box; By minimizing the Hungarian matching loss, a one-to-one matching of multiple predicted targets and true values ​​is achieved; S24, based on steps S21 and S22, target recognition is performed under the condition of few samples or zero samples, that is, pre-training prior knowledge is transferred by using the diffusion model on the basis of capturing the target feature set, and then entering step S25; First, we add noise layer by layer based on the diffusion model, and use KL divergence to constrain the open set feature x so that its feature distribution tends to the Gaussian distribution N(μ,I). We then integrate the open set feature x into the capture target feature set, while keeping the original features unchanged and incorporating the open set knowledge. Then, based on the diffusion model, layer-by-layer denoising is performed, that is, inverse denoising. The pre-trained CLIP model is trained to learn the conditional distribution probability to remove the Gaussian noise with a distribution function of N(0,I). The training loss expression is as follows: Among them, μ represents the mean, I represents the standard deviation, t1 represents the current number of diffusion layers, and x t represents the input image of the current diffusion layer, ε represents the standard normal distribution, ε θ represents the noise distribution removed in the back propagation of the diffusion model; S25. For few-shot and zero-shot target recognition, a prototype network is proposed. The sample data in each category is mapped into a space using metric learning, and the "mean" is extracted as the prototype of the category. Perform open set recognition task training, set the number of features in the multimodal fusion feature set to N, and each feature is F n , the corresponding real text feature is By minimizing the cosine distance, the Unified Transformer network model embedded in the semantic structure of the pre-trained CLIP model maps the features of the query feature set and the target semantic features extracted by the CLIP text encoder into the same feature space to achieve alignment; In the few-shot classification task, S = {(x1,u1),...,(x N ,u N )} represents a small-scale support dataset; x represents the vectorized representation of the original data in D dimensions, u represents its corresponding category, S k Represents a data set with category k; The prototype network computes a prototype representation v for each class j , through an embedding function f Φ R D →R M Map the sample data of dimension D to the space of dimension M. The embedding function uses a simple linear mapping, f Φ =Wx, W represents a parameter matrix, the purpose is to transform the dimension of x, R represents the letter representation of the dimensional space; it supports the average value of all vectorized sample data to obtain the prototype representation v of the category k , the expression is as follows: The prototype network uses softmax to measure the query vector point to v k The distance is expressed as follows: Among them, p Φ (·) represents the probability function, d(·) represents the distance from the query vector to v k The distance, k represents the true category label of the training sample, and k′ represents the set of all k values; The training process minimizes the objective function through stochastic gradient descent: J(Φ) = -logp Φ (y=k|x); S26: Based on steps S22-S25, enter the prediction stage to complete the positioning of anchor frames and identification of types for few-sample and zero-sample targets; Based on steps S22-S25, the training phase is completed and the prediction phase is entered. First, the category text to be recognized is preprocessed with a template prompt, and the template is "a photo of {}"; Among them, {} is any open class to be identified; Then, the category text is sent to the pre-trained CLIP text encoder to extract text features, and a non-parametric similarity calculation is performed between it and the query feature set to find the text feature that is most similar to each feature in the feature set. The corresponding word is the category to be identified. Finally, knowledge transfer from visible classes to invisible classes is achieved, and anchor box R is localized for few-shot and zero-shot targets. box and identification type R cls ; The step S3 is specifically as follows: S31, input the multimodal fusion feature set, use the conditional score network CSN to estimate the weight of the multimodal fusion feature and use it as a guide to selectively fuse the different modal features through the adaptive fusion network; The multimodal fusion feature set F fuse As input, we first use the conditional score network (CSN) to estimate the weights of multimodal fusion features. The conditional score network models the probability distribution gradient of data in different scenarios as a score function and uses a scoring mechanism to obtain the matching score. The multimodal fusion weights estimated by the conditional score network are used as a guide to selectively fuse different modal features through an adaptive fusion network. S32, based on step S31, using the Unified Transformer network model to jointly process the randomly initialized target feature set and the target single-frame associated fusion features to obtain a captured target feature set, and perform feature time series expansion; The Unified Transformer network model is used to jointly process the randomly initialized target feature set {q} and the target single-frame associated fusion features to obtain the captured target feature set {f}, and then expand it to multiple frames of data in the time dimension to form a multi-frame captured target feature set {f} T ; Among them, the randomly initialized target feature set of the current frame will replace the captured target feature set of the previous frame {f target } t-1 ,t∈T is taken into the initialization as input, T represents the overall time series; S33, based on step S32, designing a spatiotemporal transformer to automatically enhance feature relationships and infer target tracks, and outputting target tracks with spatiotemporal consistency; The space-time transformer includes: a space domain encoder and a time domain decoder, which are respectively composed of a Unified Transformer network model encoder and a decoder; The spatial domain encoder is used to extract spatial context and infer visual relationships within a frame, while the temporal domain decoder takes the output of the spatial domain encoder as input, captures temporal dependencies between frames and infers dynamic relationships between objects; The spatiotemporal transformer continuously mines the associations between targets in the multi-frame embedded feature set in the spatial and temporal domains to output a target track R with spatiotemporal consistency. track ; In the process of training the spatiotemporal interactive reasoning track association and enhancement based on the Unified Transformer network model, the loss function consists of the per-frame positioning recognition loss and the tracking query embedding loss, and the expression is as follows: Among them, N q Indicates the number of target features, N track Indicates the number of targets successfully detected in the previous frame, L object Represents the loss function for each target feature during training.

Images