Language specified multi-target visual tracking method and system based on space-time enhancement

Abstract

The invention discloses a language-specified multi-target visual tracking method and system based on spatio-temporal enhancement. The method comprises the following steps: acquiring a time-frequency sequence and a corresponding natural language expression; inputting the video frame into a visual backbone network to obtain a visual feature, and inputting the natural language expression into a semantic backbone network to obtain a text feature; inputting the visual features and the text features into an early fusion module for cross-modal fusion to obtain cross-modal features; the cross-modal features are input into a Transform module for encoding and decoding, and an initial target detection frame and query features are obtained; and inputting the initial target detection frame and the query features into a space-time enhancement module for optimization to obtain an optimized target detection frame. According to the invention, the sensing capability of motion information can be effectively improved, and the accuracy and robustness of target tracking are enhanced.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer vision and natural language processing, and in particular to a language specified multi-target visual tracking method based on space-time enhancement, a system, a storage medium and an electronic device. BACKGROUND

[0002] Humans can quickly locate and track specific targets in complex dynamic scenes relying on language descriptions. This natural language-based target reference and continuous perception capability is an important research direction in human-computer collaboration and intelligent perception. Referring Multi-Object Tracking (RMOT) is a new task proposed to simulate this human capability. Compared with traditional Multi-Object Tracking (MOT) which only relies on visual features, RMOT introduces natural language instructions, combining high-level semantic information, making the system more flexible and adaptable in the face of complex environments, occlusions, similar target interference, etc. In recent years, it has received extensive attention.

[0003] Existing referring multi-object tracking methods, especially end-to-end modeling schemes, mainly focus on the fusion of visual and language features within a single frame, and perform target detection and matching based on current frame information. This single-frame modeling approach can be compatible with various existing detection and matching frameworks, but it is relatively rough for dynamic modeling across time dimensions, and there is a problem of insufficient modeling of target motion states. Due to the lack of modeling of temporal information between consecutive frames, existing methods generate noisy feature representations and weak temporal coherence when dealing with complex spatio-temporal changes such as occlusions, rapid motion, target deformation, and reappearance out of the field of view, resulting in a significant gap in tracking accuracy and robustness compared to static scenes. SUMMARY

[0004] The embodiments of the present application provide a language specified multi-target visual tracking method based on space-time enhancement, a system, a storage medium and an electronic device, which can effectively improve the perception of motion information and enhance the accuracy and robustness of target tracking.

[0005] The embodiments of the present application provide a language specified multi-target visual tracking method based on space-time enhancement, which comprises: Obtaining a time-frequency sequence and a corresponding natural language expression; wherein the video sequence comprises a plurality of video frames; Inputting the video frames and the natural language expression into a multi-target visual tracking model to obtain a visual tracking result; The multi-target visual tracking model comprises a visual backbone network, a semantic backbone network, an early fusion module, a Transformer module and a space-time enhancement module; the processing process of the multi-target visual tracking model comprises: inputting the video frame into the visual backbone network to obtain visual features, and inputting the natural language expression into the semantic backbone network to obtain text features; inputting the visual features and the text features into the early fusion module for cross-modal fusion to obtain cross-modal features; inputting the cross-modal features into the Transformer module for encoding and decoding to obtain an initial target detection frame and query features; inputting the initial target detection frame and the query features into the spatio-temporal enhancement module for optimization to obtain an optimized target detection frame.

[0006] Further, the above language specified multi-target visual tracking method based on spatio-temporal enhancement, wherein the processing process of the early fusion module comprises: linearly projecting the visual features and the text features respectively to map to a unified feature dimension space; calculating the correlation between the visual spatial positions of the linearly projected visual features and the text semantic units of the linearly projected text features based on a multi-head attention mechanism to obtain a cross-modal attention score matrix; fusing the visual features and the text features based on the attention score matrix to obtain cross-modal features.

[0007] Further, the above language specified multi-target visual tracking method based on spatio-temporal enhancement, wherein the calculating the correlation between the visual spatial positions of the linearly projected visual features and the text semantic units of the linearly projected text features based on a multi-head attention mechanism to obtain a cross-modal attention score matrix comprises: performing a flattening operation on the linearly projected visual features to transform them into a sequence form wherein, represents the number of spatial positions; the text features remain as each position corresponds to the representation of a text token; the visual features are taken as the query, the text features are taken as the key and the value, and a cross-modal attention score matrix is obtained through a standard attention calculation formula:

[0008] wherein, is a learnable linear transformation matrix, is set to , is the number of attention heads, represents the a relative matching degree between a text token and a visual position.

[0009] Further, the above-mentioned language specified multi-object visual tracking method based on spatio-temporal enhancement, wherein the fusion of the visual feature and the text feature based on the attention score matrix obtains a cross-modal feature, including: weighting the text feature based on the cross-modal attention score matrix to generate a text representation corresponding to a visual spatial position one by one , the calculation formula is:

[0010] wherein, is a value projection matrix, is a text feature, is a weighted text feature; element-wise addition of the weighted text feature and the visual feature to obtain a fused cross-modal feature representation : .

[0011] Further, the above-mentioned language specified multi-object visual tracking method based on spatio-temporal enhancement, wherein the Transformer module includes an encoder, a decoder and a multi-layer perception machine, and the processing process of the Transformer module includes: in the encoder, the cross-modal feature is processed based on the multi-head attention mechanism to obtain the context-enhanced encoder feature; in the decoder, the query feature is generated based on the query-decoder mechanism, and the query feature is updated; in the multi-layer perception machine, each query feature is detected to obtain an initial target frame, a predicted category and a probability distribution of the predicted category.

[0012] Further, the above-mentioned language specified multi-object visual tracking method based on spatio-temporal enhancement, wherein the query feature is generated based on the query-decoder mechanism, and the query feature is updated, including: initialize a set of learnable query features wherein, is a predefined number of queries; based on the self-attention mechanism, model within the query feature, the formula is as follows:

[0013] wherein, indicates the self-attention mechanism; ​Based on the cross attention mechanism, the query feature retrieves the most relevant region information from the visual features; Perform a linear transformation on the query features and encoder features to obtain the query, key, and value required for cross-attention:

[0014]

[0015]

[0016] in, is the learnable parameter matrix, 、 、 Query, key and value respectively; Perform cross-attention calculation, the calculation formula is:

[0017] in, The operation is normalized along the dimension of Key. for cross-attentional results; Based on the cross-attention results, update the query features:

[0018] in, The updated features.

[0019] Furthermore, in the above-mentioned language-specified multi-target visual tracking method based on spatiotemporal enhancement, the spatiotemporal enhancement module includes an object decoder and a timing decoder; The processing process of the spatiotemporal enhancement module includes: In the timing decoder, and stored in the historical memory; Interacting the preliminary query representation of the current frame with the query features in the historical memory through a cross-attention mechanism to obtain query features enhanced by temporal order; The query features and the initial target detection frame after temporal enhancement are input into the object decoder to obtain an optimized target detection frame.

[0020] Furthermore, in the above-mentioned language-specified multi-target visual tracking method based on spatiotemporal enhancement, the object decoder includes a detection head network, and the query features after temporal enhancement and the initial target detection box are input into the object decoder to obtain the optimized target detection box, including: The internal interaction modeling of the time-enhanced query features is performed through the multi-head self-attention mechanism to obtain the updated time-enhanced query features; input the updated temporal enhancement query feature into the detection head network to obtain predicted target bounding box parameters and class confidence; optimize the initial target detection box based on the predicted target bounding box parameters to obtain an optimized target detection box.

[0021] The embodiment of the application further provides a language specified multi-target visual tracking system based on space-time enhancement, comprising: An acquisition module is configured to acquire a time-frequency sequence and a corresponding natural language expression; wherein the video sequence comprises a plurality of video frames; A multi-target visual tracking module is configured to input the video frames and the natural language expression into a multi-target visual tracking model to obtain a visual tracking result. The multi-target visual tracking model comprises a visual backbone network, a semantic backbone network, an early fusion module, a Transformer module and a space-time enhancement module; and a processing procedure of the multi-target visual tracking model comprises: The video frames are input into the visual backbone network to obtain visual features, and the natural language expression is input into the semantic backbone network to obtain text features; The visual features and the text features are input into the early fusion module for cross-modal fusion to obtain cross-modal features; The cross-modal features are input into the Transformer module for encoding and decoding to obtain an initial target detection box and a query feature; The initial target detection box and the query feature are input into the space-time enhancement module for optimization to obtain an optimized target detection box.

[0022] The embodiment of the application further provides a computer readable storage medium, wherein a plurality of instructions are stored in the computer readable storage medium, and the instructions are suitable for being loaded by a processor to execute any one of the above language specified multi-target visual tracking methods based on space-time enhancement.

[0023] The embodiment of the application further provides an electronic device comprising a processor and a memory, wherein the processor is electrically connected with the memory, the memory is configured to store instructions and data, and the processor is configured to execute steps in any one of the above language specified multi-target visual tracking methods based on space-time enhancement.

[0024] The language specified multi-target visual tracking method based on spatio-temporal enhancement provided by the application, system, storage medium and electronic device significantly improve the accuracy and robustness of target tracking. By effectively fusing video sequences and natural language descriptions, combining visual backbone networks and semantic backbone networks, extracting and fusing cross-modal features, the motion information of the target can be more accurately captured. The Transformer module is introduced to encode and decode the cross-modal features, effectively generating the initial target detection box, and the spatio-temporal enhancement module further combines multi-frame temporal information and semantic information to optimize the prediction of the target detection box. This method enhances the perception of motion information, and shows higher robustness and accuracy than traditional methods in multi-target tracking tasks, and is suitable for more complex dynamic environments, providing a more efficient and reliable solution for practical applications. BRIEF DESCRIPTION OF DRAWINGS

[0025] The technical solutions and other beneficial effects of the application will become apparent from the following detailed description of the specific embodiments of the application, combined with the accompanying drawings.

[0026] Figure 1 The flowchart of the language specified multi-target visual tracking method based on spatio-temporal enhancement provided by the embodiment of the application.

[0027] Figure 2 The flowchart of the early fusion module and the spatio-temporal enhancement module provided by the embodiment of the application.

[0028] Figure 3 The visual result graph of the multi-target tracking provided by the embodiment of the application.

[0029] Figure 4 The structure schematic diagram of the language specified multi-target visual tracking system based on spatio-temporal enhancement provided by the embodiment of the application.

[0030] Figure 5 The structure schematic diagram of the electronic device provided by the embodiment of the application. DETAILED DESCRIPTION

[0031] The technical solutions in the embodiments of the application will be described clearly and completely below with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are only part of the embodiments of the application, not all the embodiments. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the application.

[0032] With the rapid development of computer science and artificial intelligence, referring expression resolution, as an important research direction in the intersection of computer vision and natural language processing, has gradually attracted the attention of many scholars. However, the traditional referring expression resolution task faces two challenges: on the one hand, a single expression may refer to multiple targets in an open world; on the other hand, the expression often only covers the local time state of the target, and cannot accurately describe the complete trajectory change of the target. To solve these problems, some scholars have proposed the task of referring multi-object tracking, which aims to detect and track all targets matching the given semantic expression in the video sequence, so as to realize accurate tracking and positioning of multiple targets in the video.

[0033] The previous end-to-end referring multi-object tracking method usually only establishes the object matching relationship based on the correlation between adjacent frames, ignores the effective modeling of long-term time clues, and thus the overall performance is limited. Specifically, the lack of time modeling brings two problems: on the one hand, the identification accuracy of the moving state target is limited. In the RMOT task, expressions closely related to the moving state, such as "moving car" or "person walking", are crucial to understanding complex scenes, and such dynamic changes often need to rely on long-time span information to infer. It is difficult to accurately capture dynamic features such as speed changes based on the correlation between adjacent frames, which may cause target omission. On the other hand, a model that only relies on short-term information performs poorly when dealing with temporary occlusion, and it is difficult to accurately recover the target trajectory, resulting in re-identification errors after occlusion, causing ID switching, and seriously affecting the tracking stability. The present application believes that the lack of time dynamic modeling in the existing method limits the comprehensive understanding and modeling of the target evolution process, which becomes an important bottleneck restricting the performance improvement of referring multi-object tracking.

[0034] To solve the above problems, the embodiment of the present application provides a language specified multi-object visual tracking method based on spatiotemporal enhancement, a system, a storage medium and an electronic device. The language specified multi-object visual tracking system based on spatiotemporal enhancement provided by the embodiment of the present application can be integrated in an electronic device, which can be a terminal, a server, etc. The terminal can include a tablet computer, a notebook computer, a personal computer (PC), a micro processing box, or other devices, etc.

[0035] Please refer to Figure 1 , Figure 1 The flowchart of the language specified multi-object visual tracking method based on spatiotemporal enhancement provided by the embodiment of the present application is applied in an electronic device, and the language specified multi-object visual tracking method based on spatiotemporal enhancement includes the following steps: S1, obtaining a time-frequency sequence and a corresponding natural language expression; wherein the video sequence includes a plurality of video frames.

[0036] Divide the original input video into a frame sequence at a fixed interval . And for each frame image , perform standardization processing to adjust to WxH (default 640x480) resolution. At the same time, obtain the natural language expression describing the tracking target.

[0037] S2, input the video frame and the natural language expression into the multi-target visual tracking model to obtain the visual tracking result.

[0038] The multi-target visual tracking model includes a visual backbone network, a semantic backbone network, an early fusion module, a Transformer module, and a space-time enhancement module; the processing process of the multi-target visual tracking model includes: S21, input the video frame into the visual backbone network to obtain the visual feature, and input the natural language expression into the semantic backbone network to obtain the text feature.

[0039] The pre-trained visual backbone network (such as ResNet, ViT) is used to extract the hierarchical visual features of the video frame, capturing the local texture and global spatial information of the target; in parallel, the semantic backbone network (such as BERT, CLIP text encoder) is used to extract the deep semantic features of the natural language expression, modeling the context association between words. The dual-modal features are represented as visual feature and text feature .

[0040] S22, input the visual feature and the text feature into the early fusion module for cross-modal fusion to obtain the cross-modal feature.

[0041] Figure 2 For the processing flowchart of the early fusion module and the space-time enhancement module, please refer to Figure 2 In an embodiment, the processing process of the early fusion module includes: S221, respectively linearly project the visual feature and the text feature to map to a unified feature dimension space.

[0042] The visual feature and the text feature are respectively mapped to a unified feature dimension space , wherein is a preset feature dimension. This dimension unification processing ensures the operability of subsequent cross-modal fusion.

[0043] S222, based on the multi-head attention mechanism, calculate the correlation between the visual spatial position of the linearly projected visual feature and the text semantic unit of the linearly projected text feature to obtain a cross-modal attention score matrix.

[0044] The visual features after linear projection are flattened and transformed into a sequence wherein, represents the number of spatial positions; the text features remain as Each position corresponds to the representation of a text token. The visual features are taken as the query, and the text features are taken as the key and the value, and the cross-modal attention score matrix is obtained through the standard attention calculation formula :

[0045] wherein, is a learnable linear transformation matrix, is set to , is the number of attention heads, represents the relative matching degree between the th text token and the th visual position.

[0046] S223, the visual features and the text features are fused based on the attention score matrix to obtain cross-modal features.

[0047] The text features are weighted based on the cross-modal attention score matrix to generate text representations corresponding to the visual spatial positions , and the calculation formula is:

[0048] wherein, is a value projection matrix, which is used to further transform the text features, is the text feature, is the weighted text feature. This step essentially generates an adaptive semantic context description for each visual position.

[0049] The weighted text features and the visual features are added element by element to obtain the fused cross-modal feature representation : .

[0050] S23, the cross-modal features are input into the Transformer module for encoding and decoding to obtain the initial target detection frame and the query feature.

[0051] In one embodiment, the Transformer module includes an encoder, a decoder, and a multi-layer perception. After completing the visual-text cross-modal fusion, the obtained fusion features contains rich visual spatial structure and semantic context information. In order to further mine the high-order correlation between features and improve the accuracy of the generated bounding box, a standard Transformer structure is used for encoding and decoding. The processing process of the Transformer module includes: S231, in the encoder, the cross-modal features are processed based on the multi-head attention mechanism to obtain the context-enhanced encoder features.

[0052] The cross-modal features are input into the Transformer encoder module. The encoder is composed of multiple layers of stacked multi-head self-attention and feedforward networks, which can model long-distance dependencies within the fused features, strengthen the interaction between local regions and different semantic entities, and obtain context-enhanced encoding features.

[0053] Step S231 specifically includes: S2311, in the self-attention module, the features are first respectively through the learnable matrix to generate queries, keys and values:

[0054]

[0055]

[0056] S2312, calculate the self-attention score:

[0057] S2313, after multi-layer stacking, output the context-enhanced encoder features:

[0058] S232, in the decoder, generate query features based on the query-decoder mechanism, and update the query features.

[0059] Step S232 specifically includes: S2321, initialize a set of learnable query features , wherein is the predefined number of queries.

[0060] Each query can be understood as an initial representation of a potential target, and is randomly initialized, then gradually optimized in the training process, with the ability to actively learn specific target classes and location information.

[0061] S2322, based on the self-attention mechanism, model the relationship between different potential targets within the query features to avoid repeated detection or omission, which is represented by the following formula:

[0062] wherein, denotes the self-attention mechanism.

[0063] S2323, based on the cross-attention mechanism, the query feature retrieves the most relevant area information from the visual feature, realizes cross-modal information fusion and enhancement.

[0064] S2324, linear transformation is performed on the query feature and the encoder feature to obtain the query, key and value required for cross-attention:

[0065]

[0066]

[0067] wherein, is a learnable parameter matrix, , , query, key and value respectively; S2325, cross-attention calculation is performed, and the calculation formula is:

[0068] wherein, the operation is normalized along the dimension of Key to ensure the interpretability and stability of the attention weight, is the cross-attention result.

[0069] S2326, according to the cross-attention result, the query feature is updated:

[0070] wherein, is the updated feature.

[0071] S233, in the multi-layer perception, each query feature is detected to obtain the initial target frame, the predicted category and the probability distribution of the predicted category.

[0072] This step uses a multi-layer perceptron (MLP) module to process each query individually, complete the detection frame parameter regression and category prediction of the target instance. The specific steps are as follows: S2331, based on a regression branch, the updated query feature is input into the multi-layer perception to output the regression results of the center coordinates and the width and height size of the detection frame:

[0073]

[0074] S2332, input the updated query feature into the classifier based on the other two independent classification branches. The classifier is used to output the class prediction probability distribution corresponding to each query and the probability distribution matched with the text:

[0075]

[0076] S24, input the initial target detection frame and the query feature into the spatio-temporal enhancement module for optimization to obtain the optimized target detection frame.

[0077] Please continue to refer to Figure 2 , the spatio-temporal enhancement module can combine multi-frame time sequence information and semantic information to optimize the prediction of the target detection frame. To further improve the spatio-temporal consistency and semantic accuracy of the detection frame, the method introduces a spatio-temporal enhancement module after the initial detection frame is generated, fully utilizes multi-frame time sequence information and text semantic information, and optimizes the initial prediction result.

[0078] The spatio-temporal enhancement module includes an object decoder and a time sequence decoder, and the processing process of the spatio-temporal enhancement module includes: S241, in the time sequence decoder, a historical detection query set is constructed based on the updated query feature and stored in a historical memory bank.

[0079] To introduce the continuity information in the time dimension, the system continuously maintains a dynamic memory bank for storing the detection query representation of the previous several frames. Let the current frame be the th frame, and the historical detection query set is denoted as:

[0080] wherein, represents the th query feature in the th frame, is the feature dimension, is the number of historical frames. The memory bank is updated in real time in the form of a sliding window, and only the latest frame representation is retained.

[0081] S242, the preliminary query representation of the current frame and the query features in the historical memory bank are interacted through a cross-attention mechanism to obtain the query features enhanced by time sequence.

[0082] For the preliminary query representation of the current frame, the cross-attention mechanism is used to interact with the query features in the historical memory bank to explicitly model the consistency and continuity of the target in the time dimension:

[0083] in, The query features after time series enhancement have stronger target identity and time series expression capabilities. This process effectively aggregates the historical trajectory information of the unified target in the historical frame, improving the stability and discrimination ability of the query.

[0084] S243: Input the query features after temporal enhancement and the initial target detection frame into the object decoder to obtain an optimized target detection frame.

[0085] The object decoder aims to further optimize the structural relationship and category differentiation ability between objects in the current frame. The object decoder includes a detection head network. Step S243 includes: S2431, through the multi-head self-attention mechanism, the internal interaction modeling of the query features after time series enhancement is performed to obtain the updated time series enhanced query features. The multi-head self-attention mechanism is used to model the internal interactions of each query in the current frame, capturing the contextual dependencies and spatial structure information between different objects in the same frame, thereby improving the accuracy of intra-frame recognition:

[0086] S2432, input the updated temporal enhanced query features into the detection head network to obtain the predicted target bounding box parameters and category confidence, The optimized query vector It is fed into an independent detection head network, including a multi-layer perceptron branch, to predict the bounding box parameters (center coordinates and width and height offsets) and category confidence of each object, as follows:

[0087]

[0088]

[0089] S2433: Optimize the initial target detection frame based on the predicted target bounding box parameters to obtain an optimized target detection frame.

[0090] The prediction result of the current frame is dynamically corrected based on the temporal information of the historical frames to obtain the final bounding box parameters:

[0091] Furthermore, the method further includes: constructing a loss function based on the predicted bounding boxes and the true labels, and training the multi-target visual tracking model based on the loss function. The loss function includes a detection loss and a spatiotemporal loss.

[0092] Detection loss This loss is used to optimize the model's target detection ability on single-frame images, and to constrain the model's accuracy in judging target visibility, bounding box position, and language reference relationship in the current frame. For the initial detection result predicted by the model for the first frame , and the corresponding true label , the loss function is defined as follows:

[0093] wherein, represents the classification loss of detection, which uses Focal Loss to suppress the unevenness of positive and negative samples. is the position regression loss, which comprehensively uses loss and IOU loss. represents the reference correlation prediction loss, which is also trained using Focal Loss. is a weighting coefficient that controls the contribution of each sub-task to the overall loss.

[0094] Temporal-spatial loss This part of the loss is aimed at the time series enhanced prediction result that integrates historical information, further guiding the model to learn time-consistent representations and strengthen the cross-time modeling ability of semantics and motion. The loss is defined as follows:

[0095] wherein, represents the classification loss, which uses Focal Loss to suppress the unevenness of positive and negative samples. is the position regression loss, which comprehensively uses loss and IOU loss. represents the reference correlation prediction loss, which is also trained using Focal Loss. is a weighting coefficient that controls the contribution of each sub-task to the overall loss.

[0096] A specific embodiment is shown below: Dataset The present application is trained and tested on two public reference multi-target tracking datasets Refer-KITTI, Refer-KITT-V2 and one public multi-target tracking dataset KITTI.

[0097] ​For Refer-KITTI, Refer-KITT-V2 and KITTI datasets, the following strategies are adopted. In terms of parameter initialization, the early fusion module and the spatio-temporal enhancement module are randomly initialized, the parameters of the text encoder are kept frozen, and the parameters of other modules are loaded from the official Deformable DETR weights pre-trained on the COCO dataset. To improve the multi-scale perception ability of the model, the random cropping data augmentation strategy is introduced, and the image shortest side size range is set to 800-1536. In addition, referring to MOTRv2, target erasing and target inserting operations are further added to simulate the entry and exit of targets. In terms of the optimizer, the AdamW optimizer is used for training, the base learning rate is set to 1e-4, and the backbone network learning rate is set to 1e-5. The total training rounds are set to 100, and the learning rate is reduced to 1 / 10 at the 50th epoch. Training is performed on 4 RTX 4090 GPUs, with a single-card batch size of 1.

[0098] The HOTA (Higher Order Tracking Accuracy) index is used to evaluate the performance of the model on the Refer-KITTI, Refer-KITTI-V2 and KITTI datasets. This index consists of three parts: Detection, Association and HOTA total score. The Detection index is used to evaluate the accuracy of the model in the target detection task; the Association index is used to measure the consistency of the target identity in multi-target tracking; and the HOTA total score is a weighted evaluation of the previous two, to comprehensively reflect the overall performance of the model in the target detection and tracking task.

[0099] As shown in Table 1, the proposed method achieves the best performance on the Refer-KITTI dataset, verifying its effectiveness in the referential multi-target tracking task. In terms of the HOTA total index, the proposed method improves the previous best TransRMOT by 5.96%, fully demonstrating the superior performance of TRMOT in complex referential scenarios. In addition, thanks to the introduction of the spatio-temporal enhancement module, the association performance of the model is significantly enhanced, with AssA (association accuracy) improving from 62.48% to 66.56%; at the same time, DetA (localization accuracy) also rises from 37.97% to 41.53%, with an improvement of 3.56%. The above results show that the proposed method has made significant progress in both detection and association dimensions, demonstrating good comprehensive tracking capability.

[0100] Table 1 Comparison of performance of different methods and the proposed method on Refer-KITTI

[0101] 4.2 As shown in Table 2, we also conducted a system comparison on Refer-KITTI-V2. The experimental results show that the present method achieves the optimal performance in all evaluation indexes, and the HOTA total score reaches 31.79%, which is improved by 6.01% compared with the second highest TransRMOT, showing a significant advantage. Combined with the performance on the Refer-KITTI dataset, it further confirms the robustness and superiority of the algorithm proposed in the present application in diversified reference scenarios, indicating that it has good adaptability under different data distributions and complexity.

[0102] Table 2 Performance comparison results of different invented methods and the present method on Refer-KITTI-V2

[0103] 4.3 To further verify the generalization ability and scalability of TRMOT, we compared it with the current mainstream SOTA multi-target tracker on the KITTI dataset, and the experimental results are shown in Table 3. To ensure fairness of comparison, we followed the training set and test set division method of KITTI dataset in Refer-KITTI.

[0104] As can be seen from the table, TRMOT still achieves the optimal performance in the full target detection task, showing excellent comprehensive performance. In terms of key indicators, the HOTA score of TRMOT is 62.76%, which is improved by 1.24% compared with TransRMOT; in terms of AssA (association accuracy), it achieves a significant improvement, from 66.50% to 70.88%, with an increase of 4.38%. This result fully shows that the spatiotemporal enhancement module proposed in the present application plays an important role in improving the association modeling ability between targets.

[0105] Table 3 Performance comparison results of different invented methods and the present method on KITTI

[0106] 5. Qualitative results of experiments To more intuitively show the performance advantage of the present method in complex semantic instruction understanding and multi-target continuous tracking, Figure 3 the visualization result graph of the reference multi-target tracking provided by the embodiments of the present application is shown in FIG. 1. Figure 3 As shown in FIG. 1, Figure 3(a), (b), (c), (d), and (e) are the semantic segmentation results obtained when given the natural language instructions (based on semantics) related to movement: "moving cars", "car in front of ours", "left cars which are parking", "parking cars", and "right persons who are walking". The method of the present invention can accurately identify semantic targets and achieve stable cross-frame tracking, reflecting powerful semantic parsing and dynamic perception capabilities.

[0107] Compared to the existing method, TransRMOT, this method significantly outperforms the comparison method in maintaining target trajectory continuity and significantly reduces tracking interruptions. This performance improvement is primarily due to the newly introduced timing enhancement module in the method. This module accurately captures target velocity changes by modeling multi-frame information, thereby achieving more precise motion understanding. For example, in the "moving cars" task, TransRMOT repeatedly lost the target, while this method consistently maintained stable tracking of the target, fully demonstrating its temporal perception advantage.

[0108] Furthermore, this method possesses excellent state discrimination capabilities, effectively distinguishing between similar but different targets. When processing the commands "left cars which are parking" and "parking cars," TransRMOT tends to misjudge slow-moving vehicles as stationary, while the proposed method can accurately identify their actual motion state. In the "right persons who are walking" task, TransRMOT often confuses stationary pedestrians with walking groups, leading to false detections. In contrast, this method can accurately lock onto walking individuals, demonstrating a stronger ability to understand motion semantics.

[0109] like Figure 3 As shown in (d), our method also demonstrates excellent adaptability in scenarios with temporary state changes. When a red vehicle pauses at a traffic light and then restarts, our method accurately determines that it is in the "moving" semantic state and enables uninterrupted tracking. This result further demonstrates the robustness and practicality of our method in complex scenarios.

[0110] According to the method described in the above embodiment, this embodiment will be further described from the perspective of a language specified multi-target visual tracking system based on spatio-temporal enhancement, which can be implemented as an independent entity or integrated in an electronic device, which can be a terminal, a server, or the like. The terminal can include a tablet computer, a notebook computer, a personal computer (PC), a micro processing box, or other devices, etc.

[0111] Please refer to Figure 4 , Figure 4 The language specified multi-target visual tracking system based on spatio-temporal enhancement provided by the embodiments of the present application is specifically described, which is applied to an electronic device. The language specified multi-target visual tracking system based on spatio-temporal enhancement can include: An acquisition module configured to acquire a time-frequency sequence and a corresponding natural language expression. The video sequence includes a plurality of video frames. A multi-target visual tracking module configured to input the video frames and the natural language expression into a multi-target visual tracking model to obtain a visual tracking result. The multi-target visual tracking model includes a visual backbone network, a semantic backbone network, an early fusion module, a Transformer module, and a spatio-temporal enhancement module. The processing process of the multi-target visual tracking model includes: The video frames are input into the visual backbone network to obtain visual features, and the natural language expression is input into the semantic backbone network to obtain text features. The visual features and the text features are input into the early fusion module for cross-modal fusion to obtain cross-modal features. The cross-modal features are input into the Transformer module for encoding and decoding to obtain initial target detection boxes and query features. The initial target detection boxes and the query features are input into the spatio-temporal enhancement module for optimization to obtain optimized target detection boxes.

[0112] In specific implementation, each of the above modules and / or units can be implemented as an independent entity, or can be combined as the same or several entities. The specific implementation of each of the above modules and / or units can be referred to the method embodiments above, and the beneficial effects that can be achieved can also be referred to the beneficial effects in the method embodiments above, which will not be described here again.

[0113] In addition, the application further provides an electronic device, which can be a computer, a tablet computer, or the like. The electronic device can implement the steps in any embodiment of the language specified multi-target visual tracking method based on space-time enhancement provided by the application, and thus can implement the beneficial effects of any language specified multi-target visual tracking method based on space-time enhancement provided by the application. For details, see the foregoing embodiments, which will not be described here again.

[0114] Figure 5 A specific structural block diagram of an electronic device provided by the embodiments of the application is shown, which can be used to implement the language specified multi-target visual tracking method based on space-time enhancement provided in the foregoing embodiments. The electronic device 500 can be a terminal, a server, or the like, wherein the terminal can include a tablet computer, a notebook computer, a personal computer (PC), a micro processing box, or other devices, and the like.

[0115] RF circuit 510 is used to receive and transmit electromagnetic waves, converting them into electrical signals, thereby enabling communication with a communications network or other devices. RF circuit 510 may include various existing circuit components for performing these functions, such as an antenna, a radio frequency transceiver, a digital signal processor, an encryption / decryption chip, a subscriber identity module (SIM) card, memory, and the like. RF circuit 510 can communicate with various networks, such as the Internet, an intranet, or a wireless network, or with other devices via a wireless network. These wireless networks may include cellular telephone networks, wireless local area networks, or metropolitan area networks. The wireless networks may utilize various communication standards, protocols, and technologies, including but not limited to Global System for Mobile Communication (GSM), Enhanced Data GSM Environment (EDGE), Wideband Code Division Multiple Access (WCDMA), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Wireless Fidelity (Wi-Fi) (such as Institute of Electrical and Electronics Engineers standards IEEE 802.11a, IEEE 802.11b, IEEE802.11g, and / or IEEE802.11n), Voice over Internet Protocol (VoIP), Worldwide Interoperability for Microwave Access (Wi-Max), other protocols for email, instant messaging, and short messaging, and any other suitable communication protocols, including those currently undeveloped.

[0116] The memory 520 can be used to store software programs and modules, such as the corresponding program instructions / modules in the above-described embodiments, and the processor 580 can execute various functions and data processing by running the software programs and modules stored in the memory 520, i.e., realize functions such as front camera shooting, processing of the captured image, and switching of display color of the display content on the display screen. The memory 520 can include a high-speed random access memory, and can further include a non-volatile memory such as one or more magnetic storage devices, flash memories, or other non-volatile solid-state memories. In some examples, the memory 520 can further include memories disposed remotely with respect to the processor 580, which can be connected to the electronic device 500 through a network. Examples of the network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0117] The input unit 530 can be used to receive inputted digital or character information, and generate a keyboard, a mouse, and the like related to user settings and function control. The display unit 540 can be used to display information inputted by the user or provided to the user, and various graphical user interfaces which can be constituted by graphics, text, icons, video, and any combination thereof. The display unit 540 can include a display panel 541, which can be configured in the form of an LCD (Liquid Crystal Display), an OLED (Organic Light-Emitting Diode), or the like.

[0118] The audio circuit 560, the speaker 561, and the microphone 562 can provide an audio interface between the user and the electronic device 500. The audio circuit 560 can convert received audio data into an electrical signal, transmit the electrical signal to the speaker 561, and convert the electrical signal into a sound signal output by the speaker 561; on the other hand, the microphone 562 can convert a sound signal collected into an electrical signal, and the audio circuit 560 can convert the electrical signal into audio data, output the audio data to the processor 580 for processing, and then transmit the audio data to another terminal through the RF circuit 510, or output the audio data to the memory 520 for further processing. The audio circuit 560 can further include an earphone jack to provide communication between an external earphone and the electronic device 500.

[0119] The electronic device 500 can help the user to receive requests, send information, and the like through the transmission module 570 (e.g., a Wi-Fi module), which provides the user with wireless broadband Internet access. Although the transmission module 570 is shown, it can be understood that it does not belong to the essential components of the electronic device 500, and can be omitted as needed without changing the essence of the application.

[0120] The processor 580 is the control center of the electronic device 500, which connects various parts of the entire mobile phone through various interfaces and lines, executes various functions of the electronic device 500 and processes data by running or executing software programs and / or modules stored in the memory 520 and calling data stored in the memory 520, thereby monitoring the entire electronic device. Optionally, the processor 580 can include one or more processing cores; in some embodiments, the processor 580 can integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communication. It can be understood that the above-mentioned modem processor can also not be integrated into the processor 580.

[0121] The electronic device 500 further includes a power supply 590 (such as a battery) for supplying power to various components, and in some embodiments, the power supply can be logically connected to the processor 580 through a power management system, so that the power management system can realize functions such as management of charging, discharging, and power consumption management. The power supply 590 can also include one or more direct or alternating power supplies, recharging systems, power failure detection circuits, power converters or inverters, power status indicators, and any other components.

[0122] Although not shown, the electronic device 500 also includes a camera (such as a front camera or a rear camera), a Bluetooth module, and the like, which are not described here in detail. In particular, in the present embodiment, the display unit of the electronic device is a touch screen display, and the mobile terminal further includes a memory and one or more programs, wherein one or more programs are stored in the memory and configured to be executed by one or more processors, and the one or more programs include instructions for: obtaining a time-frequency sequence and a corresponding natural language expression; wherein the video sequence includes a plurality of video frames; inputting the video frames and the natural language expression into a multi-target visual tracking model to obtain a visual tracking result; The multi-target visual tracking model includes a visual backbone network, a semantic backbone network, an early fusion module, a Transformer module, and a space-time enhancement module; the processing process of the multi-target visual tracking model includes: inputting the video frames into the visual backbone network to obtain visual features, and inputting the natural language expression into the semantic backbone network to obtain text features; inputting the visual features and the text features into the early fusion module for cross-modal fusion to obtain cross-modal features; inputting the cross-modal feature into the Transformer module for encoding and decoding to obtain an initial target detection frame and a query feature; inputting the initial target detection frame and the query feature into the spatio-temporal enhancement module for optimization to obtain an optimized target detection frame.

[0123] In practice, the above various modules can be implemented as independent entities, or can be combined as the same or several entities. The specific implementation of the above various modules can be referred to the method embodiments above, and will not be described here.

[0124] Those skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructions, or by controlling relevant hardware by the instructions, which can be stored in a computer readable storage medium and loaded and executed by a processor. Therefore, the embodiments of the present application provide a storage medium, which stores a plurality of instructions that can be loaded by a processor to execute the steps of any embodiment of the language specified multi-target visual tracking method based on spatio-temporal enhancement provided by the embodiments of the present application.

[0125] The computer readable storage medium can include a read only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0126] Since the instructions stored in the storage medium can execute the steps in any embodiment of the language specified multi-target visual tracking method based on spatio-temporal enhancement provided by the embodiments of the present application, the beneficial effects of any language specified multi-target visual tracking method based on spatio-temporal enhancement provided by the embodiments of the present application can be achieved, which are described in detail in the above embodiments and will not be described here.

[0127] The language specified multi-target visual tracking method based on spatio-temporal enhancement, system, storage medium and electronic device provided by the embodiments of the present application are described in detail above, and the principles and implementation manners of the present application are described by applying specific examples; the above embodiment descriptions are only used to help understand the method and its core idea of the present application; meanwhile, for those skilled in the art, according to the idea of the present application, the specific implementation manner and application range will be changed, and the above description should not be understood as limiting the present application.

Claims

1. A language-specified multi-target visual tracking method based on spatiotemporal enhancement, characterized in that: The method comprises: Obtain a time-frequency sequence and a corresponding natural language expression; wherein the video sequence includes a plurality of video frames; Inputting the video frame and the natural language expression into a multi-target visual tracking model to obtain a visual tracking result; The multi-target visual tracking model includes a visual backbone network, a semantic backbone network, an early fusion module, a Transformer module, and a spatiotemporal enhancement module. The processing process of the multi-target visual tracking model includes: Inputting the video frame into the visual backbone network to obtain visual features, and inputting the natural language expression into the semantic backbone network to obtain text features; Inputting the visual features and the text features into the early fusion module for cross-modal fusion to obtain cross-modal features; Inputting the cross-modal features into the Transformer module for encoding and decoding to obtain initial target detection boxes and query features; The initial target detection frame and the query feature are input into the spatiotemporal enhancement module for optimization to obtain an optimized target detection frame.

2. The language-specified multi-target visual tracking method based on spatiotemporal enhancement according to claim 1 is characterized in that: The processing of the early fusion module includes: Performing linear projection on the visual features and the text features respectively, and mapping them into a unified feature dimension space; Based on the multi-head attention mechanism, the correlation between the visual space position of the linearly projected visual features and the text semantic units of the linearly projected text features is calculated to obtain the cross-modal attention score matrix; The visual features and the text features are fused based on the attention score matrix to obtain cross-modal features.

3. The language-specified multi-target visual tracking method based on spatiotemporal enhancement according to claim 2 is characterized in that: The multi-head attention mechanism is based on calculating the correlation between the visual space position of the linearly projected visual features and the text semantic unit of the linearly projected text features to obtain a cross-modal attention score matrix, including: Flatten the visual features after linear projection and transform them into sequence form ,in, Represents the number of spatial locations; the text features are maintained as , each position corresponds to the representation of a text token; Visual features As query, text features As key and value, the cross-modal attention score matrix is ​​obtained through the standard attention calculation formula : in, is a learnable linear transformation matrix, Set to , is the number of attention heads, Indicates the The text token and The relative matching degree between the visual positions.

4. The language-specified multi-target visual tracking method based on spatiotemporal enhancement according to claim 2 is characterized in that: The fusing of the visual features and the text features based on the attention score matrix to obtain cross-modal features includes: The text features are weighted based on the cross-modal attention score matrix to generate text representations that correspond one-to-one to visual space positions. , the calculation formula is: in, is the value projection matrix, is the text feature, is the weighted text feature; The weighted text features are added to the visual features element by element to obtain the fused cross-modal feature representation : 。 5. The language-specified multi-target visual tracking method based on spatiotemporal enhancement according to claim 1 is characterized in that: The Transformer module includes an encoder, a decoder, and a multi-layer perceptron. The processing of the Transformer module includes: In the encoder, the cross-modal features are processed based on a multi-head attention mechanism to obtain context-enhanced encoder features; In the decoder, query features are generated based on a query-decoder mechanism, and the query features are updated; In the multi-layer perceptron, each query feature is detected to obtain an initial target box, a predicted category, and a probability distribution of the predicted category.

6. The language-specified multi-target visual tracking method based on spatiotemporal enhancement according to claim 5 is characterized in that: The query-decoder mechanism is used to generate query features and update query features, including: Initialize a set of learnable query features ,in, is the predefined number of queries; Based on the self-attention mechanism, the query feature is modeled internally, and the formula is expressed as follows: in, Represents the self-attention mechanism; Based on the cross attention mechanism, the query feature retrieves the most relevant region information from the visual features; Perform a linear transformation on the query features and encoder features to obtain the query, key, and value required for cross-attention: in, is the learnable parameter matrix, 、 、 Query, key and value respectively; Perform cross-attention calculation, the calculation formula is: in, The operation is normalized along the dimension of Key. for cross-attentional results; Based on the cross-attention results, update the query features: in, The updated features.

7. The language-specified multi-target visual tracking method based on spatiotemporal enhancement according to claim 6 is characterized in that: The spatiotemporal enhancement module includes an object decoder and a temporal decoder; The processing process of the spatiotemporal enhancement module includes: In the temporal decoder, a historical detection query set is constructed based on the updated query features and stored in the historical memory. The preliminary query representation of the current frame and the query features in the historical memory are interacted through a cross-attention mechanism to obtain the query features enhanced by temporal sequence. The query features and the initial target detection frame after temporal enhancement are input into the object decoder to obtain an optimized target detection frame.

8. The language-specified multi-target visual tracking method based on spatiotemporal enhancement according to claim 7 is characterized in that: The object decoder includes a detection head network, and the query features after temporal enhancement and the initial object detection box are input into the object decoder to obtain an optimized object detection box, including: The internal interaction modeling of the time-enhanced query features is performed through the multi-head self-attention mechanism to obtain the updated time-enhanced query features; Inputting the updated temporal enhanced query features into the detection head network to obtain predicted object bounding box parameters and category confidence; The initial target detection frame is optimized based on the predicted target bounding box parameters to obtain an optimized target detection frame.

9. A language-specified multi-target visual tracking system based on spatiotemporal enhancement, characterized in that: include: An acquisition module is configured to acquire a time-frequency sequence and a corresponding natural language expression; wherein the video sequence includes a plurality of video frames; A multi-target visual tracking module, configured to input the video frame and the natural language expression into a multi-target visual tracking model to obtain a visual tracking result; The multi-target visual tracking model includes a visual backbone network, a semantic backbone network, an early fusion module, a Transformer module and a spatiotemporal enhancement module; The processing process of the multi-target visual tracking model includes: Inputting the video frame into the visual backbone network to obtain visual features, and inputting the natural language expression into the semantic backbone network to obtain text features; Inputting the visual features and the text features into the early fusion module for cross-modal fusion to obtain cross-modal features; Inputting the cross-modal features into the Transformer module for encoding and decoding to obtain initial target detection boxes and query features; The initial target detection frame and the query feature are input into the spatiotemporal enhancement module for optimization to obtain an optimized target detection frame.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a plurality of instructions, which are suitable for being loaded by a processor to execute the language-specified multi-target visual tracking method based on spatiotemporal enhancement according to any one of claims 1 to 8.

Images