Multi-target referring tracking method based on cross-modal fusion and collaborative query matching

Abstract

The invention discloses a referring multi-target tracking method based on cross-modal fusion and collaborative query matching, and relates to a computer vision technology. Respectively extracting visual features and language features from the video sequence and the language description of the training data set, and constructing a target query formed by splicing a detection query and a tracking query; performing cross-modal fusion on the visual features, the language features and the target query through a triple fusion module to generate multi-modal features, and inputting the multi-modal features into a decoder after residual connection and encoder optimization; the decoder is combined with a collaborative query matching mechanism to realize efficient matching of target query, a new target and a tracked target; and outputting a category, a bounding box and a reference score of the target through a prediction head module, predicting a target trajectory and calculating loss to train the model. The cross-modal feature consistency is enhanced through a triple fusion module, the detection query training efficiency is improved by means of a collaborative query matching mechanism, a target corresponding to language description is accurately tracked in a complex scene, and good adaptability and tracking precision are achieved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, and particularly relates to a referring multi-object tracking method based on cross-modal fusion and collaborative query matching. BACKGROUND

[0002] Multi-object tracking, as a fundamental and important task in the field of computer vision, has gradually become a research hotspot in recent years. The core goal is to detect the positions of multiple objects in a video or image sequence and assign a unique identifier to each object, thereby correctly associating the same object in the time dimension and continuously tracking its motion trajectory. Thanks to the rapid development of deep neural networks, researchers have proposed a series of multi-object tracking methods, which have achieved excellent performance even in the presence of interference factors such as frequent occlusion, appearance change and irregular motion. However, traditional multi-object tracking methods based on single modal vision have certain application limitations: they usually track all pre-defined class targets indiscriminately, lacking the ability to focus on specific semantic targets. This defect is particularly significant in scenarios that require selective tracking based on semantic attributes.

[0003] With the rapid development of visual-linguistic models, referring multi-object tracking has emerged. This task introduces natural language understanding technology based on traditional multi-object tracking, enabling the tracker to locate and continuously track semantic-consistent targets according to given language descriptions. Unlike traditional multi-object tracking, referring multi-object tracking takes video sequences and language descriptions as input, requiring not only target detection and trajectory association, but also establishing semantic alignment relationships between visual and linguistic information. This feature gives it greater flexibility, allowing the model to better adapt to diverse task requirements. However, the cross-modal nature of referring multi-object tracking also presents new technical challenges: first, existing methods perform poorly in visual-linguistic feature fusion, making it difficult to achieve effective cross-modal information interaction; second, as a visual-linguistic multi-modal task, there is an internal sub-task optimization conflict; finally, this task requires simultaneous processing of target detection, trajectory association and multi-modal alignment, further increasing the complexity of algorithm design. SUMMARY

[0004] The present application aims to solve the problems of insufficient cross-modal feature fusion and sub-task optimization conflict in existing referring multi-object tracking, and provides a referring multi-object tracking method based on cross-modal fusion and collaborative query matching, which aims to promote cross-modal fusion of visual and language features and alleviate the optimization conflict between sub-tasks. The method proposes a triple fusion module, which guides cross-modal fusion through target query, maps target query, visual and language features to a shared semantic space, effectively alleviates the domain difference between the three, thereby enhancing the consistency of cross-modal features and building more accurate semantic correspondence. In addition, a collaborative query matching mechanism is designed, which allows the detection query to match both new-born targets and tracked targets, effectively expanding the number of training samples and improving the training efficiency of the detection query and the detection ability of the model for new-born targets. Therefore, the method can accurately track the target corresponding to the language description and has good adaptability to complex scenes.

[0005] A referring multi-object tracking method based on cross-modal fusion and collaborative query matching comprises the following steps:

[0006] A. Given a training data set containing video sequences and object language descriptions;

[0007] B. Input the video sequence in step A into the visual feature extractor to extract the visual features; input the language description into the language feature extractor to extract the language features; and at the same time, splice the detection query and the tracking query as the target query;

[0008] C. Input the visual features and language features extracted in step B and the target query into the triple fusion module to obtain multi-modal features;

[0009] D. Add the multi-modal features obtained in step C and the visual features extracted in step B through residual connection, and then input the encoder to further extract and optimize the multi-modal feature representation;

[0010] E. Input the multi-modal features output by the encoder in step D and the target query into the decoder, interact through the multi-head attention mechanism, and realize the label matching of the target query in combination with the collaborative query matching mechanism;

[0011] F. Input the visual-language multi-modal feature information output by the decoder into the prediction head module, predict the target trajectory according to the predicted category, bounding box and reference score, and generate loss to train the model.

[0012] In step A, the given training data set, specifically a given one referring multi-object tracking data set, contains several videos, each of which is composed of video frames; each video provides location annotations and language annotations for the target of interest; the location of the target is represented by a bounding box, and the bounding box annotation is provided in each video frame from the time the target enters the scene to the time it exits the scene; the language annotation is represented by a natural language description, which contains the appearance characteristics and state information of the target.

[0013] In step B, the extraction of visual features is to input the video sequence in step A into a visual feature extractor, and use the convolutional neural network ResNet-50 to extract the pyramid feature map of the first video frame , where , and represent the channel number, height and width of the first layer feature map respectively; in order to reduce the computational complexity and unify the feature dimension, a 1x1 convolution operation is used to reduce the channel number of the feature map to 256 and flatten it into a two-dimensional tensor , where d=256 is the dimension of the feature, and the last three layers are selected as the visual feature .

[0014] The extraction of language features is to input the language description in step A into a language feature extractor, use the pre-trained language model RoBERTa to extract language features, and map them to the same dimension as the visual features through a fully connected layer to obtain the language features , where is the length of the language feature;

[0015] The target query is represented as , which is composed of detection queries and tracking queries; where, represents the total number of target queries.

[0016] In step C, the visual features and language features extracted in step B and the target query are input into a triple fusion module, and are fused through a triple multi-head attention layer to obtain a multi-modal feature representation with rich semantic information; the triple multi-head attention layer aims to enable the visual features to simultaneously focus on the target query, language features and their own information during the updating process, so as to realize triangular sampling between features and enhance the consistency between different modalities in the semantic space; its process is defined as follows:

[0017]

[0018] where, denotes a triple multi-head attention layer, denotes the generated multi-modal feature; in a specific implementation, the triple multi-head attention layer concatenates the target query, the language and visual features together as a query (Query), and the language feature as a key (Key) and a value (Value); then, the weighted feature is obtained through a cross-attention mechanism ; the specific calculation formula of the process is as shown below:

[0019]

[0020]

[0021]

[0022] wherein, is a trainable parameter; in the triple attention layer, the target query dynamically controls the weighting ratio of the visual feature and the language feature by participating in the construction of the attention weight matrix, so that the model focuses more on the language information highly related to the target semantics in the cross-modal fusion process; next, the part corresponding to the visual modality is intercepted from the weighted feature to generate a multi-modal feature :

[0023]

[0024] wherein, denotes a linear transformation matrix of the output layer.

[0025] In step D, the multi-modal feature obtained in step C and the visual feature extracted in step B are added through a residual connection, and the multi-modal feature representation is further extracted and optimized by an encoder; specifically, the visual feature extracted in step B and the multi-modal feature obtained in step C are added through a residual connection to obtain a final multi-modal feature as the input of the subsequent encoder; the calculation process is represented as follows:

[0026]

[0027] wherein, denotes the final multi-modal feature, P denotes the visual feature, denotes the generated multi-modal feature.

[0028] In step E, the multi-modal feature output by the encoder in step D is input into a decoder together with the target query, and the target query is matched through a multi-head attention mechanism and a collaborative query matching mechanism; specifically, assuming that the prediction result of the detection query is and the prediction result of the tracking query is The true label of the new student's goal is The true label of the tracked target is The label assignment results for detection queries and tracking queries are respectively denoted as... and For the first video sequence The first frame One decoding layer ( The label assignment for the detection query is achieved by optimizing the following objective function:

[0029]

[0030] in, It is the set of all possible binary matches between the query and both newly appearing and already tracked targets. This is the matching cost function. It's important to note that to avoid trajectory redundancy during the inference phase, the cooperative matching mechanism is only applied to the first L-1 layers of the decoder. The original matching strategy is retained in the last layer of the decoder, meaning that only the detection query is matched with the new target. For the label assignment of the tracking query, the matching results of the tracking query and the detection query in the previous frame are merged, and the calculation formula is as follows:

[0031]

[0032] in, Indicates the first Track the matching results of the query within the frame. No. Track the matching results of the query within the frame. Indicates the first The first frame The matching results of the query are detected in each decoding layer.

[0033] In step F, the loss comprises two parts: tracking loss and detection loss, which are used to jointly supervise the learning of the model. The tracking loss is based on... , and The calculation yielded the following. The value range is from 0 to 1. This indicates that the target is identified as foreground in frame t; This indicates the center coordinates, width, and height of the predicted bounding box; This represents the probability of association between the target instance and the linguistic description. The tracking loss function is defined as follows:

[0034]

[0035] in, , Indicates L1 loss, denotes the generalized intersection over union loss. and focal loss is adopted. , , and are the corresponding weight coefficients, respectively.In order to enhance the target positioning ability of the model, a detection loss is introduced in the design.Specifically, the detection query is used to generate a detection embedding to predict the bounding box and target classification score.Then the detected candidate target is matched with the real new target, and the loss is calculated in a similar way:

[0036]

[0037] For the frame video, the total loss is the sum of the tracking loss and the detection loss of each frame.Because there is no tracked target in the first frame, the tracking query is empty, and only the detection query is used to identify the new target.The total loss is defined as follows:

[0038]

[0039] Under the supervision of the total loss function, the model optimizes the class and positioning accuracy of target detection, the consistency of visual-linguistic semantic alignment, and improves the overall performance of the referential multi-target tracking.

[0040] Compared with the prior art, the present application has the following advantages:

[0041] 1. The present application proposes a triple fusion module, which maps the target query, visual and language features to a shared semantic space guided by the target query.Different from the shortcomings of traditional methods in visual-linguistic feature fusion, it effectively alleviates the domain difference among the three, enhances the consistency of cross-modal features, deeply excavates the complementarity of target query, visual and language modalities, makes the semantic alignment more accurate, and helps the model to locate the target more accurately according to the language description.

[0042] 2. The present application designs a cooperative query matching mechanism, which breaks the limitation of the detection query only matching the new target in the traditional label assignment.The detection query is allowed to match the new and tracked targets at the same time in the middle layer of the decoder, which expands the training samples, improves the training efficiency, and strengthens the model's detection ability for new targets;By reasonably setting the matching strategy of different layers of the decoder, the inference trajectory redundancy is avoided, the conflict between sub-tasks is optimized, and the target detection and trajectory association are more cooperative.

[0043] ​3、Experiments show that, compared with mainstream methods such as FairMOT and ByteTrack, the method has advantages in multiple key indicators such as HOTA (comprehensive detection and association performance), DetA (detection accuracy), AssA (association accuracy), LocA (positioning accuracy), and is more accurate and robust in semantic target tracking in complex scenes, and has more outstanding adaptability to diversified task requirements. BRIEF DESCRIPTION OF DRAWINGS

[0044] Figure 1 The overall flowchart of the embodiment of the application.

[0045] Figure 2 The schematic diagram of the collaborative query matching mechanism. DETAILED DESCRIPTION

[0046] The method of the application will be described in detail below in combination with the drawings and embodiments. The embodiments are implemented on the premise of the technical solution of the application, and implementation modes and specific operation processes are given, but the protection scope of the application is not limited to the following embodiments.

[0047] As shown in Figure 1 , the embodiment of the application provides a referential multi-target tracking method based on cross-modal fusion and collaborative query matching, which includes the following steps:

[0048] A. Given a referential multi-target tracking dataset, containing a plurality of videos, each video is composed of a plurality of video frames. Each video provides location annotation and language annotation for the target of interest. The location of the target is represented by a bounding box, and the bounding box annotation is provided in each video frame from the time the target enters the scene to the time it exits the scene. The language annotation is represented by a natural language description, which contains the appearance features and state information of the target.

[0049] B. Input the video sequence in step A into a visual feature extractor, and use a convolutional neural network ResNet-50 to extract the pyramid feature map of the first video frame , wherein , and represent the channel number, height and width of the first layer feature map respectively. In order to reduce the computational complexity and unify the feature dimension, a 1x1 convolution operation is used to reduce the channel number of the feature map to 256, and it is flattened into a two-dimensional tensor , wherein d=256 is the dimension of the feature, and the last three layers are selected as the visual feature Meanwhile, the language description in step A is input into the language feature extractor, the language features are extracted by using the pre-trained language model RoBERTa, and are mapped to the same dimension as the visual features by a fully connected layer to obtain the language features wherein is the length of the language features. The target query is represented as , which is composed of detection queries and tracking queries. Wherein, represents the total number of target queries.

[0050] C. The visual features, language features and target queries extracted in step B are input into the triple fusion module, and are fused by a triple multi-head attention layer to obtain a multi-modal feature representation with rich semantic information. The triple multi-head attention layer aims to enable the visual features to simultaneously focus on the target query, language features and their own information in the updating process, so as to realize triangular sampling between the features and enhance the consistency between different modalities in the semantic space. The process is defined as follows:

[0051]

[0052] wherein, represents the triple multi-head attention layer, represents the generated multi-modal feature. In specific implementation, the triple multi-head attention layer concatenates the target query, language and visual features together as a query (Query), and takes the language features as a key (Key) and a value (Value). Then, the weighted features are obtained by cross-attention mechanism. The specific calculation formula of the process is as follows:

[0053]

[0054]

[0055]

[0056] wherein, is a trainable parameter. In this triple attention layer, the target query dynamically controls the weighting ratio of the visual features and the language features by participating in the construction of the attention weight matrix, so that the model focuses more on the language information highly related to the target semantics in the cross-modal fusion process. Next, the part corresponding to the visual modality is intercepted from the weighted features to generate the multi-modal feature :

[0057]

[0058] wherein, A linear transformation matrix representing an output layer.

[0059] D. The visual features extracted in step A are added to the multi-modal features obtained in step C through a residual connection to obtain the final multi-modal features as the input of the subsequent encoder. The calculation process is represented as follows:

[0060]

[0061] E. The multi-modal features output by the encoder in step D are input into the decoder together with the target query to interact through a multi-head attention mechanism, and the label matching of the target query is realized by combining the collaborative query matching mechanism. The schematic diagram of the collaborative query matching mechanism is shown in Figure 2

[0062] Specifically, assuming that the prediction result of the detection query is , the prediction result of the tracking query is , the real label of the new target is , and the real label of the tracked target is . The label assignment results of the detection query and the tracking query are denoted as and respectively. For the th decoding layer of the th frame of the video sequence (L ), the label assignment of the detection query is realized by optimizing the following objective function:

[0063]

[0064] wherein is the set of all possible bipartite matches between the detection query and the newly appearing target and the tracked target, is the matching cost function. It should be noted that, in order to avoid trajectory redundancy in the inference stage, the collaborative matching mechanism is only applied to the first L-1 layers of the decoder. While in the last layer of the decoder, the original matching strategy is still retained, i.e. only the detection query is matched with the newly appearing target. For the label assignment of the tracking query, the matching results of the tracking query in the previous frame and the matching results of the detection query are combined, and the calculation formula is as follows:

[0065]

[0066] wherein represents the matching result of the tracking query in the th frame, the matching result of the tracking query in the th frame, represents the th decoding layer of the ​​The matching result of the detection query in the decoding layer is detected.

[0067] F. The visual-language multimodal feature information output by the decoder is sent into a prediction head module, the target trajectory is predicted according to the predicted category, bounding box and reference score, and a loss is generated to train the model.

[0068] The main innovations of the present application are: a. Cross-modal feature fusion is performed through a triple fusion module; b. Label matching of the target query is realized based on a collaborative query matching mechanism.

[0069] In order to alleviate the domain difference problem between the visual and language modalities, and fully play the guiding role of the rich information contained in the target query in cross-modal fusion, the present application proposes a triple fusion module. The module introduces a triple multi-head attention layer, uses the target query to guide the cross-modal fusion between visual and language features, and effectively coordinates the semantic space distribution of the three types of features of the target query, language and vision. Through this cross-modal fusion mode, the complementarity between the target query, visual and language modalities can be fully tapped, and the alignment consistency of different modalities in the semantic space is enhanced, so that the network can more accurately locate the target according to the language description. At the same time, in order to improve the problem of insufficient training of the detection query in the existing label assignment mechanism, a collaborative query matching mechanism is designed. The mechanism introduces an auxiliary branch in the middle layer of the decoder, so that the detection query can match both new targets and tracked targets. Then, the loss calculated by the auxiliary branch and the output loss of the decoder jointly supervise the model training, thereby optimizing the training process. Through this strategy, the number of training samples of the detection query is effectively expanded, and its training frequency is significantly improved, thereby enhancing the detection ability of the model for new targets and improving the overall training efficiency and tracking performance.

[0070] In order to verify the effectiveness of the referential multi-target tracking method based on cross-modal fusion and collaborative query matching proposed by the present application, the present application carries out comparative experiments on the Refer-KITTI dataset and the Refer-KITTI-v2 dataset, compares the performance of the present method (TriCo-RMOT) with the current mainstream referential multi-target tracking methods (such as FairMOT, ByteTrack, iKUN, TransRMOT, DeepRMOT, etc.), and the comparison indicators cover HOTA, DetA, AssA, DetRe, DetPr, AssRe, AssPr, LocA, and other commonly used evaluation indicators of multi-target tracking. Each indicator reflects the detection, association and positioning performance of the tracking method from different dimensions. The comparative experiment results on the Refer-KITTI dataset are shown in Table 1, and the comparative experiment results on the Refer-KITTI-v2 dataset are shown in Table 2.

[0071] Table 1

[0072]

[0073] Table 2

[0074]

[0075] On the Refer-KITTI dataset, the HOTA index of TriCo-RMOT (the present method) is 49.08, which is higher than that of FairMOT (22.78), DeepSORT (25.59), ByteTrack (24.95), CSTrack (27.91), TransTrack (32.77), TrackFormer (33.26), iKUN (48.84), TransRMOT (45.21), MGLT (47.95), MLS-Track (49.05), DeepRMOT (48.46) and other methods, which again verifies the advantage of the present method in comprehensive tracking performance and better cope with the tracking task under the scene of the dataset. DetA (detection accuracy) is 40.21, and AssA (association accuracy) is 60.07. DetA performs outstandingly among the compared methods, ensuring the accuracy of target detection; AssA is also at a high value, indicating that the association accuracy of the present method is good, and the same target track can be effectively associated. DetRe (detection recall rate) is 55.54, and DetPr (detection accuracy) is 58.17. The performance is balanced in terms of target detection recall and accuracy, which can effectively detect targets and reduce the false detection rate, laying a good foundation for subsequent tracking tasks. The AssRe of the present invention is 63.42, and the AssPr is 90.99. Both the association recall and accuracy perform excellently, indicating that in the track association, the same target in different frames can be associated as much as possible (high AssRe), and the high accuracy of the association result can be ensured (high AssPr), which can stably track the target motion track. The LocA (positioning accuracy) of the present method is 91.48, which is the highest value among the compared methods, reflecting the accurate positioning ability of the target position, which can accurately obtain the position information such as the target bounding box, and help precise tracking.

[0076] On the Refer-KITTI-v2 dataset, TriCo-RMOT (the present method) also exhibits excellent performance: the HOTA value of TriCo-RMOT reaches 32.09, higher than that of FairMOT (22.53), ByteTrack (24.59), iKUN (10.32), TransRMOT (31.00), DeepRMOT (31.70) and other methods, indicating that the present method is better in overall tracking performance and can better balance the detection and association tasks. The DetA (detection accuracy) of TriCo-RMOT is 22.70, and the AssA (association accuracy) is 45.49. DetA is higher than most of the comparison methods, reflecting the accuracy of target detection; AssA is also at a good level among the comparison methods, indicating an advantage in the accuracy of target association and a more accurate allocation of unique identifiers for the same target. The DetRe (detection recall rate) is 32.64, and the DetPr (detection precision) is 42.00. DetRe reflects the ability to detect real targets, and DetPr reflects the proportion of real targets in the detection results, both of which indicate that the present method can effectively capture targets in the scene and ensure the reliability of the detection results in the target detection link. The AssRe (association recall rate) reaches 48.30, and the AssPr (association precision) is 88.09. A high AssRe means that it can effectively associate different frames of the same target, and a high AssPr indicates a high proportion of correct associations in the association results, indicating that the present method performs well in the trajectory association task and can stably track targets in the time dimension. The LocA (localization accuracy) is 90.59, leading the comparison methods in numerical value, reflecting the accurate positioning ability of the present method for target positions and the ability to accurately obtain position information such as the bounding box of the target in the video frame.

[0077] Experiments show that, whether on the Refer-KITTI or Refer-KITTI-v2 dataset, the TriCo-RMOT method proposed by the present application is superior to the mainstream comparison methods in multiple-dimensional evaluation indicators. This indicates that the present method, through the triple fusion module to promote cross-modal fusion and the collaborative query matching mechanism to alleviate the design of sub-task optimization conflicts, effectively improves the performance of referential multi-object tracking and has good adaptability in complex scenes, can more accurately track the targets corresponding to the language description, and achieves better tracking accuracy. The method proposed by the present application has good adaptability in complex scenes and achieves better accuracy than mainstream referential multi-object tracking methods.

[0078] The above embodiments are only preferred embodiments of the present application and cannot be considered as limiting the scope of the present application. Any equivalent changes and improvements made within the scope of the present application should still belong to the patent coverage of the present application.

Claims

1. A method for referring multi-target tracking based on cross-modal fusion and collaborative query matching, characterized in that The method comprises the following steps: A. Given a training data set containing video sequences and language descriptions of objects; B. Input the video sequence in step A into a visual feature extractor to extract visual features; Input the language description into a language feature extractor to extract language features; At the same time, splice the detection query and the tracking query as a target query; C. Input the visual features and language features extracted in step B and the target query into a triple fusion module to obtain multi-modal features; D. Add the multi-modal features obtained in step C and the visual features extracted in step B through residual connection, and then input the multi-modal features into an encoder to further extract and optimize the multi-modal feature representation; E. Input the multi-modal features output by the encoder in step D and the target query into a decoder to interact through a multi-head attention mechanism, and realize label matching of the target query by combining a collaborative query matching mechanism; F. Input the visual-linguistic multi-modal feature information output by the decoder into a prediction head module, predict the target trajectory according to the predicted category, bounding box and reference score, and generate a loss to train the model.

2. The method of claim 1, wherein the method is based on cross-modal fusion and collaborative query matching. In step A, the given training data set, specifically a referred multi-target tracking data set, contains a plurality of videos, each video being composed of a plurality of video frames; each video provides location annotation and language annotation for the target of interest; the location of the target is represented by a bounding box, and the bounding box annotation is provided in each video frame from the time when the target enters the scene to the time when the target exits the scene; the language annotation is represented by a natural language description, which contains the appearance characteristics and state information of the target.

3. The method of claim 1, wherein the method is based on cross-modal fusion and collaborative query matching. In step B, the extracting visual features is inputting the video sequence in step A into a visual feature extractor, extracting pyramid feature maps of the first video frame by using a convolutional neural network ResNet-50 wherein , and respectively represent the number of channels, height and width of the feature map of the first layer; in order to reduce the computational complexity and unify the feature dimension, a 1x1 convolution operation is used to reduce the number of channels of the feature map to 256 and flatten it into a two-dimensional tensor , wherein d=256 is the dimension of the feature, and the last three layers are selected as the visual features .

4. The method of claim 1, wherein the method is based on cross-modal fusion and collaborative query matching. In step B, the language feature extraction is to input the language description in step A into the language feature extractor, extract the language feature by using the pre-trained language model RoBERTa, and map it to the same dimension as the visual feature through a full connection layer to obtain the language feature wherein is the length of the language feature.

5. The method of claim 1, wherein In step B, the target query is represented as , which is composed of detection queries and tracking queries; wherein, represents the total number of target queries.

6. The method of claim 1, wherein the method is based on cross-modal fusion and collaborative query matching. In step C, the visual features and language features extracted in step B and the target query are input into a triple fusion module to obtain multi-modal feature representation with rich semantic information through a triple multi-head attention layer; the triple multi-head attention layer aims to enable the visual features to simultaneously focus on the target query, language features and their own information during the updating process, so as to realize triangular sampling among the features and enhance the consistency between different modalities in the semantic space; the process is defined as follows: wherein, denotes a triple multi-head attention layer, denotes generated multi-modal features; in a specific implementation, the triple multi-head attention layer concatenates the target query, language and visual features together as a query, and takes the language features as keys and values; weighted features are obtained through a cross-attention mechanism ; the specific calculation formula of the process is as follows: wherein, are trainable parameters; in the triple attention layer, the target query dynamically regulates the weighting ratio of the visual features and the language features by participating in the construction of the attention weight matrix, so that the model focuses more on the language information highly related to the target semantics in the cross-modal fusion process; the weighted features are intercepted from the part corresponding to the visual modality to generate multi-modal features : wherein denotes a linear transformation matrix of the output layer.

7. The method of claim 1, wherein the method is based on cross-modal fusion and collaborative query matching. The multi-modal features obtained in step C and the visual features extracted in step B are added through residual connection, and then input into an encoder to further extract and optimize the multi-modal feature representation, and the calculation process is represented as follows: wherein, represents the final multimodal feature, P represents the visual feature, represents the generated multimodal feature.

8. The method of claim 1, wherein In step E, the multi-modal features output by the encoder in step D are input into the decoder together with the target query, and the multi-head attention mechanism is used to interact with the target query, and a cooperative query matching mechanism is used to realize the label matching of the target query; specifically, assuming that the prediction result of the detection query is , the prediction result of the tracking query is , the real label of the new target is , and the real label of the tracked target is ; The label assignment results for the detection query and the tracking query are denoted as follows: and For the first video sequence The first frame One decoding layer ( The label assignment for the detection query is achieved by optimizing the following objective function: wherein, is to detect all possible bipartite matching sets between queries and newly appeared targets and tracked targets, is a matching cost function; it is noted that, in order to avoid trajectory redundancy in the reasoning phase, the collaborative matching mechanism is only applied to the first L-1 layers of the decoder; while the original matching strategy is still retained in the last layer of the decoder, i.e. only matching the detection queries with the newly appeared targets; for the label assignment of the tracking queries, the matching results of the detection queries in the previous frame are combined with the matching results of the tracking queries, and the calculation formula is as follows: wherein, indicates the matching results of the detection query in the matching results of the detection query in the matching results of the detection query in the indicates the matching results of the detection query in the matching results of the detection query in the 9. The method of claim 1, wherein In step F, the loss includes two parts of tracking loss and detection loss for jointly supervising the model; the tracking loss is based on , and calculated; wherein the value range is 0 to 1, indicates that the target is identified as foreground in the t-th frame; indicates the center coordinates, width and height of the predicted bounding box; indicates the association probability between the target instance and the language description; the tracking loss function is defined as follows: wherein, , represents the L1 loss, represents the generalized intersection over union loss; and focal loss is adopted; , , and are corresponding weight coefficients; in order to enhance the target positioning ability of the model, a detection loss is introduced in the design; specifically, a detection query is used to generate a detection embedding to predict a bounding box and a target classification score; subsequently, the candidate target obtained by detection is matched with the real new target, and the loss is calculated in a similar manner: For For frame video, the total loss is the sum of tracking loss and detection loss for each frame; since there is no tracked target for the first frame, the tracking query is empty set, and only the detection query is used to identify new-born targets; the total loss is defined as follows: wherein, is a tracking loss; is a detection loss.

Images