Memory prompt-based multi-modal visual single target tracking method, device, equipment and medium

Abstract

The application provides a memory prompt-based multi-modal visual single-target tracking method and device, equipment and medium, and relates to the field of target tracking. The method comprises the following steps: encoding a template image frame and a current search image frame to obtain a current image feature; encoding a template multi-modal frame corresponding to the template image frame and a current search multi-modal frame corresponding to the current search image frame to obtain a current multi-modal feature; processing the current image feature and the current multi-modal feature to obtain an initial template feature and a current search region feature; obtaining a current template feature according to the initial template feature and a memory feature corresponding to the current search image frame; and obtaining a target tracking result of the current search image frame based on the current template feature and the current search region feature, so as to perceive long-term appearance dynamics, alleviate potential interference from a single memory, and effectively improve the generalization of a multi-modal tracker in a complex scene.

Description

Technical Field

[0001] This invention relates to the field of target tracking, and more particularly to a multimodal visual single-target tracking method, apparatus, device, and medium based on memory cues. Background Technology

[0002] Visual object tracking (VOT) is a fundamental computer vision task widely used in numerous applications such as autonomous driving, video surveillance, and robotics. In recent years, multimodal visual tracking has attracted significant research attention, with multimodal trackers gaining prominence due to their robust tracking capabilities in complex scenes. Additional visual modalities, such as thermal infrared, event, and depth sensors, provide complementary information to RGB images and effectively compensate for the limitations of RGB-based trackers through specialized feature fusion strategies.

[0003] However, current multimodal trackers (such as the current RGB+X tracker, where X includes infrared (T), event (E), and depth (D)) often rely solely on spatial matching between the initial target template and the search area, or only integrate historical information from a single frame. As the target deforms, scales change, or partially occludes over time, the difficulty of target tracking increases, and the target appearance information provided by the initial template may not be able to meet the needs of subsequent tracking.

[0004] Furthermore, most current trackers that incorporate temporal information either require retraining the entire network or re-extracting image features, which incurs additional computational resources and significant computational costs. This is because most current trackers, in order to incorporate temporal information, typically require designing an additional temporal information extraction network or re-extracting image features from online target templates. These designs necessitate retraining the entire tracking network, resulting in additional training computational burden and reduced inference speed. Summary of the Invention

[0005] In view of the above-mentioned technical problems, the present invention provides a multimodal visual single-target tracking method, apparatus, device and medium based on memory cues, which aims to overcome the above problems or at least partially solve the above problems.

[0006] The first aspect of this invention provides a multimodal visual single-target tracking method based on memory cues, the method comprising:

[0007] The template image frame and the current search image frame are encoded to obtain the current image features; and the template multimodal frame corresponding to the template image frame and the current search multimodal frame corresponding to the current search image frame are encoded to obtain the current multimodal features.

[0008] The current image features and the current multimodal features are processed to obtain the initial template features and the current search region features;

[0009] The current template features are obtained based on the initial template features and the memory features corresponding to the current search image frame; the memory features corresponding to the current search image frame include: the memory features of multiple historical search image frames, and the memory features of a historical search image frame are obtained based on the historical search region features of the historical search image frame and the target tracking results of the historical search image frame.

[0010] Based on the current template features and the current search region features, the target tracking result of the current search image frame is obtained.

[0011] A second aspect of the present invention provides a multimodal visual single-target tracking device based on memory cues, the device comprising:

[0012] The encoding module is used to encode the template image frame and the current search image frame to obtain the current image features; and to encode the template multimodal frame corresponding to the template image frame and the current search multimodal frame corresponding to the current search image frame to obtain the current multimodal features.

[0013] The first processing module is used to process the current image features and the current multimodal features to obtain initial template features and current search region features;

[0014] The second processing module is used to obtain the current template features based on the initial template features and the memory features corresponding to the current search image frame; the memory features corresponding to the current search image frame include: the memory features of multiple historical search image frames, and the memory features of a historical search image frame are obtained based on the historical search region features of the historical search image frame and the target tracking results of the historical search image frame;

[0015] The tracking module is used to obtain the target tracking result of the current search image frame based on the current template features and the current search region features.

[0016] A third aspect of the present invention provides an electronic device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the memory-based multimodal visual single-target tracking method as described in the first aspect of the present invention.

[0017] A fourth aspect of the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the memory-cue-based multimodal visual single-target tracking method as described in the first aspect of the present invention.

[0018] In the multimodal visual single-target tracking method based on memory cues proposed in this invention, firstly, the template image frame and the current search image frame in the multimodal video tracking sequence are encoded to obtain the current image features; and secondly, the template multimodal frame corresponding to the template image frame and the current search multimodal frame corresponding to the current search image frame are encoded to obtain the current multimodal features; thirdly, the current image features and the current multimodal features are processed to obtain the initial template features and the current search region features; fourthly, the current template features are obtained based on the initial template features and the memory features corresponding to the current search image frame; and finally, the target tracking result of the current search image frame is obtained based on the current template features and the current search region features, thereby completing the target tracking of each current search image frame in the video tracking sequence. Thus, this invention fully considers the temporal connections of the tracking sequence and directly incorporates previously extracted search region features as cues into the temporal information, avoiding secondary extraction of temporal information. This invention directly reuses pre-extracted historical target features (obtained based on historical search region features of historical search image frames and target tracking results of those historical search image frames) as memory cues. Temporal dependencies can be established without redundant feature extraction or specially designed temporal feature extraction networks. This effectively combines initial template features with multiple historical memory cues to generate enhanced current template features, thereby perceiving long-term appearance dynamics and mitigating potential interference from individual memories. This effectively improves the generalization of multimodal trackers in complex scenarios. Attached Figure Description

[0019] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a flowchart illustrating the steps of a multimodal visual single-target tracking method based on memory cues, as shown in an embodiment of the present invention.

[0021] Figure 2 This is a schematic diagram illustrating a memory feature generation method according to an embodiment of the present invention;

[0022] Figure 3This is a schematic diagram of a memory prompt fusion module and a coding layer with an inserted memory adapter, as shown in an embodiment of the present invention.

[0023] Figure 4 This is a schematic diagram illustrating the framework of a multimodal visual single-target tracking method based on memory cues, according to an embodiment of the present invention.

[0024] Figure 5 This is a structural block diagram of a multimodal visual single-target tracking device based on memory cues provided in an embodiment of the present invention;

[0025] Figure 6 This is a schematic diagram of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0026] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0027] Currently, multimodal trackers have attracted much attention due to their robust tracking capabilities in complex scenes. However, many multimodal trackers rely solely on spatial matching between the initial target template and the search region, or only integrate historical information from a single frame, failing to fully utilize the temporal correlations within the tracking sequence. Furthermore, most trackers incorporating temporal modeling either require retraining the entire network or designing dedicated temporal feature extraction modules, resulting in significant computational costs. In other words, current RGB+X multimodal fusion trackers suffer from insufficient utilization of temporal information, and some single-modal trackers require specialized temporal information extraction networks, adding additional computational burdens and increasing training or inference costs.

[0028] Based on this, in order to at least partially solve one or more of the above-mentioned problems and other potential problems, the present invention considers that when humans track targets visually, they unconsciously remember the shape changes of the target's appearance to achieve continuous tracking without needing to re-observe the target's appearance. Therefore, embodiments of the present invention propose a multimodal visual single-target tracking method based on memory cues. When designing the tracker, the appearance features obtained in the historical prediction process are additionally retained as memory cues and participate in the subsequent tracking process. There is no need to extract features a second time based on the prediction results. The aim is to build a temporal connection for the RGB+X tracker based on visual completion cues through memory cues, so as to improve the generalization of the multimodal tracker in complex tracking scenarios.

[0029] Please refer to Figure 1 , Figure 1 This is a flowchart illustrating the steps of a multimodal visual single-object tracking method based on memory cues, as shown in an embodiment of the present invention. Figure 1 As shown, the multimodal visual single-target tracking method based on memory cues provided in this embodiment includes at least the following steps:

[0030] Step S11: Encode the template image frame and the current search image frame to obtain the current image features; and encode the template multimodal frame corresponding to the template image frame and the current search multimodal frame corresponding to the current search image frame to obtain the current multimodal features.

[0031] This embodiment can perform single-target tracking based on a multimodal video tracking sequence. The multimodal video tracking sequence includes: an image tracking sequence (i.e., an RGB image tracking sequence) and a corresponding multimodal tracking sequence. In this embodiment, the multimodality includes at least: infrared (T) mode, event (E) mode, and depth (D) mode; that is, the multimodal tracking sequence can include: an infrared tracking sequence, an event tracking sequence, and a depth tracking sequence. The image tracking sequence includes: a template image frame and multiple search image frames. The template image frame is the first image frame in the image tracking sequence where the target appears, and the search image frames are the image frames following the template image frame in the image tracking sequence. Similarly, the multimodal tracking sequence corresponding to the image tracking sequence includes: a template multimodal frame and multiple search multimodal frames. The template multimodal frame is the first multimodal frame in the multimodal tracking sequence where the target appears, and it is the multimodal frame corresponding to the template image frame; the search multimodal frames are the multimodal frames following the template multimodal frame in the multimodal tracking sequence, and they are the multimodal frames corresponding to the search image frames.

[0032] For the current search image frame (which can be any search image frame in the image tracking sequence), the template image frame and the current search image frame can be encoded to obtain the current image features. In addition, the template multimodal frame corresponding to the template image frame and the current search multimodal frame corresponding to the current search image frame can be encoded to obtain the current multimodal features.

[0033] In one optional example, the template image frame and the current search image frame are input into the embedding layer for encoding to obtain image tokens (i.e., RGB tokens or image tokens) as current image features; and the template multimodal frame corresponding to the template image frame and the current search multimodal frame corresponding to the current search image frame are input into the embedding layer for encoding to obtain multimodal tokens (i.e., T / E / D tokens or multimodal tokens) as current multimodal features.

[0034] Step S12: Process the current image features and the current multimodal features to obtain the initial template features and the current search region features.

[0035] In this embodiment, after obtaining the current image features and the current multimodal features, the current image features and the current multimodal features can be processed to obtain initial template features and current search region features. The initial template features are fused from the current image features and the current multimodal features to obtain features related to the template region; the current search region features are the search region features corresponding to the current search image frame, and the current search region features are fused from the current image features and the current multimodal features to obtain features related to the search region.

[0036] Step S13: Obtain the current template features based on the initial template features and the memory features corresponding to the current search image frame.

[0037] In this embodiment, the template features can be updated based on the initial template features and the memory features corresponding to the current search image frame to obtain current template features that reflect the current (latest) state of the target and contain information on changes in the target's appearance. This also avoids completely replacing the target's initial appearance, which contains reliable information. The memory features corresponding to the current search image frame include: the memory features of multiple historical search image frames. A historical search image frame is a search image frame in the image tracking sequence that precedes the current search image frame. The memory features of a historical search image frame are obtained based on its historical search region features and its target tracking result. The historical search region features are the search region features corresponding to the historical search image frame, and the target tracking result of the historical search image frame is the target tracking result obtained by performing target detection on the historical search image frame.

[0038] Step S14: Based on the current template features and the current search region features, obtain the target tracking result of the current search image frame.

[0039] In this embodiment, after updating the current template features, the new template (i.e., the current template features) is used in the subsequent feature extraction and association modeling process: target tracking processing is performed on the current template features and the current search region features to obtain the target tracking result of the current search image frame, and then the target tracking result of each current search image frame is obtained, thus completing single target tracking for the multimodal video tracking sequence.

[0040] In this embodiment, pre-extracted historical target features (obtained based on historical search region features of historical search image frames and target tracking results of those historical search image frames) are directly reused as memory cues. Temporal dependencies can be established without redundant feature extraction or specially designed temporal feature extraction networks. This effectively combines initial template features with multiple historical memory cues to generate enhanced current template features, thereby perceiving long-term appearance dynamics and mitigating potential interference from individual memories. This effectively improves the generalization of multimodal trackers in complex scenarios.

[0041] In conjunction with the above embodiments, in one implementation, the present invention also provides a multimodal visual single-target tracking method based on memory cues, wherein the method may further include steps S21 to S23 before step S12:

[0042] Step S21: Input the current image features and the current multimodal features into the first modal completion prompter to obtain the first modal completion prompt, and input the first modal completion prompt into the second modal completion prompter to obtain the second modal completion prompt.

[0043] This embodiment proposes a multimodal visual single-target tracker based on memory cues. The tracker includes at least a backbone network that performs single-stream, single-stage feature extraction and association modeling based on initial template features and the current search image frame for target tracking. The backbone network comprises multiple coding layers, each including at least: a regularization layer (LN), a multi-head self-attention layer (MSA), a feedforward layer (FFN), and residual connections. In an optional example, the backbone network can be a ViT backbone network, and the coding layers within the ViT backbone network can be Transformer EncoderLayer.

[0044] In this embodiment, a modal completion cue (such as a pre-trained modal completion cue) is deployed in the backbone network to fuse multimodal information using multimodal completion cues. Compared with other methods that utilize Siamese networks to extract auxiliary modal features and design complex fusion networks, this cue-based learning method proposed in this embodiment can consume fewer computational resources. In one example, modal completion cues can be deployed before and after the input and output of each coding layer in the backbone network to provide modal completion cues for each coding layer.

[0045] After obtaining the current image features and the current multimodal features, the current image features and the current multimodal features can be input into the first modality completion prompter. The first modality completion prompter can use 1x1 convolution to map the input features (current image features and current multimodal features) into a lower-dimensional latent embedding, and obtain the first modality completion prompt output by the first modality completion prompter.

[0046] Then, the first modal completion cue is input into the second modal completion cue. A 1x1 convolution is used to map the first modal completion cue to the original dimension to generate a visual cue, resulting in the second modal completion cue output by the second modal completion cue. This second modal completion cue is a cue feature transformed into the original shape (original dimension). The first modal completion cue is the modal completion cue corresponding to the input of the first encoding layer, and the second modal completion cue is the modal completion cue corresponding to the output of the first encoding layer.

[0047] Step S22: Input the current image features, the current multimodal features, and the first modality completion prompt into the first encoding layer to obtain the first encoding result.

[0048] In this embodiment, after obtaining the first modal completion prompt, in addition to inputting the first modal completion prompt into the second modal completion prompter, the first modal completion prompt, along with the current image features and the current multimodal features, are also input into the first coding layer for processing, thereby obtaining the first coding result output by the first coding layer.

[0049] In a specific example, a spatial fovea operation can be performed on the current image features and the current multimodal features first, and then added to the first modality completion prompt to generate a multimodal internal complementary representation. The multimodal internal complementary representation is then input into the first coding layer to obtain the first coding result.

[0050] Step S23: Based on the first encoding result and the second modal completion prompt, obtain the initial template features and the current search region features.

[0051] In this embodiment, after the first encoding layer outputs the first encoding result, the initial template features and the current search region features can be obtained based on the first encoding result and the second modal completion prompt output by the second modal completion prompt. For example, the second modal completion prompt, which has been transformed into its original shape, can be directly added element-wise to the first encoding result to obtain the initial template features and the current search region features, which can then be used as input for the next encoding layer.

[0052] In other words, the modal completion cue in this embodiment includes at least three operations: 1) using 1x1 convolution to map the input features to a lower-dimensional latent embedding; 2) performing a spatial fovea operation on the input features and adding it to the multimodal cue from the previous layer to generate complementary representations within the multimodal layer; 3) using 1x1 convolution to map the features to the original dimension to generate visual cues, and finally using the cue features transformed into their original shape to directly add them element-wise to the output features of the frozen base model encoder from the previous layer to obtain the input of the next layer encoder.

[0053] In conjunction with any of the above embodiments, the present invention also provides a multimodal visual single-object tracking method based on memory cues. In this method, the current search image frame is the t-th search image frame; the memory features corresponding to the t-th search image frame include: the memory features of the ti-th search image frame, the memory features of the (t-2i)-th search image frame, the memory features of the (t-3i)-th search image frame up to the memory features of the (t-ni)-th search image frame, where n, t, and i are all integers greater than 0; and the memory features of the ti-th search image frame are obtained according to the following steps:

[0054] Step S31: For the ti-th search image frame, obtain the features of the ti-th search region and the target tracking results of the ti-th search image frame.

[0055] In this embodiment, during the tracker inference process, a fixed number of frames i are saved every i frames. That is, the search region features extracted from the coding layer during the historical tracking process are saved every i frames. Based on these features, the prediction results are cropped to serve as memory features.

[0056] In this embodiment, the current search image frame is the t-th search image frame, which is exactly the specific image frame from which memory features need to be extracted. The update interval in this embodiment is i frames. For example, in the search image frame sequence, the i-th search image frame, the 2i-th search image frame, the 3i-th search image frame, ..., the ni-th search image frame are the specific image frames from which memory features need to be extracted. Then, the memory features corresponding to the (t-i+1)-th to the t-th search image frame include: the memory features of the ti-th search image frame, the memory features of the (t-2i)-th search image frame, the memory features of the (t-3i)-th search image frame, and so on up to the (t-ni)-th search image frame, where n, t, and i are all integers greater than 0.

[0057] For example, if t is 100 and i is 20, then the memory features corresponding to the 81st to the 100th search image frame (i.e. the 100th frame in the search image frame sequence) include: the memory features of the 80th search image frame, the memory features of the 60th search image frame, the memory features of the 40th search image frame, and the memory features of the 20th search image frame.

[0058] In this embodiment, when using the aforementioned method to track a target for the ti-th search image frame, the ti-th search region features corresponding to the ti-th search image frame, as well as the target tracking result of the ti-th search image frame, can be obtained.

[0059] Step S32: Based on the target tracking results of the ti-th search image frame, crop the features of the ti-th search region to obtain the memory features of the ti-th search image frame.

[0060] In this embodiment, the features of the ti-th search region can be deformed and cropped according to the target tracking results of the ti-th search image frame, and the cropped features can be determined as the memory features of the ti-th search image frame.

[0061] It should be noted that in this embodiment, the memory features of the t-th search image frame, the memory features of the (t-2i)-th search image frame, the memory features of the (t-3i)-th search image frame, and so on up to the (t-ni)-th search image frame are generated in the same or similar ways as the memory features of the ti-th search image frame, and will not be described again.

[0062] For example, if t is 100, and the current search image frame is the 100th search image frame, reaching the update interval, during the tracking process of the 100th search image frame, the memory features corresponding to the 100th search image frame include: the memory features of the 80th search image frame, the 60th search image frame, the 40th search image frame, and the 20th search image frame. The features of the 100th search region can be saved, and based on the 100th search region features and the corresponding memory features of the 100th search image frame, the target tracking result of the 100th search image frame is obtained, completing the inference for the 100th search image frame. Then, according to the target tracking result of the 100th search image frame, the features of the 100th search region are cropped to obtain the memory features of the 100th search image frame (e.g., mapping the target tracking result of the 100th search image frame onto the deformed features of the 100th search region, and cropping the target features of the target tracking result from the deformed features of the 100th search region to serve as the memory features of the 100th search image frame). Thus, the memory features corresponding to the 101st to the 120th search image frames can be obtained (memory features of the 100th search image frame, memory features of the 80th search image frame, memory features of the 60th search image frame, memory features of the 40th search image frame, and memory features of the 20th search image frame).

[0063] In an alternative embodiment, such as Figure 2 As shown, Figure 2 This is a schematic diagram illustrating a memory feature generation method according to an embodiment of the present invention. Figure 2Taking the ti-th search image frame as an example, the ti-th search region features (search tokens) output by the specific coding layer corresponding to the ti-th search image frame can be reshaped. Then, based on the target tracking result of the ti-th search image frame, the ti-th search region features output by the specific coding layer corresponding to the reshaped ti-th search image frame are cropped to obtain the cropped search region features corresponding to the specific coding layer. Finally, the cropped search region features corresponding to the specific coding layer are reshaped and concatenated to obtain the memory tokens of the ti-th search image frame. To preserve the complete target appearance features, when encountering non-integer edges during cropping, a rounding method that maximizes the area of ​​the selected region can be adopted.

[0064] In one optional example, a specific coding layer can be a coding layer specified in the backbone network. This can be specified based on needs and experience. For example, in a backbone network with a total of 12 coding layers, the 3rd, 6th, and 9th coding layers can be specified as specific coding layers. The above is just an example and is not a limitation.

[0065] In this embodiment, during the tracker inference process, the search region features extracted during the historical tracking process are saved at a specific coding level at fixed intervals of a fixed number of frames. The search region features are deformed and cropped according to the target tracking results, and the cropped features are regarded as memory cues. In this way, the initial template features are fused with memory features from multiple frames at different times, which can build a perception of the target appearance over a longer period of time for target tracking at the current moment, while reducing the interference that may be caused by unreliable memory in a single instance.

[0066] In conjunction with any of the above embodiments, in one implementation, the present invention also provides a multimodal visual single-target tracking method based on memory cues. In this method, step S13 specifically includes steps S41 to S43:

[0067] Step S41: Concatenate the initial template features and the memory features corresponding to the current search image frame to obtain the first concatenated features.

[0068] In this embodiment, the initial target features and the memory features corresponding to the current search image frame can be concatenated to obtain the first concatenated features.

[0069] Step S42: The memory cue fusion module uses a multi-head self-attention mechanism to process the first splicing feature to obtain the memory cue fusion feature.

[0070] In this embodiment, the memory cue fusion module can use a multi-head self-attention mechanism to process the first splicing feature to obtain the memory cue fusion feature.

[0071] Step S43: Determine the current template feature from the memory prompt fusion feature based on the position and number of the initial template feature in the first splicing feature.

[0072] In this embodiment, after obtaining the memory cue fusion feature, the current template feature can be determined from the memory cue fusion feature based on the position and quantity of the initial template feature in the first concatenation feature. For example, the position and quantity of the initial template feature in the first concatenation feature are as follows: the initial template feature is on the far left of the first concatenation feature and occupies two positions; therefore, after obtaining the memory cue fusion feature, the leftmost feature occupying two positions can be determined as the current template feature.

[0073] In this embodiment, using the initial template features as a medium, the multi-head self-attention mechanism with powerful long-distance context modeling capabilities in the memory cue fusion module is used to fuse multiple time-dimensional memory features (i.e., memory cues) to obtain a new template (current template features) containing information about changes in the target's appearance. This effectively achieves feature fusion and interaction. The new template participates in the subsequent feature extraction and association modeling process, enabling the perception of changes in the target's appearance over a longer period of time, which can improve the robustness of the tracker.

[0074] In conjunction with any of the above embodiments, in one implementation, the present invention also provides a multimodal visual single-target tracking method based on memory cues. In this method, S14 specifically includes steps S51 to S54:

[0075] Step S51: Concatenate the current template features and the current search region features to obtain the second concatenated features.

[0076] In this embodiment, the current template features and the current search region features can be concatenated to obtain the second concatenated features.

[0077] Step S52: Process the second splicing feature by inserting a coding layer with a memory adapter to obtain the output of the coding layer with the inserted memory adapter.

[0078] Since the new target template (i.e. the current template features) has a different feature distribution, the subsequent feature extraction and association modeling process with the current search area may differ. Therefore, this embodiment proposes a memory adapter to transform the original feature extraction for the initial target template into feature extraction of the template that incorporates memory features.

[0079] Specifically, in this embodiment, a memory adapter is inserted into the encoding layer with input memory features and frozen parameters, resulting in an encoding layer with the memory adapter inserted. The memory adapter is used to transform the feature extraction of the encoding layer for the initial template features into feature extraction of the current template features fused with memory features by the encoding layer with the memory adapter inserted. In this embodiment, the second concatenated feature can be processed by the encoding layer with the memory adapter inserted to obtain the output of that encoding layer.

[0080] Step S53: Use the output of the encoding layer with the inserted memory adapter as the input of the next encoding layer to obtain the output of the last encoding layer.

[0081] In this embodiment, the output of the encoding layer with the inserted memory adapter can be used as the input of the next encoding layer. After processing by multiple encoding layers, the output of the last encoding layer is finally obtained. During this process, for each encoding layer (whether it is a normal encoding layer with frozen parameters (such as the first encoding layer) or an encoding layer with the inserted memory adapter), modal completion prompts are still deployed before and after its input and output to perform modal completion processing. Specifically, the processing of the modal completion prompts for each encoding layer can be found in the aforementioned embodiments, and will not be repeated here.

[0082] Step S54: Input the output of the last coding layer into the prediction head to obtain the target tracking result of the current search image frame.

[0083] In this embodiment, the last coding layer is connected to a prediction head. After obtaining the output of the last coding layer, the output of the last coding layer is input into the prediction head to obtain the target tracking result of the current search image frame output by the prediction head.

[0084] In this embodiment, since the new template obtained may have differences in extracted features compared with the initial template, in order to introduce time information while freezing the backbone network, a memory adapter is used to adjust the feature extraction and correlation modeling process to adapt to the enhanced template, that is, to adapt the feature extraction and correlation modeling of the new target template that incorporates memory features.

[0085] In an optional embodiment, a memory cue fusion module is paired with each encoding layer with an inserted memory adapter. After the current template features are obtained by the memory extraction fusion module, the current template features and the current search region features are processed by the encoding layer with the inserted memory adapter.

[0086] In conjunction with any of the above embodiments, in one implementation, the present invention also provides a multimodal visual single-target tracking method based on memory cues. In this method, the coding layer with inserted memory adapters includes at least: a first memory adapter and a second memory adapter. Furthermore, in this embodiment, step S52 may specifically include steps S61 to S62:

[0087] Step S61: Process the second splicing feature through the first memory adapter and the multi-head self-attention layer to obtain a first processing result, and fuse the first processing result with the second splicing feature to obtain a first fusion result.

[0088] In this embodiment, for each coding layer to which the memory adapter is inserted, two identical memory adapters are inserted, namely a first memory adapter and a second memory adapter. The first memory adapter is parallel to the multi-head self-attention layer in the coding layer where the memory adapter is inserted; the second memory adapter is parallel to the feedforward layer in the coding layer where the memory adapter is inserted. In an optional example, the memory adapter includes two fully connected layers and an intermediate activation layer, with the two fully connected layers used for dimensionality reduction and dimensionality recovery, respectively.

[0089] In this embodiment, the second concatenated feature is inserted into the encoding layer with a memory adapter. The encoding layer with the memory adapter can first process the second concatenated feature through a first memory adapter and a multi-head self-attention layer, respectively, to obtain the output of the first memory adapter and the output of the multi-head self-attention layer. Then, based on the output of the first memory adapter and the output of the multi-head self-attention layer, a first processing result is obtained. After obtaining the first processing result, the first processing result and the second concatenated feature are fused to obtain a first fusion result.

[0090] Step S62: Process the first fusion result through the second memory adapter and the feedforward layer to obtain a second processing result, and fuse the second processing result with the first fusion result to obtain the output of the encoding layer with the inserted memory adapter.

[0091] In this embodiment, after obtaining the first fusion result, the first fusion result can be processed by the second memory adapter and the feedforward layer respectively to obtain the output of the second memory adapter and the output of the feedforward layer. Then, based on the output of the second memory adapter and the output of the feedforward layer, a second processing result is obtained, and the second processing result is fused with the first fusion result to obtain the output of the coding layer with the memory adapter inserted.

[0092] In an optional example, such as Figure 3 As shown, Figure 3 This is a schematic diagram illustrating the structure of a memory prompt fusion module and an encoding layer with an inserted memory adapter, according to an embodiment of the present invention. Figure 3 (a) represents the Memory Prompt Fusion module, in Figure 3 In (a), the memory cue fusion module may sequentially include at least: a normalization layer, a multi-head self-attention layer, a normalization layer, and a feedforward layer. After the first concatenated feature (including the initial template feature and the memory feature corresponding to the current search image frame) is input into the memory cue fusion module, it is first processed sequentially through the normalization layer and the multi-head self-attention layer to obtain the first feature. Then, the first feature and the first concatenated feature are fused to obtain the second feature. Next, the second feature is processed sequentially through the normalization layer and the feedforward layer to obtain the third feature. Finally, the third feature is fused with the second feature to obtain the memory cue fusion feature output by the memory cue fusion module, so as to determine the current template feature from the memory cue fusion feature. It should be noted that during the training of the memory cue fusion module, the network parameters of the normalization layer, the multi-head self-attention layer, the normalization layer, and the feedforward layer all need to be updated.

[0093] Figure 3 (b) represents the Memory Adaptation (MeA) and the coding layer with the Memory Adaptation inserted. Figure 3In (a), the memory adapter MeA includes two fully connected layers and an intermediate activation layer (GELU), with the two fully connected layers used for dimensionality reduction and dimensionality recovery, respectively. The encoding layer with the inserted memory adapter includes at least: a normalization layer, a multi-head self-attention layer, a normalization layer, a feedforward layer, and two memory adapters, wherein the first memory adapter is parallel to the multi-head self-attention layer; and the second memory adapter is parallel to the feedforward layer. After inserting the second concatenated feature input into the encoding layer with the memory adapter, it is first normalized by the normalization layer to obtain a first normalization result. This first normalization result is then processed by both the first memory adapter and the multi-head self-attention layer to obtain the outputs of the first memory adapter and the multi-head self-attention layer. A first processing result is then obtained based on the outputs of the first memory adapter and the multi-head self-attention layer. After obtaining the first processing result, the first processing result and the second concatenated feature are fused to obtain a first fusion result. After obtaining the first fusion result, it is first normalized by a normalization layer to obtain a second normalized result. Then, the second normalized result is processed by a second memory adapter and a feedforward layer, respectively, to obtain the outputs of the second memory adapter and the feedforward layer. Based on these outputs, a second processed result is obtained, and this second processed result is fused with the first fusion result to obtain the output of the encoding layer with the inserted memory adapter. It should be noted that during the training of the encoding layer with the inserted memory adapter, the network parameters of the normalization layer, multi-head self-attention layer, and feedforward layer are frozen; only the network parameters of the two memory adapters are updated.

[0094] In conjunction with any of the above embodiments, in one implementation, the present invention also provides a multimodal visual single-object tracking method based on memory cues. In this method, the multimodal visual single-object tracking method based on memory cues is implemented through a multimodal visual single-object tracking model based on memory cues; the multimodal visual single-object tracking model based on memory cues includes at least: a pre-trained multimodal visual single-object tracking model; and, in addition to the above steps, may further include steps S71 to S72:

[0095] Step S71: Insert a memory cue fusion module to be trained before the target encoding layer in the pre-trained multimodal visual single-target tracking model, insert a memory adapter to be trained in the target encoding layer, and replace the prediction head in the pre-trained multimodal visual single-target tracking model with the prediction head to be trained to obtain the multimodal visual single-target tracking model with memory cue to be trained.

[0096] In this embodiment, the pre-trained multimodal visual single-object tracking model includes multiple encoding layers with frozen parameters and a trained prediction head. A memory-cue fusion module to be trained can be inserted before the target encoding layer in the pre-trained multimodal visual single-object tracking model. A memory adapter to be trained is inserted into the target encoding layer. Furthermore, since the features extracted in this embodiment change, the prediction head needs to be fine-tuned. Therefore, this embodiment replaces the prediction head in the pre-trained multimodal visual single-object tracking model with the prediction head to be trained, resulting in a multimodal visual single-object tracking model with memory cues to be trained. The target encoding layer is the encoding layer in the multimodal visual single-object tracking model where the input memory features are frozen.

[0097] Step S72: Using the sample template image frames carrying target tracking labels and the sample search image frame sequence carrying target tracking labels, as well as the sample template multimodal frames corresponding to the sample template image frames and the sample search multimodal frames corresponding to the sample search image frame sequence, as training samples, the model parameters of the memory adapter to be trained, the prediction head to be trained, and the memory cue fusion module to be trained in the multimodal visual single-target tracking model to be trained are updated to obtain the multimodal visual single-target tracking model based on memory cue.

[0098] In this embodiment, during model training, sample template image frames carrying target tracking labels and sample search image frame sequences carrying target tracking labels, as well as sample template multimodal frames corresponding to sample template image frames and sample search multimodal frames corresponding to sample search image frame sequences, are used as training samples. The model parameters of the memory adapter to be trained, the prediction head to be trained, and the memory cue fusion module to be trained in the multimodal visual single-target tracking model with memory cue are updated to obtain a multimodal visual single-target tracking model based on memory cue.

[0099] In one optional embodiment, the multimodal visual single-object tracking model to be trained includes a pre-trained embedding layer and a pre-trained multimodal visual single-object tracking model. A memory-cue fusion module to be trained is inserted before the target encoding layer in the pre-trained multimodal visual single-object tracking model. A memory adapter to be trained is inserted into the target encoding layer, and the prediction head in the pre-trained multimodal visual single-object tracking model is replaced with the prediction head to be trained, thus obtaining the multimodal visual single-object tracking model to be trained. The process involves inputting sample template image frames carrying target tracking labels and a sequence of sample search image frames carrying target tracking labels, as well as the sample template multimodal frames corresponding to the sample template image frames and the sample search multimodal frames corresponding to the sample search image frame sequence, into a multimodal visual single-target tracking model with memory cues to be trained. This model is then processed (in the same or similar manner as any of the aforementioned embodiments) to obtain the sample target tracking results of the sample search image frame sequence output by the multimodal visual single-target tracking model with memory cues. Loss is calculated based on the sample target tracking results and target tracking labels. Based on the loss, the model parameters of the memory adapter, the prediction head, and the memory cues fusion module in the multimodal visual single-target tracking model with memory cues to be trained are updated until the loss converges. Finally, the model parameters of the memory adapter, the prediction head, and the memory cues fusion module are fixed to obtain a multimodal visual single-target tracking model based on memory cues.

[0100] In this embodiment, the entire system framework (i.e., the multimodal visual single-object tracking model based on memory cue) freezes most of the backbone network weights (such as the embedding layer and the encoding layer in the pre-trained multimodal visual single-object tracking model) during training. Only the additional memory cue fusion module, memory adapter and prediction head need to be trained, which realizes the introduction of temporal information and improves the robustness of the tracker in complex environments.

[0101] In one embodiment, such as Figure 4 As shown, Figure 4 This is a schematic diagram illustrating the framework of a multimodal visual single-object tracking method based on memory cues, as shown in an embodiment of the present invention. Figure 4 In this invention, the overall network structure includes: a pre-trained embedding layer, a backbone network based on a pre-trained modal completion cue (MCP) (i.e., a multimodal visual single-object tracking model), a memory cue fusion module (MPF) to be trained, a memory adapter (MeA) to be trained, and a prediction head to be trained.

[0102] During model training, firstly, the template image frame (RGB Template) and the current search image frame (RGB Search Frame) can be input into the pre-trained embedding layer. The pre-trained embedding layer encodes the template image frame and the current search image frame to obtain the current image features (RGB Tokens). Secondly, the template multimodal frame (T / E / D Template) corresponding to the template image frame and the current search multimodal frame (T / E / D Search Frame) corresponding to the current search image frame are input into the pre-trained embedding layer. The pre-trained embedding layer encodes the template multimodal frame and the current search multimodal frame to obtain the current multimodal features (T / E / D Tokens).

[0103] Then, the current image features and current multimodal features can be input into a pre-trained modal completion prompter (such as a first modal completion prompter) to obtain a first modal completion prompt. The current image features, current multimodal features, and the first modal completion prompt are then input into the encoding layer of the backbone network (i.e., the first encoding layer, a pre-trained TransformerEncoder Layer) to obtain a first encoding result. The first modal completion prompt is then input into a second modal completion prompter to obtain a second modal completion prompt. Based on the first encoding result and the second modal completion prompt, the initial template features and the current search region features are obtained. For example, the backbone network shown in this embodiment includes 12 encoding layers (including ordinary encoding layers (such as the first encoding layer) and encoding layers with inserted memory adapters to be trained). Each encoding layer uses a vit structure, and each encoding layer also adds a corresponding modal completion prompter. That is, there are also 12 corresponding modal completion prompter layers. Figure 4 The encoding layer and modal completion prompt shown are drawn with the encoding layer and modal completion prompt omitted.

[0104] Combine the initial template features and the memory features corresponding to the current search image frame (the current search image frame is the Mth frame). t If an image frame is searched, the memory features (library) corresponding to the currently searched image frame include: the Mth... t-i Search image frame memory features, Mth t-2i Search image frame memory features, Mth t-3i Search for memory features of image frames up to the Mth frame. t-ni The memory features of the searched image frames are concatenated and then input into the memory cue fusion module to be trained to obtain the current template features. Among them, the Mth... t The memory features of the search image frame are obtained by cropping the current search region features corresponding to the Mt search image frame according to the target tracking results of the Mt search image frame, and then updating them to the memory feature library.

[0105] After obtaining the current template features, the current template features and the current search region features are concatenated and then input into the encoding layer of the memory adapter to be trained for processing. This process is repeated through M encoding layers of the memory adapter to be trained. Simultaneously, a memory cue fusion module to be trained is used before each of the M encoding layers of the memory adapter to be trained. Furthermore, a pre-trained modal completion cue is used before and after each encoding layer to perform modal completion, until the output of the last encoding layer (the pre-trained Transformer Encoder Layer) is obtained. The output of the last encoding layer is then input into the prediction head to be trained to obtain the target tracking result output by the prediction head to be trained (including the coordinates (xt, yt) of the top left corner of the target box, as well as the width wt and height ht of the target box).

[0106] Thus, the key point of this embodiment is that it directly uses previously extracted search region features as cues to introduce temporal information, avoiding secondary extraction of temporal information. Simultaneously, it maintains a frozen single-stream, single-stage backbone network for multimodal fusion tracking, reducing the computational resources required for pre-trained multimodal visual single-object tracking models (such as RGB+X trackers) to incorporate temporal information. The core innovation of this method lies in the design of memory cues, which introduce temporal information to improve the tracker's robustness and fully utilize the features extracted by the original tracker, avoiding the need to design an additional temporal information extraction network to re-extract image features, thereby reducing the computational burden.

[0107] Specifically, to make fuller use of temporal information, this invention uses an initial target template as a medium to achieve association modeling for a new template that incorporates historical prediction features during the feature extraction stage. The initial template possesses original and reliable target appearance information. This invention proposes a memory cue fusion module that fuses the initial target with multiple memory information to obtain a new template. Therefore, this new template simultaneously contains both the original template information and the latest target appearance information. Since a single memory cue may contain interfering information beyond the target, this invention uses memory cuees from multiple time periods to achieve perception of the target's appearance over a longer period, while reducing the impact of interference from potentially unreliable single memories. Furthermore, to enhance the adaptability of the original backbone network to the obtained new template, this invention adds a memory feature fusion adapter to the encoder layer where historical cue is inserted. This achieves feature extraction and association modeling for the new target template with smaller training parameters, providing new possibilities for multimodal visual target tracking in complex scenarios. In addition, to further explore the potential of the base model and adapt to the backbone network that incorporates memory cue, this invention further fine-tunes the prediction head during training on different datasets.

[0108] It should be noted that, for the sake of simplicity, the method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments of the present invention are not limited to the described order of actions, because according to the embodiments of the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions involved are not necessarily essential to the embodiments of the present invention.

[0109] Based on the same inventive concept, one embodiment of the present invention provides a multimodal visual single-target tracking device based on memory cues. (Reference) Figure 5 , Figure 5 This is a structural block diagram of a multimodal visual single-target tracking device based on memory cues, provided in an embodiment of the present invention. Figure 5 As shown, the device includes:

[0110] The encoding module is used to encode the template image frame and the current search image frame to obtain the current image features; and to encode the template multimodal frame corresponding to the template image frame and the current search multimodal frame corresponding to the current search image frame to obtain the current multimodal features.

[0111] The first processing module is used to process the current image features and the current multimodal features to obtain initial template features and current search region features;

[0112] The second processing module is used to obtain the current template features based on the initial template features and the memory features corresponding to the current search image frame; the memory features corresponding to the current search image frame include: the memory features of multiple historical search image frames, and the memory features of a historical search image frame are obtained based on the historical search region features of the historical search image frame and the target tracking results of the historical search image frame;

[0113] The tracking module is used to obtain the target tracking result of the current search image frame based on the current template features and the current search region features.

[0114] Optionally, the device further includes:

[0115] The modal completion module is used to input the current image features and the current multimodal features into a first modal completion prompter to obtain a first modal completion prompt before processing the current image features and the current multimodal features to obtain the initial template features and the current search region features, and to input the first modal completion prompt into a second modal completion prompter to obtain a second modal completion prompt;

[0116] The third processing module is used to input the current image features, the current multimodal features, and the first modality completion prompt into the first encoding layer to obtain the first encoding result;

[0117] The feature processing module is used to obtain the initial template features and the current search region features based on the first encoding result and the second modal completion prompt.

[0118] Optionally, the current search image frame is the t-th search image frame; the memory features corresponding to the (t-i+1)-th to the t-th search image frame include: the memory features of the ti-th search image frame, the memory features of the (t-2i)-th search image frame, the memory features of the (t-3i)-th search image frame up to the (t-ni)-th search image frame, where n, t, and i are all integers greater than 0; the device further includes:

[0119] A memory feature determination module is used to determine the memory features of the ti-th search image frame; the memory feature determination module includes:

[0120] The first determining module is used to obtain the features of the search region and the target tracking result of the search image frame ti for the search image frame ti.

[0121] The cropping module is used to crop the features of the ti-th search region according to the target tracking results of the ti-th search image frame, so as to obtain the memory features of the ti-th search image frame.

[0122] Optionally, the second processing module includes:

[0123] The first splicing module is used to splice the initial template features and the memory features corresponding to the current search image frame to obtain the first splicing features;

[0124] The fourth processing module is used to process the first spliced ​​features through the memory cue fusion module using a multi-head self-attention mechanism to obtain the memory cue fusion features;

[0125] The extraction module is used to determine the current template feature from the memory prompt fusion feature based on the position and number of the initial template features in the first splicing feature.

[0126] Optionally, the tracking module includes:

[0127] The second splicing module is used to splice the current template features and the current search region features to obtain the second spliced ​​features;

[0128] The fifth processing module is used to process the second concatenated feature by inserting a coding layer with a memory adapter to obtain the output of the coding layer with the memory adapter; the memory adapter is used to: transform the feature extraction of the coding layer for the initial template feature into the feature extraction of the coding layer with the memory adapter for the current template feature fused with the memory feature.

[0129] The sixth processing module is used to take the output of the encoding layer with the inserted memory adapter as the input of the next encoding layer to obtain the output of the last encoding layer;

[0130] The tracking determination module is used to input the output of the last coding layer into the prediction head to obtain the target tracking result of the current search image frame.

[0131] Optionally, the coding layer with the inserted memory adapter includes:

[0132] The first memory adapter is parallel to the multi-head self-attention layer in the coding layer in which the memory adapter is inserted;

[0133] The second memory adapter is parallel to the feedforward layer in the coding layer where the memory adapter is inserted;

[0134] The fifth processing module includes:

[0135] The seventh processing module is used to process the second splicing feature through the first memory adapter and the multi-head self-attention layer to obtain a first processing result, and to fuse the first processing result with the second splicing feature to obtain a first fusion result;

[0136] The eighth processing module is used to process the first fusion result through the second memory adapter and the feedforward layer to obtain a second processing result, and to fuse the second processing result with the first fusion result to obtain the output of the encoding layer with the inserted memory adapter.

[0137] Optionally, the memory-cue-based multimodal visual single-target tracking device is implemented using a memory-cue-based multimodal visual single-target tracking model; the memory-cue-based multimodal visual single-target tracking model includes at least a pre-trained multimodal visual single-target tracking model; the device further includes:

[0138] The model initialization module is used to insert a memory cue fusion module to be trained before the target encoding layer in the pre-trained multimodal visual single-object tracking model, insert a memory adapter to be trained in the target encoding layer, and replace the prediction head in the pre-trained multimodal visual single-object tracking model with the prediction head to be trained, so as to obtain a multimodal visual single-object tracking model with memory cue to be trained.

[0139] The model training module is used to use sample template image frames carrying target tracking labels and sample search image frame sequences carrying target tracking labels, as well as sample template multimodal frames corresponding to the sample template image frames and sample search multimodal frames corresponding to the sample search image frame sequences, as training samples to update the model parameters of the memory adapter, the prediction head, and the memory cue fusion module in the multimodal visual single-target tracking model to be trained, thereby obtaining the memory cue-based multimodal visual single-target tracking model.

[0140] Based on the same inventive concept, another embodiment of the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps in the memory-cue-based multimodal visual single-target tracking method described in any of the above embodiments of the present invention.

[0141] Based on the same inventive concept, another embodiment of the present invention provides an electronic device, such as... Figure 6 As shown, Figure 6 This is a schematic diagram of an electronic device according to an embodiment of the present invention. The electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When executed by the processor, the program implements the steps of the memory-based multimodal visual single-target tracking method described in any of the above embodiments of the present invention.

[0142] As the apparatus embodiment is basically similar to the method embodiment, it is described in a relatively simple manner. For relevant details, please refer to the description of the method embodiment.

[0143] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0144] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, apparatus, or computer program products. Therefore, embodiments of the present invention can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, embodiments of the present invention can take the form of computer program products implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0145] Embodiments of the present invention are described with reference to flowchart illustrations and / or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, as well as combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0146] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0147] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0148] Although preferred embodiments of the present invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present invention.

[0149] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0150] The foregoing has provided a detailed description of the multimodal visual single-target tracking method, apparatus, device, and medium based on memory cues provided by the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A memory cue based multi-modal visual single object tracking method, characterized in that, The method comprises: encoding the template image frame and the current search image frame to obtain current image features; and encoding a template multi-modal frame corresponding to the template image frame and a current search multi-modal frame corresponding to the current search image frame to obtain current multi-modal features; processing the current image features and the current multi-modal features to obtain initial template features and current search region features; obtaining current template features according to the initial template features and memory features corresponding to the current search image frame; the memory features corresponding to the current search image frame include memory features of a plurality of historical search image frames respectively; the memory features of a historical search image frame are obtained according to historical search region features of the historical search image frame and a target tracking result of the historical search image frame; obtaining a target tracking result of the current search image frame based on the current template features and the current search region features.

2. The memory-prompted multi-modal visual single object tracking method according to claim 1, wherein, Before processing the current image features and the current multi-modal features to obtain initial template features and current search region features, the method further comprises: inputting the current image features and the current multi-modal features into a first modal completion prompter to obtain a first modal completion prompt, and inputting the first modal completion prompt into a second modal completion prompter to obtain a second modal completion prompt; inputting the current image features and the current multi-modal features and the first modal completion prompt into a first encoding layer to obtain a first encoding result; obtaining the initial template features and the current search region features according to the first encoding result and the second modal completion prompt.

3. The memory-prompted multi-modal visual single object tracking method according to claim 2, wherein, The current search image frame is a tth search image frame; the memory features corresponding to the tth-i+1 search image frame to the tth search image frame include memory features of a tth-i search image frame, memory features of a tth-2i search image frame, memory features of a tth-3i search image frame, and up to memory features of a tth-ni search image frame, n, t and i are all integers greater than 0; the memory features of the tth-i search image frame are obtained by the following steps: obtaining tth-i search region features and a target tracking result of the tth-i search image frame for the tth-i search image frame; cropping the tth-i search region features according to the target tracking result of the tth-i search image frame to obtain the memory features of the tth-i search image frame.

4. The memory-prompted multi-modal visual single object tracking method according to claim 1, wherein, Obtaining current template features according to the initial template features and memory features corresponding to the current search image frame, comprising: splicing the initial template features and the memory features corresponding to the current search image frame to obtain first spliced features; processing the first spliced features by a memory prompt fusion module using a multi-head self-attention mechanism to obtain memory prompt fusion features; determining the current template features from the memory prompt fusion features according to positions and quantities of the initial template features in the first spliced features.

5. The memory-prompted multi-modal visual single object tracking method according to claim 1, wherein, Obtaining a target tracking result of the current search image frame based on the current template features and the current search region features, comprising: stitching the current template feature and the current search region feature to obtain a second stitched feature; processing the second stitched feature through the encoding layer inserted with the memory adapter to obtain an output of the encoding layer inserted with the memory adapter; the memory adapter is used for: converting feature extraction of the encoding layer for the initial template feature into feature extraction of the encoding layer inserted with the memory adapter for the current template feature fused with the memory feature; taking the output of the encoding layer inserted with the memory adapter as an input of a next encoding layer to obtain an output of a last encoding layer; inputting the output of the last encoding layer into a prediction head to obtain a target tracking result of the current search image frame.

6. The memory-prompted multi-modal visual single object tracking method according to claim 5, wherein, The encoding layer inserted with the memory adapter comprises: a first memory adapter, which is parallel to a multi-head self-attention layer in the encoding layer inserted with the memory adapter; a second memory adapter, which is parallel to a feedforward layer in the encoding layer inserted with the memory adapter; processing the second stitched feature through the encoding layer inserted with the memory adapter to obtain an output of the encoding layer inserted with the memory adapter, comprising: processing the second stitched feature through the first memory adapter and the multi-head self-attention layer to obtain a first processing result, and fusing the first processing result with the second stitched feature to obtain a first fusion result; processing the first fusion result through the second memory adapter and the feedforward layer to obtain a second processing result, and fusing the second processing result with the first fusion result to obtain the output of the encoding layer inserted with the memory adapter.

7. The memory-prompted multi-modal visual single object tracking method according to any one of claims 1 to 6, characterized in that, The memory prompt-based multi-modal visual single-target tracking method is implemented through a memory prompt-based multi-modal visual single-target tracking model; The memory prompt-based multi-modal visual single-target tracking model at least comprises: a pre-trained multi-modal visual single-target tracking model; the method further comprises: inserting a to-be-trained memory prompt fusion module before a target encoding layer in the pre-trained multi-modal visual single-target tracking model, inserting a to-be-trained memory adapter in the target encoding layer, and replacing a prediction head in the pre-trained multi-modal visual single-target tracking model with a to-be-trained prediction head to obtain a to-be-trained memory prompt-based multi-modal visual single-target tracking model; taking a sample template image frame carrying a target tracking label and a sample search image frame sequence carrying a target tracking label, and a sample template multi-modal frame corresponding to the sample template image frame and a sample search multi-modal frame corresponding to the sample search image frame sequence as training samples, performing model parameter updating on the to-be-trained memory adapter, the to-be-trained prediction head, and the to-be-trained memory prompt fusion module in the to-be-trained memory prompt-based multi-modal visual single-target tracking model to obtain the memory prompt-based multi-modal visual single-target tracking model.

8. A memory cue based multi-modal visual single object tracking apparatus, characterized by, The device comprises: The encoding module is configured to encode the template image frame and the current search image frame to obtain current image features, and encode a template multi-modal frame corresponding to the template image frame and a current search multi-modal frame corresponding to the current search image frame to obtain current multi-modal features; The first processing module is configured to process the current image features and the current multi-modal features to obtain initial template features and current search region features; The second processing module is configured to obtain current template features according to the initial template features and memory features corresponding to the current search image frame; the memory features corresponding to the current search image frame include memory features of a plurality of historical search image frames respectively, and the memory features of a historical search image frame are obtained according to historical search region features of the historical search image frame and a target tracking result of the historical search image frame; The tracking module is configured to obtain a target tracking result of the current search image frame based on the current template features and the current search region features.

9. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The computer program is executed by the processor to implement the memory prompt-based multi-modal visual single-target tracking method according to any one of claims 1 to 7.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that The computer program is executed by the processor to implement the memory prompt-based multi-modal visual single-target tracking method according to any one of claims 1 to 7.

Images