An interactive target recognition and tracking algorithm for intelligent robots

Abstract

The present invention belongs to the technical fields of machine learning, interactive video target recognition, and single-target visual tracking, and proposes an interactive target recognition and tracking algorithm for intelligent robots. The method takes a manual single-point click in the initial frame as the starting point, extracts deep features of the image and spatial position cues through the Transformer structure, combines the gated expert network to achieve dynamic fusion of multimodal information, and predicts the initial bounding box of the target. On this basis, a multimodal reference prompt is constructed, including visual cues and spatial position cues, to achieve gradual and fine optimization of the bounding box. The optimized bounding box can be used as an initial condition to use a tracker to realize the propagation of the target in the video sequence. This method has the advantages of low initialization cost, high annotation accuracy, lightweight network structure, and strong generalization. It is suitable for scenarios such as intelligent robot interaction in real environments, and effectively improves the practicality and scalability of the interactive tracking system.

Description

Technical Field

[0001] The present invention belongs to the fields of machine learning, interactive video target initialization, and single target visual tracking, and relates to an interactive target recognition and tracking algorithm for intelligent robots. Background Art

[0002] Visual object tracking is a basic task in computer vision, which aims to estimate the position of a specified target in a video sequence based on the initial state. ext TrackingNet and UAV123 typically provide a precisely annotated bounding box for the first frame of the tracking initialization. However, accurately annotating the initial bounding box inevitably introduces a time delay of seconds in real-world scenarios. In contrast, single-click object recognition only requires a delay of milliseconds, which is much more efficient than bounding box annotation and is more suitable for real-world applications such as human-computer interaction and intelligent robotic systems.

[0003] Compared with finely annotated bounding boxes, the information gap introduced by single clicks poses significant challenges. First, traditional trackers such as SiamFC, ATOM, TransT, Stark, and SwinTrack are usually initialized by cropping the reference region at a fixed scale. Single-click-based initialization relies only on a single coordinate representing the position for target localization, lacking scale and boundary information. Scaling the entire frame image to 256×256 or 384×384 may result in the loss of key appearance information. This lack of scale and boundary cues makes single-click-based initialization more challenging than bounding box-based initialization methods. Second, the initial bounding box predicted by the click-based model may have positioning errors, which will further accumulate in subsequent tracking, leading to serious tracking failures.

[0004] In practice, click-based tracking tasks typically combine a tracking model with a detector, such as Yolov10, or interaction models, such as SAM and SAM2. In the first frame, the detector or interaction model first generates an initial target bounding box based on a single click location, which is then followed by the tracking model. However, this modular tracking framework has several limitations. First, the additional detection or interaction model increases the number of parameters, memory usage, and computational overhead. Second, target localization requires post-processing. Both the detection and interaction models generate multiple candidate bounding boxes, and the final bounding box is selected from these multiple candidate bounding boxes using predefined rules, which may increase the error of the initialization result. Compared with tracking initialized with precise bounding boxes, click-based tracking methods have a significant performance gap, making them difficult to apply to real-world scenarios. Despite this, research focused on achieving robust click-based tracking methods remains limited.

[0005] Parameter-Efficient Fine-Tuning (PEFT) significantly reduces the training cost of adapting large models to downstream tasks by freezing the pretrained model parameters and optimizing only a small number of additional modules. Adapter-based methods insert small bottleneck structures between Transformer layers to achieve task-specific learning (Houlsby N et al., “Parameter-efficient transfer learning for NLP”, 2019). LoRA introduces a low-rank matrix factorization strategy to compress parameter updates to less than 0.01% of the original model (Hu EJ et al., “LoRA: Low-Rank Adaptation of Large Language Models”, 2021). Prompt Tuning guides the model by adding learnable vectors to the input sequence (Lester et al., “The Power of Scale for Parameter-Efficient Prompt Tuning”, 2021). To improve the capabilities of the PEFT approach, researchers designed a mixture of experts (MoE) architecture (Fedus et al., "Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity", 2022). MoE uses a gating mechanism to dynamically route data from different tasks to multiple experts, significantly improving performance compared to a single adapter in multi-task learning scenarios. Building on this approach, we introduce the MoE architecture to the task of click-cue-based object recognition and tracking, and propose an efficient parameter fine-tuning scheme for this scenario. By constructing a click-guided expert selection mechanism, we achieve adaptive fusion of object initialization and tracking tasks without burdening the main model. Summary of the Invention

[0006] This paper aims to provide an interactive target recognition and tracking method that combines the Transformer structure, the gated expert network (MoE) and the multi-stage bounding box optimization mechanism. It can complete high-quality target recognition and tracking tasks with just a single click by the user, and has the advantages of both high efficiency and high precision.

[0007] The technical solution of the present invention:

[0008] An interactive target recognition and tracking algorithm for intelligent robots, the steps are as follows:

[0009] Step 1: Manual click initialization and input preprocessing;

[0010] Manually in the initial frame of the video A single click in the video is the starting point; Upper click target Any point on the , and then normalized to Interval, expressed as click coordinates ; At the same time, the initial frame of the video It is scaled to a fixed size of 320×320 and is recorded as image ;image With click coordinates As a joint input, it provides a target hint for manual click initialization;

[0011] Step 2: Use a multi-task unified model for target coarse positioning;

[0012] The multi-task unified model mainly consists of an image encoder, a coordinate encoder, a feature encoder, a gated expert network, and a prediction network;

[0013] The image encoder consists of a convolutional network with a convolution step size of 16×16, which is used to transform the image Convert to deep feature representation ;

[0014] The coordinate encoder is composed of a two-layer linear feedforward network, which is used to convert the click coordinates Convert to deep feature representation ;

[0015] The feature encoder uses the backbone network of the visual tracking network OSTrack, which consists of N layers of Transformer structures stacked with a gated expert network;

[0016] The Transformer structure consists of an attention mechanism and a linear feedforward layer. The attention mechanism in the Transformer structure is defined as follows:

[0017]

[0018] Among them, Q, K, and V represent the query, key, and value of the input feature vector respectively. represents the dimension of the feature vector;

[0019] The Transformer structure with a gated expert network consists of an attention mechanism, a gated expert network, and a linear feedforward layer; deep feature representation , deep feature representation When passing through the Transformer structure with a gated expert network in the feature encoder, it first passes through the attention mechanism and the deep feature representation Position information and deep feature representation in The image information in the image is fused together; deep feature representation , deep feature representation Then pass through the gated expert network; the gated expert network consists of 3 expert sub-networks and a gating module. Each expert sub-network consists of a two-layer linear feedforward network. The gating module represents the , deep feature representation Dynamically allocate computing paths based on the content, activating only some experts to participate in reasoning; deep feature representation , deep feature representation Finally, pass through the linear feedforward layer;

[0020] The prediction network consists of three prediction heads, which are used to predict the probability distribution map of the click target center point coordinates, the click target center point coordinate offset and the click target size; the output of the three prediction heads is used to obtain the rough bounding box of the click target. , that is, click coordinates In the image The approximate spatial position of the target corresponding to ; 、 Respectively represent the click target center coordinates, 、 Respectively represent the width and height of the click target.

[0021] Step 3: Construct the input information for the multimodal prompt optimization phase, specifically:

[0022] Getting a rough bounding box After that, we further construct the fuzzy search area and two different modal reference hints as auxiliary input to refine the initial bounding box:

[0023] Fuzzy search area: with a rough bounding box Center the rough bounding box The width and height of 4 are expanded to form the image area containing the context background, which is recorded as ;

[0024] Visual reference: a rough bounding box Centered on the target clicked in step 1, expand the rough bounding box by 2 times and its adjacent areas, denoted as ;

[0025] Spatial position reference tip: Set the initial frame of the video in step 1 Click coordinates Convert it into the relative position coordinates based on the center point of the fuzzy search area and use it as the click target The spatial position reference in the fuzzy search area is denoted as .

[0026] Step 4: Accurately locate the bounding box based on multimodal cue optimization;

[0027] Will 、 and As a joint input, it is sent to the multi-task unified model in step 2 again, and the task goal is changed from "coarse positioning" to "multimodal prompt optimization"; the multi-task unified model receives 、 and , extract three-way features, represented as fuzzy search area features , visual reference area features Reference features with spatial positions ; Among them, the fuzzy search area features Provide long-range context clues and visual reference area features Provide high-confidence target structure information and spatial position reference features Direct attention to the core target area;

[0028] The prediction network of the final multi-task unified model outputs the optimized click target bounding box , as the initial frame of the video The final label.

[0029] Step 5: Use the Transformer-based visual tracking network OSTrack to implement video target tracking;

[0030] (1) Crop the current frame search area: For the tth frame image of the video , the target tracking result of frame t-1 The bounding box is centered, when t=2, Refers to the initialization result obtained in step 4 , construct a fixed-size search area ;

[0031] (2) Template and search feature extraction: Using the click target bounding box obtained in step 4 , at the initial frame of the video The target image clicked by the user is cut out from the image as the tracking template T; the tracking template T is Input the image encoder together to extract template features and search area features;

[0032] (3) Feature matching and fusion: The template features and search area features are matched and fused with attention through the Transformer structure to capture the temporal consistency of the target;

[0033] (4) Bounding box prediction: Use OSTrack’s prediction network to predict the current frame The target bounding box in , As the current frame The final tracking result will also be used to search the area of ​​the t+1 frame in step 5 (1). of cutting.

[0034] Beneficial effects of the present invention:

[0035] (1) The proposed annotation method has a significant low-cost startup advantage. Compared with the traditional manual fine-grained initial bounding box initialization method, this method only requires a single click in the initial frame of the video to complete the target positioning, reducing the manual annotation cost and operation complexity. At the same time, the simplified interaction process significantly shortens the time overhead of the initialization phase.

[0036] (2) A phased reasoning mechanism is used to decouple the rough positioning of the target from the subsequent fine optimization based on multimodal cues. By introducing contextual information from different modalities at different stages, the target position is gradually refined, effectively improving the positioning accuracy and stability of the bounding box, providing a solid foundation for high-quality automated labeling. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 Schematic diagram of the Transformer-based tracker OSTrack structure.

[0038] Figure 2 Schematic diagram of the multi-task unified network structure.

[0039] Figure 3 Flowchart of the proposed interactive target recognition and tracking method. DETAILED DESCRIPTION

[0040] The specific implementation of the present invention is further described below in conjunction with the accompanying drawings and technical solutions.

[0041] Figure 1 This figure shows the architecture of the Transformer-based tracker, OSTrack. It consists of an image encoder, a coordinate encoder, a feature encoder, a gated expert network, and a prediction network. The feature encoder uses a stack of 12 standard Transformer layers. Compared to commonly used correlation operations, the use of the Transformer module improves the fusion of template features and search region features. The fused features output by the feature encoder are used in the prediction network to extract center coordinates and size information, resulting in the coordinates of the object's bounding box in the input image.

[0042] Figure 2 This is a schematic diagram of the multi-task unified model structure. Figure 3 This is a flowchart for the interactive object recognition and tracking method. The process begins with a single manual click in the initial frame. The normalized click point and a scaled image are fed into the network together to provide semantic cues about the object. The multi-task unified model uses OSTrack as its base model. The feature encoder consists of a stack of Transformer layers with a gated expert network (MoE), enabling joint modeling and efficient inference of image and position information. The gated expert network consists of three expert subnetworks and a gated module with a dynamically allocated input path. Each expert subnetwork consists of a two-layer linear feedforward network, improving computational efficiency and expressiveness.

[0043] After initially predicting the bounding box, the system constructs multimodal auxiliary prompts, including a 4-fold expansion of the fuzzy search area, a 2-fold expansion of the visual reference area, and converts the click point into relative position coordinates as a spatial position prompt. The multimodal prompt information and the image are again input into the multi-task unified model. With the help of the Transformer's attention mechanism, features from different modalities and scales are integrated to achieve multimodal joint optimization of the bounding box and output a more accurate initialization bounding box. Finally, full video target propagation is achieved through the OSTrack tracker: the optimized bounding box is cropped as a template and fused and matched with the search area features of each frame, the target position is predicted frame by frame, and complete video target tracking is completed. The entire system has the characteristics of strong scalability and high tracking stability, and is suitable for real-world applications.

[0044] OSTrack in the multi-task unified model can be arbitrarily replaced with other Transformer-based trackers. Training for the first-stage coarse prediction and the second-stage multimodal cue optimization is performed in stages to ensure high performance gains. The training set uses all video sequences from the LaSOT and Got10k datasets. The optimizer chosen is AdamW, with an initial learning rate of 0.0001. The learning rate is reduced by a factor of 10 every 400 epochs, for a total of 800 epochs.

[0045] The feature extraction and fusion network structures in the tracker OSTrack and the Transformer-based tracker OSTrack are as follows:

[0046] .

Claims

1. An interactive target recognition and tracking algorithm for intelligent robots, characterized by: Here are the steps: Step 1: Manual click initialization and input preprocessing; Manually in the initial frame of the video A single click in the video is the starting point; Upper click target Any point on the , and then normalized to Interval, expressed as click coordinates ; At the same time, the initial frame of the video It is scaled to a fixed size of 320×320 and is recorded as image ;image With click coordinates As a joint input, it provides a target hint for manual click initialization; Step 2: Use a multi-task unified model for target coarse positioning; Step 3: Construct input information for the multimodal prompt optimization phase; Step 4: Accurately locate the bounding box based on multimodal cue optimization; Step 5: Use the Transformer-based visual tracking network OSTrack to implement video target tracking; The specific implementation process of step 2 is as follows: The multi-task unified model mainly consists of an image encoder, a coordinate encoder, a feature encoder, a gated expert network, and a prediction network; The image encoder consists of a convolutional network with a convolution step size of 16×16, which is used to transform the image Convert to deep feature representation ; The coordinate encoder is composed of a two-layer linear feedforward network, which is used to convert the click coordinates Convert to deep feature representation ; The feature encoder uses the backbone network of the visual tracking network OSTrack, which consists of N layers of Transformer structures stacked with a gated expert network; The Transformer structure consists of an attention mechanism and a linear feedforward layer. The attention mechanism in the Transformer structure is defined as follows: ； Among them, Q, K, and V represent the query, key, and value of the input feature vector respectively. represents the dimension of the feature vector; The Transformer structure with a gated expert network consists of an attention mechanism, a gated expert network, and a linear feedforward layer; deep feature representation , deep feature representation When passing through the Transformer structure with a gated expert network in the feature encoder, it first passes through the attention mechanism and the deep feature representation Position information and deep feature representation in The image information in the image is fused together; deep feature representation , deep feature representation Then pass through the gated expert network; the gated expert network consists of 3 expert sub-networks and a gating module. Each expert sub-network consists of a two-layer linear feedforward network. The gating module represents the , deep feature representation Dynamically allocate computing paths based on the content, activating only some experts to participate in reasoning; deep feature representation , deep feature representation Finally, pass through the linear feedforward layer; The prediction network consists of three prediction heads, which are used to predict the probability distribution map of the click target center point coordinates, the click target center point coordinate offset and the click target size; the output of the three prediction heads is used to obtain the rough bounding box of the click target. , that is, click coordinates In the image The approximate spatial position of the target corresponding to ; 、 Respectively represent the coordinates of the click target center, 、 Respectively represent the width and height of the click target.

2. The interactive target recognition and tracking algorithm for intelligent robots according to claim 1, characterized in that: The specific implementation process of step 3 is as follows: Getting a rough bounding box After that, we further construct the fuzzy search area and two different modal reference hints as auxiliary input to refine the initial bounding box: Fuzzy search area: with a rough bounding box Center the rough bounding box The width and height of 4 are expanded to form the image area containing the context background, which is recorded as ; Visual reference: a rough bounding box Centered on the target clicked in step 1, expand the rough bounding box by 2 times and its adjacent areas, denoted as ; Spatial position reference tip: Set the initial frame of the video in step 1 Click coordinates Convert it into the relative position coordinates based on the center point of the fuzzy search area and use it as the click target The spatial position reference in the fuzzy search area is denoted as .

3. The interactive target recognition and tracking algorithm for intelligent robots according to claim 2, characterized in that: The specific implementation process of step 4 is as follows: Will 、 and As a joint input, it is sent to the multi-task unified model in step 2 again, and the task goal is changed from "coarse positioning" to "multimodal prompt optimization"; the multi-task unified model receives 、 and , extract three-way features, expressed as , visual reference area features Reference features with spatial positions ; Among them, the fuzzy search area features Provide long-range context clues and visual reference area features Provide high-confidence target structure information and spatial position reference features Direct attention to the core target area; The prediction network of the final multi-task unified model outputs the optimized click target bounding box , as the initial frame of the video The final label.

4. The interactive target recognition and tracking algorithm for intelligent robots according to claim 3 is characterized in that: The specific implementation process of step 5 is as follows: (1) Crop the current frame search area: For the tth frame image of the video , the target tracking result of frame t-1 The bounding box is centered, when t=2, Refers to the initialization result obtained in step 4 , construct a fixed-size search area ; (2) Template and search feature extraction: Using the click target bounding box obtained in step 4 , at the initial frame of the video The target image clicked by the user is cut out from the image as the tracking template T; the tracking template T is Input the image encoder together to extract template features and search area features; (3) Feature matching and fusion: The template features and search area features are matched and fused with attention through the Transformer structure to capture the temporal consistency of the target; (4) Bounding box prediction: Use OSTrack’s prediction network to predict the current frame The target bounding box in , As the current frame The final tracking result will also be used to search the area of ​​the t+1 frame in step 5 (1). of cutting.

Images