Robot-oriented open-vocabulary object tracking and semantic interaction method and robot

Abstract

The application provides a kind of open vocabulary target tracking and semantic interaction method and robot for robot, including steps: collecting the multi-modal perception data of environment, synchronization alignment generates image frame sequence;Set detection period and divide image frame sequence into detection frame and non-detection frame;Establish feature mapping relationship, associate visual feature vector with corresponding target area image temporary tracking identifier, construct target feature library;Construct hierarchical trajectory memory module and identity recovery strategy, realize the identity consistency in long-time tracking of history target;Receive natural language input, determine the task focus target matched with natural language input description, generate control instruction of executing mechanism based on task focus target, drive robot to adjust motion state.The application solves the technical problems that there are still deficiencies and tracking effect is poor when applied to resource-constrained quadruped robot platform.

Description

Technical Field

[0001] This invention relates to the field of robot intelligent control technology, specifically to an open-vocabulary target tracking and semantic interaction method for robots and a robot. Background Technology

[0002] In recent years, with the rapid iteration of intelligent robot technology, quadruped robots have been increasingly widely used in fields such as inspection, search and rescue, and following. One of their core capabilities is the detection and continuous tracking of dynamic targets. Existing technologies have made many explorations for this task.

[0003] In terms of visual tracking algorithms, common approaches include traditional feature-based tracking methods and deep learning-based tracking methods. For example, some studies use color tracking algorithms to track targets by acquiring hue and brightness features of the target area; others have built tracking models based on Siamese networks and established quadruped robot target tracking datasets. Regarding system control and navigation, existing technologies have proposed collision-free target tracking methods that combine guiding vector fields and disturbance suppression controllers, or combine target tracking, multi-sensor fusion SLAM (Simultaneous Localization and Mapping), and terrain accessibility assessment to enhance the robot's autonomous perception and path planning capabilities in complex environments.

[0004] However, existing technologies still have shortcomings when applied to resource-constrained quadruped robot platforms: on the one hand, their semantic understanding capabilities in open scenarios are significantly insufficient, and traditional detection and tracking models mostly rely on predefined fixed categories, making it difficult to understand and respond to arbitrary unknown targets described in natural language; on the other hand, identity consistency in long-term tracking is difficult to guarantee. When a target is severely occluded, disappears briefly, or reappears, the model lacks a comprehensive re-identification mechanism that integrates semantic context, appearance features, and motion trajectory, which can easily lead to frequent changes and confusion in the target's identity. Summary of the Invention

[0005] The purpose of this invention is to overcome the above-mentioned technical deficiencies and provide an open vocabulary target tracking and semantic interaction method and robot for robots, solving the technical problems of existing technologies still having shortcomings and poor tracking effects when applied to resource-constrained quadruped robot platforms.

[0006] To achieve the above-mentioned technical objectives, in a first aspect, the present invention provides an open-vocabulary target tracking and semantic interaction method for robots, comprising the following steps:

[0007] Collect multimodal perception data of the environment, including RGB (Red-Green-Blue) image data and depth image data, synchronize and align the RGB image data and the depth image data to generate an image frame sequence that can be used for model inference;

[0008] The image frame sequence is divided into detection frames and non-detection frames by setting a detection period. In the detection frame, the RGB image and the task prompt are input into the fine-tuned visual language model to generate structured target state information.

[0009] In non-detection frames, the Kalman filter prediction process is initiated based on the target's historical motion state to generate Kalman filter prediction results;

[0010] Based on the target state information, the target region image is cropped from the original image, and a normalized visual feature vector is generated based on the target region image. A feature mapping relationship is established, and the visual feature vector is associated with the corresponding temporary tracking identifier of the target region image to construct a target feature library.

[0011] The system receives natural language input, converts the natural language input into a text semantic feature vector through the text encoding branch of the visual language model, calculates the cross-modal similarity between the text semantic feature vector and the visual feature vectors of each instance in the target feature library, determines the task focus target that matches the description of the natural language input, and generates control instructions for the actuator based on the task focus target to drive the robot to adjust its motion state.

[0012] Compared with the prior art, the beneficial effects of the present invention include:

[0013] (1) This invention combines a heterogeneous dual-rate tracking mechanism with a hierarchical trajectory memory module to achieve a mode that combines semantic perception of detection frames with motion prediction of non-detection frames, thereby significantly reducing latency while maintaining the accuracy of open vocabulary. By utilizing multi-factor joint matching and a dual-buffer structure, it effectively solves the problem of label loss and switching caused by target occlusion or temporary disappearance, ensuring the continuity and stability of long-term tracking.

[0014] (2) The present invention constructs a natural language driven closed-loop control system that deeply integrates open vocabulary semantic perception, cross-modal target locking and robot servo motion. It can map natural language instructions into three-dimensional spatial pose deviations and generate control instructions, enabling quadruped robots to autonomously lock and follow targets in unstructured environments based on semantics, breaking through the dependence of traditional methods on preset categories or manual annotation.

[0015] According to some embodiments of the present invention, a hierarchical trajectory memory module and an identity recovery strategy are constructed to achieve identity consistency in long-term tracking of historical targets, including the following steps:

[0016] Create two ordered dictionaries, active trajectory and historical trajectory, set the maximum number of historical trajectories and the survival time, and maintain a global identifier mapping table to ensure identity consistency across frames;

[0017] The system monitors the tracking status in real time. If a target is not detected for a set number of consecutive frames, it is determined that the target is lost. The current target is removed from the active trajectory cache and moved to the historical trajectory cache, retaining the most recent motion state and visual semantic features. At the same time, the expiration time is marked as the current frame number + the set value.

[0018] When a new target is detected, two levels of similarity are calculated between the new target and the historical trajectory: at the spatial continuity level, the intersection over union (IoU) ratio between the predicted box and the new detection box is calculated; at the visual semantic consistency level, the cosine similarity between the features of the new target and the historical features is calculated. Finally, a weighted summation strategy is used to determine whether the new target matches the candidate.

[0019] According to some embodiments of the present invention, a weighted summation strategy is used to determine whether a new target matches a candidate, including the following steps:

[0020] The IOU value is calculated based on a weighted summation strategy. If the IOU value is greater than 0.5 and the feature similarity is greater than 0.6, the new target is determined to be the original target that was lost and then reappeared. The global fixed identifier of the historical trajectory is assigned to the new target, the trajectory is moved back to the active trajectory cache, the trajectory state is updated with the new detection box and features, the lost frame count is reset, the historical motion model parameters are inherited, and a continuous trajectory spanning the occlusion period is formed.

[0021] According to some embodiments of the present invention, the RGB image data and the depth image data are synchronized and aligned to generate an image frame sequence that can be used for model inference, including the following steps:

[0022] Through camera intrinsic parameter calibration and spatiotemporal synchronization mechanism, the RGB image data and the depth image data are aligned in pixel space, and millisecond-level time synchronization is achieved based on hardware timestamps to eliminate misaligned, dropped, and invalid data, forming standardized video stream data;

[0023] The video stream data is decoded and its quality is assessed to generate an image frame sequence that can be used for model inference.

[0024] According to some embodiments of the present invention, the image frame sequence is divided into detection frames and non-detection frames by setting a detection period, including the following steps:

[0025] Let the detection period parameter T be. In the main loop, the frame counter collects one frame every T frames as a detection frame, triggering the full visual language model inference. The image frame sequence other than the detection frame is set as a non-detection frame. The non-detection frame pauses the calling of the visual language model, reuses the semantic results of the previous detection period, and continues the trajectory only through motion prediction.

[0026] According to some embodiments of the present invention, in a detection frame, the RGB image and the task prompt are input into a fine-tuned visual language model to generate structured target state information, including the following steps:

[0027] The RGB image and task prompt are input into the fine-tuned Florence-2 visual language model, and the output includes the target bounding box, semantic label and confidence score.

[0028] In non-detection frames, a Kalman filter prediction process is initiated based on the target's historical motion state to generate Kalman filter prediction results, including the following steps:

[0029] The Kalman filter is activated based on the uniform motion model. Based on the target's position and velocity at the previous moment, the center coordinates and width and height of the target in the current frame are predicted as the Kalman filter prediction result to maintain the continuity of the trajectory.

[0030] According to some embodiments of the present invention, a target region image is cropped from the original image based on the target state information, and a normalized visual feature vector is generated based on the target region image. A feature mapping relationship is established, and the visual feature vector is associated with the corresponding temporary tracking identifier of the target region image to construct a target feature library, including the following steps:

[0031] The target region image is cropped based on the target bounding box and input into a pre-trained visual encoder to generate a 512-dimensional normalized visual feature vector.

[0032] Receive natural language commands from the interactive interface and input them into a pre-trained text encoder to generate normalized text feature vectors;

[0033] Establish a mapping relationship between text feature vectors and visual feature vectors, associate visual feature vectors with corresponding temporary tracking labels of target region images, and store them in the target feature library for subsequent cross-modal matching and trajectory association.

[0034] According to some embodiments of the present invention, generating control commands for the actuator based on the task focus target to drive the robot to adjust its motion state includes the following steps:

[0035] Extract the center pixel coordinates of the bounding box of the target focus of the task, take the median effective depth of the neighborhood of the center pixel coordinates to calculate the actual distance, calculate the horizontal deviation between the target and the image center and the distance deviation between the target and the preset following distance, input the distance deviation into the PID (Proportional-Integral-Derivative) controller, and calculate the forward speed.

[0036] By combining dead zone control strategy, when the horizontal deviation is less than the rotational dead zone or the lateral dead zone, the corresponding speed is set to zero to avoid system motion jitter. Then, the three-axis speed is smoothed by exponential smoothing filter to limit the maximum forward speed and rotational speed and ensure motion stability.

[0037] Based on the completed quadruped robot motion control instruction set, the calculated three-axis velocity parameters are converted into standard instruction format, and an independent quadruped robot control thread is started.

[0038] According to some embodiments of the present invention, based on the constructed quadruped robot motion control instruction set, the calculated three-axis velocity parameters are converted into a standard instruction format, and an independent quadruped robot control thread is started, including the following steps:

[0039] The motion control instruction set is packaged into JSON format and sent to the quadruped robot's drive unit via UDP protocol (User Datagram Protocol);

[0040] Based on the state machine's automatic switching between standing and moving states, a complete physical closed loop is completed, from natural language command parsing, target locking, deviation calculation to quadruped robot motion execution, enabling intelligent target tracking of quadruped robots driven by natural language.

[0041] In a second aspect, the present invention provides a quadruped robot equipped with a target tracking and semantic interaction system, wherein the target tracking and semantic interaction system applies the open vocabulary target tracking and semantic interaction method for robots as described in any one of the first aspects.

[0042] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0043] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, wherein the abstract drawings are to be completely consistent with one of the drawings in the specification:

[0044] Figure 1 This is a schematic diagram of the overall architecture of a quadruped robot target tracking system provided in an embodiment of the present invention;

[0045] Figure 2 A flowchart of the heterogeneous dual-rate tracking mechanism provided in an embodiment of the present invention;

[0046] Figure 3 This is a schematic diagram of the hierarchical trajectory memory and identity recovery mechanism provided in an embodiment of the present invention;

[0047] Figure 4 A flowchart of the natural language interaction and control closed loop provided for embodiments of the present invention. Detailed Implementation

[0048] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0049] It should be noted that although functional modules are divided in the system diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than the module division in the system or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and the aforementioned figures are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0050] Reference Figures 1 to 4 , Figure 1 This is a schematic diagram of the overall architecture of a quadruped robot target tracking system provided in an embodiment of the present invention; Figure 2 A flowchart of the heterogeneous dual-rate tracking mechanism provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the hierarchical trajectory memory and identity recovery mechanism provided in an embodiment of the present invention; Figure 4 A flowchart of the natural language interaction and control closed loop provided for embodiments of the present invention.

[0051] In one embodiment, an open-vocabulary target tracking and semantic interaction method for robots is characterized by the following steps: acquiring multimodal perception data of the environment, including RGB image data and depth image data; synchronously aligning the RGB image data and depth image data to generate an image frame sequence for model inference; setting a detection period to divide the image frame sequence into detection frames and non-detection frames; in the detection frames, inputting the RGB image and task prompts into a fine-tuned visual language model to generate structured target state information; in the non-detection frames, initiating a Kalman filter prediction process based on the target's historical motion state to generate Kalman filter prediction results; and cropping the target region image from the original image based on the target state information. Based on the target region image, normalized visual feature vectors are generated, feature mapping relationships are established, and the visual feature vectors are associated with the corresponding temporary tracking markers of the target region image to construct a target feature library. A hierarchical trajectory memory module and identity recovery strategy are constructed to achieve identity consistency in long-term tracking of historical targets. Natural language input is received, and the natural language input is converted into text semantic feature vectors through the text encoding branch of the visual language model. The cross-modal similarity between the text semantic feature vectors and the visual feature vectors of each instance in the target feature library is calculated to determine the task focus target that matches the description of the natural language input. Based on the task focus target, control instructions for the actuator are generated to drive the robot to adjust its motion state.

[0052] Specifically, the following steps are included:

[0053] Step 1, Multimodal Sensing Preprocessing:

[0054] The multimodal perception data consists of RGB and depth image data acquired by the depth camera on the head of the quadruped robot. The RGB and depth images are spatially aligned and time-stamped to form standardized video stream data. Edge computing nodes start independent receiving threads to acquire video stream data, decode and judge the quality of the video stream, and generate image frame sequences that can be used for model inference.

[0055] Step 2, Open-vocabulary target detection and semantic extraction based on visual language model: By introducing a large visual-language model (VLM) as a global semantic prior, zero-sample detection of any text description category is achieved. On this basis, a multimodal target state estimation and semantic embedding pipeline is constructed, which combines computationally expensive deep semantic reasoning with lightweight motion prediction and feature matching. This significantly reduces system latency while ensuring the accuracy of open-vocabulary detection, and outputs structured target state information and high-dimensional semantic vector representation for subsequent continuous target tracking tasks.

[0056] The core of the open vocabulary target detection and semantic extraction module lies in the heterogeneous dual-rate tracking mechanism. This mechanism balances the semantic accuracy and real-time requirements of the open vocabulary through an architecture that alternates between detection updates and trajectory prediction. The specific process is as follows:

[0057] 1) Construction and Coordination Strategy of the Two-Stage Tracking Mechanism: Design a working mode in which detection frames and prediction frames are executed alternately. Set the detection frequency parameter T, and perform a full inference of the visual language model on the current image every T frames; in non-detection frames, the visual language model is not invoked, the structured semantic results of the previous detection cycle are reused, and the target trajectory is continued through the motion prediction Kalman filter algorithm.

[0058] 2) Open vocabulary detection and semantic structured output: In the detection frame, the image and task prompt are input into the fine-tuned visual language model to generate detection results containing target bounding boxes (Bboxes), semantic labels, and confidence scores. The detection results are post-processed to remove invalid detection boxes with widths and heights smaller than preset thresholds, and redundant boxes are removed using the non-maximum suppression (NMS) algorithm, retaining the target detection boxes with the highest confidence scores. For each detected target region, the image description task of the visual language model is further invoked to generate fine-grained descriptions, such as "a woman wearing a red jacket," forming structured target state information.

[0059] 3) Kalman Filter Trajectory Prediction and Feature Association: In non-detection frames, the Kalman filter prediction process is initiated based on the target's historical motion state. The target state vector is updated over time based on the target's position and velocity information from the previous frame. The center coordinates, width, height, and velocity of the target in the current frame are predicted, and the predicted bounding box is output. The Kalman filter prediction result is compared with the visual semantic features in the target feature library using cosine similarity. If the similarity is higher than a threshold... Then the prediction results will be associated with the existing trajectory to avoid mismatches caused by motion blur.

[0060] 4) Construction and embedding alignment of multimodal feature libraries:

[0061] Based on the target bounding box (Bbox) output in step 2, crop the target region image from the original image;

[0062] Image of the target area Input pre-trained image encoder Generate normalized visual feature vectors The formula is as follows:

[0063]

[0064] Normalize is a normalization operation.

[0065] At the same time, using a text encoder Process natural language instructions input from the interactive interface to generate text semantic feature vectors. The formula is as follows:

[0066]

[0067] Establish feature mapping relationships and map visual feature vectors It is associated with the corresponding temporary tracking identifier and stored in the target feature database.

[0068] Step 3, Layered Trajectory Memory: The layered trajectory memory module includes: an active trajectory cache unit, used to store and update the trajectory status, motion parameters, and visual semantic feature sequences of currently visible targets in real time; a historical trajectory cache unit, configured with a survival time management mechanism, used to temporarily store the trajectory context of temporarily lost targets; and an identity recovery unit, which, based on a multi-factor joint matching algorithm of spatial continuity, temporal consistency, and visual semantic similarity, compares the newly input target detection with the entries in the historical trajectory cache, and remaps the temporary detection identifier to the original global identifier when the matching conditions are met, thereby realizing identity consistency recovery and trajectory continuation after cross-frame interruption.

[0069] The core of the hierarchical trajectory memory module lies in the multi-factor joint identity recovery mechanism. It solves the trajectory breakage problem caused by target occlusion and temporary disappearance through a hierarchical memory structure and cross-modal similarity calculation. The specific steps are as follows:

[0070] 1) Initialize the active trajectory cache and historical trajectory cache, set the trajectory lifetime threshold (TTL), establish a global identifier mapping table, maintain the mapping relationship between temporary detection identifiers and globally unique identifiers, and ensure the consistency of identity across frames.

[0071] 2) Real-time monitoring of target tracking status: If a target is not detected within N consecutive frames, it is removed from the active trajectory cache and added to the historical trajectory cache, retaining its most recent motion state and visual semantic features, while marking its expiration time as the current frame number + TTL;

[0072] 3) Trajectory matching: For a newly detected target, calculate the predicted bounding box at the spatial continuity factor level. With the new detection frame Intersection over Union (IoU) This reflects motion consistency; at the visual semantic factor level, it calculates the features of new targets. With historical characteristics cosine similarity cos( It measures the consistency between appearance and semantics; a weighted summation strategy is used. ,like Then it is considered a candidate match, where and This corresponds to the weight value.

[0073] 4) Identity restoration and trajectory continuation: If the IoU value between the new target and a certain historical trajectory is greater than a preset threshold... And the feature similarity is greater than a preset threshold. If the target is identified as the same target, the fixed identifier of the historical trajectory is assigned to the new target and moved back to the active trajectory cache. At the same time, the trajectory state is updated with the newly detected bounding box and features, the lost frame counter is reset, and the motion model parameters of the historical trajectory are inherited. Finally, the current frame is inserted into the sequence as a new node of the trajectory to form a continuous trajectory that spans the occlusion period.

[0074] 5) Dynamic cleanup of expired data: Periodically traverse the historical trajectory cache and remove expired trajectory data that has exceeded its time-to-live (TTL).

[0075] Step 4, Natural Language Interaction and Control: The natural language interaction and control module includes: an instruction parsing unit, which receives natural language input and converts it into text semantic feature vectors through the text encoding branch of the visual language model; a target binding unit, which determines the task focus target matching the instruction description by calculating the cross-modal similarity between the text semantic features and the visual semantic features of each instance in the target feature library; a pose calculation unit, which fuses the image coordinates and depth information of the focus target to calculate the three-dimensional spatial pose deviation of the target relative to the robot; and a motion control unit, which generates control instructions for the actuators based on the pose deviation, driving the robot to adjust its motion state to reduce the deviation, forming a closed-loop control system from language understanding to physical motion.

[0076] The core of the natural language interaction and control module lies in cross-modal servo control, which maps natural language into spatial constraints to achieve a closed loop from language understanding to physical motion. The specific process is as follows:

[0077] 1) Functional integration of the visual human-computer interaction interface: Build a visual human-computer interaction interface, configure video display window, system log bar, target list and command input box to realize real-time display of tracking screen, system status monitoring and natural language command input;

[0078] 2) Instruction parsing and cross-modal target locking: The user's natural language input instructions are fed into the text feature encoder to generate text semantic feature vectors. , and visual feature vectors in the target feature library Calculate cosine similarity:

[0079]

[0080] Select the target with the highest similarity and a value greater than a preset threshold as the unique target to lock and track;

[0081] 3) Solving spatial pose deviation: Extracting the center pixel coordinates of the target bounding box The relative horizontal offset is obtained by normalizing the image width and height:

[0082]

[0083] in Given the image width, and combining it with an RGB-D image (an image pair containing an RGB color image and a depth map), the median effective depth of the target center neighborhood is taken and converted into the actual distance. The formula is as follows:

[0084]

[0085] in The set of depth pixels in the neighborhood of the center of the target bounding box, in millimeters; calculate the distance deviation. with horizontal deviation :in The distance to the target.

[0086]

[0087]

[0088] 4) Motion command generation and smoothing strategy: Input the distance deviation into the PID controller, and output the forward speed.

[0089]

[0090] in , and These are the proportional, integral, and differential coefficients, respectively.

[0091] Based on horizontal deviation Calculate the rotational and lateral velocities, and apply an exponential smoothing filter to the three-axis velocities:

[0092]

[0093] in The speed of the last output. For the speed of this output, It is the target speed. The smoothing coefficient is used to obtain the final control values ​​for the forward, lateral, and rotation axes;

[0094] 5) Closed-loop control execution and state management: The calculated three-axis velocity components are integrated into low-level motion commands and sent to the quadruped robot's drive unit via UDP protocol; the system dynamically switches between standing, moving and transitional states based on a state machine. When the target is lost due to timeout, it automatically enters the stop state and restores the standing posture, thus completing the physical closed loop from perception and decision-making to action.

[0095] In one embodiment, a hierarchical trajectory memory module and an identity recovery strategy are constructed to achieve identity consistency for long-term tracking of historical targets. The steps include: creating two ordered dictionaries, active trajectory and historical trajectory, setting a maximum number of historical trajectories and a lifespan, and maintaining a global identifier mapping table to ensure cross-frame identity consistency; the system monitors the tracking status in real time, and if a target is not detected for a set number of consecutive frames, it is determined that the target is lost, the current target is removed from the active trajectory cache and moved to the historical trajectory cache, the recent motion state and visual semantic features are retained, and the expiration time is marked as the current frame number + a set value; when a new target is detected, the two-layer similarity between the new target and the historical trajectory is calculated: at the spatial continuity level, the intersection-union ratio of the predicted box and the new detection box is calculated, and at the visual semantic consistency level, the cosine similarity between the features of the new target and the historical features is calculated. Finally, a weighted summation strategy is used to determine whether the new target matches the candidate.

[0096] In real-world monitoring and tracking environments, scenarios such as target loss and occlusion often occur, interrupting the tracking process. In this embodiment, two ordered dictionaries, active trajectory and historical trajectory, are created to effectively address the problem of inconsistent identity during long-term tracking. When a target is severely occluded, temporarily disappears, or reappears, this invention provides a comprehensive re-identification mechanism that integrates semantic context, appearance features, and motion trajectory to avoid frequent changes and confusion in target identity.

[0097] The process of determining whether a new target matches a candidate using a weighted summation strategy includes the following steps: calculating the IoU (Intersection over Union) value based on the weighted summation strategy; if the IoU value is greater than 0.5 and the feature similarity is greater than 0.6, then the new target is determined to be the original target that was lost and then reappeared; the global fixed identifier of the historical trajectory is assigned to the new target; the trajectory is moved back to the active trajectory cache; the trajectory state is updated with the new detection box and features; the lost frame count is reset; the parameters of the historical motion model are inherited; and a continuous trajectory spanning the occlusion period is formed.

[0098] The process of synchronizing and aligning RGB image data and depth image data to generate an image frame sequence that can be used for model inference includes the following steps: aligning RGB image data and depth image data in pixel space through camera intrinsic parameter calibration and spatiotemporal synchronization mechanism, and achieving millisecond-level time synchronization based on hardware timestamps, eliminating misaligned, dropped, and invalid data to form standardized video stream data; decoding and quality assessment of the video stream data to generate an image frame sequence that can be used for model inference.

[0099] Furthermore, the detection period is set to divide the image frame sequence into detection frames and non-detection frames, including the following steps: setting the detection period parameter T, in the main loop, the frame counter collects one frame every T frames as a detection frame, triggering the full visual language model inference; the image frame sequence except for the detection frames is set as non-detection frames, the visual language model is paused in the non-detection frames, the semantic results of the previous detection period are reused, and the trajectory is continued only through motion prediction.

[0100] In the detection frame, the RGB image and task prompt are input into the fine-tuned visual language model to generate structured target state information. This includes the steps of: inputting the RGB image and task prompt into the fine-tuned Florence-2 visual language model and outputting detection results containing target bounding boxes, semantic labels, and confidence scores. In the non-detection frame, a Kalman filter prediction process is initiated based on the target's historical motion state to generate Kalman filter prediction results. This includes the steps of: initiating a Kalman filter based on a uniform motion model and predicting the center coordinates and width and height of the target in the current frame as the Kalman filter prediction results based on the target's position and velocity in the previous moment, in order to maintain the continuity of the trajectory.

[0101] The process involves cropping a target region image from the original image based on target state information, generating a normalized visual feature vector based on the target region image, establishing a feature mapping relationship, associating the visual feature vector with the corresponding temporary tracking marker of the target region image, and constructing a target feature library. The steps include: cropping the target region image based on the target bounding box, inputting it into a pre-trained visual encoder to generate a 512-dimensional normalized visual feature vector; receiving natural language instructions from the interactive interface, inputting them into a pre-trained text encoder to generate a normalized text feature vector; establishing a mapping relationship between the text feature vector and the visual feature vector, associating the visual feature vector with the corresponding temporary tracking marker of the target region image, and storing it in the target feature library for subsequent cross-modal matching and trajectory association.

[0102] The process involves generating control commands for the actuators based on the task focus target, driving the robot to adjust its motion state. This includes the following steps: extracting the center pixel coordinates of the bounding box of the task focus target; calculating the actual distance by taking the median effective depth of the neighborhood of the center pixel coordinates; calculating the horizontal deviation between the target and the image center, and the distance deviation from the preset following distance; inputting the distance deviation into the PID controller to calculate the forward speed; combining a dead-zone control strategy, setting the corresponding speed to zero when the horizontal deviation is less than the rotational or lateral dead zone to avoid system motion jitter; then smoothing the three-axis speeds using exponential smoothing filtering to limit the maximum forward and rotational speeds and ensure motion stability; and finally, converting the calculated three-axis speed parameters into a standard command format based on the constructed quadruped robot motion control command set, and starting an independent quadruped robot control thread.

[0103] Furthermore, based on the completed quadruped robot motion control instruction set, the calculated three-axis velocity parameters are converted into a standard instruction format, and an independent quadruped robot control thread is started. This includes the following steps: packaging the motion control instruction set into JSON format and sending it to the quadruped robot's drive unit via UDP protocol; and automatically switching between standing and moving states based on the state machine to complete the entire physical closed loop from natural language instruction parsing, target locking, deviation calculation to quadruped robot motion execution, thereby realizing intelligent target tracking of the quadruped robot driven by natural language.

[0104] In one embodiment, the process of constructing an open-vocabulary target tracking and semantic interaction system for robots includes:

[0105] Step 1, Construction of the multimodal sensing preprocessing module:

[0106] This embodiment uses an RGB-D depth camera mounted on the head of a quadruped robot as the perception front end to simultaneously acquire RGB images and depth point cloud data. Through camera intrinsic parameter calibration and spatiotemporal synchronization mechanisms, the RGB images and depth maps are aligned in pixel space, and millisecond-level time synchronization is achieved based on hardware timestamps to eliminate invalid data such as misalignment and missing frames, generating a standardized RGB-D paired frame sequence.

[0107] The system starts an independent Transmission Control Protocol (TCP) receive thread on the edge computing node, listening to the video stream sent by the quadruped robot through a fixed port. It employs a header identifier parsing combined with a precise length reading mechanism: parsing the packet header's 0x01 for color frames, 0x02 for depth frames, and 0x03 for depth metadata identifiers, reading the frame length according to a fixed format, and completely receiving JPEG-encoded color frames and 16-bit lossless encoded depth frames. Using the OpenCV vision library, it performs JPEG decoding on the color frames and 16-bit grayscale decoding on the depth frames. It then performs brightness, contrast, and distortion checks on the decoded images, filtering out invalid frames that are too dark, overexposed, or distorted, and outputting a multimodal frame sequence with uniform resolution and continuous temporal order, directly used for subsequent model inference and tracking algorithms.

[0108] Step 2: Construct an open vocabulary target detection and semantic extraction module based on a visual language model. In specific implementation, as shown in Figure 2, a heterogeneous dual-rate inference architecture is adopted. The detection frame and the prediction frame are executed alternately to balance the semantic accuracy and real-time requirements of the open vocabulary.

[0109] (1) Heterogeneous dual-rate mechanism control

[0110] Let the detection period parameter be T. In the main loop, a full inference of the visual language model is triggered every T frames to generate a fine-grained description of the scene; in non-detection frames, the invocation of the visual language model is paused, the semantic results of the previous detection period are reused, and the trajectory is continued only through motion prediction, reducing the computational load by about 60%.

[0111] (2) Open vocabulary detection and semantic structured output

[0112] For each detection frame, the RGB image and task prompt are input into the fine-tuned Florence-2 visual language model, which outputs detection results including the target bounding box (Bbox), semantic label (Label), and confidence score. The post-processing workflow includes:

[0113] Detection boxes whose width is less than the minimum box width threshold or whose height is less than the minimum box height threshold are identified as invalid boxes and are removed.

[0114] Redundant bounding boxes are removed by nonmaximum suppression, and pedestrian targets with a confidence level ≥ 0.6 are retained;

[0115] For each target region, the image is cropped, and the image description task of the visual language model is invoked to generate a fine-grained semantic description, forming a structured output of {Bbox, Label, Confidence, Description}.

[0116] (1) Kalman filter trajectory prediction

[0117] In non-detection frames, the system does not invoke the visual language model, but directly reuses the prediction results output by the tracker. A Kalman filter is activated based on a uniform motion model to predict the center coordinates and dimensions of the target in the current frame, maintaining trajectory continuity, based on the target's position and velocity from the previous moment.

[0118] (4) Construction and embedding alignment of multimodal feature library

[0119] The target region image is cropped based on the detection box and input into a pre-trained visual encoder to generate a 512-dimensional normalized visual feature vector.

[0120] Receive natural language commands from the interactive interface and input them into a pre-trained text encoder to generate normalized text feature vectors;

[0121] Establish a mapping relationship between target identifiers and visual features, and store it in the target feature library for subsequent cross-modal matching and trajectory association.

[0122] In non-detection frames, to reduce computational load, the system suspends inference from the large visual-language model and instead relies entirely on the Kalman filter within the tracker to maintain the trajectory. This process, based on a uniform motion model, recursively estimates the target's state vector and includes the following prediction and update steps:

[0123] 1) Define the state vector: The system defines the state vector of the target. It is an 8-dimensional vector containing the center point coordinates, aspect ratio, height, and their corresponding rates of change of the target in the image coordinate system:

[0124]

[0125] in, The center pixel coordinates of the target bounding box. Aspect ratio, For height; This is the velocity estimate for the corresponding component.

[0126] 2) State prediction (time update): Using the state transition matrix to update the state from the previous time step. The posterior state estimate is extrapolated to the current time step. The prior state. Assume a time interval. For one frame period, the system uses a constant-rate model to construct the state transition matrix:

[0127]

[0128] in It is an identity matrix. Predicted state. With predicted covariance The calculation formula is as follows:

[0129]

[0130]

[0131] Where F is the state transition matrix, This is the process noise covariance matrix, used to characterize the uncertainty of the target motion.

[0132] 3) Observation and correction: When entering the detection frame, the bounding box output by the visual language model is used as the observation value. The system uses the observation matrix. Mapping the state space to the observation space:

[0133]

[0134] Calculate Kalman gain The residual between the observed and predicted values ​​is used to correct the state, thereby obtaining the optimal posterior estimate for the current frame. :

[0135]

[0136]

[0137] in The noise covariance matrix is ​​observed to reflect the positioning error of the detection box.

[0138] 4) Based on the predicted state vector The position and size components are converted into the predicted center coordinates and predicted bounding box width and height of the target in the current frame, which are used as the target prediction result for the current frame.

[0139] 5) The predicted bounding box output by the Kalman filter is associated and matched with the visual semantic features in the target feature library to correct the motion trajectory, so that a stable and continuous tracking trajectory can be maintained even without new detection results.

[0140] Step 3: Construct a hierarchical trajectory memory module and identity recovery mechanism. As shown in Figure 3, create two ordered dictionaries: active trajectories and historical trajectories. Set the maximum number of historical trajectories to 50 and the lifespan to 500 frames, and maintain a global identifier mapping table to ensure identity consistency across frames.

[0141] The system monitors the tracking status in real time. If a target is not detected for 50 consecutive frames, it is determined that the target is lost. It is removed from the active trajectory cache and moved to the historical trajectory cache, retaining the most recent motion state and visual semantic features. At the same time, the expiration time is marked as the current frame number + TTL.

[0142] When a new target is detected, its two-layer similarity with the historical trajectory is calculated. At the spatial continuity level, the intersection-union ratio of the predicted box and the new detection box is calculated. At the visual semantic consistency level, the cosine similarity between the features of the new target and the historical features is calculated. Finally, a weighted summation strategy is used to determine whether the target is a candidate match.

[0143] If the IoU value is greater than 0.5 and the feature similarity is greater than 0.6, the new target is determined to be the original target that was lost and then reappeared. The global fixed identifier of the historical trajectory is assigned to the new target, the trajectory is moved back to the active trajectory cache, the trajectory state is updated with the new detection box and features, the lost frame count is reset, the historical motion model parameters are inherited, and a continuous trajectory spanning the occlusion period is formed.

[0144] The system periodically traverses the historical trajectory cache, automatically deleting expired trajectories that have exceeded their TTL (Time To Live) and releasing storage resources.

[0145] Step 4: Construct a closed-loop module for natural language interaction and control.

[0146] As shown in Figure 4, a visual user interface is first built to monitor the process and display the results. A 640×480 pixel video canvas is set on the left side of the interface to display the tracking screen and target box in real time. The right side includes a system status log bar, a target list window, and an instruction input box, which allows operators to input natural language instructions to specify the tracking target.

[0147] The system pre-builds a standardized quadruped robot motion control instruction set, which is defined based on three-axis motion dimensions and includes three core motion control instructions: forward, lateral translation, and rotation. It is also equipped with auxiliary control parameters such as speed threshold, dead zone parameters, and state switching instructions, forming a standardized and directly parsable and executable instruction system. When in use, the motion parameters generated by closed-loop calculation are encapsulated according to the instruction set format to ensure that the sent instructions are fully compatible with the underlying control protocol of the quadruped robot.

[0148] After the user inputs natural language commands through the interactive interface, the system generates text features from the input natural language commands using a text encoder. The system then calculates the cosine similarity between the text features and visual features in the target feature library, selecting the target with the highest similarity as the unique target to lock onto and track, and marking it with a red border on the interface. The system extracts the center pixel coordinates of the target bounding box, combines them with the RGB-D depth map, and calculates the actual distance by taking the median effective depth of the center neighborhood. The system calculates the horizontal deviation between the target and the image center, and the distance deviation from the preset following distance. The distance deviations are then input into a PID controller to calculate the forward speed. By combining a dead-zone control strategy, the corresponding velocity is set to zero when the horizontal deviation is less than the rotational or lateral dead zone to avoid system motion jitter. Then, exponential smoothing filtering is used to smooth the three-axis velocities, limiting the maximum forward and rotational speeds to ensure motion stability. Based on the constructed quadruped robot motion control instruction set, the system converts the calculated three-axis velocity parameters into a standard instruction format and starts an independent quadruped robot control thread.

[0149] Standardized motion commands are packaged into JSON format and sent to the quadruped robot's drive unit via UDP protocol. Based on the state machine, the quadruped robot automatically switches between standing and moving states, completing the entire physical closed loop from natural language command parsing, target locking, deviation calculation to quadruped robot motion execution, thus realizing intelligent target tracking of the quadruped robot driven by natural language.

[0150] The system of this invention was tested on a quadruped robot platform equipped with edge computing nodes, and the results show that:

[0151] (1) In terms of open vocabulary detection, the detection accuracy rate for natural language description targets such as "woman wearing a red top" reaches 87% in typical indoor scenarios, breaking through the dependence of traditional methods on preset categories;

[0152] (2) In terms of anti-occlusion performance, through the hierarchical trajectory memory and multi-factor matching mechanism, the success rate of identity recovery after the target is briefly occluded for 3 to 8 seconds reaches 91%, which effectively suppresses the switching of identifiers;

[0153] (3) In terms of real-time performance, under the condition of limited computing power, the average processing frame rate of the system is maintained at 8 to 10 FPS through the heterogeneous dual-rate inference architecture, which balances the semantic accuracy and motion control requirements.

[0154] (4) In terms of control accuracy, the distance deviation is controlled within ±0.35m at a following distance of 1.2m to 2m, and the horizontal angle deviation is ≤ 6%, which meets the requirements of the basic following task of the quadruped robot.

[0155] In summary, this invention verifies the feasibility of implementing open vocabulary semantic tracking on a low-computing-power platform, and provides an effective solution for natural language-driven robot perception and control.

[0156] In one embodiment, a quadruped robot is equipped with a target tracking and semantic interaction system, which applies the open-vocabulary target tracking and semantic interaction method for robots described above.

[0157] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0158] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0159] Furthermore, one embodiment of the present invention provides a computer-readable storage medium storing computer-executable instructions that are executed by a processor or controller, for example, by a processor in the above-described terminal embodiment, such that the processor performs the robot-oriented open vocabulary target tracking and semantic interaction method described in the above-described embodiment.

[0160] It will be understood by those skilled in the art that all or some of the steps and systems in the methods disclosed above can be implemented as software, firmware, hardware, and suitable combinations thereof. Some or all of the physical components can be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable medium, which can include computer storage media (or non-transitory media) and communication media (or transient media). As is known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, as is known to those skilled in the art, communication media typically contain computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information transmission medium.

[0161] The above is a detailed description of the preferred embodiments of the present invention. However, the present invention is not limited to the above embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention. All such equivalent modifications or substitutions are included within the scope defined by the claims of the present invention.

[0162] The specific embodiments of the present invention described above do not constitute a limitation on the scope of protection of the present invention. Any other corresponding changes and modifications made in accordance with the technical concept of the present invention should be included within the scope of protection of the claims of the present invention.

Claims

1. An open-lexical target tracking and semantic interaction method for robots, characterized in that, Including the following steps: Collect multimodal perception data of the environment, including RGB image data and depth image data, synchronize and align the RGB image data and the depth image data to generate an image frame sequence that can be used for model inference; The image frame sequence is divided into detection frames and non-detection frames by setting a detection period. In the detection frame, the RGB image and the task prompt are input into the fine-tuned visual language model to generate structured target state information. In non-detection frames, the Kalman filter prediction process is initiated based on the target's historical motion state to generate Kalman filter prediction results; Based on the target state information, the target region image is cropped from the original image, and a normalized visual feature vector is generated based on the target region image. A feature mapping relationship is established, and the visual feature vector is associated with the corresponding temporary tracking identifier of the target region image to construct a target feature library. A hierarchical trajectory memory module and identity recovery strategy are constructed to achieve identity consistency in long-term tracking of historical targets; The system receives natural language input, converts the natural language input into a text semantic feature vector through the text encoding branch of the visual language model, calculates the cross-modal similarity between the text semantic feature vector and the visual feature vectors of each instance in the target feature library, determines the task focus target that matches the description of the natural language input, and generates control instructions for the actuator based on the task focus target to drive the robot to adjust its motion state.

2. The open-vocabulary target tracking and semantic interaction method for robots according to claim 1, characterized in that, Constructing a hierarchical trajectory memory module and identity recovery strategy to achieve identity consistency during long-term tracking of historical targets includes the following steps: Create two ordered dictionaries, active trajectory and historical trajectory, set the maximum number of historical trajectories and the survival time, and maintain a global identifier mapping table to ensure identity consistency across frames; The system monitors the tracking status in real time. If a target is not detected for a set number of consecutive frames, it is determined that the target is lost. The current target is removed from the active trajectory cache and moved to the historical trajectory cache, retaining the most recent motion state and visual semantic features. At the same time, the expiration time is marked as the current frame number + the set value. When a new target is detected, two levels of similarity are calculated between the new target and the historical trajectory: at the spatial continuity level, the intersection-union ratio of the predicted box and the new detection box is calculated; at the visual semantic consistency level, the cosine similarity between the features of the new target and the historical features is calculated. Finally, a weighted summation strategy is used to determine whether the new target matches the candidate.

3. The open-vocabulary target tracking and semantic interaction method for robots according to claim 2, characterized in that, The weighted summation strategy is used to determine whether a new target matches a candidate, including the following steps: The intersection-union ratio (IoU) is calculated based on a weighted summation strategy. If the IoU value is greater than 0.5 and the feature similarity is greater than 0.6, the new target is determined to be the original target that was lost and then reappeared. The global fixed identifier of the historical trajectory is assigned to the new target, the trajectory is moved back to the active trajectory cache, the trajectory state is updated with the new detection box and features, the lost frame count is reset, the parameters of the historical motion model are inherited, and a continuous trajectory spanning the occlusion period is formed.

4. The open-vocabulary target tracking and semantic interaction method for robots according to claim 1, characterized in that, The process of synchronizing and aligning the RGB image data with the depth image data to generate an image frame sequence suitable for model inference includes the following steps: Through camera intrinsic parameter calibration and spatiotemporal synchronization mechanism, the RGB image data and the depth image data are aligned in pixel space, and millisecond-level time synchronization is achieved based on hardware timestamps to eliminate misaligned, dropped, and invalid data, forming standardized video stream data; The video stream data is decoded and its quality is assessed to generate an image frame sequence that can be used for model inference.

5. The open-vocabulary target tracking and semantic interaction method for robots according to claim 1, characterized in that, The image frame sequence is divided into detection frames and non-detection frames by setting a detection period, including the following steps: Let the detection period parameter T be. In the main loop, the frame counter collects one frame every T frames as a detection frame, triggering the full visual language model inference. The image frame sequence other than the detection frame is set as a non-detection frame. The visual language model is paused in the non-detection frame, the semantic results of the previous detection period are reused, and the trajectory is continued only through motion prediction.

6. The open-vocabulary target tracking and semantic interaction method for robots according to claim 5, characterized in that, In the detection frame, the RGB image and the task prompt are input into the fine-tuned visual language model to generate structured target state information, including the following steps: The RGB image and task prompt are input into the fine-tuned Florence-2 visual language model, and the output includes the target bounding box, semantic label and confidence score. In non-detection frames, a Kalman filter prediction process is initiated based on the target's historical motion state to generate Kalman filter prediction results, including the following steps: The Kalman filter is activated based on the uniform motion model. Based on the target's position and velocity at the previous moment, the center coordinates and width and height of the target in the current frame are predicted as the Kalman filter prediction result to maintain the continuity of the trajectory.

7. The open-vocabulary target tracking and semantic interaction method for robots according to claim 6, characterized in that, Based on the target state information, a target region image is cropped from the original image, and a normalized visual feature vector is generated based on the target region image. A feature mapping relationship is established, and the visual feature vector is associated with the corresponding temporary tracking identifier of the target region image to construct a target feature library, including the following steps: The target region image is cropped based on the target bounding box and input into a pre-trained visual encoder to generate a 512-dimensional normalized visual feature vector. Receive natural language commands from the interactive interface and input them into a pre-trained text encoder to generate normalized text feature vectors; Establish a mapping relationship between text feature vectors and visual feature vectors, associate visual feature vectors with corresponding temporary tracking labels of target region images, and store them in the target feature library for subsequent cross-modal matching and trajectory association.

8. The open-vocabulary target tracking and semantic interaction method for robots according to claim 7, characterized in that, Based on the task focus objective, control commands are generated for the actuator to drive the robot to adjust its motion state, including the following steps: Extract the center pixel coordinates of the bounding box of the target focus of the task, take the median effective depth of the neighborhood of the center pixel coordinates to calculate the actual distance, calculate the horizontal deviation between the target and the image center and the distance deviation from the preset following distance, input the distance deviation into the PID controller, and calculate the forward speed. By combining dead zone control strategy, when the horizontal deviation is less than the rotational dead zone or the lateral dead zone, the corresponding speed is set to zero to avoid system motion jitter. Then, the three-axis speed is smoothed by exponential smoothing filter to limit the maximum forward speed and rotational speed and ensure motion stability. Based on the completed quadruped robot motion control instruction set, the calculated three-axis velocity parameters are converted into standard instruction format, and an independent quadruped robot control thread is started.

9. The open-vocabulary target tracking and semantic interaction method for robots according to claim 8, characterized in that, Based on the completed quadruped robot motion control instruction set, the calculated three-axis velocity parameters are converted into a standard instruction format, and an independent quadruped robot control thread is started, including the following steps: The motion control instruction set is packaged into JSON format and sent to the drive unit of the quadruped robot via UDP protocol; Based on the state machine's automatic switching between standing and moving states, a complete physical closed loop is completed, from natural language command parsing, target locking, deviation calculation to quadruped robot motion execution, enabling intelligent target tracking of quadruped robots driven by natural language.

10. A quadruped robot, characterized in that, A target tracking and semantic interaction system is provided, which applies the open vocabulary target tracking and semantic interaction method for robots as described in any one of claims 1-9.

Images