Multi-robot visual pursuit method and system for unknown target

Abstract

The application belongs to the field of multi-robot cooperative navigation, and provides a multi-robot visual pursuit method and system for unknown targets, which comprises the following steps: performing semantic analysis according to a first visual angle image of a current frame of each pursuit robot and a pursuit instruction, determining a target to be pursued and an initial target bounding box; based on the initial target bounding box and a plurality of continuous frames of target images for tracking the target to be pursued, a pseudo video sequence is formed, the pseudo video sequence is segmented, and a tracking result is obtained; based on the first visual angle image and the tracking result, a target relative position of the target to be pursued relative to each pursuit robot is obtained; a state of the target to be pursued in the current frame is predicted, and the target relative position is corrected to obtain a target state estimation result; a local observation state is constructed based on the target state estimation result, and a pursuit control instruction is generated according to the local observation state. The application improves the pursuit success rate, real-time response capability and practical application value in a complex environment.

Description

Technical Field

[0001] This invention belongs to the field of multi-robot cooperative navigation technology, specifically relating to a multi-robot visual tracking method and system for unknown targets. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] Multi-robot pursuit involves target recognition, tracking, state estimation, and cooperative control, representing a deeply coupled perception-decision-control problem. Existing methods mainly fall into two categories. The first category is based on model-based, geometric rule-based, or optimization control methods, such as differential game theory, Voronoi partitioning, and model predictive control. These methods assume that the target state is known accurately in real time and that observations are sufficient. However, in reality, the initial position of the target is unknown, and relying solely on local sensors is susceptible to interference, occlusion, and time delays, making them difficult to apply. The second category is based on deep reinforcement learning or end-to-end visual policy learning methods, which directly map images to control actions and possess a certain degree of adaptability. However, the learning targets are complex, training is unstable, state representation is lacking, interpretability is poor, and it is difficult to locate failure points. Furthermore, the simulated strategies are severely degraded in real systems due to perception delays, limited field of view, and chassis constraints, making transfer difficult.

[0004] Practical pursuit and capture also requires solving problems such as locking onto targets among interfering targets based on natural language commands, converting image detection into measurable relative positions, compensating for visual latency, and meeting differential chassis and collision avoidance constraints. Existing technologies mostly focus on single aspects and lack an overall solution that organically integrates language disambiguation, visual tracking, state estimation, latency compensation, and cooperative control, thus limiting pursuit and capture performance in complex environments. Summary of the Invention

[0005] To address the aforementioned issues, this invention proposes a multi-robot visual tracking method and system for unknown targets. This invention can determine the target to be tracked based on natural language instructions, continuously track the target using onboard visual information and estimate its relative state online. Based on this, it generates collaborative tracking control instructions suitable for execution by real robots, thereby improving the tracking success rate, real-time response capability, and practical application value in complex environments.

[0006] According to some embodiments, the first aspect of the present invention provides a multi-robot visual tracking method for unknown targets, employing the following technical solution: Multi-robot visual tracking methods for unknown targets include: Semantic analysis is performed on the first-view image of each pursuit robot in the current frame and the pursuit command to determine the target to be pursued and the initial target bounding box. A pseudo-video sequence is formed by combining the initial target bounding box with several consecutive frames of target images of the target to be tracked online. The pseudo-video sequence is then segmented using a video target segmentation model to obtain the tracking result of the current frame. Based on the first-person view images and tracking results, the target to be pursued is transformed from the image space to the robot chassis coordinate system to obtain the target relative to each pursuit robot. The state of the target to be tracked in the current frame is predicted to obtain the predicted position and velocity of the current frame. The prediction is then corrected based on the relative position of the target to obtain the target state estimation result. Based on the target state estimation results, the current state of the pursuit robot itself, and the states of other pursuit robots, a local observation state is constructed, and pursuit control commands for the current pursuit robot are generated based on the local observation state. The pursuit control commands are converted into short-time domain reference target points, and path planning is performed on the short-time domain reference target points to obtain the underlying control variables of the pursuit robot at the current moment.

[0007] Furthermore, the step of performing semantic analysis based on the first-view image of each pursuit robot in the current frame and the pursuit command to determine the target to be pursued and the initial target bounding box includes: Acquire first-person view images and pursuit commands for each pursuit robot in the current frame; A visual big language model is used to perform joint semantic analysis on first-person view images and pursuit instructions to determine the semantic correspondence between pursuit instructions and candidate targets in first-person view images, and to obtain the matching score of each candidate target and pursuit instruction. The candidate target with the highest matching score is selected as the target to be pursued, and the initial target bounding box of the target to be pursued is extracted using a visual large language model.

[0008] Furthermore, a pseudo-video sequence is formed based on the initial target bounding box and several consecutive frames of target images of the target to be tracked online. The pseudo-video sequence is then segmented using a video target segmentation model to obtain the tracking result of the current frame, including: Following the chronological order, using the initial target bounding box as a visual cue, the target images of the current frame and several consecutive frames preceding it are organized to form a pseudo-video sequence; Based on pseudo-video sequences, a pre-trained video target segmentation model is used for segmentation to obtain the target mask, target bounding box, and reliability score of the current frame, which are used as the tracking results.

[0009] Furthermore, the step of transforming the target to be pursued from image space to robot chassis coordinate system based on first-view images and tracking results to obtain the target's relative position with respect to each pursuing robot includes: Target pixels are selected based on the target bounding box in the tracking results, and the coordinates of the target pixels in the camera coordinate system are determined by combining the depth information corresponding to the first-view image and the camera calibration parameters. Based on the rotation matrix and translation vector from the camera coordinate system to the robot chassis coordinate system, and the coordinates of the target pixel in the camera coordinate system, the coordinates of the target pixel in the robot chassis coordinate system are determined, which serve as the target relative position of the target to be pursued relative to each pursuing robot.

[0010] Furthermore, the target state in the current frame is predicted to obtain the predicted position and velocity of the current frame, and then corrected according to the target's relative position to obtain the target state estimation result, including: Based on the target position and target velocity estimates of the previous frame, the state of the target to be pursued in the current frame is predicted to obtain the predicted position and predicted velocity of the current frame. The predicted position and velocity of the current frame are corrected based on the relative position of the target to obtain the updated predicted position and velocity, which are used as the target state prediction result.

[0011] Furthermore, the process of converting pursuit control commands into short-time domain reference target points, performing path planning on the short-time domain reference target points, and obtaining the underlying control variables of the pursuit robot at the current moment includes: Based on the pursuit control command and the current position information of the pursuit robot, the short-time domain reference target point is obtained by integral calculation according to the preset look-ahead time. Based on the short-time domain reference target point, combined with the current pose information of the current pursuit robot, environmental obstacle information, and the position information of other pursuit robots, a local path planning algorithm is used to plan the path and obtain the optimal speed control quantity. The optimal speed control value is used as the underlying control value of the pursuit robot at the current moment.

[0012] According to some embodiments, the second aspect of the present invention provides a multi-robot visual tracking system for unknown targets, employing the following technical solution: A multi-robot visual tracking system for unknown targets, including: The target initialization module is used to perform semantic analysis based on the first-view image of each pursuit robot in the current frame and the pursuit command to determine the target to be pursued and the initial target bounding box. The visual tracking module is used to form a pseudo-video sequence based on the initial target bounding box and several consecutive frames of target images of the target to be tracked online. The pseudo-video sequence is then segmented using a video target segmentation model to obtain the tracking result of the current frame. The state estimation module is used to transform the target to be pursued from the image space to the robot chassis coordinate system based on the first-view image and the tracking results, so as to obtain the target relative position of the target to be pursued with respect to each pursuit robot. The delay compensation module is used to predict the state of the target to be pursued in the current frame, obtain the predicted position and predicted velocity of the current frame, and correct it according to the relative position of the target to obtain the target state estimation result. The collaborative pursuit control module is used to construct a local observation state based on the target state estimation results and the target's relative position, and to generate pursuit control commands for the current pursuit robot based on the local observation state. The navigation control module is used to convert pursuit control commands into short-time domain reference target points, perform path planning on the short-time domain reference target points, and obtain the underlying control variables of the pursuit robot at the current moment.

[0013] According to some embodiments, a third aspect of the present invention provides a computer-readable storage medium.

[0014] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the multi-robot visual tracking method for unknown targets as described in the first embodiment above.

[0015] According to some embodiments, a fourth aspect of the present invention provides a computer device.

[0016] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the multi-robot visual tracking method for unknown targets as described in the first embodiment above.

[0017] According to some embodiments, a fifth aspect of the present invention provides a computer program product or computer program.

[0018] A computer program product or computer program includes computer instructions stored in a computer-readable storage medium, wherein a processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the steps in the multi-robot visual tracking method for unknown targets as described in the first embodiment above.

[0019] Compared with the prior art, the beneficial effects of the present invention are as follows: This invention addresses two-dimensional or near-ground plane environments containing obstacles, occlusions, and interfering targets. Multiple pursuit robots and at least one target are deployed in this environment. At the start of the pursuit, the target's initial position is not directly observable by the robots. Each robot acquires RGB images, depth images, and its own pose information via onboard visual sensors and determines the target based on natural language commands. Subsequently, through continuous target tracking, geometric state estimation, time delay compensation, and multi-robot cooperative control, the spatial distance between the robot and the target is gradually reduced, forming a cooperative pursuit posture. Ultimately, stable tracking and effective capture of the target are achieved. This invention forms an integrated methodological architecture consisting of target initialization via language commands, visual tracking, geometric state estimation, time delay compensation, multi-robot cooperative pursuit control, and navigation execution interface, thus organically combining target designation, perception, estimation, decision-making, and actual execution.

[0020] To address the shortcomings of existing end-to-end visual tracking methods, such as weak interpretability, high training difficulty, and difficulty in troubleshooting perception and control faults, this invention proposes a perception-estimation link consisting of visual-language initialization, video target tracking, depth-assisted geometric mapping, and time-delay-aware filtering. This method progressively transforms target observations in image space into target relative states with clear physical meaning, enabling control decisions to be based on structured state inputs. This improves the system's interpretability, module debuggability, and engineering implementation capabilities in complex scenarios. Attached Figure Description

[0021] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0022] Figure 1 This is a flowchart of a multi-robot visual tracking method for unknown targets in an embodiment of the present invention. Detailed Implementation

[0023] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0024] It should be noted that the following detailed description is illustrative and intended to provide further explanation of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0025] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0026] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0027] As mentioned in the background section, unlike single-robot target tracking, multi-robot pursuit not only requires the system to detect and lock onto the target in a complex environment, but also requires multiple robots to form a cooperative encirclement based on the target's motion state and spatial relationships, thereby continuously compressing the target's feasible escape space and ultimately achieving effective pursuit. Therefore, this type of task involves multiple stages, including target recognition, target tracking, state estimation, multi-robot cooperative decision-making, and real robot execution control, making it a typical problem of deep coupling of perception, decision-making, and control.

[0028] The first type of method typically establishes a pursuit strategy under the conditions that the target state is known, observations are sufficient, obstacle structures are regular, or global information is available, thus possessing a certain theoretical foundation and provability. However, this type of method often relies on strong idealized assumptions, such as that the target's position, velocity, or orientation can be measured accurately in real time, or that each pursuit robot can continuously obtain relatively complete global state information. In practical applications, these assumptions are often difficult to hold: the target may start moving from an unknown position; the robot can only rely on local sensors such as onboard cameras and depth cameras for observation; there may be interfering objects in the scene that resemble the target in appearance; obstacle occlusion, robot rotation, and sensor processing delays can all lead to unstable target state acquisition, making it difficult to directly apply traditional pursuit methods based on ideal state input to real systems.

[0029] The second type of method reduces the manual modeling process and has the potential for end-to-end learning. However, in multi-robot pursuit scenarios, end-to-end vision methods still generally have significant limitations. First, the policy network needs to simultaneously complete multiple sub-tasks such as target recognition, target disambiguation, temporal tracking, relative localization, and cooperative control. The learning objectives are complex, the training cost is high, and slow convergence and training instability are common problems. Second, there is a lack of clear intermediate state representation between images and actions, resulting in weak system interpretability. When target loss, localization deviation, cooperative failure, or control anomalies occur, it is difficult to quickly determine whether the problem originates from the perception, estimation, or control module, which is detrimental to engineering debugging and reliable deployment. Third, real robot platforms generally suffer from problems such as perception latency, limited field of view, severe occlusion, unstable communication, and chassis kinematic constraints. This often leads to significant performance degradation of end-to-end policies trained in simulation environments on real systems, making simulation-to-real transfer difficult.

[0030] Furthermore, in actual pursuit missions, merely "seeing the target" does not equate to achieving stable pursuit. For multi-robot systems, the following key issues need to be addressed: First, how to identify the true target to be pursued from multiple candidate targets with similar colors or appearances based on natural language or task instructions; second, how to convert the target bounding box or segmentation results in image space into a quantifiable relative positional state that can be used for control decisions; third, how to compensate for and predict the target state when there is a time delay in visual processing to avoid control decision lag; and fourth, how to ensure that the multi-robot control strategy maintains cooperative pursuit capabilities while being compatible with real robot navigation and control systems, thereby meeting practical constraints such as differential chassis, obstacle avoidance, and safe distances. Existing technologies typically only improve one aspect of these processes, lacking a holistic technical solution that organically combines target language disambiguation, online visual tracking, target state estimation, time delay compensation, and multi-robot cooperative pursuit control.

[0031] Therefore, this invention proposes a language-guided vision-based multi-robot cooperative pursuit method for unknown targets. This method is designed for two-dimensional or near-ground plane environments with obstacles, occlusions, and interfering targets. It forms an integrated method architecture consisting of "language command target initialization—visual tracking—geometric state estimation—time delay compensation—multi-robot cooperative pursuit control—navigation execution interface," thus organically combining target designation, perception, estimation, decision-making, and actual execution.

[0032] Example 1 like Figure 1As shown, this embodiment provides a multi-robot visual tracking method for unknown targets. This embodiment uses the application of this method to a server as an example for illustration. It is understood that this method can also be applied to terminals, and can also be applied to systems including terminals, servers, and other components, and is implemented through interaction between the terminal and the server. The server can be an independent physical server, a server cluster composed of multiple physical servers, or a distributed system. It can also be a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network servers, cloud communication, middleware services, domain name services, CDN security services, and big data and artificial intelligence platforms. The terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, etc., but is not limited to these. The terminal and server can be directly or indirectly connected via wired or wireless communication, which is not limited herein. In this embodiment, the method includes the following steps: Step S1: Perform semantic analysis based on the first-view image of each pursuit robot in the current frame and the pursuit command to determine the target to be pursued and the initial target bounding box, including: Step S1.1: Obtain the first-person view image of each pursuit robot in the current frame and the pursuit command; Among them, the pursuit command is a natural language command used to describe the category and local appearance features of the target to be pursued. It is preferably a text phrase composed of category words and appearance attribute words, such as "Blue car". First-person view images refer to first-person view images captured by the pursuit robot in the current frame and from the current viewpoint. Step S1.2: Use a visual big language model to perform joint semantic analysis on the first-view image and the pursuit command, determine the semantic correspondence between the pursuit command and the candidate targets in the first-view image, and obtain the matching score of each candidate target and the pursuit command. Specifically, the first-person view image captured at the current moment and the pursuit command are input into the visual big language model. The visual big language model is used to perform cross-modal alignment of text semantics and image region features. Based on the attention mechanism, the matching relationship between the text and the candidate target region is calculated to establish the semantic correspondence between the command description and the candidate target in the image. Multiple candidate targets are matched and discriminated. The joint analysis process of text and image in the visual large language model is as follows: First, the pursuit command is input into the text encoding module of the visual big language model for encoding, generating a text semantic feature vector; Simultaneously, the first-view image is input into the visual encoding module of the visual big language model to extract the features of each candidate target and its corresponding candidate target region in the first-view image; Subsequently, in the multimodal fusion module of the visual large language model, a cross-modal attention mechanism is adopted to enable the text semantic feature vector and the candidate target region features to interact and fuse, thereby achieving semantic alignment between text information and image region information. Finally, by calculating the matching score between the text semantic feature vector and the features of each candidate target region, the candidate target with the highest score is selected as the target to be hunted, and its initial target bounding box is output.

[0033] Preferably, the visual large language model can adopt, but is not limited to, the Qwen3-VL model.

[0034] Step S1.3: Select the candidate target with the highest matching score as the target to be pursued, and use the visual big language model to extract the initial target bounding box of the target to be pursued.

[0035] After identifying the target object, the visual-language model further outputs the target's location in the first frame image, preferably an initial target bounding box. The initial target bounding box is output by the visual-language model during joint semantic analysis and takes the form of a rectangular region of the target in the image coordinate system.

[0036] Subsequently, the coordinates of the center point are calculated based on the geometric center of the initial target bounding box, which are used as the pixel sampling positions of the subsequent 3D target points extracted in the image space for subsequent visual tracking and geometric state estimation. The model outputs the initial target bounding box in the first frame image, and uses the coordinates of the center point of the initial target bounding box as the target image position for subsequent visual tracking and state estimation.

[0037] Through the above processing, this embodiment can complete target semantic understanding, candidate target disambiguation, and initial position determination based on natural language instructions at the beginning of the pursuit task. This allows for the identification of the target to be pursued in complex scenarios with multiple similar targets or interfering objects, and establishes a unified target reference for subsequent visual tracking and collaborative control. Addressing the problem of existing multi-robot pursuit methods struggling to accurately determine the target in complex scenarios, this embodiment proposes a target initialization method based on a combination of natural language instructions and a vision-language model. This method can accurately identify the target to be pursued among multiple robots or interfering targets with similar appearances, and output a unified initial target localization result. This solves the problem of unstable target determination in traditional pursuit systems, improving the consistency, accuracy, and reliability of target designation.

[0038] Step S2: Based on the initial target bounding box and several consecutive frames of target images of the target to be tracked online, a pseudo-video sequence is formed. The pseudo-video sequence is segmented using a video target segmentation model to obtain the tracking result of the current frame, including: Step S2.1: Following the time sequence, using the initial target bounding box as a visual cue, organize the target images of the current frame and several consecutive frames before it to form a pseudo-video sequence for online tracking of the target to be tracked in the current frame; This embodiment does not directly input a long video file. Instead, the pursuit robot continuously acquires several frames of target images of the target to be pursued, and organizes the first cue image and the subsequent consecutive target images into a pseudo-video sequence in chronological order, which serves as the input to the video target segmentation model. The construction of the pseudo-video sequence is only used for online visual tracking of the target in the current frame. Its input consists of the current frame and several consecutive frames preceding it, and does not involve information from future frames. The purpose of this step is to enhance the temporal consistency of target tracking using short temporal images, preparing for the subsequent acquisition of the target bounding box.

[0039] The initial bounding box of the target is obtained as a visual cue. In subsequent moments, the pursuit robot continuously acquires image frames and organizes them into a short image sequence in chronological order. This image sequence is then used as a pseudo-video sequence and input into the video target segmentation model for continuous target tracking.

[0040] Step S2.2: Based on the pseudo-video sequence, segment the target using a pre-trained video target segmentation model to obtain the target mask, target bounding box, and reliability score of the current frame, which are used as the tracking result.

[0041] Specifically, the pseudo-video sequence is input into the video target segmentation model, which establishes the temporal correlation of targets on the pseudo-video sequence and outputs the corresponding target mask, target bounding box, and reliability score for each frame in the pseudo-video sequence. In actual online tracking, only the output result of the frame corresponding to the current moment is used as the target tracking result for the current moment. In this way, the temporal continuity of the target tracking result can be maintained even when the target appearance changes or there is short-term local occlusion.

[0042] It is important to note that in this step, the output of the video object segmentation model is the target mask, target bounding box, and reliability score of the frame corresponding to the current time step. This means that the video object segmentation model outputs the corresponding target mask, target bounding box, and reliability score for each frame in the pseudo-video sequence; however, during online tracking, only the output of the frame corresponding to the current time step is used as the target tracking result for that time step. Therefore, this step describes the frame-by-frame inference output of the video object segmentation model for a multi-frame sequence, while the final tracking result of the frame at the current time step is actually used. The video target segmentation model may be, but is not limited to, the SAM2 model; To reduce the computational latency caused by directly using video target segmentation for closed-loop tracking, this embodiment designs a sliding window pseudo-video mechanism. Specifically, this embodiment does not process the entire historical image sequence from the start of tracking to the current time each time it updates, but only retains the first frame of the target cue and the most recent frame. Each frame image forms a short pseudo-video window for inference of the current frame. By analyzing only the first frame and the most recent k frames, rather than the entire historical image sequence, the inference burden is reduced, and the real-time performance of online tracking is improved.

[0043] The first frame is used to maintain the model's consistency constraint on who the currently tracked object is, while the most recent frame... Frame images are used to provide temporal information on the target's recent motion and appearance changes. Subsequently, the first frame and the most recent k frames are input into the video target segmentation model in chronological order to analyze the target state within this short time window, and output the target mask, target bounding box, and reliability score corresponding to the current frame. Through this sliding window processing method, this embodiment significantly reduces the inference burden caused by the accumulation of historical images, improving online tracking speed while maintaining temporal consistency, thus making it more suitable for the real-time processing requirements of closed-loop pursuit systems. For short-term tracking loss caused by robot rotation, changes in field of view, or partial occlusion, this embodiment also utilizes target position information provided by other pursuit robots that are still maintaining effective observation to generate a recommended recovery yaw angle for the current robot, thereby increasing the probability of the target re-entering the field of view and being recaptured.

[0044] Step S3: Based on the first-view image and the tracking results of the current frame, the target to be tracked is transformed from the image space to the robot chassis coordinate system to obtain the target relative to each tracking robot. Since the pursuit controller requires a quantified positional state of the target relative to the pursuit robot, and the video target segmentation model outputs target bounding boxes, target masks, and reliability scores on consecutive image frames as tracking results, these results are still in image space and cannot be directly used as input to the pursuit controller. Therefore, this embodiment uses the tracking results as input for state estimation. Target pixels are selected from the target image region corresponding to the target bounding box, and combined with the depth information and camera calibration parameters corresponding to the current first-view image of the pursuit robot, the target is transformed from image space to the robot chassis coordinate system, thereby obtaining the quantified positional state of the target relative to the pursuit robot. The state estimation step, based on the tracking results obtained at the current moment and combined with historical state information, uses recursive methods such as Kalman filtering to predict and correct the target state at the current moment to compensate for visual processing delays and measurement noise.

[0045] Step S3.1: Select target pixels based on the target bounding box in the tracking results, and determine the coordinates of the target pixels in the camera coordinate system by combining the depth information corresponding to the first-view image and the camera calibration parameters; This invention selects target pixels from the target bounding box of the tracking result based on the coordinates of the center point of the bounding box, preferably pixels near the center of the lower part of the bounding box, and combines the depth value aligned with the RGB image (first-view image) and the camera intrinsic parameter matrix. The 3D coordinates of the target pixel in the camera coordinate system are obtained through inverse projection, calculated as follows: (1); in, They represent the first The first pursuit robot and the first Frame image index, Indicates the first The first pursuit robot in the The homogeneous coordinates of the target pixel in the frame image. Indicates the first-person perspective image The first pursuit robot in the The corresponding depth value of the frame image. It is the camera intrinsic parameter matrix. Indicates the first The camera coordinate system of a pursuit robot Indicates the first The first pursuit robot in the The coordinates of the target pixel in the frame image in the camera coordinate system.

[0046] Step S3.2: Based on the rotation matrix and translation vector from the camera coordinate system to the robot chassis coordinate system, and the coordinates of the target pixel in the camera coordinate system, determine the coordinates of the target pixel in the robot chassis coordinate system, which will be used as the target relative position of the target to be pursued relative to each pursuing robot.

[0047] Specifically, based on the three-dimensional coordinates of the target pixel in the camera coordinate system, and combined with the rigid body transformation relationship between the camera coordinate system and the chassis coordinate system, the coordinates of the target to be tracked in the robot chassis coordinate system, i.e., the target's relative position, are obtained, calculated as follows: (2); in, Indicates the first The chassis coordinate system of a pursuit robot This represents the coordinates of the 3D target point in the robot chassis coordinate system, obtained by backprojection of the target pixel. and Let represent the rotation matrix and translation vector from the camera coordinate system to the robot chassis coordinate system, respectively. The relative position of the target in this frame is defined as the observation. This is used for subsequent Kalman filter state estimation.

[0048] Through the above process, the tracking results of the target to be pursued can be converted into a geometrically meaningful measurement of the target's relative position that can be directly used for control decisions.

[0049] Step S4: Predict the state of the target to be tracked in the current frame to obtain the predicted position and predicted velocity of the current frame, and correct it according to the relative position of the target to obtain the target state estimation result. To address the processing latency issue in visual tracking and geometric mapping, this invention constructs a time-delay-aware recursive estimation and compensation method for the target state, based on the obtained target's relative position. Specifically, this step further addresses the issues of time lag and noise in the target's relative position by performing recursive estimation and forward compensation of the target state.

[0050] Step S4.1: Based on the target position and target velocity estimates of the previous frame, predict the state of the target to be tracked in the current frame to obtain the predicted position and predicted velocity of the current frame. Specifically, the first The target to be tracked in the frame relative to the first frame The state of a pursuit robot is defined as a state vector. ,in Indicates the first The target to be tracked in the frame relative to the first frame The location of the tracking robot Indicates the first The target to be tracked in the frame relative to the first frame The speed of the pursuit robot. This invention uses a time-motion model to model the state of the target to be pursued, i.e. ,in, Indicates the first The predicted state of the frame, Indicates the first -1 frame target state Let be the state transition matrix, representing the uniform motion model.

[0051] Therefore, the predicted target position and velocity in the current frame are recursively derived from the state of the previous frame. Subsequently, the relative target position obtained in step S3 is used as the observation. The input Kalman filter is used for state updates.

[0052] Step S4.2: Correct the predicted position and predicted velocity of the current frame according to the relative position of the target to obtain the updated predicted position and predicted velocity, which are used as the target state prediction results.

[0053] Specifically, the residual between the predicted state and the observed state is calculated. ,in This is the observation matrix. The Kalman gain is calculated based on this residual. The predicted state is then updated to obtain the updated state. ,in It is used to fuse predicted and observed information to obtain an updated target state estimate.

[0054] Through the aforementioned prediction-correction recursive process, this invention continuously updates the target's position and velocity relative to the pursuit robot at consecutive time points. Furthermore, this invention incorporates observation timestamps to perform forward propagation compensation on the updated target state; that is, based on the currently estimated target velocity, a short-time prediction of the target position is made, thereby compensating for the time delay errors caused by visual perception and computational processing. Therefore, what is ultimately sent to the pursuit controller is no longer a lag position measurement directly obtained from geometric mapping, but rather the target position-velocity state after recursive estimation and time delay compensation. This method improves the temporal consistency and effectiveness of the target state input, thereby enhancing closed-loop pursuit performance.

[0055] To address the problems of large online computation latency, delayed state input, and easily invalidated tracking results when existing visual tracking methods are directly applied to closed-loop pursuit, this embodiment proposes a target state update method that combines sliding window video tracking with latency compensation. This method reduces tracking latency by shortening the video inference time window and mitigates the adverse effects of perception lag on the controller through state prediction compensation, thereby improving the real-time performance and accuracy of target state estimation and the overall robustness of the closed-loop pursuit system. It maintains relatively stable pursuit performance even under conditions of robot posture changes or discontinuous observation.

[0056] Step S5: Construct local observation state based on target state estimation results and target relative position, and generate current pursuit control commands for the pursuit robot based on local observation state; This invention further designs a multi-robot cooperative pursuit control method based on structured relative state input, addressing the problem of how each pursuit robot should coordinate its movement to achieve effective pursuit. Unlike end-to-end visual pursuit methods that directly input the raw image into the control network, this invention does not directly use the first-view image as control input. Instead, it constructs a structured state representation based on target state estimation, the robot's own state, and the states of other pursuit robots, and uses this representation as input to the multi-robot cooperative pursuit controller, thereby establishing an interpretable closed loop consisting of "perception—state—control." This step is a local decision-making process executed separately for each pursuit robot, with each robot independently generating control commands based on its own structured state.

[0057] Specifically, for any pursuing robot, its local observation state input into the multi-agent reinforcement learning policy network includes at least the following: the pursuing robot's own position, velocity, and orientation; the relative positions, orientations, and velocities of other pursuing robots; and the relative position of the target relative to the current pursuing robot. These observations collectively constitute the local observation state of the pursuing robot at the current moment, wherein the target's relative position is provided by the aforementioned visual tracking, geometric transformation, and time delay compensation steps.

[0058] The first The first tracking robot The local observation state of a frame image is denoted as Multi-agent reinforcement learning policy network is ,but: (3); in, The output of the multi-agent reinforcement learning policy network is targeted at the third agent. The first pursuit robot in the The discrete behavior decision for each frame, which will be explained later as a velocity index, is calculated after all frames have been processed to obtain the current frame. The pursuit control commands for a pursuit robot.

[0059] In this way, the collaborative pursuit controller no longer needs to learn target recognition and localization directly from the original image, but instead makes action decisions based on state variables with clear physical meaning, thereby reducing the learning complexity of the control strategy. Regarding control output, this invention uses a two-dimensional discrete velocity action form to generate pursuit control commands.

[0060] Specifically, in shaft and Several discrete velocity values ​​are pre-set along the axial direction, forming two velocity sets; the multi-agent reinforcement learning policy network receives the local observation state of the current pursuit robot, and after network mapping, outputs... Axial direction and The selection probability of each discrete velocity action in the axial direction is used to select a discrete velocity value from two velocity sets based on the selection probability corresponding to each velocity action. This combination forms the current planar velocity command for the pursuit robot, i.e.: (4); in, and They are respectively Axial direction and A preset set of discrete velocities along the axial direction. and The first The first pursuit robot in the The frame selects the velocity value from the corresponding velocity set.

[0061] In other words, the output of the multi-agent reinforcement learning policy network is not a continuous control quantity, but rather discrete action selection results corresponding to two directions. By combining the action selection results in the two directions, the two-dimensional planar velocity action used for pursuit control at the current moment can be obtained. Multiple pursuit robots generate action outputs based on their respective observation states and under the same cooperative policy constraints, thus forming a cooperative pursuit behavior.

[0062] To guide the pursuit robots to rapidly approach the target and form effective cooperation while ensuring safety, this invention preferably employs a reward mechanism consisting of target approach rewards, successful pursuit rewards, penalties for robots getting too close, and time penalties. Specifically, target approach rewards encourage pursuit robots to reduce the distance to the target; successful pursuit rewards enhance positive incentives when the target is successfully captured; penalties for robots getting too close mitigate the risk of collisions caused by insufficient distance; and time penalties promote the rapid completion of the pursuit process. Through the above design of observation inputs, action outputs, and reward constraints, this invention enables the multi-agent reinforcement learning policy network to learn control strategies suitable for multi-robot cooperative pursuit tasks, thereby outputting two-dimensional discrete velocity actions that reflect changes in target position and the relative relationships of teammates, improving the efficiency and stability of multi-robot cooperative pursuit.

[0063] To address the challenges of existing multi-robot pursuit control methods on real robot platforms, such as difficulties in policy transfer, inability to meet chassis kinematic constraints, and insufficient obstacle avoidance safety, this embodiment proposes a multi-agent reinforcement learning cooperative pursuit method based on structured relative state input and constructs an execution interface adapted to real robot navigation systems. This method transforms high-level cooperative pursuit actions into reference target points that can be stably executed by differential robots, and combines local path planning to achieve obstacle avoidance and maintain safe distances. This improves the method's deployability, adaptability to real platforms, and application promotion value, making it applicable to various pursuit scenarios, including indoor, outdoor, and obstacle-containing scenarios.

[0064] Step S6: Convert the pursuit control command into a short-time domain reference target point, perform path planning on the short-time domain reference target point, and obtain the underlying control quantity of the pursuit robot at the current moment.

[0065] Finally, addressing the issue that multi-agent reinforcement learning outputs are difficult to directly apply to the chassis of real robots, this invention constructs a navigation execution interface for real robots. Considering that real pursuit robots typically have differential chassis and cannot directly and stably execute the two-dimensional planar velocity vector output by the multi-agent reinforcement learning strategy, this invention does not directly send the high-level action result to the low-level motors. Instead, it first converts the two-dimensional planar velocity command into a short-time domain reference target point, and then the navigation module and local path planning algorithm further convert the reference target point into a low-level control quantity that the robot can execute.

[0066] Specifically, based on the pursuit control command and the current position information of the pursuit robot, the short-time domain reference target point is obtained by integral calculation according to the preset look-ahead time. Let the first Frame number The positions of the pursuit robots in the plane are: The first output of the multi-agent reinforcement learning policy network Frame number The planar velocity command for each pursuit robot is: The present invention is based on a preset look-ahead time. Convert the planar velocity command into a short-time domain reference target point. ,Right now: (5); In this way, abstract planar velocity motion can be converted into spatial targets that the navigation module can process.

[0067] Based on the short-time domain reference target point, combined with the current pose information of the current pursuit robot, environmental obstacle information, and the position information of other pursuit robots, a local path planning algorithm is used to plan the path and obtain the optimal speed control quantity. The optimal speed control value is used as the underlying control value of the pursuit robot at the current moment.

[0068] After obtaining the short-time domain reference target point, the navigation module combines the current pose of the pursuit robot, environmental obstacle information, and the position information of other pursuit robots, calls the local path planning algorithm to search and evaluate candidate motion trajectories, and selects the motion control quantity that meets the constraints.

[0069] Preferably, the local path planning algorithm is a Dynamic Window Method (DWA), but it is not limited to this. In this process, the local path planning algorithm uses the reference target point as the local navigation target, comprehensively considering the feasible range of the robot chassis's linear and angular velocities, obstacle avoidance requirements, and safety distance constraints with other pursuit robots, to generate the underlying control quantity corresponding to the current moment. The underlying control quantity is preferably a linear and angular velocity control command that can be directly executed by the differential robot, or it can be further converted into left and right wheel drive commands. Through this execution interface, the pursuit strategy can be smoothly migrated from the simulation environment to the real robot platform, thereby improving the engineering deployability and practical application capability of the method.

[0070] The optimal speed control value is output as the linear and angular velocity commands that the differential robot can execute, and then used for execution by the robot chassis.

[0071] It is understood that this invention can be deployed in differential wheeled robot systems, and can also be extended to omnidirectional mobile robots, tracked platforms, quadruped robots or other robot platforms with onboard visual perception and autonomous navigation capabilities; the environment can also be extended from two-dimensional near-ground plane scenes to more complex indoor and outdoor mixed scenes, dynamic obstacle scenes or heterogeneous robot collaborative scenes.

[0072] Through the above technical solution, this embodiment realizes a complete technical closed loop, from specifying the pursuit target in natural language, online visual tracking, target relative state estimation and compensation, to multi-robot collaborative pursuit control and execution by real robots. This method can adapt to complex pursuit scenarios with unknown initial target positions, the presence of interfering objects, partial occlusion, and significant perception delays. It also considers interpretability, real-time performance, collaboration, and engineering deployability, demonstrating promising application prospects.

[0073] Example 2 This embodiment provides a multi-robot visual tracking system for unknown targets, including: The target initialization module is used to perform semantic analysis based on the first-view image of each pursuit robot in the current frame and the pursuit command to determine the target to be pursued and the initial target bounding box. The visual tracking module is used to form a pseudo-video sequence based on the initial target bounding box and several consecutive frames of target images of the target to be tracked online. The pseudo-video sequence is then segmented using a video target segmentation model to obtain the tracking result of the current frame. The state estimation module is used to transform the target to be pursued from the image space to the robot chassis coordinate system based on the first-view image and the tracking results, so as to obtain the target relative position of the target to be pursued with respect to each pursuit robot. The delay compensation module is used to predict the state of the target to be pursued in the current frame, obtain the predicted position and predicted velocity of the current frame, and correct it according to the relative position of the target to obtain the target state estimation result. The collaborative pursuit control module is used to construct a local observation state based on the target state estimation results and the target's relative position, and to generate pursuit control commands for the current pursuit robot based on the local observation state. The navigation control module is used to convert pursuit control commands into short-time domain reference target points, perform path planning on the short-time domain reference target points, and obtain the underlying control variables of the pursuit robot at the current moment.

[0074] The examples and application scenarios implemented by the above modules and corresponding steps are the same, but are not limited to the content disclosed in Embodiment 1 above. It should be noted that the above modules, as part of the system, can be executed in a computer system such as a set of computer-executable instructions.

[0075] The descriptions of each embodiment in the above embodiments have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0076] The proposed system can be implemented in other ways. For example, the system embodiments described above are merely illustrative, and the division of modules described above is only a logical functional division. In actual implementation, there may be other division methods. For example, multiple modules may be combined or integrated into another system, or some features may be ignored or not executed.

[0077] Example 3 This embodiment provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the multi-robot visual tracking method for unknown targets as described in Embodiment 1 above.

[0078] Example 4 This embodiment provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps in the multi-robot visual tracking method for unknown targets as described in Embodiment 1 above.

[0079] Example 5 This embodiment provides a computer program product or computer program, including computer instructions stored in a computer-readable storage medium. The processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the steps in the multi-robot visual tracking method for unknown targets described in Embodiment 1 above.

[0080] Those skilled in the art will understand that embodiments of the present invention can provide methods, systems, or computer program products. Therefore, the present invention can take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage and optical storage) containing computer-usable program code.

[0081] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, as well as combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0082] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0083] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0084] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0085] While the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A multi-robot visual tracking method for unknown targets, characterized in that, include: Semantic analysis is performed on the first-view image of each pursuit robot in the current frame and the pursuit command to determine the target to be pursued and the initial target bounding box. A pseudo-video sequence is formed by combining the initial target bounding box with several consecutive frames of target images of the target to be tracked online. The pseudo-video sequence is then segmented using a video target segmentation model to obtain the tracking result of the current frame. Based on the first-person view images and tracking results, the target to be pursued is transformed from the image space to the robot chassis coordinate system to obtain the target relative to each pursuit robot. The state of the target to be tracked in the current frame is predicted to obtain the predicted position and velocity of the current frame. The prediction is then corrected based on the relative position of the target to obtain the target state estimation result. Based on the target state estimation results, the current state of the pursuit robot itself, and the states of other pursuit robots, a local observation state is constructed, and pursuit control commands for the current pursuit robot are generated based on the local observation state. The pursuit control commands are converted into short-time domain reference target points, and path planning is performed on the short-time domain reference target points to obtain the underlying control variables of the pursuit robot at the current moment.

2. The multi-robot visual tracking method for unknown targets as described in claim 1, characterized in that, The step of performing semantic analysis based on the first-view image of each pursuit robot in the current frame and the pursuit command to determine the target to be pursued and the initial target bounding box includes: Acquire first-person view images and pursuit commands for each pursuit robot in the current frame; A visual big language model is used to perform joint semantic analysis on first-person view images and pursuit instructions to determine the semantic correspondence between pursuit instructions and candidate targets in first-person view images, and to obtain the matching score of each candidate target and pursuit instruction. The candidate target with the highest matching score is selected as the target to be pursued, and the initial target bounding box of the target to be pursued is extracted using a visual large language model.

3. The multi-robot visual tracking method for unknown targets as described in claim 1, characterized in that, A pseudo-video sequence is constructed based on an initial target bounding box and several consecutive frames of target images of the target to be tracked online. A video target segmentation model is used to segment the pseudo-video sequence to obtain the tracking result of the current frame, including: Following the chronological order, using the initial target bounding box as a visual cue, the target images of the current frame and several consecutive frames preceding it are organized to form a pseudo-video sequence; Based on pseudo-video sequences, a pre-trained video target segmentation model is used for segmentation to obtain the target mask, target bounding box, and reliability score of the current frame, which are used as the tracking results.

4. The multi-robot visual tracking method for unknown targets as described in claim 1, characterized in that, The process of transforming the target to be pursued from image space to robot chassis coordinate system based on first-view images and tracking results to obtain the target's relative position with respect to each pursuing robot includes: Target pixels are selected based on the target bounding box in the tracking results, and the coordinates of the target pixels in the camera coordinate system are determined by combining the depth information corresponding to the first-view image and the camera calibration parameters. Based on the rotation matrix and translation vector from the camera coordinate system to the robot chassis coordinate system, and the coordinates of the target pixel in the camera coordinate system, the coordinates of the target pixel in the robot chassis coordinate system are determined, which serve as the target relative position of the target to be pursued relative to each pursuing robot.

5. The multi-robot visual tracking method for unknown targets as described in claim 1, characterized in that, The target state in the current frame is predicted to obtain the predicted position and velocity. These are then corrected based on the target's relative position to obtain the target state estimation result, including: Based on the target position and target velocity estimates of the previous frame, the state of the target to be pursued in the current frame is predicted to obtain the predicted position and predicted velocity of the current frame. The predicted position and velocity of the current frame are corrected based on the relative position of the target to obtain the updated predicted position and velocity, which are used as the target state prediction result.

6. The multi-robot visual tracking method for unknown targets as described in claim 1, characterized in that, The process of converting pursuit control commands into short-time domain reference target points, performing path planning on the short-time domain reference target points, and obtaining the underlying control variables of the pursuit robot at the current moment includes: Based on the pursuit control command and the current position information of the pursuit robot, the short-time domain reference target point is obtained by integral calculation according to the preset look-ahead time. Based on the short-time domain reference target point, combined with the current pose information of the current pursuit robot, environmental obstacle information, and the position information of other pursuit robots, a local path planning algorithm is used to plan the path and obtain the optimal speed control quantity. The optimal speed control value is used as the underlying control value of the pursuit robot at the current moment.

7. A multi-robot visual tracking system for unknown targets, characterized in that, include: The target initialization module is used to perform semantic analysis based on the first-view image of each pursuit robot in the current frame and the pursuit command to determine the target to be pursued and the initial target bounding box. The visual tracking module is used to form a pseudo-video sequence based on the initial target bounding box and several consecutive frames of target images of the target to be tracked online. The pseudo-video sequence is then segmented using a video target segmentation model to obtain the tracking result of the current frame. The state estimation module is used to transform the target to be pursued from the image space to the robot chassis coordinate system based on the first-view image and the tracking results, so as to obtain the target relative position of the target to be pursued with respect to each pursuit robot. The delay compensation module is used to predict the state of the target to be pursued in the current frame, obtain the predicted position and predicted velocity of the current frame, and correct it according to the relative position of the target to obtain the target state estimation result. The collaborative pursuit control module is used to construct a local observation state based on the target state estimation results and the target's relative position, and to generate pursuit control commands for the current pursuit robot based on the local observation state. The navigation control module is used to convert pursuit control commands into short-time domain reference target points, perform path planning on the short-time domain reference target points, and obtain the underlying control variables of the pursuit robot at the current moment.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps in the multi-robot vision tracking method for unknown targets as described in any one of claims 1-6.

9. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps in the multi-robot visual tracking method for unknown targets as described in any one of claims 1-6.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the steps of the multi-robot visual tracking method for unknown targets as described in any one of claims 1-6.

Images