Multi-unmanned aerial vehicle cooperative tracking method based on target semantics and spatial position information

Abstract

The invention discloses a multi-unmanned aerial vehicle cooperative tracking method based on target semantics and spatial position information. Two or more unmanned aerial vehicles are allowed to realize cooperative tracking of the same target in a low-communication information interaction mode. The method comprises the following steps: S1, extracting target semantic information; each unmanned aerial vehicle obtains target image data through camera shooting, detects all targets and processes the targets into low-dimensional semantic embedding vectors to represent high-level semantic information of the targets. S2, acquiring target position information; and each unmanned aerial vehicle obtains spatial pose information in real time through a built-in GPS and an IMU, and estimates the position of the target in the world coordinate system based on the visual information of the target and the pose information of the unmanned aerial vehicle. And S3, the target semantics and the position information are combined and transmitted. And combining the target semantic embedding information with the spatial position information to form a target description with the position information, and transmitting the target description in a low communication mode. And S4, cross-unmanned aerial vehicle target association and tracking. And associating the same target under different visual angles by comparing the target semantic similarity and the spatial position distance. And if the association is successful, continuous tracking is carried out. The method can effectively realize cooperation of multiple unmanned aerial vehicles, reduce the communication load and improve the tracking precision.

Description

Technical Field

[0001] The present invention relates to the fields of computer vision, UAV technology and multi-agent collaborative perception, and in particular to a multi-UAV collaborative tracking method that combines target semantic information and spatial position information. The method integrates target detection and semantic information extraction, pose estimation, cross-view target association, and real-time information sharing and transmission technology to achieve efficient and stable collaborative tracking of the same target by multiple UAVs in a dynamic and complex environment. Technical Background

[0002] With the rapid development of drone technology, drones are increasingly used in environmental monitoring, logistics distribution, search and rescue missions, security patrols and other fields. However, in many practical scenarios, a single drone is difficult to complete complex tasks efficiently due to limitations in field of view, computing power and communication range. Therefore, multi-drone collaborative operations have gradually become a hot research topic. Especially in target tracking tasks, the collaboration of multiple drones can effectively expand the monitoring range, enhance the robustness of the system and enhance the efficiency of task completion.

[0003] Traditional target tracking methods usually rely on the visual detection and tracking capabilities of a single drone. These methods often have difficulty maintaining tracking continuity when faced with complex dynamic environments, such as frequent occlusion of the target, changes in lighting, and rapid movement of the target. At the same time, the perception range of a single drone is limited. When the target is beyond its field of view, the system will lose control of the target. In order to solve these problems, multi-drone collaborative tracking has become a key technology. The core lies in how to achieve efficient collaboration between multiple drones, including information sharing, target association, and task allocation.

[0004] In multi-UAV cooperative target tracking, information interaction is the key to achieving collaboration. On the one hand, UAVs need to share target information in a low-latency and high-efficiency manner; on the other hand, in order to adapt to the multi-view environment, it is necessary to combine the semantic information and spatial position information of the target to accurately identify and associate the same target observed by different UAVs. However, this process faces many technical challenges. The appearance and position information of the target may be different from the perspective of different UAVs. How to ensure the consistency of the target identity is the core issue. The communication bandwidth between UAVs is limited. The transmission of large-scale data (such as images or videos) will lead to increased latency or even data loss. Therefore, it is necessary to design efficient data compression and key information transmission strategies. Lighting changes, occlusions, and parallax in complex scenes may lead to a decrease in the accuracy of target detection and association. It is necessary to design a robust association strategy, and how to restore the tracking of the target through the mutual cooperation of multiple UAVs when a UAV in the system fails to track. Summary of the invention

[0005] The purpose of the present invention is to overcome the shortcomings of the prior art and provide a multi-UAV collaborative tracking method based on target semantics and spatial position information. The present invention adopts the following technical solution: a multi-UAV collaborative tracking method based on target semantics and spatial position information, the method comprising the following steps:

[0006] A multi-UAV collaborative tracking method based on target semantics and spatial position information, the method comprising the following steps:

[0007] S1: Each drone acquires target image data through camera and detects all targets in the field of view. The target box is generated and processed into a low-dimensional semantic embedding vector to represent the high-level semantic information of the target.

[0008] S2: Target position information acquisition. Each drone obtains spatial posture information in real time through the built-in GPS and IMU, and estimates the position of the target in the world coordinate system by combining the target detection results and the drone posture data. During the estimation, an optimization algorithm is used to reduce the estimation error.

[0009] S3: Combine and transmit the target semantics with the location information. Combine the target’s semantic embedding information with the spatial location information to form a target description with location information, and transmit it in a low-traffic manner, sharing only key data without transmitting the complete image or video.

[0010] S4: Cross-UAV target association and tracking. By comparing the semantic similarity and spatial position distance of the target, the same target from different perspectives is associated. After successful association, continuous tracking is performed and the latest real-time information is broadcasted to the outside; if the association fails, the shared information from other UAVs is continuously received, while searching and matching the target in the field of view. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings of the embodiments will be briefly introduced below. Obviously, the drawings in the following description only relate to some embodiments of the present invention, but are not intended to limit the present invention.

[0012] Figure 1 : Flowchart of multi-UAV collaborative target tracking method (demonstrated with two UAVs).

[0013] Figure 2 : Schematic diagram of multi-UAV target detection and target semantic information generation.

[0014] Figure 3 : Schematic diagram of multi-UAV collaborative tracking based on target semantics and spatial position information.

[0015] Figure 4 : Schematic diagram of multi-UAV collaborative tracking test process. DETAILED DESCRIPTION

[0016] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with specific embodiments and with reference to the accompanying drawings.

[0017] S1 Target semantic information extraction: A single drone collects image data within the field of view through a camera, detects all targets and extracts their semantic information, such as Figure 2 As shown in Figure 2, the target semantic information is represented in the form of a low-dimensional embedding vector for subsequent cross-agent target association. Suppose the visual feature of the target is F i ∈R d (i=1, 2...n) where d is the embedding dimension, F i Represents the high-level semantic information of the detected i-th target. In traditional methods, target detection and semantic embedding are usually performed separately, that is, the detector is used to extract the target box first, and then each target box is processed one by one through the embedding network. This process will cause the inference time to increase linearly with the number of targets n. det +n×T embed , where T det and T embed The time overhead of target detection and single target embedding is respectively. To solve this problem, the present invention integrates the detector and semantic embedding network into a single framework, and realizes end-to-end reasoning of the two tasks by sharing feature extractors, thereby optimizing computational efficiency. In order to avoid the competition problem between detection and semantic embedding tasks that may be caused by task inheritance, the present invention introduces a reciprocal network design, which allocates multi-scale feature maps F for detection tasks and embedding tasks respectively. det and F embed , to achieve feature decoupling, the specific process is:

[0018] F det =SelfAttention(F) (1)

[0019] F embed =CrossAttention(F) (2)

[0020] Where F is the shared basic feature map, SelfAttention and CrossAttention are the self-correlation and cross-correlation modules respectively. In the association process based only on the target semantics, the present invention transmits semantic information through low communication bandwidth, limits the dimension of the embedding vector to d <= 512, and calculates the semantic similarity of the target by the following formula:

[0021]

[0022] When Sim(F i , F j)≥τ (similarity threshold, 0.7 in the experiment), it is determined that targets i and j belong to the same target and the targets are associated. Through the above method, the system realizes the efficient extraction and sharing of semantic information, significantly improves the target detection and cross-view association performance in complex scenes, and ensures the real-time and robustness in multi-agent collaboration.

[0023] S2 Target position information acquisition: The invention first establishes a standard world coordinate system in the global scope, which provides a unified reference frame for all drones. On this basis, the flight attitude and relative height information of each drone is obtained through the flight control system of the drone. Specifically, the drone's position information is usually represented by the quaternion q=[q w ,q x ,q y ,q z ] is used to describe the rotation state of the drone camera relative to the world coordinate system. The height information H is used to describe the vertical position information of the drone camera. In addition, the intrinsic parameter matrix K of the drone camera is the basis of visual positioning and is used to link the camera coordinate system with the image coordinate system. The intrinsic parameter matrix of the camera is as follows:

[0024]

[0025] Among them, f x and f y are the focal lengths of the intelligent camera in the horizontal and vertical directions, O x and O y is the pixel coordinate of the principal point. In order to determine the position of the target in the world coordinate system, it is first necessary to calculate the rotation matrix R through the quaternion q to describe the spatial rotation relationship of the drone camera. The quaternion q can be converted into a rotation matrix through the following formula:

[0026]

[0027] This matrix describes the posture of the camera in the world coordinate system, and the image coordinates are converted to world coordinates through the rotation matrix. In order to extract the three-dimensional information of the target from the image coordinates, the present invention adopts a target positioning method based on the perspective projection model. The image coordinates (u, v) captured by the drone camera can be projected and transformed into three-dimensional coordinates (X, Y, Z) through the intrinsic parameter matrix K and the rotation matrix R. The relationship can be expressed in the following matrix form:

[0028]

[0029] Among them, s is a scaling factor used to adjust the relative proportion of targets of different sizes in the image; the translation vector t = (0, 0, H) represents the relative position of the camera in the world coordinate system. Through the inverse perspective transformation, the three-dimensional position of the target in the world coordinate system can be solved. A single drone can only estimate the position of the target in the world coordinate system based on its own position. In order to ensure that multiple drones can lock the same target at the same time, the present invention adopts a joint target positioning method of multi-drone collaboration. In this method, three drones independently calculate the position coordinates of the target and compare their respective calculation results through communication. Multiple drones determine whether the same target is locked by the deviation between their estimated target coordinate values. When the target coordinate deviation estimated by each drone is within a certain threshold, it can be considered that all drones have successfully locked the same target. Otherwise, the system will further adjust the attitude information and camera parameters of each drone to optimize the accuracy of target positioning. In view of the problem of low estimation accuracy caused by sensor error, the present invention introduces a posture correction mechanism based on Kalman filtering, dynamically adjusts the posture data of each drone, and reduces the deviation between the estimated values ​​of each target through multiple iterative optimization. Reduce the impact of task accuracy caused by sensor error as much as possible. Specifically, Kalman Filter (KF) is a recursive algorithm widely used in state estimation, especially for noise suppression and state tracking in linear systems. In the joint positioning problem of multi-UAV observers, the system regards the posture of each UAV as the state of the system, and dynamically adjusts and corrects these posture data through Kalman filtering. The core idea of ​​Kalman filtering is to use sensor data and system models to predict and update state estimates. It is based on two main steps, namely: 1. Prediction step: According to the motion model of the system, the state and control input of the previous moment are used to predict the state of the next moment. 2. Update step: Combine the measurement data to calculate the estimate of the current state, and adjust the prediction error through the Kalman gain. In the multi-UAV joint positioning system, the posture X of the UAV is the state of the system. t Contains its position and orientation. At time t, the position and orientation of the drone can be represented by the following state vector:

[0030]

[0031] Among them, p x , p y , p z is the position of the drone in the world coordinate system, q w ,q x ,q y ,q z is a quaternion describing its orientation. The Kalman filter algorithm is used to modify this state vector based on sensor measurements (such as IMU and GPS) and feedback from the visual positioning system.

[0032] The prediction step is based on the kinematic model of the drone, using the state X at the previous time t-1 t-1 The process of posture prediction can be expressed as:

[0033]

[0034] in is the predicted state vector at time t, F t is the state transfer matrix, which describes the motion model of the system. t U t is the control input, describing the influence of external control on the attitude, such as the propulsion system of the UAV. t is a bias that describes the noise and uncertainty in the system.

[0035] In the update step, this study combines the actual observation value Z of the sensor t To correct the predicted state. The corrected state estimate can be obtained by the following update equation:

[0036]

[0037] Where K t is the Kalman gain matrix, which is used to balance the error between the predicted state and the measured state. When the measurement uncertainty is large, the Kalman gain tends to rely on the predicted state. Conversely, when the measurement is more accurate, the system will rely more on the measured value.

[0038] Through each cycle, the Kalman filter dynamically adjusts the drone's pose data based on the error between the prediction and the measurement. This method can effectively suppress errors caused by sensor noise or other interference. Over multiple iterations, the pose estimate of each drone gradually stabilizes, and its estimated target position is therefore closer to the true value. When three drones jointly locate the same target, the system calculates each drone's independent estimate of the target position and takes a weighted average. After the pose is corrected by the Kalman filter, the deviation between the target estimates of the three drones will be greatly reduced, and the final target position will have a higher accuracy.

[0039] S3 Combining and transmitting target semantics with position information: In the implementation process of combining and transmitting target semantics with position information, each drone first captures the image data of the target through the camera, and uses the target detection algorithm and semantic embedding method to extract the high-level features of the target to generate a low-dimensional semantic embedding vector F. At the same time, the drone obtains its own posture information in real time through the built-in GPS and IMU sensors, and combines the position of the visual detection box of the target to obtain the spatial position P of the target in the world coordinate system through geometric calculation. After the semantic information of the target is combined with the spatial position information, a complete target description data packet T = {F, P} is generated for sharing the target among multiple drones. In order to reduce the communication bandwidth requirements and transmission delay, only key data such as the semantic embedding information and spatial position information of the target are transmitted, without sending the original image or video data. The generated data packet structure includes target ID, semantic embedding vector, spatial position information and timestamp. Calculated with 512 dimensions and 8 bits per dimension for each semantic embedding vector, the size of the semantic information is 512 bytes. With the spatial position information and additional metadata, the size of a single target data packet is about 540 bytes, which greatly reduces the communication load and supports efficient transmission even in the case of limited bandwidth. The transmission uses the UAV's digital transmission module, which supports low-latency (about 20ms) and high-stability communication. Data packets can be transmitted to cooperative UAVs in a broadcast or point-to-point manner. After receiving the data packet, the receiving UAV decodes it to extract the semantic embedding and spatial location information of the target, and uses this information to complete the matching and association of the target, ensuring the consistency of the target identity under different perspectives. When the target is successfully matched, the cooperative UAV will continue to relay tracking the target, while updating the latest status of the target and broadcasting it to other UAVs to form a cooperative tracking closed loop. In order to improve the robustness of communication, the digital transmission module has packet loss retransmission and error correction mechanisms to ensure that key target information can still be accurately transmitted in complex environments or signal interference. This method is particularly suitable for multi-UAV collaboration to overcome the shortcomings of existing technologies. For example, in a complex dynamic environment, the main UAV shares the semantics and location information of the target through the digital transmission module, ensuring that the cooperative UAV can use the shared information to quickly locate and relay tracking the target after the target leaves the main UAV's field of view, so as to achieve continuous management and efficient collaboration of the target. In addition, the system can also operate effectively in multi-target scenarios, supporting the parallel transmission and processing of multiple target information, thereby meeting the real-time, reliability and robustness requirements of multi-agent collaborative tasks.

[0040] S4 combines target semantics and location information to associate and track cross-UAV targets: Its overall architecture is as follows Figure 3As shown, firstly, the position information is used to make a preliminary rough association, and after the suspected matching target is determined, the semantic features are used to make a refined association. The specific association steps of each method have been described in S2 and S3. Specifically, when a drone that has not successfully tracked the target receives the real-time semantic information of the target and its own posture information from the successfully tracked drone, the current drone first estimates the approximate range of the target based on these posture data, and then the drone turns to roughly estimate the position of the target, locks the suspected target, and makes a preliminary target association. Then, among these suspected targets, the semantic similarity between the targets (such as cosine similarity) is calculated to further refine the association of the same target from the perspective of different drones. When a drone temporarily loses the target, it relies on information transmitted by other drones to reposition the target to ensure that the system can continue to track the target even in the case of occlusion or loss of vision. The flow chart of the entire invention is as follows. Figure 1 shown.

[0041] The present invention can be physically tested in an outdoor drone scene. The test results show that by adopting the method of the present invention, multiple drones can coordinately track the target stably, thereby verifying the feasibility and practicality of the present invention.

Claims

1. A multi-UAV collaborative tracking method based on target semantics and spatial position information, characterized in that: The method comprises the following steps: S1: Each drone acquires target image data through camera and detects all targets in the field of view. The target box is generated and processed into a low-dimensional semantic embedding vector to represent the high-level semantic information of the target. S2: Target position information acquisition. Each drone obtains spatial posture information in real time through the built-in GPS and IMU, and estimates the position of the target in the world coordinate system by combining the target detection results and the drone posture data. During the estimation, an optimization algorithm is used to reduce the estimation error. S3: Combine and transmit the target semantics with the location information. Combine the target’s semantic embedding information with the spatial location information to form a target description with location information, and transmit it in a low-traffic manner, sharing only key data without transmitting the complete image or video. S4: Cross-UAV target association and tracking. By comparing the semantic similarity and spatial position distance of the target, the same target from different perspectives is associated. After the association is successful, continuous tracking is performed and the latest real-time information is broadcasted to the outside; If the association fails, it continues to receive shared information from other drones while searching and matching targets in its field of view.

2. The method for multi-UAV collaborative tracking based on target semantics and spatial position information according to claim 1, characterized in that: In step 1, each drone in the system obtains image information of the target through a camera, extracts the features of the target using deep learning or traditional image processing methods, and then enters a branch, one of which detects the target in the field of view, and the other branch uses a convolutional neural network or other visual extraction network to convert the extracted visual features into a low-dimensional semantic embedding vector of length 512 to represent the high-level semantic information of the target.

3. The method for multi-UAV collaborative tracking based on target semantics and spatial position information according to claim 1, characterized in that: In step 2, each drone obtains its spatial position information in real time through built-in sensors (such as GPS, IMU) to form the position information of the drone. The posture information of the drone is usually represented by quaternion q = [q w ,q x ,q y ,q z ] is used to describe the rotation state of the drone camera relative to the world coordinate system. The intrinsic parameter matrix K of the drone camera is the basis of visual positioning and is used to link the camera coordinate system with the image coordinate system. The intrinsic parameter matrix of the camera is as follows: Among them, f x and f y are the focal lengths of the intelligent camera in the horizontal and vertical directions, O x and O y is the pixel coordinate of the principal point. In order to determine the position of the target in the world coordinate system, it is first necessary to calculate the rotation matrix R through the quaternion q to describe the spatial rotation relationship of the drone camera. Then, based on the target detection results and the position information of the drone, the position of the target in the world coordinate system is estimated. The image coordinates (u, v) captured by the drone camera can be projected and transformed into three-dimensional coordinates (X, Y, Z) through the intrinsic parameter matrix K and the rotation matrix R. The relationship can be expressed in the following matrix form: Among them, s is the scaling factor, which is used to adjust the relative proportions of objects of different sizes in the image; the translation vector t = (0, 0, H) represents the relative position of the camera in the world coordinate system. Through the inverse perspective transformation, the three-dimensional position of the target in the world coordinate system can be solved. Then the Kalman filter algorithm is used to correct the posture of the drone to reduce the estimation error of the target position.

4. The method for multi-UAV collaborative tracking based on target semantics and spatial position information according to claim 1, characterized in that: In step 3, the semantic embedding information of the target is combined with the spatial location information to form a target description with location information, which is transmitted in a low-traffic manner, sharing only key data to reduce the consumption of transmission bandwidth between drones. The specific content transmitted includes the high-level semantic information of the target, with a length of 512, and the quaternion matrix of the current drone. This information is summarized and used in the target association step across drones.

5. The method for multi-UAV collaborative tracking based on target semantics and spatial position information according to claim 1, characterized in that: In step 4, the location information is first used for preliminary rough association. After the suspected matching target is determined, it is then refined by semantic features. Specifically, when a drone that has not successfully tracked the target receives the real-time semantic information of the target and its own posture information from the successfully tracked drone, the current drone first estimates the approximate range of the target based on these posture data, and then the drone turns to roughly estimate the position of the target, locks the suspected target, and performs preliminary target association. Then, among these suspected targets, the semantic similarity between the targets (such as cosine similarity) is calculated to further refine the association of the same target from different drone perspectives. When a drone temporarily loses the target, it relies on the information transmitted by other drones to relocate the target to ensure that the system can continue to track the target even in the case of occlusion or loss of vision.

Images