A method for long-term target tracking in a multi-unmanned system

Abstract

This invention discloses a long-term target tracking method for multi-unmanned systems, comprising: selecting an arbitrary j-th unmanned system, using its camera to capture an image frame as the current frame, cropping a target region image patch, determining the target's three-dimensional position, and determining self-localization information; sending the above information to teammates and receiving the above information from teammates; a target collaborative localization module fusing the above information from itself and teammates to determine target position information and predict target information, and transmitting the target position information, target prediction information, and the target region image patch from teammates to a target tracking module for updating; the camera capturing a new image frame as the current frame, obtaining multiple target candidate regions based on the target position information and target prediction information, and further determining the target's three-dimensional position; repeating the above steps until the tracking task is completed. This invention improves the accuracy and success rate of target tracking.

Description

Technical Field

[0001] This invention belongs to the field of unmanned systems technology, and in particular relates to a long-term target tracking method based on multi-unmanned system cooperation. Background Technology

[0002] Target tracking, a core and highly challenging research topic in computer vision, has been extensively studied by researchers for many years. However, significant challenges remain when tracking moving targets in multi-unmanned systems under complex environments. Currently, mainstream visual tracking algorithms mainly involve image input, target appearance modeling (target description, statistical model), motion estimation, target localization, and target appearance model updating. In recent years, deep learning-based target tracking algorithms have also achieved good performance. Existing methods primarily address the target tracking problem based on image sequences.

[0003] However, in practical applications of multi-unmanned systems, the target tracking problem becomes more complex, becoming a distributed collaborative tracking problem. Its tracking performance is affected by sensor type, target appearance, external environment, and multi-sensor information fusion framework. As a result, the accuracy and success rate of target tracking in existing multi-unmanned systems are relatively low. Summary of the Invention

[0004] To address the technical problems existing in the prior art, the present invention provides a method for long-term target tracking through multi-unmanned system cooperation, the method comprising the following steps:

[0005] S1: Construct a multi-unmanned system, where each unmanned system is numbered i = 1, 2, 3, ..., n. Each unmanned system is equipped with a self-localization module, a communication module, a target tracking module, a target cooperative localization module, and a camera ci for acquiring image data. The target tracking module consists of a tracking thread and a detection thread. The self-localization information of the i-th unmanned system is the rotation and translation transformation relationship between the camera coordinate system and the world coordinate system of camera ci.

[0006] S2: Take the j-th unmanned system, its self-localization information is as follows: The image frame acquired by camera cj of the j-th unmanned system is used as the current frame, and the target region image block I is cropped. cj The target tracking module is used to determine the three-dimensional position p of the target. wj ;

[0007] S3: Transfer the image patch I of the target region tracked by the j-th unmanned system. cj Target three-dimensional position p wj and the self-localization information of the j-th unmanned system The j-th unmanned system's communication module sends information to other teammates, and the j-th unmanned system's communication module receives target area image patches, target 3D positions, and teammate self-localization information I obtained from other teammates by performing the same steps as described in S2. ci ,p wi , The information received by the communication module of the j-th unmanned system is transmitted to the target cooperative localization module;

[0008] S4: The target cooperative localization module of the j-th unmanned system fuses the target area image patch tracked by the j-th unmanned system, the target's 3D position, and the self-localization information of the j-th unmanned system. cj ,p wj , Image patches of the target area tracked by other teammates, target 3D position, and teammate self-localization information I ci ,p wi , The system determines the target's position and velocity information, predicts the target's possible image coordinates in the next frame, and transmits this target prediction information (including the target's position and potential image coordinates) to the target tracking module. It also transmits the target region image patch I from other teammates. ci The target tracking module of the j-th unmanned system is passed to the target tracking module of the j-th unmanned system, where i = 1, 2, ..., n and i ≠ j.

[0009] S5: The j-th unmanned system utilizes the target region image patch I obtained by itself. cj Target area image patch I with other teammates ci For i = 1, 2, ..., n, where i ≠ j, update its target tracking module;

[0010] S6: The camera cj of the j-th unmanned system acquires a new image frame as the current frame. Based on the target position information provided in step S4 and the image coordinate prediction information that the target may appear in the next frame, the target candidate region is obtained, and the target tracking module is used to further determine the three-dimensional position of the target.

[0011] S7: Repeat steps S3-S6 until the tracking task is completed.

[0012] Furthermore, in the above-mentioned multi-unmanned system collaborative long-term target tracking method, all unmanned systems in the multi-unmanned system execute the steps S1-S7 to achieve multi-unmanned system collaborative long-term target tracking.

[0013] Furthermore, in the aforementioned multi-unmanned system cooperative long-term target tracking method, the self-localization information of the i-th unmanned system is confirmed as follows: the self-localization module determines the pose information of each unmanned system, and obtains the rotation and translation transformation relationship between the coordinate system of the self-localization module of the i-th unmanned system and the world coordinate system. By offline calibration of the self-localization module and the camera coordinate system of the camera CI, the rotation and translation transformation relationship between the camera coordinate system of the camera CI and the world coordinate system is obtained.

[0014] Furthermore, in the above-mentioned multi-unmanned system collaborative long-term target tracking method, the self-localization module is a combined navigation and positioning system or a visual-inertial navigation and positioning system, and all unmanned systems define the same world coordinate system and have unified system time.

[0015] Furthermore, in the above-mentioned multi-unmanned system collaborative long-term target tracking method, the tracking sub-thread of the target tracking module is a target tracking algorithm that can output the pixel coordinates of the target, and the detection sub-thread of the target tracking module is a target detection algorithm or a deep convolutional neural network algorithm that can determine whether the target exists in the entire frame image.

[0016] Furthermore, in the above-mentioned multi-unmanned system cooperative long-term target tracking method, the specific implementation process of step S2 includes:

[0017] S21. Camera cj acquires the current frame image. The tracking thread's tracking sub-taper outputs the pixel coordinates of the target image. The detection thread's detection sub-taper determines whether the target exists in the entire frame image. If the target does not exist, camera cj acquires the next frame image; if the target exists, the target region is determined, and the target region image block I is cropped. cj ;

[0018] S22. Initialize the tracking and detection threads of the target tracking module. If an offline target dataset is available, train the detector of the detection thread using the offline dataset. Use the detector to determine the target image region in the current frame image and initialize the tracking thread's tracker. If no offline target dataset is available, manually determine the target image region in the current frame image, and then initialize the tracker and detector using the visual features of the target region. After determining the target image region, further determine the target's 3D position p. wj .

[0019] Furthermore, in the above-mentioned multi-unmanned system cooperative long-term target tracking method, step S22 determines the target's three-dimensional position p. wj The specific implementation process includes:

[0020] (1) Based on the intrinsic parameters of camera cj and the twist parameter k cj ,d cj The pixel homogeneous coordinates of the target image are transformed to camera coordinates according to the following formula:

[0021]

[0022] Where, pj =[uv 1] T p represents the homogeneous pixel coordinates of the target image. cj =[xyz] T The camera coordinates of the target image;

[0023] (2) When camera cj is a monocular camera, for the j-th unmanned system, the target is located in the ray l of three-dimensional space. j Above, the ray l j The ray equation is expressed in the following world coordinate system:

[0024]

[0025] Where k j For the coefficient, v = [x / zy / z 1] T Let P(k) be the direction vector of the ray equation in the camera coordinate system. The ray passes through the origin and passes through P(k). cj, d cj The inverse transform yields:

[0026]

[0027] The target three-dimensional position p wj From the ray l where the target is located j The normalized representation of the unit direction vector, i.e. Where norm represents the normalization of the vector;

[0028] (3) When the camera cj is a stereo camera or a monocular camera with an integrated range sensor, the camera coordinates p of the target image can be directly solved. cj =[xyz] T The three-dimensional position of the target in the world coordinate system is obtained as follows:

[0029] Furthermore, in the above-mentioned multi-unmanned system cooperative long-term target tracking method, the specific implementation process of step S5 includes:

[0030] S51: The tracking thread first initializes, then acquires image frames, obtains tracking results through the target tracking algorithm, performs a state self-check on the results, and continues tracking if the tracking result confidence is high, and transmits the results and image frames to the detection thread; if the result confidence is low, it re-initializes.

[0031] S52: The detection thread is initialized first, then the image frames and tracking results transmitted by the tracking thread are processed to further determine the tracking status. If the tracking status is good, the detection model is further learned and updated; if the tracking status is poor, the target is re-detected from the image frame and transmitted to the tracking thread, and the tracking thread is re-initialized.

[0032] The multi-unmanned system cooperative long-term target tracking method of the present invention has the following advantages and beneficial effects: by deeply integrating target tracking and target detection, the accuracy and success rate of target tracking are improved. At the same time, by deeply integrating the tracking results of different unmanned systems, robust and fast long-term target tracking can be further achieved. Attached Figure Description

[0033] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. In the drawings:

[0034] Figure 1 This is a schematic diagram illustrating the implementation of the multi-unmanned system cooperative long-term target tracking method of the present invention;

[0035] Figure 2 This is a schematic diagram illustrating the implementation of the target tracking module in the multi-unmanned system cooperative long-term target tracking method of the present invention. Detailed Implementation

[0036] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0037] Target tracking primarily utilizes the temporal and spatial correlation of targets to estimate their position information in consecutive image frames. Tracking algorithms offer good real-time performance and low computational resource consumption, but they frequently fail and cannot re-search for the target. Target detection, on the other hand, determines whether a target exists in an image. These algorithms are robust but require significant computational resources and processing time.

[0038] The multi-unmanned system collaborative long-term target tracking method provided in this invention deeply integrates target tracking and target detection, which can improve the overall performance of the tracking algorithm, increase the accuracy and success rate of target tracking, and further achieve robust and fast long-term target tracking by deeply integrating the tracking results of different unmanned systems.

[0039] The multi-unmanned system cooperative long-term target tracking method provided in this embodiment of the invention specifically includes the following steps:

[0040] S1: Construct a multi-unmanned system, where each unmanned system is numbered i = 1, 2, 3, ..., n, such as... Figure 1 As shown, each unmanned system is equipped with a self-localization module, a communication module, a target tracking module, a target cooperative localization module, and a camera ci for acquiring image data. The target tracking module consists of a tracking thread and a detection thread. In the multi-unmanned system, the rotation and translation transformation relationship between the coordinate system of the self-localization module of the i-th unmanned system and the world coordinate system is as follows: By offline calibration of the extrinsic parameter relationship between the self-localization module and the camera coordinate system of camera ci, the self-localization information of the i-th unmanned system is determined. This self-localization information is the rotation and translation transformation relationship between the camera coordinate system of camera cu and the world coordinate system.

[0041] Specifically, for multi-unmanned systems, the world coordinate system of all unmanned systems is unified. Through the self-localization module, the pose information of each unmanned system can be determined, and finally the rotation and translation transformation relationship between the self-localization module coordinate system and the world coordinate system of the i-th unmanned system can be obtained. By calibrating the extrinsic parameters between the self-localization module and the camera coordinate system (CI) offline, the rotation and translation transformation relationship between the camera coordinate system (CI) and the world coordinate system can be obtained. That is, the self-localization information of the i-th unmanned system.

[0042] S2: Take the j-th unmanned system, its self-localization information is as follows: The image frame acquired by camera cj of the j-th unmanned system is used as the current frame, and the target region image block I is cropped. cj The target tracking module is used to determine the three-dimensional position p of the target. wj .

[0043] More specifically, the specific implementation process of step S2 above includes:

[0044] S21. Camera cj acquires the current frame image. The tracking thread's tracking sub-taper outputs the pixel coordinates of the target image. The detection thread's detection sub-taper determines whether the target exists in the entire frame image. If the target does not exist, camera cj acquires the next frame image; if the target exists, the target region is determined, and the target region image block I is cropped. cj ;

[0045] S22. Initialize the tracking and detection threads of the target tracking module. If an offline target dataset is available, the detector of the detection thread can be trained using the offline dataset. The detector determines the target region in the current frame image, and the tracking thread's tracker is then initialized. If no offline target dataset is available, the target image region in the current frame image is manually determined, and then the tracker and detector are initialized using the visual features of the target region. After determining the target image region, the target's 3D position p is further determined. wj .

[0046] In step S22 above, the three-dimensional position p of the target is determined. wj The specific implementation process includes:

[0047] (1) Based on the camera's intrinsic parameters and distortion parameter k cj ,d cj Transform the pixel homogeneous coordinates of the target image to camera coordinates:

[0048]

[0049] Where, p j =[uv 1] T p represents the homogeneous pixel coordinates of the target image. cj =[xyz] T The camera coordinates of the target image;

[0050] (2) When camera cj is a monocular camera, for the j-th unmanned system, the target is located in the ray l of three-dimensional space. j Above, the ray l j The ray equation is expressed in the following world coordinate system:

[0051]

[0052] Where k j For coefficients, Let v = [x / zy / z 1] be the rotation and translation transformation relationship between the camera coordinate system and the world coordinate system of the j-th unmanned system camera cj. T Let P(k) be the direction vector of the ray equation in the camera coordinate system. The ray passes through the origin and passes through P(k). cj, d cj The inverse transform yields:

[0053]

[0054] Target 3D position p wj From the ray l where the target is located j The normalized representation of the unit direction vector, i.e. Where norm represents the normalization of the vector.

[0055] (3) When the camera cj is a stereo camera or a monocular camera with an integrated range sensor, the camera coordinates p of the target image can be directly solved. cj =[xyz] T The three-dimensional position p of the target in the world coordinate system is obtained. wj :

[0056]

[0057] in, Let cj be the camera coordinate system of the j-th unmanned system, and let cj be the camera coordinate system under rotation and translation with respect to the world coordinate system. Using Kalman filtering, the estimated 3D positions of the target from different unmanned systems can be fused, resulting in more accurate results.

[0058] In (2) above, the direction of the target ray is obtained. To estimate the three-dimensional position of the target, a cooperative localization algorithm based on ray intersection is needed to obtain a more accurate result. Specifically:

[0059] (2.1) The ray containing the j-th unmanned system target is l. j :k j p wj +t cj ,k j >0, because l j Includes unknown parameter k j There may be multiple rays. For any two non-parallel rays, we can find a three-dimensional point p that is closest to both rays. w Each ray is a distance p w The nearest point is defined as q j q j With p w Minimize the distance d between j for:

[0060] d j =||k j p wj +t cj —p w || 2

[0061] exist time q j With pw The distance between them is minimized, at which point the ray is at its minimum distance from p. w Recently click q j for:

[0062]

[0063] (p w -t cj ) || Indicates (p) w -t cj The projection of ) onto vector p wj direction, q j With p w Square distance between for:

[0064]

[0065] I is the identity matrix, (p w -t cj ) ⊥ Indicates (p) w -t cj (relative to p) wj The vertical component of the direction. A least-squares method is used, by accumulating the values ​​of all unmanned systems. The sum of these can be used to solve for p. w The following system of equations is obtained:

[0066]

[0067] (2.2) Establish a model with a minimum sampling set of potential k=2 and a three-dimensional point p as the measurement unit. w The sample set M = {p wi}, i = 1, 2, ..., n, and randomly select m samples from set M as a subset S, where m is greater than k, construct the system of equations in (2.1) above, and solve for the target position p. w .

[0068] (2.3) The complement set of set M is SC = MS, and the target position is p. w With the remaining concentrated rays l j :k j p wj +t cj ,k j Distances greater than 0 represent errors. The set of samples with error values ​​less than a certain set threshold t, along with S, constitutes the interior point set S*.

[0069] (2.4) Randomly select a new subset S and repeat the above process.

[0070] (2.5) After N samplings, the target position p corresponds to the largest potential in the interior point set S*. wj The final target three-dimensional position p was determined. wj .

[0071] S3: Transfer the image patch I of the target region tracked by the j-th unmanned system. cj Target three-dimensional position p wj and the self-localization information of the j-th unmanned system The j-th unmanned system's communication module sends information to other teammates, and the j-th unmanned system's communication module receives target area image patches, target 3D position, and teammate self-localization information I obtained from other teammates by performing the same steps as S2 described above. ci ,p wi , The information received by the communication module is ultimately transmitted to the target cooperative positioning module.

[0072] S4: The target cooperative localization module of the j-th unmanned system fuses the target area image patch tracked by the j-th unmanned system, the target's 3D position, and the self-localization information of the j-th unmanned system. cj ,p wj , Image patches of the target area tracked by other teammates, target 3D position, and teammate self-localization information I ci ,p wi , The system determines the target's position and velocity information, predicts the target's possible image coordinates in the next frame, and transmits this information to the target tracking module. It also transmits the target region image patch I from other teammates. ci The target tracking module of the j-th unmanned system is passed to i = 1, 2, ..., n, i ≠ j.

[0073] More specifically, the specific implementation process of step S4 above includes:

[0074] S41. Determine the target location information. For a monocular camera, the 3D target position p of multiple unmanned systems. wi The values ​​of i = 1, 2, ..., n essentially represent rays emanating from the optical center of the camera ci. The intersection of rays from multiple unmanned systems represents the three-dimensional position of the target, thus determining the target's position information. For binocular cameras or monocular cameras integrated with a ranging sensor, the three-dimensional positions p of the target from multiple unmanned systems are... wi Kalman filtering is used to fuse information from multiple sensors for i = 1, 2, ..., n, thereby determining the target location information;

[0075] S42. Determine target velocity information. Based on the known recorded target position and time information, estimate the target velocity information using a least squares algorithm;

[0076] S43. Predict the possible image coordinates of the target in the next frame. The target's current position and velocity are known. Assuming that the target moves at a constant speed relative to the unmanned platform for a short period of time, the target's three-dimensional position at the next moment can be obtained. With the camera's intrinsic parameters known, the possible image coordinates of the target in the next frame can be predicted through projective transformation.

[0077] S5: The j-th unmanned system utilizes the target region image patch I obtained by itself. cj Target area image patch I with other teammates ci Update the target tracking module for i = 1, 2, ..., n, i ≠ j.

[0078] More in detail, such as Figure 2 As shown, the specific implementation process of step S5 above includes:

[0079] S51: The tracking thread first initializes, then acquires image frames, obtains tracking results through the target tracking algorithm, performs a state self-check on the results, and continues tracking if the tracking result confidence is high, and transmits the results and image frames to the detection thread; if the result confidence is low, it re-initializes.

[0080] S52: The detection thread is first initialized, and then the image frames and tracking results transmitted by the tracking thread are processed to further determine the tracking status. If the tracking status is good, the detection model is further learned and updated; if the tracking status is poor, the target is re-detected from the image frame and transmitted to the tracking thread, and the tracking thread is re-initialized.

[0081] S6: The camera cj of the j-th unmanned system acquires a new image frame as the current frame. Based on the target position information provided in step S4 above and the image coordinate prediction information that the target may appear in the next frame, the target candidate region is obtained, and the target tracking module is used to further determine the three-dimensional position of the target.

[0082] S7: Repeat steps S3-S6 above until the tracking task is completed.

[0083] In the multi-unmanned system cooperative long-term target tracking method provided in the embodiments of the present invention, all unmanned systems in the multi-unmanned system execute the steps S1-S7, thereby ultimately realizing multi-unmanned system cooperative long-term target tracking.

[0084] In the multi-unmanned system cooperative long-term target tracking method provided in the embodiments of the present invention, the self-localization module can be a combined navigation and positioning system, a visual inertial navigation and positioning system, etc. Moreover, the world coordinate system defined by all unmanned systems must be consistent, and the system time must be unified.

[0085] In the multi-unmanned system cooperative long-term target tracking method provided in the embodiments of the present invention, the tracking sub-thread of the target tracking module can be any target tracking algorithm, as long as it can output the pixel coordinates of the target; the detection sub-thread of the target tracking module can be a traditional target detection algorithm or a deep convolutional neural network algorithm, which can determine whether the target exists in the entire frame image.

[0086] In summary, the multi-unmanned system collaborative long-term target tracking method of the present invention deeply integrates target tracking and target detection, which can improve the accuracy and success rate of target tracking. At the same time, by deeply integrating the tracking results of different unmanned systems, it can further achieve robust and fast long-term target tracking.

[0087] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Furthermore, some specific parameters given herein are merely illustrative, and their values ​​may be varied accordingly in different implementations.

[0088] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for long-term target tracking through multi-unmanned system cooperation, characterized in that, Including the following steps: S1: Construct a multi-unmanned system. Each unmanned system in the multi-unmanned system is numbered i = 1, 2, 3, ..., n. Each unmanned system is equipped with a self-localization module, a communication module, a target tracking module, a target cooperative localization module, and a camera ci for acquiring image data. The target tracking module consists of a tracking thread and a detection thread. The self-localization information of the i-th unmanned system is the rotation and translation transformation relationship between the camera coordinate system and the world coordinate system of camera ci. S2: Take the j-th unmanned system, its self-localization information is as follows: The image frame acquired by camera cj of the j-th unmanned system is used as the current frame, and the target region image block I is cropped. cj The target tracking module is used to determine the three-dimensional position p of the target. wj ; S3: Transfer the image patch I of the target region tracked by the j-th unmanned system. cj Target three-dimensional position p wj and the self-localization information of the j-th unmanned system t cj The j-th unmanned system's communication module sends information to other teammates, and the j-th unmanned system's communication module receives target area image patches, target 3D position, and teammate self-localization information obtained from other teammates by performing the same steps as described in S2. The information received by the communication module of the j-th unmanned system is transmitted to the target cooperative localization module; S4: The target cooperative localization module of the j-th unmanned system fuses the target area image patch tracked by the j-th unmanned system, the target's 3D position, and the self-localization information of the j-th unmanned system. Image patches of the target area tracked by other teammates, the target's 3D position, and teammates' self-localization information. The system determines the target's position and velocity information, predicts the target's possible image coordinates in the next frame, and transmits this target prediction information (including the target's position and potential image coordinates) to the target tracking module. It also transmits the target region image patch I from other teammates. ci The target tracking module of the j-th unmanned system is passed to the target tracking module of the j-th unmanned system, where i = 1, 2, ..., n and i ≠ j. S5: The j-th unmanned system utilizes the target region image patch I obtained by itself. cj Target area image patch I with other teammates ci For i = 1, 2, ..., n, i ≠ j, update its target tracking module; S6: The camera cj of the j-th unmanned system acquires a new image frame as the current frame. Based on the target position information provided in step S4 and the image coordinate prediction information that the target may appear in the next frame, the target candidate region is obtained, and the target tracking module is used to further determine the three-dimensional position of the target. S7: Repeat steps S3-S6 until the tracking task is completed.

2. The multi-unmanned system cooperative long-term target tracking method according to claim 1, characterized in that, All unmanned systems in the multi-unmanned system execute steps S1-S7 to achieve collaborative long-term target tracking among the multi-unmanned systems.

3. The multi-unmanned system cooperative long-term target tracking method according to claim 1, characterized in that, The self-localization information of the i-th unmanned system is confirmed as follows: the self-localization module determines the pose information of each unmanned system, and obtains the rotation and translation transformation relationship between the coordinate system of the self-localization module of the i-th unmanned system and the world coordinate system. , t i By calibrating the extrinsic parameter relationship between the self-localization module and the camera coordinate system of camera CI offline, the rotation and translation transformation relationship between the camera coordinate system of camera CI and the world coordinate system is obtained. , t ci .

4. The multi-unmanned system cooperative long-term target tracking method according to claim 1, characterized in that, The self-localization module is a combined navigation and positioning system or a visual-inertial navigation and positioning system. All unmanned systems define the same world coordinate system and have a unified system time.

5. The multi-unmanned system cooperative long-term target tracking method according to claim 1, characterized in that, The tracking thread of the target tracking module is a target tracking algorithm that can output the pixel coordinates of the target, and the detection thread of the target tracking module is a target detection algorithm or a deep convolutional neural network algorithm that can determine whether the target exists in the entire frame image.

6. The multi-unmanned system cooperative long-term target tracking method according to any one of claims 1 to 5, characterized in that, The specific implementation process of step S2 includes: S21. Camera cj acquires the current frame image. The tracking thread's tracking sub-taper outputs the pixel coordinates of the target image. The detection thread's detection sub-taper determines whether the target exists in the entire frame image. If the target does not exist, camera cj acquires the next frame image; if the target exists, the target region is determined, and the target region image block I is cropped. cj ; S22. Initialize the tracking and detection threads of the target tracking module. If an offline target dataset is available, train the detector of the detection thread using the offline dataset. Use the detector to determine the target image region in the current frame image and initialize the tracking thread's tracker. If no offline target dataset is available, manually determine the target image region in the current frame image, and then initialize the tracker and detector using the visual features of the target region. After determining the target image region, further determine the target's 3D position p. wj .

7. The multi-unmanned system cooperative long-term target tracking method according to claim 6, characterized in that, In step S22, the three-dimensional position p of the target is determined. wj The specific implementation process includes: (1) Based on the intrinsic parameters of camera cj and the twist parameter k cj d cj The pixel homogeneous coordinates of the target image are transformed to camera coordinates according to the following formula: Where, p j =[uv 1] T p represents the homogeneous pixel coordinates of the target image. cj =[xyz] T The camera coordinates of the target image; (2) When camera cj is a monocular camera, for the j-th unmanned system, the target is located in the ray l of three-dimensional space. j Above, the ray l j The ray equation is expressed in the following world coordinate system: Where k j For the coefficient, v = [x / zy / z 1] T Let P(k) be the direction vector of the ray equation in the camera coordinate system. The ray passes through the origin and passes through P(k). cj d cj Inverse transformation yields: The target three-dimensional position p wj From the ray l where the target is located j The normalized representation of the unit direction vector, i.e. Where norm represents the normalization of the vector; (3) When the camera cj is a stereo camera or a monocular camera with an integrated range sensor, the camera coordinates p of the target image can be directly solved. cj =[xyz] T The three-dimensional position of the target in the world coordinate system is obtained as follows:

8. The multi-unmanned system cooperative long-term target tracking method according to any one of claims 1 to 5, characterized in that, The specific implementation process of step S5 includes: S51: The tracking thread first initializes, then acquires image frames, obtains tracking results through the target tracking algorithm, performs a state self-check on the results, and continues tracking if the tracking result confidence is high, and transmits the results and image frames to the detection thread; if the result confidence is low, it re-initializes. S52: The detection thread is first initialized, and then the image frames and tracking results transmitted by the tracking thread are processed to further determine the tracking status. If the tracking status is good, the detection model is further learned and updated; if the tracking status is poor, the target is re-detected from the image frame and transmitted to the tracking thread, and the tracking thread is re-initialized.

Images