A multi-target perception fusion method in a multi-unmanned aerial vehicle search process

Abstract

The application discloses a multi-target perception fusion method in a multi-unmanned aerial vehicle search process. First, an unmanned aerial vehicle A uses a camera to search and track a target, obtains a target image position and number, and calculates a three-dimensional position of each target. If no new target appears, target information in a fusion target pool is updated; otherwise, the information component of unmanned aerial vehicle A is deleted from the fusion target pool, a target pool subset to be matched is obtained, an adjacent relationship bipartite graph is constructed, the bipartite graph is simplified, and matching is performed by taking the maximum matching number and the minimum shape gap as a correlation measure. Finally, an updated multi-target fusion result is calculated from each target information in the fusion target pool. The application can realize multi-target perception and fusion in a multi-unmanned aerial vehicle search process, retains shape features of original calculation results, uses an adjacent judgment and a shape matching strategy, and solves the problem of a high correlation matching error rate of a traditional matching fusion algorithm when a target is low-speed, gathered, and the like.

Description

Technical Field

[0001] This invention belongs to the field of multi-UAV cooperative situational awareness technology, and in particular relates to a multi-target perception fusion method in the multi-UAV search process. Background Technology

[0002] In recent years, with the development of science and technology, drones have been widely used in various industries due to their low cost, high flexibility, and ease of operation. Compared with manned aircraft, drones are smaller, more economical, and pose no risk of human injury, making them more suitable for hazardous environments or environments unsuitable for direct human access. Target search using drones is a typical application scenario. Currently, this work is mainly done manually by operators and through visual observation. Large-scale searches require high manpower, resulting in low efficiency. Furthermore, when there are many targets with low distinguishability, it is difficult to match and merge numerous duplicate targets, ultimately hindering subsequent decision-making processes.

[0003] In multi-UAV cooperative search and perception fusion tasks, the fusion process of multiple targets is closely related to the UAV perception measurement results. In traditional fusion algorithms, multi-target fusion often requires accurate target position and velocity information. However, the UAV measurement process is noisy. When the target movement speed is low and the distance between them is close, the low signal-to-noise ratio makes it difficult for traditional fusion algorithms to obtain accurate similarity judgment conditions during the matching process. Therefore, it is difficult to perform association matching for such targets and to identify erroneous association matching. Therefore, multi-UAV cooperative search and perception fusion tasks require a unified and coherent method. This method should address the characteristic that the calculated target positions still maintain high shape similarity under different noise types. It should not rely excessively on the absolute value of the calculated target positions, but use adjacency judgment and shape matching as similarity judgment conditions. It should also fully consider the mutual influence of each link in the perception to fusion process to achieve an effective and continuous perception fusion method, avoiding the problems that may occur with traditional methods. Summary of the Invention

[0004] The purpose of this invention is to address the problems mentioned in the background section by achieving multi-drone, multi-view perception and fusion of multi-target information in a large scene during multi-drone target search. Therefore, a multi-target perception and fusion method is proposed for multi-drone search, enabling real-time acquisition of target positions by multiple drones, real-time multi-target fusion among multiple drones, and generation of a fused target pool to provide scene multi-target information reference for subsequent decision-making processes.

[0005] To achieve the objective of this invention, a multi-target perception fusion method is disclosed in the multi-UAV search process, comprising the following steps:

[0006] Step 1: A single UAV A acquires images from its onboard camera and performs multi-target tracking on a given search target to obtain the image positions and tracking numbers of the multiple targets;

[0007] Step 2: Calculate the positions of the multiple targets in the world coordinate system based on the multi-target image positions and camera pose information;

[0008] Step 3: Using the results from Steps 1 and 2, update the target pool information for UAV A and send it to the fusion target pool;

[0009] Step 4: Remove targets that have been lost from the fusion target pool;

[0010] Step 5: If no new target appears in the target pool of UAV A, update the information components of UAV A in the fusion target pool and proceed to step 10; if a new target appears, delete the multi-target information of the current UAV from the fusion target pool and re-match and fuse it.

[0011] Step 6: Remove the information component of UAV A from the fusion target pool to obtain the fusion target pool subset to be matched and the target pool subset to be matched for UAV A.

[0012] Step 7: Sequentially determine the adjacency relationships of each target in the fusion target pool subset to be matched and the target pool subset to be matched of UAV A, and construct an adjacency relationship bipartite graph;

[0013] Step 8: Simplify the adjacency relationship bipartite graph and perform association matching using the maximum number of matches and the minimum shape difference as association metrics;

[0014] Step 9: Directly assign unassociated targets in the UAV A target pool as new fusion targets;

[0015] Step 10: Update the position information of each target in the fusion target pool to obtain the multi-target fusion result updated by UAV A. Repeat the above steps to obtain the multi-target fusion result obtained by multiple UAVs.

[0016] Furthermore, in step 1, a multi-target tracking algorithm is used to detect and track multiple targets in the acquired image. The acquisition device has a resolution of 1920x1080. The YOLO (You Only Look Once) algorithm is used to detect the pixel position of the given search target in the image. The Simple Online and Realtime Tracking (SORT) multi-target tracking algorithm is used to correlate the detection results, and finally the pixel position of the multiple targets in the image acquired by UAV A and the corresponding tracking number are obtained.

[0017] Further, in step 2, the target pixel position and camera pose information are used to calculate the target position, wherein the camera is mounted on a stabilization gimbal and attached to the underside of the drone, specifically including the following steps:

[0018] Step 2-1: Obtain the position (X) of the projection point on the normalized image plane in front of the camera in the world coordinate system based on the target pixel position. w ,Y w Z w Its projection formula is:

[0019]

[0020] Where (u,v) are the pixel coordinates of the target in the distortion-removed image, and their values ​​are the number of horizontal and vertical pixels from the target center to the top-left corner of the image. d is the distance-depth between the normalized image plane and the camera, K is the camera's intrinsic parameter matrix, and X... w ,Y w Z w () represents the position of the projection point in the world coordinate system. Let be the extrinsic parameter matrix of the camera, whose value is equivalent to the inverse of the camera pose matrix, specifically:

[0021]

[0022] Where R c Let C be the rotation matrix corresponding to the camera's current pose, and let C be the column vector of the camera's position in the world coordinate system. Combining the above two equations, then (X... w ,Y w Z w It can be obtained directly from the following formula:

[0023]

[0024] Where d is 1m, R c The position is obtained from the attitude of the gimbal and the drone body, where C is the position of the drone, specifically the position relative to the origin of the coordinate system obtained from GPS.

[0025] Step 2-2: Taking the position in the camera's world coordinate system as the starting point, and using the projection point (X) obtained in Step 2-1... w ,Y w Z w Construct a ray that intersects the ground plane, and set the ground plane height to 0m. Then, the position of the target's projection point on the ground plane in the x and y directions of the world coordinate system is obtained by the following formula:

[0026]

[0027] Where (X,Y,Z) represents the position of the camera in the world coordinate system;

[0028] Step 2-3: Repeat step 2-1 until the position calculation of all targets in the image is completed, and finally obtain the position of multiple targets in the world coordinate system.

[0029] Furthermore, step 3 updates the target pool for UAV A. The target pool for UAV A refers to the set containing the targets searched by UAV A, including target location, location calculation variance, tracking number, and timestamp information. The update process specifically includes the following steps:

[0030] Step 3-1: Construct a new target pool for UAV A, the contents of which are the positions, tracking numbers and current timestamps of multiple targets in the world coordinate system obtained from Step 1 and Step 2;

[0031] Step 3-2: Record targets whose tracking numbers appear in the old target pool but not in the new target pool as lost targets in the current search process of UAV A, and delete them from the new target pool;

[0032] Step 3-3: Record targets whose tracking numbers appear in the new target pool but not in the old target pool, delete targets that do not meet the initial conditions, and use the remaining targets as new targets to be acquired during the current search process of UAV A. The specific process is as follows:

[0033] Step 3-3-1: Before the search process begins, construct an empty list to record the tracking number and timestamp;

[0034] Step 3-3-2: Traverse the list and delete the tracking numbers and their corresponding timestamps that do not appear in the new target pool;

[0035] Step 3-3-3: Traverse the list and compare the duration of the timestamps recorded in the list with the duration of the current timestamp. If the duration is greater than the set threshold, delete the timestamp and its corresponding tracking number, and mark the drone as having captured a new target at the current moment. Otherwise, delete the target with the corresponding tracking number from the new target pool.

[0036] Step 3-3-4: Traverse the new target pool and add the tracking number and current timestamp of the target whose tracking number is not in the list to the list;

[0037] Steps 3-4: Use the new target pool as the current UAV A target pool to complete a UAV target pool update, and estimate the variance σ of each target position based on the distance of the targets in the target pool from the UAV. 2 This serves as a basis for adjacency determination. Specifically, according to the 3σ criterion, the Euclidean distance between the target's calculated position under the current state and its actual position is no greater than 3σ, where the value of σ is obtained by the following formula:

[0038]

[0039] Where, σ e The straight-line distance l between the target and the drone e The standard deviation of the position calculation error, k e Let l be the coefficient of variation of the standard deviation with the straight-line distance between the target and the UAV, and l be the straight-line distance between the UAV and the target for calculating the three-dimensional coordinates.

[0040] Furthermore, in step 4, the fusion target pool refers to a set of fused targets obtained by fusing the perception results of multiple UAV targets. The fused target includes an association set, fused target location, fused target location variance, fusion number, and timestamp information. The association set is the set of corresponding targets in each UAV target pool associated with the fused target. The specific formulas for calculating the fused target location and the fused target location variance are as follows:

[0041]

[0042] Where μ i and J represents the position and variance of the fusion target i, respectively. i Let J be the set of associations for fusion target i, where j∈J i For the associated target index, μ j and Here, represents the location and variance of the associated target j, respectively. This fusion process is similar to multiplying and superimposing multiple location information that follow a normal distribution.

[0043] In step 4, if the association set of a certain fusion target contains only one target of UAV A, and that target is the lost target in step 3-2, then the fusion target is deleted, the fusion target pool is traversed, and the deletion of the tracked lost target in the fusion target pool is completed.

[0044] Furthermore, in step 5, if there are no newly captured targets in the current UAV target pool, the information components of UAV A in the fusion target pool are updated directly. That is, the target position, target position calculation variance, and timestamp information of the targets associated with UAV A in the fusion target pool are updated. The fusion target pool is traversed to update the information components of UAV A in the fusion target pool, and then the process proceeds to step 10.

[0045] Further, in step 6, the information component of UAV A is deleted from the fusion target pool to obtain the fusion target pool subset to be matched and the target pool subset to be matched for UAV A. Specifically, this includes the following steps:

[0046] Step 6-1: Construct the fusion target pool subset to be matched and the target pool subset to be matched for UAV A;

[0047] Step 6-2: Fill all targets in the target pool of UAV A into the target pool subset to be matched by UAV A;

[0048] Step 6-3: Traverse the fusion target pool. If the association set of a certain fusion target contains a target from UAV A, and the number of associations in the association set of the fusion target is greater than 1, then delete the association from UAV A from the association set and recalculate the position and variance of the fusion target. Add the fusion target to the fusion target pool subset to be matched and go to step 6-5.

[0049] Step 6-4: If the association set of a certain fusion target only contains targets from drone A, then remove this target from drone A from the target pool set to be matched by drone A.

[0050] Step 6-5: Repeat step 6-3 until all fusion targets have been traversed, resulting in the final subset of fusion target pools to be matched and the subset of target pools to be matched for UAV A.

[0051] Furthermore, step 7 constructs an adjacency bipartite graph using adjacency relationship judgment, specifically including the following steps:

[0052] Step 7-1: Construct an empty bipartite graph G = (X, Y), where X stores targets from the target pool set to be matched from UAV A, and Y stores targets from the target pool set to be matched and fused.

[0053] Step 7-2: Take a target from the target pool set of drone A to be matched;

[0054] Step 7-3: Traverse the fusion target pool. If a target satisfies an adjacency relationship with a target in the subset of the target pool to be matched, add the target to X and add the adjacent fusion targets to Y. If the target has already been added to Y, ignore it. At the same time, connect the target to all fusion targets with adjacency relationships in G. The adjacency relationship is determined as follows:

[0055]

[0056] Where x i ,y i ,σ i To fuse the position and standard deviation of target i, x, y, σ represent the position and standard deviation of the target;

[0057] Step 7-4: Repeat step 7-2 until all targets in the target pool set to be matched by UAV A are removed, resulting in the final bipartite graph G.

[0058] Furthermore, in step 8, the bipartite graph obtained in step 7 is simplified and shape matching is performed according to the maximum matching principle to construct the association between the targets in the UAV A target pool and the fused targets in the fused target pool. Specifically, this includes the following steps:

[0059] Step 8-1: Construct an empty bipartite graph G′=(XN,Y′), where X′ stores targets from X and Y′ stores targets from Y;

[0060] Step 8-2: Construct an empty set S to record the target to be searched next, and select a target from the X set of the bipartite graph G and put it into S. If the X set is empty, the algorithm ends.

[0061] Step 8-3: Find all targets in S in the X set of the bipartite graph G, select all the merged targets connected to it from the Y set of the bipartite graph G as the new set S, and move all the found targets from X to X′. If S is an empty set, go to step 8-5.

[0062] Step 8-4: Find all fusion targets in set S in set Y of bipartite graph G. Select all targets connected to it from set X of bipartite graph G as a new set S. Move all the fusion targets found from Y to YN. If S is not an empty set, go to step 8-3.

[0063] Step 8-5: Use the Hungarian algorithm to obtain the maximum number of pairs n in the bipartite graph G′. If n is 0, go to step 8-2.

[0064] Step 8-6: Traverse all pairs with number n in G′, and select a pair as the matching relationship between the target in the UAV target pool and the fused target in the fused target pool, based on the minimum shape difference as the association metric. The shape difference is specifically defined as:

[0065]

[0066]

[0067] in and E shape Let M be the overall offset error vector and shape difference of this pairing, M be the target set corresponding to the X′ set in the bipartite graph G′ in this pairing, f(·) be the matching mapping relationship of this pairing, and x m and y m For the solution position x and y of the target m, x f(m) and y f(m) Let x and y be the positions of the corresponding fused targets paired with target m, and λ be the offset error influence coefficient.

[0068] Step 8-7: Based on the pairing obtained in Step 8-6, associate the targets in the target pool of UAV A with the targets in the fused target pool that are paired with it. If the X set is empty, the algorithm ends; otherwise, go to Step 8-1.

[0069] Furthermore, in step 9, all targets in the target pool of UAV A that are not associated with the fusion targets in the fusion target pool are directly assigned as new fusion targets and added to the fusion target pool. In step 10, the position and variance of each fusion target in the fusion target pool are recalculated to obtain the multi-target fusion result updated by UAV A. Repeating the above steps for other UAVs B, C, etc., can obtain the multi-target fusion result obtained by continuous updates from multiple UAVs.

[0070] Compared with the prior art, the significant advancements of this invention are: 1) This invention is a multi-target perception fusion method for multi-UAV search processes. A typical application scenario is multi-UAV top-down search for multiple moving targets. It can automatically generate fusion results based on UAV search results and reflect the position and movement status of the real targets in real time; 2) In areas where the field of view of multiple UAVs overlaps locally, this invention avoids the problem of difficulty in matching low-speed, clustered targets in traditional fusion algorithms through adjacency judgment and shape matching; 3) This invention can run continuously, is insensitive to the order in which UAVs update search results, and does not require specifying the number of search UAVs and the number of targets.

[0071] To more clearly illustrate the functional characteristics and structural parameters of the present invention, further explanation is provided below in conjunction with the accompanying drawings and specific embodiments. Attached Figure Description

[0072] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings:

[0073] Figure 1 This is a flowchart illustrating a multi-target perception fusion method in a multi-UAV search process;

[0074] Figure 2 This is a schematic diagram illustrating the results of single-UAV target detection and multi-target tracking.

[0075] Figure 3 This is a schematic diagram showing the target positions and their maximum error range in the fusion target pool subset to be matched and the UAV target pool subset to be matched in a two-dimensional plane;

[0076] Figure 4 This is a bipartite graph representing the adjacency relationships constructed during the fusion process.

[0077] Figure 5A simplified diagram of the adjacency relationship bipartite graph and the remaining part of the original bipartite graph;

[0078] Figure 6 This is a schematic diagram showing the target positions and their maximum error range in the fused target pool after fusion. Detailed Implementation

[0079] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0080] A multi-target perception fusion method for multi-UAV search process, characterized by the following steps:

[0081] Step 1: Use a multi-target tracking algorithm to perform multi-target detection and tracking on the acquired images. The acquisition device has a resolution of 1920x1080. The YOLO algorithm is used to detect the pixel position of the given search target in the image. The SORT multi-target tracking algorithm is used to correlate the detection results and finally obtain the pixel position of the multiple targets in the image acquired by UAV A and the corresponding tracking number.

[0082] Step 2: Calculate the target position based on the target pixel position and camera pose information. The camera is mounted on a stabilization gimbal and attached to the underside of the drone. This includes the following steps:

[0083] Step 2-1: Obtain the position (X) of the projection point on the normalized image plane in front of the camera in the world coordinate system based on the target pixel position. w ,Y w Z w Its projection formula is:

[0084]

[0085] Where (u,v) are the pixel coordinates of the target in the distortion-removed image, and their values ​​are the number of horizontal and vertical pixels from the target center to the top left corner of the image. d is the distance-depth between the normalized image plane and the camera, and K is the camera's intrinsic parameter matrix. w ,Y w Z w () represents the position of the projection point in the world coordinate system. Let be the extrinsic parameter matrix of the camera, whose value is equivalent to the inverse of the camera pose matrix, specifically:

[0086]

[0087] Where R c Let C be the rotation matrix corresponding to the camera's current pose, and let C be the column vector of the camera's position in the world coordinate system. Combining the above two equations, then (X... w ,Y w Z w It can be obtained directly from the following formula:

[0088]

[0089] Where d is 1m, R c The position is obtained from the attitude of the gimbal and the drone body, where C is the position of the drone, specifically the position relative to the origin of the coordinate system obtained from GPS.

[0090] Step 2-2: Taking the position in the camera's world coordinate system as the starting point, and using the projection point (X) obtained in Step 2-1... w ,Y w Z w Construct a ray that intersects the ground plane, taking the ground plane height as 0m. Then, the position of the target's projection point on the ground plane in the x and y directions of the world coordinate system can be obtained by the following formula:

[0091]

[0092] Where (X,Y,Z) represents the position of the camera in the world coordinate system;

[0093] Step 2-3: Repeat step 2-1 until the position calculation of all targets in the image is completed, and finally obtain the position of multiple targets in the world coordinate system.

[0094] Step 3: Update the target pool for UAV A. The target pool for UAV A refers to the set of targets searched by UAV A, including target location, location variance, tracking number, and timestamp information. The update process specifically includes the following steps:

[0095] Step 3-1: Construct a new target pool for UAV A. The content of the pool is given by the results obtained in Step 1 and Step 2, specifically including the position of multiple targets in the world coordinate system, tracking number, and current timestamp.

[0096] Step 3-2: Record targets whose tracking numbers appear in the old target pool but not in the new target pool as lost targets in the current search process of UAV A, and delete them from the new target pool;

[0097] Step 3-3: Record targets whose tracking numbers appear in the new target pool but not in the old target pool, delete targets that do not meet the initial conditions, and use the remaining targets as new targets to be acquired during the current search process of UAV A. The specific process is as follows:

[0098] Step 3-3-1: Before the search process begins, construct an empty list to record the tracking number and timestamp;

[0099] Step 3-3-2: Traverse the list and delete the tracking numbers and their corresponding timestamps that do not appear in the new target pool;

[0100] Step 3-3-3: Traverse the list and compare the time interval between the timestamps recorded in the list and the current timestamp. If the interval is greater than the set threshold, delete the timestamp and its corresponding tracking number, and mark the drone as having captured a new target at the current moment. Otherwise, delete the target with the corresponding tracking number from the new target pool.

[0101] Step 3-3-4: Traverse the new target pool and add the tracking number and current timestamp of the target whose tracking number is not in the list to the list;

[0102] Steps 3-4: Use the new target pool as the current UAV target pool to complete a UAV target pool update, and estimate the variance σ of each target position based on the distance of the targets in the target pool from the UAV. 2 This serves as a basis for adjacency determination, specifically meaning that, according to the 3σ criterion, the Euclidean distance between the target's calculated position under the current state and its actual position is no greater than 3σ. The value of σ is obtained from the following formula:

[0103]

[0104] Where, σ e The straight-line distance l between the target and the drone e The standard deviation of the position calculation error, k e This is the coefficient of variation of the standard deviation with respect to the straight-line distance between the target and the UAV.

[0105] Step 4: If the association set of a certain fused target contains only one target of UAV A, and this target is the lost target in Step 3-2, then delete the fused target. Iterate through the fused target pool to complete the deletion of the tracked lost targets from the fused target pool. The fused target pool refers to a set containing multiple fused targets. Each fused target contains information such as the association set, fused target location, fused target location variance, and fused number. The association set is the set of corresponding targets in each UAV target pool associated with the fused target. The specific formulas for calculating the fused target location and fused target location variance are as follows:

[0106]

[0107] Where μ i and J represents the position and variance of the fusion target i, respectively. i For the set of associations of fusion target i, μ jand These are the position and variance of the corresponding target i associated with the target in a certain UAV target pool. This fusion process is similar to multiplying and superimposing multiple position information that follow a normal distribution.

[0108] Step 5: If there are no newly captured targets in the current UAV target pool, directly update the information components of UAV A in the fusion target pool. That is, update the target position, target position calculation variance, timestamp, and other information of the targets associated with UAV A in the fusion target pool. Traverse the fusion target pool to complete the update of the information components of UAV A in the fusion target pool, and then go to step 10.

[0109] Step 6: Remove the information component of UAV A from the fusion target pool to obtain the fusion target pool subset to be matched and the target pool subset to be matched for UAV A. This includes the following steps:

[0110] Step 6-1: Construct the fusion target pool subset to be matched and the target pool subset to be matched for UAV A;

[0111] Step 6-2: Fill all targets in the target pool of UAV A into the target pool subset to be matched by UAV A;

[0112] Step 6-3: Traverse the fusion target pool. If the association set of a certain fusion target contains a target from UAV A, and the number of associations in the association set of the fusion target is greater than 1, then delete the association from UAV A from the association set and recalculate the position and variance of the fusion target. The specific calculation formula is shown in the relevant calculation formula in step 4. Then add the fusion target to the fusion target pool subset to be matched and go to step 6-5.

[0113] Step 6-4: If the association set of a certain fusion target only contains targets from drone A, then remove this target from drone A from the target pool set to be matched by drone A.

[0114] Step 6-5: Repeat step 6-3 until all fusion targets have been traversed, resulting in the final subset of fusion target pools to be matched and the subset of target pools to be matched for UAV A.

[0115] Step 7: An adjacency relationship bipartite graph was constructed using the adjacency relationship judgment method. This includes the following steps:

[0116] Step 7-1: Construct an empty bipartite graph G = (X, Y) and an empty bipartite graph G′ = (X′, Y′), where X′ stores targets from X and Y′ stores targets from Y.

[0117] Step 7-2: Take a target from the target pool set of drone A to be matched;

[0118] Step 7-3: Traverse the fusion target pool. If a target satisfies an adjacency relationship with a target in the subset of the target pool to be matched, add the target to X and add the adjacent fusion targets to Y (ignore targets already added to Y). Simultaneously, connect the target to all adjacent fusion targets in G. The adjacency relationship is determined as follows:

[0119]

[0120] Where x i ,y i ,σ i To fuse the position and standard deviation of target i, x, y, σ represent the position and standard deviation of the target;

[0121] Step 7-4: Repeat step 7-2 until all targets in the target pool set to be matched by UAV A are removed, resulting in the final bipartite graph G.

[0122] Step 8: Simplify the bipartite graph obtained in Step 7 and perform shape matching according to the maximum matching principle to construct the association between the targets in the UAV A target pool and the fused targets in the fused target pool. This specifically includes the following steps:

[0123] Step 8-1: Construct an empty bipartite graph G′=(X′,Y′), where X′ stores targets from the UAV A target pool and Y′ stores fused targets from the fused target pool;

[0124] Step 8-2: Construct an empty set S to record the target to be searched next, and select a target from the X set of the bipartite graph G and put it into S. If the X set is empty, the algorithm ends.

[0125] Step 8-3: Find all targets in S in the X set of the bipartite graph G, select all the merged targets connected to it from the Y set of the bipartite graph G as the new set S, and move all the found targets from X to X′. If S is an empty set, go to step 8-5.

[0126] Step 8-4: Find all fusion targets in set S in set Y of bipartite graph G. Select all targets connected to it from set X of bipartite graph G as a new set S. Move all the fusion targets found from Y to Y′. If S is not an empty set, go to step 8-3.

[0127] Step 8-5: Use the Hungarian algorithm to obtain the maximum number of pairs n in the bipartite graph G′. If n is 0, go to step 8-2.

[0128] Step 8-6: Traverse all pairs with number n in G′, and select a pair as the matching relationship between the target in the UAV target pool and the fused target in the fused target pool, based on the minimum shape difference as the association metric. The shape difference is specifically defined as:

[0129]

[0130]

[0131] in and E shape Let M be the overall offset error vector and shape difference of this pairing, M be the target set corresponding to the X′ set in the bipartite graph G′ in this pairing, f(·) be the matching mapping relationship of this pairing, and x m and y m For the solution position x and y of the target m, x f(m) and y f(m) Let x and y be the positions of the corresponding fused targets paired with target m, and λ be the offset error influence coefficient.

[0132] Step 8-7: Based on the pairing obtained in Step 8-6, associate the targets in the target pool of UAV A with the targets in the fused target pool that are paired with it. If the X set is empty, the algorithm ends; otherwise, go to Step 8-1.

[0133] Step 9: Directly assign all targets in the UAV A target pool that are not associated with any fusion targets in the fusion target pool as new fusion targets and add them to the fusion target pool.

[0134] Step 10: Recalculate the position and variance of each fusion target in the fusion target pool for other UAVs B, C, etc. The specific calculation formula is shown in the relevant calculation formula in Step 4. The multi-target fusion result updated by UAV A is obtained. Repeat the above steps to obtain the multi-target fusion result obtained by continuous updates from multiple UAVs.

[0135] Example

[0136] Combination Figure 1 The multi-target perception fusion method proposed in this invention for UAV search process includes the following steps:

[0137] Step 1: Use a multi-target tracking algorithm to detect and track multiple targets in the acquired images, obtaining the pixel positions and corresponding tracking numbers of the multiple targets in the images acquired by UAV A. The multi-target detection and tracking results are as follows: Figure 2 As shown, the dashed box represents the target pixel position obtained by multi-target detection, and A1 to A6 represent the tracking numbers obtained by multi-target tracking.

[0138] Step 2: Calculate the target position based on the target pixel position and camera pose information to obtain the position of multiple targets in the world coordinate system.

[0139] Step 3: Update the target pool for drone A to obtain a new target pool for drone A.

[0140] Step 4: Remove targets that have been lost from the fusion target pool;

[0141] Step 5: If a new target is captured in the current drone target pool, continue to Step 6 to complete the matching and fusion process.

[0142] Step 6: Remove the information component of UAV A from the fusion target pool to obtain the fusion target pool subset to be matched and the target pool subset to be matched for UAV A. Their representation in the xOy coordinate system is as follows: Figure 3 As shown, the dashed circle represents the maximum error range of the target. The fused target pool is {1, 2, 3, 4, 5, 6}, and the target pool to be matched is {A1, A2, A3, A4, A5, A6}. At this time, the subsets of the fused target pool and the subsets of the target pool to be matched are the same as the fused target pool and the target pool to be matched, respectively.

[0143] Step 7: Construct a bipartite graph with adjacency relationships using the adjacency relationship judgment method, where... Figure 3 The bipartite graph G obtained from the target information is... Figure 4 As shown.

[0144] Step 8: Simplify the bipartite graph obtained in Step 7 and perform shape matching according to the maximum matching principle, where... Figure 4 After simplification, the bipartite graph G can be obtained as the simplified bipartite graph G′ and the remaining part of the bipartite graph G, as shown below. Figure 5 As shown, the maximum number of matches in the bipartite graph G′ is 3. The matching results obtained by using the minimum shape difference as the association metric are {A1-2, A2-3, A3-4}. The matching results of the remaining part of the bipartite graph G are {A5-6}. The association between the UAV A target pool target and the fused target in the fused target pool is constructed based on the above association relationship.

[0145] Step 9: Directly assign all targets in the UAV A target pool that are not associated with any fusion targets in the fusion target pool as new fusion targets and add them to the fusion target pool.

[0146] Step 10: Recalculate the position and variance of each target in the fusion target pool. The specific calculation formula is shown in the relevant calculation formula in Step 4, obtaining the multi-target fusion result updated by UAV A. The fusion result obtained by updating by UAV A is as follows: Figure 6 As shown, in addition to the targets matched in step 8, targets A4 and A6 are assigned as new fusion targets 7 and 8, and the dashed circle represents the maximum error of the fused targets.

[0147] Repeating the above steps for other drones B, C, etc., yields the multi-target fusion result obtained through continuous updates from multiple drones. It should be noted that in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus.

[0148] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A multi-target perception fusion method in a multi-UAV search process, characterized in that, Includes the following steps: Step 1: A single UAV A acquires images from its onboard camera and performs multi-target tracking on a given search target to obtain the image positions and tracking numbers of the multiple targets; Step 2: Calculate the positions of the multiple targets in the world coordinate system based on the multi-target image positions and camera pose information; Step 3: Using the results from Steps 1 and 2, update the target pool information for UAV A and send it to the fusion target pool; Step 4: Remove targets that have been lost from the fusion target pool; Step 5: If no new target appears in the target pool of UAV A, update the information components of UAV A in the fusion target pool and proceed to step 10; if a new target appears, delete the multi-target information of the current UAV from the fusion target pool and re-match and fuse it. Step 6: Remove the information component of UAV A from the fusion target pool to obtain the fusion target pool subset to be matched and the target pool subset to be matched for UAV A. Step 7: Sequentially determine the adjacency relationships of each target in the fusion target pool subset to be matched and the target pool subset to be matched of UAV A, and construct an adjacency relationship bipartite graph; Step 8: Simplify the adjacency relationship bipartite graph and perform association matching using the maximum number of matches and the minimum shape difference as association metrics; Step 9: Directly assign unassociated targets in the UAV A target pool as new fusion targets; Step 10: Update the position information of each target in the fusion target pool to obtain the multi-target fusion result updated by UAV A. Repeat the above steps to obtain the multi-target fusion result obtained by multiple UAVs. Step 3 involves updating the target pool for UAV A. The target pool for UAV A refers to the set of targets searched by UAV A, including target location, location variance, tracking number, and timestamp information. The update process specifically includes the following steps: Step 3-1: Construct a new target pool for UAV A, the contents of which are the positions, tracking numbers and current timestamps of multiple targets in the world coordinate system obtained from Step 1 and Step 2; Step 3-2: Record targets whose tracking numbers appear in the old target pool but not in the new target pool as lost targets in the current search process of UAV A, and delete them from the new target pool; Step 3-3: Record targets whose tracking numbers appear in the new target pool but not in the old target pool, delete targets that do not meet the initial conditions, and use the remaining targets as new targets to be acquired during the current search process of UAV A. The specific process is as follows: Step 3-3-1: Before the search process begins, construct an empty list to record the tracking number and timestamp; Step 3-3-2: Traverse the list and delete the tracking numbers and their corresponding timestamps that do not appear in the new target pool; Step 3-3-3: Traverse the list and compare the duration of the timestamps recorded in the list with the duration of the current timestamp. If the duration is greater than the set threshold, delete the timestamp and its corresponding tracking number, and mark the drone as having captured a new target at the current moment. Otherwise, delete the target with the corresponding tracking number from the new target pool. Step 3-3-4: Traverse the new target pool and add the tracking number and current timestamp of the target whose tracking number is not in the list to the list; Step 3-4, taking the new target pool as the current UAV A target pool, completing a UAV target pool update, and estimating the position solution variance of each target according to the distance of the target in the target pool from the UAV and as the basis for adjacent judgment, specifically, according to criteria, the Euclidean distance between the target position obtained in the current state and the actual position of the target is not greater than wherein the value is obtained by the following formula: wherein, is the straight-line distance of the UAV from the target is the standard deviation of the position solution error, is the coefficient of the standard deviation of the position solution error as a function of the straight-line distance of the UAV from the target, and l is the straight-line distance of the UAV from the target for which the three-dimensional coordinates are solved. In step 4, the fusion target pool refers to a set of fused targets obtained by fusing the perception results of multiple UAV targets. A fused target includes an association set, fused target location, fused target location variance, fusion number, and timestamp information. The association set is the set of corresponding targets in each UAV target pool associated with that fused target. The specific formulas for calculating the fused target location and the fused target location variance are as follows: wherein and are the position and variance of the fusion target , is the association set of the fusion target , is the association target index, and are the position and variance of the association target , the fusion process is similar to the product superposition of multiple position information subject to normal distribution; In step 4, if the association set of a certain fusion target contains only one target of UAV A, and that target is the lost target in step 3-2, then the fusion target is deleted, the fusion target pool is traversed, and the deletion of the tracked lost target in the fusion target pool is completed. In step 5, if no new target is captured in the target pool of UAV A, the information components of UAV A in the fusion target pool are updated directly. That is, the target position, target position calculation variance, and timestamp information of the targets associated with UAV A in the fusion target pool are updated. The fusion target pool is traversed to complete the update of the information components of UAV A in the fusion target pool, and then the process proceeds to step 10.

2. The method of claim 1, wherein, In step 1, a multi-target tracking algorithm is used to detect and track multiple targets in the acquired image. The acquisition device has a resolution of 1920x1080. The YOLO algorithm is used to detect the pixel position of the given search target in the image. The SORT multi-target tracking algorithm is used to correlate the detection results, and finally the pixel position of the multiple targets in the image acquired by UAV A and the corresponding tracking number are obtained.

3. The method of claim 1, wherein, Step 2 involves calculating the target position based on the target pixel position and camera pose information. The camera is mounted on a stabilization gimbal and attached to the underside of the drone. This process includes the following steps: Step 2-1, get the position of the projection point on the normalized image plane in front of the camera in the world coordinate system from the target pixel position The projection formula is: wherein is the pixel coordinate in the image after removing the distortion, whose value is the horizontal and vertical pixel number from the top-left corner of the image, is the depth of the distance between the normalized image plane and the camera, is the intrinsic matrix of the camera, is the position of the projection point in the world coordinate system, is the extrinsic matrix of the camera, whose value is equivalent to the inverse matrix of the camera pose matrix, specifically: wherein R is a rotation matrix corresponding to the camera rotation to the current pose, is a position column vector of the camera in the world coordinate system, combining the above two equations, then is directly obtained from the following equation: wherein is 1m, obtained from the gimbal and the unmanned aerial vehicle body attitude, is the unmanned aerial vehicle position, specifically the relative position of the position obtained from the GPS and the coordinate origin; Step 2-2, taking the position in the camera world coordinate system as the starting point, constructing a ray and intersecting with the ground plane, setting the ground plane height as 0 m, then the position of the projection point of the target on the ground plane in the x, y direction of the world coordinate system is obtained by the following formula: Step 2-2, taking the position in the camera world coordinate system as the starting point, constructing a ray and intersecting with the ground plane, setting the ground plane height as 0 m, then the position of the projection point of the target on the ground plane in the x, y direction of the world coordinate system is obtained by the following formula: wherein is the position of the camera in the world coordinate system; Step 2-3: Repeat step 2-1 until the position calculation of all targets in the image is completed, and finally obtain the position of multiple targets in the world coordinate system.

4. The method of claim 1, wherein, Step 6 involves removing the information component of UAV A from the fusion target pool to obtain the fusion target pool subset to be matched and the target pool subset to be matched for UAV A. This specifically includes the following steps: Step 6-1: Construct the fusion target pool subset to be matched and the target pool subset to be matched for UAV A; Step 6-2: Fill all targets in the target pool of UAV A into the target pool subset to be matched by UAV A; Step 6-3: Traverse the fusion target pool. If the association set of a certain fusion target contains a target from UAV A, and the number of associations in the association set of the fusion target is greater than 1, then delete the association from UAV A from the association set and recalculate the position and variance of the fusion target. Add the fusion target to the fusion target pool subset to be matched and go to step 6-5. Step 6-4: If the association set of a certain fusion target only contains targets from drone A, then remove this target from drone A from the target pool set to be matched by drone A. Step 6-5: Repeat step 6-3 until all fusion targets have been traversed, resulting in the final subset of fusion target pools to be matched and the subset of target pools to be matched for UAV A.

5. The method of claim 1, wherein, Step 7 constructs an adjacency bipartite graph using adjacency relationship judgment, specifically including the following steps: Step 7-1, constructing an empty bipartite graph wherein storing targets from the subset of the target pool of the UAV A to be matched, storing targets from the subset of the target pool to be matched; Step 7-2: Take a target from the target pool set of drone A to be matched; Step 7-3, traversing the fusion target pool, if the target has adjacent relationship with the target in the subset of the target pool to be matched, the target is added to the subset of the target pool to be matched Step 7-3, traversing the fusion target pool, if the target has adjacent relationship with the target in the subset of the target pool to be matched, the target is added to the subset of the target pool to be matched Step 7-3, traversing the fusion target pool, if the target has adjacent relationship with the target in the subset of the target pool to be matched, the target is added to the subset of the target pool to be matched Step 7-3, traversing the fusion target pool, if the target has adjacent relationship with the target in the subset of the target pool to be matched, the target is added to the subset of the target pool to be matched Step 7-3, traversing the fusion wherein is the position and standard deviation of the fusion target is the position and standard deviation of the fusion target is the position and standard deviation of the fusion target Step 7-4, repeat step 7-2 until all targets in the subset of the target pool that drone A is waiting to match are taken out, resulting in a final bipartite graph .

6. The method of claim 5, wherein, Step 8 simplifies the bipartite graph obtained in Step 7 and performs shape matching according to the maximum matching principle to construct the association between the targets in the UAV A target pool and the fused targets in the fused target pool. Specifically, this includes the following steps: Step 8-1, constructing an empty bipartite graph wherein storing the target from storing the target from storing the target from storing the target from Step 8-2: Construct an empty set Record the target you are looking for and analyze it from the bipartite graph. of Select a target from the set and put it into In the middle, if The algorithm terminates when the set is empty. Step 8-3, find all targets in the set of bipartite graph Step 8-5, if the set is empty, go to Step 8-6​​​​​​​​ Step 8-4, find all fusion targets in the bipartite graph Step 8-3, if the set is not empty, go to Step 8-4​​​​​​​​ Step 8-5 - Obtain the maximum number of pairings n of the bipartite graph using the Hungarian algorithm, and go to step 8-2 if n is 0. Step 8-5 - Obtain the maximum number of pairings n of the bipartite graph using the Hungarian algorithm, and go to step 8-2 if n is 0. Step 8-6, traverse all In the pairing of n, select a set of pairs as the matching relationship between the targets in the UAV target pool and the fusion targets in the fusion target pool according to the minimum shape difference as the correlation measure, wherein the shape difference is specifically defined as: in and The overall offset error vector and shape difference for this pairing. This pair is located in the bipartite graph. of The target set corresponding to the set, This is the matching mapping relationship for this pairing. and To calculate the solution positions x and y for target m, and For the positions x and y of the corresponding fusion target paired with target m, This is the offset error influence coefficient; Step 8-7: Based on the pairing obtained in Step 8-6, associate the targets in the target pool of UAV A with the targets in the fused target pool that are paired with it. If the set is empty, the algorithm ends; otherwise, proceed to step 8-1.

7. The multi-target perception fusion method in a multi-UAV search process according to claim 1, characterized in that, In step 9, all targets in the target pool of UAV A that are not associated with the fusion targets in the fusion target pool are directly assigned as new fusion targets and added to the fusion target pool. In step 10, the position and variance of each fusion target in the fusion target pool are recalculated to obtain the multi-target fusion result updated by UAV A. Repeating the above steps can obtain the multi-target fusion result obtained by continuous updates from multiple UAVs.

Images