Aircraft cluster cooperative positioning method based on multi-view indirect observation

Abstract

The application discloses a kind of aircraft cluster cooperative positioning method based on multi-view indirect observation, 2D-3D correspondence is constructed based on semantic target in shared field of view, and the linear initial value of relative pose is solved efficiently using EPnP algorithm, which overcomes the problem that blind iteration is easy to fall into local minimum value;On this basis, a nonlinear cost function based on reprojection error is constructed, and the Levenberg-Marquardt algorithm is used to perform high-precision iterative optimization on the rotation matrix and translation vector, which significantly improves the relative positioning accuracy and robustness of the aircraft cluster under complex dynamic configuration.

Description

Technical Field

[0001] This invention relates to the field of collaborative positioning and navigation technology for aircraft swarms, and in particular to a collaborative positioning method for aircraft swarms based on multi-view indirect observation. Background Technology

[0002] In recent years, with the rapid development of aircraft technology, the demand for collaborative applications in complex environments has been increasing. High-precision relative positioning is the foundation for multiple aircraft to perform collaborative tasks.

[0003] Traditional collaborative positioning for aircraft swarms typically relies on individual aircraft autonomous positioning and direct relative measurements, such as positioning methods based on GNSS, inertial measurement units, and UWB. While these methods are applicable in open environments, they are prone to positioning drift, error accumulation, and even collaborative positioning failure in scenarios with GNSS rejection, dynamic interference, feature duplication, or missing or limited direct line-of-sight measurements. This leads to a degradation to single-aircraft navigation, making it difficult to meet the requirements of high-precision, high-robustness collaborative missions.

[0004] Unlike traditional methods that rely solely on geometric feature matching or direct line-of-sight ranging between nodes, this invention achieves "indirect observation" by collaboratively sharing semantic targets within the field of view. It utilizes the multi-constraint Hungarian algorithm to construct robust cross-view semantic associations and combines EPnP with nonlinear optimization cascaded solutions to calculate relative poses to achieve passive, high-precision collaborative localization without direct interaction. This results in overcoming direct line-of-sight constraints and maintaining high positioning accuracy and robustness even in complex rejection environments such as weak textures and dynamic interference. Summary of the Invention

[0005] The technical problem this invention aims to solve is to address the shortcomings of direct observation-based ranging and direction finding methods such as UWB, which are rendered ineffective or unable to obtain relative position information due to interference from complex electromagnetic environments, obstructed lines of sight caused by obstacles, or limited communication links. This invention provides a collaborative localization method for aircraft swarms based on multi-view indirect observation. By utilizing the indirect observation of common semantic targets in the environment by swarm aircraft, robust semantic geometric constraints are constructed across views. This enables high-precision collaborative pose calculation even when there is no direct relative measurement data between nodes, thereby significantly improving the survivability and collaborative localization robustness of the swarm in denied environments.

[0006] To solve the above-mentioned technical problems, the present invention adopts the following technical solution:

[0007] A collaborative localization method for an aircraft swarm based on multi-view indirect observation, wherein the aircraft swarm comprises q aircraft capable of acquiring images, and among the q image-acquiring aircraft are several reference aircraft with autonomous localization capabilities and several aircraft to be localized, the method includes the following steps:

[0008] Let P be the set of the aircraft that have been located in the aircraft cluster, and Q be the set of the aircraft to be located;

[0009] Step 1), select an unpositioned aircraft j from Q;

[0010] Step 2) Construct a common view of the aircraft j to be located, determine the reference aircraft i with the optimal common view relationship with the aircraft j based on the common view, and obtain the multi-view semantic target set of aircraft i and aircraft j and their attribute features.

[0011] Step 3), determine the time difference between the coordination of the reference aircraft i and the aircraft j to be located and the motion state of the semantic target set;

[0012] Step 4) Construct a set of target features to be matched based on the judgment results;

[0013] Step 5), calculate the comprehensive matching cost matrix of semantic target pairs between aircraft i and aircraft j;

[0014] Step 6) Solve the optimal semantic target matching relationship between aircraft i and aircraft j based on the multi-constraint Hungarian algorithm;

[0015] Step 7), calculate the initial relative pose of the reference aircraft i and the aircraft j to be positioned based on the EPnP algorithm;

[0016] Step 8) Iteratively solve the initial relative pose value based on a nonlinear optimization algorithm, and output the relative pose parameters after iterative convergence;

[0017] Step 9) Determine if the set of aircraft to be located is empty. If it is empty, the aircraft cluster positioning task is completed. If it is not empty, transfer aircraft j from set Q to set P and jump to step 1) to locate the next aircraft to be located.

[0018] As a further optimization of the collaborative localization method for aircraft swarms based on multi-view indirect observation of the present invention, step 2) includes the following specific steps:

[0019] Step 2.1) Perform semantic target detection on images acquired by aircraft j and all reference aircraft in set P, count the types N of semantic targets, and construct the N-dimensional semantic distribution vector of aircraft j and each reference aircraft according to the following formula:

[0020] ;

[0021] ;

[0022] In the formula, Indicates the first The number of semantic targets of a class in the images acquired by aircraft j. A natural number greater than or equal to 1 and less than or equal to N; Indicates the first Class semantic goals in the first The number of images acquired by the reference aircraft greater than or equal to 1 and less than or equal to 1 natural numbers, Let P be the number of reference aircraft in set P;

[0023] Step 2.2) Calculate the semantic similarity between aircraft j and each reference aircraft in set P according to the following formula:

[0024] ;

[0025] In the formula, Indicates that aircraft j and the first Semantic similarity between benchmark aircraft;

[0026] Step 2.3), the semantic similarity between aircraft j and each reference aircraft in set P is compared with the set similarity threshold. For comparison, if the reference aircraft The semantic similarity between the aircraft j and the aircraft j is greater than If so, it is considered that the benchmark is unmanned. There is a possibility of co-location between the reference aircraft and aircraft j. Both the aircraft j and the graph are defined as nodes, and a connecting edge is established between them;

[0027] After traversing set P, the common view of aircraft j is finally constructed. Let E be the set of edges formed by all pairs of aircraft nodes that satisfy the similarity threshold condition. S is the similarity value corresponding to the aircraft pair. ;

[0028] Step 2.4), determine if the common view G is empty.

[0029] Step 2.4.1): If G is not empty, select from set S. The aircraft i with the highest similarity is used as the benchmark;

[0030] Building a benchmark aircraft i Semantic target set at time , In the formula Indicates that the reference aircraft i is in The A-th target detected at time t, where A is greater than or equal to 1 and less than or equal to 1. natural numbers, The total number of targets detected by the baseline aircraft i. In the formula This represents the pixel coordinates of the center of the h-th target detection box in the image coordinate system. This indicates the width and height of the detection frame. Indicates semantic category label, This represents the confidence level of object detection. Indicates the image acquisition timestamp;

[0031] Building the aircraft j in Semantic target set at time , In the formula Indicates that aircraft j is in The A-th target detected at time t, where A is greater than or equal to 1 and less than or equal to 1. natural numbers, The total number of targets detected by aircraft j. In the formula This represents the pixel coordinates of the center of the h-th target detection box in the image coordinate system. These represent the width and height of the detection frame, respectively. Indicates semantic category label, This represents the confidence level of object detection. Indicates the image acquisition timestamp;

[0032] (Step 2.4.2) If G is empty, it means there is no available reference, and proceed to step 9.

[0033] As a further optimization of the aircraft swarm cooperative localization method based on multi-view indirect observation of the present invention, step 3) includes the following specific steps:

[0034] Step 3.1), calculate the absolute value of the time difference between image acquisition of aircraft i and j. ;

[0035] Step 3.2), based on the historical frame data of aircraft i, calculate the target's position. Speed ​​of movement within a time period Its expression is:

[0036] ;

[0037] in, , For the same goal and The location of the detection at any given moment.

[0038] As a further optimization of the aircraft swarm cooperative localization method based on multi-view indirect observation of the present invention, step 4) includes the following specific steps:

[0039] Step 4.1), and Compare and and Comparison, The preset time synchronization threshold, The preset speed threshold;

[0040] Step 4.1.1), if And all targets Then use the target set. and As a set to be matched;

[0041] Step 4.1.2), if And there is a target :

[0042] Step 4.1.2.1), then for the target set and Perform static and dynamic separation, and Divided into:

[0043] ;

[0044] ;

[0045] In the formula, , Let i be the set of stationary targets in aircraft i and j. , Let i and j be the sets of moving targets in aircraft i and aircraft j, respectively.

[0046] Step 4.1.2.2), set the target set and As a set to be matched;

[0047] Step 4.1.3), if If so, it is determined to be a scenario with a time difference:

[0048] Step 4.1.3.1), place spacecraft i at a historical moment arrive The acquired image sequence is defined as follows:

[0049] ;

[0050] in Indicates that aircraft 1 is at time The captured images, The set of semantic targets in the corresponding image At the current moment, aircraft j Image acquisition Its detection target set is: ;

[0051] Step 4.1.3.2), for any candidate anchor point frame of aircraft i in history. Construct the total semantic matching cost between it and the current spacecraft j. :

[0052] ;

[0053] Represents the history frames of aircraft i The single-target cost between the a-th target and the b-th target of the aircraft j in the current frame, where a is greater than or equal to 1 and less than or equal to 1. For any natural number b, b is greater than or equal to 1 and less than or equal to 1. Natural numbers, spacecraft i historical frames The a-th target The b-th target in the current frame of aircraft j ; Represents the history frames of aircraft i The total number of matching pairs formed between the target in the middle and the target in the current frame of the aircraft j;

[0054] ;

[0055] in, , , , These are the weight coefficients for the preset normalized center point Euclidean distance penalty, width-to-height difference normalized scale penalty, semantic category penalty, and confidence-based linear reward. To construct the normalized Euclidean distance between the center points, ; , These are the horizontal and vertical resolutions of the images captured by the cameras on the aircraft, respectively.

[0056] To construct the scale penalty after normalizing the width-to-height difference, ; The intersection-union ratio of two detection boxes. The bounding box of the target detected by the aircraft. , ;

[0057] To construct a semantic category penalty term, ;

[0058] To construct a linear reward term based on confidence level, ;

[0059] Step 4.1.3.3), define the time difference penalty term. ;

[0060] In the formula, This represents the time difference between the current frame and the candidate anchor frame. This is the time difference weighting coefficient, used to balance semantic consistency and temporal proximity;

[0061] Step 4.1.3.4) combines the semantic matching cost with the temporal difference penalty to obtain the comprehensive evaluation function of the anchor frame. for:

[0062] ;

[0063] Step 4.1.3.5) determines the optimal anchor frame by minimizing the comprehensive cost function. and the optimal objective set :

[0064] ;

[0065] Step 4.1.3.5), set the target set and As a set to be matched.

[0066] As a further optimization of the aircraft swarm cooperative localization method based on multi-view indirect observation of the present invention, step 5) includes the following specific steps:

[0067] Step 5.1): For the set of targets to be matched, calculate the single-target cost between the a-th target in the image of aircraft i and the b-th target in the current frame of aircraft j. :

[0068] ;

[0069] Step 5.2), construct the cost matrix between the semantic objectives of aircraft i and aircraft j. Construct cost matrix C:

[0070] .

[0071] As a further optimization of the aircraft swarm cooperative localization method based on multi-view indirect observation of the present invention, step 6) includes the following specific steps:

[0072] Step 6.1) The Hungarian algorithm is used to perform linear allocation optimization on the cost matrix C to find the optimal one-to-one matching set. :

[0073] ;

[0074] Where M represents the set of matching relationships that satisfy the one-to-one constraint, that is, each semantic target participates in the matching result at most once;

[0075] Step 6.2) Through the above optimization process, the optimal matching set is obtained. The specific solution expression form:

[0076] ;

[0077] This represents the number of the Z-th semantic target selected for matching in the image of aircraft i. This indicates the number of the corresponding semantic target in the image of aircraft j that is matched with it;

[0078] Step 6.3), based on the optimal matching result Extract the pixel center coordinates of the target in the two images and construct matching two-dimensional pixel pairs. ,in , , represents the center pixel coordinates of the Z-th semantic target in the images of aircraft i and aircraft j, respectively.

[0079] As a further optimization of the aircraft swarm cooperative localization method based on multi-view indirect observation of the present invention, step 7) includes the following specific steps:

[0080] Step 7.1), convert the two-dimensional pixel coordinates Represented in homogeneous form: ;

[0081] Step 7.2), combining the flight altitude (height) of aircraft i and the camera intrinsic parameter matrix K, perform a planar back projection on the Z-th matching point from the perspective of aircraft i to obtain its 3D point in the camera coordinate system of aircraft i:

[0082] ;

[0083] in The scale factor is determined jointly by the flight altitude (height) and the ground assumption. ;

[0084] in This represents the third component of the vector; This represents its three-dimensional point in the coordinate system of the aircraft's camera;

[0085] Step 7.3), constructing 3D points The set of 3D–2D correspondences between the corresponding two-dimensional pixels in the image of aircraft j:

[0086] ;

[0087] Step 7.4) uses the EPnP algorithm to estimate the relative pose between aircraft i and aircraft j. Its optimization objective function can be expressed as:

[0088] ;

[0089] Where R and T represent the rotation matrix and translation vector of aircraft i relative to aircraft j, respectively, and the function π(⋅) represents the homogeneous normalized projection operator; by minimizing the above reprojection error, the optimal relative pose estimation result between the two aircraft is obtained.

[0090] As a further optimization scheme of the collaborative localization method for aircraft swarms based on multi-view indirect observation of the present invention, step 8) includes using the Levenberg-Marquardt LM algorithm as a nonlinear optimization algorithm, and its iterative update formula is:

[0091] , ;

[0092] Where r represents the residual vector of reprojection error; J is the Jacobian matrix of the residual with respect to the pose parameters; and λ is the damping factor. It is the identity matrix. These are the pose parameters to be optimized.

[0093] Compared with the prior art, the present invention, employing the above technical solution, has the following technical effects:

[0094] This invention comprehensively utilizes target location, size, semantic category, and detection confidence information to optimize traditional scene matching methods, making feature matching more robust in scenarios with feature repetition, missing features, or large viewpoint changes. Simultaneously, it designs a motion compensation mechanism and a time difference penalty term to address the time difference and target motion issues during the collaborative process, ensuring high reliability of the positioning system even in scenarios with non-strict time synchronization or dynamic interference. Combined with adaptive image enhancement preprocessing and nonlinear pose optimization strategies, it further enhances the system's adaptability and positioning accuracy in complex environments such as strong light and occlusion. Compared to traditional positioning algorithms based on geometric features (such as SIFT and ORB), this invention achieves higher positioning accuracy and robustness even in environments with GPS rejection, weak texture, and dynamic interference. Attached Figure Description

[0095] Figure 1 This is a schematic diagram of the overall framework of the multi-view scene semantic matching and localization algorithm implemented in this invention;

[0096] Figure 2 This is a schematic diagram of the semantic matching and localization algorithm in a static scene in this invention;

[0097] Figure 3 This is a schematic diagram of the motion-static separation and motion compensation matching positioning algorithm in this invention;

[0098] Figure 4 This is a schematic diagram of the anchor frame selection and time-weighted matching positioning algorithm in the time difference scenario of this invention;

[0099] Figures 5(a), 5(b), and 5(c) are schematic diagrams showing the absolute error variation curves between the estimated and actual values ​​in the x, y, and z directions when the x-axis spacing is 3m according to the present invention. Detailed Implementation

[0100] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings:

[0101] This invention can be implemented in many different forms and should not be considered limited to the embodiments described herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully express the scope of the invention to those skilled in the art. In the drawings, components are enlarged for clarity.

[0102] This invention constructs an end-to-end relative positioning system, organically combining semantic target detection, multi-constraint feature matching, and pose estimation. By fusing position, scale, category, and confidence information, the matching cost matrix is ​​optimized, making the semantic correspondence across views more accurate and reasonable. At the same time, a dynamic-static separation and time-weighted compensation mechanism is introduced to address the cooperative time difference and target motion, thereby comprehensively utilizing the EPnP algorithm and nonlinear optimization to recover high-precision relative pose. This also enhances the adaptability of the swarm aircraft positioning algorithm to complex scenarios such as GPS rejection, dynamic interference, and large viewpoint changes.

[0103] like Figure 1 As shown, a collaborative localization method for an aircraft swarm based on multi-view indirect observation is presented. The aircraft swarm comprises q aircraft capable of acquiring images. Among the q image-acquiring aircraft, there are several reference aircraft with autonomous localization capabilities and several aircraft to be localized. The method includes the following steps:

[0104] Let P be the set of the aircraft that have been located in the aircraft cluster, and Q be the set of the aircraft to be located;

[0105] Step 1), select an unpositioned aircraft j from Q;

[0106] Step 2) Construct a common view of the aircraft j to be located, determine the reference aircraft i with the optimal common view relationship with the aircraft j based on the common view, and obtain the multi-view semantic target set of aircraft i and aircraft j and their attribute features.

[0107] Step 2.1) Perform semantic target detection on images acquired by aircraft j and all reference aircraft in set P, count the types N of semantic targets, and construct the N-dimensional semantic distribution vector of aircraft j and each reference aircraft according to the following formula:

[0108] ;

[0109] ;

[0110] In the formula, Indicates the first The number of semantic targets of a class in the images acquired by aircraft j. A natural number greater than or equal to 1 and less than or equal to N; Indicates the first Class semantic goals in the first The number of images acquired by the reference aircraft greater than or equal to 1 and less than or equal to 1 natural numbers, Let P be the number of reference aircraft in set P;

[0111] Step 2.2) Calculate the semantic similarity between aircraft j and each reference aircraft in set P according to the following formula:

[0112] ;

[0113] In the formula, Indicates that aircraft j and the first Semantic similarity between benchmark aircraft;

[0114] Step 2.3), the semantic similarity between aircraft j and each reference aircraft in set P is compared with the set similarity threshold. For comparison, if the reference aircraft The semantic similarity between the aircraft j and the aircraft j is greater than If so, it is considered that the benchmark is unmanned. There is a possibility of co-location between the reference aircraft and aircraft j. Both the aircraft j and the graph are defined as nodes, and a connecting edge is established between them;

[0115] After traversing set P, the common view of aircraft j is finally constructed. Let E be the set of edges formed by all pairs of aircraft nodes that satisfy the similarity threshold condition. S is the similarity value corresponding to the aircraft pair. ;

[0116] Step 2.4), determine if the common view G is empty.

[0117] Step 2.4.1): If G is not empty, select from set S. The aircraft i with the highest similarity is used as the benchmark;

[0118] Building a benchmark aircraft i Semantic target set at time , In the formula Indicates that the reference aircraft i is in The A-th target detected at time t, where A is greater than or equal to 1 and less than or equal to 1. natural numbers, The total number of targets detected by the baseline aircraft i. In the formula This represents the pixel coordinates of the center of the h-th target detection box in the image coordinate system. This indicates the width and height of the detection frame. Indicates semantic category label, This represents the confidence level of object detection. Indicates the image acquisition timestamp;

[0119] Building the aircraft j in Semantic target set at time , In the formula Indicates that aircraft j is in The A-th target detected at time t, where A is greater than or equal to 1 and less than or equal to 1. natural numbers, The total number of targets detected by aircraft j. In the formula This represents the pixel coordinates of the center of the h-th target detection box in the image coordinate system. These represent the width and height of the detection frame, respectively. Indicates semantic category label, This represents the confidence level of object detection. Indicates the image acquisition timestamp;

[0120] (Step 2.4.2) If G is empty, it means there is no available reference, and proceed to step 9.

[0121] Step 3), determine the time difference between the coordination of the reference aircraft i and the aircraft j to be located and the motion state of the semantic target set;

[0122] Step 3.1), calculate the absolute value of the time difference between image acquisition of aircraft i and j. ;

[0123] Step 3.2), based on the historical frame data of aircraft i, calculate the target's position. Speed ​​of movement within a time period Its expression is:

[0124] ;

[0125] in, , For the same goal and The location of the detection at any given moment.

[0126] Step 4) Construct a set of target features to be matched based on the judgment results;

[0127] Step 4.1), and Compare and and Comparison, The preset time synchronization threshold, The preset speed threshold;

[0128] Step 4.1.1), if And all targets Then use the target set. and As a set to be matched;

[0129] Step 4.1.2), if And there is a target :

[0130] Step 4.1.2.1), then for the target set and Perform static and dynamic separation, and Divided into:

[0131] ;

[0132] ;

[0133] In the formula, , Let i be the set of stationary targets in aircraft i and j. , Let i and j be the sets of moving targets in aircraft i and aircraft j, respectively.

[0134] Step 4.1.2.2), set the target set and As a set to be matched;

[0135] Step 4.1.3), if If so, it is determined to be a scenario with a time difference:

[0136] Step 4.1.3.1), place spacecraft i at a historical moment arrive The acquired image sequence is defined as follows:

[0137] ;

[0138] in Indicates that aircraft 1 is at time The captured images, The set of semantic targets in the corresponding image At the current moment, aircraft j Image acquisition Its detection target set is: ;

[0139] Step 4.1.3.2), for any candidate anchor point frame of aircraft i in history. Construct the total semantic matching cost between it and the current spacecraft j. :

[0140] ;

[0141] Represents the history frames of aircraft i The single-target cost between the a-th target and the b-th target of the aircraft j in the current frame, where a is greater than or equal to 1 and less than or equal to 1. For any natural number b, b is greater than or equal to 1 and less than or equal to 1. Natural numbers, spacecraft i historical frames The a-th target The b-th target in the current frame of aircraft j ; Represents the history frames of aircraft i The total number of matching pairs formed between the target in the middle and the target in the current frame of the aircraft j;

[0142] ;

[0143] in, , , , These are the weight coefficients for the preset normalized center point Euclidean distance penalty, width-to-height difference normalized scale penalty, semantic category penalty, and confidence-based linear reward. To construct the normalized Euclidean distance between the center points, ; and This refers to the horizontal resolution (i.e., image width) and vertical resolution (i.e., image height) of images captured by cameras mounted on aircraft.

[0144] To construct the scale penalty after normalizing the width-to-height difference, ; The intersection-union ratio of two detection boxes. The bounding box of the target detected by the aircraft. , ;

[0145] To construct a semantic category penalty term, ;

[0146] To construct a linear reward term based on confidence level, ;

[0147] Step 4.1.3.3), define the time difference penalty term. ;

[0148] In the formula, This represents the time difference between the current frame and the candidate anchor frame. This is the time difference weighting coefficient, used to balance semantic consistency and temporal proximity;

[0149] Step 4.1.3.4) combines the semantic matching cost with the temporal difference penalty to obtain the comprehensive evaluation function of the anchor frame. for:

[0150] ;

[0151] Step 4.1.3.5) determines the optimal anchor frame by minimizing the comprehensive cost function. and the optimal objective set :

[0152] ;

[0153] Step 4.1.3.5), set the target set and As a set to be matched.

[0154] Step 5), calculate the comprehensive matching cost matrix of semantic target pairs between aircraft i and aircraft j;

[0155] Step 5.1): For the set of targets to be matched, calculate the single-target cost between the a-th target in the image of aircraft i and the b-th target in the current frame of aircraft j. :

[0156] ;

[0157] Step 5.2), construct the cost matrix between the semantic objectives of aircraft i and aircraft j. Construct cost matrix C:

[0158] .

[0159] Step 6) Solve the optimal semantic target matching relationship between aircraft i and aircraft j based on the multi-constraint Hungarian algorithm;

[0160] Step 6.1) The Hungarian algorithm is used to perform linear allocation optimization on the cost matrix C to find the optimal one-to-one matching set. :

[0161] ;

[0162] Where M represents the set of matching relationships that satisfy the one-to-one constraint, that is, each semantic target participates in the matching result at most once;

[0163] Step 6.2) Through the above optimization process, the optimal matching set is obtained. The specific solution expression form:

[0164] ;

[0165] This represents the number of the Z-th semantic target selected for matching in the image of aircraft i. This indicates the number of the corresponding semantic target in the image of aircraft j that is matched with it;

[0166] Step 6.3), based on the optimal matching result Extract the pixel center coordinates of the target in the two images and construct matching two-dimensional pixel pairs. ,in , , represents the center pixel coordinates of the Z-th semantic target in the images of aircraft i and aircraft j, respectively.

[0167] Step 7), calculate the initial relative pose of the reference aircraft i and the aircraft j to be positioned based on the EPnP algorithm;

[0168] Step 7.1), convert the two-dimensional pixel coordinates Represented in homogeneous form: ;

[0169] Step 7.2), combining the flight altitude (height) of aircraft i and the camera intrinsic parameter matrix K, perform a planar back projection on the Z-th matching point from the perspective of aircraft i to obtain its 3D point in the camera coordinate system of aircraft i:

[0170] ;

[0171] in The scale factor is determined jointly by the flight altitude (height) and the ground assumption. ;

[0172] in This represents the third component of the vector; This represents its three-dimensional point in the coordinate system of the aircraft's camera;

[0173] Step 7.3), constructing 3D points The set of 3D–2D correspondences between the corresponding two-dimensional pixels in the image of aircraft j:

[0174] ;

[0175] Step 7.4) uses the EPnP algorithm to estimate the relative pose between aircraft i and aircraft j. Its optimization objective function can be expressed as:

[0176] ;

[0177] Where R and T represent the rotation matrix and translation vector of aircraft i relative to aircraft j, respectively, and the function π(⋅) represents the homogeneous normalized projection operator; by minimizing the above reprojection error, the optimal relative pose estimation result between the two aircraft is obtained.

[0178] Step 8) Iteratively solve the initial relative pose value based on a nonlinear optimization algorithm, and output the relative pose parameters after iterative convergence;

[0179] The nonlinear optimization algorithm used is the Levenberg-Marquardt LM algorithm, and its iterative update formula is as follows:

[0180] , ;

[0181] Where r represents the residual vector of reprojection error; J is the Jacobian matrix of the residual with respect to the pose parameters; and λ is the damping factor. It is the identity matrix. These are the pose parameters to be optimized.

[0182] Step 9) Determine if the set of aircraft to be located is empty. If it is empty, the aircraft cluster positioning task is completed. If it is not empty, transfer aircraft j from set Q to set P and jump to step 1) to locate the next aircraft to be located.

[0183] To verify the effectiveness of the proposed multi-view indirect observation-based collaborative localization method for aircraft swarms, a Gazebo digital simulation analysis and real-world scenario verification were conducted. A dual-aircraft collaborative scenario was built in the simulation, with the x, y, and z distances of aircraft 2 relative to aircraft 1 set to 3m, 0m, and 0.5m, respectively, and the flight altitude of aircraft 1 set to 4m. Figures 5(a), 5(b), and 5(c) show the absolute error curves between the estimated and actual values ​​at different distances using the method of this invention.

[0184] Simulation results show that the positioning errors using the algorithm of this invention are 29%, 35%, and 22%, respectively. Compared with traditional positioning schemes based on geometric features (such as SIFT and ORB), traditional methods often have positioning errors exceeding 30% and are prone to failure in complex scenes with similar appearances or rotation. However, the method of this invention, after multi-constraint Hungarian matching and nonlinear optimization, can effectively resist interference caused by changes in illumination and target occlusion. Furthermore, practical verification shows that as scene complexity increases, the method of this invention, benefiting from the introduction of semantic information and motion compensation mechanisms, exhibits better robustness and environmental adaptability compared to traditional methods, effectively solving the cooperative positioning problem under GPS denial conditions and demonstrating significant application value.

[0185] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. It should also be understood that terms such as those defined in general dictionaries should be understood to have the same meaning as in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless defined as herein.

[0186] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A collaborative localization method for an aircraft swarm based on multi-view indirect observation, wherein the aircraft swarm comprises q aircraft capable of acquiring images, and among the q image-acquiring aircraft are several reference aircraft with autonomous localization capabilities and several aircraft to be localized, characterized in that, Includes the following steps: Let P be the set of the aircraft that have been located in the aircraft cluster, and Q be the set of the aircraft to be located; Step 1), select an unpositioned aircraft j from Q; Step 2) Construct a common view of the aircraft j to be located, determine the reference aircraft i with the optimal common view relationship with the aircraft j based on the common view, and obtain the multi-view semantic target set of aircraft i and aircraft j and their attribute features. Step 3), determine the time difference between the coordination of the reference aircraft i and the aircraft j to be located and the motion state of the semantic target set; Step 4) Construct a set of target features to be matched based on the judgment results; Step 5), calculate the comprehensive matching cost matrix of semantic target pairs between aircraft i and aircraft j; Step 6) Solve the optimal semantic target matching relationship between aircraft i and aircraft j based on the multi-constraint Hungarian algorithm; Step 7), calculate the initial relative pose of the reference aircraft i and the aircraft j to be positioned based on the EPnP algorithm; Step 8) Iteratively solve the initial relative pose value based on a nonlinear optimization algorithm, and output the relative pose parameters after iterative convergence; Step 9) Determine if the set of aircraft to be located is empty. If it is empty, the aircraft cluster positioning task is considered complete. If not empty, transfer aircraft j from set Q to set P, and jump to step 1) to locate the next aircraft to be located.

2. The collaborative localization method for aircraft swarms based on multi-view indirect observation according to claim 1, step 2) includes the following specific steps: Step 2.1) Perform semantic target detection on images acquired by aircraft j and all reference aircraft in set P, count the types N of semantic targets, and construct the N-dimensional semantic distribution vector of aircraft j and each reference aircraft according to the following formula: ； ； In the formula, Indicates the first The number of semantic targets of a class in the images acquired by aircraft j. A natural number greater than or equal to 1 and less than or equal to N; Indicates the first Class semantic goals in the first The number of images acquired by the reference aircraft greater than or equal to 1 and less than or equal to 1 natural numbers, Let P be the number of reference aircraft in set P; Step 2.2) Calculate the semantic similarity between aircraft j and each reference aircraft in set P according to the following formula: ； In the formula, Indicates that aircraft j and the first Semantic similarity between benchmark aircraft; Step 2.3), the semantic similarity between aircraft j and each reference aircraft in set P is compared with the set similarity threshold. For comparison, if the reference aircraft The semantic similarity between the aircraft j and the aircraft j is greater than If so, it is considered that the benchmark is unmanned. There is a possibility of co-location between the reference aircraft and aircraft j. Both the aircraft j and the graph are defined as nodes, and a connecting edge is established between them; After traversing set P, the common view of aircraft j is finally constructed. Let E be the set of edges formed by all pairs of aircraft nodes that satisfy the similarity threshold condition. S is the similarity value corresponding to the aircraft pair. ; Step 2.4), determine if the common view G is empty. Step 2.4.1): If G is not empty, select from set S. The aircraft i with the highest similarity is used as the benchmark; Building a benchmark aircraft i Semantic target set at time , In the formula Indicates that the reference aircraft i is in The A-th target detected at time t, where A is greater than or equal to 1 and less than or equal to 1. natural numbers, The total number of targets detected by the baseline aircraft i. In the formula This represents the pixel coordinates of the center of the h-th target detection box in the image coordinate system. This indicates the width and height of the detection frame. Indicates semantic category label, This represents the confidence level of object detection. Indicates the image acquisition timestamp; Building the aircraft j in Semantic target set at time , In the formula Indicates that aircraft j is in The A-th target detected at time t, where A is greater than or equal to 1 and less than or equal to 1. natural numbers, The total number of targets detected by aircraft j. In the formula This represents the pixel coordinates of the center of the h-th target detection box in the image coordinate system. These represent the width and height of the detection frame, respectively. Indicates semantic category label, This represents the confidence level of object detection. Indicates the image acquisition timestamp; (Step 2.4.2) If G is empty, it means there is no available reference, and proceed to step 9.

3. The collaborative localization method for aircraft swarms based on multi-view indirect observation according to claim 2, step 3) includes the following specific steps: Step 3.1), calculate the absolute value of the time difference between image acquisition of aircraft i and j. ; Step 3.2), based on the historical frame data of aircraft i, calculate the target's position. Speed ​​of movement within a time period Its expression is: ； in, , For the same goal and The location of the detection at any given moment.

4. The collaborative localization method for aircraft swarms based on multi-view indirect observation according to claim 3, step 4) includes the following specific steps: Step 4.1), and Compare and and Comparison, The preset time synchronization threshold, The preset speed threshold; Step 4.1.1), if And all targets Then use the target set. and As a set to be matched; Step 4.1.2), if And there is a target : Step 4.1.2.1), then for the target set and Perform static and dynamic separation, and Divided into: ； ； In the formula, , Let i be the set of stationary targets in aircraft i and j. , Let i and j be the sets of moving targets in aircraft i and aircraft j, respectively. Step 4.1.2.2), set the target set and As a set to be matched; Step 4.1.3), if If so, it is determined to be a scenario with a time difference: Step 4.1.3.1), place spacecraft i at a historical moment arrive The acquired image sequence is defined as follows: ； in Indicates that aircraft 1 is at time The captured images, The set of semantic targets in the corresponding image ; At the current moment, aircraft j Image acquisition Its detection target set is: ; Step 4.1.3.2), for any candidate anchor point frame of aircraft i in history. Construct the total semantic matching cost between it and the current spacecraft j. : ； Represents the history frames of aircraft i The single-target cost between the a-th target and the b-th target of the aircraft j in the current frame, where a is greater than or equal to 1 and less than or equal to 1. For any natural number b, b is greater than or equal to 1 and less than or equal to 1. Natural numbers, spacecraft i historical frames The a-th target The b-th target in the current frame of aircraft j ; Represents the history frames of aircraft i The total number of matching pairs formed between the target in the middle and the target in the current frame of the aircraft j; ； in, , , , These are the weight coefficients for the preset normalized center point Euclidean distance penalty, width-to-height difference normalized scale penalty, semantic category penalty, and confidence-based linear reward. To construct the normalized Euclidean distance between the center points, ; , These are the horizontal and vertical resolutions of the images captured by the cameras on the aircraft, respectively. To construct the scale penalty after normalizing the width-to-height difference, ; The intersection-union ratio of two detection boxes. The bounding box of the target detected by the aircraft. , ; To construct a semantic category penalty term, ; To construct a linear reward term based on confidence level, ; Step 4.1.3.3), define the time difference penalty term. ; In the formula, This represents the time difference between the current frame and the candidate anchor frame. This is the time difference weighting coefficient, used to balance semantic consistency and temporal proximity; Step 4.1.3.4) combines the semantic matching cost with the temporal difference penalty to obtain the comprehensive evaluation function of the anchor frame. for: ； Step 4.1.3.5) determines the optimal anchor frame by minimizing the comprehensive cost function. and the optimal objective set : ； Step 4.1.3.5), set the target set and As a set to be matched.

5. The collaborative localization method for aircraft swarms based on multi-view indirect observation according to claim 4, step 5) includes the following specific steps: Step 5.1): For the set of targets to be matched, calculate the single-target cost between the a-th target in the image of aircraft i and the b-th target in the current frame of aircraft j. : ； Step 5.2), construct the cost matrix between the semantic objectives of aircraft i and aircraft j. Construct cost matrix C: 。 6. The collaborative localization method for aircraft swarms based on multi-view indirect observation according to claim 5, step 6) includes the following specific steps: Step 6.1) The Hungarian algorithm is used to perform linear allocation optimization on the cost matrix C to find the optimal one-to-one matching set. : ； Where M represents the set of matching relationships that satisfy the one-to-one constraint, that is, each semantic target participates in the matching result at most once; Step 6.2) Through the above optimization process, the optimal matching set is obtained. The specific solution expression form: ； This represents the number of the Z-th semantic target selected for matching in the image of aircraft i. This indicates the number of the corresponding semantic target in the image of aircraft j that is matched with it; Step 6.3), based on the optimal matching result Extract the pixel center coordinates of the target in the two images and construct matching two-dimensional pixel pairs. ,in , , represents the center pixel coordinates of the Z-th semantic target in the images of aircraft i and aircraft j, respectively.

7. The collaborative localization method for aircraft swarms based on multi-view indirect observation according to claim 6, step 7) includes the following specific steps: Step 7.1), convert the two-dimensional pixel coordinates Represented in homogeneous form: ; Step 7.2), combining the flight altitude (height) of aircraft i and the camera intrinsic parameter matrix K, perform a planar back projection on the Z-th matching point from the perspective of aircraft i to obtain its 3D point in the camera coordinate system of aircraft i: ； in The scale factor is determined jointly by the flight altitude (height) and the ground assumption. ; in This represents the third component of the vector; This represents its three-dimensional point in the coordinate system of the aircraft's camera; Step 7.3), constructing 3D points The set of 3D–2D correspondences between the corresponding two-dimensional pixels in the image of aircraft j: ； Step 7.4) uses the EPnP algorithm to estimate the relative pose between aircraft i and aircraft j. Its optimization objective function can be expressed as: ； Where R and T represent the rotation matrix and translation vector of aircraft i relative to aircraft j, respectively, and the function π(⋅) represents the homogeneous normalized projection operator; by minimizing the above reprojection error, the optimal relative pose estimation result between the two aircraft is obtained.

8. The collaborative localization method for aircraft swarms based on multi-view indirect observation according to claim 7, step 8) includes employing the Levenberg-Marquardt LM algorithm as a nonlinear optimization algorithm, whose iterative update formula is: ， ； Where r represents the residual vector of reprojection error; J is the Jacobian matrix of the residual with respect to the pose parameters; and λ is the damping factor. It is the identity matrix. These are the pose parameters to be optimized.

Images