A multi-view cooperative perception method based on spatio-temporal prior guidance and communication adaptation

Abstract

The application discloses a kind of multi-view cooperative perception methods based on space-time prior guidance and communication adaptation, it is applied to by multiple observation unmanned aerial vehicle and cooperative processing node composition Multi-UAV cooperative perception system. Each observation unmanned aerial vehicle collects environmental image and synchronously obtains pose information, generates local target observation record;According to the pose of reference observation unmanned aerial vehicle and cooperative observation unmanned aerial vehicle, camera parameter and pose error, target space sight is projected to cooperative observation image plane, and generates geographic prior search area ROI;In the ROI, pre-trained depth feature matching network is called to perform cross-view feature matching, and ROI space constraint, matching confidence screening and optional adaptive inference mechanism are used to reduce invalid calculation amount, and multi-UAV target ID association table is established or updated;Further according to target association state, local tracking state and communication link state, trigger lightweight identification abstract transmission, and adaptively adjust abstract sending frequency and compression degree;Cooperative processing node carries out grouping to multi-source abstract, space-time alignment, consistency review and multi-view three-dimensional positioning, and outputs global target state.The application can reduce cross-view matching calculation redundancy and communication overhead, improve the stability of multi-UAV cooperative perception and target positioning under the condition of limited resources.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and multi-agent cooperative control technology. Specifically, it relates to a multi-UAV multi-view cooperative perception and target localization method for low-altitude defense scenarios, based on spatiotemporal prior guidance and communication adaptation. Background Technology

[0002] With the rapid development of the low-altitude economy, the widespread application of "low, slow, and small" drones has brought economic benefits, but it has also posed new challenges to public airspace safety.

[0003] To achieve real-time and precise defense against unauthorized drone targets, building multi-drone cooperative sensing networks has become a crucial direction for technological evolution. Compared to single-drone observation, multi-drone collaboration can effectively overcome blind spots from a single perspective and maintain continuous tracking even when targets are obscured. However, in real-world and complex low-altitude defense scenarios, existing multi-drone cooperative sensing technologies face numerous severe bottlenecks in the actual deployment of airborne edge devices.

[0004] The most pressing issue is the high redundancy and poor geometric consistency in cross-view target association calculations. Under multiple UAV perspectives, the appearance of the same target is significantly affected by differences in viewpoint, lighting variations, and background interference. Existing image feature matching algorithms typically employ a global search and matching mode across the entire image. This not only has extremely high computational complexity, making it impossible for computationally limited UAV edge computing nodes to meet real-time requirements, but also, without spatial prior information guidance, the global search is prone to generating numerous erroneous matches. This leads to drastic fluctuations in the target's 3D coordinates subsequently calculated based on epipolar triangulation, failing to provide a reliable spatial reference.

[0005] Meanwhile, existing methods severely lack dynamic adaptability at the computational level and communication robustness in weak network environments. In real-world defense scenarios, the difficulty of target observation is dynamically changing, but most existing deep matching models use fixed network depth and static inference graphs. Regardless of whether facing clear targets with simple backgrounds or blurred targets with complex textures, they consume the same amount of computational resources. This "one-size-fits-all" static architecture results in a huge waste of computing power.

[0006] On the other hand, another major pain point in multi-UAV collaborative perception lies in the limited and highly unstable inter-UAV communication bandwidth. Most existing multi-UAV collaborative frameworks assume ideal communication links, often requiring the transmission of raw high-definition image streams or uncompressed high-dimensional dense feature maps between nodes. In low-altitude dynamic defense, communication bandwidth can be significantly reduced at any time due to distance changes or electromagnetic interference. This strategy of directly transmitting large amounts of raw data easily leads to network congestion and extremely high transmission latency, causing the collaborative fusion of multiple perspectives to completely fail. In summary, there is an urgent need for a multi-UAV collaborative perception method that can simultaneously address the problems of low cross-view matching efficiency, limited edge computing resources, and unstable inter-UAV communication, in order to achieve high-precision, low-latency target collaborative identification and localization under stringent resource constraints. Summary of the Invention

[0007] The purpose of this invention is to address the problems of high redundancy in cross-view matching calculations, poor geometric consistency, and low collaborative efficiency in environments with limited computing power and weak networks in existing technologies. This invention aims to provide a multi-view collaborative sensing method based on spatiotemporal prior guidance and communication adaptation. This method can achieve high-precision, low-latency collaborative identification and global localization of targets under stringent edge-side computing power constraints and dynamically fluctuating communication bandwidth constraints.

[0008] The technical solution of the present invention: To achieve the above-mentioned objective, according to the first aspect of the present invention, a multi-view collaborative perception method based on spatiotemporal prior guidance and communication adaptation is proposed, applicable to multi-UAV scenarios, wherein each UAV is equipped with an airborne visual sensor, a GPS / RTK positioning module, an IMU inertial measurement unit, and a target tracker; comprising the following steps: Step 1: Each UAV node uses its onboard vision sensor to acquire environmental images within its field of view according to a preset sampling period, and simultaneously acquires the UAV pose information and pose error information corresponding to the acquired images at the same time. Each UAV node performs local target detection and local target tracking on the acquired environmental images to obtain the detection box, target center pixel coordinates, target category, detection confidence, and local tracking identifier of the candidate targets. Visual feature descriptors for cross-view target association are extracted for the image regions corresponding to each candidate target. Subsequently, the UAV node identifier, image acquisition timestamp, camera calibration parameters, current pose information, pose error information, and the local tracking identifier, detection box, target center pixel coordinates, target category, detection confidence, and visual feature descriptors of each candidate target are associated and encapsulated to generate a local target observation record set, which serves as the input data for subsequent geographic prior search region generation, cross-view feature matching, multi-UAV target ID association, and target 3D localization.

[0009] Step 2: For any local target observation record generated by any UAV node in Step 1, the first observation UAV that has detected a candidate target and provided the center pixel coordinates of the candidate target is determined as the reference observation UAV. The second observation UAV, which has acquired the image to be matched within a preset collaborative time window and has an effective observation baseline with the reference observation UAV, is selected as the collaborative observation UAV. Based on the target center pixel coordinates of the reference observation UAV, the pose information of the reference observation UAV and the collaborative observation UAV at the corresponding time, camera intrinsic parameters, and camera extrinsic parameters, the candidate target is back-projected from the image plane of the reference observation UAV into a spatial line of sight, and the spatial line of sight is projected onto the image plane of the collaborative observation UAV to obtain the epipolar search segment of the target in the image of the collaborative observation UAV. Further, based on the pose estimation error covariance matrix of the reference observation UAV and the collaborative observation UAV, and the image measurement error, the epipolar search segment is expanded into a geographic prior search area (ROI) containing the true projection position of the target, which is used to limit the search range for subsequent cross-view target feature matching.

[0010] Step 3: Within the geographic prior search area (ROI) generated in Step 2, perform cross-view feature matching on candidate targets in the corresponding images of the collaborative observation UAV; specifically, only extract and retain candidate feature points and their feature descriptors located within the geographic prior search area (ROI), input the target features of the reference observation UAV and the candidate features within the ROI of the collaborative observation UAV into a pre-trained deep feature matching network, and use the pre-trained deep feature matching network to establish cross-view feature point matching relationships; In one possible embodiment, the pre-trained deep feature matching network can be implemented using a LightGlue network, a SuperGlue network, or other local feature matching networks based on graph neural networks or attention mechanisms.

[0011] This invention is not limited by the specific network structure, training method, or parameter update method of the deep feature matching network. The improvement of this invention lies in using the geographic prior search region (ROI) to constrain the range of candidate feature points in the collaborative observation image, and combining the deep feature matching results with target category consistency, appearance similarity, geometric consistency, and the update process of the multi-UAV target ID association table; based on the output cross-view feature point matching relationship, target category consistency, appearance similarity, and geometric consistency, it is determined whether the candidate target in the reference observation UAV and the candidate target in the collaborative observation UAV correspond to the same physical target; Repeat steps 2 and 3 to iterate through all cooperative observation UAVs that meet the conditions, and establish or update the multi-UAV target ID association table based on the cross-view feature matching results. The multi-UAV target ID association table is used to record the correspondence between the global target ID and the local tracking identifier of each UAV.

[0012] In one possible embodiment, establishing or updating the multi-UAV target ID association table based on the matching results specifically includes: For candidates that are confirmed to correspond to the same physical target, if there is no candidate target with a global target ID, the collaborative processing node will assign a new global target ID and write the corresponding UAV node identifier and local tracking identifier into the global target ID field; if there is a target with a global target ID, the newly added UAV node identifier and its local tracking identifier will be appended to the corresponding global target ID field. For candidate targets that have not yet passed the consistency judgment, assign a confirmation identifier and include them in the set of targets to be confirmed, and return to step 2 when there are subsequent image frames, new observation data from other UAVs, or changes in the target status.

[0013] Step 4: The collaborative processing node collects multi-UAV observation data associated with the same global target ID or unconfirmed identifier based on the multi-UAV target ID association table in Step 3, and groups and aligns the multi-UAV observation data in time and space; for targets that already have a global target ID, the cross-view association relationship is checked for consistency by combining the target center pixel coordinates, target category, detection confidence, target visual feature descriptor, UAV pose information and target motion state. For multi-source observation data that has been verified to correspond to the same physical target and has at least two effective observation perspectives, the three-dimensional position of the target in a unified world coordinate system is calculated based on the camera calibration parameters of each observation UAV, the UAV pose information at the corresponding image acquisition time, and the target center pixel coordinates. The method of minimizing the multi-view reprojection error is used. The credibility of the positioning result is judged based on the reprojection error and the number of effective observation perspectives. The final output includes global target status information, including global target ID, target category, comprehensive confidence, three-dimensional position, motion state, and a list of associated observation UAVs.

[0014] In one possible embodiment, the multi-UAV target ID association table is updated when a new observation node is added, an association conflict occurs, or a target to be confirmed is confirmed.

[0015] In one possible embodiment, each observation UAV generates a lightweight identification summary based on local target observation records and the multi-UAV target ID association table. The lightweight identification summary includes at least the UAV node identifier, image acquisition timestamp, local tracking identifier, global target ID or unconfirmed identifier, target center pixel coordinates, target category, detection confidence level, and compressed feature summary.

[0016] Specifically, the lightweight recognition summary is shown in the following formula:

[0017] in, Indicates the drone node identifier, Indicates the image acquisition timestamp. Indicates drone Regarding the first Local tracking identifiers are established for each candidate target. This indicates the global target ID assigned to this target in step 3; if the candidate target has not yet completed the global target ID association, then... Take the confirmation icon; Represents the target detection box. Indicates the center pixel coordinates of the target. Indicates the target category. Indicates the confidence level of the detection. This represents the pose of the UAV relative to the unified world coordinate system when it acquires the corresponding image frame. This represents a compressed feature summary obtained by dimensionality reduction, quantization, or hash encoding of the target visual feature descriptor.

[0018] The lightweight recognition summary is different from the local target observation record generated in step 1: the local target observation record is used for local processing and cross-view matching calculation of UAV nodes, while the lightweight recognition summary is used for inter-machine communication transmission when the trigger condition is met, so as to reduce the communication resource occupation caused by the direct transmission of raw image data or high-dimensional feature data.

[0019] In one possible embodiment, the system monitors the available bandwidth of communication links between drones or between drones and collaborative processing nodes in real time. Average transmission delay and packet loss rate And based on preset thresholds, the current link status is divided into three levels: When satisfied When the condition is good, the system sends the complete lightweight identification summary corresponding to the trigger target at the preset normal sending frequency. When satisfied Or average transmission delay Approaching the maximum allowable transmission delay When this occurs, the system is deemed to be in a restricted state; the system reduces the frequency of sending lightweight recognition digests and reduces the compression feature digest. The dimensions are limited to retaining only the target category, detection confidence, target center pixel coordinates, local tracking identifier, global target ID or unconfirmed identifier, and low-dimensional feature summary. When satisfied or When the network is in a very weak state, the system only sends the minimum identification summary of high-priority targets. The minimum identification summary includes at least the UAV node identifier, timestamp, local tracking identifier, global target ID or unconfirmed identifier, target category, target center pixel coordinates and detection confidence. in, This represents the bandwidth threshold required for normal transmission of a complete identification digest. This indicates the minimum bandwidth threshold required to maintain the transmission of critical target summaries. Indicates the maximum allowable average transmission delay. This indicates the maximum allowed packet loss rate.

[0020] Each UAV node does not perform frame-by-frame summary broadcasting for all candidate targets. Instead, it determines whether to trigger the identification summary transmission based on the local tracking status and the association status of multiple UAV target IDs.

[0021] In one possible implementation, for any global target ID If the multi-drone target ID association table already contains no less than If all UAVs complete consistent association with the target within a preset time window, and the local tracking status of each corresponding observation UAV meets the stability condition, then the target is determined to have entered a stable cooperative tracking state. Once the target enters a stable cooperative tracking state, each observation UAV suspends the frame-by-frame lightweight recognition summary broadcast for the target and enters a communication silence state. In the communication silence state, each observation UAV maintains local tracking of the target solely based on local image observation results and prediction filters, and sends target status confirmation information according to a preset low-frequency period.

[0022] The stability conditions include at least the following: 1. The current target detection confidence is greater than the stable confidence threshold. ; 2. The number of consecutive stable tracking frames for this target by the local target tracker is greater than the stable frame count threshold. ; 3. The prediction residual of the prediction filter is not greater than the stable residual threshold. ; 4. The difference between the current visual feature summary and the historical feature template corresponding to the global target ID is no greater than the visual feature mutation threshold. ; 5. No related conflict events occurred within the current preset time window.

[0023] In one possible embodiment, when a visual feature mutation, local tracking loss, association conflict, or prediction residual exceeding a threshold is detected, the communication silence state is immediately exited and summary broadcasting and cross-view association updates are retried.

[0024] In one possible embodiment, when multiple targets trigger identification summary broadcasting simultaneously, and the current communication link is insufficient to send lightweight identification summaries corresponding to all targets, the sending priority of each target is calculated based on the target threat level, detection confidence, target association status, local tracking status, and spatial relationship between the target and the defense area. Candidate Targets At any moment The sending priority score is represented as follows:

[0025] in, Indicates the target threat level score; Indicates the confidence level of target detection; This indicates the score corresponding to when the target is in a state of pending association confirmation; This represents the score corresponding to a local tracking loss event of the target; This indicates a score determined based on the spatial distance between the target and the defense zone; , , , and Let be the corresponding weight coefficients, and satisfy:

[0026] The system selects lightweight recognition digests to be sent in descending order of priority score.

[0027] When the communication link is in a very weak network state, priority is given to sending the first appearance event of a new target, the local tracking loss event, the identification summary data corresponding to the unconfirmed target and the high threat level target; for targets that have entered a stable cooperative tracking state, the communication silence state is maintained, or only low-frequency status confirmation information is sent.

[0028] In one possible embodiment, for drones At any moment Detected candidate targets A new target first appearance event is determined to have occurred when the candidate target meets any of the following conditions: 1. There is no associated record in the multi-UAV target ID association table that matches the UAV node identifier and local tracking identifier of the candidate target; 2. The candidate targets are in continuous The target was detected in all frames, and the detection confidence score of each frame was greater than the new target confirmation threshold. ; 3. The appearance similarity between the targets corresponding to each global target ID in the candidate target and multi-UAV target ID association table is lower than the preset appearance association threshold, and the geometric consistency score is lower than the preset geometric association threshold.

[0029] When a new target appears for the first time, the candidate target is set as a target to be confirmed, a lightweight identification summary containing the identifier to be confirmed is generated for it, and the lightweight identification summary is immediately broadcast to other UAV nodes or collaborative processing nodes participating in collaborative perception to trigger the geographic prior search area generation process in step 2 and the cross-view target matching and global target ID association process in step 3.

[0030] For candidate targets that have been set as pending targets but have not yet completed the global target ID association, unconditional retransmission will not be performed on them.

[0031] Let the time when the target to be confirmed last sent the lightweight identification digest be . The current number of retries has been performed. The preset retry interval is 1. The preset maximum number of retry attempts is... When the following conditions are met:

[0032] and:

[0033] At that time, the lightweight identification digest corresponding to the target to be confirmed can be sent again.

[0034] When the number of retries for a target to be confirmed reaches the preset maximum number of retries, and no cross-view observation results that meet the target consistency judgment conditions are still obtained, the candidate target is marked as a low-confidence target to be confirmed, and the periodic repetitive transmission for the target is suspended until new local observation evidence, relevant observation data actively reported by other UAVs, or changes in the target status are obtained, and then the association confirmation process is retried.

[0035] In one possible embodiment, for a global target ID that has already been established in a multi-UAV target ID association table For the tracked target, based on the degree of difference between the compressed feature summary generated in the current image frame and the historical feature template corresponding to the global target ID, it is determined whether the target's visual features have undergone a sudden change.

[0036] When compressing feature summary and historical feature templates When the vector is continuous or quantized, cosine distance is used to calculate the feature difference between the current feature summary and the historical feature template:

[0037] When the following conditions are met:

[0038] When a sudden change in visual features occurs in the current target, it is determined that the target has undergone a change in visual characteristics. This represents the threshold for visual feature mutations set for continuous or quantized vectors.

[0039] When compressing feature summary and historical feature templates For length is When performing binary hash encoding, the normalized Hamming distance is used to calculate the feature difference between the two:

[0040] in, This represents an indicator function that takes the value 1 when the condition in parentheses is true, and 0 otherwise.

[0041] When the following conditions are met:

[0042] When a sudden change in visual features occurs in the current target, it is determined that the target has undergone a change in visual characteristics. This represents the threshold for visual feature mutations set for binary hash encoding.

[0043] The visual feature mutation event can be caused by changes in target pose, changes in target scale, partial occlusion, changes in illumination, or local tracking errors.

[0044] When a visual feature mutation event occurs, the drone node sends a lightweight identification summary of the target at the current moment and triggers step 3 to re-evaluate the cross-view consistency of the target, so as to reduce the risk of incorrect association or incorrect switching of the global target ID caused by changes in the target's appearance.

[0045] In one possible implementation, for a globally assigned target ID To achieve the target, each UAV node uses a local target tracker and a predictive filter to maintain continuous tracking of the target.

[0046] In one implementation, a drone is determined to be a drone when any of the following conditions are met. For global target ID A local tracking loss event occurred for the corresponding target: 1. In continuous No match was detected in the frame image for the global target ID. Corresponding candidate targets; 2. The tracking confidence level output by the local target tracker is continuously lower than the preset tracking confidence level threshold; 3. The residual between the predicted position output by the prediction filter and the current observation position exceeds the preset tracking loss threshold.

[0047] The predicted position of the output of the prediction filter is denoted as... The currently detected observation position is recorded as The predicted residuals are expressed as:

[0048] When the following conditions are met:

[0049] When the current observation results are inconsistent with the predicted tracking status, it is determined that the current tracking status is inconsistent. This indicates the preset tracking loss threshold.

[0050] When a local tracking loss event occurs, the UAV node broadcasts the last valid lightweight identification summary and tracking loss flag of the target to other UAV nodes or cooperative processing nodes, enabling other UAV nodes that are still able to stably observe the target to provide auxiliary observation data and re-execute the target association update process.

[0051] In one possible embodiment, when a candidate target still carries an existing global target ID, but its geometric consistency, appearance consistency, or motion continuity does not meet the consistency verification conditions defined in step 4, an association conflict event is determined to have occurred. When an association conflict event occurs, the current observation data is paused from being incorporated into the fused positioning data set corresponding to the original global target ID, and steps 2 and 3 are retried for the target to redetermine its cross-view target matching relationship and target ID association status.

[0052] In one possible embodiment, step 1 specifically includes the following steps: Step 1.1: For A collaborative perception system composed of drones will... The drone is recorded as ,in At any moment drones Image frames are acquired using an onboard vision sensor. It also reads the data output by the airborne positioning module and inertial measurement unit corresponding to the image frame.

[0053] drones The pose representation in a unified world coordinate system is as follows:

[0054] in, Indicates drone At any moment The attitude rotation matrix, Indicates drone At any moment The position vector.

[0055] Based on the fusion calculation results of the GPS / RTK positioning module and the IMU inertial measurement unit, the error covariance matrix corresponding to the pose is obtained:

[0056] in, , , These represent the variances of the position estimation error, respectively. , , These represent the variances of the estimation errors for roll angle, pitch angle, and yaw angle, respectively.

[0057] Meanwhile, the drones are pre-calibrated and stored. Intrinsic parameter matrix of the mounted camera And the extrinsic transformation matrix between the camera coordinate system and the UAV body coordinate system. The image frame Collection timestamp , pose Pose error covariance matrix Camera intrinsic parameter matrix and extrinsic transformation matrix Together they constitute the spatiotemporal calibration data of the current image frame.

[0058] Step 1.2: Drone Image frames Input a lightweight object detection network pre-deployed on an onboard computing unit, and output a set of candidate objects in the current image frame:

[0059] in, Indicates drone At any moment The number of candidate targets detected; candidate targets At least including:

[0060] in, A detection bounding box representing a candidate target; This indicates the pixel coordinates corresponding to the center of the detection box; Indicates the target category; This indicates the confidence level of the detection.

[0061] For detection confidence levels that meet the preset detection threshold Candidate targets are identified and associated with the detection results from the previous time step using the local target tracker. If the candidate target successfully matches an existing local trajectory, the existing local tracking identifier is inherited. If no match is found with an existing trajectory, a new local tracking identifier will be assigned to it. .

[0062] The local tracking identifier is only used to distinguish different targets observed by the same UAV in consecutive image frames, and does not directly indicate that the targets observed by multiple UAVs belong to the same physical target; the unified target identifier among multiple UAVs will be established in the subsequent cross-view association step.

[0063] Step 1.3: For each candidate target whose detection confidence level meets the preset detection threshold, use its detection bounding box. The defined target image region is used as input, and a lightweight feature extraction network is used to extract visual feature descriptors to describe the appearance attributes of the target:

[0064] in, This refers to the feature extraction model deployed on the UAV's onboard computing unit. Representing dimensions The target visual feature descriptor.

[0065] It should be noted that the lightweight feature extraction network can be implemented using existing mature visual feature extraction neural networks, such as MobileNet, ShuffleNet, EfficientNet-Lite, Lightweight ResNet, or other pre-trained neural networks capable of outputting target appearance feature vectors. The lightweight feature extraction network is used for detecting bounding boxes. The defined target image region is feature-encoded, and a visual feature descriptor is output for subsequent cross-view target matching. This invention is not limited by the specific network structure, training method, or parameter update method of the lightweight feature extraction network. The lightweight feature extraction network can directly adopt existing mature models or be deployed on the UAV's onboard computing unit after conventional training and fine-tuning according to specific application scenarios. The improvement of this invention lies in combining the visual feature descriptor with subsequent geographic prior search region (ROI) constraints, cross-view feature matching, multi-UAV target ID association, and adaptive communication transmission processes, rather than improving the structure of the lightweight feature extraction network itself.

[0066] Step 1.4: For drones At any moment The detected and tracked first For each candidate target, generate a local target observation record:

[0067] in, This indicates the identifier of the UAV node that generated the local target observation record. This indicates the timestamp of the corresponding environmental image acquisition. Indicates the first The drones targeted the first Local tracking identifiers are established for each candidate target; Indicates the first The detection bounding boxes of candidate targets at time t. This indicates the pixel coordinates corresponding to the center of the detection box. Indicates the candidate target category, This represents the detection confidence level of the candidate target. This represents the visual feature descriptor extracted and normalized for the candidate target; Indicates drone The pose relative to the unified world coordinate system when acquiring the corresponding image frame. This represents the estimation error covariance matrix corresponding to the pose. Indicates drone The intrinsic parameter matrix of the mounted camera, This represents the extrinsic transformation matrix between the camera coordinate system and the UAV body coordinate system. The local target observation record serves as the foundational data for subsequent cross-view associations; it is used when generating the geographic prior search area (ROI) and establishing a multi-UAV target ID association table.

[0068] drones At any moment All generated local target observation records are combined into a local target observation record set:

[0069] The target center pixel coordinates, camera calibration parameters, UAV pose, and pose estimation error covariance matrix in the local target observation record set are used to subsequently deduce the epipolar projection range of the target on other UAV image planes and generate a geographic prior search region; the local tracking identifier, detection confidence, and visual feature descriptor are used for subsequent cross-view target matching and association of global unique IDs for multiple UAV targets.

[0070] In one possible embodiment, in step 1.3, the visual feature descriptors are normalized:

[0071] in, Indicates drone At any moment For the Normalized pre-visual feature descriptors obtained from the extraction of candidate targets Indicates the dimension of the visual feature descriptor; This represents the visual feature descriptor obtained after normalizing the aforementioned visual feature descriptor; Indicates the drone's serial number. Indicates the candidate target number. Indicates the time of image acquisition or target detection; The L2 norm of a vector. Visual feature descriptors The Euclidean length is calculated as follows:

[0072] in, Visual feature descriptors The Each component. The above normalization process means dividing the visual feature descriptor by its own L2 norm, so that the normalized visual feature descriptor has a unit length, thereby reducing the impact of different lighting conditions and different image scales on subsequent cross-view appearance similarity calculations.

[0073] This reduces the impact of different lighting conditions and image scales on subsequent cross-view appearance similarity calculations.

[0074] It should be noted that the visual feature descriptors generated in this step are used to support subsequent cross-view association; in the subsequent adaptive transmission process, they can be further compressed or encoded into low-dimensional feature summaries according to the current link bandwidth status, instead of requiring the direct transmission of the original high-dimensional feature data in this step.

[0075] In one possible embodiment, step 2 specifically includes the following process: Step 2.1: For the local target observation record set generated in Step 1, when the UAV... At any moment Candidate target detected When the detection confidence of the candidate target meets the preset detection threshold, the drone will be... The UAV was identified as the reference observation UAV, and its corresponding local target observation record was recorded as follows:

[0076] The local target observation record includes at least the target center pixel coordinates. Reference observation of UAV attitude Pose estimation error covariance matrix And camera calibration parameters.

[0077] Among the other UAV nodes participating in collaborative sensing, the UAVs that acquire the image frames to be matched within the preset collaborative time window and have an effective observation baseline with the reference observation UAV will be selected. It was determined to be a collaborative observation UAV; the image acquisition time corresponding to the collaborative observation UAV is denoted as... Its pose and pose estimation error covariance matrix are denoted as follows: and .

[0078] The preset collaborative time window is used to limit the maximum allowable time difference between the reference observation image and the collaborative observation image; when:

[0079] At that time, it is assumed that the two image frames meet the time synchronization requirements for cooperative matching, where, This is the preset maximum time difference threshold.

[0080] It should be noted that the reference observation UAV and the cooperative observation UAV in this step only represent the observation relationship in the current cross-view target search process, and do not limit a certain UAV to be fixed as the transmitter or receiver in the long term; the cooperative processing node that performs subsequent calculations and data fusion can be the cooperative observation UAV, the master UAV, or the ground control station.

[0081] For reference observation UAVs At any moment Candidate targets detected Based on its target center pixel coordinates and reference observation UAV camera intrinsic parameter matrix The target center pixel coordinates are back-projected onto the reference observation UAV camera coordinate system to obtain the normalized imaging direction vector. Then, based on the reference observation UAV pose and camera extrinsic parameters, the normalized imaging direction vector is transformed to the world coordinate system to obtain the spatial line-of-sight direction vector. and form candidate space rays The space ray within the effective depth range The two endpoints within are projected onto the collaborative observation UAV. From the image plane, obtain the projection point. and Connecting the two projection points yields the candidate target. Epipolar search segment in collaboratively observed images .

[0082] Step 2.2: For the reference observation UAV Candidate targets detected Based on the target center pixel coordinates:

[0083] And the intrinsic parameter matrix of the camera carried by the reference observation drone. The target center pixel coordinates are then projected backwards onto the camera coordinate system of the reference observation UAV to obtain the normalized imaging direction vector:

[0084] Based on the pose information of the reference observation UAV and the external parameter transformation relationship between the camera coordinate system and the UAV body coordinate system, the normalized imaging direction vector is transformed to a unified world coordinate system to obtain the spatial line-of-sight direction vector corresponding to the target:

[0085] in, Indicates the reference observation drone at time The attitude rotation matrix relative to the world coordinate system, This represents the rotation matrix of the camera coordinate system on the reference observation drone relative to the drone's body coordinate system.

[0086] Therefore, the candidate spatial position of the target in the world coordinate system can be represented as:

[0087] in, This indicates the position of the optical center of the reference observation UAV camera in the world coordinate system. This represents the unknown depth parameter of the target along the line of sight in space.

[0088] Step 2.3: Based on the application scenario of the target, the effective detection range of the airborne sensors, or the target's mission airspace boundary, pre-set the effective depth range of the target along the spatial line-of-sight direction:

[0089] By taking the two endpoints of the effective depth range, the two endpoints of the target candidate spatial line segment are obtained:

[0090] Transform the two endpoints to a collaborative observation drone. The camera coordinate system, and based on the camera intrinsic parameter matrix of the collaborative observation UAV. Projecting onto its image plane, we obtain:

[0091] in, The projection function represents the perspective projection of a point in three-dimensional space onto an image plane.

[0092] Connect projection points and Candidate targets were obtained. In collaborative observation of drones Epipolar search segment on the image plane:

[0093] The epipolar search segment is used to represent, under ideal conditions where positioning errors and image measurement errors are not considered, the epipolar search segment in the image acquired by the collaborative observation UAV that is closest to the candidate target. The corresponding actual projection position should be located on this search segment.

[0094] Step 2.4: Considering that both the positioning module and inertial measurement unit of the reference observation UAV and the cooperative observation UAV have measurement errors, the pose errors of the reference observation UAV and the cooperative observation UAV are expressed as zero-mean Gaussian random variables, and their covariance matrices are respectively... and .

[0095] The pose error covariance matrices of the reference observation UAV and the cooperative observation UAV are combined into a relative observation error covariance matrix:

[0096] For any candidate spatial projection point on the epipolar search segment, its projection position on the collaborative observation UAV image plane is represented as a function of pose variables:

[0097] in, and These represent the pose parameter vectors of the reference observation UAV and the cooperative observation UAV, respectively.

[0098] According to the first-order error propagation principle, the covariance matrix of the projected position on the image plane of the cooperative observation UAV is expressed as:

[0099] in, Represents the projection function Jacobian matrix for pose variables of the reference observation UAV and the cooperative observation UAV; This represents the pixel measurement noise covariance matrix caused by target center point detection error, image noise, and feature localization error.

[0100] In obtaining candidate targets In collaborative observation of drones Epipolar search segment in the image plane Then, the polar search segment Discrete sampling is performed to obtain multiple candidate projection points. ,in For each candidate projection point Combined with reference observation UAV pose error covariance matrix Collaborative observation drones pose error covariance matrix Relative observation error covariance matrix and pixel measurement noise covariance matrix The covariance matrix of the projection position of the candidate projection point in the co-observed image plane is calculated by first-order error propagation. Based on the projection position covariance matrix and pre-set reliability level At each candidate projection point Construct a two-dimensional probability confidence region around it. and all two-dimensional probability confidence regions By merging, the geographic prior search region of candidate target m in the image plane of the collaborative observation UAV is obtained. ,Right now:

[0101] in, Indicates the search segment for the polar line. The number of candidate projection points obtained by discrete sampling.

[0102] Step 2.5: Based on the epipolar search segment obtained in Step 2.3 and the projection position covariance matrix obtained in step 2.4 In the collaborative observation UAV image plane, a geographic prior search region (ROI) containing the true projection location of candidate targets is constructed.

[0103] In one possible embodiment, when the relative baseline between the reference observation UAV and the cooperative observation UAV is too short, the line-of-sight angle between the two UAVs is too small, or the target pole falls into the effective image area of ​​the cooperative observation UAV, resulting in a reduction in the epipolar constraint capability, the geographical prior search area is expanded to reduce the risk of target missed detection due to geometric constraint degradation.

[0104] Specifically, let the spatial baseline length between the reference observation UAV and the cooperative observation UAV be:

[0105] When the baseline length satisfies:

[0106] Alternatively, if the angle between the two observation lines is less than a preset angle threshold, the current observation combination is deemed to have a risk of geometric constraint degradation; in this case, a preset expansion coefficient is used. Adjust the projection position covariance matrix:

[0107] The extended geographic prior search area is then regenerated based on the adjusted projection location covariance matrix.

[0108] If the area of ​​the expanded geographic prior search region exceeds the preset maximum area threshold, the combination of the reference observation UAV and the cooperative observation UAV is marked as a low geometric constraint confidence combination, and other cooperative observation UAVs with longer effective baselines or larger line-of-sight intersection angles are preferentially selected to perform cross-view target matching.

[0109] In one possible embodiment, the specific process of constructing the geographic prior search area (ROI) in step 2.5 includes: For each discrete sampling projection point on the epipolar search segment According to the preset reliability level Constructing a two-dimensional probability confidence region:

[0110] in, This indicates a degree of freedom of 2 and a confidence level of 1. The corresponding chi-square distribution threshold.

[0111] The probability confidence regions corresponding to each sampling projection point on the epipolar search segment are merged to obtain the candidate target. Geographic prior search area in the collaborative observation UAV image plane:

[0112] in, This represents the number of projection points obtained by discretely sampling the epipolar search segment.

[0113] The geographic prior search area Indicate candidate target The search range is probabilistically limited within the collaborative observation UAV image plane. In subsequent cross-view target matching, only candidate feature points located within the aforementioned geographic prior search area are extracted and matched, thereby reducing the unnecessary computational load from full-image search and incorrect matching caused by background interference.

[0114] In one possible embodiment, step 3 specifically includes the following process: In the cross-view feature matching stage, the geographical prior search area (ROI) generated in step 2 is first used to limit the range of candidate feature points in the collaborative observation image. Then, the reference feature point set and the collaborative candidate feature point set within the ROI are input into the pre-trained deep feature matching network to obtain the cross-view feature point matching relationship, and target consistency judgment and multi-UAV target ID association are performed accordingly.

[0115] Step 3.1: For the reference observation UAV determined in Step 2 and collaborative observation drones Based on the candidate targets in the reference observation UAV Local target observation records are used to obtain the target image region, target center pixel coordinates, visual feature descriptors, and local tracking markers; simultaneously, in collaborative observation of UAVs... In the image, the geographic prior search area generated in step 2 Filter the set of candidate feature points located within this region:

[0116] in, Indicates the first in the collaborative observation UAV image 1 candidate feature point.

[0117] Extract a set of reference feature points from the target image region of the reference observation UAV:

[0118] in, Indicates the first [unclear] in the target image region of the reference observation UAV. One feature point, This indicates the number of reference feature points.

[0119] Obtain the reference feature point set respectively and collaborative candidate feature point set The corresponding subset of visual feature descriptions is denoted as:

[0120] in, Indicates that it is located in the geographic prior search area The number of candidate feature points within.

[0121] This step limits the input of the graph matching network to the probability search region generated in step 2, avoiding a global search of the entire image of the collaborative observation UAV, thereby reducing unnecessary computation and background interference.

[0122] Step 3.2: Set the reference feature points and collaborative candidate feature point set Each feature point in the graph is used as a graph node, and the two-dimensional position, visual feature descriptor, and position offset relative to the center of the geographic prior search area of ​​each feature point are encoded to obtain the initial node state vector:

[0123] in, This represents a multilayer perceptron encoder. and Let represent the initial state vectors of feature points in the reference observation image and the cooperative observation image, respectively. It indicates the center location of the geographic prior search area.

[0124] By introducing This enables graph matching networks to perceive the positional relationship of candidate feature points relative to the geographic prior search area, thereby enhancing the geometric constraint capability of cross-view matching.

[0125] Step 3.3: Set the reference feature points from the reference observation image. and its visual feature description subset A set of collaborative candidate feature points located within the geographic prior search region (ROI) in the collaboratively observed image. and its visual feature description subset Input the pre-trained deep feature matching network.

[0126] In one specific embodiment, the pre-trained deep feature matching network employs the LightGlue network. Local feature points and feature descriptors are extracted from the target image region in the reference observation image using a local feature extractor. In the co-observation image, local feature points and feature descriptors are extracted only from candidate regions located within the geographic prior search area (ROI). The position coordinates of the two sets of local feature points and their feature descriptors are input into the LightGlue network, which then outputs the local feature matching relationship between the two images.

[0127] In other embodiments, the pre-trained deep feature matching network may also employ a SuperGlue network or other local feature matching networks based on graph neural networks or attention mechanisms. The local feature extractor may employ SuperPoint, DISK, SIFT, ORB, or other feature extraction methods capable of outputting local feature points and descriptors. This invention is not limited to the internal network structure, training method, or parameter update method of a specific deep feature matching network. The aforementioned LightGlue network, SuperGlue network, and the accompanying local feature extraction methods such as SuperPoint, DISK, SIFT, and ORB are all existing mature local feature extraction or local feature matching methods. In this invention, they are only used as pre-trained deep feature matching modules or feature extraction modules, and it is not necessary to redesign the network structure for this invention.

[0128] In an optional embodiment, the deep feature matching network includes multiple attention aggregation layers, in the... In each layer, self-attention aggregation is performed on feature points within a single image, and cross-attention aggregation is performed on feature points between two images.

[0129] For feature points in the reference observation image Its state update can be represented as:

[0130] For feature points in collaborative observation images Its state update can be represented as:

[0131] in, Indicates the first The set of feature point states of the reference observation image in the layer. Indicates the first The set of feature point states in the collaboratively observed image within the layer. This represents the self-attention operation. This represents the cross-attention operation. This represents the feature fusion update function.

[0132] Through the aggregation of self-attention and cross-attention, the network simultaneously utilizes the internal contextual information of a single image and the appearance similarity information across different viewpoints to obtain a high-dimensional node state representation for matching judgment.

[0133] Step 3.4: When the pre-trained deep feature matching network uses a LightGlue network with feature filtering capabilities or other graph matching networks, in the graph matching network's... After the layer output, the matching confidence score is calculated for each feature point that has not yet been removed. For feature points in the reference observation image... Its matching confidence level is:

[0134] For feature points in the collaborative observation image Its matching confidence level is:

[0135] in, This represents the Sigmoid activation function. and This represents the learnable confidence prediction parameter.

[0136] Set feature pruning threshold When the matching confidence of a feature point is lower than the feature pruning threshold, the feature point is marked as an invalid feature point and removed from the attention calculation and matching assignment calculation of subsequent network layers. or

[0137] Then the corresponding feature point will no longer participate in the first step. Feature aggregation calculation of the layer and its subsequent layers.

[0138] Through dynamic feature pruning, graph matching networks can reduce the number of invalid nodes layer by layer during inference, reduce the computational complexity of attention, and reduce the interference of background feature points on the matching results of the real target.

[0139] Step 3.5: When the pre-trained deep feature matching network outputs the layer-by-layer matching allocation results, after each layer of the graph matching network outputs, calculate the matching allocation matrix between the reference observation image and the co-observation image based on the current effective feature point state:

[0140] in, and They represent the first The effective feature point state matrix output by the layer. This represents the dimension of the feature state vector.

[0141] Calculate the difference rate between the matching assignment matrices of two adjacent layers:

[0142] in, Describing the Frobenius norm, To prevent constants with a denominator of zero.

[0143] Set dynamic early retirement threshold .when:

[0144] Furthermore, when the average matching confidence of the current layer exceeds a preset confidence threshold, the matching assignment matrix is ​​considered to have converged, the inference of subsequent network layers is terminated early, and the matching assignment matrix output by the current layer is taken as the final matching result. To maintain consistency with subsequent matching and filtering processes, the final output matching assignment matrix is ​​denoted as the matching result matrix. In this embodiment, .

[0145] Through a dynamic early termination mechanism, graph matching networks can adaptively adjust the inference depth according to the matching difficulty of different scenarios, reducing unnecessary network layer computation when the target texture is obvious, the viewpoint difference is small, or the ROI constraint is strong.

[0146] Step 3.6: Based on the matching result matrix obtained in Step 3.5 Feature points in the reference observation image and the collaborative observation image are selected using a one-to-one matching constraint; among them, Represents reference feature point With collaborative candidate feature points The matching score, matching probability, or matching confidence between the reference feature points; and collaborative candidate feature points If it satisfies:

[0147] And satisfy:

[0148] Then determine the reference feature point With collaborative candidate feature points Satisfy the bidirectional nearest neighbor constraint and determine the two as effective cross-view feature point matching pairs; This yields the set of cross-view feature point matching pairs corresponding to the candidate targets:

[0149] in, This indicates the preset matching confidence threshold.

[0150] Step 3.7: Based on the cross-view feature point matching pair set obtained in Step 3.6, perform consistency judgment on the candidate targets in the reference observation UAV and the candidate targets in the cooperative observation UAV.

[0151] Step 3.8: When determining the reference observation drone Candidate targets With collaborative observation drones Candidate targets When dealing with the same physical target, establish a multi-UAV target ID association table.

[0152] The multi-UAV target ID association table includes a global target ID, UAV node identifier, local tracking identifier, observation timestamp, target detection box, target center pixel coordinates, target category, detection confidence, target visual feature descriptor or its hash code, and a list of observed UAVs associated with the global target ID.

[0153] In one possible embodiment, the specific process of establishing the multi-UAV target ID association table in step 3.8 includes: For candidate targets that are confirmed to correspond to the same physical target through target consistency judgment and do not yet have a global target ID, a new global target ID is assigned, and the UAV node identifiers and corresponding local tracking identifiers of the reference observation UAV and the cooperative observation UAV are written into the global target ID field. For targets that already have a global target ID, the newly added UAV node identifier and its local tracking identifier are appended to the corresponding global target ID field. Each confirmed association record in the multi-UAV target ID association table uses the global target ID as the main index and records the UAV node identifier and local tracking identifier corresponding to the global target ID. For candidate targets that fail the target consistency judgment, they are not written into the confirmed multi-UAV target ID association table, but are assigned a pending confirmation identifier and stored in the pending confirmation target set. For the target to be confirmed, the collaborative processing node continues to retain its UAV node identifier, local tracking identifier, image acquisition timestamp, target center pixel coordinates, target category, detection confidence, and visual feature descriptor or compressed feature summary; when subsequent image frames, new observation data from other UAVs, or changes in target status meet the reconfirmation conditions, the geographic prior search area (ROI) generation process in step 2 and the cross-view feature matching and target consistency judgment process in step 3 are retried to further confirm the target to be confirmed.

[0154] The multi-UAV target ID association table can be represented as:

[0155] in, Indicates the first A global target ID, Indicates the drone node identifier, This indicates the local tracking marker of the drone targeting the corresponding target. This indicates the corresponding observation timestamp.

[0156] The multi-UAV target ID association table can clearly record which UAVs observed the same physical target within the corresponding time window, and provides a target-level association basis for subsequent steps such as multi-source identification summary collection, spatiotemporal alignment, global association cost matrix construction, and three-dimensional coordinate calculation.

[0157] In one possible embodiment, the specific process of performing the consistency judgment in step 3.7 includes: Calculate the target-level matching score between the two:

[0158] in, The geometric consistency score is derived from the number of valid feature point matching pairs, the reprojection error of matching points, or the epipolar distance. This represents the appearance consistency score obtained from the similarity of the target visual feature descriptors; This represents the category consistency score, obtained by determining whether the target category is consistent. , , For the corresponding weight coefficients, and satisfying:

[0159] When the target-level matching score Greater than the preset target association threshold Furthermore, the number of effective cross-view feature point matching pairs is not less than a preset threshold. At that time, the reference observation drone was determined. Candidate targets With collaborative observation drones Candidate targets They correspond to the same physical target.

[0160] In one possible embodiment, in step 4, the collaborative processing node receives lightweight identification summaries sent by each UAV node according to the event-triggered communication and adaptive transmission mechanism, performs group management, spatiotemporal alignment, and consistency verification on the received multi-source observation data, and completes cross-view collaborative 3D positioning and status update, specifically including the following sub-steps.

[0161] Step 4.1: Grouping and managing observation data and its association with targets The collaborative processing nodes parse the global target ID or unconfirmed identifier carried in each lightweight identification summary and group the observation data from different UAV nodes: For targets for which a global target ID has been established in step 3, lightweight identification summaries with the same global target ID are grouped into the same target observation set. Let the global target ID be The target's corresponding multi-source observation data set is:

[0162] in, Indicates drone At any moment A lightweight identification digest is sent for the m-th target. This indicates that the multi-UAV target ID association table established in step 3 is related to the global target ID. The corresponding associated records.

[0163] For unconfirmed targets that have not yet been assigned a global target ID, their summary is compared with the historical observation information of existing targets, and combined with the observation results of other UAVs received subsequently, it is determined whether they meet the conditions for being incorporated into an existing global target ID. For targets that do not meet the inclusion criteria but have stable observational evidence, they are submitted to the target ID association mechanism described in step 3 to establish a new global target ID.

[0164] Step 4.2: Spatiotemporal alignment and motion compensation based on timestamps Since there are differences in the image sampling time, summary sending time and communication transmission delay among different UAV nodes, the collaborative processing node establishes a sliding time window based on the image acquisition timestamp in each summary.

[0165] Let the unified reference time to be merged be Observations that satisfy the following formula will be included in the same spatiotemporally aligned observation set:

[0166] in, This indicates the preset maximum allowable time difference threshold for spatiotemporal alignment.

[0167] Regarding the data collection time and the unified reference time For observation data that is not entirely consistent but still within the allowable time range, time compensation is applied to the observed position based on the target's historical motion state and the output of the prediction filter. The target... At any moment State estimation is denoted as Its state transition matrix is ​​denoted as Then compensate to the unified reference time. The predicted state is represented as:

[0168] The aforementioned time compensation reduces the impact of asynchronous sampling and communication delays on subsequent consistency verification and 3D positioning.

[0169] Step 4.3: Multi-view observation consistency verification For multi-source observation data that has completed spatiotemporal alignment, the collaborative processing nodes perform consistency verification based on the global target ID association established in step 3. For the target... In drones With drones Based on two observations, a comprehensive consistency cost function is constructed:

[0170] in, Represents the cost of geometric consistency. This indicates the cost of appearance consistency. This represents the cost of motion continuity. , and Let represent the weight coefficients of the corresponding cost terms, and satisfy:

[0171] The geometric consistency cost is determined based on the target center pixel coordinates, the UAV's pose at the corresponding time, and camera calibration parameters. In one implementation, the geometric consistency cost is represented by the distance between the target observation point and the corresponding epipolar line.

[0172] in, Indicate target In drones Homogeneous pixel coordinates in the image Indicate target In drones Homogeneous pixel coordinates in the image Indicated by drones With drones The basic matrix is ​​determined by the relative pose and camera calibration parameters.

[0173] The appearance consistency cost is determined based on the cosine distance between the compressed feature summaries uploaded by different drones:

[0174] in, and Representing the target In drones With drones The corresponding compressed feature summary.

[0175] The motion continuity cost is determined based on the Mahalanobis distance between the target predicted position output by the prediction filter and the current observation fusion position:

[0176] in, According to the drone With drones The current preliminary observations of the target location are as follows. Indicates the predictive filter at the unified reference time. Output target predicted state, This represents the covariance matrix corresponding to the predicted residuals.

[0177] When the overall consistency cost is satisfied:

[0178] At that time, it is determined that the two observation data still correspond to the same physical target, where, This indicates the preset consistency verification threshold. For newly added observation data that meets the consistency verification conditions, its UAV node identifier and local tracking identifier are appended to the global target ID. In the corresponding associated records, for observation data that does not meet the consistency verification conditions, it is marked as associated conflict data, and steps 2 and 3 are triggered to re-execute the geographic prior search area generation and cross-view target matching process for the target.

[0179] Step 4.4: Multi-view weighted fusion 3D localization For multiple valid viewpoint observations corresponding to the same physical target, which have been confirmed through consistency verification, the collaborative processing node reads the target center pixel coordinates, camera intrinsic parameter matrix, camera extrinsic parameter matrix, and UAV pose information at the corresponding image acquisition time for each valid observation. Let the target be... have The first effective perspective observation, the first The projection matrix corresponding to each effective viewpoint is: The homogeneous pixel coordinates of the target in this view are:

[0180] Target The three-dimensional position in a unified world coordinate system is denoted as:

[0181] Based on the projection relationship of the pinhole camera, each effective observation satisfies:

[0182] Among them, symbols It represents the equality relationship in the sense of homogeneous coordinates.

[0183] Due to detection errors, pose estimation errors, and image measurement noise, multiple spatial observation rays typically cannot precisely converge at the same point. Therefore, the collaborative processing node employs a method that minimizes multi-view reprojection errors to solve for the target's 3D position.

[0184] in, Indicate target The three-dimensional fusion localization results This represents a projection function that converts homogeneous projected coordinates into two-dimensional pixel coordinates. Indicates the first Positioning weights for each effective perspective observation.

[0185] In one implementation, the first The positioning weights for each effective viewpoint observation are determined based on the detection confidence and pose error:

[0186] in, Indicates the first Target from an effective perspective The detection confidence level, This represents the covariance matrix of the corresponding UAV pose estimation error. Represents the matrix trace operation. This represents a positive constant to prevent the denominator from being zero. By setting the weights as described above, observation data with high detection confidence and small pose errors contribute more significantly to the 3D position fusion process.

[0187] Step 4.5: Determining the effectiveness of the location and assessing overall confidence. In obtaining the target 3D fusion localization results Then, calculate its average reprojection error:

[0188] When the following conditions are met:

[0189] At that time, the current 3D positioning result is confirmed to be valid, whereby, This indicates the preset average reprojection error threshold. This represents the minimum number of effective observation angles required to complete reliable 3D positioning.

[0190] When the average reprojection error exceeds the preset average reprojection error threshold, or the number of effective observation views is insufficient, the current positioning result is marked as a low-confidence result. For low-confidence results, the collaborative processing node triggers the expansion of the observation time window to be fused, requests other UAVs with effective observation baselines to provide lightweight identification summaries, or re-executes steps 2 and 3 to obtain cross-view observation data that meets the positioning accuracy requirements.

[0191] For global targets with valid positioning results The collaborative processing nodes generate a comprehensive confidence score based on the detection confidence score of each effective viewpoint, the consistency verification cost, and the average reprojection error. In one implementation, the comprehensive confidence score of the target is expressed as:

[0192] in, Indicate target The corresponding average cost of consistency verification Indicate target The average reprojection error, , and To calculate the weights for the overall confidence level, and satisfy the following conditions: .

[0193] Step 4.6: Global Target State Update and Closed-Loop Feedback The collaborative processing node will target 3D fusion localization results An input prediction filter is used to update the target's motion state and predicted position at the next moment, and a global target state record is generated. The global target state record includes at least the global target ID, target category, overall confidence score, 3D position, motion state, most recent update timestamp, a list of UAV nodes involved in association and 3D localization, and the current association status.

[0194] When the communication silence condition is met, the collaborative processing node feeds back the confirmed and stable global target status to the corresponding observation UAV, enabling each observation UAV to continue performing local target tracking with low communication overhead based on the global target ID and local tracking status. When subsequent visual feature mutations, local tracking loss, correlation conflicts, or reduced reliability of 3D positioning results occur, the collaborative processing node reassembles the relevant lightweight identification summaries and performs spatiotemporal alignment, consistency verification, and 3D position calculation again, thereby achieving continuous updates of target identity, spatial position, and motion status during multi-UAV collaborative perception.

[0195] Compared with the prior art, the present invention has the following significant advantages: 1) Based on the target observation data of the reference observation UAV, the pose information of the reference observation UAV and the cooperative observation UAV, the camera calibration parameters and the pose estimation error, this invention generates a geographic prior search region (ROI) of the target in the image plane of the cooperative observation UAV. This enables subsequent cross-view target matching to be performed within the spatially constrained candidate region without performing a global search on the entire image, thereby reducing the number of invalid candidate feature points participating in the matching calculation and reducing the interference of complex backgrounds on the target association results.

[0196] 2) This invention invokes a pre-trained deep feature matching network within the geographic prior search area (ROI) to perform cross-view feature matching, and utilizes ROI spatial constraints, matching confidence filtering, and an optional adaptive inference mechanism to reduce unnecessary computation. Since the input of the pre-trained deep feature matching network is restricted to the ROI, it reduces the participation of irrelevant background feature points in the matching calculation and reduces the interference of complex backgrounds on the target association results. Simultaneously, when using an existing deep feature matching network with adaptive inference capabilities, unnecessary network layer inference can be further reduced, thereby reducing the processing load on the onboard computing unit and improving the efficiency of collaborative perception under resource-constrained conditions.

[0197] 3) This invention establishes or updates a multi-UAV target ID association table based on cross-view target matching results, and further sets up an event-triggered communication mechanism according to the target association status and local tracking status. For targets that are stably tracked and have formed a consistent association, each observation UAV can reduce the transmission of frame-by-frame identification summary; for newly appearing targets, targets with abrupt changes in visual features, targets lost during local tracking, or targets awaiting confirmation, a lightweight identification summary is triggered. Simultaneously, the system adaptively adjusts the summary transmission frequency, feature compression dimension, and target transmission priority based on the available bandwidth, transmission latency, and packet loss rate of the communication link, thereby reducing unnecessary data transmission and helping to maintain the collaborative perception process of key targets under conditions of limited communication resources or fluctuating communication status.

[0198] 4) This invention uses a multi-UAV target ID association table as a foundation to aggregate and spatiotemporally align lightweight identification summaries from different observing UAVs. It also verifies cross-view association results by combining geometric consistency, appearance consistency, and motion continuity. For multi-view observation data confirmed to belong to the same physical target, a multi-view 3D positioning method is further employed to obtain the target's spatial position in a unified world coordinate system. The reliability of the positioning results is judged based on reprojection error and the number of effective observation views. This improves the stability of target identity association and spatial position estimation during multi-UAV collaborative perception and enhances the system's tolerance to single-UAV missed detections, local tracking anomalies, or observation errors. Attached Figure Description

[0199] Figure 1 This is an overall flowchart of the multi-view collaborative perception method of the present invention; Figure 2 This is a schematic diagram illustrating the cross-viewpoint epipolar geometry and spatial line-of-sight projection principle between the reference observation UAV and the cooperative observation UAV in this invention. Figure 3 This is a schematic diagram illustrating the generation of geographic prior search areas (ROIs) based on pose uncertainty in this invention. Figure 4 This is a schematic diagram of the cross-view deep feature matching processing flow based on the geographical prior search area ROI constraint in this invention. Figure 5 This is a block diagram of the event-triggered communication, adaptive digest transmission, and multi-source fusion positioning logic in this invention. Detailed Implementation

[0200] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0201] It should be noted that the algorithm provided in this application is not a purely mathematical method, but rather is combined with a specific technical field to solve practical technical problems. The following detailed embodiments, in conjunction with the accompanying drawings, illustrate in detail how the technical solution of this application utilizes this algorithm to process technical data and achieve technical effects.

[0202] This invention provides a multi-view collaborative perception method based on spatiotemporal prior guidance and communication adaptation, which is applied to a multi-UAV collaborative perception system consisting of multiple observation UAVs and at least one collaborative processing node.

[0203] like Figure 1 As shown, the multi-view collaborative perception method of the present invention includes the generation of local target observation records, the generation of geographic prior search areas (ROIs), cross-view deep feature matching and target ID association based on ROI constraints, event-triggered communication and adaptive summary transmission, multi-source consistency verification, and multi-view three-dimensional positioning process. Figure 2 This illustrates the cross-viewpoint epipolar geometry and spatial line-of-sight projection relationship between the reference observation UAV and the cooperative observation UAV; Figure 3 This illustrates the process of generating a geographic prior search region (ROI) based on pose uncertainty; Figure 4 This demonstrates the process of cross-view feature matching by calling a pre-trained deep feature matching network under the constraint of the geographic prior search region (ROI). Figure 5The logical relationship between event-triggered communication, adaptive summary transmission, and multi-source fusion localization is illustrated.

[0204] Each observation UAV is equipped with an airborne visual sensor, GPS / RTK positioning module, IMU inertial measurement unit, wireless communication module, and airborne computing unit. These are used to acquire environmental images, obtain their own pose information, perform local target detection and tracking, generate local target observation records, and send lightweight identification summaries based on event triggering conditions. The collaborative processing node can be deployed in a main UAV with fusion computing capabilities or in a ground control station. It is used to organize cross-view target search, manage multi-UAV target ID association tables, receive lightweight identification summaries, perform multi-source fusion positioning, and output global target status information.

[0205] In this embodiment, the multi-UAV cooperative perception system can be applied to low-altitude defense scenarios. For example, multiple observation UAVs are deployed around the defense airspace or along patrol routes where targets may enter. When any observation UAV detects a suspected intrusion target within its field of view, the system does not require all UAVs to perform an indiscriminate search of the entire image. Instead, it uses the local target observation record generated by that observation UAV, combined with the pose information of other cooperative observation UAVs, to generate a geographic prior search area (ROI) with spatial probability constraints in the cooperative observation UAV image. Subsequently, the system performs cross-view target matching and target identity association only within the geographic prior search area (ROI), and triggers necessary data transmission and fusion positioning when the target identity or tracking status changes.

[0206] Example 1: Multi-UAV cooperative perception and target localization process under normal communication conditions like Figure 1 As shown, this embodiment illustrates the overall processing flow of the method of the present invention when the communication link is in good condition. It includes each observation UAV generating local target observation records, generating a geographic prior search area (ROI) based on the pose and camera parameters of the reference observation UAV and the cooperative observation UAV, performing cross-view target matching and target ID association within the geographic prior search area (ROI), generating and transmitting a lightweight identification summary based on the target association status, and having the cooperative processing node perform consistency verification and multi-view fusion positioning on the multi-source observation data.

[0207] In this embodiment, multiple observation drones are deployed around the airspace to be defended. Each observation drone is equipped with an airborne visual sensor, a positioning module, an inertial measurement unit, a wireless communication module, and an airborne computing unit. The collaborative processing node is located in the main drone or a ground control station. The communication links between each observation drone and the collaborative processing node are in good condition, and their available bandwidth, average transmission latency, and packet loss rate all meet the normal transmission requirements for a complete lightweight identification summary.

[0208] Step 1, Local Target Observation Record Generation: Each observation UAV acquires environmental images according to a preset sampling period and simultaneously obtains the UAV pose information and pose estimation error information at the corresponding time. When the first observation UAV detects a candidate target in the current image frame, it uses a local target tracker to determine the local tracking identifier of the candidate target, extracts the visual feature descriptor of the candidate target, and associates and encapsulates the UAV node identifier, image acquisition timestamp, target detection box, target center pixel coordinates, target category, detection confidence, local tracking identifier, visual feature descriptor, UAV pose information, pose estimation error information, and camera calibration parameters to generate a local target observation record.

[0209] Step 2, Geographic Prior Search Area (ROI) Generation: The first observation UAV that detected the candidate target is identified as the reference observation UAV. A second observation UAV, which acquired the image to be matched within a preset collaborative time window and has a valid observation baseline with the reference observation UAV, is selected as the collaborative observation UAV. Based on the target center pixel coordinates in the reference observation UAV, the pose information of the two UAVs, camera calibration parameters, and pose estimation error, the spatial line of sight corresponding to the target is projected onto the image plane of the collaborative observation UAV to obtain the epipolar search segment. The geographic prior search area (ROI) is then generated by combining the pose error and image measurement error.

[0210] Step 3, Cross-view matching and target ID association within the ROI: Cooperative candidate feature points are extracted only within the geographic prior search area (ROI) and input into a pre-trained deep feature matching network along with reference feature points in the target image region of the reference observation UAV. The pre-trained deep feature matching network can employ LightGlue, SuperGlue, or other local feature matching networks based on graph neural networks or attention mechanisms. These networks are all existing mature local feature matching networks; in this embodiment, they are only used as pre-trained matching modules and do not require redesigning the network structure or performing special training for this invention. When using networks with adaptive inference capabilities, such as LightGlue, their matching confidence filtering or adaptive inference mechanisms can be used to reduce the participation of low-confidence feature points or low-contribution matching relationships in subsequent calculations. Based on the cross-view feature point matching results output by the graph matching network, target category consistency, appearance similarity, and geometric consistency, it is confirmed that the candidate targets in the first and second observation UAVs correspond to the same physical target.

[0211] For targets that do not yet have a global target ID, the collaborative processing node assigns a new global target ID to it and writes the local tracking identifiers corresponding to the two UAVs into the global target ID field, thus establishing a multi-UAV target ID association table.

[0212] Step 4, Event-Triggered Communication and Adaptive Transmission: Under normal communication conditions, each observation UAV generates a lightweight identification summary based on its local target observation records and a multi-UAV target ID association table, and sends the complete lightweight identification summary to the collaborative processing node at a preset normal transmission frequency. For targets that have entered a stable collaborative tracking state, the observation UAV stops repeatedly transmitting frame by frame and only sends status confirmation information at a preset low-frequency period.

[0213] Step 5, Multi-source fusion positioning and status output: The collaborative processing node collects multi-UAV observation data associated with the same global target ID or unconfirmed identifier based on the multi-UAV target ID association table established or updated in Step 3, and performs grouping, spatiotemporal alignment and consistency verification on the multi-UAV observation data; for multi-source observation data that has been confirmed by consistency verification to correspond to the same physical target and have at least two effective observation perspectives, the three-dimensional position of the target in a unified world coordinate system is calculated, and the corresponding global target status information is output.

[0214] Through the complete processing of this embodiment, the system can complete a continuous collaborative perception process from single-machine target discovery, cross-view search, target identity association, summary transmission to multi-view fusion positioning under good communication conditions.

[0215] Example 2: Adaptive Cooperative Sensing and Target Localization Process under Dynamic Weak Network Conditions This embodiment illustrates how the method of the present invention completes the entire implementation process of local observation, cross-view matching, target identity association, adaptive summary transmission, and multi-source fusion positioning in low-altitude defense scenarios where communication bandwidth between UAVs fluctuates dynamically, transmission latency increases, or packet loss rate rises.

[0216] Step 1: Each observation UAV continuously acquires environmental images, synchronously obtains UAV pose and pose estimation error information, and performs local target detection, local target tracking, and visual feature extraction to generate a local target observation record. When an observation UAV detects a candidate target that has not yet appeared in the multi-UAV target ID association table, it sets the candidate target as a target to be confirmed and assigns it a confirmation identifier.

[0217] Step 2: The UAV that detected the target to be confirmed is identified as the reference observation UAV. From the other UAVs, a cooperative observation UAV that meets the cooperative time window requirements and has an effective observation baseline is selected. Based on the target center pixel coordinates in the reference observation UAV, the pose information of the two UAVs, camera intrinsic and extrinsic parameters, and pose estimation error, a geographic prior search region (ROI) corresponding to the target is generated in the image plane of the cooperative observation UAV. If the relative baseline between the two UAVs is too short or the angle between their lines of sight is too small, the ROI is expanded. If the expanded search region still cannot meet the preset geometric confidence requirements, another UAV with better observation geometry is selected to perform cooperative matching.

[0218] Step 3: Within the geographic prior search area (ROI), a pre-trained deep feature matching network is invoked to perform cross-view feature matching. ROI spatial constraints, matching confidence filtering, and optional adaptive inference mechanisms are utilized to reduce unnecessary computation. If candidate targets in the reference observation UAV and the collaborative observation UAV pass the target-level consistency judgment, the collaborative processing node assigns them a global target ID, or adds a new local tracking identifier to an existing global target ID. If it cannot be confirmed that they belong to the same physical target, the pending confirmation identifier is retained, and the geographic prior search area (ROI) generation, cross-view feature matching, and target consistency judgment process are re-triggered in subsequent image frames, when other UAVs add observation data, or when the target state changes.

[0219] Furthermore, under dynamic weak network conditions, each observation UAV determines the transmission method of the lightweight identification summary based on the target association status, local tracking status, and current communication link status. When a new target appears for the first time, visual features change abruptly, local tracking is lost, prediction residuals exceed a threshold, or target association conflicts occur, each observation UAV triggers the transmission of the lightweight identification summary for the corresponding target. For targets that have entered a communication silence state, when visual features change abruptly, local tracking is lost, prediction residuals exceed a threshold, or target association conflicts occur, the UAV exits the communication silence state and re-triggers the transmission of the lightweight identification summary.

[0220] When the communication link is in good condition, a complete lightweight identification summary containing a high-dimensional compressed feature summary is sent at the normal transmission frequency. When the available bandwidth decreases or the average transmission delay approaches the maximum allowable transmission delay, the transmission frequency and feature summary dimension are reduced, and only the target category, detection confidence, target center pixel coordinates, local tracking identifier, global target ID or unconfirmed identifier, and low-dimensional feature summary are retained. When the communication link enters an extremely weak network state, only the minimum identification summary of high-priority targets is sent. The minimum identification summary includes the UAV node identifier, timestamp, local tracking identifier, global target ID or unconfirmed identifier, target category, target center pixel coordinates, and detection confidence.

[0221] When multiple targets trigger digest transmission simultaneously, and the current communication resources are insufficient to transmit all digests, the transmission priority is calculated based on the target threat level, detection confidence, target association status, local tracking status, and spatial relationship between the target and the defense zone. Digests corresponding to newly emerging targets, unconfirmed targets, locally tracked lost targets, and high-threat-level targets are transmitted first.

[0222] Step 4: The collaborative processing node groups, caches, and spatiotemporally aligns the received lightweight identification summaries, and performs a consistency check based on the currently obtainable target center pixel coordinates, compressed feature summaries, UAV pose information, and historical motion states in the summaries. When the number of received effective observation views meets the 3D positioning requirements, the 3D position of the target is calculated; when the number of effective observation views is insufficient, the reprojection error is too large, or some key summaries are not delivered in time due to weak networks, the current positioning result is marked as low confidence, and the node waits for subsequent summaries, requests auxiliary observation data from other observation UAVs, or re-executes target association and fusion positioning after the communication link is restored.

[0223] Once the communication link is restored, each observation UAV sends necessary lightweight identification summaries based on the latest target status. The collaborative processing node updates the multi-UAV target ID association table and global target status information, thereby maintaining continuous perception of target identity and target spatial location under dynamic weak network conditions.

[0224] Effect verification method: To verify the effectiveness of the method of the present invention under conditions of limited airborne computing resources and dynamic changes in communication link status, a test system comprising multiple observation UAVs, a wireless communication link simulation module, and collaborative processing nodes is constructed in one verification method. Each observation UAV is equipped with an airborne visual sensor, a positioning module, an inertial measurement unit, a wireless communication module, and an edge computing module with model inference capabilities; the collaborative processing node is located in a ground control terminal or a main UAV platform with fusion computing capabilities.

[0225] The test scenarios include one or more of the following: target entering the defense zone, target being obscured, target rapidly maneuvering, multiple targets appearing simultaneously, and dynamic changes in communication link bandwidth, average transmission latency, and packet loss rate.

[0226] During the test, the following data were recorded: the generated geographic prior search area, the number of candidate feature points participating in cross-view matching, the actual inference layers of the graph matching network, the target association results, the lightweight recognition summary transmission volume, the summary transmission latency, the 3D positioning results, and the target continuous tracking status.

[0227] In one possible embodiment, it can be combined with Figures 1 to 5The technical effects of this invention were verified. Specifically, when verifying the role of the geographic prior search region (ROI), a local matching method based on the ROI and a full-map matching method without the ROI were used respectively. The number of candidate feature points involved in the calculation, the computational cost of feature matching, the matching processing time, and the number of incorrect matches were compared between the two methods. The verification results show that after using the ROI, the number of candidate feature points and irrelevant background feature points in the collaboratively observed image is reduced, and the cross-view matching search range is limited to the probability region near the true projection location of the target. Therefore, it can reduce the computational cost of invalid feature matching and reduce incorrect matches caused by background interference.

[0228] To verify the performance of pre-trained deep feature matching networks under the constraints of geographic prior search areas (ROIs), we employed deep feature matching networks with optional feature filtering or adaptive inference mechanisms, as well as a fixed-depth deep feature matching network. We compared the actual number of inference layers, the number of feature points involved in the computation, the cross-view matching processing time, and the target association results under both methods. The verification results show that, under conditions of strong ROI constraints, obvious target texture, or rapid convergence of two-view matching relationships, pre-trained deep feature matching networks with adaptive inference capabilities can reduce the participation of low-confidence feature points or low-contribution matching relationships in subsequent computations and reduce unnecessary network layer inference. This is because the ROI spatial constraints have pre-excluded a large number of irrelevant background areas, and the matching network only needs to confirm cross-view feature relationships within a relatively small candidate range.

[0229] In verifying the adaptive communication mechanism, two methods were employed: one sending target observation data at fixed intervals, and the other triggering lightweight identification summary transmission based on target association status, local tracking status, and communication link status. The data transmission volume, average transmission latency, key target summary transmission success rate, and the number of valid summaries received by the collaborative processing nodes were compared between the two methods. Verification results show that event-triggered communication and adaptive summary transmission can reduce the repeated summary transmission of stable targets and prioritize summary transmission for new targets, unconfirmed targets, locally tracked lost targets, and high-threat targets when communication links are limited. This reduces communication bandwidth consumption and improves the continuity of collaborative perception of key targets.

[0230] To verify the multi-source fusion localization capability, target 3D position estimation results were obtained based on two and three or more effective observation perspectives, respectively. The target position estimation error, reprojection error, and stability of the localization results across different numbers of effective observation perspectives were compared. The verification results show that when the number of effective observation perspectives involved in the fusion increases and the cross-perspective target ID association remains consistent, the reprojection error and position fluctuation across consecutive frames of the multi-view 3D localization results decrease, and the stability of target spatial localization improves.

[0231] Through the above verification methods, the effects of the geographic prior search area (ROI), matching confidence screening and optional adaptive reasoning mechanism, communication adaptive mechanism, and multi-source fusion positioning process on the collaborative sensing process of this invention can be evaluated respectively.

Claims

1. A multi-view collaborative sensing method based on spatiotemporal prior guidance and communication adaptation, characterized in that, Includes the following steps: Step 1: Each UAV performs local target detection, local tracking, and visual feature extraction to form a local target observation record; Step 2: The first observation drone that has detected the candidate target and provided the center pixel coordinates of the candidate target is identified as the reference observation drone, and the second observation drone that has acquired the image to be matched within the preset cooperative time window and has an effective observation baseline with the reference observation drone is selected as the cooperative observation drone from the remaining observation drones. Based on the observation relationship between the reference observation UAV and the cooperative observation UAV, the geographic prior search region (ROI) corresponding to the candidate target is generated by projecting it onto the image plane of the cooperative observation UAV. Step 3: Within the geographic prior search area (ROI), invoke a pre-trained deep feature matching network to perform cross-view feature matching on candidate targets in the reference observation UAV and candidate targets in the cooperative observation UAV; Repeat steps 2-3 to iterate through all cooperative observation UAVs that meet the conditions, and establish or update the multi-UAV target ID association table based on the cross-view feature matching results; Step 4: The collaborative processing node performs a consistency check based on the multi-UAV target ID association table in Step 3. For multi-source observation data that have been confirmed to correspond to the same physical target and have at least two valid observation perspectives after the consistency check, the method of minimizing the multi-view reprojection error is used to calculate the three-dimensional position of the target in the unified world coordinate system.

2. The multi-view collaborative sensing method based on spatiotemporal prior guidance and communication adaptation according to claim 1, characterized in that, Step 1 specifically includes the following process: Step 1.1: For A collaborative perception system composed of drones will... The drone is recorded as ,in At any moment drones Image frames are acquired using an onboard vision sensor. It also reads the data output by the airborne positioning module and inertial measurement unit corresponding to the image frame; Step 1.2: Drone Image frames Input a lightweight object detection network pre-deployed on an onboard computing unit, and output a set of candidate objects in the current image frame: in, Indicates drone At any moment The number of candidate targets detected; candidate targets At least including: in, A detection bounding box representing a candidate target; This indicates the pixel coordinates corresponding to the center of the detection box; Indicates the target category; Indicates the confidence level of the detection; For detection confidence levels that meet the preset detection threshold Candidate targets are identified and associated with the detection results from the previous time step using the local target tracker; if the candidate target successfully matches an existing local trajectory, the existing local tracking identifier is inherited. If no match is found with an existing trajectory, a new local tracking identifier will be assigned to it. ; Step 1.3: For each candidate target whose detection confidence level meets the preset detection threshold, use its detection bounding box. The defined target image region is used as input, and a lightweight feature extraction network pre-deployed on the UAV's onboard computing unit is used to extract visual feature descriptors to describe the target's appearance attributes: in, This refers to the feature extraction model deployed on the UAV's onboard computing unit. Representing dimensions Target visual feature descriptor; Step 1.4: For drones At any moment The detected and tracked first For each candidate target, generate a local target observation record: drones At any moment All generated local target observation records are combined into a single local target observation record: 。 3. The multi-view collaborative sensing method based on spatiotemporal prior guidance and communication adaptation according to claim 2, characterized in that, In step 1.3, the visual feature descriptor is normalized using the following formula: This reduces the impact of different lighting conditions and image scales on subsequent cross-view appearance similarity calculations.

4. The multi-view collaborative sensing method based on spatiotemporal prior guidance and communication adaptation according to claim 1, characterized in that, Step 2 specifically includes the following process: Step 2.1: For the local target observation record set generated in Step 1, when the UAV... At any moment Candidate target detected When the detection confidence of the candidate target meets the preset detection threshold, the drone will be... The UAV was identified as the reference observation UAV, and its corresponding local target observation record was recorded as follows: The local target observation record includes at least the target center pixel coordinates. Reference observation of UAV attitude Pose estimation error covariance matrix And camera calibration parameters; Among the other UAV nodes participating in collaborative sensing, the UAVs that acquire the image frames to be matched within the preset collaborative time window and have an effective observation baseline with the reference observation UAV will be selected. It was determined to be a collaborative observation UAV; the image acquisition time corresponding to the collaborative observation UAV is denoted as... Its pose and pose estimation error covariance matrix are denoted as follows: and ; The preset collaborative time window is used to limit the maximum allowable time difference between the reference observation image and the collaborative observation image; when: At that time, it is assumed that the two image frames meet the time synchronization requirements for cooperative matching, where, The preset maximum time difference threshold; Step 2.2: For the reference observation UAV Candidate targets detected Based on the target center pixel coordinates: And the intrinsic parameter matrix of the camera carried by the reference observation drone. The target center pixel coordinates are then projected backwards onto the camera coordinate system of the reference observation UAV to obtain the normalized imaging direction vector: Based on the pose information of the reference observation UAV and the external parameter transformation relationship between the camera coordinate system and the UAV body coordinate system, the normalized imaging direction vector is transformed to a unified world coordinate system to obtain the spatial line-of-sight direction vector corresponding to the target: in, Indicates the time of reference observation drone The attitude rotation matrix relative to the world coordinate system, This represents the rotation matrix of the camera coordinate system on the reference observation drone relative to the drone's body coordinate system; Therefore, the candidate spatial position of the target in the world coordinate system can be represented as: in, This indicates the position of the optical center of the reference observation UAV camera in the world coordinate system. This represents the unknown depth parameter of the target along the line of sight in space. Step 2.3: Based on the application scenario of the target, the effective detection range of the airborne sensors, or the target's mission airspace boundary, pre-set the effective depth range of the target along the spatial line-of-sight direction: By taking the two endpoints of the effective depth range, the two endpoints of the target candidate spatial line segment are obtained: Transform the two endpoints to a collaborative observation drone. The camera coordinate system, and based on the camera intrinsic parameter matrix of the collaborative observation UAV. Projecting onto its image plane, we obtain: in, The projection function represents the perspective projection of a point in three-dimensional space onto an image plane; Connect projection points and , obtain candidate targets Collaborative observation drones Epipolar search segment on the image plane: Step 2.4: Express the pose errors of the reference observation UAV and the cooperative observation UAV as zero-mean Gaussian random variables, with their covariance matrices as follows: and ; The pose error covariance matrices of the reference observation UAV and the cooperative observation UAV are combined into a relative observation error covariance matrix: For any candidate spatial projection point on the epipolar search segment, its projection position on the collaborative observation UAV image plane is represented as a function of pose variables: in, and These represent the pose parameter vectors of the reference observation UAV and the cooperative observation UAV, respectively; According to the first-order error propagation principle, the covariance matrix of the projected position on the image plane of the cooperative observation UAV is expressed as: in, Represents the projection function Jacobian matrix for pose variables of the reference observation UAV and the cooperative observation UAV; This represents the pixel measurement noise covariance matrix caused by target center point detection error, image noise, and feature localization error. Step 2.5: Based on the epipolar search segment obtained in Step 2.3 and the projection position covariance matrix obtained in step 2.4 In the collaborative observation UAV image plane, a geographic prior search region (ROI) containing the true projection location of candidate targets is constructed.

5. A multi-view collaborative sensing method based on spatiotemporal prior guidance and communication adaptation according to claim 4, characterized in that, In step 2.5, the specific process of constructing the geographic prior search area (ROI) includes: For each discrete sampling projection point on the epipolar search segment According to the preset reliability level Constructing a two-dimensional probability confidence region: in, This indicates a degree of freedom of 2 and a confidence level of 1. The corresponding chi-square distribution threshold; The probability confidence regions corresponding to each sampling projection point on the epipolar search segment are merged to obtain the candidate target. Geographic prior search area in the collaborative observation UAV image plane: in, This represents the number of projection points obtained by discretely sampling the epipolar search segment.

6. The multi-view collaborative sensing method based on spatiotemporal prior guidance and communication adaptation according to claim 4, characterized in that, Step 3 specifically includes the following process: Step 3.1: Collaborative observation of UAVs In the image, the geographic prior search area generated in step 2 Filter the set of candidate feature points located within this region: in, Indicates the first in the collaborative observation UAV image 1 candidate feature point; Extract a set of reference feature points from the target image region of the reference observation UAV: in, Indicates the first [unclear] in the target image region of the reference observation UAV. One feature point, Indicates the number of reference feature points; Obtain the reference feature point set respectively and collaborative candidate feature point set The corresponding subset of visual feature descriptions is denoted as: in, Indicates that it is located in the geographic prior search area The number of candidate feature points within; Step 3.2: Set the reference feature points and collaborative candidate feature point set Each feature point in the graph is used as a graph node, and the two-dimensional position, visual feature descriptor, and position offset relative to the center of the geographic prior search area of ​​each feature point are encoded to obtain the initial node state vector: in, This represents a multilayer perceptron encoder. and Let represent the initial state vectors of feature points in the reference observation image and the cooperative observation image, respectively. Indicates the center location of the geographic prior search area; Step 3.3: Set the reference feature points from the reference observation image. and its visual feature description subset A set of collaborative candidate feature points located within the geographic prior search region (ROI) in the collaboratively observed image. and its visual feature description subset Input the pre-trained deep feature matching network; Step 3.4: Based on the feature point matching confidence, matching score, or feature point validity judgment result output by the pre-trained deep feature matching network, filter out low-confidence feature points or low-confidence matching relationships to obtain candidate cross-view feature point matching relationships; Step 3.5: Based on the matching assignment result, matching probability matrix, or matching score matrix output by the pre-trained deep feature matching network, determine the cross-view feature matching result between the reference observation image and the co-observation image, and represent the cross-view feature matching result as a matching result matrix. ;in, Represents reference feature point With collaborative candidate feature points The matching score, matching probability, or matching confidence between them; when using a deep feature matching network with adaptive inference capabilities, the deep feature matching network adaptively determines the number of inference layers based on the matching confidence or the convergence state of the matching results; when using a deep feature matching network with fixed depth, fixed-depth matching is performed on candidate feature points within the geographic prior search area (ROI) under the constraint of the ROI. Step 3.6: Based on the matching result matrix obtained in Step 3.5 Feature points in the reference observation image and the collaborative observation image are selected through a one-to-one matching constraint; for the reference feature points... and collaborative candidate feature points If it satisfies: And it satisfies: Then determine the reference feature point With collaborative candidate feature points Satisfy the bidirectional nearest neighbor constraint and determine the two as effective cross-view feature point matching pairs; This yields the set of cross-view feature point matching pairs corresponding to the candidate targets: in, This indicates the preset matching confidence threshold; Step 3.7: Based on the cross-view feature point matching pair set obtained in Step 3.6, perform consistency judgment on the candidate targets in the reference observation UAV and the candidate targets in the cooperative observation UAV; Step 3.8: When determining the reference observation drone Candidate targets With collaborative observation drones Candidate targets When corresponding to the same physical target, establish or update the multi-UAV target ID association table.

7. A multi-view collaborative sensing method based on spatiotemporal prior guidance and communication adaptation according to claim 6, characterized in that, In step 3.7, the specific process for performing the consistency judgment includes: Calculate the target-level matching score between the two: in, The geometric consistency score is derived from the number of valid feature point matching pairs, the reprojection error of matching points, or the epipolar distance. This represents the appearance consistency score obtained from the similarity of the target visual feature descriptors; This represents the category consistency score, obtained by determining whether the target category is consistent. , , For the corresponding weight coefficients, and satisfying: When the target-level matching score Greater than the preset target association threshold Furthermore, the number of effective cross-view feature point matching pairs is not less than a preset threshold. At that time, the reference observation drone was determined. Candidate targets With collaborative observation drones Candidate targets They correspond to the same physical target.

8. A multi-view collaborative sensing method based on spatiotemporal prior guidance and communication adaptation according to claim 6, characterized in that, In step 3.8, the specific process of establishing or updating the multi-UAV target ID association table includes: For candidate targets that are confirmed to correspond to the same physical target through target consistency judgment and do not yet have a global target ID, a new global target ID is assigned, and the UAV node identifiers and corresponding local tracking identifiers of the reference observation UAV and the cooperative observation UAV are written into the global target ID field. For targets that already have a global target ID, the newly added UAV node identifier and its local tracking identifier are appended to the corresponding global target ID field. Each confirmed association record in the multi-UAV target ID association table uses the global target ID as the main index and records the UAV node identifier and local tracking identifier corresponding to the global target ID. For candidate targets that fail the target consistency judgment, a pending confirmation identifier is assigned to them and stored in the pending confirmation target set. When subsequent image frames, new observation data from other UAVs, or changes in target status meet the reconfirmation conditions, the geographic prior search area (ROI) generation process in step 2 and the cross-view feature matching and target consistency judgment process in step 3 are retried to further confirm the pending confirmation targets.

9. A multi-view collaborative sensing method based on spatiotemporal prior guidance and communication adaptation according to claim 1, characterized in that, After establishing or updating the multi-UAV target ID association table in step 3, each observation UAV generates a lightweight identification summary based on its local target observation records and the multi-UAV target ID association table. Based on the event-triggered communication and adaptive transmission mechanism, and according to the real-time communication link status and target dynamic characteristics, the sending strategy of the identification summary is dynamically adjusted to achieve an adaptive balance between communication bandwidth resources and collaborative tracking reliability.

10. A multi-view collaborative sensing method based on spatiotemporal prior guidance and communication adaptation according to claim 9, characterized in that, In step 4, the collaborative processing node receives lightweight identification summaries sent by each UAV node according to the event-triggered communication and adaptive transmission mechanism, performs group management, spatiotemporal alignment, and consistency verification on the received multi-source observation data, and completes cross-view collaborative 3D positioning and status update.

Images