Unmanned aerial vehicle swarm cooperative imaging system and method based on image fusion

Abstract

The application discloses a UAV swarm cooperative imaging system and method based on image quality fusion, relates to the technical field, and comprises an image quality demand modeling module, a scheduling parameter management module, an intelligent mapping model training module, a real-time data acquisition module, an environment mutation detection module and a closed-loop optimization module; the image quality demand modeling module is used for constructing and updating a multi-dimensional image quality demand matrix in real time; the scheduling parameter management module is used for receiving adjusted scheduling parameters; the intelligent mapping model training module is used for establishing a dynamic mapping relationship between the image quality demand matrix and the scheduling parameter matrix; the real-time data acquisition module is used for collecting environment data, image quality data, swarm state data and UAV type information through an onboard sensor; the environment mutation detection module is used for detecting environment data mutation conditions and triggering a model emergency iteration mode; and the closed-loop optimization module is used for triggering corresponding mapping sub-model iteration optimization based on a fusion efficiency quantization result and outputting adjusted scheduling parameters.

Description

Technical Field

[0001] This invention relates to the field of drone swarm collaborative imaging technology, specifically to a drone swarm collaborative imaging system and method based on image quality fusion. Background Technology

[0002] Unmanned aerial vehicle (UAV) swarm collaborative imaging technology enables large-scale, high-efficiency imaging tasks through distributed multi-UAV cooperation, and is widely used in military reconnaissance, emergency rescue, and civilian surveying and mapping. However, existing technologies have significant flaws in the binding logic between swarm task scheduling and image quality fusion requirements: most adopt passive image quality adaptation driven by task priority, without considering the stitching compatibility of multi-UAV images during the scheduling process; or they use static collaborative scheduling with fixed parameters, which cannot be adjusted in real time according to scene changes, resulting in problems such as stitching gaps and uneven colors during fusion; and they only focus on binding a single image quality indicator with scheduling, lacking multi-dimensional collaborative optimization, making it difficult to meet the high-precision imaging requirements in complex scenarios.

[0003] In existing technologies, the linkage between scheduling and image quality is unidirectional; scheduling determines imaging parameters, and image quality feedback does not adjust scheduling accordingly, forming an open-loop process. Image quality metrics are fragmented, failing to achieve multi-dimensional collaborative optimization. Scene adaptation is rigid, unable to dynamically adapt to complex environmental changes, and lacks differentiated mapping rules designed for the sensor characteristics of heterogeneous beehives. Anti-interference capabilities are weak; facing sudden scenarios such as strong electromagnetic interference and extreme lighting, it is difficult to quickly adjust scheduling strategies, resulting in insufficient imaging effectiveness and low fusion efficiency. Furthermore, existing mapping models have low training data coverage and poor generalization ability, failing to provide effective comparison with traditional algorithms and lacking creative support. Therefore, there is an urgent need for an intelligent mapping scheme that can achieve dynamic adaptation of multi-dimensional image quality metrics and scheduling parameters, scene-specific adaptation, heterogeneous adaptation, and anti-interference adaptation, addressing the shortcomings of existing technologies. Summary of the Invention

[0004] The purpose of this invention is to provide a drone swarm collaborative imaging system and method based on image quality fusion, so as to solve the problems raised in the prior art.

[0005] To achieve the above objectives, the present invention provides the following technical solution: a drone swarm cooperative imaging method based on image quality fusion, the method comprising the following steps:

[0006] Step S100: Construct a multi-dimensional image quality requirement matrix and set the threshold range corresponding to each image quality indicator in the matrix. Analyze the priority weight of each image quality indicator based on the application scenario type of the drone swarm collaborative imaging event record.

[0007] Step S200: Construct a swarm scheduling parameter matrix. The scheduling parameters include flight altitude, shooting angle, UAV formation spacing, sensor exposure parameters, and task allocation weights. Considering the heterogeneous characteristics of fixed-wing and multi-rotor UAVs, preset the value range of each scheduling parameter and the optimal value for fixed-wing and multi-rotor UAV applications respectively.

[0008] Step S300: Train an intelligent mapping model based on the DQN reinforcement learning algorithm. The model includes a fixed wing model and a multi-rotor wing model. Establish a dynamic mapping relationship between the image quality requirement matrix and the scheduling parameter matrix.

[0009] Step S400: In swarm collaborative imaging, environmental data, image quality data and UAV type information are collected in real time, and the image quality requirement matrix is ​​dynamically updated; if the change in environmental data exceeds the preset threshold, the model emergency iteration mode is triggered; otherwise, real-time scheduling parameters are output through the mapping sub-model of the corresponding type of UAV.

[0010] Step S500: Verify the fusion effect in real time based on the fusion effect quantification formula. If the fusion effect does not meet the standard or any image quality index exceeds the threshold range, trigger the corresponding mapping sub-module to iterate and optimize again, and output the adjusted scheduling parameters.

[0011] Step S100 includes the following specific steps:

[0012] Image quality demand matrix refers to the matrix composed of the i-th type of image quality index Q i The matrix Q is formed, Q=[Q1,Q2,...,Q m The image quality requirement matrix includes at least resolution Q1, color consistency Q2, geometric accuracy Q3, signal-to-noise ratio Q4, and fusion overlap rate Q5; the corresponding threshold range is represented as [Q...]. i,min Q i,max ];

[0013] Color consistency is calculated by summing the brightness difference by the square of the product of the brightness weighting coefficient and the brightness correction coefficient, adding the square of the product of the chroma difference by the chroma weighting coefficient and the chroma correction coefficient, adding the square of the product of the hue difference by the hue weighting coefficient and the hue correction coefficient, and then adding the product of the rotation term and the product of the chroma difference by the chroma weighting coefficient and the chroma correction coefficient, the product of the hue difference by the hue weighting coefficient and the hue correction coefficient, and finally taking the square root of the sum of these four terms. The brightness difference, chroma difference, and hue difference are the differences between the fused image and the original image in the brightness, chroma, and hue dimensions, respectively. The brightness weighting coefficient, chroma weighting coefficient, and hue weighting coefficient are preset color calibration weighting parameters, and the brightness correction coefficient, chroma correction coefficient, and hue correction coefficient are preset color deviation correction parameters. The rotation term refers to the compensation coefficient for correcting non-linear deviations in the hue dimension.

[0014] Where the geometric accuracy Q3 = {1 / N*∑[(X k1 -X k2 ) 2 +(Y k1 -Y k2 ) 2 +(Z k1 -Z k2 ) 2 ]} 1 / 2 N represents the number of feature points, k∈[1,N], (X k1 ,Y k1 Z k1 (X) represents the measured coordinates of the k-th feature point. k2 ,Y k2 Z k2 () represents the true coordinates of the k-th feature point;

[0015] The fusion overlap rate Q5 is calculated as (S1 / S2)*100%, where S1 represents the area of ​​overlap between images acquired by two adjacent drones, and S2 represents the actual coverage area of ​​a single frame image from a single drone. For multi-drone collaborative scenarios, the average overlap rate of all adjacent drone images is taken as the final fusion overlap rate.

[0016] Step S100 also includes the following specific details:

[0017] Application scenarios include military reconnaissance, civilian surveying and mapping, and emergency rescue.

[0018] For military reconnaissance scenarios, the priority of various image quality indicators is as follows: signal-to-noise ratio > resolution > geometric accuracy > color consistency > fusion overlap rate;

[0019] For civilian surveying and mapping scenarios, the priority of various image quality indicators is as follows: geometric accuracy > resolution > color consistency > fusion overlap rate > signal-to-noise ratio;

[0020] In emergency rescue scenarios, the priority of each image quality indicator is as follows: resolution > signal-to-noise ratio > geometric accuracy > fusion overlap rate > color consistency.

[0021] Based on the priority ranking of image quality indicators corresponding to each type of application scenario, priority weights are assigned in sequence, and the sum of the priority weights of all image quality indicators in the same application scenario is 1.

[0022] Step S300 includes the following specific steps:

[0023] Step S310: Dynamic mapping relationship P=M(Q,Scene,UAV-Type), where Q represents the image quality requirement matrix, P represents the scheduling parameter matrix, and M is the intelligent mapping model; the reward function R of the mapping model satisfies:

[0024] R = ɑ(Scene)·∑[w i ·(Q i -Q i,min ) / (Q i,max -Q i,min )] - β(Scene)·∑[(P j -P j,opt(UVA-Type) ) / (P j,max -P j,min )];

[0025] Where ɑ(Scene) represents the weight of scene - based image quality gain, β(Scene) represents the weight of scene - based scheduling cost, and ɑ(Scene)+β(Scene)=1; wi represents the priority weight of the i - th type of image quality index; P j represents the i - th type of scheduling parameter, [P j,min -P j,max represents the value range corresponding to the j - th type of scheduling parameter; P j,opt(UVA-Type) represents the optimal value of the j - th type of scheduling parameter under the applications of fixed - wing and multi - rotor UAVs respectively;

[0026] Step S320: Obtain the scores F1 and F2 of the image quality compliance priority and resource consumption sensitivity under various application scenarios. Based on the score magnitudes, set several different weight combinations of ɑ(Scene) and β(Scene). When F1≥F2, set ɑ(Scene)≥β(Scene); when F1<F2, set ɑ(Scene)<β(Scene). Based on the UAV swarm collaborative imaging events under the same weight combination, calculate the corresponding image quality compliance rate U, U=(the number of compliant image quality indicators / the total number of image quality indicators)*100%, and the resource consumption ratio V, V=(the resource consumption of the current weight combination / the resource consumption of the reference weight combination)*100%. The reference weight combination refers to the weight combination when ɑ(Scene)=β(Scene);

[0027] Step S330: Based on the image quality compliance rates and resource consumption ratios output under different weight combinations for the same application scenario, screen out the weight combinations where the resource consumption ratio is less than or equal to the scenario threshold when the image quality compliance rate is greater than the compliance rate threshold as the application weights of the function model for the corresponding application scenario.

[0028] Step S400 includes the following content:

[0029] The situation where the mutation amplitude of environmental data exceeds the preset threshold means that the light intensity changes by greater than or equal to 500 lux within 1 s, the communication bandwidth drops by greater than or equal to 30% within 1 s, and the wind speed changes by greater than or equal to 5 m / s per unit time;

[0030] Emergency iteration mode refers to shortening the model iteration cycle to 20-50ms and prioritizing the top 3 image quality indicators with the highest priority weight.

[0031] Step S500 includes the following specific steps:

[0032] Step S510: The fusion effectiveness quantification formula includes the information entropy increment ΔH and the total mutual information MI. 总 Target recognition accuracy improvement rate r and fault redundancy efficiency d;

[0033] ΔH=H 融合后 -max(H 源1 ,...,H 源n );

[0034] MI 总 =∑MI(F,S k );

[0035] r={[A 融合后 -max(A 源1 ,...,A 源n )] / max(A 源1 ,...,A 源n )}*100;

[0036] d=(W 故障状态完成度 / W 无故障完成度 )*100;

[0037] Where H represents 融合后 The information entropy of the fused image, max(H) 源1 ,...,H 源n H represents the maximum value selected from the information entropy of all original images; 源1 ,...,H 源n The entropy represents the information of the original images collected by the 1st, ..., nth UAVs, where n represents the number of UAVs participating in the collaborative imaging.

[0038] MI(F,S k ) represents the mutual information between the fused image F and the original image of the k-th UAV; mutual information is used to measure the degree of information correlation between the two images, and the larger the value, the more effective information they share; ∑MI(F,S) k This indicates that all terms from k=1 to k=n are summed.

[0039] A 融合后 A represents the accuracy of the fused image in completing the target recognition task. 源1 ,...,A 源n This is represented as the target recognition accuracy of single-source images of the 1st, ..., nth UAVs;

[0040] W 故障状态完成度 This indicates the actual degree of completion of the fusion imaging task when some drones in the swarm are malfunctioning; W 无故障完成度 This indicates the degree of completion of the fusion imaging task when all drones in the swarm are working normally;

[0041] Step S520: Set the evaluation thresholds for the four types of indicators. Output "fusion effect meets the standard" when all results meet the evaluation thresholds; otherwise, output "fusion effect does not meet the standard".

[0042] The normal iteration cycle for closed-loop optimization is 50-200ms. During each iteration, the reward function weights α(Scene) and β(Scene) of the intelligent mapping model are adjusted by ±5%-±15%; under emergency iteration mode, the adjustment range is ±10%-±20%.

[0043] The UAV swarm collaborative imaging system based on image quality fusion includes an image quality requirement modeling module, a scheduling parameter management module, an intelligent mapping model training module, a real-time data acquisition module, an environmental change detection module, and a closed-loop optimization module.

[0044] The image quality requirement modeling module is used to build and update a multi-dimensional image quality requirement matrix in real time, set scene-based image quality priority weights, and verify whether the image quality indicators meet the threshold range.

[0045] The scheduling parameter management module is used to store the scheduling parameter matrix of fixed-wing / multi-rotor UAVs and receive the adjusted scheduling parameters;

[0046] The intelligent mapping model training module is used to train a mapping model containing sub-models based on the DQN reinforcement learning algorithm, and to establish a dynamic mapping relationship between the image quality requirement matrix and the scheduling parameter matrix.

[0047] The real-time data acquisition module is used to collect environmental data, image quality data, swarm status data, and drone type information through airborne sensors;

[0048] The environmental mutation detection module is used to detect sudden changes in environmental data and trigger the model's emergency iteration mode.

[0049] The closed-loop optimization module is used to trigger iterative optimization of the corresponding mapping sub-model based on the quantification results of the fusion effect, and output the adjusted scheduling parameters.

[0050] The intelligent mapping model training module also includes a model storage unit, which stores pre-trained mapping models for different application scenarios. These models can be directly called when switching scenarios, with a switching response time of ≤100ms.

[0051] Compared with the prior art, the beneficial effects of the present invention are:

[0052] 1. This invention constructs a multi-dimensional image quality requirement matrix with scene-based priority weights, and combines it with a scene-based reward function to achieve accurate adaptation of image quality-scheduling parameters under different scenes, thus solving the problem of rigid scene adaptation in existing technologies; it optimizes the heterogeneous swarm differential mapping rules so that the scheduling strategies of fixed-wing and multi-rotor UAVs are adapted to their sensor characteristics respectively, thereby improving the collaborative imaging effect of heterogeneous swarms.

[0053] 2. This invention adds an environmental mutation detection and emergency iteration mode, which greatly improves the anti-interference ability in complex environments and ensures that core image quality indicators do not meet the standards in sudden scenarios; the training data coverage is wide and the model has strong generalization ability;

[0054] 3. This invention forms a closed loop of "image quality feedback-scheduling optimization", which can adapt to environmental changes, swarm status and scene requirements in real time, improve the imaging effectiveness in complex scenes, and has strong versatility. It can be widely used in military reconnaissance, emergency rescue, civilian surveying and mapping and other fields. Attached Figure Description

[0055] Figure 1 This is a schematic diagram of the structure of the UAV swarm collaborative imaging method based on image quality fusion according to the present invention. Detailed Implementation

[0056] Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0057] Example: Figure 1 As shown, this invention provides a drone swarm cooperative imaging method based on image quality fusion, the method comprising the following steps:

[0058] Step S100: Construct a multi-dimensional image quality requirement matrix and set the threshold range corresponding to each image quality indicator in the matrix. Analyze the priority weight of each image quality indicator based on the application scenario type of the drone swarm collaborative imaging event record.

[0059] Step S200: Construct a swarm scheduling parameter matrix. The scheduling parameters include flight altitude, shooting angle, UAV formation spacing, sensor exposure parameters, and task allocation weights. Considering the heterogeneous characteristics of fixed-wing and multi-rotor UAVs, preset the value range of each scheduling parameter and the optimal value for fixed-wing and multi-rotor UAV applications respectively.

[0060] Step S300: Train an intelligent mapping model based on the DQN reinforcement learning algorithm. The model includes a fixed wing model and a multi-rotor wing model. Establish a dynamic mapping relationship between the image quality requirement matrix and the scheduling parameter matrix.

[0061] Step S400: In swarm collaborative imaging, environmental data, image quality data and UAV type information are collected in real time, and the image quality requirement matrix is ​​dynamically updated; if the change in environmental data exceeds the preset threshold, the model emergency iteration mode is triggered; otherwise, real-time scheduling parameters are output through the mapping sub-model of the corresponding type of UAV.

[0062] Step S500: Verify the fusion effect in real time based on the fusion effect quantification formula. If the fusion effect does not meet the standard or any image quality index exceeds the threshold range, trigger the corresponding mapping sub-module to iterate and optimize again, and output the adjusted scheduling parameters.

[0063] Step S100 includes the following specific steps:

[0064] Image quality demand matrix refers to the matrix composed of the i-th type of image quality index Q i The matrix Q is formed, Q=[Q1,Q2,...,Q m The image quality requirement matrix includes at least resolution Q1, color consistency Q2, geometric accuracy Q3, signal-to-noise ratio Q4, and fusion overlap rate Q5; the corresponding threshold range is represented as [Q...]. i,min Q i,max ];

[0065] Color consistency is calculated by summing the brightness difference by the square of the product of the brightness weighting coefficient and the brightness correction coefficient, adding the square of the product of the chroma difference by the chroma weighting coefficient and the chroma correction coefficient, adding the square of the product of the hue difference by the hue weighting coefficient and the hue correction coefficient, and then adding the product of the rotation term and the product of the chroma difference by the chroma weighting coefficient and the chroma correction coefficient, the product of the hue difference by the hue weighting coefficient and the hue correction coefficient, and finally taking the square root of the sum of these four terms. The brightness difference, chroma difference, and hue difference are the differences between the fused image and the original image in the brightness, chroma, and hue dimensions, respectively. The brightness weighting coefficient, chroma weighting coefficient, and hue weighting coefficient are preset color calibration weighting parameters, and the brightness correction coefficient, chroma correction coefficient, and hue correction coefficient are preset color deviation correction parameters. The rotation term refers to the compensation coefficient for correcting non-linear deviations in the hue dimension.

[0066] Where the geometric accuracy Q3 = {1 / N*∑[(X k1 -X k2 ) 2 +(Y k1 -Y k2 ) 2 +(Z k1 -Z k2 ) 2 ]} 1 / 2 N represents the number of feature points, k∈[1,N], (X k1 ,Y k1 Z k1(X) represents the measured coordinates of the k-th feature point. k2 ,Y k2 Z k2 () represents the true coordinates of the k-th feature point;

[0067] The fusion overlap rate Q5 is calculated as (S1 / S2)*100%, where S1 represents the area of ​​overlap between images acquired by two adjacent drones, and S2 represents the actual coverage area of ​​a single frame image from a single drone. For multi-drone collaborative scenarios, the average overlap rate of all adjacent drone images is taken as the final fusion overlap rate.

[0068] Q1 = [(flight altitude * sensor pixel size) / focal length] * 100;

[0069] Q4 = 10log 10 (S 信号功率 / N 噪声功率 ).

[0070] Step S100 also includes the following specific details:

[0071] Application scenarios include military reconnaissance, civilian surveying and mapping, and emergency rescue.

[0072] For military reconnaissance scenarios, the priority of various image quality indicators is as follows: signal-to-noise ratio > resolution > geometric accuracy > color consistency > fusion overlap rate;

[0073] For civilian surveying and mapping scenarios, the priority of various image quality indicators is as follows: geometric accuracy > resolution > color consistency > fusion overlap rate > signal-to-noise ratio;

[0074] In emergency rescue scenarios, the priority of each image quality indicator is as follows: resolution > signal-to-noise ratio > geometric accuracy > fusion overlap rate > color consistency.

[0075] Based on the priority ranking of image quality indicators corresponding to each type of application scenario, priority weights are assigned in sequence, and the sum of the priority weights of all image quality indicators in the same application scenario is 1.

[0076] Step S300 includes the following specific steps:

[0077] Step S310: Dynamic mapping relationship P=M(Q,Scene,UAV-Type), where Q represents the image quality requirement matrix, P represents the scheduling parameter matrix, and M is the intelligent mapping model; the reward function R of the mapping model satisfies:

[0078] R=ɑ(Scene)·∑[w i ·(Q i -Q i,min ) / (Q i,max -Q i,min) - β(Scene)·∑[(P j -P j,opt(UVA-Type) ) / (P j,max -P j,min )];

[0079] Where α(Scene) represents the weight of scene - specific image quality gain, β(Scene) represents the weight of scene - specific scheduling cost, and α(Scene)+β(Scene)=1; wi represents the priority weight of the i - th type of image quality index; P j represents the i - th type of scheduling parameter, [P j,min -P j,max represents the value range corresponding to the j - th type of scheduling parameter; P j,opt(UVA-Type) represents the optimal value of the j - th type of scheduling parameter under the applications of fixed - wing and multi - rotor UAVs respectively;

[0080] Step S320: Obtain the scores F1 and F2 of the image quality compliance priority and resource consumption sensitivity under various application scenarios. Based on the magnitudes of the scores, set several different weight combinations of α(Scene) and β(Scene). When F1≥F2, set α(Scene)≥β(Scene); when F1<F2, set α(Scene)<β(Scene). Based on the UAV swarm collaborative imaging events under the same weight combination, calculate the corresponding image quality compliance rate U, U = (the number of compliant image quality indices / the total number of image quality indices) * 100%, and the resource consumption ratio V, V=(the resource consumption of the current weight combination / the resource consumption of the reference weight combination) * 100%. The reference weight combination refers to the weight combination when α(Scene)=β(Scene);

[0081] Step S330: Based on the image quality compliance rates and resource consumption ratios output under different weight combinations for the same application scenario, screen out the weight combinations whose resource consumption ratios are less than or equal to the scenario threshold when the image quality compliance rates are greater than the compliance rate threshold as the application weights of the function model for the corresponding application scenario.

[0082] As shown in the embodiment:

[0083] Select 4 dimensions strongly related to the core tasks of the scene and set the dimension weights (ensure that the sum of the weights is 1)

[0084] Relevance of the core task image quality index: The weight accounts for 0.4. If the index compliance directly determines the success or failure of the task (such as the geometric accuracy of mapping), it is 10 points; if there is no direct impact, it is 1 point;

[0085] Compliance with industry standards: The weight accounts for 0.3. If the index needs to meet the mandatory requirements of national standards / military standards, it is 10 points; if there are no mandatory requirements, it is 1 point;

[0086] Image quality anomaly tolerance: weighted at 0.2, a score of 10 is given if the indicator is slightly below standard and causes the task to fail; a score of 1 is given if the tolerance is high.

[0087] Customer needs priority: weighted by 0.1, 10 points for customers who explicitly require priority; 1 point for customers who do not explicitly require priority.

[0088] Taking a civilian surveying scenario as an example: F1 = 10*0.4 + 10*0.3 + 9*0.2 + 8*0.1 = 9.6;

[0089] Similarly, F2 = 2.4, indicating that F1 > F2. Therefore, the weight combinations can be set as follows:

[0090] α(Scene) = 0.6, β(Scene) = 0.4; correspondingly, U = 92%, V = 105%.

[0091] α(Scene) = 0.7, β(Scene) = 0.3; correspondingly, U = 96%, V = 110%.

[0092] α(Scene) = 0.8, β(Scene) = 0.2; correspondingly, U = 100%, V = 132%;

[0093] α(Scene) = 0.9, β(Scene) = 0.1; corresponding to U = 100%, V = 180%;

[0094] The compliance rate threshold is 95%, and the scene threshold is 150%. There are two scenes that meet the requirements. Since the scene is a civilian surveying and mapping scene, the image accuracy requirements are higher. Therefore, it is necessary to ensure a higher image quality compliance rate. So, we finally chose α(Scene)=0.8 and β(Scene)=0.2 as the application weights of the function model.

[0095] Step S400 includes the following:

[0096] An environmental data change exceeding the preset threshold means that the light intensity changes by more than or equal to 500 lux within 1 second, the communication bandwidth decreases by more than or equal to 30% within 1 second, and the wind speed changes by more than or equal to 5 m / s per unit time.

[0097] Emergency iteration mode refers to shortening the model iteration cycle to 20-50ms and prioritizing the top 3 image quality indicators with the highest priority weight.

[0098] Step S500 includes the following specific steps:

[0099] Step S510: The fusion effectiveness quantification formula includes the information entropy increment ΔH and the total mutual information MI. 总 Target recognition accuracy improvement rate r and fault redundancy efficiency d;

[0100] ΔH=H 融合后 -max(H 源1 ,...,H 源n );

[0101] MI 总 =∑MI(F,S k );

[0102] r={[A 融合后 -max(A 源1 ,...,A 源n )] / max(A 源1 ,...,A 源n )}*100;

[0103] d=(W 故障状态完成度 / W 无故障完成度 )*100;

[0104] Where H represents 融合后 The information entropy of the fused image, max(H) 源1 ,...,H 源n H represents the maximum value selected from the information entropy of all original images; 源1 ,...,H 源n The entropy represents the information of the original images collected by the 1st, ..., nth UAVs, where n represents the number of UAVs participating in the collaborative imaging.

[0105] MI(F,S k ) represents the mutual information between the fused image F and the original image of the k-th UAV; mutual information is used to measure the degree of information correlation between the two images, and the larger the value, the more effective information they share; ∑MI(F,S) k This indicates that all terms from k=1 to k=n are summed.

[0106] A 融合后 A represents the accuracy of the fused image in completing the target recognition task. 源1 ,...,A 源n This is represented as the target recognition accuracy of single-source images of the 1st, ..., nth UAVs;

[0107] W 故障状态完成度 This indicates the actual degree of completion of the fusion imaging task when some drones in the swarm are malfunctioning; W 无故障完成度 This indicates the degree of completion of the fusion imaging task when all drones in the swarm are working normally;

[0108] Step S520: Set the evaluation thresholds for the four types of indicators. Output "fusion effect meets standard" when all results meet the evaluation thresholds; otherwise, output "fusion effect does not meet standard". Generally, the thresholds are set as follows: ΔH ≥ 0.5, MI...总 ≥1.0, r≥10%, d≥90%.

[0109] As shown in the example:

[0110] If the verification fails, the corresponding mapping sub-model is triggered to iterate and optimize again. The specific iteration process is as follows:

[0111] Fault location and parameter correlation: First, analyze the source of the non-compliant indicators to determine whether the fusion efficiency quantitative indicators are not up to standard or the image quality indicators exceed the threshold. Then, through the preset "indicator-scheduling parameter correlation table", locate the scheduling parameters that are strongly correlated with the non-compliant indicators, such as the sensor exposure parameters corresponding to the non-compliant signal-to-noise ratio and the drone formation spacing corresponding to the non-compliant fusion overlap rate.

[0112] Iterative initialization: Call the pre-trained sub-model corresponding to the current scene and drone type, take the real-time values ​​of the unmet indicators, the current scheduling parameters, and environmental data as the initial input, and record the values ​​of the current reward function weights α and β at the same time;

[0113] Model Iterative Calculation: Based on the DQN reinforcement learning iterative mechanism, with the goal of "bringing the non-compliant indicators back to the qualified range and keeping the compliant indicators stable", the values ​​of the associated scheduling parameters are adjusted, and the adjustment step size is determined according to the iteration mode: the step size is 5%-15% in the normal mode and 10%-20% in the emergency iteration mode, and the α and β weights of the reward function are updated simultaneously.

[0114] Intermediate result verification: After each parameter adjustment, recalculate the fusion efficiency quantitative index and image quality index to determine whether the target conditions are met; if not, repeat step 3 to continue iterating until the preset iteration limit is reached or the index is met.

[0115] Optimal parameter output: When the indicator meets the target or the maximum number of iterations is reached, the iteration stops, and the set of parameters with the largest reward function value during the iteration is selected as the adjusted scheduling parameter Pa and sent to the corresponding UAV for execution. In the normal mode, the iteration cycle is 50-200ms and the maximum number of iterations is 10. In the emergency iteration mode, the iteration cycle is 20-50ms and the maximum number of iterations is 5, to ensure rapid response to scene changes. For example, if the fusion overlap rate is <40%, the associated parameter is first located as the formation spacing P3. In the normal mode, P3 is reduced by 10% step size. Q5 is calculated once for each adjustment until Q5 ≥ 40% and the final P3 is output. If the signal-to-noise ratio is <30dB, the exposure time is extended by 15% step size in the emergency mode to quickly improve the signal-to-noise ratio to ≥30dB.

[0116] The normal iteration cycle for closed-loop optimization is 50-200ms. During each iteration, the reward function weights α(Scene) and β(Scene) of the intelligent mapping model are adjusted by ±5%-±15%; under emergency iteration mode, the adjustment range is ±10%-±20%.

[0117] The UAV swarm collaborative imaging system based on image quality fusion includes an image quality requirement modeling module, a scheduling parameter management module, an intelligent mapping model training module, a real-time data acquisition module, an environmental change detection module, and a closed-loop optimization module.

[0118] The image quality requirement modeling module is used to build and update a multi-dimensional image quality requirement matrix in real time, set scene-based image quality priority weights, and verify whether the image quality indicators meet the threshold range.

[0119] The scheduling parameter management module is used to store the scheduling parameter matrix of fixed-wing / multi-rotor UAVs and receive the adjusted scheduling parameters;

[0120] The intelligent mapping model training module is used to train a mapping model containing sub-models based on the DQN reinforcement learning algorithm, and to establish a dynamic mapping relationship between the image quality requirement matrix and the scheduling parameter matrix.

[0121] The real-time data acquisition module is used to collect environmental data, image quality data, swarm status data, and drone type information through airborne sensors;

[0122] The environmental mutation detection module is used to detect sudden changes in environmental data and trigger the model's emergency iteration mode.

[0123] The closed-loop optimization module is used to trigger iterative optimization of the corresponding mapping sub-model based on the quantification results of the fusion effect, and output the adjusted scheduling parameters.

[0124] The intelligent mapping model training module also includes a model storage unit, which stores pre-trained mapping models for different application scenarios. These models can be directly called when switching scenarios, with a switching response time of ≤100ms.

[0125] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A drone swarm cooperative imaging method based on image quality fusion, characterized in that: The method includes the following steps: Step S100: Construct a multi-dimensional image quality requirement matrix and set the threshold range corresponding to each image quality indicator in the matrix. Analyze the priority weight of each image quality indicator based on the application scenario type of the drone swarm collaborative imaging event record. Step S200: Construct a swarm scheduling parameter matrix, wherein the scheduling parameters include flight altitude, shooting angle, UAV formation spacing, sensor exposure parameters and task allocation weight; and, considering the heterogeneous characteristics of fixed-wing and multi-rotor UAVs, preset the value range of each scheduling parameter and the optimal value for fixed-wing and multi-rotor UAV applications respectively. Step S300: Train an intelligent mapping model based on the DQN reinforcement learning algorithm. The model includes a fixed wing model and a multi-rotor wing model. Establish a dynamic mapping relationship between the image quality requirement matrix and the scheduling parameter matrix. Step S400: In swarm collaborative imaging, environmental data, image quality data and UAV type information are collected in real time, and the image quality requirement matrix is ​​dynamically updated; if the change in environmental data exceeds the preset threshold, the model emergency iteration mode is triggered; otherwise, real-time scheduling parameters are output through the mapping sub-model of the corresponding type of UAV. Step S500: Verify the fusion effect in real time based on the fusion effect quantification formula. If the fusion effect does not meet the standard or any image quality index exceeds the threshold range, trigger the corresponding mapping sub-module to iterate and optimize again, and output the adjusted scheduling parameters.

2. The UAV swarm cooperative imaging method based on image quality fusion according to claim 1, characterized in that: Step S100 includes the following specific steps: The image quality requirement matrix refers to the matrix composed of the i-th type of image quality index Q. i The matrix Q is formed, Q=[Q1,Q2,...,Q m The image quality requirement matrix includes at least resolution Q1, color consistency Q2, geometric accuracy Q3, signal-to-noise ratio Q4, and fusion overlap rate Q5; the corresponding threshold range is represented as [Q...]. i,min Q i,max ]; The color consistency is calculated by adding the square of the product of the brightness difference and the brightness correction coefficient, the square of the product of the chroma difference and the chroma correction coefficient, the square of the product of the hue difference and the hue correction coefficient, the square of the product of the hue difference and the hue correction coefficient, the product of the rotation term and the product of the chroma difference and the hue difference, and finally taking the square root of the sum of the above four terms. The brightness difference, chroma difference, and hue difference are the differences between the fused image and the original image in the brightness, chroma, and hue dimensions, respectively. The brightness weight coefficient, chroma weight coefficient, and hue weight coefficient are preset color calibration weight parameters, and the brightness correction coefficient, chroma correction coefficient, and hue correction coefficient are preset color deviation correction parameters. The rotation term refers to the compensation coefficient for correcting non-linear deviations in the hue dimension. Where the geometric accuracy Q3 = {1 / N*∑[(X k1 -X k2 ) 2 +(Y k1 -Y k2 ) 2 +(Z k1 -Z k2 ) 2 ]} 1 / 2 N represents the number of feature points, k∈[1,N], (X k1 ,Y k1 Z k1 (X) represents the measured coordinates of the k-th feature point. k2 ,Y k2 Z k2 () represents the true coordinates of the k-th feature point; The fusion overlap rate Q5 is calculated as (S1 / S2)*100%, where S1 represents the area of ​​overlap between images acquired by two adjacent drones, and S2 represents the actual coverage area of ​​a single frame image from a single drone. For multi-drone collaborative scenarios, the average overlap rate of all adjacent drone images is taken as the final fusion overlap rate.

3. The UAV swarm cooperative imaging method based on image quality fusion according to claim 1, characterized in that: Step S100 also includes the following specific contents: The application scenarios include military reconnaissance scenarios, civilian surveying and mapping scenarios, and emergency rescue scenarios; The priority of image quality indicators for the military reconnaissance scenario is as follows: signal-to-noise ratio > resolution > geometric accuracy > color consistency > fusion overlap rate. For the civilian mapping scenario, the priority of each image quality index is geometric accuracy > resolution > color consistency > fusion overlap rate > signal-to-noise ratio; For the emergency rescue scenario, the priority of each image quality index is resolution > signal-to-noise ratio > geometric accuracy > fusion overlap rate > color consistency; Based on the priority ranking of the image quality indexes corresponding to various types of application scenarios, priority weights are allocated in order, and the sum of the priority weights of all image quality indexes under the same application scenario is 1.

4. The UAV swarm cooperative imaging method based on image quality fusion according to claim 1, characterized in that: The step S300 includes the following specific steps: Step S310: The dynamic mapping relationship P = M(Q, Scene, UAV-Type), where Q represents the image quality requirement matrix, P represents the scheduling parameter matrix, and M is the intelligent mapping model; the reward function R of the mapping model satisfies: R=ɑ(Scene)·∑[w i ·(Q i -Q i,min ) / (Q i,max -Q i,min )]-β(Scene)·∑[(P j -P j,opt(UVA-Type) ) / (P j,max -P j,min )]; Where α(Scene) represents the scene-based image quality gain weight, β(Scene) represents the scene-based scheduling cost weight, and α(Scene) + β(Scene) = 1; wi represents the priority weight of the i-th type of image quality metric; P j Represents the i-th type of scheduling parameter, [P j,min -P j,max ] represents the range of values ​​corresponding to the j-th type of scheduling parameter; P j,opt(UVA-Type) This represents the optimal value of the j-th type of scheduling parameter in both fixed-wing and multi-rotor UAV applications; Step S320: Obtain the scores F1 and F2 of the image quality compliance priority and resource consumption sensitivity under various application scenarios. Based on the score magnitudes, set several groups of different weight combinations of ɑ(Scene) and β(Scene). When F1 ≥ F2, set ɑ(Scene) ≥ β(Scene); when F1 < F2, set ɑ(Scene) < β(Scene); Based on the UAV swarm collaborative imaging events under the same weight combination, calculate the corresponding image quality compliance rate U, U = (the number of compliant image quality indexes / the total number of image quality indexes) * 100%, and the resource consumption ratio V, V = (the resource consumption of the current weight combination / the resource consumption of the reference weight combination) * 100%; The reference weight combination refers to the weight combination corresponding to ɑ(Scene) = β(Scene); Step S330: Based on the image quality compliance rate and resource consumption ratio output under different weight combinations in the same application scenario, select the weight combination with a resource consumption ratio less than or equal to the scenario threshold when the image quality compliance rate is greater than the compliance rate threshold as the application weight of the function model under the corresponding application scenario.

5. The UAV swarm cooperative imaging method based on image quality fusion according to claim 1, characterized in that: The step S400 includes the following content: The situation where the mutation amplitude of the environmental data exceeds the preset threshold means that the light intensity changes by greater than or equal to 500 lux within 1 s, the communication bandwidth drops by greater than or equal to 30% within 1 s, and the wind speed changes by greater than or equal to 5 m / s per unit time; The emergency iteration mode means that the model iteration period is shortened to 20 - 50 ms, and the top 3 image quality indexes with priority weights are preferentially guaranteed.

6. The UAV swarm cooperative imaging method based on image quality fusion according to claim 1, characterized in that: The step S500 includes the following specific steps: Step S510: The fusion efficiency quantification formula includes the information entropy increment ΔH and the total mutual information MI. 总 Target recognition accuracy improvement rate r and fault redundancy efficiency d; ΔH=H 融合后 -max(H 源1 ,...,H 源n ); MI 总 =∑MI(F,S k ); r={[A 融合后 -max(A 源1 ,...,A 源n )] / max(A 源1 ,...,A 源n )}*100; d=(W 故障状态完成度 / W 无故障完成度 )*100; Where H represents 融合后 The information entropy of the fused image, max(H) 源1 ,...,H 源n H represents the maximum value selected from the information entropy of all original images; 源1 ,...,H 源n The entropy represents the information of the original images collected by the 1st, ..., nth UAVs, where n represents the number of UAVs participating in the collaborative imaging. MI(F,S k ) represents the mutual information between the fused image F and the original image of the k-th UAV; ∑MI(F,S) k This indicates that all terms from k=1 to k=n are summed. A 融合后 A represents the accuracy of the fused image in completing the target recognition task. 源1 ,...,A 源n This is represented as the target recognition accuracy of single-source images of the 1st, ..., nth UAVs; W 故障状态完成度 This indicates the actual degree of completion of the fusion imaging task when some drones in the swarm are malfunctioning. W 无故障完成度 This indicates the degree of completion of the fusion imaging task when all drones in the swarm are working normally; Step S520: Set the evaluation thresholds corresponding to the four types of indexes. When all results meet the evaluation thresholds, output that the fusion effect is compliant; otherwise, output that the fusion effect is non-compliant; The normal iteration period of the closed-loop optimization is 50 - 200 ms. Each time of iteration, adjust the reward function weights ɑ(Scene) and β(Scene) of the intelligent mapping model, and the adjustment range is ±5% - ±15%; the adjustment range in the emergency iteration mode is ±10% - ±20%.

7. A drone swarm cooperative imaging system based on image quality fusion, using the drone swarm cooperative imaging method based on image quality fusion as described in any one of claims 1-6, characterized in that: The system includes an image quality requirement modeling module, a scheduling parameter management module, an intelligent mapping model training module, a real-time data acquisition module, an environmental mutation detection module, and a closed-loop optimization module; The image quality requirement modeling module is used to construct and update a multi-dimensional image quality requirement matrix in real time, set scene-based image quality priority weights, and verify whether the image quality indicators meet the threshold range. The scheduling parameter management module is used to store the scheduling parameter matrix of fixed-wing / multi-rotor UAVs and receive the adjusted scheduling parameters. The intelligent mapping model training module is used to train a mapping model containing sub-models based on the DQN reinforcement learning algorithm, and to establish a dynamic mapping relationship between the image quality requirement matrix and the scheduling parameter matrix. The real-time data acquisition module is used to collect environmental data, image quality data, swarm status data and drone type information through airborne sensors; The environmental mutation detection module is used to detect environmental data mutations and trigger the model's emergency iteration mode. The closed-loop optimization module is used to trigger iterative optimization of the corresponding mapping sub-model based on the fusion efficiency quantification results, and output the adjusted scheduling parameters.

8. The UAV swarm cooperative imaging system based on image quality fusion according to claim 7, characterized in that: The intelligent mapping model training module also includes a model storage unit for storing pre-trained mapping models for different application scenarios. These models can be directly called when switching scenarios, with a switching response time of ≤100ms.

Images