An unmanned aerial vehicle domain adaptive detection method of image pixel-level hybrid enhancement

Abstract

The application discloses an unmanned aerial vehicle domain adaptive detection method based on image pixel-level hybrid enhancement, and belongs to the field of infrared target detection applications. The application faces infrared unmanned aerial vehicle data applications, collects infrared unmanned aerial vehicle images by using an infrared camera, and realizes domain adaptive target detection of the infrared unmanned aerial vehicle images through steps such as data preparation, pseudo-label generation, region division and confidence calculation, hybrid image generation based on the confidence, bidirectional dynamic region pseudo-label filtering, hybrid label generation and loss calculation and model training optimization. The application can be used for infrared unmanned aerial vehicle target detection tasks in scenes such as night monitoring, disaster rescue and military reconnaissance, can effectively cope with the performance decline problem in infrared image cross-domain applications (such as different devices and environmental conditions), and improves the accuracy and robustness of target detection.

Description

Technical Field

[0001] This invention belongs to the field of unmanned aerial vehicle (UAV) applications, and more specifically, relates to an infrared image UAV domain adaptive detection method based on pixel-level hybrid enhancement of region pseudo-label confidence images. Background Technology

[0002] With the development of UAV technology, infrared imaging technology has been widely used in UAV target detection tasks, especially in scenarios such as night surveillance, disaster relief, and military reconnaissance, where it has unique advantages. Infrared images can achieve target identification under low light or complex weather conditions by capturing the thermal radiation information of targets. However, acquiring infrared UAV images is relatively difficult, and the lack of samples can seriously affect the detection performance of deep learning-based infrared target detection models; if models trained from visible light images are directly applied to infrared images, the performance of the detection models is often not high due to differences in domain distribution.

[0003] To address the challenges posed by domain distribution differences, domain adaptation techniques have become a research hotspot in recent years. Domain-adaptive object detection methods aim to adapt models trained in the source domain (a domain with abundant labeled data) to the target domain (a domain with scarce or no labeled data) through transfer learning or feature alignment. Currently, common domain adaptation methods include adversarial training, Mean Teacher-based methods, and pseudo-label self-training methods. For example, adversarial training aligns the feature distributions of the source and target domains through a generator and discriminator; pseudo-label self-training uses the source domain model to generate initial labels for the target domain data, followed by iterative optimization. However, these methods still face several problems in infrared UAV scenarios: first, feature alignment based on adversarial training struggles to capture regional-level thermal radiation semantic information, affecting detection accuracy; second, the small target scale and indistinct thermal features in infrared images lead to low-quality pseudo-labels; and third, existing methods struggle to meet the practical needs of UAV platforms in terms of computational complexity and real-time performance.

[0004] To address the aforementioned issues, some studies have attempted to introduce data augmentation or confidence screening strategies, such as enhancing data diversity through image region-level fusion methods. However, these methods are mostly designed for visible light images and do not fully consider the thermal imaging characteristics of infrared images and the dynamic nature of UAV scenarios. Therefore, how to combine region information and pixel-level features in infrared UAV target detection to improve pseudo-label quality and optimize detection performance remains a pressing technical challenge. Summary of the Invention

[0005] To address the shortcomings of existing technologies in adaptive detection of UAVs in infrared image domains, this invention proposes a UAV domain-adaptive detection method based on pixel-level fusion enhancement of region pseudo-label confidence images. This method aims to solve the performance degradation problem caused by domain distribution differences in UAV detection tasks using infrared images. Through a region pixel-level fusion strategy and a bidirectional dynamic region pseudo-label filtering mechanism, it achieves effective knowledge transfer from the source domain (visible light image) to the target domain (infrared image), improving the model's detection accuracy and robustness in infrared UAV images.

[0006] To achieve the above objectives, the present invention provides a UAV domain adaptation detection method with image pixel-level hybrid enhancement, comprising the following steps:

[0007] S1. Read Data: Read the visible light image x S and its tag y S (Source domain data), read infrared image x T (Target domain data);

[0008] S2. Pseudo-tag generation: The infrared image x... T The data is input into the object detector F (obtained in the first iteration using source domain data) to generate a set of pseudo-labels for the target domain. Where N T The number of targets detected. The pseudo-label for the i-th target (including bounding box coordinates and confidence score);

[0009] S3. Region segmentation and confidence calculation: The visible light image x... S and infrared image x T Divide into four equal regions, denoted as and According to pseudo-labels Within the region where the center point is located, each pseudo-label is assigned to the corresponding region, and the average pseudo-label confidence score for each region is calculated, denoted as . This represents the average pseudo-label confidence level of the j-th region;

[0010] S4. Two-way dynamic region pseudo-label filtering: A two-way dynamic region pseudo-label filtering mechanism is adopted for infrared image x T generated pseudo tags Perform filtering, and denote the set of pseudo-labels after filtering as follows:

[0011] S5. Confidence-based hybrid image generation: For infrared image x T and visible light image x S Perform region confidence-weighted pixel-by-pixel fusion to generate a hybrid image x M ;

[0012] S6. Hybrid Label Generation: Generate hybrid labels from the source domain image y S And target domain pseudo-labels filtered by bidirectional dynamic region pseudo-labels Merge the images to generate hybrid image pseudo-labels.

[0013] S7. Model Training and Optimization: Utilizing the source domain image x S and the generated mixed image x M The target detector F is trained by domain adaptation to obtain the updated target detector F. The updated target detector F is then used to replace the target detector F used in the previous round. Steps S2 to S7 are repeated. Through multiple rounds of iteration, the final target detector F is obtained.

[0014] Furthermore, in the bidirectional dynamic region pseudo-label filtering step, this invention designs a bidirectional dynamic region pseudo-label filtering mechanism. By combining global filtering from loose to strict and bridging filtering from strict to loose, more reliable pseudo-labels can be obtained, thereby improving the effectiveness of domain adaptive learning.

[0015] Furthermore, in the global filtering process, which ranges from loose to strict, a strategy for dynamically adjusting the false label confidence level is adopted, using a pre-set fixed threshold C. th The process involves filtering out pseudo-labels with a confidence score greater than a threshold, achieving a shift from lenient to strict pseudo-label filtering. Specifically, in the early stages of training, a relatively lenient pseudo-label confidence scoring strategy is adopted, allowing more pseudo-labels to participate in training. As the number of training epochs increases, the strictness of the confidence score is gradually increased to filter out unreliable pseudo-labels. The pseudo-label confidence score C(t) is obtained using the following formula:

[0016]

[0017] δ(t) is a factor that dynamically adjusts with each iteration t, and its value ranges from (0,1); M is the number of samples used in each iteration; α is a constant (e.g., a value of 5); C det (t) and C bbx (t) represents the category confidence and location confidence of the pseudo-label in each iteration, which are obtained by the object detector F.

[0018] Furthermore, in the bridging filtering process, which proceeds from strict to lenient, an adaptive dynamic threshold adjustment strategy is adopted. The average confidence level of the highest confidence region is used as a benchmark, and the threshold C is dynamically adjusted accordingly. p (t), which implements the introduction of pseudo-labels from non-highest confidence regions from strict to lenient, and the dynamic threshold is calculated as follows:

[0019]

[0020] Where Cb and C e Let be the initial and final thresholds (e.g., 1 and 0.9 respectively), and n(t) and N be the current training iteration round and the total training iteration rounds respectively. is the average confidence level of the pseudo-labels in the t-th round of the region with the highest confidence level.

[0021] Furthermore, in the confidence-based hybrid image generation step, a pixel-level image fusion method with region confidence weighting is designed. First, the average pseudo-label confidence of each region is used as the basis for the fusion. The four regions are sorted by size, and based on their ranking from highest to lowest, they are corresponding to four confidence levels: high, medium, medium, and low. Then, for each of the three confidence levels (high, medium, and low), a Beta(α,β) function with three different parameters is used to randomly generate a weight ratio λ for each region during the mixed image generation. i For high-confidence regions, a Beta function with a value distribution biased towards 1 is used to make the visible image x... S The values ​​of the visible light image x are given a higher weight; for low-confidence regions, a Beta function with a value distribution biased towards 0 is used, making the visible light image x... S The weighting ratio is too small; for the medium confidence region, a Beta function with a value distribution biased towards 0.5 is used to make the visible light image x S The infrared image has the same weight ratio as the infrared image. After obtaining the weights, the four regions are blended according to the following formula to obtain a blended image of the four regions. The final blended image x can then be obtained through merging. M .

[0022]

[0023] Here, ⊙ represents pixel-level blending operations.

[0024] Furthermore, in model training and optimization, supervised loss and consistency loss are used as constraints, and the overall loss function can be expressed as:

[0025] L total =L det +γL cons

[0026] in To monitor the loss, the target detector is optimized using labeled visible light images. S The detector results For the corresponding reference tags; For consistency loss, The target detector's detection results for the mixed image. γ represents the reference label corresponding to the mixed image; γ is the dynamic weight, adjusted according to the reliability of the pseudo-label; L() is the general cross-entropy loss, used to measure the classification loss.

[0027] In summary, the technical solutions conceived by this invention have the following beneficial effects compared with the prior art:

[0028] (1) Utilization of global information: By using a regional pixel-level mixing strategy and dynamically adjusting the mixing ratio based on the pseudo-label confidence, the limitations of existing methods that only focus on high-confidence regions are overcome, and the model's ability to learn global thermal features of the target domain in infrared images is enhanced. This is particularly suitable for scenarios where the target scale is small and the thermal features are blurred.

[0029] (2) Improved reliability of pseudo-tags: A two-way dynamic region pseudo-tag filtering mechanism is adopted to introduce more infrared target information in the early stage of training and to screen reliable pseudo-tags in the later stage, thereby improving the quality of pseudo-tags and effectively improving the detection accuracy of infrared UAVs.

[0030] (3) High efficiency and real-time performance: Based on a single-stage detector (such as YOLOv8), combined with pixel-level mixing and consistency loss optimization, it meets the requirements of infrared UAV platforms for real-time performance and lightweight design. Attached Figure Description

[0031] Figure 1 The following is a flowchart of the infrared image-based UAV domain adaptive target detection process provided in this embodiment of the invention;

[0032] Figure 2 The embodiment of this invention provides a domain adaptive network structure diagram based on pixel-level mixing of regional pseudo-label confidence. Detailed Implementation

[0033] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0034] To address the performance degradation issue caused by domain distribution differences in adaptive target detection in the infrared target domain of unmanned aerial vehicles (UAVs), this invention provides an adaptive target detection method for the UAV image domain based on pixel-level fusion of region pseudo-label confidence. Through a region pixel-level fusion strategy and a bidirectional dynamic region pseudo-label filtering mechanism, this method achieves effective knowledge transfer from the source domain to the target domain, improving the model's detection accuracy and generalization ability in UAV scenarios. The specific implementation of this invention is described in detail below using a YOLOv8 single-stage detector as an example.

[0035] refer to Figures 1-2 This embodiment takes the domain-adaptive target detection task of UAV images from a clear sky scene (source domain) to a foggy sky scene (target domain) as an example. The specific implementation steps are as follows:

[0036] Data preparation: Prepare a source domain dataset, such as visible light images of drones acquired using a visible light imaging device (denoted as x). S The system then annotates the drones in the images, generating corresponding annotation data y. S (Including the bounding box and category of the UAV target). Prepare a target domain dataset, such as infrared images of UAVs acquired using infrared thermal imaging equipment (denoted as x). T ).

[0037] Source Domain Detector Training: Using YOLOv8 as the object detection network and the source domain dataset, the network is trained using supervised learning to obtain the source domain object detector F. VL and with F VL The initial model for the target detector in the target domain is the target detector F = F IR =F VL .

[0038] Pseudo-label generation: using the target detector F IR Perform UAV target detection on the target domain image, and use the detection output as pseudo-labels (including the bounding boxes of detected targets and confidence scores) to form a pseudo-label set. Where N T The number of targets detected. The pseudo-label for the i-th target (containing bounding box coordinates and confidence score).

[0039] Region partitioning and confidence calculation: The source domain image x... S and target domain infrared image x T Divide the area into four equal regions using a 2×2 grid, denoted as follows: and Assign each pseudo-label to a corresponding region based on its center coordinates, calculate the average confidence score for each region, and record the result as follows: This represents the average pseudo-label confidence level of the j-th region; if there are no pseudo-labels in the region, the confidence level is set to 0.

[0040] Confidence-based region-level pixel blending: based on the average confidence level of each region. Ranking and Beta function parameter assignment: the highest confidence region uses a Beta function biased by 1 (32, 18), the lowest confidence region uses a Beta function biased by 0 (18, 32), and the medium confidence region uses a Beta function biased by 0.5 (32, 32). Let λ be the weight ratio generated by the Beta function with the corresponding parameters for the i-th region. i After obtaining the weights, the four regions are blended according to the following formula to obtain the blended image of the four regions. The final blended image x can then be obtained through merging. M :

[0041]

[0042] Two-way dynamic region pseudo-label filtering: A two-way dynamic region pseudo-label filtering mechanism is used to filter infrared images x T generated pseudo tags Perform filtering, and denote the set of pseudo-labels after filtering as follows: In global filtering, which ranges from loose to tight, the filtering threshold C th Set to 0.3. In the bridging filtering process, from strictest to loosest, the dynamic threshold C is... p (t) from C b =0.9 to C e =0.8 transition.

[0043] Hybrid label generation: This involves combining the source domain image labels y... S And target domain pseudo-labels filtered by bidirectional dynamic region pseudo-labels Merge the images to generate hybrid image pseudo-labels.

[0044] Model Training and Optimization: Utilizing Source Domain Image x S and the generated mixed image x M The target detector F is trained by domain adaptation to obtain the updated target detector, and the updated target detector is used to replace the target detector F used in the previous round.

[0045] Repeat the above six steps, from pseudo-label generation to model training and optimization, for 100 iterations to obtain the final target detector F. Then, the final target detector F is applied to infrared UAV target detection applications.

[0046] The above content is readily understood by those skilled in the art. The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. An unmanned aerial vehicle domain adaptive detection method of image pixel level hybrid enhancement, characterized in that, The method comprises the following steps: S1, reading data: reading source domain data and target domain data, taking the source domain data as visible light image x S and its label y S , and the target domain data as infrared image x T ; S2, pseudo label generation: input the infrared image x T to the target detector F to generate a set of pseudo labels of the target domain where N T is the number of detected targets, is the pseudo label of the i-th target; wherein the target detector F is initially trained using source domain data, and the pseudo label contains the bounding box coordinates and the confidence. S3, Region division and confidence calculation: visible light image x S and infrared image x T are respectively divided into four equal regions, denoted as and respectively represent the jth equal region of visible light image x S and infrared image x T ; according to the region where the center point of the pseudo label is located, each pseudo label is assigned to the corresponding equal region, and the average pseudo label confidence of each region is calculated, denoted as represents the average pseudo label confidence of the jth region. S4, bidirectional dynamic region pseudo-label filtering: a bidirectional dynamic region pseudo-label filtering mechanism is adopted to filter the infrared image x T Generated pseudo-labels Screening, and the target domain pseudo-label after screening is denoted as S5, confidence-based hybrid image generation: region confidence weighted pixel-wise fusion is performed on the infrared image x T and the visible light image x S to generate a hybrid image x M ; S6, mixed label generation: the label y of the source domain data is generated S and the target domain pseudo label filtered through bidirectional dynamic region pseudo label filtering are merged to generate a mixed image pseudo label S7, model training and optimization: using visible light image x S and the generated mixed image x M The target detector F is updated and trained in the field to obtain an updated target detector F, and the updated target detector is used to replace the target detector F used in the previous round. The steps of S2 to S7 are repeated, and through iteration, the final target detector F is obtained, so as to perform infrared image unmanned aerial vehicle field self-adaptive detection based on image pixel-level mixed enhancement.

2. The UAV domain adaptive detection method of image pixel-level hybrid enhancement according to claim 1, characterized in that, In the bidirectional dynamic region pseudo-label filtering step, the bidirectional dynamic region pseudo-label filtering mechanism adopts a combination of a global filtering process from loose to strict and a bridging process filtering process from strict to loose; In the global filtering process from loose to strict, the pseudo label confidence dynamic adjustment strategy is adopted, and the pseudo labels with confidence greater than the threshold are screened out through the pre-set fixed threshold C th , to realize the filtering of pseudo labels from loose to strict. In the bridge filtering process from strict to loose, the threshold adaptive dynamic adjustment strategy is adopted, and the average confidence of the highest confidence area is taken as the benchmark p (t), and the pseudo label is introduced from the non-highest confidence area from strict to loose.

3. The UAV domain adaptive detection method of claim 2, wherein, The global filtering process from loose to strict specifically comprises: In the initial training stage, a relatively loose pseudo-label confidence score strategy is adopted to allow more pseudo-labels to participate in training, and with the increase of the training round number, the strictness of the confidence score is gradually improved to filter out untrustworthy pseudo-labels, and the pseudo-label confidence C(t) is obtained by using the following calculation formula: δ(t) is a factor dynamically adjusted with iteration round t, and its value range is (0, 1); M is the number of samples used in each iteration; a is a preset constant; C det (t) and C bbx (t) are the class confidence and position confidence of the pseudo label at each iteration round, respectively, which are obtained by prediction of the target detector F.

4. The UAV domain adaptive detection method of claim 2, wherein, In the bridge filtering process from strict to loose, the threshold adaptive dynamic adjustment strategy is adopted, and the average confidence of the highest confidence area is taken as the benchmark. By dynamically adjusting the threshold C p (t), the pseudo label is introduced from the non-highest confidence area from strict to loose, and the threshold C p (t) is calculated as: where C b and C e are the initial and final thresholds, respectively, n(t) and N are the current training iteration round t and the total training iteration rounds, is the average confidence of the highest confidence region for the t-th round of pseudo labels.

5. The UAV domain adaptive detection method of image pixel-level hybrid enhancement of claim 1, wherein, The confidence-based mixed image generation step specifically comprises the following steps: First, according to the average pseudo-label confidence of each region The size ranks the four regions, and the four regions are sequentially corresponded to high, medium, medium and low confidence regions in descending order of ranking. Then for the three types of high, medium and low confidence, Beta(α,β) function with three different parameters is used to randomly generate a weight ratio λ for each region during the generation of a mixed image i ; for the high confidence region, Beta function with distribution biased to 1 is used to make the visible image x S have a higher weight ratio; for the low confidence region, Beta function with distribution biased to 0 is used to make the visible image x S have a smaller weight ratio; for the medium confidence region, Beta function with distribution biased to 0.5 is used to make the visible image x S and the infrared image have equal weight ratio values; After generating the weight ratio λ i After that, the four regions are mixed respectively according to the following formula to obtain a mixed image of the four regions The final mixed image x is obtained through the merging process M : Wherein, ⊙ represents a pixel-level mixing operation.

6. The UAV domain adaptive detection method of image pixel-level hybrid enhancement of claim 1, wherein, In the model training and optimization step, when the domain adaptation update training is performed on the target detector F, the supervised loss and the consistency loss are used for constraint, and the total loss function L is total is represented as: L total = L det + γL cons wherein is a supervision loss, the target detector is optimized by the labeled visible image pair, y S is a detector detection result, is a corresponding reference label; is a consistency loss, is a detection result of the target detector on the mixed image, is a corresponding reference label of the mixed image; γ is a dynamic weight, which is adjusted according to the pseudo label reliability; L() is a general cross-entropy loss, which is used to measure the classification loss.

Images