Visible light and thermal infrared image target fusion detection method from the perspective of UAV vision

Abstract

The present invention provides a method for target fusion detection of visible light and thermal infrared images from the visual angle of a drone, which belongs to the field of image processing technology, and includes: preprocessing and data enhancement of the data set of visible light and thermal infrared images of the detection area acquired by the drone to obtain a number of initial images; analyzing the initial images at different scales based on a multi-scale dynamic feature pyramid network, fusing the initial images in the visible light mode with the initial images in the thermal infrared mode based on an attention fusion mechanism, and locking the candidate area, thereby highlighting small targets, long-distance targets and candidate areas; splitting all target samples to obtain a training set and a verification set, and training a preset network model based on the training set to obtain an initial detection model, and performing model performance evaluation and model optimization on the initial detection model based on the verification set to obtain a target detection model. The problem of target detection challenges of drones in complex environments is solved.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a method for detecting a target by fusing visible light and thermal infrared images from a visual angle of an unmanned aerial vehicle. Background Art

[0002] With the widespread application of drones in disaster relief, security monitoring, public safety and other fields, higher requirements are placed on their target detection capabilities in complex environments. Traditional visible light images and thermal infrared images each have their own advantages in drone target detection: visible light images can provide rich texture and color information, while thermal infrared images can accurately detect heat source targets in low light and bad weather conditions. However, the detection methods using visible light or thermal infrared images alone have limitations and cannot meet the diverse needs in complex environments. Therefore, the fusion of visible light and thermal infrared image information has become the key to improving detection robustness and adaptability.

[0003] The current visible light and thermal infrared image fusion detection technology has improved the detection accuracy to a certain extent, but there are still some problems when applied under the dynamic flight perspective of drones: (1) Existing methods mostly use the multi-branch structure of deep neural networks to achieve the alignment and fusion of multimodal features, but the complexity of the algorithm leads to high computational cost and delay, which cannot meet the real-time and low power consumption requirements of drones; (2) There are bottlenecks in improving the accuracy of small target and long-distance target detection, and there is also a lack of effective data enhancement methods to improve the generalization ability of the model; (3) Perspective change: The image perspective of the drone will change continuously due to changes in position and altitude during flight, and the shape, size, and angle of the target will change accordingly. Traditional detection algorithms based on plane perspective are difficult to adapt to these changes; (4) Target posture change: Under drone observation, the posture of the target may change rapidly, especially non-rigid targets (such as humans and vehicles) are more likely to change posture during movement, making it difficult for single-modal feature extraction to stably detect the target.

[0004] Invention patent: A visible light-thermal infrared image target detection method from the perspective of a drone (CN202210577635.2), which has achieved certain results in visible light and thermal infrared image fusion target detection from the perspective of a drone, but still has the following shortcomings:

[0005] 1. High computational complexity

[0006] Due to the use of the FasterR-CNN architecture and multi-branch dynamic symmetric feature extraction network, and the introduction of normalized Wasserstein distance in the distribution of positive and negative samples, the computational complexity is high, which will affect real-time performance. In particular, the resources of the UAV platform are limited, and the computing resources and power are limited. This high-complexity algorithm may not be suitable for real-time processing.

[0007] 2. Limited adaptability to weather and environmental changes

[0008] Under some extreme weather conditions (such as strong wind, rain, snow, haze, etc.), the quality of both visible light and infrared images may decrease, thus affecting the detection accuracy. The current methods may not fully consider the adaptability in such situations.

[0009] 3. Lack of efficient cross-modal alignment mechanism

[0010] The feature distribution of visible light and infrared images is quite different. Existing feature fusion methods may not be sufficient to fully align cross-modal features, which may lead to a decrease in detection accuracy, especially when the target scale and morphology vary greatly.

[0011] 4. Lack of dedicated optimization for small goals

[0012] Although the normalized Wasserstein distance is used to improve the detection effect of small targets, the detection effect for ultra-small targets (long distance or extremely small target size) may need further optimization.

[0013] 5. Limited optimization in adapting to changes in viewpoint and target pose

[0014] In response to changes in perspective and target posture, this patent only introduces deformable convolutions into the backbone network, achieving partial optimization, but there is still room for further improvement.

[0015] Therefore, the present invention proposes a visible light and thermal infrared image target fusion detection method from the visual angle of an unmanned aerial vehicle. Summary of the invention

[0016] The present invention provides a visible light and thermal infrared image target fusion detection method from the visual angle of an unmanned aerial vehicle (UAV) to solve the problem of target detection challenges of UAVs in complex environments.

[0017] The present invention provides a method for detecting targets by fusion of visible light and thermal infrared images from the visual angle of an unmanned aerial vehicle, comprising:

[0018] Step 1: Preprocess the dataset of visible light and thermal infrared images of the detection area acquired by the drone, and perform data enhancement on the preprocessed images to obtain several initial images;

[0019] Step 2: hierarchically abstract and combine the image features of the initial image at different scales based on a multi-scale dynamic feature pyramid network to lock tiny targets and long-distance targets in the initial image;

[0020] Step 3: Based on the attention fusion mechanism, the initial image in the visible light mode is fused with the initial image in the thermal infrared mode, and the candidate area is locked, and then the small targets, long-distance targets and candidate areas are highlighted to obtain several target samples;

[0021] Step 4: Split all target samples into a training set and a validation set, and train a preset network model based on the training set to obtain an initial detection model, and perform model performance evaluation and model optimization on the initial detection model based on the validation set to obtain a target detection model;

[0022] Wherein, locking the tiny target and the long-distance target in the initial image comprises:

[0023] Based on the execution logic of each feature layer in the structural design of the multi-scale dynamic feature pyramid network, hierarchical feature extraction is performed on different initial images;

[0024] Based on a dynamic weighting mechanism, each hierarchical feature is assigned a channel weight, and all hierarchical features are fused to detect tiny targets and long-distance targets in the initial image;

[0025] The process of fusing the initial image in the visible light mode with the initial image in the thermal infrared mode based on the attention fusion mechanism includes:

[0026] Applying the initial image in the visible light mode and the initial image in the thermal infrared mode to the self-attention fusion mechanism respectively, obtaining the autocorrelation weights in different modes and constructing an attention matrix, wherein the attention matrix represents the correlation weights in different viewing angles of different modes;

[0027] Applying the initial image in the visible light modality and the initial image in the thermal infrared modality to a cross-modal attention fusion mechanism, obtaining the cross-correlation weights between different modalities and constructing a weight matrix, wherein the weight matrix represents the complementarity of the visible light modality and the thermal infrared modality at the viewing angle position;

[0028] Based on the attention matrix and the weight matrix, the detection requirements under different viewing angles and postures during the flight of the drone are met, and the detection of the current target in the initial image is achieved by combining the attention weights.

[0029] Preferably, the preprocessing includes: image normalization processing and image denoising processing.

[0030] Preferably, data enhancement is performed on the preprocessed image, including:

[0031] Determine the drone shooting angle and shooting posture of each pre-processed image to obtain a shooting array;

[0032] Adaptively adjusting the photographing array according to a rotation angle of 1° to obtain a plurality of adaptive images corresponding to the preprocessed image;

[0033] Determine the data richness of each preprocessed image separately;

[0034] ;

[0035] in, represents the richness of the extracted features of the i1th preprocessed image; represents the area occupied by the target of interest in the i1th pre-processed image determined under the corresponding shooting angle and shooting posture; represents the area occupied by the target of interest determined after adjusting the i1th pre-processed image based on the reset shooting angle and shooting posture; ln represents the logarithmic function symbol;

[0036] according to Determine a first enhancement quantity, where F0 represents a preset richness; Indicates the clarity of the i1th preprocessed image; Q0 indicates the preset clarity; Indicates that the i1th preprocessed image is based on the drone shooting perspective Shooting angle of view with theoretical level The number to be supplemented due to the visual difference; ceiling indicates the symbol for rounding up;

[0037] Each preprocessed image is divided into unit blocks to obtain a block set;

[0038] Performing data enhancement processing on each unit block in the block set, wherein the data enhancement processing is determined according to the number of block feature coefficients in the block set that are greater than the effective coefficients;

[0039] Determining a second enhancement quantity of the corresponding preprocessed image according to a data enhancement processing result of each unit block in the corresponding preprocessed image;

[0040] sorting the image richness of all preprocessed images, and assigning a shooting weight to each shooting array according to the image richness;

[0041] Matching a third enhancement quantity consistent with the shooting weight from a weight-quantity comparison table;

[0042] Determine the corresponding final enhancement number according to max(Dn1, Dn2, Dn3), and randomly select the same number of first images from the multiple adaptive images under the same preprocessed image and continue to perform adaptive adjustment according to the rotation angle of 1° to obtain multiple adaptive images of each first image, where Dn1 represents the first enhancement number corresponding to the same preprocessed image, Dn2 represents the second enhancement number corresponding to the same preprocessed image, and Dn3 represents the third enhancement number corresponding to the same preprocessed image;

[0043] Determine the shooting environment of the drone when determining each pre-processed image, and perform index analysis on the shooting environment to construct an environment vector;

[0044] Element-adjusting the environment vector to obtain a new vector, simulating a new environment of the new vector, and fusing the new vector with each corresponding first image to obtain a second image;

[0045] Among them, all adaptive images and the second image are used as initial images.

[0046] Preferably, the process of detecting the target in the initial image by combining attention weighting includes:

[0047] Obtain the important weights of the convolution kernels involved in the detection process of each current target, and remove the convolution kernels whose important weights are less than the preset weights;

[0048] The response values ​​of the channels involved in the detection process of each current target are obtained, and the channels with response values ​​less than the preset threshold are eliminated.

[0049] Preferably, the highlighting process refers to enhancing the details of tiny targets, distant targets and candidate areas.

[0050] Preferably, performing model performance evaluation and model optimization on the initial detection model based on the validation set to obtain a target detection model includes:

[0051] Capturing the current participating parameters of each detection layer during the analysis of each verification sample in the verification set by the initial detection model, and judging whether the current participating parameters meet the following requirements: the number of participating parameters of the corresponding layer is consistent with the preset number, the role played by each participating parameter is greater than the preset role, and the connection and coordination relationship between the current participating parameters and the participating parameters of the adjacent layers meets the model operation setting standard;

[0052] If satisfied, the corresponding detection layer is judged to be running properly;

[0053] Otherwise, extracting the unsatisfied index from the judgment result, and judging the supporting behavior of the unsatisfied index to the initial detection model;

[0054] Based on the state fullness coefficient of the supporting behavior and the set weight of the corresponding unsatisfied index based on the initial detection model, the coefficient to be optimized of the corresponding supporting behavior is obtained;

[0055] ;

[0056] in, represents the coefficient to be optimized for the j1th supporting behavior in the corresponding detection layer; represents the normalized function of the j1th support behavior in the corresponding detection layer; represents the set weight of the unsatisfied index in the corresponding detection layer that is consistent with the j1-th supporting behavior; represents the state fullness coefficient under the j1th supporting behavior in the corresponding detection layer; Nc represents the preset number of parameters planned for the corresponding detection layer; Represents the weight sum of all indicators in the corresponding detection layer;

[0057] Function fitting is performed on all coefficients to be optimized to obtain the final optimization factor, and based on the comparison relationship between the final optimization factor and each supporting behavior, the optimization parameters based on the corresponding detection layer are obtained and the model is optimized.

[0058] Compared with the prior art, the beneficial effects of this application are as follows:

[0059] Through the multi-scale dynamic feature pyramid network, cross-modal alignment mechanism, lightweight and fast reasoning strategy, feature fusion based on adaptive weights, and multi-view and multi-pose fusion data enhancement methods, the fusion detection accuracy and real-time performance of drones under the conditions of changing view angles and target poses are effectively improved. This method is not only suitable for drone target detection in complex dynamic environments, but also reduces computational costs while improving detection efficiency, and can meet the real-time requirements of practical applications.

[0060] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures particularly pointed out in the written description and the accompanying drawings.

[0061] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0063] Figure 1The present invention is a flowchart of a method for fusion detection of visible light and thermal infrared images of targets from the visual angle of a UAV in an embodiment of the present invention. DETAILED DESCRIPTION

[0064] The preferred embodiments of the present invention are described below in conjunction with the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0065] The present invention provides a method for detecting targets by fusion of visible light and thermal infrared images from the visual angle of an unmanned aerial vehicle. Figure 1 As shown, including:

[0066] Step 1: Preprocess the dataset of visible light and thermal infrared images of the detection area acquired by the drone, and perform data enhancement on the preprocessed images to obtain several initial images;

[0067] Step 2: hierarchically abstract and combine the image features of the initial image at different scales based on a multi-scale dynamic feature pyramid network to lock tiny targets and long-distance targets in the initial image;

[0068] Step 3: Based on the attention fusion mechanism, the initial image in the visible light mode is fused with the initial image in the thermal infrared mode, and the candidate area is locked, and then the small targets, long-distance targets and candidate areas are highlighted to obtain several target samples;

[0069] Step 4: Split all target samples into training sets and validation sets, and train the preset network model based on the training set to obtain an initial detection model, and perform model performance evaluation and model optimization on the initial detection model based on the validation set to obtain a target detection model.

[0070] In this embodiment, the training set and the validation set include samples from multiple perspectives, multiple postures, different weather conditions, and small targets, ensuring the comprehensiveness of the data to achieve comprehensive training and comprehensive validation of the model.

[0071] In this embodiment, the optimized model and the known model are compared on the same data set to comprehensively evaluate the model performance using indicators such as mean average precision (mAP) and inference speed.

[0072] In this embodiment, the preset network model refers to a neural network model.

[0073] Preferably, the preprocessing includes: image normalization processing and image denoising processing.

[0074] Preferably, locking onto a tiny target and a distant target in the initial image comprises:

[0075] Based on the execution logic of each feature layer in the structural design of the multi-scale dynamic feature pyramid network, hierarchical feature extraction is performed on different initial images;

[0076] Based on a dynamic weighting mechanism, channel weighting is assigned to each hierarchical feature respectively, and feature fusion is performed on all hierarchical features to detect tiny targets and long-distance targets in the initial image.

[0077] Preferably, the process of fusing the initial image in the visible light modality with the initial image in the thermal infrared modality based on the attention fusion mechanism includes:

[0078] Applying the initial image in the visible light mode and the initial image in the thermal infrared mode to the self-attention fusion mechanism respectively, obtaining the autocorrelation weights in different modes and constructing an attention matrix, wherein the attention matrix represents the correlation weights in different viewing angles of different modes;

[0079] Applying the initial image in the visible light modality and the initial image in the thermal infrared modality to a cross-modal attention fusion mechanism, obtaining the cross-correlation weights between different modalities and constructing a weight matrix, wherein the weight matrix represents the complementarity of the visible light modality and the thermal infrared modality at the viewing angle position;

[0080] Based on the attention matrix and the weight matrix, the detection requirements under different viewing angles and postures during the flight of the drone are met, and the detection of the current target in the initial image is achieved by combining the attention weights.

[0081] Preferably, the process of detecting the target in the initial image by combining attention weighting includes:

[0082] Obtain the important weights of the convolution kernels involved in the detection process of each current target, and remove the convolution kernels whose important weights are less than the preset weights;

[0083] The response values ​​of the channels involved in the detection process of each current target are obtained, and the channels with response values ​​less than the preset threshold are eliminated.

[0084] The highlighting process refers to enhancing the details of small targets, distant targets and candidate areas.

[0085] In this embodiment, the image is normalized and denoised in order to ensure that the training image of the input model is stable and consistent.

[0086] In this embodiment, data enhancement is to improve the adaptability of the model under different viewing angles and postures.

[0087] In this embodiment, the yolov8 neural network is used as the basic network, and the specific improvement and optimization methods are as follows:

[0088] By dynamically constructing a multi-scale feature pyramid structure, the problem of target size and position changes caused by changes in perspective under the drone's perspective is solved. The network can hierarchically abstract and combine image features at different scales, and improve the detection accuracy of small and long-distance targets through hierarchical feature fusion, including:

[0089] Multi-scale feature abstraction and fusion mechanism:

[0090] The core of the FPN network lies in its adaptive multi-scale feature extraction structure. Traditional convolutional neural networks usually extract features at only one scale, while FPN introduces a multi-level pyramid structure that enables the network to transfer and fuse information between high-resolution and low-resolution feature maps.

[0091] Structural design: FPN includes a bottom-up feature pyramid construction process, which mainly includes the following levels:

[0092] Bottom feature layer: retains high-resolution information of the image and is suitable for detecting small targets.

[0093] Middle feature layer: contains abstract semantic information and is used to detect medium-sized objects.

[0094] High-level feature layer: further abstracts semantic information and is suitable for large object detection.

[0095] By transferring and fusing information layer by layer in these feature layers, FPN can generate adaptive feature pyramids at different scales and improve the detection accuracy of objects of different sizes.

[0096] Construction and dynamic fusion of feature pyramid:

[0097] The feature pyramid is implemented through adaptive feature fusion. This fusion strategy allows information interaction between high-resolution layers and low-resolution layers, enabling the feature pyramid to adaptively adjust feature weights of different scales.

[0098] Feature pyramid construction formula:

[0099] ;

[0100] in, Indicates Layer characteristics; Indicates that by upsampling The resolution of layer features is increased to Layer scale.

[0101] This formula realizes the layer-by-layer fusion of features, allowing small target features to be clearly extracted in the underlying high-resolution feature maps. Adjacent features of each layer of features are upsampled or downsampled so that multi-scale features can be integrated into the final pyramid features.

[0102] Dynamic feature weighting mechanism:

[0103] The dynamic weighting mechanism can weight features of different scales according to the detection task, mainly used to improve the detection accuracy of small and distant targets. Through adaptive channel weighting, different scale features are given different weights, thereby enhancing the response to key scales.

[0104] Dynamic feature weighting formula:

[0105] ;

[0106] in, represents the fused multi-scale features, is the adaptive feature weighting coefficient, obtained through learning, is the feature of each layer in the pyramid. Weighting coefficient It is dynamically generated during the training process through the attention mechanism and can be adaptively adjusted according to the feature distribution of the input image and the scale of the detection target.

[0107] Cross-modal alignment mechanism:

[0108] In order to better fuse the visible light and infrared image features, the present invention adopts a cross-modal alignment model based on the self-attention mechanism to make the features from the two modalities more consistent in the feature space to enhance the final detection effect.

[0109] The role of the self-attention mechanism is to generate correlation weights based on its own features to capture the relationship between features of different modalities. By applying self-attention to visible light and infrared images separately, the autocorrelation weights of each modality can be obtained, and then these weights can be combined to enhance cross-modal fusion.

[0110] Self-attention calculation formula:

[0111] ;

[0112] Where: Q, K, V represent the query matrix, key matrix and value matrix respectively, which are obtained by linear transformation of input feature F.

[0113] is a scaling factor used to balance the size of the matrix multiplication result.

[0114] The calculated attention matrix characterizes the relevance weights of different positions, enabling the network to capture long-range dependencies between modalities.

[0115] The cross-modal attention mechanism utilizes the feature maps of two modalities and To generate the cross-correlation weights, the features of the two modalities can be better aligned and integrated. The cross-modal attention mechanism calculates the correlation between different modalities by generating query (Q), key (K), and value (V) matrices for the features of each modality.

[0116] Cross-modal attention calculation formula:

[0117] ;

[0118] The weight matrix generated by the interactive attention can enhance the complementarity of visible and infrared modalities and thus align features under different viewpoints.

[0119] Modal feature fusion and output:

[0120] Based on the dynamic attention mechanism and cross-modal alignment mechanism, the fused features can effectively meet the detection requirements of different viewing angles and postures during the flight of the drone. Finally, target detection is achieved by merging the attention-weighted modal feature maps:

[0121] ;

[0122] This adaptive cross-modal fusion method can not only dynamically focus on the key features of different modalities, but also improve the accuracy and robustness of detection in cross-modal situations.

[0123] Lightweight and fast reasoning strategy:

[0124] Model pruning is a technique to simplify the model structure by removing redundant parameters. By identifying and removing convolution kernels or neurons that contribute less to model predictions, the model parameters can be reduced while maintaining the accuracy of the model.

[0125] Pruning strategy:

[0126] Pruning based on weight threshold: First, the weight of each convolution layer is evaluated and the importance weight of each convolution kernel is calculated. If the weight of some convolution kernels is less than the preset threshold, it is considered that they have little impact on the model performance and can be safely removed.

[0127] Pruning based on channel importance: By calculating the response value or gradient size of each channel, the impact of the channel on the output is determined and unimportant channels are removed. This pruning method is suitable for lightweight convolutional neural networks.

[0128] Pruning formula:

[0129] The pruning loss function can be implemented by introducing the regularization term R(W), and the formula is as follows:

[0130] ;

[0131] in: is the original task loss function; R(W) is the regularization term, which is used to weaken the weight of unimportant parameters (for example, L1 regularization can achieve sparsity); is the regularization weight coefficient, which is used to control the strength of model pruning.

[0132] Through pruning, the structure of the model is streamlined, significantly reducing memory requirements and computational complexity, thereby improving inference speed.

[0133] Local region of interest pooling (RoI Pooling) reduces the waste of computing resources by performing detailed analysis on candidate regions. During the candidate region generation process, this technology extracts the region of interest for pooling, thereby accurately passing the region in the feature map that may contain the target to the subsequent detection layer.

[0134] Principle analysis:

[0135] RoI Pooling divides the candidate region into fixed-size grids (e.g. 7×7) and uses the maximum pooling operation on each grid to generate fixed-size features. This allows for more efficient feature extraction in the candidate region, thereby reducing the computation of irrelevant areas.

[0136] Region division: Divide the candidate region in the input feature map into uniform sub-regions, each sub-region is .

[0137] Max pooling: A max pooling operation is performed in each sub-region to compress each sub-region into a single pixel value, thereby generating an output feature map of a fixed size.

[0138] The formula for RoI Pooling is:

[0139] ;

[0140] in: is the pooling result of the i-th and j-th sub-regions; R(i,j) represents the pixel set of the i-th and j-th sub-regions; F(x,y) is the value at position (x,y) on the input feature map.

[0141] Advantages of RoI Pooling:

[0142] By extracting and processing features only on candidate regions that may contain targets, RoI Pooling greatly reduces the amount of computation, allowing the model to achieve real-time target detection with limited resources.

[0143] The lightweight and fast reasoning strategy of the present invention reduces the complexity and computational requirements of the model through the above methods. Model pruning simplifies the network structure, and RoI Pooling avoids global feature calculation, ultimately achieving fast reasoning in resource-constrained environments and meeting the needs of real-time drone detection.

[0144] Feature fusion based on adaptive weights:

[0145] This paper proposes a feature fusion mechanism based on adaptive weights, aiming to improve the detection capability of small targets. The main challenge of small target detection is that its response in the feature map is weak, which is easily affected by the feature extraction and fusion process and leads to missed detection. By adaptively adjusting the feature weights of small target areas, the detection sensitivity and accuracy of the model for small targets can be effectively improved.

[0146] Feature fusion principle based on adaptive weights

[0147] The core idea of ​​adaptive weighted feature fusion is to dynamically adjust the weights of each region in the feature map according to the size and distance of the target, so that small target regions receive higher attention during the fusion process. This mechanism increases the significance of small target features in the final detection results by adding an adaptive weighting module during the feature fusion process.

[0148] This mechanism can be expressed as assigning an adaptive weight w(i, j) to each pixel in different feature layers, so that the fusion feature has stronger expressive power. Specifically:

[0149] For small target areas, higher weights are given to enhance their features.

[0150] For non-target areas or background areas, lower weights are given to reduce their influence on the final features.

[0151] Adaptive weight calculation formula

[0152] In order to achieve this adaptability, the present invention introduces a weight calculation method to map the size information and distance information of the target into a weight adjustment factor. The adaptive weight w(i,j) can be calculated by the following formula:

[0153] ;

[0154] Where: w(i,j) represents the adaptive weight at the feature map position (i,j);

[0155] s(i,j) represents the target size at position (i,j);

[0156] d(i,j) represents the distance or other scale information of the target;

[0157] α and β are hyperparameters that control the contribution of each.

[0158] Calculation method explanation:

[0159] Size function : It is used to indicate the importance of small targets in the feature map. This function can be designed to be inversely proportional to the target size to enhance the weight of small targets. The common function form is:

[0160] ;

[0161] in, is a small constant used to avoid zero denominator.

[0162] Distance function : It is used to indicate the importance of distant targets. The farther the target is, the lower its resolution in the image, so a higher feature weight is needed to enhance its detection effect. The function can be designed as:

[0163] ;

[0164] in: Control the degree to which distance affects weight; Indicates a specific distance threshold. When the distance is greater than , the weight increases.

[0165] Weight application in feature fusion process

[0166] During the feature fusion process, the calculated adaptive weights are applied to each feature map position to generate an enhanced fused feature map The fusion formula is as follows:

[0167] ;

[0168] in: and Represent the features of the visible light image and infrared image at position (i, j); Represents the fused features.

[0169] This fusion method assigns higher weights to small target locations, making the features of small target areas more prominent and thus more sensitive in subsequent detection.

[0170] Advantages of Adaptive Feature Fusion

[0171] The following advantages can be achieved through the feature fusion mechanism of adaptive weights:

[0172] Improve the detection sensitivity of small targets: The increase of adaptive weights ensures that small target features will not be masked by large targets or background information during the fusion process, thereby reducing the missed detection rate.

[0173] Improve the robustness of the model to long-distance targets: By increasing the weight at long-distance positions, the detection accuracy of long-distance small targets can be improved.

[0174] Dynamic response to perspective changes: This adaptive mechanism can automatically adjust feature weights according to different detection requirements and adapt to changes in the perspective of the drone during flight.

[0175] This adaptive weight-based feature fusion mechanism greatly improves the detection accuracy and robustness of small targets from the perspective of drones.

[0176] In this embodiment, training data construction and batch generation: a diversified data set enhanced with multi-view and multi-posture is used to train the model to ensure the adequacy of the training data, and high-frequency training is specially designed for small target samples to ensure that the model focuses on small targets.

[0177] Loss function design and optimization: Focal Loss: Focal Loss is added to the loss function to deal with the problem of category imbalance and improve the model's attention to small targets and difficult samples.

[0178] Bimodal input training is adopted, and the cross-modal attention module is used to dynamically fuse the features of different modalities.

[0179] During the training process, the posture perception module is used to adaptively adjust the feature weights to improve the multi-view alignment effect.

[0180] Model lightweight training: Pruning strategy: Prune in stages to remove redundant nodes and maintain accuracy to provide guarantee for real-time detection.

[0181] The beneficial effects of the above technical solution are: through multi-scale dynamic feature pyramid network, cross-modal alignment mechanism, lightweight and fast reasoning strategy, feature fusion based on adaptive weight, and multi-view and multi-posture fusion data enhancement method, the fusion detection accuracy and real-time performance of drones under the conditions of changing view angle and target posture are effectively improved. This method is not only suitable for drone target detection in complex dynamic environments, but also reduces the computational cost while improving detection efficiency, and can meet the real-time requirements of practical applications.

[0182] The present invention provides a visible light and thermal infrared image target fusion detection method from the visual angle of an unmanned aerial vehicle, which performs data enhancement on the preprocessed image, comprising:

[0183] Determine the drone shooting angle and shooting posture of each pre-processed image to obtain a shooting array;

[0184] Adaptively adjusting the photographing array according to a rotation angle of 1° to obtain a plurality of adaptive images corresponding to the preprocessed image;

[0185] Determine the data richness of each preprocessed image separately;

[0186] ;

[0187] in, represents the richness of the extracted features of the i1th preprocessed image; represents the area occupied by the target of interest in the i1th pre-processed image determined under the corresponding shooting angle and shooting posture; represents the area occupied by the target of interest determined after adjusting the i1th pre-processed image based on the reset shooting angle and shooting posture; ln represents the logarithmic function symbol;

[0188] according to Determine a first enhancement quantity, where F0 represents a preset richness; Indicates the clarity of the i1th preprocessed image; Q0 indicates the preset clarity; Indicates that the i1th preprocessed image is based on the drone shooting perspective Shooting angle of view with theoretical level The number to be supplemented due to the visual difference; ceiling indicates the symbol for rounding up;

[0189] Each preprocessed image is divided into unit blocks to obtain a block set;

[0190] Performing data enhancement processing on each unit block in the block set, wherein the data enhancement processing is determined according to the number of block feature coefficients in the block set that are greater than the effective coefficients;

[0191] Determining a second enhancement quantity of the corresponding preprocessed image according to a data enhancement processing result of each unit block in the corresponding preprocessed image;

[0192] sorting the image richness of all preprocessed images, and assigning a shooting weight to each shooting array according to the image richness;

[0193] Matching a third enhancement quantity consistent with the shooting weight from a weight-quantity comparison table;

[0194] Determine the corresponding final enhancement number according to max(Dn1, Dn2, Dn3), and randomly select the same number of first images from the multiple adaptive images under the same preprocessed image and continue to perform adaptive adjustment according to the rotation angle of 1° to obtain multiple adaptive images of each first image, where Dn1 represents the first enhancement number corresponding to the same preprocessed image, Dn2 represents the second enhancement number corresponding to the same preprocessed image, and Dn3 represents the third enhancement number corresponding to the same preprocessed image;

[0195] Determine the shooting environment of the drone when determining each pre-processed image, and perform index analysis on the shooting environment to construct an environment vector;

[0196] Element-adjusting the environment vector to obtain a new vector, simulating a new environment of the new vector, and fusing the new vector with each corresponding first image to obtain a second image;

[0197] Among them, all adaptive images and the second image are used as initial images.

[0198] In this embodiment, each image initially acquired by the drone has its corresponding viewing angle and posture, and therefore, can be directly determined. For example, the drone obtains image A1 based on 10° in the horizontal direction and in posture action 1, and then image A1 is pre-processed, wherein the shooting array = {shooting viewing angle, shooting posture}.

[0199] In this embodiment, adaptive adjustment refers to rotation based on the shooting angle of view in units of 1°, and the operation can be implemented based on corresponding image processing software, which belongs to the prior art, and 360 adaptive images can be obtained.

[0200] In this embodiment, the posture of the drone also changes adaptively during the angle rotation process. The comparison relationship between the change in viewing angle and the change in posture is pre-planned, and the changes can be directly matched.

[0201] In this embodiment, the value range of the richness of the extracted features is from 0 to 1, and the more features there are in the corresponding image, the larger the corresponding value.

[0202] In this embodiment, ,in, represents the size of the region of interest of the i1th preprocessed image determined under the corresponding shooting angle and shooting posture; represents the size of the area involved in the i1th pre-processed image determined under the corresponding shooting angle and shooting posture;

[0203] In this embodiment, the preset definition value is 0.6, and the definition value range of the image is 0 to 1.

[0204] In this embodiment, the theoretical horizontal viewing angle The value of is 0°.

[0205] In this embodiment, It is obtained from a difference-quantity comparison table, which contains different visual differences and the quantities to be supplemented that match the visual differences, and the value range of the quantity to be supplemented is 0 to 100.

[0206] In this embodiment, unit block segmentation refers to segmenting the preprocessed image into pieces of 1 cm×1 cm in size, and each preprocessed image corresponds to a block set.

[0207] In this embodiment, the block feature coefficient = the number of feature pixel points of the corresponding unit block / the total number of all pixel points in the corresponding unit block, and the value of the effective coefficient is 0.6.

[0208] If there are 10 unit blocks in the block set whose characteristic coefficients are greater than the effective coefficients, then each unit block is increased by an integer result of the corresponding block characteristic coefficient × 10 pictures. That is, based on each unit block, the unit block is randomly adjusted an integer number of times of the corresponding block characteristic coefficient × 10 pictures of the viewing angle.

[0209] In this embodiment, the average value of the enhancement quantity of each unit block in the corresponding preprocessed image is calculated, and the calculated result is rounded up to obtain a result which is the second enhancement quantity.

[0210] In this embodiment, the shooting weight of the shooting array=(the image richness of the corresponding image / the sum of the richness of all images)×(1+ln(2-sort number / total number)).

[0211] In this embodiment, the weight-quantity comparison table includes different shooting weights and enhancement quantities corresponding to the weights.

[0212] In this embodiment, the final enhancement number refers to obtaining the maximum number from the corresponding first enhancement number, the second enhancement number, and the third enhancement number.

[0213] In this embodiment, the shooting environment refers to the weather conditions of the corresponding place when the drone takes the image, such as specific information such as temperature, humidity, visibility, rain, wind, snow, etc., and the indicators are related to temperature, humidity, visibility, rain, wind, snow, etc.

[0214] In this embodiment, the environment vector={values ​​under different indicators}.

[0215] In this embodiment, element adjustment refers to randomly adjusting the number of each indicator in the environment vector. As long as there is a change in the value of the element in the environment vector, it can be regarded as generating a new vector, and then as generating a new environment.

[0216] In this embodiment, the new environment is obtained by inputting the new vector into the environment analysis model, and the model is obtained by training a neural network model based on different new vectors and the environment determination results of the new vectors as samples.

[0217] In this embodiment, the purpose of the fusion process is to perform image update on the first image to expand the sample.

[0218] The beneficial effect of the above technical solution is: by performing data enhancement processing, training samples of different perspectives, postures, and weather are effectively generated, which further improves the generalization ability of the model and its adaptability in complex environments. Specifically, the final enhancement quantity is comprehensively determined by three aspects: the quantity is determined based on the data richness of the image, the quantity is determined based on the data enhancement processing results of the unit block, and the quantity is determined based on the shooting weight, so as to ensure the maximization of the expansion, and subsequently, the first image is further fused with the environmental vector to obtain several second images, and the comprehensiveness of the sample is ensured by two enhancement methods.

[0219] The present invention provides a visible light and thermal infrared image target fusion detection method from the visual angle of an unmanned aerial vehicle, and performs model performance evaluation and model optimization on the initial detection model based on the verification set to obtain a target detection model, including:

[0220] Capturing the current participating parameters of each detection layer during the analysis of each verification sample in the verification set by the initial detection model, and judging whether the current participating parameters meet the following requirements: the number of participating parameters of the corresponding layer is consistent with the preset number, the role played by each participating parameter is greater than the preset role, and the connection and coordination relationship between the current participating parameters and the participating parameters of the adjacent layers meets the model operation setting standard;

[0221] If satisfied, the corresponding detection layer is judged to be running properly;

[0222] Otherwise, extracting the unsatisfied index from the judgment result, and judging the supporting behavior of the unsatisfied index to the initial detection model;

[0223] Based on the state fullness coefficient of the supporting behavior and the set weight of the corresponding unsatisfied index based on the initial detection model, the coefficient to be optimized of the corresponding supporting behavior is obtained;

[0224] ;

[0225] in, represents the coefficient to be optimized for the j1th supporting behavior in the corresponding detection layer; represents the normalized function of the j1th support behavior in the corresponding detection layer; represents the set weight of the unsatisfied index in the corresponding detection layer that is consistent with the j1-th supporting behavior; represents the state fullness coefficient under the j1th supporting behavior in the corresponding detection layer; Nc represents the preset number of parameters planned for the corresponding detection layer; Represents the weight sum of all indicators in the corresponding detection layer;

[0226] Function fitting is performed on all coefficients to be optimized to obtain the final optimization factor, and based on the comparison relationship between the final optimization factor and each supporting behavior, the optimization parameters based on the corresponding detection layer are obtained and the model is optimized.

[0227] In this embodiment, the log generated by the process of the model analyzing the sample can be directly captured based on the background capture tool. Therefore, the current participation parameters are obtained by extracting the parameters in the log. For example, the model has 3 layers, and there are participation parameters 1, 2 and 3 in detection layer 1, but the preset number is 4. At this time, it is considered that the number is inconsistent. If the preset number is 3, then it is considered that the number is consistent.

[0228] In this embodiment, the role played refers to the role played by the corresponding parameters in the process of analyzing the sample, and the preset role is pre-set. For example, the role played was originally to perform image enhancement processing on micro-images, but image enhancement processing was not performed in the actual analysis process. At this time, the role played is less than the preset role.

[0229] In this embodiment, if there is a next layer based on the detection layer, the participation parameter of the adjacent layer refers to the next layer; if there is no next layer based on the detection layer, the participation parameter of the adjacent layer refers to the previous layer.

[0230] In this embodiment, the connection and coordination relationship refers to the actual relationship between the mutual coordination of parameters. The model operation setting standard is pre-set, that is, how different detection layers analyze the image is planned, and then the actual coordination relationship is captured for comparison with the standard plan. If the two are consistent, they are considered to be satisfied.

[0231] In this embodiment, there are three judgment conditions when judging whether the current participation parameters are satisfied. Each judgment condition can be regarded as an existing indicator. Not satisfying the indicator means not satisfying the corresponding judgment condition.

[0232] In this embodiment, supporting behavior refers to the behavior of implementing the judgment conditions. For example, the number of participants is 3 and the preset number is 4. The state saturation coefficient at this time is 3 / 4, that is, the role played is 80%. At this time, it is not greater than the preset role, and then the state saturation coefficient is 80% / 100%, that is, the connection and coordination relationship is only 10% away from meeting the standard. At this time, the state saturation coefficient is: (percentage of the set standard - 10%) / percentage of the set standard.

[0233] In this embodiment, the weights are all preset, and the sum of the weights of different indicators in all detection layers is 1.

[0234] In this embodiment, function fitting refers to performing linear fitting on the coefficients, and then obtaining the average value of the fitting line as the final optimization factor.

[0235] In this embodiment, the control relationship refers to the numerical conversion (standardization) of the final optimization factor to ensure adaptation with the initial parameters, and then optimize the model. For example, the parameter A1 in the detection layer 1 is converted from the original value a1 to the value a2 under the action of the final optimization factor and the control relationship.

[0236] The beneficial effect of the above technical solution is: by capturing the parameters in the analysis process of the verification sample to determine the satisfied situation, to analyze and judge the unsatisfied indicators to obtain the coefficients to be optimized, and then the function fitting and control relationship ensure the reasonable adjustment of the parameters, thereby achieving reliable optimization of the model and further ensuring the accuracy of image fusion detection.

[0237] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A method for detecting targets by fusion of visible light and thermal infrared images from the visual angle of an unmanned aerial vehicle, characterized in that: include: Step 1: Preprocess the dataset of visible light and thermal infrared images of the detection area acquired by the drone, and perform data enhancement on the preprocessed images to obtain several initial images; Step 2: hierarchically abstract and combine the image features of the initial image at different scales based on a multi-scale dynamic feature pyramid network to lock tiny targets and long-distance targets in the initial image; Step 3: Based on the attention fusion mechanism, the initial image in the visible light mode is fused with the initial image in the thermal infrared mode, and the candidate area is locked, and then the small targets, long-distance targets and candidate areas are highlighted to obtain several target samples; Step 4: Split all target samples into a training set and a validation set, and train a preset network model based on the training set to obtain an initial detection model, and perform model performance evaluation and model optimization on the initial detection model based on the validation set to obtain a target detection model; Wherein, locking the tiny target and the long-distance target in the initial image comprises: Based on the execution logic of each feature layer in the structural design of the multi-scale dynamic feature pyramid network, hierarchical feature extraction is performed on different initial images; Based on a dynamic weighting mechanism, each hierarchical feature is assigned a channel weight, and all hierarchical features are fused to detect tiny targets and long-distance targets in the initial image; The process of fusing the initial image in the visible light mode with the initial image in the thermal infrared mode based on the attention fusion mechanism includes: Applying the initial image in the visible light mode and the initial image in the thermal infrared mode to the self-attention fusion mechanism respectively, obtaining the autocorrelation weights in different modes and constructing an attention matrix, wherein the attention matrix represents the correlation weights in different viewing angles of different modes; Applying the initial image in the visible light modality and the initial image in the thermal infrared modality to a cross-modal attention fusion mechanism, obtaining the cross-correlation weights between different modalities and constructing a weight matrix, wherein the weight matrix represents the complementarity of the visible light modality and the thermal infrared modality at the viewing angle position; Based on the attention matrix and the weight matrix, the detection requirements under different viewing angles and postures during the flight of the drone are met, and the detection of the current target in the initial image is achieved by combining the attention weights.

2. The visible light and thermal infrared image target fusion detection method of the UAV visual angle according to claim 1 is characterized in that: The preprocessing includes: image normalization processing and image denoising processing.

3. The visible light and thermal infrared image target fusion detection method of the UAV visual angle according to claim 1 is characterized in that: Perform data augmentation on the preprocessed images, including: Determine the drone shooting angle and shooting posture of each pre-processed image to obtain a shooting array; Adaptively adjusting the photographing array according to a rotation angle of 1° to obtain a plurality of adaptive images corresponding to the preprocessed image; Determine the data richness of each preprocessed image separately; ； in, represents the richness of the extracted features of the i1th preprocessed image; represents the area occupied by the target of interest in the i1th pre-processed image determined under the corresponding shooting angle and shooting posture; represents the area occupied by the target of interest determined after adjusting the i1th pre-processed image based on the reset shooting angle and shooting posture; ln represents the logarithmic function symbol; according to Determine a first enhancement quantity, wherein F0 represents a preset richness; Indicates the clarity of the i1th preprocessed image; Q0 indicates the preset clarity; Indicates that the i1th preprocessed image is based on the drone shooting perspective Shooting angle of view with theoretical level The number to be supplemented due to the visual difference; ceiling indicates the symbol for rounding up; Each preprocessed image is divided into unit blocks to obtain a block set; Performing data enhancement processing on each unit block in the block set, wherein the data enhancement processing is determined according to the number of block feature coefficients in the block set that are greater than the effective coefficients; Determining a second enhancement quantity of the corresponding preprocessed image according to a data enhancement processing result of each unit block in the corresponding preprocessed image; sorting the image richness of all preprocessed images, and assigning a shooting weight to each shooting array according to the image richness; Matching a third enhancement quantity consistent with the shooting weight from a weight-quantity comparison table; Determine the corresponding final enhancement number according to max(Dn1, Dn2, Dn3), and randomly select the same number of first images from the multiple adaptive images under the same preprocessed image and continue to perform adaptive adjustment according to the rotation angle of 1° to obtain multiple adaptive images of each first image, where Dn1 represents the first enhancement number corresponding to the same preprocessed image, Dn2 represents the second enhancement number corresponding to the same preprocessed image, and Dn3 represents the third enhancement number corresponding to the same preprocessed image; Determine the shooting environment of the drone when determining each pre-processed image, and perform index analysis on the shooting environment to construct an environment vector; Element-adjusting the environment vector to obtain a new vector, simulating a new environment of the new vector, and fusing the new vector with each corresponding first image to obtain a second image; Among them, all adaptive images and the second image are used as initial images.

4. The visible light and thermal infrared image target fusion detection method of the UAV visual angle according to claim 1 is characterized in that: The process of detecting the target in the initial image by combining the attention weights includes: Obtain the important weights of the convolution kernels involved in the detection process of each current target, and remove the convolution kernels whose important weights are less than the preset weights; The response values ​​of the channels involved in the detection process of each current target are obtained, and the channels with response values ​​less than the preset threshold are eliminated.

5. The visible light and thermal infrared image target fusion detection method of the UAV visual angle according to claim 1 is characterized in that: The highlighting process refers to enhancing the details of small targets, distant targets and candidate areas.

6. The visible light and thermal infrared image target fusion detection method of the UAV visual angle according to claim 1 is characterized in that: Based on the validation set, the initial detection model is evaluated for model performance and optimized to obtain a target detection model, including: Capturing the current participating parameters of each detection layer during the analysis of each verification sample in the verification set by the initial detection model, and judging whether the current participating parameters meet the following requirements: the number of participating parameters of the corresponding layer is consistent with the preset number, the role played by each participating parameter is greater than the preset role, and the connection and coordination relationship between the current participating parameters and the participating parameters of the adjacent layers meets the model operation setting standard; If satisfied, the corresponding detection layer is judged to be running properly; Otherwise, extracting the unsatisfied index from the judgment result, and judging the supporting behavior of the unsatisfied index to the initial detection model; Based on the state fullness coefficient of the supporting behavior and the set weight of the corresponding unsatisfied index based on the initial detection model, the coefficient to be optimized of the corresponding supporting behavior is obtained; ； in, represents the coefficient to be optimized for the j1th supporting behavior in the corresponding detection layer; represents the normalized function of the j1th support behavior in the corresponding detection layer; represents the set weight of the unsatisfied index in the corresponding detection layer that is consistent with the j1-th supporting behavior; represents the state fullness coefficient under the j1th supporting behavior in the corresponding detection layer; Nc represents the preset number of parameters planned for the corresponding detection layer; Qc represents the weight sum of all indicators in the corresponding detection layer; Function fitting is performed on all coefficients to be optimized to obtain the final optimization factor, and based on the comparison relationship between the final optimization factor and each supporting behavior, the optimization parameters based on the corresponding detection layer are obtained and the model is optimized.

Images