A multi-modal aerial image fusion and target detection joint learning method

Abstract

The application discloses a kind of multi-modal aerial image fusion and target detection joint learning method, belong to remote sensing image processing technical field, comprising: using paired visible light image and infrared image preliminary training image fusion branch, generate first fusion image;Based on first fusion image, design target detection branch with expert guidance information;Using first fusion image, target detection branch is preliminarily trained;Through the output expert feature map of target detection branch after preliminary training;Expert feature map is processed by feature alignment;And using the expert feature map after feature alignment to the image fusion branch is fine-tuned, generate second fusion image;Using second fusion image, target detection branch is fine-tuned, realizes optimization target detection task;The method simultaneously improves the performance of visible light and infrared aerial image fusion and target detection two tasks, provides more accurate, efficient data analysis and decision support for unmanned aerial vehicle application.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of remote sensing image processing, and more particularly relates to a multi-modal aerial image fusion and target detection joint learning method. BACKGROUND

[0002] In recent years, multi-modal aerial images taken by unmanned aerial vehicles (UAVs) have received increasing attention and can be applied to many fields such as environmental investigation, urban planning, and disaster rescue. Among them, the fusion of visible light and infrared aerial images and target detection are two important tasks in the application of unmanned aerial vehicles.

[0003] Visible light images capture the reflected light of a scene and present rich texture details, but are often affected by light conditions. In contrast, infrared images have strong anti-interference ability and can capture thermal radiation information, although they are suitable for various complex environments, but lack detailed texture information. The fusion of visible light and infrared images takes advantage of the complementary information between the two modalities to generate a fusion image containing more information, which can improve the performance of other high-level tasks. The target detection task can use the generated high-quality fusion image and combine it with deep learning-based image processing algorithms to quickly and accurately detect target objects.

[0004] Existing multi-modal image fusion and target detection methods mostly improve the fusion effect by designing networks and introducing constraint conditions, ignoring the potential benefits of the target detection network. In 2023, Zhao Wenda et al. designed a meta-feature embedding model to enable the features of the target detection network to guide the visible light and infrared image fusion network. However, this method uses an internal and external two-stage updating method, which complicates the training process, and only focuses on simple natural images without considering complex aerial images. Unlike natural images, the field of view of a UAV is wide, the targets in the images are small and dense, and the background noise is complex, which increases the difficulty of target detection.

[0005] The progress of large language base model GPT-4 has sparked significant attention to the development of base models in the field of computer vision. Among them, segmentation base models such as SAM (Segment Anything Model) and MobileSAM are novel interactive models designed specifically for image segmentation tasks and subsequent downstream applications, which also provide a new solution to the problem of complex background, small target size, and poor fusion and detection effect of multi-modal aerial images.

[0006] Therefore, how to use the segmentation base model to enhance the detection ability of the target detection branch to small targets in aerial images, and at the same time improve the performance of visible light and infrared aerial image fusion and target detection, and provide more accurate and efficient data analysis and decision support for unmanned aerial vehicle application, is a problem that those skilled in the art need to solve. SUMMARY

[0007] In view of the above problems, the present application provides a multi-modal aerial image fusion and target detection joint learning method to at least solve some of the technical problems mentioned in the background art.

[0008] In order to achieve the above purpose, the present application adopts the following technical solutions:

[0009] A multi-modal aerial image fusion and target detection joint learning method, comprising the following steps

[0010] S1, using a pair of visible light images and infrared images to preliminarily train an image fusion branch to generate a first fusion image;

[0011] S2, based on the first fusion image, designing a target detection branch with expert guidance information;

[0012] S3, using the first fusion image to preliminarily train the target detection branch; the expert feature map output by the preliminarily trained target detection branch;

[0013] S4, performing feature alignment processing on the expert feature map; and using the expert feature map after feature alignment processing to fine-tune the image fusion branch to generate a second fusion image;

[0014] S5, using the second fusion image to fine-tune the target detection branch to realize the optimization of the target detection task.

[0015] Further, the step S2 specifically comprises:

[0016] using a segmentation base model to segment the first fusion image into a plurality of image blocks;

[0017] According to a preset area interval and a segmentation coding module, the plurality of image blocks are classified and coded, and a hybrid expert gate mechanism is used to adaptively learn the coded features to form a target detection branch with expert guidance information.

[0018] Further, the preset area interval setting process comprises:

[0019] The target real box in the pair of visible light images and infrared images is normalized respectively, and the area of the normalized target real box is calculated, and one of the areas is selected as the first cluster center;

[0020] The K-Means++ clustering algorithm is used to calculate the shortest Euclidean distance between the area of the other target real frame and the first cluster center; the area of the other target real frame is the area of the target real frame other than the first cluster center;

[0021] According to the shortest Euclidean distance, the probability of the area of each target real frame being selected as the next cluster center is calculated until K clusters are clustered, obtaining K area intervals; the K area intervals are used as the preset area interval.

[0022] Further, the segmentation encoding module specifically includes:

[0023] The areas of the minimum circumscribed rectangles of the plurality of image blocks are calculated respectively;

[0024] The plurality of minimum circumscribed rectangle areas are divided into K classes according to the preset area interval, and the target area of each class is set to 1 and the other areas are set to 0, obtaining a K-channel Mask matrix;

[0025] The Mask matrix is flattened and mapped to a fixed dimension; based on the Patch embedding encoding feature and the Position embedding encoding feature, a Transformer encoder is used for self-attention operation to obtain a feature map;

[0026] The feature map is down-sampled by a convolution module to obtain a down-sampled feature map.

[0027] Further, the size of the down-sampled feature map is the same as the size of the feature map output by the target detection branch.

[0028] Further, the adaptive learning of the encoding feature using the hybrid expert gate mechanism specifically includes:

[0029] First, the feature map output by the target detection branch and the down-sampled feature map are spliced along the channel dimension; then the spliced feature map is processed through a gate network to generate a weight; finally, the two feature maps and the corresponding weight are linearly combined to generate an expert feature map.

[0030] Further, the step S4 specifically includes:

[0031] The expert feature map is subjected to feature alignment processing; and a first loss function is constructed based on the expert feature map after the feature alignment processing;

[0032] The first loss function and the second loss function of the image fusion branch are linearly combined to obtain an image fusion loss function;

[0033] The image fusion loss function is optimized through back propagation until an optimal image fusion branch is obtained; and a second fused image is generated based on the optimal image fusion branch.

[0034] Compared with the prior art, the multi-modal aerial image fusion and target detection joint learning method provided by the present application has the following beneficial effects:

[0035] (1) The joint learning method adopted by the present application not only improves the fusion quality of visible light and infrared images and improves the fusion effect, but also improves the target detection performance of aerial images, realizing the cooperative optimization of the two tasks.

[0036] (2) The present application focuses on the target detection task of multi-modal aerial images, fully utilizes the advantage that the segmentation-based model can segment everything, and provides expert guidance information for the target detection branch by constructing a Mask matrix through a clustering algorithm, thereby enhancing the ability of the detector to detect small targets in complex working environments of unmanned aerial vehicles.

[0037] (3) The present application adopts a mixed expert gate mechanism to model the guidance information provided by the segmentation-based model, which can dynamically select the degree of dependence of each expert information and effectively suppress the complex background noise in aerial images, so that the detector can focus more on reliable target objects. BRIEF DESCRIPTION OF DRAWINGS

[0038] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced below. Obviously, the drawings in the following description are only embodiments of the present application, and those skilled in the art can obtain other drawings according to the provided drawings without creative labor.

[0039] Figure 1 The multi-modal aerial image fusion and target detection joint learning method provided by the present application is shown in the flowchart.

[0040] Figure 2 The multi-modal aerial image fusion and target detection joint learning method provided by the present application is shown in the framework diagram.

[0041] Figure 3 The schematic diagram of SAM guiding the target detection branch provided by the present application is shown in the schematic diagram.

[0042] Figure 4 The image block segmented by SAM and the corresponding 3-channel visualized Mask matrix provided by the present application are shown in the schematic diagram.

[0043] Figure 5The expert feature map of the target detection branch provided by the embodiment of the present application refines the image fusion branch through feature alignment.

[0044] Figure 6 The fusion image generated by the post-joint optimization image fusion branch provided by the embodiment of the present application is shown in the schematic diagram.

[0045] Figure 7 The detection effect diagram of the post-joint optimization target detection branch provided by the embodiment of the present application is shown in the schematic diagram. DETAILED DESCRIPTION

[0046] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of the present application.

[0047] Referring to Figure 1 The embodiment of the present application discloses a multi-modal aerial image fusion and target detection joint learning method, which is implemented based on a segmentation-based model SAM, and the implementation process includes the following steps:

[0048] S1, a pair of visible light images and infrared images are used to preliminarily train an image fusion branch to generate a first fusion image;

[0049] S2, based on the first fusion image, a target detection branch with expert guidance information is designed;

[0050] S3, the first fusion image is used to preliminarily train the target detection branch; and the expert feature map is output by the post-preliminary training target detection branch;

[0051] S4, the expert feature map is processed through feature alignment; and the image fusion branch is refined by using the expert feature map processed through feature alignment to generate a second fusion image;

[0052] S5, the second fusion image is used to refine the target detection branch to realize the optimization of the target detection task.

[0053] Next, each of the above steps will be described.

[0054] In the above step S1, referring to Figure 2As shown, the image fusion branch includes a plurality of feature fusion modules and an image reconstruction module; in the training process, first, input a pair of visible light images and infrared images into the plurality of feature fusion modules, and sequentially extract image features of the visible light images and the infrared images by the plurality of feature fusion modules respectively, and perform image feature fusion processing to obtain image fusion features; then, an image reconstruction module is used to construct a fusion image based on the obtained image fusion features, that is, a first fusion image;

[0055] In the embodiment of the present application, the image fusion branch includes 3 feature fusion modules and 1 image reconstruction module; each feature fusion module is composed of 2 3x3 convolutions and a Relu activation function, and the features are fused between the modules by splicing; the image reconstruction module is composed of 6 3x3 convolutions and a Relu activation function.

[0056] In the above step S2, referring to Figure 3 As shown, the first fusion image generated in step S1 is segmented into a plurality of image blocks using a segmentation base model; the plurality of image blocks are classified and coded according to a preset area interval and a segmentation coding module, the coding features are adaptively learned using a hybrid expert gate mechanism, and a target detection branch with expert guidance information is formed;

[0057] The setting process of the above-mentioned preset area interval includes:

[0058] (1) The target real boxes in the pair of visible light images and infrared images are normalized respectively; represented by the formula as:

[0059]

[0060]

[0061] Wherein, w represents the width of the target real box; h represents the height of the target real box; W represents the width of the visible light image or the infrared image; H represents the height of the visible light image or the infrared image;

[0062] Then, the area of the normalized target real box is calculated, and one of the areas is selected as the first cluster center;

[0063] (2) The K-Means++ clustering algorithm is used to calculate the shortest Euclidean distance d(s,c) between the area of the other target real box and the first cluster center; the area of the other target real box is the area of the target real box except the first cluster center; the shortest Euclidean distance d(s,c) is represented as:

[0064]

[0065] Wherein, s n And c nrepresents an area vector of the target real box in the n-th dimension; N represents the dimension of the vector;

[0066] (3) According to the shortest Euclidean distance, the area of each target real box is selected as the probability P(s) of the next clustering center until K clusters are clustered, and K area intervals are obtained; the K area intervals are taken as the preset area interval;

[0067] The probability P(s) selected as the next clustering center is represented as:

[0068]

[0069] Wherein, c' represents other clustering centers.

[0070] The above segmentation encoding module specifically includes:

[0071] (1) The areas of the minimum circumscribed rectangles of the plurality of image blocks are calculated respectively; the plurality of minimum circumscribed rectangle areas are divided into K classes according to the preset area interval, and then the target area of each class is set to 1 and the other areas are set to 0, to obtain a K-channel Mask matrix; Figure 4 The SAM segmented image block and the corresponding 3-channel visual Mask matrix;

[0072] (2) The above Mask matrix is flattened and mapped to a fixed dimension; based on the Patch embedding encoding feature and the Position embedding encoding feature, a Transformer encoder is used for self-attention operation Attention to obtain a feature map; represented as:

[0073] K=LN(Reshape(k,R1)·W) (5)

[0074] V=LN(Reshape(v,R2)·W) (6)

[0075]

[0076] Wherein, Q represents a query; k represents a key; v represents a key value; LN represents layer normalization; R1 and R2 both represent a reduction ratio; W represents linear mapping; d represents a scale factor.

[0077] (3) A convolution module is used to downsample the feature map to obtain a downsampled feature map; the size of the downsampled feature map is the same as the size of the feature map output by the target detection branch.

[0078] The above uses a hybrid expert gating mechanism to adaptively learn the encoding feature, which specifically includes:

[0079] First, the feature map output from the object detection branch and the downsampled feature map are concatenated along the channel dimension; then, the concatenated feature map is processed by a gating unit to generate weights; finally, the two feature maps and their corresponding weights are combined linearly with a weighted average to generate an expert feature map; the entire process is represented as follows:

[0080] f i =concatenate(F di ,F si (8)

[0081] ω i =g(flatten(f) i (9)

[0082] F mi =ω i ·F di +(1-ω i )F si (10)

[0083] Where i = 1, 2, 3, 4, F di F represents the feature map output by the object detection branch. si The feature map is represented by the downsampled feature map; g represents the gated unit function; f i This represents the concatenated feature map; w i F represents the weight; mi Represents expert feature maps.

[0084] In step S3 above, see Figure 2 As shown, the object detection branch includes a backbone network (using ResNet50), a feature pyramid network, a hybrid expert gating system, and a detection head. During training, the first fused image is input into the backbone network. The backbone network extracts image features of the first fused image at different scales. Then, the feature pyramid network fuses the image features at different scales to obtain a high-resolution, semantically strong fused feature map. This fused feature map is input into the hybrid expert gating network, and object classification and regression are achieved through the detection head in conjunction with expert guidance information.

[0085] The object detection branch also includes SAM (Segment Anything Model) and a segmentation coding module. The first fused image is input into SAM and segmented into multiple image blocks. The segmentation coding module encodes the segmented image blocks to further enhance the semantic effect of the feature maps. Then, the feature maps output by the segmentation coding module are downsampled and input into the hybrid expert gating network to provide expert guidance information for the object detection branch.

[0086] In step S4, the features of the image fusion task and the target detection task are not compatible due to the difference between the two tasks, and the features output by the target detection branch cannot be directly used to assist the image fusion branch, and the feature alignment module can match the features of the two tasks; in the embodiment of the application, the expert feature map is processed by the feature alignment module; and based on the expert feature map processed by the feature alignment, a first loss function L g is constructed to fine-tune the image fusion branch to generate more target semantic information, as shown in Figure 5 ;

[0087] Step S4 specifically includes:

[0088] Based on the expert feature map output by the target detection branch after preliminary training, a first loss function L g is constructed; the first loss function L g is a SmoothL1 loss function; and is expressed as:

[0089]

[0090] Wherein, F ui represents the feature output by the feature alignment module; F vi represents the feature of the image fusion branch;

[0091] A second loss function L f of the image fusion branch is obtained.

[0092] The first loss function and the second loss function are linearly combined to obtain an image fusion loss function L.

[0093] The image fusion loss function L is continuously optimized through back propagation until an optimal image fusion branch is obtained; and is expressed as:

[0094] L=L f +λL g (12)

[0095] L f =θ1·(1-SSIM(I f ,I1))+θ2·(1-SSIM(I f ,I2)) (13)

[0096]

[0097]

[0098] Wherein, λ, θ1 and θ2 represent weight ratios; SSIM represents a structural similarity loss; I1 represents a visible light image; I2 represents an infrared image; and I frepresents the average pixel of the first fusion image or the second fusion image; μ1 represents the average pixel of the visible light image; μ2 represents the average pixel of the infrared image; μ f represents the average pixel of the first fusion image or the second fusion image; σ1 represents the standard deviation of the pixel of the visible light image; σ2 represents the standard deviation of the pixel of the infrared image; σ f represents the standard deviation of the pixel of the first fusion image or the second fusion image; σ f1 represents the pixel covariance of the fusion image and the visible light image; σ f2 is the pixel covariance of the fusion image and the infrared image; wherein the fusion image is the first fusion image or the second fusion image; k1 and k2 both represent constants.

[0099] After fine-tuning of the image fusion branch, the second fusion image is generated; since the image fusion branch has been optimized, the quality of the generated second fusion image is higher, and specific details can be seen from Figure 6 .

[0100] In the above step S5, the second fusion image is used to further fine-tune the target detection branch, and the optimized target detection task is realized, and specific target detection effects are as shown in Figure 7 . Figure 7 It can be seen that the fine-tuned target detection branch can accurately distinguish easily confused target categories, and even in the case of insufficient light, small targets can still be recognized.

[0101] In summary, the embodiment of the present application provides a multi-modal aerial image fusion and target detection joint learning method, which uses the expert guidance information provided by the segmentation base model to enhance the detection ability of the target detection branch to small targets in aerial images, and uses the guided target detection branch to assist the image fusion branch to generate more target semantic information. This method can simultaneously improve the performance of the visible light and infrared aerial image fusion and target detection two tasks, and provides more accurate and efficient data analysis and decision support for unmanned aerial vehicle applications.

[0102] The embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple, and the related parts can be referred to the method part.

[0103] The foregoing description of the disclosed embodiments enables a person skilled in the art to make or use the application. Modifications of these embodiments will occur to persons of skill in the art, and that the appended claims are intended to cover all such modifications that do not depart from the true spirit and scope of the application. Therefore, the application is not limited to the embodiments shown but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A joint learning method for multimodal aerial image fusion and target detection, characterized in that, Includes the following steps S1. Initially train the image fusion branch using paired visible light and infrared images to generate the first fused image; S2. Based on the first fused image, design a target detection branch with expert guidance information; S3. The target detection branch is initially trained using the first fused image; After initial training, the target detection branch outputs an expert feature map. S4. Perform feature alignment processing on the expert feature map; The image fusion branch is then fine-tuned using an expert feature map that has undergone feature alignment processing to generate a second fused image; S5. Fine-tune the target detection branch using the second fused image to optimize the target detection task; Step S2 specifically includes: The first fused image is segmented into multiple image patches using a segmentation base model; The image blocks are classified and encoded according to the preset area range and the segmentation coding module. The hybrid expert gating mechanism is used to adaptively learn the coding features to form a target detection branch with expert guidance information. The process of setting the preset area range includes: The target bounding boxes in the paired visible light image and infrared image are normalized respectively, and the area of ​​the target bounding box after normalization is calculated. One of the areas is selected as the first cluster center. The K-Means++ clustering algorithm is used to calculate the shortest Euclidean distance between the area of ​​other ground truth boxes and the first cluster center; the area of ​​other ground truth boxes is the area of ​​the ground truth boxes excluding the first cluster center. Based on the shortest Euclidean distance, calculate the probability that the area of ​​each target ground truth box is selected as the next cluster center, until K clusters are formed, resulting in K area intervals; use the K area intervals as the preset area intervals; The segmentation and encoding module specifically includes: Calculate the area of ​​the minimum bounding rectangle for each of the image blocks; The areas of the multiple minimum bounding rectangles are divided into K categories according to the preset area range. The target area of ​​each category is set to 1, and the other areas are set to 0, to obtain a K-channel Mask matrix. The Mask matrix is ​​flattened and mapped to a fixed dimension; based on the Patch embedding encoded features and Position embedding encoded features, a Transformer encoder is used to perform self-attention operation to obtain the feature map; The feature map is downsampled using a convolution module to obtain a downsampled feature map.

2. The multimodal aerial image fusion and target detection joint learning method according to claim 1, characterized in that, The size of the feature map after downsampling is the same as the size of the feature map output by the target detection branch.

3. The multimodal aerial image fusion and target detection joint learning method according to claim 1, characterized in that, The adaptive learning of encoded features using a hybrid expert gating mechanism specifically includes: First, the feature map output by the object detection branch and the downsampled feature map are concatenated along the channel dimension; then, the concatenated feature map is passed through a gating network to generate weights; finally, the two feature maps and their corresponding weights are combined linearly to generate an expert feature map.

4. The multimodal aerial image fusion and target detection joint learning method according to claim 1, characterized in that, Step S4 specifically includes: The expert feature map is subjected to feature alignment processing; and a first loss function is constructed based on the feature-aligned expert feature map. The first loss function and the second loss function of the image fusion branch are linearly combined to obtain the image fusion loss function; The image fusion loss function is optimized through backpropagation until the optimal image fusion branch is obtained; a second fused image is generated based on the optimal image fusion branch.

Images