An adaptive object detection method based on semantically guided image fusion

Abstract

The present invention discloses an adaptive target detection method based on semantic guided image fusion, a pedestrian visible light image I V Input the visible light detection branch to obtain the visible light detection loss, and optimize the visible light detection branch; Pedestrian infrared image I I Input the infrared light detection branch to obtain the infrared light detection loss, and optimize the infrared light detection branch; pedestrian visible light image I V and pedestrian infrared image I I After inputting the fusion branch, the fused image I is obtained F The fused image is input into the visible light detection branch and the infrared light detection branch to obtain the detection drive loss. The fusion loss and the detection drive loss obtained in the above training process jointly optimize the fusion branch; the fused image is input into the fixed detection network to obtain the detection result: the fused image obtained by the image fusion training network inference guided by semantic features is input into a fixed detection network to obtain the final detection result. The present invention can effectively improve the accuracy of target detection.

Description

Technical Field

[0001] The present invention relates to the technical field of image detection, and in particular to an adaptive target detection method based on semantically guided image fusion. Background Art

[0002] In the era of information explosion, the rapid development of sensor technology has led to an increasing number of sensor types, and the captured information has also shown a diversified trend. Due to the differences in their imaging principles, different types of sensors have significant differences in the images they capture. Therefore, the images obtained by a single sensor optoelectronic imaging system often cannot fully and accurately describe the entire scene, and the accuracy of the target detection results of the acquired images is also affected. In order to solve the problem of incomplete time, space and spectral information in a single sensor imaging system, image fusion technology came into being. Image fusion technology aims to integrate multiple image information from the same scene but different modalities, different locations or different times to generate a fused image that is rich in information, highly reliable and highly targeted. Through image fusion technology, the characteristics of multiple sensors can be comprehensively utilized to make up for their respective limitations, so as to provide more comprehensive and accurate image information, thereby improving the accuracy of target detection.

[0003] At present, the existing algorithms for multimodal fusion are mainly divided into two categories: traditional image fusion methods and deep learning-based image fusion methods. Traditional fusion methods measure the activity level of images in different feature domains to obtain the decomposed feature coefficients of the source image, and then execute specific fusion rules on these feature coefficients to generate a fused image; infrared and visible light image fusion methods based on deep learning are mainly classified into four categories: methods based on convolutional neural networks (CNN), methods based on auto-encoders (AE), methods based on generative adversarial networks (GAN), and methods based on transformers. However, the image fusion process is mainly constrained by the relationship between the generated results and the source image, and lacks the definition of the target and background in the image, which has a certain impact on the accuracy of downstream tasks, such as target detection tasks. Summary of the invention

[0004] In order to solve the above technical problems, the present invention proposes an adaptive target detection method based on semantically guided image fusion, in which the semantic information of the target is provided to the image fusion network through the target detection task, and an infrared and visible light image fusion framework with embedded target detection features is constructed. However, in addition to the semantic information of the target, the target detection feature may also contain some information related to the target detection task, which may have a positive effect on the image fusion task. In order to solve this problem, a sharing module is also designed to extract the semantic features in the detection process and highlight the target information by adjusting the target weight of the fused feature. By obtaining a fused image with the semantic information of the target for target detection, the accuracy of target detection can be effectively improved.

[0005] In order to achieve the above object, the technical solution of the present invention is as follows:

[0006] An adaptive target detection method based on semantic guided image fusion comprises the following steps:

[0007] The pedestrian visible light image I V Input the visible light detection branch to obtain the visible light detection feature F V and visible light detection loss L vdet , and based on the visible light detection loss L vdet Optimize the visible light detection branch to obtain the optimized visible light detection branch; Pedestrian infrared image I I Input the infrared light detection branch and obtain the infrared light detection feature F I and infrared light detection loss L Idet , and based on the infrared light detection loss L Idet Optimizing the infrared light detection branch to obtain an optimized infrared light detection branch;

[0008] The pedestrian visible light image I V and pedestrian infrared image I I Input the fusion branch to get the fused image I F and the fusion loss L fusion ; The fused image I F Input the visible light detection branch and infrared light detection branch respectively to obtain the detection drive loss L det , and optimize the fusion branch according to the fusion loss and the detection-driven loss to obtain the optimized fusion branch;

[0009] The visible light image of the pedestrian to be detected and the visible light image of the pedestrian to be detected are respectively input into the optimized infrared light detection branch and the optimized infrared light detection branch to obtain the detection results; the detection results are input into the optimized fusion branch to obtain the fused image; the fused image is input into the fixed detection network for recognition to obtain the target position and target category.

[0010] Preferably, the visible light detection branch and the infrared light detection branch have the same structure but do not share parameters.

[0011] Preferably, the backbone network of the visible light detection branch adopts ResNet-FPN, and the detection head adopts FasterR-CNN; the backbone network of the infrared light detection branch adopts ResNet-FPN, and the detection head adopts FasterR-CNN.

[0012] Preferably, the fusion branch includes two encoders, a shared module and a decoder, wherein the pedestrian visible light image I V and pedestrian infrared image I I Input two encoders respectively to obtain the detection feature F V and F I , and detect the feature F V and F I Add the elements to obtain the fusion feature F F ;

[0013] The feature F V 、F I and F F Input the shared module together to obtain the weighted fusion feature F′ F ; Decode the weighted fusion features through the decoder to obtain the fused image I F .

[0014] Preferably, the detection drive loss L det , defined as follows:

[0015] L det =L vdet +L Idet

[0016] Among them, the visible light detection loss L vdet and infrared light detection loss L Idet Both include classification loss and regression loss.

[0017] Taking the visible light detection branch as an example, the visible light detection loss L vdet Contains classification loss and regression loss as follows,

[0018]

[0019] Where p represents the predicted probability of each category, t u represents the predicted bounding box result, u represents the true category, and v represents the true bounding box regression target. The Iverson bracket indicator function [u≥1] is the cross entropy loss function. It takes the value of 1 when u≥1 and 0 otherwise. All background classes are marked as u=0. is the Smooth L1 loss function.

[0020] Preferably, the fixed detection network adopts the yolov8 network.

[0021] Based on the above technical solution, the beneficial effects of the present invention are as follows: the present invention provides an adaptive target detection method based on semantic guided image fusion, which transforms the pedestrian visible light image I V Input the visible light detection branch to obtain the visible light detection feature F V and visible light detection loss L vdet , and based on the visible light detection loss L vdet Optimize the visible light detection branch to obtain the optimized visible light detection branch; Pedestrian infrared image I I Input the infrared light detection branch and obtain the infrared light detection feature F I and infrared light detection loss L Idet , and based on the infrared light detection loss L Idet The infrared light detection branch is optimized to obtain an optimized infrared light detection branch; the pedestrian visible light image I V and pedestrian infrared image I I Input the fusion branch to get the fused image I F and the fusion loss L fusion ; The fused image I F Input the visible light detection branch and infrared light detection branch respectively to obtain the detection drive loss L det , and optimize the fusion branch according to the fusion loss and the detection-driven loss to obtain the optimized fusion branch; input the visible light image of the pedestrian to be detected and the visible light image of the pedestrian to be detected into the optimized infrared light detection branch and the optimized infrared light detection branch respectively to obtain the detection result; input the detection result into the optimized fusion branch to obtain the fused image; input the fused image into the fixed detection network for recognition to obtain the target position and target category. The present invention provides the semantic information of the target to the image fusion network through the target detection task, and constructs an infrared and visible light image fusion framework with embedded target detection features. However, in addition to the semantic information of the target, the target detection feature may also contain some information related to the target detection task, which may have a positive effect on the image fusion task. In order to solve this problem, a shared module is also designed to extract the semantic features in the detection process and highlight the target information by adjusting the target weight of the fusion feature. By obtaining a fused image with the semantic information of the target for target detection, the target detection accuracy can be effectively improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 is a flowchart of an adaptive target detection method based on semantic guided image fusion in an embodiment;

[0023] Figure 2 is a schematic diagram of the structure of an image fusion network in an adaptive target detection method based on semantically guided image fusion in an embodiment;

[0024] Figure 3 This is a schematic diagram of the structure of the yolov8 network. DETAILED DESCRIPTION

[0025] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0026] like Figure 1 The present embodiment provides an adaptive target detection method based on semantically guided image fusion, which uses an image fusion network (such as an image fusion network) based on semantic feature-guided infrared and visible light image fusion for pedestrian infrared and visible light images. Figure 2 As shown in Figure 1, the network structure of the image fusion network includes three branches: fusion branch, visible light detection branch and infrared light detection branch. It uses the object-related information learned in the object detection network to guide the fusion of multimodal images to obtain the fused image; the fused image is input into a fixed detection network to obtain the final detection result. Specifically, it includes the following steps:

[0027] Step 1: Transform the pedestrian visible light image I V Input the visible light detection branch to obtain the visible light detection feature F V and visible light detection loss L vdet , and based on the visible light detection loss L vdet Optimize the visible light detection branch to obtain the optimized visible light detection branch; Pedestrian infrared image I I Input the infrared light detection branch and obtain the infrared light detection feature F I and infrared light detection loss L Idet , and based on the infrared light detection loss L Idet The infrared light detection branch is optimized to obtain an optimized infrared light detection branch.

[0028] In this embodiment, the network structures of the two detection branches are the same and the parameters are not shared. Taking the visible light detection branch as an example, the following is explained:

[0029] Pedestrian visible light image I V Input into the visible light detection branch backbone network to obtain the visible light detection feature F V ;

[0030] The visible light detection branch backbone network uses ResNet-FPN to extract features;

[0031] Visible light detection feature F V After passing through the visible light detection branch detection head, the visible light detection loss L is obtained. vdet ;

[0032] The visible light detection branch detection head uses the detection head in FasterR-CNN to classify and predict the position of the target box, and obtains the visible light detection loss L vdet .

[0033] The infrared light detection branch has the same design as the visible light detection branch, but the parameters of the two detection branches are not shared, and they are measured through their own detection loss (L vdet or L Idet ) are optimized independently.

[0034] Step 2: Transform the pedestrian visible light image I V and pedestrian infrared image I I Input the fusion branch to get the fused image I F and the fusion loss L fusion ; The fused image I F Input the visible light detection branch and infrared light detection branch respectively to obtain the detection drive loss L det , and optimize the fusion branch according to the fusion loss and detection-driven loss to obtain the optimized fusion branch.

[0035] In this embodiment, the fusion branch includes two encoders, a shared module and a decoder. The encoder and decoder adopt the design of the encoder and decoder in DenseFuse. The specific processing process is as follows:

[0036] First, the pedestrian visible light image I V Grayscale and get I′ V , for the pedestrian infrared image I I Extract one of the channels and get I′ I ; The two encoders are respectively I′ V and I′ I As input to extract features of the two modalities;

[0037] Then concatenate the output features of the two encoders to obtain the fused feature F F (The fusion process is jointly optimized by the subsequent fusion loss and detection-driven loss);

[0038] The detection features F obtained by the two detection branches (visible light detection branch and infrared light detection branch) V and F I , the feature FV 、F I and F F Inputting the shared module together can obtain the weighted fusion feature F′ F , weighted fusion feature F′ F Can obtain information related to the target semantic features;

[0039] The shared module uses element-by-element multiplication to extract the uniqueness scores of the two modal features and performs upsampling operations with F F Keeping the same shape, the weighted features of the shared module are passed through the decoder to obtain the fused image I F .

[0040] Specifically, the shared module design is as follows:

[0041] After the infrared image and the visible light image are passed through their respective detection branch backbone networks, the detection features are obtained. After semantic extraction of the feature map, the corresponding score matrix is ​​obtained. The score matrices of both are four-dimensional. The four-dimensional tensor can be operated element by element, and the result of element-by-element multiplication will remain as a four-dimensional tensor. The element-by-element multiplication operation of the four-dimensional tensor is to multiply the elements in the corresponding positions, which is implemented in PyTorch using the '*' symbol.

[0042] For example, the score matrix corresponding to the infrared image is multiplied by the complementary matrix of the score matrix of the visible light image. The element-by-element multiplication of the two four-dimensional tensors can obtain a uniqueness score matrix, where high scores indicate that targets that can be detected in the infrared image cannot be detected in the visible light image.

[0043] Similarly, we get the score matrix of visible light images, where high scores indicate targets that can be detected in visible light images but not in infrared images. In this way, we get different detection targets between infrared images and visible light images, and then use the score weights to guide the fusion features obtained by the fusion branch to pay more attention to the different information between the two.

[0044] Specifically, the design for detecting drive loss is as follows:

[0045] The fused image is input into the infrared light detection branch and the visible light detection branch respectively, and the detection loss is calculated by comparing the respective prediction results with the true value. The detection loss of the fused image in the two detectors is expressed as L vdet and L Idet . Detection drive loss L det The definition is as follows:

[0046] L det =L vdet +L Idet

[0047] Taking the visible light detection branch as an example, L vdetContains classification loss and regression loss

[0048]

[0049] Where p represents the predicted probability of each category, t u Represents the predicted bounding box result, u represents the true category, and v represents the true bounding box regression target. It is a cross entropy loss function Iverson bracket indicator function, [u≥1] takes the value of 1 when u≥1, otherwise it is 0, and all background classes are marked as u=0. is the Smooth L1 loss function.

[0050] The detection-driven loss does not participate in the gradient back-propagation of the two detection branches. The fusion branch is optimized by jointly optimizing the detection-driven loss and the fusion loss, L′ fusion =L det +L fusion After training the fusion branch, the fused image is obtained by inference.

[0051] Step 3: Input the visible light image of the pedestrian to be detected and the visible light image of the pedestrian to be detected into the optimized infrared light detection branch and the optimized infrared light detection branch respectively to obtain the detection results; input the detection results into the optimized fusion branch to obtain the fused image; input the fused image into the fixed detection network for identification to obtain the target position and target category.

[0052] In this embodiment, the visible light image of the pedestrian to be detected and the visible light image of the pedestrian to be detected are respectively input into the optimized infrared light detection branch and the optimized infrared light detection branch to obtain the detection result; the detection result is input into the optimized fusion branch to obtain the fused image; the fused image is input into the fixed detection network to obtain the detection result. The fixed detection network adopts the yolov8 network, and its detection network structure is as follows Figure 3 shown.

[0053] Data preparation: Prepare training datasets and test datasets. The training dataset contains images with labeled boxes for training the model; the test dataset is used to evaluate the performance of the model.

[0054] Model construction: The yolov8 network uses Darknet as the basic network architecture. Darknet is a lightweight deep learning framework for building convolutional neural networks. The yolov8 network builds the network by stacking multiple convolutional layers and pooling layers.

[0055] Feature extraction: Based on the model construction, the yolov8 network uses convolutional layers to extract the features of the image. These features can capture information such as the shape and texture of different objects in the image.

[0056] Anchor box generation: In order to detect objects of different sizes, the Yolov8 network introduces the concept of anchor boxes. Anchor boxes are a set of predefined bounding boxes used to represent objects of different sizes and aspect ratios.

[0057] Target detection: After feature extraction and Anchor box generation, the yolov8 network uses convolutional layers and fully connected layers to predict whether each Anchor box contains a target object, and predicts the target category and the location of the bounding box.

[0058] experiment

[0059] The fusion performance evaluation index is calculated by comparing the input infrared image and visible light image group with the output fusion image. The implementation effect is shown in Table 1. Seafusion is the first network to use high-level visual tasks to drive image fusion. Tardal proposed a target perception dual adversarial learning network for fusion; Visiable and Infrared respectively represent the detection results obtained by using visible light images alone and infrared images alone as inputs to the Faster R-CNN network. EN is entropy. The larger the information entropy, the greater the amount of information contained in the image; PSNR is peak signal-to-noise ratio. The larger the PSNR value, the higher the quality of the reconstructed image.

[0060] The fused image is used as the input image data and passed through the same detection network to obtain the detection performance evaluation indicators, where Box_P: the precision of the detection box in target detection (Box-level Precision); Box_R: the recall rate of the detection box in target detection (Box-level Recall); Box_mAP50: the mean average precision (mean Average Precision) when the IOU threshold is 0.5; Box_mAP50-95: the average precision calculated in the range of IOU thresholds from 0.5 to 0.95.

[0061] Table 1

[0062]

[0063] From the analysis of detection indicators, it can be seen that the method of the present invention has higher detection frame precision and recall rate in terms of detection indicators; from the analysis of fusion indicators, the fusion branch design in the method of the present invention is better than the classic fusion algorithm in fusion performance.

[0064] It should be understood that, although the various steps in the above-mentioned flow chart are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the above-mentioned flow chart may include a plurality of sub-steps or a plurality of stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these sub-steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

Claims

1. An adaptive target detection method based on semantically guided image fusion, characterized in that: The steps include: The pedestrian visible light image I V Input the visible light detection branch to obtain the visible light detection feature F V and visible light detection loss L vdet , and based on the visible light detection loss L vdet Optimize the visible light detection branch to obtain the optimized visible light detection branch; Pedestrian infrared image I I Input the infrared light detection branch and obtain the infrared light detection feature F I and infrared light detection loss L Idet , and based on the infrared light detection loss L Idet Optimizing the infrared light detection branch to obtain an optimized infrared light detection branch; The pedestrian visible light image I V and pedestrian infrared image I I Input the fusion branch to get the fused image I F and the fusion loss L fusion ; The fused image I F Input the visible light detection branch and the infrared light detection branch respectively to obtain the detection drive loss L det , and optimize the fusion branch according to the fusion loss and the detection-driven loss to obtain an optimized fusion branch, wherein the fusion branch includes two encoders, a shared module and a decoder, wherein the pedestrian visible light image I V and pedestrian infrared image I I Input two encoders respectively to obtain the detection feature F V and F I , and detect the feature F V and F I Add the elements to obtain the fusion feature F F ; The feature F V 、F I and F F Input the shared module together to obtain the weighted fusion feature F′ F ; The weighted fusion features are decoded by the decoder to obtain the fused image I F ; The shared module uses element-by-element multiplication to extract the uniqueness scores of the two modal features and performs upsampling operations with F F Keeping the same shape, the weighted features of the shared module are passed through the decoder to obtain the fused image I F ; The visible light image of the pedestrian to be detected and the visible light image of the pedestrian to be detected are respectively input into the optimized infrared light detection branch and the optimized infrared light detection branch to obtain the detection result; the detection result is input into the optimized fusion branch to obtain the fused image; the fused image is input into the fixed detection network for recognition to obtain the target position and target category, where, The detection drive loss L det , defined as follows: L det =L vdet +L Idet Among them, the visible light detection loss L vdet and infrared light detection loss L Idet Both include classification loss and regression loss. Visible light detection loss L vdet Contains classification loss and regression loss as follows, Where p represents the predicted probability of each category, t u represents the predicted bounding box result, u represents the true category, and v represents the true bounding box regression target. The Iverson bracket indicator function [u≥1] is the cross entropy loss function. It takes the value of 1 when u≥1 and 0 otherwise. All background classes are marked as u=0. is the Smooth L1 loss function.

2. The adaptive target detection method based on semantic guided image fusion according to claim 1, characterized in that: The visible light detection branch and the infrared light detection branch have the same structure and do not share any parameters.

3. The adaptive target detection method based on semantic guided image fusion according to claim 2, characterized in that: The backbone network of the visible light detection branch adopts ResNet-FPN, and the detection head adopts Faster R-CNN; the backbone network of the infrared light detection branch adopts ResNet-FPN, and the detection head adopts Faster R-CNN.

4. The adaptive target detection method based on semantic guided image fusion according to claim 1, characterized in that: The fixed detection network adopts the yolov8 network.

Images