Target detection method based on edge information assisted dual-spectrum fusion
By introducing edge information-assisted dual-spectral fusion method in multi-spectral object detection, the edge attention module and MSE module extract and fusion features are used to extract and fusion characteristics, the problem of insufficient utilization of multi-mode information in low-light environments is solved, and the detection performance and robustness are improved sex.
Patent Information
- Application Number
- CN202410853466.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-06-28
- Publication Date
- 2024-11-01
AI Technical Summary
The existing multispectral object detection methods fail to fully utilize multimodal edge information in low-light environments, resulting in insufficient detection performance and lack of comprehensibility in the fusion results.
A dual-spectral fusion method based on edge information assistance was designed. By adding edge attention modules and fusion modules (MSE modules) to the infrared image network, a variety of modal information is used to perform mid- and late-stage convolution feature extraction and fusion, which enhances the Semantic consistency and feature fusion capability.
The integration capability and detection performance of multi-spectral information are improved, the robustness and accuracy of object detection are enhanced, and the noise is suppressed through edge feature extraction and weighted fusion of semantic information is suppressed, which improves the credibility of the detection results.
Smart Images

Figure CN118887501A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of target detection in low-light environments, and in particular to a target detection method based on edge information-assisted dual-spectrum fusion. Background Art
[0002] In practical applications, target detection under low lighting conditions faces many difficulties, including insufficient lighting, shadows, blur, etc., making it difficult for traditional single-modality detection methods to achieve ideal results in such complex environments. In order to overcome these challenges, multispectral target detection has become a solution that has attracted much attention.
[0003] The core idea of multispectral target detection is to make full use of sensor information in different bands, usually including visible light and thermal infrared images. These two modalities provide complementary information. Visible light is suitable for detection under normal lighting conditions, while thermal infrared is more adaptable to low-light environments or insufficient lighting. Therefore, multispectral fusion can improve the accuracy and robustness of target detection.
[0004] However, current methods have some problems in multimodal information fusion, mainly in ignoring information with special significance and importance, and lacking the understandability of the fusion results. To solve these problems, this innovative method proposes a strategy that combines multimodal edge information extraction and multispectral semantic fusion to more accurately guide the fusion of multimodal information, thereby improving the performance of target detection in low-light environments.
[0005] In recent years, progress has been made in the field of multispectral target detection, including the introduction of various fusion architectures and techniques. Studies have found that late fusion architectures are superior to early fusion and traditional methods of aggregating channel features. Mid-term fusion and fusion RPN also prove the superiority of mid-term fusion. Different studies have proposed a variety of methods, such as cross-modal interactive attention, recursive fusion and refinement modules, feature alignment modules, multi-label learning, bidirectional AdaGrad attention gates, etc., to improve the performance of multispectral target detection. However, current methods have not fully explored the edge information characteristics of the target in the modality and its fusion with multispectral semantic information, which may be a direction for future research.
[0006] In the algorithm models of most existing multimodal target detection methods, the visible thermal images as input are simply stacked, and the multimodal information therein is not fully utilized, which can easily lead to the unique edge information modality being ignored in the segmentation task and failing to play a more prominent role. Summary of the invention
[0007] To solve the above problems, the present invention discloses a target detection method based on edge information assisted dual-spectrum fusion, which is used to improve the consistency between the two modes and improve the ability to integrate multispectral information from these multiple modes.
[0008] A target detection method based on edge information assisted dual-spectrum fusion, comprising the following steps:
[0009] S1. Obtain paired infrared and visible light target images, perform random flipping data enhancement operations, complete data preprocessing, and divide the training set, validation set, and test set into proportions;
[0010] S2. Construct a multispectral target detection network structure that integrates multispectral semantics and edge attention information, namely, the MSENet network structure. The MSENet adds three stages, namely stage5-7, based on the resnet18 network. Then, an edge attention module is designed and added to the infrared image network to obtain the target edge information from the infrared spectrum. Secondly, a fusion module (MSE) is designed using three modal information including infrared, visible light, and edge information. A fusion module is added to each stage, and finally the fusion result is input into the detector part to obtain the final output result.
[0011] S3. Train MSENet using the infrared and visible light target images, adjust hyperparameters, i.e. use the adam optimizer and set the learning rate to 0.0001 to obtain the best target detection model.
[0012] Beneficial effects of the present invention:
[0013] 1. This method is used to improve the consistency between the two modes and improve the ability to integrate multispectral information from these multimodalities; the multispectral semantic relationship between the two views is analyzed. This relationship is modeled to generate a multispectral representation, and the deep fusion of visible and infrared information is guided by edge information.
[0014] 2. By performing infrared imaging of the edge information of the target, the detection performance can be improved. However, edge features are shallow features. Directly using the output features of the last convolution block will force the deep network to capture shallow edge features, thus affecting the edge feature extraction performance. In addition, using the middle layer can also bring rich edge information convolution features. Therefore, it is necessary to use the middle and late convolution layers to obtain richer features.
[0015] 3. The MSE module is an effective fusion mechanism that achieves multispectral modality semantic consistency under the guidance of edge attention assistance, and achieves the fusion of visible light and infrared features under the guidance of complementary modalities and edge feature information. The MSE module constructs feature vectors of the two modalities under the guidance of deep features with rich semantic information, calculates the similarity of deep semantics between modalities, improves the consistency between modalities, suppresses modality-specific noise, pays attention to multispectral feature information in the modality, and weights the semantic information of the fused target feature context. This fusion mechanism relies on multispectral semantic consistency, enhances the perception of target information, and supports more accurate and stable target detection by including three features. BRIEF DESCRIPTION OF THE DRAWINGS
[0016] Figure 1 This is the overall network structure diagram of the present invention.
[0017] Figure 2 The figure is a structural diagram of an MSE fusion module in one implementation mode.
[0018] Figure 3 A structural diagram of an edge convolution operation in an edge feature extraction module according to one implementation.
[0019] Figure 4 A structural diagram of an edge feature extraction module according to an implementation scheme.
[0020] Figure 5 A detection result diagram of an implementation method is shown. DETAILED DESCRIPTION
[0021] The present invention is further explained below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention. It should be noted that the words "front", "rear", "left", "right", "upper" and "lower" used in the following description refer to directions in the accompanying drawings, and the words "inner" and "outer" refer to directions toward or away from the geometric center of a specific component, respectively.
[0022] This embodiment proposes a target detection method based on edge information assisted dual-spectrum fusion, such as Figure 1 It is mainly composed of infrared visible light convolution layer structure, edge feature extraction module, fusion module MSE and target detector SSD.
[0023] This embodiment designs an edge feature extraction module. First, as shown in FIG3, edge convolution is used to extract features using two convolution operators in different directions. Figure 4 As shown in the figure, in order to enhance edge features, the edge feature extraction module refines the output features of each stage using 3×3 convolution.
[0024] In some embodiments, as shown in FIG3 , the computation process of using edge convolution in the edge feature extraction module is described. Its main structure consists of two operators, one for detecting horizontal edges and the other for detecting vertical edges, and weighting the positions of pixels to reduce edge blur.
[0025] In some embodiments, the edge convolution is performed by combining a given infrared feature map The horizontal and vertical brightness difference approximations can be obtained separately.
[0026] Preferably, Figure 3 The edge convolution operation shown includes: using two convolution operators To extract the edge information of the feature map, normalize the semantic information through the BN layer, and then use the L2 norm and sigmoid function to aggregate the horizontal and vertical spatial information, and obtain the attention value at the same time. Finally, after element-by-element multiplication, the attention value will be broadcasted accordingly to obtain the edge convolution The output result of the edge convolution is as follows:
[0027]
[0028] Where σ represents the sigmoid function, β represents batch normalization and It is element-wise multiplication.
[0029] In some embodiments, the edge feature extraction module, such as Figure 2 , 3 , as shown in 4. First, use 1×1 convolution to map the feature map obtained by edge convolution into a three-channel feature map, and obtain the output feature Then, the lower branch undergoes two 3×3 convolutions and BN, ReLU structure for further feature extraction, and is cancat-joined with the original feature map to enrich the edge feature information. Finally, the 3×3 convolution retains the modality-specific information while integrating edge feature information to generate the edge attention map described in this invention.
[0030] In some embodiments, the proposed MSE module is shown in FIG2. Formally, let and Represent two sets of multimodal feature maps representing visible light and thermal images, respectively.
[0031] In order to estimate the semantic similarity between target features of different forms through the aid of edge features, the feature fusion module of the present invention proposes a cross-view feature representation:
[0032] W r =fMSE (U r ,U t ,U e )
[0033] W t =f MSE (U t ,U r ,U e )
[0034] Among them, f MSE Represents the proposed MSE module, and the final output is W r and W t , whose format is the same as the input format same.
[0035] In some embodiments, following the form of cross-attention reference, the input U is represented as Q, K, V by linear mapping. The query vector Qe of edge features is then added to the visible light and infrared query vectors Qr, Qt, respectively, to help guide the fusion of the two modes.
[0036] Preferably, the fusion module is based on a cross-attention mechanism, using the keys of the modality to respectively calculate the dot product attention of the remaining modality queries. The calculation formula is as follows:
[0037]
[0038] Preferably, the method is experimented on a multispectral object detection dataset, which includes a total of 95,328 color thermal image pairs, containing 103,128 object annotations and 1,182 unique objects.
[0039] In some embodiments, the annotation uses a total of 25086 frames for training. Based on the provided true labels, the present invention is rigorously evaluated on a test set consisting of 2252 images, including 1455 daytime images ("day") and 797 nighttime images ("night").
[0040] Preferably, the image resolution of the FLIR data set is 512×640.
[0041] In some embodiments, the present invention uses the commonly used log mean missing rate method to sample the false positive rate (FPPI) of each image within a specified range of (0.01, 1). Due to its precision and accuracy, this method is the most commonly used metric in object detection tasks.
[0042] In some embodiments, a detection framework based on the resnet18 backbone network is constructed. A multispectral target detection framework based on Pytorch is implemented.
[0043] In some embodiments, an optimization algorithm (Adam) training framework based on a momentum algorithm and an AdaGrad learning rate is used, with hyperparameters betas=(0.9, 0.998), eps=2e-8.
[0044] Preferably, the framework is trained on 2 NVIDIA 3090 GPUs with a total of 8 sets of images per batch size. The detector is trained for 60 epochs with an initial learning rate of 1e-4.
[0045] In some implementations, in the MSE module, dropout is set to 0.15, and the number of repetitions of the MSE module in each stage is set to N=3.
[0046] The network framework proposed in the present invention is superior to the existing ResNet-based method, and the method of the present invention significantly exceeds the performance of MBNet using ResNet-50. It also achieves higher detection accuracy. The experimental results prove the effectiveness of the present invention and its practical value.
[0047] From the above, this application proposes a target detection method based on edge information assisted dual-spectral fusion, which solves the problems in multispectral target detection, especially the problem of insufficient consistency in modal feature extraction and fusion. The edge feature extraction module is densely embedded in resnet18 at each stage to perform comprehensive edge feature extraction. The MSE module embedded in the backbone network combines visible, thermal and edge features, allowing edge features to guide the fusion of modalities. This enhances the consistency of multi-modal multispectral information at different semantic levels and improves the robustness of the detector under different lighting conditions. The experimental results verify the effectiveness of the key components of a target detection method based on edge information assisted dual-spectral fusion, showing superior performance on the dataset compared to some state-of-the-art methods.
[0048] The technical means disclosed in the scheme of the present invention are not limited to the technical means disclosed in the above-mentioned implementation mode, but also include technical schemes composed of any combination of the above technical features.
Claims
1. A target detection method based on edge information assisted dual-spectrum fusion, characterized in that: The following steps are involved: S1. Obtain paired infrared and visible light target images, perform random flipping data enhancement operations, complete data preprocessing, and divide the training set, validation set, and test set into proportions; S2. Construct a multispectral target detection network structure that integrates multispectral semantics and edge attention information, namely, the MSENet network structure. The MSENet adds three stages, namely stage5-7, based on the resnet18 network. Then, an edge attention module is designed and added to the infrared image network to obtain the target edge information from the infrared spectrum. Secondly, a fusion module (MSE) is designed using three modal information including infrared, visible light, and edge information. A fusion module is added to each stage, and finally the fusion result is input into the detector part to obtain the final output result. S3. Train MSENet using the infrared and visible light target images, adjust hyperparameters, i.e. use the adam optimizer and set the learning rate to 0.0001 to obtain the best target detection model.
2. The target detection method based on edge information assisted dual-spectrum fusion according to claim 1, characterized in that: The multimodal target detection method of S2 is mainly composed of a feature extraction and detection structure, an edge feature extraction module and a fusion module; the extraction structure adopts the resnet18 network architecture and adds a Stage5-7 structure, each of which contains two convolutional layers to extract deep semantic features from multimodal images; The SSD algorithm is used for final fusion and bounding box regression prediction; the edge feature extraction module is designed to extract edge features for feature enhancement by taking the convolutional network as the backbone; considering the modal characteristics of the infrared image modality, only the features after the fourth stage are selected as the input of the edge detection module.
3. The target detection method based on edge information assisted dual-spectrum fusion according to claim 1, characterized in that: In the detection process described in step S3, the model is evaluated on the test set; the evaluation indicators used are MR-rate, MR-scale, and MR-occlusion.
4. The target detection method based on edge information assisted dual-spectrum fusion according to claim 1, characterized in that: In S2: the edge attention module includes: using two convolution operators To extract the edge information of the feature map, the semantic information is normalized by the BN layer, and then the L2 norm and sigmoid function are used to aggregate the horizontal and vertical spatial information, and the attention value is obtained at the same time; finally, after element-by-element multiplication, the attention value will be broadcasted accordingly to obtain the edge convolution The output result of the edge feature extraction module is as follows: Where σ represents the sigmoid function, β represents batch normalization and It is element-wise multiplication.
5. The target detection method based on edge information assisted dual-spectrum fusion according to claim 1, characterized in that: In order to estimate the semantic similarity between target features of different forms through the assistance of edge features, the feature fusion module in S2 proposes a cross-view feature representation: W r =f MSE (U r ,U t ,U e ) W t =f MSE (U t ,U r ,U e ) Among them, fMSE represents the proposed MSE module, and the final output is Wr and Wt, whose format is the same as the input format same.
6. The target detection method based on edge information assisted dual-spectrum fusion according to claim 5, characterized in that: The feature fusion module described in S2 follows the form of cross-attention reference, and the input U is represented as Q, K, and V respectively through linear mapping; then the query vector Qe of the edge feature is added to the visible light and thermal query vectors Qr and Qt respectively to help guide the fusion of the two modes. Finally, a novel form of cross-attention reference is constructed.
7. The target detection method based on edge information assisted dual-spectrum fusion according to claim 5, characterized in that: The fusion module is based on a cross-attention mechanism, using the keys of the modality to calculate the dot product attention of the remaining modality queries respectively.
8. The target detection method based on edge information assisted dual-spectrum fusion according to claim 5, characterized in that: The calculation formula of the fusion module described above based on the cross attention mechanism is as follows: where softmax represents the activation function softmax, T represents the transposed vector, d represents the dimension of the vector, t, r, e represent the features from infrared, visible light, and edge respectively, and K, V, Q represent the features of different vector spaces respectively;
Citation Information
Cited By
Double-spectrum asymmetric fusion identification method and device under single-mode degradation condition
CN121121679A
