Non-registration multi-mode fusion real-time target detection method based on infrared-visible light image pair
Through the non-registered cross-modal feature interaction learning network, the position movement problem in multimodal object detection is solved, the detection efficiency and accuracy are improved, and real-time object detection under complex conditions is realized.
Patent Information
- Application Number
- CN202411050121.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-01
- Publication Date
- 2024-11-29
AI Technical Summary
Under complex conditions such as insufficient light, occlusion and cluttered background, there are location movement problems in multimodal object detection, and existing methods are difficult to effectively solve the problem of weak alignment, resulting in insufficiency of detection.
A non-registered cross-modal feature interaction learning network based on infrared-visible image pairs is adopted. Through feature extraction, interaction, enhancement and splicing modules, feature similarity is directly calculated, image registration steps are cancelled, and network efficiency is improved.
The performance and efficiency of multi-spectral vehicle detection is significantly improved, real-time object detection under weak alignment conditions is achieved, and the accuracy and speed of detection results are improved.
Smart Images

Figure CN119048731A_ABST
Abstract
Description
Technical Field
[0001] The invention belongs to the field of real-time target detection, and in particular relates to a real-time target detection method based on non-registration multimodal fusion of infrared-visible light image pairs. Background Art
[0002] As computer vision technology has gradually become an important research topic in the computer field, cross-modal vision-based target detection has gradually demonstrated remarkable performance, greatly promoting its wide application in the fields of autonomous driving, remote sensing, video surveillance, etc. Although it has achieved certain success in most scenarios, under complex conditions such as insufficient lighting, occlusion, and cluttered background, the same object has different positions in different modes of different sensors, resulting in great difficulties in detecting vehicles in extreme cases. Therefore, in recent years, researchers have proposed a variety of different research solutions to solve the problem of weak alignment of sensors in order to address the position offset problem of multispectral vehicle detection.
[0003] At present, the research on multimodal object detection is very extensive, but few detection methods involve the problem of position shift in weakly aligned image pairs. Due to hardware and environmental reasons, the problem of position shift is inevitable in real scenes, which can be alleviated independently through preprocessing and postprocessing. Figure 1 As shown in the figure, the preprocessing solves the camera calibration problem by designing a modality alignment system. However, the hardware of such a system is inevitably susceptible to external interference, and due to the characteristics of the additional modality, the recalibration process is very difficult. The image registration process requires large computing resources, which leads to low efficiency of modality registration after camera calibration. A common example of post-processing is spatial alignment. As a pixel-level solution to the position shift problem, post-processing geometrically aligns the actual image and the perceived image. This task mainly includes four processes: feature detection, mapping function design, feature matching, image transformation, and resampling. Most of the existing solutions to the weak alignment problem are based on the assumption of full alignment, directly fusing the features of different modalities at the corresponding pixel positions without calculating the similarity values of each region. This not only hinders the use of weakly aligned datasets, but also limits the further development of multispectral vehicle detection. Summary of the invention
[0004] The present invention provides a fusion detection method based on a non-registered cross-modal feature interactive learning network of infrared-visible light image pairs to solve the problem of position movement of weakly aligned image pairs in multispectral vehicle detection data.
[0005] A fusion detection method based on a non-registered cross-modal feature interactive learning network of infrared-visible light image pairs includes the following steps:
[0006] S1. Data preprocessing: FLIR dataset and LLVIP dataset are selected. The datasets contain infrared-visible light image pairs. The infrared-visible light image pairs are divided into training sets and test sets. According to the ratio of training set: test set = 4:1, the visible light images and their corresponding labels are randomly translated by [-50, 50] pixels to obtain weakly aligned visible light images.
[0007] S2. Feature extraction: Construct a feature extraction network, which includes two branches with the same structure. The preprocessed infrared-visible light image pair is input into the feature extraction network, and the two branches extract visible light features F respectively. rgb 、Infrared characteristics F ir ;
[0008] S3. Feature interaction: Construct a non-registration cross-modal feature interaction learning network NRCMFINet to transform the visible light feature F rgb With infrared characteristics F ir Input the NRCMFINet for training and extract the visible light enhancement feature Z rgb and infrared enhanced feature Z ir ;
[0009] S4, feature enhancement: construct a feature enhancement splicing module ESM, and transform the Z rgb and Z ir Input the feature enhancement splicing module ESM to perform calculations to obtain the multi-scale fusion feature Z splic ;
[0010] S5. Output the multi-scale fusion feature Z splic , and obtain the final test results.
[0011] Furthermore, the feature extraction network is the CSPDarknet53 feature extraction network in YOLOv5. in Represents a three-dimensional matrix, W is the width, H is the height, and C represents the number of channels.
[0012] Furthermore, in S3, the non-registration cross-modal feature interaction learning network NRCMFINet includes a cross-modal weakly aligned feature interaction module NRCMFI, and the cross-modal weakly aligned feature interaction module NRCMFI includes a sliding window feature interaction module SWFIM.
[0013] Furthermore, the execution process of the non-registration cross-modal feature interaction learning network NRCMFINet includes the following steps:
[0014] S31: The visible light feature F rgb With infrared characteristics Fir Divide into Non-overlapping local windows, i.e. visible light feature windows X rgb With infrared characteristic window X ir , Where S is the internal window size;
[0015] S32: The X rgb and X ir Calculate the cosine similarity based on each other, take the value with the largest cosine similarity, and add it element by element into X rgb With X ir In the above equation, we get their respective cross attention similarity SWMSC:
[0016]
[0017] In the formula, x ir represents the infrared local feature window, x rgb represents the visible light local feature window, i represents the i-th non-overlapping local window;
[0018] S34: Use the multi-layer perceptron MLP with GELU activation function to optimize the feature markers generated by the SWMSC to obtain the infrared enhancement feature Z rgb and visible light enhancement feature Z ir , as shown below:
[0019] Z ir =LN(MLP(LN(SWMSC(X ir ))))+LN(SWMSC(X ir ))
[0020] Z rgb =LN(MLP(LN(SWMSC(X rgb ))))+LN(SWMSC(X rgb ))
[0021] Where LN represents layer normalization.
[0022] Furthermore, in the fusion detection method based on the non-registered cross-modal feature interactive learning network of infrared-visible light image pairs, the execution process of the feature enhancement splicing module ESM includes the following steps:
[0023] S411: The visible light enhancement feature Z rgb Perform average pooling and maximum pooling operations on the channels to generate the average pooling features of the visible light channel Visible light channel maximum pooling feature The and The shared network is fed forward to the same shared network, which consists of a multi-layer perceptron MLP with a hidden layer, and outputs the visible light channel attention feature M by element-by-element addition. c (Z rgb ),Right now:
[0024]
[0025] Where M c is the channel attention feature, σ represents the sigmoid function, W0 is the weight of the ReLU function, and W1 is The weight of W2 is The weight of r is the reduction rate, AvgPool is the average pooling, MaxPool is the maximum pooling, W0, W1, and W2 in MLP are all shared, and W0 is after W1 and W2;
[0026] S412: The visible light enhancement feature Z rgb Perform average pooling and maximum pooling operations in space to generate visible light spatial average pooling features Visible light space maximum pooling feature The and Perform concatenation and convolution layer convolution to output the visible light spatial attention feature M s (Z rgb ),Right now:
[0027]
[0028] Where M s is the spatial attention feature, σ represents the sigmoid function, f 7×7 Represents a convolution operation with a convolution kernel size of 7×7;
[0029] S413: The M c (Z rgb ), M s (Z rgb ) and Z rgb Do element-by-element addition to get the visible light enhancement feature M rgb ,Right now:
[0030] M rgb =Z rgb +M c (Z rgb )+M s (Z rgb )
[0031] S421: The infrared enhancement feature Zir Perform average pooling and maximum pooling operations on the channels to generate the average pooling features of the infrared channel. Infrared channel maximum pooling feature The and Feed forward to the same shared network, which consists of a multi-layer perceptron MLP with a hidden layer, and outputs the infrared channel attention features M by element-by-element addition. c (Z ir ),Right now:
[0032]
[0033] Where W3 is The weight of W4 is The weight of
[0034] S422: The infrared enhancement feature Z ir Perform average pooling and maximum pooling operations in space to generate infrared spatial average pooling features Infrared spatial maximum pooling feature The and Perform concatenation and convolution layer convolution to output infrared spatial attention feature M s (Z ir ),Right now:
[0035]
[0036] S423: The M c (Z ir ), M s (Z ir ) and Z ir Do element-by-element addition to get the infrared enhancement feature M ir ,Right now:
[0037] M ir =Z ir +M c (Z ir )+M s (Z ir )
[0038] S43: The M rgb and M ir Do splicing to get the multi-scale fusion feature Z splic , the feature concatenation calculation is:
[0039]
[0040] In the formula, Indicates splicing.
[0041] The present invention proposes a non-registered cross-modal feature interaction learning network NRCMFINet for infrared-visible light image pairs. NRCMFINet introduces a new component called non-registered cross-modal feature interaction module NRCMFI, which establishes a channel between modal alignment and unit fusion of different features; the present invention designs a spatial and channel attention mechanism for the feature enhancement splicing module ESM, which can well enhance the enhanced features obtained in NRCMFI again. ESM can be easily embedded in existing methods, thereby bringing significant performance improvements; the present invention cancels the image registration part in target detection, directly calculates the similarity in SCMWAFIINet, and significantly improves the operation efficiency of the network. BRIEF DESCRIPTION OF THE DRAWINGS
[0042] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0043] Figure 1 This is a schematic diagram comparing the registration and non-registration cross-modal feature interactive learning networks.
[0044] Figure 2 It is a basic flow chart of the fusion detection method of the present invention.
[0045] Figure 3 It is a schematic diagram of the network architecture of the fusion detection method of the present invention.
[0046] Figure 4 It is a schematic diagram of the cross-modal feature interaction module of the present invention.
[0047] Figure 5 It is a schematic diagram of the feature enhancement splicing module of the present invention. DETAILED DESCRIPTION
[0048] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0049] like Figure 2-5As shown, the present invention provides a fusion detection method based on a non-registered cross-modal feature interactive learning network of infrared-visible light image pairs, comprising the following steps:
[0050] S1. Data preprocessing: FLIR dataset and LLVIP dataset are selected. The datasets contain infrared-visible light image pairs. The infrared-visible light image pairs are divided into training sets and test sets. According to the ratio of training set: test set = 4:1, the visible light images and their corresponding labels are randomly translated by [-50, 50] pixels to obtain weakly aligned visible light images.
[0051] S2. Feature extraction: Construct a feature extraction network, which includes two branches with the same structure. The preprocessed infrared-visible light image pair is input into the feature extraction network, and the two branches extract visible light features F respectively. rgb 、Infrared characteristics F ir ;
[0052] S3. Feature interaction: Construct a non-registration cross-modal feature interaction learning network NRCMFINet to transform the visible light feature F rgb With infrared characteristics F ir Input the NRCMFINet for training and extract the visible light enhancement feature Z rgb and infrared enhanced feature Z ir ;
[0053] S4, feature enhancement: construct a feature enhancement splicing module ESM, and transform the Z rgb and Z ir Input the feature enhancement splicing module (ESM) to perform calculations to obtain the multi-scale fusion feature Z splic ;
[0054] S5. Output the multi-scale fusion feature Z splic , and obtain the final test results.
[0055] The dataset may be replaced by the VEDAI dataset.
[0056] The feature extraction network is the CSPDarknet53 feature extraction network in YOLOv5.
[0057] The visible light characteristics Infrared signature in Represents a three-dimensional matrix, W is the width, H is the height, and C is the number of channels.
[0058] The multi-scale fusion feature Z splic The output is sent to the detection head to detect the type and location of the predicted target.
[0059] The non-registration cross-modal feature interaction learning network NRCMFINet includes a cross-modal weakly aligned feature interaction module NRCMFI, and the cross-modal weakly aligned feature interaction module NRCMFI includes a sliding window feature interaction module SWFIM.
[0060] The non-registration cross-modal feature interaction learning network NRCMFINet execution process includes the following steps:
[0061] S31: The visible light feature F rgb With infrared characteristics F ir Divide into Non-overlapping local windows, i.e. visible light feature windows X rgb With infrared characteristic window X ir , Where S is the internal window size;
[0062] S32: The X rgb and X ir Calculate the cosine similarity based on each other, take the value with the largest cosine similarity, and add it element by element into X rgb With X ir In the above equation, we get their respective cross attention similarity SWMSC:
[0063]
[0064] In the formula, x ir represents the local feature window of infrared features, x rgb represents the visible light local feature window, i represents the i-th non-overlapping local window;
[0065] S34: Use the multi-layer perceptron MLP with GELU activation function to optimize the feature markers generated by the SWMSC to obtain the infrared enhancement feature Z rgb and visible light enhancement feature Z ir , as shown below:
[0066] Z ir =LN(MLP(LN(SWMSC(X ir ))))+LN(SWMSC(X ir ))
[0067] Z rgb =LN(MLP(LN(SWMSC(X rgb ))))+LN(SWMSC(X rgb ))
[0068] The LN stands for layer normalization.
[0069] The execution process of the feature enhancement splicing module (ESM) includes the following steps:
[0070] S411: The visible light enhancement feature Z rgb Perform average pooling and maximum pooling operations on the channels to generate the average pooling features of the visible light channel Visible light channel maximum pooling feature The and The shared network is fed forward to the same shared network, which consists of a multi-layer perceptron MLP with a hidden layer, and outputs the visible light channel attention feature M by element-by-element addition. c (Z rgb ),Right now:
[0071]
[0072] Where M c is the channel attention feature, σ represents the sigmoid function, W0 is the weight of the ReLU function, and W1 is The weight of W2 is The weight of r is the reduction rate, AvgPool is the average pooling, MaxPool is the maximum pooling, W0, W1, and W2 in MLP are all shared, and W0 is after W1 and W2;
[0073] S412: The visible light enhancement feature Z rgb Perform average pooling and maximum pooling operations in space to generate visible light spatial average pooling features Visible light space maximum pooling feature The and
[0074] Perform concatenation and convolution layer convolution to output the visible light spatial attention feature M s (Z rgb ),Right now:
[0075]
[0076] Where M s is the spatial attention feature, σ represents the sigmoid function, f 7×7 Represents a convolution operation with a convolution kernel size of 7×7;
[0077] S413: The M c (Z rgb ), M s (Z rgb ) and Z rgbDo element-by-element addition to get the visible light enhancement feature M rgb ,Right now:
[0078] M rgb =Z rgb +M c (Z rgb )+M s (Z rgb )
[0079] S421: The infrared enhancement feature Z ir Perform average pooling and maximum pooling operations on the channels to generate the average pooling features of the infrared channel. Infrared channel maximum pooling feature The and Feed forward to the same shared network, which consists of a multi-layer perceptron MLP with a hidden layer, and outputs the infrared channel attention features M by element-by-element addition. c (Z ir ),Right now:
[0080]
[0081] Where W3 is The weight of W4 is The weight of
[0082] S422: The infrared enhancement feature Z ir Perform average pooling and maximum pooling operations in space to generate infrared spatial average pooling features Infrared spatial maximum pooling feature The and Perform concatenation and convolution layer convolution to output infrared spatial attention feature M s (Z ir ),Right now:
[0083]
[0084] S423: The M c (Z ir ), M s (Z ir ) and Z ir Do element-by-element addition to get the infrared enhancement feature M ir ,Right now:
[0085] M ir =Z ir +M c (Z ir )+M s(Z ir )
[0086] S43: The M rgb and M ir Do splicing to get Z splic , the feature concatenation calculation is:
[0087]
[0088] in Indicates splicing.
[0089] Table 1
[0090]
[0091]
[0092] As shown in Table 1, the present invention designs four settings to explore the effectiveness of NRCMFI and ESM. The first row indicates that both NRCMFI and ESM are removed, and the feature extraction branch directly provides two single-modal features to the feature fusion module; the second row indicates that NRCMFI in is replaced by several convolutional layers, and the third row indicates that convolution and splicing in ESM are replaced by splicing.
[0093] From Table 1, the following conclusions can be drawn:
[0094] 1. The use of ESM in the present invention further improves the NRCMFI performance, and the mAP increases from 36.7% to 36.9%, which proves that the ESM network is beneficial to target detection.
[0095] 2. Without using ESM, using NRCMFI significantly improves the performance, with mAP increased by 1.6%, which proves that NRCMFI can enhance the relevance of features, alleviate the shortcomings of unimodal features, and the refined features can better capture significant target clues.
[0096] 3. The integration of the two achieves good performance, with a mAP value of 38.9%, proving the effectiveness of the present invention. As mentioned above, NRCMFINet introduces NRCMFI to guide the generation of information features, providing a new technical reference for the field of visible-infrared target detection.
[0097] Table 2
[0098]
[0099]
[0100] Table 3
[0101] Model Dataset Type <![CDATA[mAP 50 Indicator (LLVIP)]]> Base-detector RGB 86.9% Base-detector IR 90.8% HalfwayFusion Strict alignment 95.4% YoloFusion Strict alignment 95.4% CFT Strict alignment 97.5% UA-CMDet Strict alignment 96.3% MBNet Weak alignment 95.3% ProbEn3 Weak alignment 95.5% NRCMFINet Weak alignment 95.7%
[0102] Tables 2 and 3 give the performance comparison results of NRCMFINet on the FLIR dataset and LLVIP, which shows that:
[0103] (1) The performance of the present invention in dual-modality is better than that in single-modality, which reflects the effectiveness of the multi-modal feature fusion strategy;
[0104] (2) When the NRCMFINet of the present invention is compared with the existing technologies of strict alignment and weak alignment, the performance of the method of the present invention is better;
[0105] (3) Among all the multimodal detection methods, the NRCMFINet of the present invention achieves better performance.
[0106] Table 4
[0107] Model Dataset Type Test time (ms / ing) UA-CMDet Strict alignment 370.4 CFT Strict alignment 91.2 ProbEn3 Weak alignment 846.9 MBNet Weak alignment 70.9 NRCMFINet Weak alignment 58.5
[0108] As can be seen from Table 4, when comparing NRCMFINet with the strictly aligned and weakly aligned models of the prior art, the present invention has obvious advantages in test time, which is about 17.49% faster than MBNet and about 35.86% faster than CFT, proving the significant progress of the present invention in the real-time target detection method of non-registration multimodal fusion.
[0109] In summary, the improvement of the present invention is due to three reasons:
[0110] 1. The real-time target detection method of the present invention eliminates the image registration step, and completes the similarity calculation while the non-registration cross-modal feature interactive learning network NRCMFINet performs feature extraction;
[0111] Second, the non-registration cross-modal feature interaction learning network NRCMFINet in the present invention adopts a cross-modal weak alignment feature interaction module (NRCMFI) to improve the quality of visible light features and infrared features and perform feature fusion;
[0112] 3. The present invention designs a feature enhancement splicing module ESM to enhance the feature information of target detection.
[0113] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.
[0114] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.
Claims
1. A fusion detection method based on a non-registered cross-modal feature interactive learning network for infrared-visible light image pairs, characterized in that: The following steps are involved: S1. Data preprocessing: FLIR dataset and LLVIP dataset are selected. The datasets contain infrared-visible light image pairs. The infrared-visible light image pairs are divided into training sets and test sets. According to the ratio of training set: test set = 4:1, the visible light images and their corresponding labels are randomly translated by [-50, 50] pixels to obtain weakly aligned visible light images. S2. Feature extraction: Construct a feature extraction network, which includes two branches with the same structure. The preprocessed infrared-visible light image pair is input into the feature extraction network, and the two branches extract visible light features F respectively. rgb 、Infrared characteristics F ir ; S3. Feature interaction: Construct a non-registration cross-modal feature interaction learning network NRCMFINet to transform the visible light feature F rgb With infrared characteristics F ir Input the NRCMFINet for training and extract the visible light enhancement feature Z rgb and infrared enhanced feature Z ir ; S4. Feature enhancement: Construct a feature enhancement splicing module ESM and transform the Z rgb and Z ir Input the feature enhancement splicing module ESM to perform calculations to obtain the multi-scale fusion feature Z splic ; S5. Output the multi-scale fusion feature Z splic , and obtain the final test results.
2. The fusion detection method based on the non-registration cross-modal feature interactive learning network of infrared-visible light image pairs according to claim 1 is characterized in that: In S2, the feature extraction network is the CSPDarknet53 feature extraction network in YOLOv5. in Represents a three-dimensional matrix, W is the width, H is the height, and C represents the number of channels.
3. The fusion detection method based on the non-registered cross-modal feature interactive learning network of infrared-visible light image pairs according to claim 1 is characterized in that: In S3, the non-registration cross-modal feature interaction learning network NRCMFINet includes a cross-modal weakly aligned feature interaction module NRCMFI, and the cross-modal weakly aligned feature interaction module NRCMFI includes a sliding window feature interaction module SWFIM.
4. The fusion detection method based on the non-registration cross-modal feature interactive learning network of infrared-visible light image pairs according to claim 1 or 3 is characterized in that: The execution process of the non-registration cross-modal feature interaction learning network NRCMFINet includes the following steps: S31: The visible light feature F rgb With infrared characteristics F ir Divide into Non-overlapping local windows, i.e. visible light feature windows X rgb With infrared characteristic window X ir , Where S is the internal window size; S32: The X rgb and X ir Calculate the cosine similarity based on each other, take the value with the largest cosine similarity, and add it element by element into X rgb With X ir In the above equation, we get their respective cross attention similarity SWMSC: In the formula, x ir represents the infrared local feature window, x rgb represents the visible light local feature window, i represents the i-th non-overlapping local window; S34: Use the multi-layer perceptron MLP with GELU activation function to optimize the feature markers generated by the SWMSC to obtain the infrared enhancement feature Z rgb and visible light enhancement feature Z ir , as shown below: Z ir =LN(MLP(LN(sWMSC(X ir ))))+LN(sWMSC(X ir )) Z rgb =LN(MLP(LN(SWMSC(X rgb ))))+LN(SWMSC(X rgb )) Where LN represents layer normalization.
5. The fusion detection method based on the non-registered cross-modal feature interactive learning network of infrared-visible light image pairs according to claim 1 is characterized in that: The execution process of the feature enhancement splicing module ESM includes the following steps: S411: The visible light enhancement feature Z rgb Perform average pooling and maximum pooling operations on the channels to generate the average pooling features of the visible light channel Visible light channel maximum pooling feature The and Feed forward to the same shared network, which consists of a multi-layer perceptron MLP with a hidden layer, and outputs the visible light channel attention feature Mc by element-by-element addition. ( Z rgb ),Right now: Where M c is the channel attention feature, σ represents the sigmoid function, W0 is the weight of the ReLU function, and W1 is The weight of W2 is The weight of r is the reduction rate, AvgPool is the average pooling, MaxPool is the maximum pooling, W0, W1, and W2 in MLP are all shared, and W0 is after W1 and W2; S412: The visible light enhancement feature Z rgb Perform average pooling and maximum pooling operations in space to generate visible light spatial average pooling features Visible light space maximum pooling feature The and Perform concatenation and convolution layer convolution to output the visible light spatial attention feature M s (Z rgb ),Right now: M s (Z rgb )=σ(f 7×7 ([AvgPool(Z rgb ); Where M s is the spatial attention feature, σ represents the sigmoid function, f 7×7 Represents a convolution operation with a convolution kernel size of 7×7; S413: The M c (Z rgb )、M s (Z rgb ) and Z rgb Do element-by-element addition to get the visible light enhancement feature M rgb ,Right now: M rgb =Z rgb +M c (From rgb )+M s (From rgb ) S421: The infrared enhancement feature Z ir Perform average pooling and maximum pooling operations on the channels to generate the average pooling features of the infrared channel. Infrared channel maximum pooling feature The and Feed forward to the same shared network, which consists of a multi-layer perceptron MLP with a hidden layer, and outputs the infrared channel attention features M by element-by-element addition. c (Z ir ),Right now: Where W3 is The weight of W4 is The weight of S422: The infrared enhancement feature Z ir Perform average pooling and maximum pooling operations in space to generate infrared spatial average pooling features Infrared spatial maximum pooling feature The and Perform concatenation and convolution layer convolution to output infrared spatial attention feature M s (Z ir ),Right now: S423: The M c (Z ir )、M s (Z ir ) and Z ir Do element-by-element addition to get the infrared enhancement feature M ir ,Right now: M ir =Z ir +M c (From ir )+M s (From ir ) S43: The M rgb and M ir Do splicing to get the multi-scale fusion feature Z splic , the feature concatenation calculation is: In the formula, Indicates splicing.
Citation Information
Patent Citations
Non-aligned visible light and infrared modal fusion target detection method based on style migration
CN115578246A
Infrared-visible light fusion method and system based on feature enhancement and readable storage medium
CN116258934A
Model training method, system and device for three-mode environment monitoring and detection
CN117372808A
Non-registered infrared visible light image fusion and vehicle target detection mutual promotion method
CN118279868A
Method for small object detection in drone scene based on deep learning
US11881020B1
Cited By
Unmanned aerial vehicle power station inspection fault detection method and system based on multi-modal depth feature fusion
CN120766015A
Infrared and visible light feature fusion method based on space channel decoupling coupling
CN121214132A
Visible light infrared image target detection method and system based on modal common characteristics
CN121437515A
