RGB and thermal infrared fusion detection system and method for complex visual environment

Abstract

The present application relates to the technical field of image processing, and more particularly to a RGB and thermal infrared fusion detection system and method for complex visual environment, wherein a backbone network respectively extracts multi-scale features of input RGB images and infrared images, a feature fusion network respectively enhances local features of the two kinds of multi-scale features, and then respectively enhances the two kinds of features in the row and column directions after enhancement to obtain RGB enhanced features and infrared enhanced features; the RGB enhanced features and the infrared enhanced features are fused to obtain fused features; a neck network gradually fuses the features, obtains a soft histogram of the current fused features after each fusion, and enhances the features obtained by the current fusion by using the soft histogram; and a detection network respectively inputs the multi-scale enhanced features into corresponding detection heads to obtain target detection results of targets of different sizes. The present application effectively suppresses background interference and improves the detection effect of small targets.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, and in particular relates to an RGB and thermal infrared fusion detection system and method for complex visual environments. Background Technology

[0002] Multi-source image fusion technology, relying on the collaborative sensing mechanism of heterogeneous sensors, constructs a more dimensionally interconnected information acquisition framework. At the remote sensing observation level, SAR possesses microwave penetration capabilities, effectively penetrating clouds, fog, and low-light environments; while optical imagery excels in its high spectral resolution. The two complement each other, achieving full coverage of Earth observation in both temporal and spatial dimensions, significantly improving the continuity and stability of dynamic monitoring. In target detection tasks, the fusion of mid-wave infrared thermal radiation information and structural details from visible light images restores the complete brightness and texture hierarchy of the scene. Overall, multimodal fusion not only enhances the system's anti-interference capability and environmental adaptability but also provides broader expansion space in terms of perception integrity and data value, becoming an important supporting means for intelligent image processing.

[0003] In vehicle target detection tasks, the application of multi-source image fusion technology can significantly improve the robustness and accuracy of detection. For example, in various complex lighting, occlusion, rain, and snow scenarios, fused images can retain a large amount of structural details and salient features. Therefore, combining multi-source image fusion can enhance the boundary perception capability of targets, improve the resolution of small targets, and help reduce the occurrence of false detections and false negatives. Thus, in different scenarios such as intelligent transportation, autonomous driving, and military surveillance, multi-source image fusion can assist detection models in accurately detecting vehicle targets. Vehicle detection schemes that fuse infrared, optical, and SAR images are receiving increasing attention and have become an important technical method for improving target detection systems.

[0004] The drawbacks of existing technologies include: under low-light conditions, especially in the dark, there are significant differences between different modalities. Modal fusion strategies need to be able to dynamically adjust the fusion weights to adapt to these differences and avoid mutual interference between modalities; remote sensing vehicles are small in scale, and background interference is severe, making it difficult to characterize features and resulting in low detection accuracy; the distribution of the number of remote sensing vehicle categories is severely uneven, causing the model to not learn enough vehicle features and resulting in serious missed detections. Summary of the Invention

[0005] In view of this, the present invention aims to provide an RGB and thermal infrared fusion detection system and method for complex visual environments. It comprehensively utilizes the feature information of dual-source modalities, reduces the interference of harsh environments on detection accuracy, and significantly improves detection accuracy. Specifically, it uses histogram statistics to collect global feature information of the image, suppresses background interference, and improves the detection effect of small targets. Finally, it uses loss function optimization to improve the detection effect of difficult targets.

[0006] To achieve the above objectives, the technical solution created by this invention is implemented as follows:

[0007] An RGB and infrared fusion detection system for complex visual environments includes: a backbone network that extracts multi-scale features from input RGB and infrared images to obtain multi-scale RGB and infrared features respectively; a feature fusion network that enhances the multi-scale RGB and infrared features locally, and then enhances the enhanced features in the row and column directions respectively to obtain multi-scale enhanced RGB and infrared features; the RGB and infrared enhanced features at corresponding scales are fused to obtain multi-scale fused features; a neck network that performs hierarchical feature fusion of the multi-scale fused features in a bottom-up and top-down manner to obtain multi-scale enhanced features; after each fusion, a soft histogram of the current fused features is obtained, and the soft histogram is used to enhance the features obtained by the current fusion; and a detection network that inputs the multi-scale enhanced features into corresponding detection heads to obtain target detection results for targets of different sizes.

[0008] Furthermore, the feature fusion network includes an intramodal structure enhancement module and a cross-modal difference enhancement perception module. The intramodal structure enhancement module includes parallel and structurally consistent RGB processing branches and infrared processing branches. In the RGB or infrared processing branch: continuous row-direction attention feature enhancement operations are performed on the RGB or infrared features to obtain row-enhanced features; then column-direction attention feature enhancement operations are performed on the row-enhanced features to obtain column-enhanced features; the obtained row-enhanced features and column-enhanced features are fused with the corresponding original input features to obtain RGB-enhanced features or infrared-enhanced features. In the cross-modal difference enhancement perception module: RGB... After channel-direction compression and fusion of the RGB enhancement features and the infrared enhancement features, a global semantic feature containing global information from both RGB and infrared features is obtained. Convolution operations at different scales are performed on the RGB enhancement features and the infrared enhancement features respectively to obtain RGB multi-scale features and infrared multi-scale features. After combining the global semantic feature with the RGB multi-scale features and the infrared multi-scale features and extracting weights, RGB cross-modal weights and infrared cross-modal weights are obtained. The RGB cross-modal weights and infrared cross-modal weights are then assigned to the RGB enhancement features and the infrared enhancement features respectively. Finally, the corresponding elements of the two features are added together to obtain the fused feature at the current scale.

[0009] Furthermore, the process of performing row-direction feature enhancement includes: performing a linear mapping on the input features to generate the original query matrix, original key matrix, and original value matrix of the current input features; compressing the spatial dimension of the original query matrix, original key matrix, and original value matrix in the row direction to obtain the corresponding row query matrix, row key matrix, and row value matrix; and performing attention enhancement operations on the row query matrix, row key matrix, and row value matrix to obtain row-enhanced features. The process of performing column-direction feature enhancement includes: performing a linear mapping on the input features to generate the original query matrix, original key matrix, and original value matrix of the current input features; compressing the spatial dimension of the original query matrix, original key matrix, and original value matrix in the column direction to obtain the corresponding column query matrix, column key matrix, and column value matrix; and performing attention enhancement operations on the column query matrix, column key matrix, and column value matrix to obtain column-enhanced features.

[0010] Furthermore, in the process of fusing the obtained row-enhanced features and column-enhanced features with the corresponding original input features: the original query matrix, original key matrix, and original value matrix are concatenated and then subjected to a depthwise separable convolution operation to model the local relationships in the input features and obtain local features; the corresponding elements of the row-enhanced features, column-enhanced features, and original value matrix are added together, and then the added features are multiplied by the corresponding elements of the local features to obtain the output enhanced features.

[0011] Furthermore, the process of obtaining global semantic features includes: performing linear mapping on the RGB enhancement features and infrared enhancement features respectively, adding the corresponding elements of the two features and then performing linear mapping again to complete the fusion of the RGB enhancement features and infrared enhancement features; after performing linear mapping and sigmoid activation on the fused features, the compression operation of the fused features in the channel direction is completed to obtain global semantic features.

[0012] Furthermore, in the neck network, a soft histogram of fused features is obtained through a soft histogram-sensing transformer block, and the soft histogram is used to enhance the features obtained by fusion. In the process of obtaining the soft histogram of fused features, the input features are aggregated in the channel dimension and expanded in the spatial dimension. The elements in the processed features are binned and statistically analyzed using Gaussian kernels to obtain the soft histogram.

[0013] Furthermore, in the neck network, the soft histogram-aware transformer block also includes a preprocessing operation on the fused features of the input, wherein: multiple consecutive residual operations are performed on the fused features of the input to add the corresponding elements of features at different scales.

[0014] Furthermore, the process of using soft histograms to enhance the features obtained from the fusion includes: projecting the soft histogram onto the feature space to obtain soft histogram encoding; performing convolution operations on the input features to generate multiple query matrices and multiple value matrices; and calculating the distribution-guided weights corresponding to each query matrix using the following formula:

[0015] ;

[0016] Where, α i Q represents the i-th query matrix. i The corresponding distribution guides the weights, where δ represents sigmoid activation, C represents the number of channels in the input feature, and H... d t represents the number of query matrices. i This represents the components corresponding to the i query matrices in the soft histogram encoding t; based on the weights guided by each distribution, the corresponding value matrices are weighted and modulated, and then the modulated key matrices are concatenated; the obtained features are input into the feedforward network and then connected with the input feature residuals of the feedforward network to obtain the output features.

[0017] A method for RGB and infrared fusion detection in complex visual environments includes:

[0018] S1: Obtain the dataset and preprocess it to obtain the training set; the dataset includes RGB images, corresponding infrared images, and target labels in the RGB and infrared images;

[0019] S2: Construct an RGB and infrared fusion detection system for complex visual environments as provided in this invention;

[0020] S3: Train the detection system constructed in step S2 using the dataset from step S1 to obtain the target detection model;

[0021] S4: Input the RGB image and infrared image to be detected into the target detection model obtained in step S3 to obtain the target detection result.

[0022] Furthermore, in step S3, the detection model is trained using the following loss function;

[0023] L total =λ1L cael +λ2L box +λ3L dfl ;

[0024] Among them, L total L represents the loss function for training the detection model. cael L represents the adaptive sensing loss. box L represents the bounding box loss. dflLet λ1, λ2, and λ3 represent the distribution focus loss, and let λ1, λ2, and λ3 represent the loss weights.

[0025] The adaptive sensing loss is:

[0026] ;

[0027] in, This indicates that the current detection model outputs the predicted detection result for the m-th sample. This indicates the corresponding actual test result. Let N represent the true detection result of the nth target in the mth sample, where N represents the total number of targets in the mth sample. Indicates the difficulty level. ;

[0028] The bounding box loss is:

[0029] ;

[0030] Where, ω m CIoU represents the weight of the m-th sample. m This represents the CIoU of the m-th sample;

[0031] The distribution focus loss is:

[0032] ;

[0033] in, This represents the true bounding box coordinates of the detected target in the m-th sample. and These represent the left and right boundaries of the detected target, respectively. and Let represent the predicted left and right boundary probabilities of the detected target, respectively, and CE represent the cross-entropy loss.

[0034] Compared with the prior art, the present invention can achieve the following beneficial effects:

[0035] (1) In the RGB and thermal infrared fusion detection system for complex visual environments described in this invention, the intramodal structure enhancement module and the cross-modal difference enhancement perception module work together to enhance the light perception capability of the model by modeling the long-range relationship within the modality and the complementary relationship between modalities, thereby improving the model's adaptability to low-light scenes under different modalities; a soft histogram perception transformer is proposed, which uses the differentiable Gaussian soft histogram to statistically analyze the global feature information of the image, thereby guiding the subsequent spatial attention enhancement, realizing cross-modal interaction from the statistical domain to the spatial domain, in order to solve the problem of small target size of remote sensing vehicles and confusion between background and vehicle;

[0036] (2) In the RGB and thermal infrared fusion detection method for complex visual environments described in this invention, by introducing class perception loss during training and combining dynamic threshold and dual gating branch, the problem of low accuracy of difficult samples caused by class imbalance is addressed, and the learning ability of the model for difficult samples and rare classes is significantly improved. Attached Figure Description

[0037] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments and descriptions of the invention are used to explain the invention and do not constitute an undue limitation of the invention. In the drawings:

[0038] Figure 1 A schematic diagram of the RGB and infrared fusion detection system for complex visual environments as described in the embodiments of the present invention;

[0039] Figure 2 A schematic diagram of the modal intra-structure enhancement module described in an embodiment of the present invention;

[0040] Figure 3 A schematic diagram of the cross-modal difference enhancement sensing module described in an embodiment of the present invention;

[0041] Figure 4 A schematic diagram of the soft histogram sensing transformer block described in an embodiment of the present invention;

[0042] Figure 5 A schematic diagram illustrating the preprocessing operation of the input fusion features in the soft histogram sensing transformer block as described in the embodiment of the present invention;

[0043] Figure 6 A schematic flowchart of the RGB and infrared fusion detection method for complex visual environments described in the embodiments of the present invention;

[0044] Figure 7 Comparison diagram of the methods described in the embodiments of the present invention;

[0045] Figure 8 A comparative graph showing the objective analysis results of the methods described in the embodiments of the present invention. Detailed Implementation

[0046] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and do not constitute a limitation thereof. It should be noted that, unless otherwise specified, the embodiments and features described herein can be combined with each other.

[0047] In the description of this invention, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined with "first," "second," etc., may explicitly or implicitly include one or more of that feature. In the description of this invention, unless otherwise stated, "a plurality of" means two or more.

[0048] The invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0049] like Figure 1 As shown in the embodiment of the present invention, the RGB and infrared fusion detection system for complex visual environments comprehensively utilizes the dual-modal image features of infrared-RGB for vehicle recognition and detection, including a backbone network, a feature fusion network, a neck network, and a detection network. The backbone network extracts multi-scale features from the input RGB and infrared images, respectively, obtaining multi-scale RGB features and infrared features. The feature fusion network enhances the multi-scale RGB and infrared features locally, and then enhances the enhanced features in both row and column directions, respectively, to obtain multi-scale enhanced RGB features and infrared enhanced features. The corresponding scale RGB and infrared enhanced features are then fused to obtain multi-scale fused features. The neck network performs stepwise feature fusion of the multi-scale fused features in a bottom-up and top-down manner to obtain multi-scale enhanced features. After each fusion, a soft histogram of the current fused features is obtained, and the soft histogram is used to enhance the features obtained from the current fusion. The detection network inputs the multi-scale enhanced features into the corresponding detection heads to obtain target detection results for targets of different sizes.

[0050] In this embodiment of the invention, a dual-path isomorphic DarkNet53 is used as the backbone structure to perform local modeling on the input RGB and infrared images, thereby obtaining multi-scale RGB and infrared features respectively. While ensuring the consistency of feature expression, this effectively improves the stability of cross-mode fusion and the detection performance of the overall system.

[0051] In some embodiments, the feature fusion network includes an intramodal structure enhancement module (IMSEM) and a cross-modal difference enhancement perception module (DDAPM). In visible-infrared multimodal vehicle detection tasks, the visible light modality is prone to detail loss and semantic degradation under low light, nighttime, and complex lighting conditions, while the infrared modality, although possessing stable target response capabilities, suffers from deficiencies in texture and structural representation. To fully leverage the complementary advantages of the two modalities in extreme lighting environments, this invention designs an intramodal structure enhancement module and a cross-modal difference enhancement perception module in the feature fusion network. These two modules work together to achieve modal complementarity and enhance light perception capabilities. By fusing feature information from both visible and infrared modalities, the complementary role of multimodal approaches in low-light environments is utilized to improve target perception capabilities and feature representation robustness in low-light scenes.

[0052] The intramodal structure enhancement module aims to address the structural and semantic degradation issues within a single modality. First, it performs intramodal enhancement modeling for both modalities separately to avoid introducing noise interference before cross-modal interactions. Then, through compressed attention operations in the row and column directions, it captures long-range structural dependencies while ensuring computational efficiency. This includes parallel and structurally consistent RGB processing branches and infrared processing branches. Within either the RGB or infrared processing branch: continuous row-direction attention feature enhancement operations are performed on the RGB or infrared features to obtain row-enhanced features; then, column-direction attention feature enhancement operations are performed on the row-enhanced features to obtain column-enhanced features; finally, the obtained row-enhanced features and column-enhanced features are fused with the corresponding original input features to obtain the RGB-enhanced features or infrared-enhanced features.

[0053] In some embodiments, the process of performing row-direction feature enhancement includes: performing a linear mapping on the input features to generate a first original query matrix, a first original key matrix, and a first original value matrix of the current input features; performing row-direction spatial dimension compression on the first original query matrix, the first original key matrix, and the first original value matrix respectively to obtain a row query matrix, a row key matrix, and a row value matrix; and performing an attention enhancement operation on the row query matrix, the row key matrix, and the row value matrix to obtain row-enhanced features. The process of performing row-direction feature enhancement includes: performing a linear mapping on the input features to generate a second original query matrix, a second original key matrix, and a second original value matrix of the current input features; performing column-direction spatial dimension compression on the second original query matrix, the second original key matrix, and the second original value matrix respectively to obtain a column query matrix, a column key matrix, and a column value matrix; and performing an attention enhancement operation on the column query matrix, the column key matrix, and the column value matrix to obtain column-enhanced features.

[0054] In some embodiments, the process of fusing the obtained row enhancement features and column enhancement features with the corresponding original input features includes: performing a linear mapping on the original input features to obtain a third original query matrix, a third original key matrix, and a third original value matrix; concatenating the third original query matrix, the third original key matrix, and the third original value matrix and then performing a depthwise separable convolution operation to model the local relationships in the input features and obtain local features; adding the corresponding elements of the row enhancement features, column enhancement features, and the third original value matrix, and then multiplying the added features with the corresponding elements of the local features to obtain the output enhancement features.

[0055] The modal intra-structure enhancement module provided in this embodiment of the invention is as follows: Figure 2 As shown, the specific feature processing process includes: First, RGB feature IR and infrared feature VI are processed in parallel using the RGB processing branch and the infrared processing branch. The feature processing process of the RGB processing branch for RGB feature IR is consistent with the feature processing process of the infrared processing branch for infrared feature VI; therefore, the feature processing process of the RGB processing branch for RGB feature IR is described in detail here.

[0056] In the RGB processing branch provided in this embodiment of the invention, the process of performing row direction feature enhancement includes: performing a convolution operation on the RGB feature IR to generate a first original query matrix Q1, a first original key matrix K1, and a first original value matrix V1 of the current input feature; performing row direction average pooling operations on the first original query matrix Q1, the first original key matrix K1, and the first original value matrix V1 respectively, and then performing position encoding to obtain the corresponding row query matrix Q. r and row key matrix K r And perform row-wise average pooling on the first original value matrix V1 to obtain the row value matrix V. r The process is as follows:

[0057] Q r =PE(Mean w (Q1));

[0058] K r =PE(Mean w (K1));

[0059] V r =Mean w (V1);

[0060] Where PE stands for Positional Encoding, Mean w This indicates the average pooling operation in the row direction.

[0061] The row-direction attention feature enhancement operation is performed using the following formula to obtain the row-enhanced feature Y. r :

[0062] ;

[0063] Where d represents the number of channels and Softmax represents the Softmax function.

[0064] In this embodiment of the invention, after performing row direction feature enhancement operation in the RGB processing branch, the row enhancement feature Y corresponding to the RGB feature IR is... r With the corresponding row value matrix V r After element-wise addition and normalization, the resulting features are input into the feedforward network. The output features of the feedforward network are then added to the input features of the feedforward network and normalized. After linear mapping, the resulting features are used to obtain the enhanced row features of the RGB IR feature.

[0065] The process of performing row direction feature enhancement in the infrared processing branch only requires replacing the input of the RGB processing branch. That is, the input of the infrared processing branch is infrared feature VI, and after performing the same row direction feature enhancement operation as the RGB processing branch, the corresponding row enhancement feature is obtained. It can be understood that after performing the row direction feature enhancement operation in the infrared processing branch provided in this embodiment of the invention, the following steps are also taken: the row enhancement feature corresponding to infrared feature VI is added to the corresponding elements of the corresponding row value matrix and normalized; the resulting feature is then input into the feedforward network; the output feature of the feedforward network is added to the corresponding elements of the input feature of the feedforward network and normalized; and the resulting feature is then linearly mapped to obtain the enhanced row enhancement feature of infrared feature VI.

[0066] In the RGB processing branch provided in this embodiment of the invention, the process of performing column-direction feature enhancement includes: performing a convolution operation on the enhanced row enhancement features of the RGB feature IR to generate the second original query matrix Q2, the second original key matrix K2, and the second original value matrix V2 of the current input feature; performing column-direction average pooling operations on the second original query matrix Q2 and the second original key matrix K2 respectively, and then performing position encoding to obtain the corresponding column query matrix Q. c and column bond matrix K c And perform column-wise average pooling on the second original value matrix V2 to obtain the column value matrix V. c The process is as follows:

[0067] Q c =PE(Mean h (Q2));

[0068] K c=PE(Mean h (K2));

[0069] V c =Mean h (V2);

[0070] Among them, Mean h This indicates an average pooling operation along the column direction.

[0071] The column-direction attention feature enhancement operation is performed using the following formula to obtain the column-enhanced feature Y. c :

[0072] .

[0073] In this embodiment of the invention, after completing the column direction feature enhancement operation, the RGB processing branch enhances the column feature Y corresponding to the RGB feature IR. c With the corresponding column value matrix V c After adding and normalizing the corresponding elements, the resulting features are input into the feedforward network. The output features of the feedforward network are added and normalized to the corresponding elements of the input features of the feedforward network. The resulting features are then linearly mapped to obtain the enhanced column features of the RGB feature IR.

[0074] The column-direction feature enhancement operation in the infrared processing branch only requires replacing the input of the RGB processing branch. That is, after performing the same column-direction feature enhancement operation on the enhanced row feature of infrared feature VI as on the RGB processing branch, the corresponding column-direction enhanced feature is obtained. It can be understood that after performing the column-direction feature enhancement operation in the infrared processing branch provided in this embodiment, the following steps are also taken: the enhanced row feature corresponding to infrared feature VI is added to and normalized with the corresponding elements of the column value matrix, and the resulting feature is input into the feedforward network; the output feature of the feedforward network is added to and normalized with the corresponding elements of the input feature of the feedforward network, and then the resulting feature is linearly mapped to obtain the enhanced column-direction enhanced feature of infrared feature VI.

[0075] In the RGB processing branch provided in this embodiment of the invention, the specific process of obtaining RGB enhanced features includes: performing a convolution operation on the original input RGB features to obtain a third original query matrix Q3, a third original key matrix K3, and a third original value matrix V3; concatenating the third original query matrix Q3, the third original key matrix K3, and the third original value matrix V3, and then sequentially performing a depthwise separable convolution operation and a point convolution operation to obtain the RGB local features F. RGB As shown in the following formula:

[0076] F RGB=PWConv(DWConv([Q3,K3,V3]));

[0077] Where DWConv represents depthwise separable convolution, and PWConv represents pointwise convolution operation;

[0078] This will enhance the row enhancement feature Y r '、Enhance column enhancement feature Y' c After adding the corresponding elements of the third original value matrix V3, the RGB local feature F is then used. RGB The summed features are then subjected to gating adaptive modulation following the sigmoid activation operation. Specifically, the summed features are compared with the RGB local features F. RGB Element-wise multiplication is performed, and the sigmoid activation function is used to control the activation of the multiplied features, resulting in the final RGB enhanced feature Y output by the RGB processing branch. RGB The above process can be represented by the following formula:

[0079] Y RGB =δ(F RGB ×(Y r '+Y c '+V3));

[0080] Here, δ represents sigmoid activation. Through gated adaptive modulation, salient structural regions are enhanced while redundant background is effectively suppressed.

[0081] The process of obtaining infrared enhancement features in the infrared processing branch only requires replacing the input of the RGB processing branch. Specifically, convolution is performed on the original input infrared features to obtain the corresponding third original query matrix, third original key matrix, and third original value matrix. These matrices are then concatenated and sequentially subjected to depthwise separable convolution and dot-matrix operations to obtain local infrared features. The enhanced row and column features of the infrared features are then added element-wise to the corresponding elements of the third original value matrix. This summed feature is then multiplied element-wise with the local infrared features. The sigmoid activation function is used to activate the multiplied feature, resulting in the final output infrared enhancement feature Y of the infrared processing branch. VI .

[0082] In the cross-modal difference enhancement perception module, RGB enhancement features and infrared enhancement features are compressed and fused along the channel direction to obtain global semantic features that simultaneously contain global information from both RGB and infrared features. Convolution operations at different scales are performed on the RGB and infrared enhancement features respectively to obtain RGB multi-scale features and infrared multi-scale features. The global semantic features are then combined with the RGB and infrared multi-scale features and weighted to obtain RGB cross-modal weights and infrared cross-modal weights. These weights are then assigned to the RGB and infrared enhancement features respectively, and the corresponding elements of the two features are added together to obtain the fused feature at the current scale. The cross-modal difference enhancement perception module designed in this invention can effectively enhance the complementary information of the two modalities in the target region while suppressing modality-specific noise.

[0083] In some embodiments, the process of obtaining global semantic features includes: performing linear mapping on the RGB enhancement features and the infrared enhancement features respectively, adding the corresponding elements of the two features and then performing linear mapping again to complete the fusion of the RGB enhancement features and the infrared enhancement features; performing linear mapping and sigmoid activation on the fused features, and then performing compression operation on the fused features in the channel direction to obtain global semantic features.

[0084] The cross-modal difference enhancement sensing module provided in this embodiment of the invention is as follows: Figure 3 As shown. The RGB enhancement feature Y... RGB and infrared enhancement feature Y VI After channel-direction compression, the corresponding elements of the two compressed features are added together. The resulting features are then sequentially processed through linear mapping and sigmoid activation to obtain the global semantic features G, as shown in the following equation:

[0085] G=δ(Linear(f c (Y RGB )+f c (Y VI )));

[0086] Among them, f c The compression operation represents the channel direction. In this embodiment of the invention, a linear mapping method is specifically used for channel compression. "Linear" represents the linear mapping operation. The global semantic feature G obtained here is used for subsequent modulation of the local difference weights.

[0087] Two different scale convolution operations were performed on the RGB enhancement features and the infrared enhancement features respectively to obtain the corresponding RGB multi-scale features D. RGB and infrared multi-scale features D VI The process is as follows:

[0088] DRGB =Conv k1 (Y RGB )+Conv k2 (Y RGB );

[0089] D VI =Convk1(Y VI )+Conv k2 (Y VI );

[0090] Among them, Conv k1 With Conv k2 This represents two convolution operations at different scales.

[0091] The global semantic feature G is compared with the RGB multi-scale feature D. RGB and infrared multi-scale features D VI After adding corresponding elements and applying sigmoid activation, the RGB multi-scale features and infrared multi-scale features are combined and weights are extracted, resulting in the corresponding RGB cross-modal weight W. RGB and infrared cross-modal weights W VI The process is as follows:

[0092] W RGB =δ(G+D RGB );

[0093] W VI =δ(G+D VI );

[0094] RGB crossmodal weights W RGB and infrared cross-modal weights W VI Assign RGB enhancement features Y respectively RGB and infrared enhancement feature Y VI Then, the corresponding elements of the two obtained features are added together and activated by sigmoid to obtain the fused feature Y at the current scale. fuse The process is as follows:

[0095] Y fuse =δ(W RGB ×Y RGB +W VI ×Y VI ).

[0096] The cross-modal difference enhancement perception module provided by this invention focuses on the efficient mining of cross-modal complementary information. By explicitly modeling the differences between visible light and infrared modes in brightness response, texture details and semantic expression, it guides the network to adaptively focus on the stable and significant response of the infrared mode in low-light environments, while supplementing the fine-grained structural and semantic information contained in the visible light mode. This enables more discriminative multimodal collaborative perception in complex lighting and extreme environments, improving overall detection performance and robustness.

[0097] In remote sensing image target detection, due to the large imaging scale, small vehicle target size, and complex background, attention mechanisms relying solely on spatial neighborhood modeling are insufficient to fully characterize the statistical differences between the target and the background. This is especially true in infrared or low-contrast visible light scenes, where the spatial structural information of the target is often insignificant, but its feature responses exhibit stronger discriminative power at the global distribution level. This invention designs a cross-stage partial bottleneck with a soft histogram-aware transformer block (CSP-HTB) and integrates it into the neck network. The CSP-HTB explicitly models global intensity distribution statistics using a differentiable soft histogram and utilizes this information to guide the spatial attention mechanism, thereby achieving distribution-aware feature enhancement and improving vehicle detection accuracy in remote sensing scenes. Specifically, in some embodiments, the neck network obtains a soft histogram of fused features through the CSP-HTB and uses this soft histogram to enhance the currently fused features. In the process of obtaining the soft histogram of fused features: the input features are aggregated along the channel dimension and expanded along the spatial dimension; the elements in the processed features are binned and statistically analyzed using Gaussian kernels to obtain the soft histogram.

[0098] In some embodiments, the process of using soft histograms to enhance the features obtained by the current fusion includes: projecting the soft histogram onto the feature space to obtain soft histogram encoding; performing convolution operations on the input features to generate multiple query matrices and multiple value matrices; and calculating the distribution-guided weights corresponding to each query matrix using the following formula:

[0099] ;

[0100] Where, α i Q represents the i-th query matrix. i The corresponding distribution guides the weights, where C represents the number of channels in the input feature, and H... d t represents the number of query matrices. iThis represents the components corresponding to the i query matrices in the soft histogram encoding t; based on the weights guided by each distribution, the corresponding value matrices are weighted and modulated, and then the modulated key matrices are concatenated; the obtained features are input into the feedforward network and then connected with the input feature residuals of the feedforward network to obtain the output features.

[0101] In some embodiments, in the neck network, the soft histogram-aware transformer block further includes: a preprocessing operation on the input fused features, wherein: multiple consecutive residual operations are performed on the input fused features, and the corresponding elements of the features at different scales are added together to complete the preprocessing operation on the input fused features. It can be understood that in the soft histogram-aware transformer block, a soft histogram is obtained from the preprocessed fused features, then the soft histogram of the current feature is obtained, and then the soft histogram is used to enhance the preprocessed fused features.

[0102] The soft histogram sensing transformer block provided in this embodiment of the invention is as follows: Figure 4 As shown, its feature processing specifically includes processing the input fused features as follows: Figure 5 The preprocessing operation is shown, and the soft histogram of the preprocessed fused features is obtained. The soft histogram is then used to enhance the preprocessed fused features.

[0103] The preprocessing operations for the input fused features specifically include: first, performing a convolution operation on the input fused features, and then performing multiple consecutive residual operations on the convolutional features to obtain features at different scales. In each residual operation, after performing two consecutive convolution operations and batch normalization on the input features, the resulting features are concatenated with the feature residuals after the first convolution to obtain the output features of the current residual operation. The output features of the multiple residual operations are then concatenated with the residuals of the convolutional fused features to complete the preprocessing operations for the input fused features.

[0104] In the soft histogram of the preprocessed fused features, a soft histogram strategy is used to perform soft binning statistics on the entire feature image. To avoid the non-differentiability problem of bin allocation in traditional histogram operations, this invention uses a Gaussian kernel as the model for each bin, outputting a differentiable global distribution vector. In this embodiment, the process of obtaining the soft histogram of the preprocessed fused features specifically includes:

[0105] For input features Aggregation along the channel dimension and expansion along the spatial dimension are performed as follows:

[0106] ;

[0107] Among them, X bThe current processed features are represented by C, B, H, W, and X. c This represents the element matrix of the c-th channel in the input feature X, where reshape represents the scaling operation of the spatial dimensions; the currently processed feature X b It is a two-dimensional feature with a size of B×(H×W);

[0108] For the b-th batch, the global distribution vector of the k-th histogram interval is:

[0109] ;

[0110] in, Let represent the global distribution vector of the k-th histogram interval in the b-th sample. Let x represent the b-th sample. b The j-th element, μ k σ represents the center of the k-th bin (i.e., the mean of the elements in the k-th bin), and σ represents the smoothing coefficient (i.e., the standard deviation of the elements in the k-th bin), used to control the flexibility of the distribution response. The global distribution vector... The combination yields the complete global distribution vector, which is now a two-dimensional vector with a size of B×(H×W).

[0111] In this embodiment of the invention, the obtained global distribution vector is also... Normalize the following formula:

[0112] ;

[0113] Where K represents the total number of bins k, and ε represents a sufficiently small constant to prevent the denominator from being zero. Then, the normalized global distribution vector... The combination yields the complete global distribution vector, which is now a two-dimensional vector with a size of B×(H×W).

[0114] In this embodiment of the invention, the process of enhancing the features obtained by the current fusion using soft histograms specifically includes:

[0115] Convolution and layer normalization operations are performed on the complete global distribution vector h to project the soft histogram onto the feature space, obtaining the soft histogram encoding t, as shown in the following equation:

[0116] t=LN(W h ×h);

[0117] Among them, W h The weights represent the convolution operation weights, and LN represents the layer normalization operation.

[0118] The input feature X is subjected to a convolution operation with a kernel size of 1×1 to generate H. d A query matrix Q i H d Key matrix K i and H d Value matrix V i Unlike traditional self-attention, this invention does not directly calculate the similarity between the query matrix and the key matrix. Instead, it introduces distributed guided weights to guide the spatial features. Specifically, the distributed guided weights corresponding to each query matrix are calculated using the following formula:

[0119] ;

[0120] Where, α i Q represents the i-th query matrix. i The corresponding distributed guided weights, t i This represents the components corresponding to the i query matrices in the soft histogram encoding t;

[0121] Guide weights t for each distribution i With the corresponding value matrix V i Multiplication is performed to guide the weights according to each distribution, weighting and modulating the corresponding value matrices. Then, the modulated key matrices are concatenated through channels and then convolved. The process is as follows:

[0122] Y1=Wo×Concat(t i1 ×V i1 ,t2×V2,...,t i ×V i ,...,t Hd ×V Hd );

[0123] Where Y1 represents the currently obtained features, Concat represents the channel concatenation operation, and Wo represents the weights of the current convolution;

[0124] The obtained feature Y1 is normalized and then input into the feedforward network, and then connected with the feature Y1 residual to obtain the output feature Y. out The above process is as follows:

[0125] Y out =Y1+FFN(LN(Y1));

[0126] Here, FFN represents a feedforward network.

[0127] This invention embeds the soft histogram sensing transformer into the top-down and bottom-up feature pyramid model to complete feature enhancement. It utilizes the mechanism of differentiable soft histograms to globally model the statistical information of intensity distribution, effectively suppressing remote sensing background interference and improving the perception capability of small targets such as vehicles in a large remote sensing background.

[0128] This invention also provides an RGB and infrared fusion detection method for complex visual environments, combining... Figure 1 and Figure 6 ,include:

[0129] S1: Obtain the dataset and preprocess it to obtain the training set; the dataset includes RGB images, corresponding infrared images, and target labels in the RGB and infrared images.

[0130] In this embodiment of the invention, the DroneVehicle remote sensing vehicle dataset and the VEDAI remote sensing vehicle dataset are specifically used. Image preprocessing operations, including mosaic, copy-paste, random perspective, blending, and random HSV format transformation, are performed on the datasets. The DroneVehicle dataset is a large-scale UAV aerial vehicle dataset collected and labeled by Tianjin University, consisting of 28,439 pairs of visible-infrared images. It includes five types of vehicles: cars, trucks, buses, vans, and box trucks. It covers various scenes from daytime to nighttime, including residential areas, urban roads, and parking lots. Through the differential redundancy features of visible-infrared cross-modal images, it provides support for smart city traffic management and disaster relief. VEDAI is constructed by cropping high-resolution wide-swath aerial images taken at a fixed flight altitude by the Automated Geographic Reference Center (AGRC) in Utah. The pixel size is approximately 16000×16000, corresponding to a ground spatial resolution of about 12.5cm. The cropped images are uniformly sized to 1024×1024 or 512×512, and contain both visible light (RGB) and infrared (IR) imaging information. The VEDAI dataset contains 1246 small-scale sample images with diverse scene types, covering various typical aerial observation environments such as natural grasslands, roads, mountainous areas, and urban scenes. Seven different types of vehicle targets with varying shapes and functions are labeled, including cars, pickup trucks, RVs, and trucks. This paper's experimental task focuses on multiple vehicle targets within this dataset.

[0131] S2: Construct an RGB and infrared fusion detection system for complex visual environments as provided in this invention.

[0132] S3: Train the detection system constructed in step S2 using the dataset from step S1 to obtain the target detection model.

[0133] Current deep learning methods for object detection typically use cross-entropy loss to obtain label loss, allowing the model to learn label information. However, in remote sensing vehicle detection or small object detection, target categories are extremely imbalanced, with a single category having an excessively large weight, affecting detection performance. This invention proposes a category-adaptive perceptual loss to improve training stability and enhance attention to difficult or rare samples, thereby improving the detection accuracy of rare vehicle categories. Specifically, this invention adaptively assesses sample difficulty by maintaining category-aware statistics for positive and negative samples. In some embodiments, the following loss function is used to train the detection model;

[0134] L total =λ1L cael +λ2L box +λ3L dfl ;

[0135] Among them, L total L represents the loss function for training the detection model. cael L represents the adaptive sensing loss. box L represents the bounding box loss. dfl λ represents the distribution focus loss, and λ1, λ2, and λ3 represent the loss weights.

[0136] Among them, the adaptive sensing loss L cael for:

[0137] ;

[0138] in, This indicates that the current detection model outputs the predicted detection result for the m-th sample. This indicates the corresponding actual test result. Let N represent the true detection result of the nth target in the mth sample, where N represents the total number of targets in the mth sample. Indicates the difficulty level. The difficulty coefficient φ should not be too large, otherwise it will cause gradient explosion and lead to training failure. This is because the prediction detection results... The coefficients for difficult samples are larger than those for easy samples, thus amplifying the difficulty of the class of samples. Bounding box loss L... box for:

[0139] ;

[0140] Where, ω m CIoU represents the weight of the m-th sample. m This represents the CIoU (Complete-IoU) of the m-th sample.

[0141] Distribution focus loss L dflfor:

[0142] ;

[0143] in, This represents the true bounding box coordinates of the detected target in the m-th sample. and These represent the left and right boundaries of the detected target, respectively. and Let represent the predicted left and right boundary probabilities of the detected target, respectively, and CE represent the cross-entropy loss.

[0144] The model training using the loss function provided by this invention solves the problem of difficulty in detecting targets of a small number of vehicle classes due to severe imbalance in vehicle class.

[0145] S4: Input the RGB image and infrared image to be detected into the target detection model obtained in step S3 to obtain the target detection result.

[0146] To verify that the system and method provided by this invention can effectively identify vehicle targets, this embodiment of the invention specifically selects multiple test images from the DroneVehicle remote sensing vehicle dataset, performs vehicle target detection on these test images using existing methods, and compares the detection results with those obtained by the system and method provided by this invention. The existing methods used in this embodiment include both unimodal and multimodal methods. Unimodal methods include YOLOV12_x, RTDETRv2_pr34, and DEIMv2_HgB0, while multimodal methods include DEYOLO_T, MRT-DETR_pr50, and MM-DETR_pr101. The comparison results are as follows: Figure 7 As shown. In Figure 7 In particular, the first and third images in low-light scenes show that the single-modal detection method performs poorly in visible light images, resulting in numerous false positives and false negatives. In contrast, the multimodal detection method can comprehensively utilize redundant information from multiple modalities, improving detection performance in low-light conditions. Furthermore, the system provided by this invention exhibits a very low false positive rate for small target detection compared to other multimodal detectors. This is due to the model's emphasis on loss design and the comprehensive utilization of statistical domain features, which significantly enhances the model's ability to model small vehicle targets in low light.

[0147] Furthermore, this invention employs mAP and AP (Average Precision) as objective evaluation metrics to objectively analyze existing methods and the method provided by this invention for vehicle target detection on multiple test images. mAP is a comprehensive metric obtained by averaging the average precision (AP) across all categories. Its calculation uses an integral method to obtain the area enclosed by the precision-recall curve and the coordinate axis for all categories. mAP is:

[0148] ;

[0149] Where Cl represents the total number of categories, r represents the recall rate, and P represents the precision rate.

[0150] Commonly used metrics in mAP include mAP 50 and mAP 50:95 mAP 50 This represents the mean average accuracy when the cross-union ratio is 0.5, while mAP 50:95 This represents the mean of the average precision of 10 different mean values ​​from 0.5 to 0.95 for the cross-union ratio (CUP). mAP 50:95 and mAP 50 They are respectively:

[0151] ;

[0152] ;

[0153] Where s represents the computed index, This represents the average accuracy of the intersection-union ratio (IU) of class cl targets at 0.5.

[0154] Furthermore, to more accurately evaluate the model's performance on targets of different scales, this embodiment of the invention divides the target into different intervals according to its size: the size is ( The objective of (∞, 322) is the small objective AP. S A target with dimensions [322, 962) is a medium-sized target (AP). M The target with dimensions [962, +∞) is the large target AP. L This is used to measure the model's detection performance on small, medium, and large targets, respectively. Objective analysis results are as follows: Figure 8 As shown, Figure 8 middle This represents the average precision with an intersection-over-union ratio (IoU) of 0.5 for the car category. This represents the average accuracy with an intersection-over-union ratio of 0.5 for the truck category. This represents the average precision with an intersection-over-union ratio (IoU) of 0.5 for the bus category. This represents the average precision with an intersection-over-union ratio of 0.5 for the target category of vans. This represents the average precision for a crossover ratio (CRO) of 0.5, targeting the van category. From Figure 8 It can be seen that the method provided by this invention is effective across five different vehicle categories. , and They all achieved optimal results. The method achieved suboptimal performance; based on detection results at three scales (small, medium, and large), the method provided by this invention is optimal for small and medium targets, and suboptimal for large targets; in terms of overall detection performance, mAP50 is optimal, surpassing the suboptimal DEYOLO_T detector by 1%, mAP50. 50:95 The difference from the optimal result is only 0.6%, which fully verifies the superiority of the system and method provided by this invention.

[0155] It should be understood that the various forms of processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this invention disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this invention can be achieved, and this is not limited herein.

[0156] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. An RGB and infrared fusion detection system for complex visual environments, characterized in that, include: The backbone network performs multi-scale feature extraction on the input RGB and infrared images respectively, obtaining multi-scale RGB and infrared features respectively; The feature fusion network performs local feature enhancement on multi-scale RGB and infrared features respectively, and then performs feature enhancement on the two enhanced features in the row and column directions respectively to obtain multi-scale RGB enhanced features and infrared enhanced features. The RGB enhancement features and infrared enhancement features at the corresponding scales are fused to obtain multi-scale fused features; The neck network fuses multi-scale features in a bottom-up and top-down manner to obtain multi-scale enhanced features. After each fusion, a soft histogram of the current fused features is obtained, and the soft histogram is used to enhance the features obtained from the current fusion. The detection network inputs multi-scale enhancement features into the corresponding detection heads to obtain target detection results for targets of different sizes.

2. The RGB and infrared fusion detection system for complex visual environments according to claim 1, characterized in that, The feature fusion network includes an intramodal structure enhancement module and a cross-modal difference enhancement perception module, wherein: The intramodal structure enhancement module includes parallel and structurally consistent RGB processing branches and infrared processing branches. In the RGB processing branch or infrared processing branch: continuous row-direction attention feature enhancement operations are performed on the RGB features or infrared features to obtain row-enhanced features; then column-direction attention feature enhancement operations are performed on the row-enhanced features to obtain column-enhanced features; the obtained row-enhanced features and column-enhanced features are fused with the corresponding original input features to obtain RGB-enhanced features or infrared-enhanced features. In the cross-modal difference enhancement perception module: After compressing and fusing the RGB enhancement features and infrared enhancement features in the channel direction, a global semantic feature containing global information from both RGB and infrared features is obtained; convolution operations at different scales are performed on the RGB enhancement features and infrared enhancement features respectively to obtain RGB multi-scale features and infrared multi-scale features; after combining the global semantic feature with the RGB multi-scale feature and infrared multi-scale feature and extracting weights, RGB cross-modal weights and infrared cross-modal weights are obtained respectively; after assigning the RGB cross-modal weights and infrared cross-modal weights to the RGB enhancement feature and infrared enhancement feature respectively, the corresponding elements of the two features are added together to obtain the fused feature at the current scale.

3. The RGB and infrared fusion detection system for complex visual environments according to claim 2, characterized in that, The process of performing row-direction feature enhancement includes: linearly mapping the input features to generate the original query matrix, original key matrix, and original value matrix of the current input features; compressing the spatial dimension of the original query matrix, original key matrix, and original value matrix in the row direction to obtain the corresponding row query matrix, row key matrix, and row value matrix; and performing attention enhancement operations on the row query matrix, row key matrix, and row value matrix to obtain the row-enhanced features. The process of performing column-direction feature enhancement includes: performing linear mapping on the input features to generate the original query matrix, original key matrix, and original value matrix of the current input features; compressing the spatial dimension of the original query matrix, original key matrix, and original value matrix in the column direction to obtain the corresponding column query matrix, column key matrix, and column value matrix; and performing attention enhancement operation on the column query matrix, column key matrix, and column value matrix to obtain the column-enhanced features.

4. The RGB and infrared fusion detection system for complex visual environments according to claim 3, characterized in that, In the process of fusing the obtained row-enhanced features and column-enhanced features with the corresponding original input features: The original query matrix, original key matrix, and original value matrix are concatenated and then subjected to depthwise separable convolution to model the local relationships in the input features and obtain local features. After adding the corresponding elements of the row enhancement features, column enhancement features, and original value matrix, the added features are then multiplied element-wise with the local features to obtain the output enhancement features.

5. The RGB and infrared fusion detection system for complex visual environments according to claim 2, characterized in that, The process of obtaining global semantic features includes: After linearly mapping the RGB enhancement features and infrared enhancement features respectively, the corresponding elements of the two features are added together and then linearly mapped again to complete the fusion of the RGB enhancement features and infrared enhancement features. After linear mapping and sigmoid activation, the fused features are compressed along the channel direction to obtain global semantic features.

6. The RGB and infrared fusion detection system for complex visual environments according to claim 1, characterized in that, In the neck network, a soft histogram of the fused features is obtained by sensing the transformer block through a soft histogram, and the soft histogram is used to enhance the features obtained by the current fusion. In the process of obtaining the soft histogram of fused features: The input features are aggregated along the channel dimension and expanded along the spatial dimension. The elements in the processed features are binned and statistically analyzed using Gaussian kernels to obtain a soft histogram.

7. The RGB and infrared fusion detection system for complex visual environments according to claim 6, characterized in that, In the neck network, the soft histogram perceptual transformer block also includes a preprocessing operation on the fused features of the input, wherein: multiple consecutive residual operations are performed on the fused features of the input, and the corresponding elements of the features at different scales are added together.

8. The RGB and infrared fusion detection system for complex visual environments according to claim 1, characterized in that, The process of using soft histograms to enhance the features obtained from the fusion includes: The soft histogram is projected onto the feature space to obtain the soft histogram encoding; The input features are convolved to generate multiple query matrices and multiple value matrices. The distribution-guided weights corresponding to each query matrix are calculated using the following formula: ； Where, α i Q represents the i-th query matrix. i The corresponding distribution guides the weights, where δ represents sigmoid activation, C represents the number of channels in the input feature, and H... d t represents the number of query matrices. i This represents the components corresponding to the i query matrices in the soft histogram encoding t; Based on the guiding weights of each distribution, the corresponding value matrix is ​​weighted and modulated, and then the modulated key matrix is ​​concatenated. The obtained features are input into the feedforward network and then connected with the input feature residuals of the feedforward network to obtain the output features.

9. A method for RGB and infrared fusion detection in complex visual environments, characterized in that, include: S1: Obtain the dataset and preprocess it to obtain the training set; the dataset includes RGB images, corresponding infrared images, and target labels in the RGB and infrared images; S2: Construct an RGB and infrared fusion detection system for complex visual environments as described in any one of claims 1 to 8; S3: Train the detection system constructed in step S2 using the dataset from step S1 to obtain the target detection model; S4: Input the RGB image and infrared image to be detected into the target detection model obtained in step S3 to obtain the target detection result.

10. The RGB and infrared fusion detection method for complex visual environments according to claim 9, characterized in that, The detection model is trained using the following loss function in step S3; L total =λ1L cael +λ2L box +λ3L dfl ; Among them, L total L represents the loss function for training the detection model. cael L represents the adaptive sensing loss. box L represents the bounding box loss. dfl Let λ1, λ2, and λ3 represent the distribution focus loss, and let λ1, λ2, and λ3 represent the loss weights. The adaptive sensing loss is: ； in, This indicates that the current detection model outputs the predicted detection result for the m-th sample. This indicates the corresponding actual test result. Let N represent the true detection result of the nth target in the mth sample, where N represents the total number of targets in the mth sample. Indicates the difficulty level. ; The bounding box loss is: ； Where, ω m CIoU represents the weight of the m-th sample. m This represents the CIoU of the m-th sample; The distribution focus loss is: ； in, This represents the true bounding box coordinates of the detected target in the m-th sample. and These represent the left and right boundaries of the detected target, respectively. and Let represent the predicted left and right boundary probabilities of the detected target, respectively, and CE represent the cross-entropy loss.

Images