Multispectral fusion target detection method and system considering similarity and complementarity

Abstract

The invention discloses a multi-spectral fusion target detection method and system considering similarity and complementarity, and the method comprises the following steps: obtaining a visible light image and an infrared image, and carrying out the preprocessing of the visible light image and the infrared image; constructing a multispectral fusion target detection model, respectively extracting multispectral features of the visible light image and the infrared image by the multispectral fusion target detection model, carrying out feature fusion on the multispectral features of the visible light image and the infrared image according to the similarity and complementarity of the two multispectral features, and finally, carrying out feature fusion on the multispectral features of the visible light image and the infrared image. Predicting the category and bounding box of the target according to the fused multispectral features; and according to the category and the bounding box of the target predicted by the multispectral fusion target detection model, setting a loss function to optimize the prediction result of the multispectral fusion target detection model, and outputting the final optimized category and bounding box of target detection. The method provided by the invention can effectively improve the multi-spectral fusion target detection accuracy and fusion efficiency.

Description

Technical Field

[0001] The present invention relates to the technical field of multispectral fusion target detection, and in particular to a multispectral fusion target detection method and system that takes both similarity and complementarity into consideration. Background Art

[0002] Visible light imaging relies on the reflection of visible light from objects to create images. These images contain rich texture and color information, clearly displaying the details and appearance of objects. This aligns with human visual perception and plays a crucial role in target recognition and scene understanding. For example, in security monitoring, visible light images can clearly show a person's appearance, clothing, and surrounding details, facilitating accurate target identification and scene interpretation.

[0003] Infrared imaging uses the thermal radiation characteristics of an object to create an image. It is unaffected by lighting conditions and can function effectively in low-visibility environments, such as at night and in inclement weather (such as fog, rain, and snow). It can detect differences in thermal distribution within an object, thereby revealing targets hidden in the background. This is crucial for applications such as military reconnaissance and fire monitoring. For example, in forest fire monitoring, infrared imagery can quickly detect the location and extent of a fire, even in dense smoke obscuration.

[0004] Due to the differences in the imaging principles of visible light and infrared images, they each have unique advantages and limitations. Fusion of the two can fully leverage their complementary characteristics, improve the integrity and accuracy of image information, and provide more comprehensive image data support for various applications.

[0005] Currently, visible light images face imaging difficulties under extreme conditions such as insufficient illumination, resulting in reduced or even ineffective target feature recognition. Therefore, existing methods employ multispectral fusion target detection technology to fuse infrared and visible light images, enabling robust and stable target detection in a variety of scenarios. Furthermore, the development and advancement of deep learning and convolutional neural networks have provided strong support for the advancement of multispectral fusion technology, gradually establishing convolutional neural networks as the mainstream fusion method in the current field.

[0006] However, current fusion methods tend to focus on fusing similar information between multispectral features, while ignoring the mining and fusion of multispectral similarity and complementary information, resulting in low multispectral fusion target detection accuracy and poor model robustness. Summary of the Invention

[0007] In order to overcome the problem that the existing technology ignores the mining and fusion of multispectral similarity and complementary information, resulting in low multispectral fusion target detection accuracy and poor model robustness, the purpose of the present invention is to propose a multispectral fusion target detection method and system that takes into account both similarity and complementarity, which can effectively mine and fuse multispectral similarity and complementary information, thereby improving the multispectral fusion target detection accuracy and model robustness.

[0008] To achieve the purpose of the present invention, the present invention adopts the following technical solutions: A multispectral fusion target detection method taking into account both similarity and complementarity, the method comprising the following steps: Acquire a visible light image and an infrared image, and preprocess the visible light image and the infrared image; Constructing a multispectral fusion target detection model, and inputting the preprocessed visible light image and infrared image into the multispectral fusion target detection model; The multispectral fusion target detection model extracts multispectral features from visible light images and infrared images respectively, and fuses the multispectral features of visible light images and infrared images based on their similarity and complementarity. Finally, the target category and bounding box are predicted based on the fused multispectral features. According to the target category and bounding box predicted by the multispectral fusion target detection model, a loss function is set to optimize the prediction results of the multispectral fusion target detection model, and the final optimized target detection category and bounding box are output.

[0009] In the above technical scheme, by preprocessing the acquired visible light images and infrared images, the reliability and practicality of the data can be effectively improved; the multispectral fusion target detection model constructed is used to extract multispectral features of the preprocessed visible light images and infrared images, and the multispectral features of the visible light images and infrared images are fused according to the similarity and complementarity of the two multispectral features. Finally, the category and bounding box of the target are predicted based on the fused multispectral features, which can effectively mine and fuse the similarity and complementary information of the multispectrals; setting a loss function to optimize the prediction results of the multispectral fusion target detection model, and outputting the final optimized target detection category and bounding box, can improve the accuracy of multispectral fusion target detection and the robustness of the model, while improving the fusion efficiency of multispectral features, thereby improving the target detection performance in multiple scenarios.

[0010] Furthermore, the process of preprocessing the visible light image and the infrared image includes: The acquired visible light images and infrared images are randomly cropped and paired, and data enhancement is performed on the paired visible light images and infrared images.

[0011] In the above technical solution, by performing random cropping and pairing operations on the acquired visible light images and infrared images, the reliability and practicality of the data can be effectively improved. By performing data enhancement on the paired visible light images and infrared images, the requirements for data calculation can be effectively reduced, thereby improving the efficiency of data processing.

[0012] Furthermore, the process of predicting the category and bounding box of the target based on the fused multispectral features includes: The constructed multispectral fusion target detection model includes several feature encoders connected one by one, several similarity and complementary cross-attention fusion modules, several feature decoders and several detection heads, among which: Several feature encoders are used to extract multispectral features of multiple scales of visible light images and infrared images respectively; Each similarity and complementary cross-attention fusion module cross-fuses the multispectral features of visible light images and infrared images of different scales to obtain fused features; Each feature decoder performs feature decoding based on the fusion features output by the corresponding connected similarity and complementary cross-attention fusion modules; Each detection head predicts the target category and bounding box based on the decoded fusion features output by the corresponding connected feature decoder, and outputs the prediction result.

[0013] Furthermore, the similarity and complementary cross-attention fusion modules cross-fuse the multispectral features of visible light images and infrared images of different scales respectively, and the process includes: According to the similarity of the multispectral features of visible light images and infrared images, the feature distribution weights of the multispectral features of visible light images and infrared images in the self-modality are calculated respectively, and the similarity of the multispectral features of visible light images and infrared images is enhanced respectively according to the feature distribution weights in the self-modality; According to the complementarity of the multispectral characteristics of visible light images and infrared images, the feature attention weights of the multispectral features of visible light images and infrared images in the self-modality are calculated respectively, and the complementary information interaction of the multispectral features of visible light images and infrared images is performed according to the feature attention weights in the self-modality; By performing similarity enhancement and complementary information interaction on the multispectral features of visible light images and infrared images respectively, the multispectral features of visible light images and infrared images of different scales are cross-fused.

[0014] Furthermore, the process of calculating the feature distribution weights of the multispectral features of the visible light image and the infrared image in the self-modality includes: Calculate the feature distribution weight of the multispectral features of the visible light image in the self-modal state. The expression is:

[0015] Calculate the feature distribution weight of the multispectral features of the infrared image in the self-modal state. The expression is:

[0016] According to the feature distribution weights of the multispectral features of the visible light image and the infrared image in the self-modal state, the cosine similarity of the multispectral features of the visible light image and the infrared image is calculated. The expression is:

[0017] According to the cosine similarity of the multispectral features of the visible light image and the infrared image, the comprehensive similarity feature distribution of the multispectral features of the visible light image and the infrared image is calculated. The expression is:

[0018] According to the comprehensive similarity feature distribution of multispectral features of visible light image and infrared image, Gaussian filtering is used Further smooth the similar areas, enhance the areas of common interest of the multispectral features of the visible light image and the infrared image, suppress the features that the multispectral features of the visible light image and the infrared image do not care about, and complete the similarity enhancement of the multispectral features of the visible light image and the infrared image. The expression is:

[0019]

[0020]

[0021] in, Indicates that it has parameters Sigmoid activation function, Represents the multispectral features of visible light images, Represents the multispectral characteristics of infrared images, represents element-wise product, represents the cosine similarity, Represents the comprehensive similarity feature distribution of multispectral features of visible light images and infrared images, Multispectral features of visible light images after similarity enhancement, Represents the multispectral features of the infrared image after similarity enhancement.

[0022] Furthermore, the process of calculating the feature attention weights of the multispectral features of the visible light image and the infrared image in the self-modal state includes: SimAM attention is used to calculate the feature attention weights of the multispectral features of visible light images and infrared images in the self-modal state, and the expression is:

[0023]

[0024] in, For the Transformer's multi-head self-attention calculation, the SimAM attention calculation formula is as follows:

[0025]

[0026]

[0027]

[0028] Among them, h, w represent the height and width of the input feature F, F.mean, F.var represent the mean and variance of the input feature F, respectively. As a formula hyperparameter The complementary feature weighted calculation is performed based on the feature attention weights of the multispectral features of the visible light image and the infrared image in the self-modal state. The expression is:

[0029]

[0030] Based on the multispectral features of the visible light image and infrared image after similarity enhancement, combined with the feature attention weights of the multispectral features of the visible light image and infrared image after feature weighting in the self-modality, the fusion of complementary information is completed. The expression is:

[0031]

[0032] in, Represents the feature attention weight of the multispectral features of the visible light image in the self-modal state, Represents the feature attention weight of the multispectral features of the infrared image in the self-modal state, represents multi-head self-attention calculation, Represents the complementary feature information provided by the multispectral features of infrared images, Represents the complementary feature information provided by the multispectral features of infrared images.

[0033] In the above technical scheme, several feature encoders, several similarity and complementary cross-attention fusion modules, several feature decoders and several detection heads in the constructed multispectral fusion target detection model are used to extract multispectral features of the preprocessed visible light images and infrared images, and the multispectral features of the visible light images and infrared images are fused according to the similarity and complementarity of the two multispectral features. Finally, the category and bounding box of the target are predicted based on the fused multispectral features, which can effectively mine and fuse the similarity and complementary information of the multispectral, thereby improving the accuracy of multispectral fusion target detection and the robustness of the model.

[0034] Furthermore, the loss function is set to optimize the prediction results of the multispectral fusion target detection model as follows:

[0035] in, and Represents the influencing factor used to balance the weights between loss functions, represents the bounding box regression loss, represents the target classification loss, represents the target confidence loss.

[0036] Furthermore, the target classification loss and target confidence loss The calculation process includes: Focal loss function is used to calculate target classification loss , the expression is:

[0037] Focal loss function is used to calculate target confidence loss , the expression is:

[0038] in, The label of the true category, Represents the probability when the model predicts category y=1, 、 Respectively represent the preset coefficients, represents the coefficient at time t.

[0039] Furthermore, the bounding box regression loss The calculation expression is:

[0040] in, Represents the preset EIoU loss function, represents the original IoU loss, represents the distance loss of the center point, represents the aspect ratio loss, , A and B both represent borders, and Represents the height and width of the minimum bounding box, represents the Euclidean distance, and the superscript gt represents the true value.

[0041] In the above technical solution, the bounding box regression loss is set , target classification loss and target confidence loss Optimizing the prediction results of the multispectral fusion target detection model and outputting the final optimized target detection category and bounding box can improve the accuracy of multispectral fusion target detection and the robustness of the model.

[0042] A multispectral fusion target detection system that takes both similarity and complementarity into account, the system comprising: A data acquisition module, used for acquiring visible light images and infrared images; A data processing module, configured to pre-process the visible light image and the infrared image; A model building module is used to build a multispectral fusion target detection model and input the preprocessed visible light image and infrared image into the multispectral fusion target detection model; The target detection module is used to extract the multispectral features of visible light images and infrared images respectively using the multispectral fusion target detection model. The multispectral features of visible light images and infrared images are then fused based on their similarity and complementarity. Finally, the target category and bounding box are predicted based on the fused multispectral features. The parameter optimization module is used to set the loss function to optimize the prediction results of the multispectral fusion target detection model according to the target category and bounding box predicted by the multispectral fusion target detection model, and output the final optimized target detection category and bounding box.

[0043] Compared with the prior art, the present invention has the following beneficial effects: The present invention proposes a multispectral fusion target detection method and system that takes into account both similarity and complementarity. By preprocessing the acquired visible light images and infrared images, the reliability and practicality of the data can be effectively improved. The multispectral fusion target detection model constructed is used to extract multispectral features of the preprocessed visible light images and infrared images, and the multispectral features of the visible light images and infrared images are fused according to the similarity and complementarity of the two multispectral features. Finally, the category and bounding box of the target are predicted according to the fused multispectral features, which can effectively mine and fuse the similarity and complementary information of the multispectrals. A loss function is set to optimize the prediction results of the multispectral fusion target detection model, and the final optimized target detection category and bounding box are output, which can improve the accuracy of multispectral fusion target detection and the robustness of the model, while improving the fusion efficiency of multispectral features, thereby improving the target detection performance in multiple scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 A flowchart of the steps of a multispectral fusion target detection method that takes into account similarity and complementarity provided in an embodiment of the present application; Figure 2 A schematic diagram of the structure of the multispectral fusion target detection model provided in the embodiment of the present application; Figure 3 A schematic diagram of the structure of the similarity and complementary cross-attention fusion module provided in an embodiment of the present application; Figure 4 A schematic diagram of the intersection and union relationship between two bounding boxes in the target detection results provided in an embodiment of the present application; Figure 5 A schematic diagram of the structure of a multispectral fusion target detection model based on Yolov5 as the basic model in the experiment provided in the embodiment of this application; Figure 6 A schematic diagram of the structure of a multispectral fusion target detection system that takes into account both similarity and complementarity, provided in an embodiment of the present application. DETAILED DESCRIPTION

[0045] To facilitate understanding of the present invention, the present invention will be described more fully below with reference to the accompanying drawings. Preferred embodiments of the present invention are shown in the accompanying drawings. However, the present invention may be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided to provide a more thorough and comprehensive understanding of the present disclosure.

[0046] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this invention pertains. The terms used in this specification of the present invention are for the purpose of describing specific embodiments only and are not intended to limit the present invention. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0047] Example 1: This embodiment provides a multi-spectral fusion target detection method that takes into account both similarity and complementarity. Figure 1 , the method comprises the following steps: Step S1: Acquire a visible light image and an infrared image, and preprocess the visible light image and the infrared image; Step S2: constructing a multispectral fusion target detection model, and inputting the preprocessed visible light image and infrared image into the multispectral fusion target detection model; Step S3: The multispectral fusion target detection model extracts multispectral features of visible light images and infrared images respectively, and fuses the multispectral features of visible light images and infrared images based on the similarity and complementarity of the two multispectral features. Finally, the target category and bounding box are predicted based on the fused multispectral features. Step S4: According to the target category and bounding box predicted by the multispectral fusion target detection model, a loss function is set to optimize the prediction results of the multispectral fusion target detection model, and the final optimized target detection category and bounding box are output.

[0048] In step S1, the process of preprocessing the visible light image and the infrared image includes: The acquired visible light images and infrared images are randomly cropped and paired, and data enhancement is performed on the paired visible light images and infrared images.

[0049] It can be understood that by performing random cropping and pairing operations on the acquired visible light images and infrared images, the reliability and practicality of the data can be effectively improved. Performing data enhancement on the paired visible light images and infrared images can effectively reduce the requirements for data calculation, thereby improving the efficiency of data processing.

[0050] In step S2, see Figure 2 The constructed multi-spectral fusion target detection model includes several feature encoders connected one by one, several similar and complementary cross attention fusion modules (i.e. SCCA, Similar and Complementary Cross Attention in the figure), several feature decoders and several detection heads.

[0051] In step S3, the process of predicting the target category and bounding box based on the fused multispectral features includes: Step S31: using several feature encoders to extract multispectral features of multiple scales of visible light images and infrared images respectively; Step S32: Each similarity and complementary cross attention fusion module cross-fuses the multispectral features of visible light images and infrared images of different scales to obtain fused features; Step S33: Each feature decoder performs feature decoding based on the fusion features output by the corresponding similarity and complementary cross attention fusion module; Step S34: Each detection head predicts the target category and bounding box based on the decoded fusion features output by the corresponding connected feature decoder, and outputs the prediction result.

[0052] Specifically, see Figure 2 At the model's input, the paired visible light and infrared images are fed into a feature encoder or image feature extraction network (such as ResNet or VGG) to extract multispectral features at different scales (P1, P2, and P3 in the figure). Feeding these multi-scale deep features into the neck network and detection head enables effective detection of objects of different scales. The Similarity and Complementary Cross Attention (SCCA) fusion module in the model is used to cross-fuse multispectral features at different scales, enabling the simultaneous mining and fusion of similar and complementary information between multiple spectra. Finally, a multi-stage detection head is used to predict object categories and bounding boxes at different scales from the multispectral fusion features at different scales.

[0053] In step S33, see Figure 3 The similarity and complementary cross attention fusion modules cross-fuse the multispectral features of visible light images and infrared images of different scales respectively, and the process includes: According to the similarity of the multispectral features of visible light images and infrared images, the feature distribution weights of the multispectral features of visible light images and infrared images in the self-modality are calculated respectively, and the similarity of the multispectral features of visible light images and infrared images is enhanced respectively according to the feature distribution weights in the self-modality; According to the complementarity of the multispectral characteristics of visible light images and infrared images, the feature attention weights of the multispectral features of visible light images and infrared images in the self-modality are calculated respectively, and the complementary information interaction of the multispectral features of visible light images and infrared images is performed according to the feature attention weights in the self-modality; By performing similarity enhancement and complementary information interaction on the multispectral features of visible light images and infrared images respectively, the multispectral features of visible light images and infrared images of different scales are cross-fused.

[0054] As a preferred embodiment, the process of respectively calculating the feature distribution weights of the multispectral features of the visible light image and the infrared image in the self-modality includes: Specifically, Figure 3 The principle block diagram of the CNN-based cross-attention fusion module SCCA is presented in detail. The overall algorithm architecture of the fusion model SCCA is divided into two parts: similarity and complementarity.

[0055] During the similarity information enhancement stage, a simple Sigmoid activation function is first applied to the multispectral features to obtain the feature distribution weights for each modality. The dot product of these weights (i.e., the multispectral features of the visible light image and the infrared image) is then calculated as the distribution of features that are considered important by both modalities. Because the output of the Sigmoid activation function ranges from 0 to 1, it maps the input feature to a weight that represents its importance. Therefore, when the feature weights of the two modalities are multiplied together, areas with higher weights indicate that both features consider this information important. This means that the visible light and infrared features are considered to be similar information that is considered important. The similarity weight distribution is then propagated back to each modality and weighted with the previous features to enhance the similarity information within each modality. This process enhances similar areas of mutual interest and suppresses features that are not of interest to either modality, thereby enhancing the similarity information between the two modalities. The specific calculation process is as follows: Calculate the feature distribution weight of the multispectral features of the visible light image in the self-modal state. The expression is:

[0056] Calculate the feature distribution weight of the multispectral features of the infrared image in the self-modal state. The expression is:

[0057] According to the feature distribution weights of the multispectral features of the visible light image and the infrared image in the self-modal state, the cosine similarity of the multispectral features of the visible light image and the infrared image is calculated. The expression is:

[0058] According to the cosine similarity of the multispectral features of the visible light image and the infrared image, the comprehensive similarity feature distribution of the multispectral features of the visible light image and the infrared image is calculated. The expression is:

[0059] According to the comprehensive similarity feature distribution of multispectral features of visible light image and infrared image, Gaussian filtering is used Further smooth the similar areas, enhance the areas of common interest of the multispectral features of the visible light image and the infrared image, suppress the features that the multispectral features of the visible light image and the infrared image do not care about, and complete the similarity enhancement of the multispectral features of the visible light image and the infrared image. The expression is:

[0060]

[0061]

[0062] in, Indicates that it has parameters Sigmoid activation function, Represents the multispectral features of visible light images, Represents the multispectral characteristics of infrared images, represents element-wise product, represents the cosine similarity, Represents the comprehensive similarity feature distribution of multispectral features of visible light images and infrared images, Multispectral features of visible light images after similarity enhancement, Represents the multispectral features of the infrared image after similarity enhancement.

[0063] As a preferred embodiment, the process of respectively calculating the feature attention weights of the multispectral features of the visible light image and the infrared image in the self-modality includes: In the complementary information interaction stage, the SimAM attention method is used to calculate the weights of the modal features of both parties respectively to quantify the importance of each modality. This process aims to capture the key features within each modality and provide a basis for subsequent information interaction. After the weight calculation is completed, the obtained self-modal weight distribution is used to weight the features of the other modality. This means that the features of the other modality are weighted according to the attention distribution of the self-modality, thereby sharing the degree of attention of the self-modality to the features with the other modality. Similarly, the self-modality will also receive the attention distribution of the other modality to the features, and through this information interaction mechanism, the information complementarity of the two modalities is achieved. The fusion module has a simple structure, lightweight parameters, and good pluggability, which facilitates migration to other models for fusion. The specific calculation process is as follows: Calculate the feature attention weights of the multispectral features of visible light images and infrared images in the self-modal state. The expression is:

[0064]

[0065] in, For the Transformer's multi-head self-attention calculation, SimAM attention is a parameter-free feature attention weight calculation method. The calculation formula is as follows:

[0066]

[0067]

[0068]

[0069] Among them, h, w represent the height and width of the input feature F, F.mean, F.var represent the mean and variance of the input feature F, respectively. as a hyperparameter in the formula.

[0070] Complementary feature weighted calculation: The weight distribution of the self-modality is used to weight the features of the other modality, and a normalization operation is introduced to ensure that the features are on the same scale. The complementary feature weighted calculation is performed based on the feature attention weights of the multispectral features of the visible light image and the infrared image in the self-modality. The expression is:

[0071]

[0072] Fusion of complementary information: Through the information exchange mechanism, the attention information of the self-modality to the features is shared with the other modality, and the feature attention distribution of the other modality is received, thus achieving information complementarity between the modalities. That is, based on the multispectral features of the visible light image and infrared image after similarity enhancement, combined with the feature attention weights of the multispectral features of the visible light image and infrared image after feature weighting in the self-modality, the fusion of complementary information is completed. The expression is:

[0073]

[0074] in, Represents the feature attention weight of the multispectral features of the visible light image in the self-modal state, Represents the feature attention weight of the multispectral features of the infrared image in the self-modal state, represents multi-head self-attention calculation, Represents the complementary feature information provided by the multispectral features of infrared images, Represents the complementary feature information provided by the multispectral features of infrared images.

[0075] In this embodiment, the similarity and complementary cross-attention fusion module SCCA improves the innovation and fusion efficiency of the similarity information enhancement and complementary information interaction stages by introducing an activation function with adaptive parameters, combining innovative methods such as cosine similarity, Gaussian filtering, and multi-head attention mechanism, thereby further improving the effect of multispectral fusion.

[0076] It can be understood that the multispectral fusion target detection model constructed by using several feature encoders, several similarity and complementary cross-attention fusion modules, several feature decoders and several detection heads is used to extract multispectral features of the preprocessed visible light images and infrared images, and the multispectral features of the visible light images and infrared images are fused according to the similarity and complementarity of the two multispectral features. Finally, the category and bounding box of the target are predicted based on the fused multispectral features, which can effectively mine and fuse the similarity and complementary information of the multispectral, thereby improving the accuracy of multispectral fusion target detection and the robustness of the model.

[0077] As a preferred embodiment, in step S4, the loss function is set to optimize the prediction result of the multispectral fusion target detection model as follows:

[0078] in, and Represents the influencing factor used to balance the weights between loss functions, represents the bounding box regression loss, represents the target classification loss, represents the target confidence loss.

[0079] As a preferred embodiment, the target classification loss and target confidence loss The calculation process includes: Specifically, the target classification loss With confidence loss The Focal loss function is designed to improve the sample imbalance problem based on the binary cross entropy (CE) function. As shown in the following formula, for the convenience of expression, the cross entropy formula can be written as .

[0080]

[0081]

[0082] in, The label of the true category, It represents the probability when the model predicts the class y=1.

[0083] The general two-category cross entropy loss will add a Factors, such as , but this method cannot distinguish the difficulty of classification. Therefore, the Focal loss function reduces the weight of simple categories and increases the attention to difficult-to-classify samples, as shown in the following formula:

[0084] Therefore, the Focal loss function is used in the model of this embodiment to calculate the target classification loss With confidence loss , specifically including: Focal loss function is used to calculate target classification loss , the expression is:

[0085] Focal loss function is used to calculate target confidence loss , the expression is:

[0086] in, The label of the true category, Represents the probability when the model predicts category y=1, 、 Respectively represent the preset coefficients, represents the coefficient at time t.

[0087] As a preferred embodiment, the attributes of the two frames and the relationship between them can be as follows: Figure 4 As shown. The intersection of union (IoU) refers to the ratio of the area of ​​the intersection and union of two bounding boxes, which is used to indicate the degree of overlap between the two. For example, Figure 4 The IoU of boxes A and B can be expressed as:

[0088] This model uses the EIoU loss function "Focal and efficient IOU loss for accuratebounding box regression" to calculate the regression loss between the predicted bounding box and the true value box. The EIoU loss function consists of three parts: the original IoU loss , the distance loss of the center point and aspect ratio loss Based on the original IOU loss, the EIoU loss comprehensively considers the offset error between the center point of the predicted box and the true value box and the aspect ratio scale error between the two, directly minimizing the height and width errors between the two, thereby helping the model converge faster and improve accuracy. The calculation expression is:

[0089] in, Represents the preset EIoU loss function, represents the original IoU loss, represents the distance loss of the center point, represents the aspect ratio loss, , A and B both represent borders, and Represents the height and width of the minimum bounding box, represents the Euclidean distance, and the superscript gt represents the true value.

[0090] Understandably, setting the bounding box regression loss , target classification loss and target confidence loss Optimizing the prediction results of the multispectral fusion target detection model and outputting the final optimized target detection category and bounding box can improve the accuracy of multispectral fusion target detection and the robustness of the model.

[0091] In this embodiment, by preprocessing the acquired visible light images and infrared images, the reliability and practicality of the data can be effectively improved; the constructed multispectral fusion target detection model is used to extract multispectral features of the preprocessed visible light images and infrared images, and the multispectral features of the visible light images and infrared images are fused according to the similarity and complementarity of the two multispectral features. Finally, the category and bounding box of the target are predicted based on the fused multispectral features, which can effectively mine and fuse the similarity and complementary information of the multispectrals; a loss function is set to optimize the prediction results of the multispectral fusion target detection model, and the final optimized target detection category and bounding box are output, which can improve the accuracy of multispectral fusion target detection and the robustness of the model, while improving the fusion efficiency of multispectral features, thereby improving the target detection performance in multiple scenarios.

[0092] Example 2: This example is based on the method described in Example 1, and corresponding experimental data are provided for illustration, as follows: This embodiment is a multi-spectral fusion pedestrian target detection task. The overall process is as follows: Figure 5As shown, Figure 2 The main algorithm logic is the same, the difference is Figure 5 The feature encoder / decoder has been instantiated and can be replaced with other feature encoders / decoders (such as VGG, ResNet, etc.). The model is based on the single-stage detector Yolov5 and rewritten into a dual-branch structure suitable for multispectral fusion detection. Yolov5 is a lightweight, single-stage object detector that effectively balances detection accuracy and speed. The model inputs paired visible light and infrared images. These images can undergo data enhancement before being fed into the backbone network for feature extraction. Here, backbone features at three different scales are fed into the model for feature fusion, using the similarity and complementarity cross-attention fusion module (SCCA). As described in Example 1 above, this fusion module is used to mine and enhance similar and complementary information, cross-fusing multispectral features at different scales. The resulting multispectral fusion features are then used with the corresponding neck network and detection head to detect objects at different scales and predict pedestrian categories, bounding box locations, and sizes.

[0093] This embodiment was experimentally trained and verified on the KAIST multispectral dataset "Multispectral Pedestrian Detection: Benchmark Dataset and Baselines", as shown in Table 1, this example model.

[0094] Table 1: Experimental results on the KAIST dataset

[0095] Example 3: This embodiment provides a multi-spectral fusion target detection system that takes into account both similarity and complementarity. Figure 6 , the system comprising: A data acquisition module, used for acquiring visible light images and infrared images; A data processing module, configured to pre-process the visible light image and the infrared image; A model building module is used to build a multispectral fusion target detection model and input the preprocessed visible light image and infrared image into the multispectral fusion target detection model; The target detection module is used to extract the multispectral features of visible light images and infrared images respectively using the multispectral fusion target detection model. The multispectral features of visible light images and infrared images are then fused based on their similarity and complementarity. Finally, the target category and bounding box are predicted based on the fused multispectral features. The parameter optimization module is used to set the loss function to optimize the prediction results of the multispectral fusion target detection model according to the target category and bounding box predicted by the multispectral fusion target detection model, and output the final optimized target detection category and bounding box.

[0096] The above descriptions are merely embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention's description and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A multispectral fusion target detection method that takes into account both similarity and complementarity, characterized in that: The method comprises the following steps: Acquire a visible light image and an infrared image, and preprocess the visible light image and the infrared image; Constructing a multispectral fusion target detection model, and inputting the preprocessed visible light image and infrared image into the multispectral fusion target detection model; The multispectral fusion target detection model extracts multispectral features from visible light images and infrared images respectively, and fuses the multispectral features of visible light images and infrared images based on their similarity and complementarity. Finally, the target category and bounding box are predicted based on the fused multispectral features. According to the target category and bounding box predicted by the multispectral fusion target detection model, a loss function is set to optimize the prediction results of the multispectral fusion target detection model, and the final optimized target detection category and bounding box are output.

2. The multispectral fusion target detection method taking into account both similarity and complementarity according to claim 1, characterized in that: The process of preprocessing the visible light image and the infrared image includes: The acquired visible light images and infrared images are randomly cropped and paired, and data enhancement is performed on the paired visible light images and infrared images.

3. The multispectral fusion target detection method taking into account both similarity and complementarity according to claim 1, characterized in that: The process of predicting the target category and bounding box based on the fused multispectral features includes: The constructed multispectral fusion target detection model includes several feature encoders connected one by one, several similarity and complementary cross-attention fusion modules, several feature decoders and several detection heads, among which: Several feature encoders are used to extract multispectral features of multiple scales of visible light images and infrared images respectively; Each similarity and complementary cross-attention fusion module cross-fuses the multispectral features of visible light images and infrared images of different scales to obtain fused features; Each feature decoder performs feature decoding based on the fusion features output by the corresponding connected similarity and complementary cross-attention fusion modules; Each detection head predicts the target category and bounding box based on the decoded fusion features output by the corresponding connected feature decoder, and outputs the prediction result.

4. The multispectral fusion target detection method taking into account both similarity and complementarity according to claim 3, characterized in that: The similarity and complementary cross attention fusion modules cross-fuse the multispectral features of visible light images and infrared images of different scales respectively. The process includes: According to the similarity of the multispectral features of visible light images and infrared images, the feature distribution weights of the multispectral features of visible light images and infrared images in the self-modality are calculated respectively, and the similarity of the multispectral features of visible light images and infrared images is enhanced respectively according to the feature distribution weights in the self-modality; According to the complementarity of the multispectral characteristics of visible light images and infrared images, the feature attention weights of the multispectral features of visible light images and infrared images in the self-modality are calculated respectively, and the complementary information interaction of the multispectral features of visible light images and infrared images is performed according to the feature attention weights in the self-modality; By performing similarity enhancement and complementary information interaction on the multispectral features of visible light images and infrared images respectively, the multispectral features of visible light images and infrared images of different scales are cross-fused.

5. The multispectral fusion target detection method taking into account both similarity and complementarity according to claim 4, characterized in that: The process of calculating the feature distribution weights of the multispectral features of the visible light image and the infrared image in the self-modality includes: Calculate the feature distribution weight of the multispectral features of the visible light image in the self-modal state. The expression is: Calculate the feature distribution weight of the multispectral features of the infrared image in the self-modal state. The expression is: According to the feature distribution weights of the multispectral features of the visible light image and the infrared image in the self-modal state, the cosine similarity of the multispectral features of the visible light image and the infrared image is calculated. The expression is: According to the cosine similarity of the multispectral features of the visible light image and the infrared image, the comprehensive similarity feature distribution of the multispectral features of the visible light image and the infrared image is calculated. The expression is: According to the comprehensive similarity feature distribution of multispectral features of visible light image and infrared image, Gaussian filtering is used Further smooth the similar areas, enhance the areas of common interest of the multispectral features of the visible light image and the infrared image, suppress the features that the multispectral features of the visible light image and the infrared image do not care about, and complete the similarity enhancement of the multispectral features of the visible light image and the infrared image. The expression is: in, Indicates that it has parameters Sigmoid activation function, Represents the multispectral features of visible light images, Represents the multispectral characteristics of infrared images, represents element-wise product, represents the cosine similarity, Represents the comprehensive similarity feature distribution of multispectral features of visible light images and infrared images, Multispectral features of visible light images after similarity enhancement, Represents the multispectral features of the infrared image after similarity enhancement.

6. The multispectral fusion target detection method taking into account both similarity and complementarity according to claim 5, characterized in that: The process of calculating the feature attention weights of the multispectral features of visible light images and infrared images in the self-modal state includes: SimAM attention is used to calculate the feature attention weights of the multispectral features of visible light images and infrared images in the self-modal state, and the expression is: in, For the Transformer's multi-head self-attention calculation, the SimAM attention calculation formula is as follows: Among them, h, w represent the height and width of the input feature F, F.mean, F.var represent the mean and variance of the input feature F, respectively. As a formula hyperparameter; The complementary feature weighted calculation is performed based on the feature attention weights of the multispectral features of the visible light image and the infrared image in the self-modal state. The expression is: Based on the multispectral features of the visible light image and infrared image after similarity enhancement, combined with the feature attention weights of the multispectral features of the visible light image and infrared image after feature weighting in the self-modality, the fusion of complementary information is completed. The expression is: in, Represents the feature attention weight of the multispectral features of the visible light image in the self-modal state, Represents the feature attention weight of the multispectral features of the infrared image in the self-modal state, represents multi-head self-attention calculation, Represents the complementary feature information provided by the multispectral features of infrared images, Represents the complementary feature information provided by the multispectral features of infrared images.

7. The multispectral fusion target detection method taking into account both similarity and complementarity according to claim 1, characterized in that: The loss function is set to optimize the prediction results of the multispectral fusion target detection model as follows: in, and Represents the influencing factor used to balance the weights between loss functions, represents the bounding box regression loss, represents the target classification loss, represents the target confidence loss.

8. The multispectral fusion target detection method taking into account both similarity and complementarity according to claim 7, characterized in that: Target classification loss and target confidence loss The calculation process includes: Focal loss function is used to calculate target classification loss , the expression is: Focal loss function is used to calculate target confidence loss , the expression is: in, The label of the true category, Represents the probability when the model predicts category y=1, 、 Respectively represent the preset coefficients, represents the coefficient at time t.

9. The multispectral fusion target detection method taking into account both similarity and complementarity according to claim 1, characterized in that: Bounding box regression loss The calculation expression is: in, Represents the preset EIoU loss function, represents the original IoU loss, represents the distance loss of the center point, represents the aspect ratio loss, , A and B both represent borders, and Represents the height and width of the minimum bounding box, represents the Euclidean distance, and the superscript gt represents the true value.

10. A multispectral fusion target detection system that takes into account both similarity and complementarity, characterized in that: The system comprises: A data acquisition module, used for acquiring visible light images and infrared images; A data processing module, configured to pre-process the visible light image and the infrared image; A model building module is used to build a multispectral fusion target detection model and input the preprocessed visible light image and infrared image into the multispectral fusion target detection model; The target detection module is used to extract the multispectral features of visible light images and infrared images respectively using the multispectral fusion target detection model. The multispectral features of visible light images and infrared images are then fused based on their similarity and complementarity. Finally, the target category and bounding box are predicted based on the fused multispectral features. The parameter optimization module is used to set the loss function to optimize the prediction results of the multispectral fusion target detection model according to the target category and bounding box predicted by the multispectral fusion target detection model, and output the final optimized target detection category and bounding box.

Images