Bimodal small target detection method based on category feature guidance

Abstract

The invention relates to a bimodal small target detection method based on category feature guidance. The bimodal small target detection method comprises the following steps: respectively cutting regions corresponding to category labeling boxes in an infrared image and a visible light image to obtain a corresponding infrared category target region map and a visible light category target region map; performing feature extraction on the infrared category target area map to obtain infrared target features, constructing an infrared category feature set based on the infrared target features of all categories, performing feature extraction on the visible light category target area map to obtain visible light target features, and constructing a visible light category feature set based on the visible light target features of all categories; constructing a target detection model; and inputting the infrared-visible light image pair into a target detection model, guiding the model to learn small target features through the infrared category feature set and the visible light category feature set, and optimizing parameters of the target detection model to obtain a trained target detection model. According to the invention, the detection precision and robustness of the small target in a complex environment are effectively improved.

Description

Technical Field

[0001] The invention belongs to the technical field of image processing and computer vision, and relates to a dual-modal small target detection method based on category feature guidance. Background Art

[0002] In the field of computer vision, small target detection has always been an important research topic, especially in application scenarios such as drone images, surveillance videos and traffic monitoring. However, due to factors such as the low proportion of small targets in the image, blurred features and complex background, traditional target detection algorithms face many challenges.

[0003] Existing small target detection methods are mainly based on the improvement of mainstream target detection network models, which can be divided into several categories: enhancement methods for insufficient and uneven distribution of small target data (such as data augmentation technology); multi-scale fusion strategies to solve the problem that a single feature layer has insufficient representation ability for small targets (for example, FPN and BiFPN); super-resolution methods to improve the visualization features of small targets (such as super-resolution reconstruction networks); contextual learning methods to strengthen the feature information carried by small targets (for example, enhancing target features through contextual information); anchor box mechanisms to deal with the poor adaptability of anchor box size to small targets (for example, strategies based on adaptive anchor boxes); attention mechanisms to improve the lack of discriminative features of small targets (such as SE modules and CBAM); and small target detection strategies based on specific scenarios (for example, methods based on vision and infrared fusion).

[0004] Dual-modal small target detection has significant advantages in complex application scenarios and can effectively address the limitations of traditional small target detection. Single-modal data has deficiencies in dealing with lighting changes, complex backgrounds, and noise interference, especially in low-resolution or unclear target features. It is difficult to obtain high-precision detection results. Infrared images show strong target detection capabilities under extreme conditions such as low light, strong reflections, and smoke, while visible light images can provide rich texture and detail information. Therefore, the fusion of infrared and visible light can effectively improve the accuracy and robustness of small target detection.

[0005] Despite this, existing dual-modal small target detection methods still have shortcomings in fully utilizing the complementarity of infrared and visible light. On the one hand, traditional methods usually directly extract and fuse features from infrared and visible light images, and do not fully explore the feature differences of targets of different categories; on the other hand, due to the weakening of the features of small targets themselves, the directly extracted infrared and visible light features often contain a lot of background noise information and lack the ability to effectively represent category features. This leads to inaccurate feature expression of small targets, which in turn affects the detection performance. Summary of the invention

[0006] In order to solve the problems existing in the background technology, the present invention provides a dual-modal small target detection method based on category feature guidance, which uses a trained target detection model to perform target detection on an infrared-visible light image pair to obtain a target detection result. The training process of the target detection model includes:

[0007] S1: According to the category annotation box in the annotation information of the infrared-visible light image pair, the area corresponding to the category annotation box is cut out in the infrared image and the visible light image to obtain the corresponding infrared category target area map and visible light category target area map;

[0008] S2: Extract features of the infrared target area map using the InfMAE model to obtain infrared target features, and build an infrared feature set based on the infrared target features of all categories. Extract features of the visible light target area map using the McMAE model to obtain visible light target features, and build a visible light feature set based on the visible light target features of all categories.

[0009] S3: constructing a target detection model, wherein the target detection model includes: an infrared multi-scale feature extraction module, a visible light multi-scale feature extraction module, an infrared category feature guidance module, a visible light category feature guidance module, a feature fusion module and a feature prediction module;

[0010] S4: Input the infrared-visible light image pair into the target detection model, guide the model to learn small target features through the infrared category feature set and the visible light category feature set, and build a loss function according to the prediction results and annotation information of the target detection model to optimize the parameters of the target detection model to obtain a trained target detection model.

[0011] The present invention has at least the following beneficial effects

[0012] The present invention extracts multi-scale features in infrared and visible light images respectively through an infrared multi-scale feature extraction module and a visible light multi-scale feature extraction module, and the feature fusion module enables the features of different modalities to interact and fuse at a deeper level, thereby enhancing the model's ability to characterize small targets. The method extracts features from infrared and visible light category target area maps through an InfMAE model and a McMAE model respectively, and constructs an infrared category feature set and a visible light category feature set; by constructing a feature set, the method can more accurately capture the feature differences of each category of targets, and improves the model's ability to distinguish different categories of targets. The infrared category feature guidance module and the visible light category feature guidance module use the constructed feature set to guide the model to learn small target features so that the model can pay more attention to the feature learning of small targets during the training process, thereby improving the detection accuracy and generalization ability of the model. By combining infrared and visible light images and using the method of dual-modal feature fusion and category feature guidance, the method can significantly improve the accuracy and robustness of small target detection. In complex application scenarios, such as illumination changes, target occlusion, and complex background, the method can still maintain a high detection performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 It is a schematic diagram of the method flow of the present invention;

[0014] Figure 2 Schematic diagram of the framework structure of the target detection model of the present invention;

[0015] Figure 3 A schematic diagram of extracting infrared category features according to the present invention;

[0016] Figure 4 It is a schematic diagram of the structure of the InfMAE model of the present invention;

[0017] Figure 5 It is a structural schematic diagram of the McMAE model of the present invention;

[0018] Figure 6 It is a structural diagram of the ResNet50 model of the present invention;

[0019] Figure 7 It is a structural schematic diagram of the infrared category feature guidance module of the present invention;

[0020] Figure 8 Schematic diagram of the structure of the dilated convolutional encoder of the present invention;

[0021] Fig. 9 It is a schematic diagram of the structure of the Deformable DETR decoder of the present invention. DETAILED DESCRIPTION

[0022] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments only illustrate the basic concept of the present invention in a schematic manner, and the following embodiments and features in the embodiments can be combined with each other without conflict.

[0023] See also Figure 1 and Figure 2 In order to solve the problems existing in the background technology, the present invention provides a dual-modal small target detection method based on category feature guidance, using a trained target detection model to perform target detection on an infrared-visible light image pair to obtain a target detection result. The training process of the target detection model includes:

[0024] S1: According to the category annotation box in the label-annotation information of the infrared-visible light image pair, the area corresponding to the category annotation box is cut out in the infrared image and the visible light image to obtain the corresponding infrared category target area map and visible light category target area map;

[0025] In this embodiment, a training set is first obtained, and each training sample in the training set includes: an infrared-visible light image pair, and corresponding annotation information, wherein the annotation information includes: the coordinates of the upper left corner and the lower right corner of the annotation box, and the corresponding category label; according to the category annotation box information in the infrared image, the corresponding position is found on the infrared image, and these coordinates are used to cut out an area containing the target, and the obtained area is the infrared category target area map; similarly, the same category annotation box information is used on the visible light image to cut out an area containing the target, and the obtained area is the visible light category target area map.

[0026] See also Figure 3 , S2: Extract features of the infrared category target area map through the InfMAE model to obtain infrared target features, and build an infrared category feature set based on the infrared target features of all categories. Extract features of the visible light category target area map through the McMAE model to obtain visible light target features, and build a visible light category feature set based on the visible light target features of all categories;

[0027] Preferably, the constructing of the infrared category feature set and the visible light category feature set comprises:

[0028] S21: scaling the sizes of all the cropped infrared target area images and visible light target area images to a preset size, so that all the cropped infrared target area images and visible light target area images are adjusted to have the same width and height;

[0029] S22: Extract features from the infrared target area map through the InfMAE model, and use the feature map output by the first coding layer of the InfMAE model as the infrared target feature; calculate the central feature vector of all infrared target features under each category as the infrared category feature of the category, and form the infrared category features of all categories into an infrared category feature set in, represents the infrared category feature of the mth category, and M represents the total number of categories;

[0030] S23: Extract features from the visible light category target area map through the MCMAE model, and use the feature map output by the first encoding stage of the MCMAE model as the visible light target feature; calculate the central feature vector of all visible light target features under each category as the visible light category feature of the category, and form the visible light category features of all categories into a visible light category feature set in, represents the visible light category feature of the mth category, M represents the total number of categories, h and w represent the height and width, and C represents the number of channels.

[0031] In this embodiment, in order to ensure the consistency and comparability of the subsequent feature extraction process, it is necessary to scale the size of all cropped infrared category target area maps and visible light category target area maps to a preset size; use image scaling techniques (such as bilinear interpolation, nearest neighbor interpolation, etc.) to adjust all cropped area maps to the same width and height. Size scaling helps to eliminate feature extraction bias caused by different image sizes and improve the generalization ability of the model. Use the pre-trained InfMAE model to extract features from the infrared category target area map. The InfMAE model is a feature extraction network based on a masked autoencoder, which learns the feature representation of an image by masking parts of the input image and reconstructing these areas. In this step, focus on the feature maps output by the first encoding layer of the InfMAE model. These feature maps contain low-level features of the infrared target area, please refer to Figure 4 For each input infrared category target area map, the feature map (F1) output by the first coding layer of the InfMAE model is taken as the infrared target feature in the present invention. For each category, the central feature vector of all infrared target features is calculated. This is usually achieved by calculating the average or median of all feature vectors under each category. The central feature vector represents the typical pattern of the infrared target feature of this category.

[0032] Similar to the infrared category feature extraction, the pre-trained McMAE model is used to extract features from the visible light category target area map. The McMAE model may have a different architecture or parameters from the InfMAE model to adapt to the characteristics of visible light images, such as Figure 5 As shown in Figure 1, the output feature map of the McMAE model in stage 1 is used as the visible light target feature. Similarly, for each category, the central feature vector of all visible light target features is calculated.

[0033] S3: constructing a target detection model, wherein the target detection model includes: an infrared multi-scale feature extraction module, a visible light multi-scale feature extraction module, an infrared category feature guidance module, a visible light category feature guidance module, a feature fusion module and a feature prediction module;

[0034] Preferably, the infrared multi-scale feature extraction module is used to perform multi-scale feature extraction on infrared images to obtain multi-scale features of infrared images; the visible light multi-scale feature extraction module is used to perform multi-scale feature extraction on visible light images to obtain multi-scale features of visible light images; the infrared category feature guidance module is used to perform cross-attention enhancement on the multi-scale features of infrared images according to the infrared category feature set to obtain multi-scale enhanced features of infrared images; the visible light category feature guidance module is used to perform cross-attention enhancement on the multi-scale features of visible light images according to the visible light category feature set to obtain multi-scale enhanced features of visible light images; the feature fusion module is used to fuse the multi-scale enhanced features of infrared images and the multi-scale enhanced features of visible light images to obtain multi-scale weighted fusion features; the feature prediction module is used to output target detection results based on the multi-scale weighted fusion features.

[0035] The visible light multi-scale feature extraction module is similar to the infrared multi-scale feature extraction module. This module processes visible light images. It also uses a convolutional neural network (which may be another ResNet50 instance, or other networks with similar functions) to extract multi-scale features. These feature maps will play a key role in the subsequent object detection process because they provide detailed information about the objects in the visible light image.

[0036] In this embodiment, the main task of the infrared multi-scale feature extraction module is to process infrared images and extract multi-scale features from them. These features are crucial for subsequent target detection because they can capture target information of different sizes and positions in the image. To achieve this goal, the module usually uses advanced convolutional neural networks (such as ResNet50) to extract features. These networks will output feature maps of different scales at the feature extraction stage at different levels, and these feature maps together constitute a multi-scale feature set of infrared images. Features of scale. These features are crucial for subsequent target detection because they can capture target information of different sizes and positions in the image. To achieve this goal, the module usually uses advanced convolutional neural networks (such as ResNet50) to extract features. These networks will output feature maps of different scales at the feature extraction stage at different levels, and these feature maps together constitute a multi-scale feature set of infrared images.

[0037] The infrared category feature guidance module uses the infrared category feature set to enhance the multi-scale features of the infrared image. The infrared category feature set may contain some predefined category features that are extracted from the infrared image in some way (such as a trained model). The module fuses these category features with the multi-scale features of the infrared image through a cross-attention mechanism to obtain enhanced multi-scale features. This enhancement helps the model better identify targets in infrared images. The visible light category feature guidance module is similar to the infrared category feature guidance module. This module uses the visible light category feature set to enhance the multi-scale features of the visible light image. It also uses a cross-attention mechanism to fuse the visible light category features with the multi-scale features of the visible light image. This enhancement helps the model to more accurately identify targets in visible light images. The task of the feature fusion module is to fuse the multi-scale enhanced features of the infrared image and the visible light image to obtain a multi-scale weighted fusion feature containing information from both modalities. To achieve this goal, the module may adopt some complex fusion strategies, such as weighted summation and splicing. These strategies aim to make full use of the complementarity between infrared and visible light images to improve the accuracy of target detection. Feature prediction module The feature prediction module outputs the target detection result based on the multi-scale weighted fusion features. This usually involves a decoder (such as the Deformable DETR decoder), which can convert the fused features into the output format required for target detection, including information such as the target's category, location, and confidence.

[0038] Preferably, the infrared multi-scale feature extraction module adopts a first ResNet50 model, and the visible light multi-scale feature extraction module adopts a second ResNet50 model;

[0039] The first ResNet50 model is used to extract features from the infrared image, and the output feature maps of the first ResNet50 model in the first feature extraction stage, the second feature extraction stage, and the third feature extraction stage are combined into a multi-scale feature map of the infrared image. i∈{1,2,3}, Represents the output feature map of the first ResNet50 model at the i-th feature extraction stage;

[0040] The second ResNet50 model is used to extract features from the visible light image, and the output feature maps of the second ResNet50 model in the first feature extraction stage, the second feature extraction stage, and the third feature extraction stage are combined into a multi-scale feature map of the visible light image. i∈{1,2,3}, Represents the output feature map of the second ResNet50 model at the i-th feature extraction stage.

[0041] See also Figure 6 ResNet50 includes the feature extraction stages of STAGE0 to STAGE4. In the present invention, the output feature maps of STAGE1, STAGE2 and STAGE3 are used to form the multi-scale features of infrared images.

[0042] Preferably, the step of performing cross-attention enhancement on multi-scale features of the infrared image according to the infrared category feature set comprises:

[0043] S301: The feature map of each scale of the infrared image Perform global average pooling respectively, concatenate the features of all scales after global average pooling and input them into the classifier to predict the probability distribution of each category in the infrared image

[0044] S302: for the k categories with the highest probability in the infrared image, infrared category features corresponding to the k categories are screened out from the infrared category feature set;

[0045] S303: Flatten each infrared category feature selected along the spatial dimension, and concatenate the flattened k infrared category features to obtain a feature sequence as a key vector K R and a numerical vector V R ;

[0046] S304: The feature map of each scale of the infrared image Divide the image into multiple image blocks along the spatial dimension, ensuring that each image block has the same width and height as the infrared category feature in the infrared category feature set;

[0047] S305: Flatten each cropped image block along the spatial dimension, and use the feature sequence of all flattened image blocks as the query vector Q Ri ;

[0048] S306: The key vector K R , numerical vector V R and the query vector Q Ri Perform cross attention processing and reshape the feature map after cross attention into a feature map The same size obtains enhanced features of each scale of infrared images i∈{1,2,3}, and then construct the multi-scale enhanced features of infrared images

[0049] in, represents the enhanced features of the ith scale of the infrared image, reshape represents the reshaping function, d represents the temperature coefficient, and T represents the transpose.

[0050] Preferably, the performing cross-attention enhancement on multi-scale features of the visible light image according to the visible light category feature set comprises:

[0051] S311: Feature maps of each scale of the visible light image Perform global average pooling respectively, concatenate the features of all scales after global average pooling and input them into the classifier to predict the probability distribution of each category in the visible light image

[0052] S312: for the k categories with the highest probability in the visible light image, select visible light category features corresponding to the k categories from the visible light category feature set;

[0053] S313: Flatten each visible light category feature selected along the spatial dimension, and concatenate the k flattened visible light category features to obtain a feature sequence as a key vector K V and a numerical vector V V ;

[0054] S314: Feature maps of each scale of the visible light image Divide the image into multiple image blocks along the spatial dimension, ensuring that each image block has the same width and height as the visible light category feature in the visible light category feature set;

[0055] S315: Flatten each cropped image block along the spatial dimension, and use the feature sequence of all flattened image blocks as the query vector Q Vi ;

[0056] S316: The key vector K V , numerical vector VV and the query vector Q Vi Perform cross attention processing and reshape the feature map after cross attention into a feature map The same size obtains enhanced features at each scale of the visible light image i∈{1,2,3}, and then construct the multi-scale enhanced features of visible light images

[0057]

[0058] in, represents the enhanced features of the i-th scale of the visible light image, reshape represents the reshaping function, d represents the temperature coefficient, and T represents the transpose.

[0059] In this embodiment, the multi-scale features of infrared and visible light images are enhanced by using the infrared and visible light category feature sets through the cross-attention mechanism. This enhancement method enables the model to pay more attention to the features related to the category, thereby improving the accuracy of target detection. Through the steps of global average pooling, category probability prediction, screening category features, constructing key vectors, numerical vectors and query vectors, and cross-attention processing, the multi-scale enhanced features of infrared and visible light images are successfully constructed, which provides strong support for subsequent feature fusion and target detection, such as Figure 7 shown.

[0060] Preferably, the feature fusion module is used to fuse the multi-scale enhancement features of the infrared image and the multi-scale enhancement features of the visible light image to obtain the multi-scale weighted fusion features, including:

[0061] S321: inputting the enhanced features of each scale of the infrared image into the first dilated convolution encoder for encoding processing to obtain multi-scale mixed features of the infrared image;

[0062] S322: Inputting the enhanced features of each scale of the visible light image into the second dilated convolutional encoder for encoding processing to obtain multi-scale mixed features of the visible light image;

[0063] S323: performing weighted fusion on the multi-scale mixed features of the infrared image and the multi-scale mixed features of the visible light image at corresponding scales to obtain a multi-scale weighted mixed feature;

[0064] S324: Concatenate the multi-scale weighted mixed features and the multi-scale mixed features of the infrared image to obtain the multi-scale weighted fusion features.

[0065] Preferably, the multi-scale weighted fusion feature includes:

[0066]

[0067] in, represents the weighted fusion feature of the i-th scale, w Ri and w Vi represents the weight factor; dc1(·) represents the first dilated convolutional encoder; dc2(·) represents the second dilated convolutional encoder; Represents the enhanced features of the i-th scale of the infrared image; represents the enhanced features of the i-th scale of the visible light image, and Concat represents the concatenation function. In this embodiment, the model structures of the first dilated convolutional encoder and the second dilated convolutional encoder are as follows: Figure 8 shown.

[0068] Preferably, the feature prediction module is used to output the target detection result according to the multi-scale weighted fusion feature, including: Flatten in the spatial dimension, and input the flattened feature sequence into the Deformable DETR decoder for decoding to obtain the target detection result.

[0069] In this embodiment, the multi-scale feature fusion and target detection of infrared images and visible light images are realized through the feature fusion module and the feature prediction module. The feature fusion module improves the feature representation capability through the dilated convolution encoder and the weighted fusion strategy; the feature prediction module uses the Deformable DETR decoder to achieve accurate target detection. This fusion method can make full use of the complementarity between infrared and visible light images to improve the accuracy and robustness of target detection. The model structure of the Deformable DETR decoder is as follows: Fig. 9 shown.

[0070] S4: Input the infrared-visible light image pair into the target detection model, guide the model to learn small target features through the infrared category feature set and the visible light category feature set, and build a loss function according to the prediction results and annotation information of the target detection model to optimize the parameters of the target detection model to obtain a trained target detection model.

[0071] Preferably, the constructing the loss function comprises:

[0072] L total =L DETR +λ R-guide L R-guided +λ V-guided L V-guided

[0073] L DETR =λ cls L cls +λ bbox L bbox

[0074]

[0075] Among them, L total Represents the total loss function, L DETR represents DETR Loss, which is responsible for optimizing the classification and bounding box of target detection; L R-guided and L V-guided They represent the infrared category guidance loss function and the visible light category guidance loss function, respectively, and are responsible for optimizing the category prediction accuracy and feature matching of the infrared category feature guidance module and the visible light category feature guidance module; N represents the number of samples; λ cls , bbox , R-guided , V-guided , r-feat , v-f , L1 and λ GIOU is a hyperparameter to balance the losses; L1(·) represents the L1 loss function; GIOU(·) is used to measure the overlap between the bounding box predicted by the target detection model and the true bounding box; represents the predicted bounding box of the target detection model, b i represents the real bounding box in the annotation information; y Ri Represents the category label of the i-th category in the annotation information, y Vi Represents the category label of the i-th category in the annotation information, y i Represents the category label of the i-th category in the annotation information, and its value is 0 or 1; represents the probability of the i-th class predicted by the classifier in the infrared category feature guidance module, represents the probability of the i-th category predicted by the classifier in the visible light category feature guidance module, Represents the probability of the i-th class predicted by the target detection model; and Respectively represent the output feature map of the first ResNet50 model at the i-th feature extraction stage and the output feature map of the second ResNet50 model at the i-th feature extraction stage; and They represent the enhanced features of the i-th scale of the infrared image and the i-th scale of the visible light image respectively, and ‖.‖2 represents the L2 loss.

[0076] In this embodiment, the total loss function L total Comprehensively considering DETR loss, infrared category guidance loss and visible light category guidance loss, it can comprehensively optimize the performance of the target detection model; through the classification loss L cls , the model can more accurately predict the category of the target and reduce classification errors; the bounding box loss L bboxCombining L1 loss and GIOU loss can more accurately predict the bounding box of the target and improve positioning accuracy. R-guided and the visible light category guided loss L V-guided The classification loss part in can optimize the category prediction ability of the guidance module and improve the accuracy of category prediction. The feature matching loss part uses L2 loss to measure the difference between the enhanced features output by the guidance module and the feature map output by the ResNet50 model, which helps the model learn more robust and accurate feature representation. By using the infrared and visible light category feature sets to guide the model to learn small target features, the complementarity of the two modal information can be fully utilized to improve the model's detection ability for small targets. In the feature fusion module, by multi-scale weighted fusion features, feature information of different scales can be further extracted and utilized, which helps the model better capture the details and contextual information of small targets. The above loss function achieves comprehensive optimization and improvement of the target detection model by comprehensively considering the classification and bounding box prediction ability of the target detection model, the effect of infrared and visible light category feature guidance, the influence of different loss terms, and the model's detection ability for small targets. This design makes the performance of the model more robust and accurate in complex scenarios.

[0077] In summary, the present invention extracts multi-scale features in infrared and visible light images respectively through an infrared multi-scale feature extraction module and a visible light multi-scale feature extraction module, and the feature fusion module enables the features of different modalities to interact and fuse at a deeper level, thereby enhancing the model's ability to characterize small targets. The method extracts features from infrared and visible light category target area maps through the InfMAE model and the McMAE model respectively, and constructs an infrared category feature set and a visible light category feature set; by constructing the feature set, the method can more accurately capture the feature differences of each category of targets, and improve the model's ability to distinguish different categories of targets. The infrared category feature guidance module and the visible light category feature guidance module use the constructed feature set to guide the model to learn small target features so that the model can pay more attention to the feature learning of small targets during the training process, thereby improving the detection accuracy and generalization ability of the model. By combining infrared and visible light images and using the method of dual-modal feature fusion and category feature guidance, the method can significantly improve the accuracy and robustness of small target detection. In complex application scenarios, such as illumination changes, target occlusion, and complex background, the method can still maintain a high detection performance.

[0078] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution, which should be included in the scope of the claims of the present invention.

Claims

1. A dual-modal small target detection method based on category feature guidance, using a trained target detection model to perform target detection on infrared-visible light image pairs to obtain target detection results, characterized in that: The training process of the target detection model includes: S1: According to the category annotation box in the annotation information of the infrared-visible light image pair, the area corresponding to the category annotation box is cut out in the infrared image and the visible light image to obtain the corresponding infrared category target area map and visible light category target area map; S2: Extract features of the infrared target area map using the InfMAE model to obtain infrared target features, and build an infrared feature set based on the infrared target features of all categories. Extract features of the visible light target area map using the McMAE model to obtain visible light target features, and build a visible light feature set based on the visible light target features of all categories. S3: constructing a target detection model, wherein the target detection model includes: an infrared multi-scale feature extraction module, a visible light multi-scale feature extraction module, an infrared category feature guidance module, a visible light category feature guidance module, a feature fusion module and a feature prediction module; S4: Input the infrared-visible light image pair into the target detection model, guide the model to learn small target features through the infrared category feature set and the visible light category feature set, and build a loss function according to the prediction results and annotation information of the target detection model to optimize the parameters of the target detection model to obtain a trained target detection model.

2. The dual-modal small target detection method based on category feature guidance according to claim 1, characterized in that: The constructing of the infrared category feature set and the visible light category feature set comprises: S21: scaling the sizes of all the cropped infrared target area images and visible light target area images to a preset size, so that all the cropped infrared target area images and visible light target area images are adjusted to have the same width and height; S22: Extract features from the infrared target area map through the InfMAE model, and use the feature map output by the first coding layer of the InfMAE model as the infrared target feature; calculate the central feature vector of all infrared target features under each category as the infrared category feature of the category, and form the infrared category features of all categories into an infrared category feature set in, represents the infrared category feature of the mth category, and M represents the total number of categories; S23: Extract features from the visible light category target area map through the McMAE model, and use the feature map output by the first encoding stage of the McMAE model as the visible light target feature; calculate the central feature vector of all visible light target features under each category as the visible light category feature of the category, and form the visible light category features of all categories into a visible light category feature set in, represents the visible light category feature of the mth category, M represents the total number of categories, h and w represent the height and width, and C represents the number of channels.

3. The dual-modal small target detection method based on category feature guidance according to claim 2, characterized in that: The infrared multi-scale feature extraction module is used to perform multi-scale feature extraction on infrared images to obtain multi-scale features of infrared images; the visible light multi-scale feature extraction module is used to perform multi-scale feature extraction on visible light images to obtain multi-scale features of visible light images; the infrared category feature guidance module is used to perform cross-attention enhancement on the multi-scale features of infrared images according to the infrared category feature set to obtain multi-scale enhanced features of infrared images; the visible light category feature guidance module is used to perform cross-attention enhancement on the multi-scale features of visible light images according to the visible light category feature set to obtain multi-scale enhanced features of visible light images; the feature fusion module is used to fuse the multi-scale enhanced features of infrared images and the multi-scale enhanced features of visible light images to obtain multi-scale weighted fusion features; the feature prediction module is used to output target detection results according to the multi-scale weighted fusion features.

4. The dual-modal small target detection method based on category feature guidance according to claim 3 is characterized in that: The infrared multi-scale feature extraction module adopts a first ResNet50 model, and the visible light multi-scale feature extraction module adopts a second ResNet50 model; The first ResNet50 model is used to extract features from the infrared image, and the output feature maps of the first ResNet50 model in the first feature extraction stage, the second feature extraction stage, and the third feature extraction stage are combined into a multi-scale feature map of the infrared image. Represents the output feature map of the first ResNet50 model at the i-th feature extraction stage; The second ResNet50 model is used to extract features from the visible light image, and the output feature maps of the second ResNet50 model in the first feature extraction stage, the second feature extraction stage, and the third feature extraction stage are combined into a multi-scale feature map of the visible light image. Represents the output feature map of the second ResNet50 model at the i-th feature extraction stage.

5. The dual-modal small target detection method based on category feature guidance according to claim 4, characterized in that: The cross-attention enhancement of the multi-scale features of the infrared image according to the infrared category feature set comprises: S301: The feature map of each scale of the infrared image Perform global average pooling respectively, concatenate the features of all scales after global average pooling and input them into the classifier to predict the probability distribution of each category in the infrared image S302: for the k categories with the highest probability in the infrared image, infrared category features corresponding to the k categories are screened out from the infrared category feature set; S303: Flatten each infrared category feature selected along the spatial dimension, and concatenate the flattened k infrared category features to obtain a feature sequence as a key vector K R and a numerical vector V R ; S304: The feature map of each scale of the infrared image Divide the image into multiple image blocks along the spatial dimension, ensuring that each image block has the same width and height as the infrared category feature in the infrared category feature set; S305: Flatten each cropped image block along the spatial dimension, and use the feature sequence of all flattened image blocks as the query vector Q Ri ; S306: The key vector K R , numerical vector V R and the query vector Q Ri Perform cross attention processing and reshape the feature map after cross attention into a feature map The same size obtains enhanced features of each scale of infrared images Then construct the multi-scale enhancement features of infrared images in, represents the enhanced features of the ith scale of the infrared image, reshape represents the reshaping function, d represents the temperature coefficient, and T represents the transpose.

6. The dual-modal small target detection method based on category feature guidance according to claim 5, characterized in that: The cross-attention enhancement of multi-scale features of the visible light image according to the visible light category feature set comprises: S311: Feature maps of each scale of the visible light image Perform global average pooling respectively, concatenate the features of all scales after global average pooling and input them into the classifier to predict the probability distribution of each category in the visible light image S312: for the k categories with the highest probability in the visible light image, select visible light category features corresponding to the k categories from the visible light category feature set; S313: Flatten each visible light category feature selected along the spatial dimension, and concatenate the k flattened visible light category features to obtain a feature sequence as a key vector K V and a numerical vector V V ; S314: Feature maps of each scale of the visible light image Divide the image into multiple image blocks along the spatial dimension, ensuring that each image block has the same width and height as the visible light category feature in the visible light category feature set; S315: Flatten each cropped image block along the spatial dimension, and use the feature sequence of all flattened image blocks as the query vector Q Vi ; S316: The key vector K V , numerical vector V V and the query vector Q Vi Perform cross attention processing and reshape the feature map after cross attention into a feature map The same size obtains enhanced features at each scale of the visible light image Then construct the multi-scale enhancement features of visible light images in, represents the enhanced features of the i-th scale of the visible light image, reshape represents the reshaping function, d represents the temperature coefficient, and T represents the transpose.

7. The dual-modal small target detection method based on category feature guidance according to claim 6, characterized in that: The feature fusion module is used to fuse the multi-scale enhancement features of the infrared image and the multi-scale enhancement features of the visible light image to obtain the multi-scale weighted fusion features, including: S321: inputting the enhanced features of each scale of the infrared image into the first dilated convolution encoder for encoding processing to obtain multi-scale mixed features of the infrared image; S322: Inputting the enhanced features of each scale of the visible light image into the second dilated convolutional encoder for encoding processing to obtain multi-scale mixed features of the visible light image; S323: performing weighted fusion on the multi-scale mixed features of the infrared image and the multi-scale mixed features of the visible light image at corresponding scales to obtain a multi-scale weighted mixed feature; S324: Concatenate the multi-scale weighted mixed features and the multi-scale mixed features of the infrared image to obtain the multi-scale weighted fusion features.

8. The dual-modal small target detection method based on category feature guidance according to claim 7, characterized in that: The multi-scale weighted fusion features include: in, represents the weighted fusion feature of the i-th scale, w Ri and w Vi represents the weight factor; dc1(·) represents the first dilated convolutional encoder; dc2(·) represents the second dilated convolutional encoder; Represents the enhanced features of the i-th scale of the infrared image; represents the enhanced features of the i-th scale of the visible light image, and Concat represents the concatenation function.

9. The dual-modal small target detection method based on category feature guidance according to claim 8, characterized in that: The feature prediction module is used to output the target detection result according to the multi-scale weighted fusion feature, including: Flatten in the spatial dimension, and input the flattened feature sequence into the Deformable DETR decoder for decoding to obtain the target detection result.

10. The dual-modal small target detection method based on category feature guidance according to claim 9, characterized in that: The construction of the loss function includes: L total =L DETR +λ R-guided L R-guided +λ V-guided L V-guided L DETR =λ cls L cls +λ bbox L bbox Among them, L total Represents the total loss function, L DETR represents DETR Loss, which is responsible for optimizing the classification and bounding box of target detection; L R-guided and L V-guided They represent the infrared category guidance loss function and the visible light category guidance loss function, respectively, and are responsible for optimizing the category prediction accuracy and feature matching of the infrared category feature guidance module and the visible light category feature guidance module; N represents the number of samples; λ cls , bbox , R-guided , V-guided , r-fea , v-fea , L1 and λ GIOU is a hyperparameter to balance the losses; L1(·) represents the L1 loss function; GIOU(·) is used to measure the overlap between the bounding box predicted by the target detection model and the true bounding box; represents the predicted bounding box of the target detection model, b i represents the real bounding box in the annotation information; y Ri Represents the category label of the i-th category in the annotation information, y Vi Represents the category label of the i-th category in the annotation information, y i Represents the category label of the i-th category in the annotation information, and its value is 0 or 1; represents the probability of the i-th class predicted by the classifier in the infrared category feature guidance module, represents the probability of the i-th category predicted by the classifier in the visible light category feature guidance module, Represents the probability of the i-th class predicted by the target detection model; and Respectively represent the output feature map of the first ResNet50 model at the i-th feature extraction stage and the output feature map of the second ResNet50 model at the i-th feature extraction stage; and They represent the enhanced features of the i-th scale of the infrared image and the i-th scale of the visible light image respectively, and ‖.‖2 represents the L2 loss.

Images