A dual-mode small target detection method based on category feature guidance

Abstract

The application relates to a dual-mode small target detection method based on category feature guidance, which comprises the following steps: cutting the regions corresponding to the category marking frames in infrared images and visible light images to obtain corresponding infrared category target region images and visible light category target region images; performing feature extraction on the infrared category target region images to obtain infrared target features, constructing an infrared category feature set based on the infrared target features of all categories, performing feature extraction on the visible light category target region images to obtain visible light target features, and constructing a visible light category feature set based on the visible light target features of all categories; constructing a target detection model; inputting an infrared-visible light image pair into the target detection model, guiding model learning of small target features through the infrared category feature set and the visible light category feature set, optimizing parameters of the target detection model, and obtaining a trained target detection model. The application effectively improves the detection precision and robustness of small targets in complex environments.

Description

Technical Field

[0001] This invention belongs to the field of image processing and computer vision technology, and relates to a dual-modal small target detection method based on category feature guidance. Background Technology

[0002] In the field of computer vision, small object detection has always been an important research topic, especially in applications such as drone imagery, surveillance video, and traffic monitoring, where accurate detection of small objects is of great significance. However, due to factors such as the low proportion of small objects in images, blurred features, and complex backgrounds, traditional object detection algorithms face many challenges.

[0003] Existing small target detection methods are mainly based on improvements to mainstream target detection network models, and can be specifically divided into several categories: enhancement methods to address insufficient and unevenly distributed small target data (such as data augmentation techniques); multi-scale fusion strategies to address the insufficient representation ability of a single feature layer for small targets (e.g., FPN and BiFPN); super-resolution methods to improve the visual features of small targets (such as super-resolution reconstruction networks); context learning methods to enhance the feature information carried by small targets (e.g., enhancing target features through contextual information); anchor box mechanisms to address the poor adaptability of anchor box size to small targets (e.g., strategies based on adaptive anchor boxes); attention mechanisms to improve the lack of discriminative features of small targets (such as SE modules and CBAM); and small target detection strategies based on specific scenarios (e.g., methods based on visual and infrared fusion).

[0004] Dual-modal small target detection offers significant advantages in complex application scenarios, effectively addressing the limitations of traditional small target detection methods. Single-modal data falls short in handling variations in illumination, complex backgrounds, and noise interference, especially at low resolutions or when target features are unclear, making it difficult to achieve high-precision detection results. Infrared images demonstrate strong target detection capabilities under extreme conditions such as low light, strong reflections, and smoke, while visible light images provide rich texture and detail information. Therefore, the fusion of infrared and visible light can effectively improve the accuracy and robustness of small target detection.

[0005] Nevertheless, existing bimodal small target detection methods still have shortcomings in fully utilizing the complementarity of infrared and visible light. On the one hand, traditional methods typically extract and fuse features directly from infrared and visible light images, failing to fully exploit the feature differences between different target categories. On the other hand, due to the weakened features of small targets themselves, the directly extracted infrared and visible light features often contain a large amount of background noise information and lack effective representation capabilities for category-specific features. This results in inaccurate feature representation of small targets, thus affecting detection performance. Summary of the Invention

[0006] To address the problems existing in the background technology, this invention provides a dual-modal small target detection method based on category feature guidance. The method utilizes a trained target detection model to perform target detection on infrared-visible image pairs, obtaining target detection results. The training process of the target detection model includes:

[0007] S1: Based on the category bounding boxes in the annotation information of the infrared-visible image pair, crop the regions corresponding to the category bounding boxes in the infrared image and the visible image respectively to obtain the corresponding infrared category target region map and the visible light category target region map;

[0008] S2: Infrared target features are obtained by extracting features from the infrared target area map using the InfMAE model. An infrared category feature set is constructed based on the infrared target features of all categories. Visible light target features are obtained by extracting features from the visible light target area map using the McMAE model. A visible light category feature set is constructed based on the visible light target features of all categories.

[0009] S3: Construct a target detection model, wherein the target detection model includes: an infrared multi-scale feature extraction module, a visible light multi-scale feature extraction module, an infrared category feature guidance module, a visible light category feature guidance module, a feature fusion module, and a feature prediction module;

[0010] S4: Input the infrared-visible light image into the target detection model, guide the model to learn small target features through the infrared category feature set and the visible light category feature set, construct a loss function based on the prediction results and annotation information of the target detection model, optimize the parameters of the target detection model, and obtain a trained target detection model.

[0011] The present invention has at least the following beneficial effects

[0012] This invention extracts multi-scale features from infrared and visible light images using infrared and visible light multi-scale feature extraction modules, respectively. A feature fusion module enables deeper interaction and fusion of features from different modalities, thereby enhancing the model's ability to represent small targets. The method uses InfMAE and McMAE models to extract features from infrared and visible light target region maps, respectively, constructing infrared and visible light category feature sets. By constructing these feature sets, the method can more accurately capture the feature differences between target categories, improving the model's ability to distinguish between different target categories. The infrared and visible light category feature guidance modules utilize the constructed feature sets to guide the model in learning small target features, enabling the model to focus more on small target feature learning during training, thus improving the model's detection accuracy and generalization ability. By combining infrared and visible light images and utilizing dual-modal feature fusion and category feature guidance, this method significantly improves the accuracy and robustness of small target detection. Even in complex application scenarios, such as changes in illumination, target occlusion, and complex backgrounds, this method maintains high detection performance. Attached Figure Description

[0013] Figure 1 This is a schematic diagram of the method flow of the present invention;

[0014] Figure 2 This is a schematic diagram of the framework structure of the target detection model of the present invention;

[0015] Figure 3 This is a schematic diagram illustrating the extraction of infrared category features according to the present invention;

[0016] Figure 4 This is a schematic diagram of the InfMAE model of the present invention;

[0017] Figure 5 This is a schematic diagram of the structure of the McMAE model of the present invention;

[0018] Figure 6 This is a schematic diagram of the ResNet50 model of the present invention;

[0019] Figure 7 This is a schematic diagram of the infrared category feature guidance module of the present invention;

[0020] Figure 8 This is a schematic diagram of the structure of the extended convolutional encoder of the present invention;

[0021] Figure 9 This is a schematic diagram of the Deformable DETR decoder of the present invention. Detailed Implementation

[0022] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0023] Please see Figure 1 and Figure 2 To address the problems existing in the background technology, this invention provides a dual-modal small target detection method based on category feature guidance. The method utilizes a trained target detection model to perform target detection on infrared-visible image pairs, obtaining the target detection results. The training process of the target detection model includes:

[0024] S1: Based on the category label boxes in the labeling information of the infrared-visible image pair, crop the regions corresponding to the category label boxes in the infrared image and the visible image respectively to obtain the corresponding infrared category target region map and the visible light category target region map;

[0025] In this embodiment, a training set is first obtained. Each training sample in the training set includes an infrared-visible light image pair and corresponding annotation information, wherein the annotation information includes the coordinates of the upper left and lower right corners of the bounding box and the corresponding category label. Based on the category bounding box information in the infrared image, the corresponding position is found on the infrared image, and a region containing the target is cropped using these coordinates. The resulting region is the infrared category target region map. Similarly, the same category bounding box information is used to crop a region containing the target on the visible light image. The resulting region is the visible light category target region map.

[0026] Please see Figure 3 S2: Infrared target features are obtained by extracting features from the infrared target area map using the InfMAE model. An infrared target feature set is constructed based on the infrared target features of all categories. Visible target features are obtained by extracting features from the visible light target area map using the McMAE model. A visible light target feature set is constructed based on the visible light target features of all categories.

[0027] Preferably, constructing the infrared category feature set and the visible light category feature set includes:

[0028] S21: Scale the size of all cropped infrared category target area maps and visible light category target area maps to the preset size, so that all cropped infrared category target area maps and visible light category target area maps are adjusted to have the same width and height;

[0029] S22: Feature extraction is performed on the infrared target region map using the InfMAE model. The feature map output from the first encoding layer of the InfMAE model is used as the infrared target feature. The center feature vector of all infrared target features under each category is calculated as the infrared category feature of that category. The infrared category features of all categories are combined into an infrared category feature set. in, Let M represent the infrared category feature of the m-th category, and M represent the total number of categories;

[0030] S23: Feature extraction is performed on the visible light category target region map using the MCMAE model. The feature map output from the first encoding stage of the MCMAE model is used as the visible light target feature. The central feature vector of all visible light target features under each category is calculated as the visible light category feature for that category. The visible light category features of all categories are combined into a visible light category feature set. in, Let m represent the visible light category feature of the m-th category, M represent the total number of categories, h and w represent the height and width, and C represent the number of channels.

[0031] In this embodiment, to ensure consistency and comparability in subsequent feature extraction processes, the sizes of all cropped infrared and visible light target region maps need to be scaled to a preset size. Image scaling techniques (such as bilinear interpolation, nearest neighbor interpolation, etc.) are used to adjust all cropped region maps to have the same width and height. Size scaling helps eliminate feature extraction bias caused by different image sizes and improves the model's generalization ability. A pre-trained InfMAE model is used to extract features from the infrared target region maps. The InfMAE model is a feature extraction network based on a mask autoencoder, which learns the feature representation of an image by masking a portion of the input image and reconstructing those regions. In this step, we focus on the feature maps output by the first encoding layer of the InfMAE model. These feature maps contain low-level features of the infrared target regions; please refer to [link to relevant documentation]. Figure 4 For each input infrared target region map of a category, this invention uses the feature map (F1) output from the first encoding layer of the InfMAE model as the infrared target feature. For each category, a central feature vector is calculated for all infrared target features. This is typically achieved by calculating the mean or median of all feature vectors for each category. The central feature vector represents the typical pattern of infrared target features for that category.

[0032] Similar to infrared category feature extraction, a pre-trained McMAE model is used to extract features from visible light category target region maps. The McMAE model may have a different architecture or parameters than the InfMAE model to adapt to the characteristics of visible light images, such as... Figure 5 As shown, the output feature map of the McMAE model in stage 1 is used as the visible light target feature. Similarly, for each category, the center feature vector of all visible light target features is calculated.

[0033] S3: Construct a target detection model, wherein the target detection model includes: an infrared multi-scale feature extraction module, a visible light multi-scale feature extraction module, an infrared category feature guidance module, a visible light category feature guidance module, a feature fusion module, and a feature prediction module;

[0034] Preferably, the infrared multi-scale feature extraction module is used to extract multi-scale features from the infrared image to obtain multi-scale features of the infrared image; the visible light multi-scale feature extraction module is used to extract multi-scale features from the visible light image to obtain multi-scale features of the visible light image; the infrared category feature guidance module is used to perform cross-attention enhancement on the multi-scale features of the infrared image according to the infrared category feature set to obtain multi-scale enhanced features of the infrared image; the visible light category feature guidance module is used to perform cross-attention enhancement on the multi-scale features of the visible light image according to the visible light category feature set to obtain multi-scale enhanced features of the visible light image; the feature fusion module is used to fuse the multi-scale enhanced features of the infrared image and the multi-scale enhanced features of the visible light image to obtain multi-scale weighted fusion features; and the feature prediction module is used to output target detection results based on the multi-scale weighted fusion features.

[0035] Similar to the infrared multi-scale feature extraction module, the visible light multi-scale feature extraction module processes visible light images. It also employs a convolutional neural network (possibly another ResNet50 instance, or another network with similar functionality) to extract multi-scale features. These feature maps play a crucial role in subsequent object detection, as they provide detailed information about the target in the visible light image.

[0036] In this embodiment, the main task of the infrared multi-scale feature extraction module is to process infrared images and extract multi-scale features from them. These features are crucial for subsequent target detection because they can capture target information of different sizes and locations within the image. To achieve this, the module typically employs advanced convolutional neural networks (such as ResNet50) to extract features. These networks output feature maps of different scales at different levels of feature extraction, and these feature maps collectively constitute the multi-scale feature set of the infrared image.

[0037] The infrared category feature guidance module utilizes an infrared category feature set to enhance the multi-scale features of infrared images. This set may contain predefined category features extracted from the infrared image through some method (such as a trained model). The module fuses these category features with the multi-scale features of the infrared image using a cross-attention mechanism, resulting in enhanced multi-scale features. This enhancement helps the model better identify targets in infrared images. The visible light category feature guidance module is similar to the infrared module, utilizing a visible light category feature set to enhance the multi-scale features of visible light images. It also employs a cross-attention mechanism to fuse visible light category features with the multi-scale features of the visible light image. This enhancement helps the model more accurately identify targets in visible light images. The feature fusion module's task is to fuse the multi-scale enhanced features of the infrared and visible light images, resulting in a multi-scale weighted fusion feature containing information from both modalities. To achieve this, the module may employ complex fusion strategies, such as weighted summation and stitching. These strategies aim to fully utilize the complementarity between infrared and visible light images to improve target detection accuracy. The feature prediction module outputs target detection results based on multi-scale weighted fused features. This typically involves a decoder (such as a Deformable DETR decoder) that can convert the fused features into the output format required for target detection, including information such as the target's category, location, and confidence level.

[0038] Preferably, the infrared multi-scale feature extraction module adopts a first ResNet50 model, and the visible light multi-scale feature extraction module adopts a second ResNet50 model;

[0039] The first ResNet50 model is used to extract features from infrared images. The output feature maps of the first ResNet50 model in the first, second, and third feature extraction stages are combined to form multi-scale features of the infrared image. i∈{1,2,3} This represents the output feature map of the first ResNet50 model in the i-th feature extraction stage;

[0040] The second ResNet50 model is used to extract features from visible light images. The output feature maps of the second ResNet50 model in the first, second, and third feature extraction stages are combined to form a multi-scale feature map of the visible light image. i∈{1,2,3} This represents the output feature map of the second ResNet50 model in the i-th feature extraction stage.

[0041] Please see Figure 6 ResNet50 includes feature extraction stages STAGE0 to STAGE4. In this invention, the output feature maps of STAGE1, STAGE2, and STAGE3 are used to form the multi-scale features of the infrared image.

[0042] Preferably, the cross-attention enhancement of multi-scale features of the infrared image based on the infrared category feature set includes:

[0043] S301: Feature maps of each scale of the infrared image Global average pooling is performed separately for each scale. The features from all scales after global average pooling are then concatenated and input into the classifier to predict the probability distribution of each category in the infrared image.

[0044] S302: For the k categories with the highest probability in the infrared image, select the infrared category features corresponding to the k categories from the infrared category feature set;

[0045] S303: Flatten each selected infrared category feature along the spatial dimension, and use the concatenated feature sequence of the flattened k infrared category features as the key vector K. R And numerical vector V R ;

[0046] S304: Feature maps of each scale of the infrared image The image is divided into multiple image blocks along the spatial dimension, ensuring that the width and height of each image block are the same as those of the infrared category features in the infrared category feature set;

[0047] S305: Flatten each cropped image patch along the spatial dimension, and use the feature sequence of all flattened image patches as the query vector Q. Ri ;

[0048] S306: Transfer the key vector K R Numerical vector V R and query vector Q Ri Perform cross-attention processing, and reshape the cross-attention feature map back to its original shape. Enhancement features at each scale of the infrared image obtained at the same size i∈{1,2,3}, and thus construct multi-scale enhancement features for infrared images.

[0049] in, Let represent the enhancement feature at the i-th scale of the infrared image, reshape represent the reshaping function, d represent the temperature coefficient, and T represent the transpose.

[0050] Preferably, the cross-attention enhancement of multi-scale features of the visible light image based on the visible light category feature set includes:

[0051] S311: Feature maps of each scale of the visible light image Global average pooling is performed separately for each scale. The features from all scales after global average pooling are then concatenated and input into the classifier to predict the probability distribution of each category in the visible light image.

[0052] S312: For the k categories with the highest probability in the visible light image, select the visible light category features corresponding to the k categories from the visible light category feature set;

[0053] S313: Flatten each selected visible light category feature along the spatial dimension, and use the concatenated feature sequence of the flattened k visible light category features as the key vector K. V And numerical vector V V ;

[0054] S314: Feature maps of visible light images at each scale The image is divided into multiple image blocks along the spatial dimension, ensuring that the width and height of each image block are the same as those of the visible light category features in the visible light category feature set.

[0055] S315: Flatten each cropped image patch along the spatial dimension, and use the feature sequence of all flattened image patches as the query vector Q. Vi ;

[0056] S316: Transfer the key vector K V Numerical vector VV and query vector Q Vi Perform cross-attention processing, and reshape the cross-attention feature map back to its original shape. Enhanced features at each scale of a visible light image are obtained at the same size. i∈{1,2,3}, and thus construct multi-scale enhancement features for visible light images.

[0057]

[0058] in, Let represent the enhancement feature at the i-th scale of the visible light image, reshape represent the reshaping function, d represent the temperature coefficient, and T represent the transpose.

[0059] In this embodiment, a cross-attention mechanism is used to enhance the multi-scale features of infrared and visible light images using infrared and visible light category feature sets. This enhancement method enables the model to focus more on category-related features, thereby improving the accuracy of target detection. Through steps such as global average pooling, category probability prediction, filtering category features, constructing key vectors, numerical vectors and query vectors, and cross-attention processing, multi-scale enhanced features for infrared and visible light images are successfully constructed, providing strong support for subsequent feature fusion and target detection. Figure 7 As shown.

[0060] Preferably, the feature fusion module is used to fuse multi-scale enhancement features of infrared images and multi-scale enhancement features of visible light images to obtain multi-scale weighted fusion features, including:

[0061] S321: Input the enhanced features of each scale of the infrared image into the first dilated convolutional encoder for encoding processing to obtain the multi-scale mixed features of the infrared image;

[0062] S322: Input the enhanced features of each scale of the visible light image into the second dilated convolutional encoder for encoding processing to obtain the multi-scale mixed features of the visible light image;

[0063] S323: Multiscale weighted fusion features are obtained by weighted fusion of multiscale fusion features of infrared images and visible light images at corresponding scales.

[0064] S324: Multiscale weighted fusion features are obtained by stitching together multiscale weighted fusion features and multiscale fusion features of infrared images.

[0065] Preferably, the multi-scale weighted fusion features include:

[0066]

[0067] in, w represents the weighted fusion feature at the i-th scale. Ri and w Vi dc1(·) represents the weight factor; dc2(·) represents the first dilated convolutional encoder; dc2(·) represents the second dilated convolutional encoder. This represents the enhancement feature at the i-th scale of the infrared image; Let represent the enhancement features at the i-th scale of the visible light image, and ... Figure 8 As shown.

[0068] Preferably, the feature prediction module is used to output target detection results based on multi-scale weighted fusion features, including: [the following is a partial translation of the original text, which is incomplete and requires further context.] The feature sequence is flattened in the spatial dimension and then input into the Deformable DETR decoder to obtain the target detection result.

[0069] In this embodiment, multi-scale feature fusion and target detection of infrared and visible light images are achieved through a feature fusion module and a feature prediction module. The feature fusion module improves the representational power of features by using an expanded convolutional encoder and a weighted fusion strategy; the feature prediction module achieves accurate target detection by utilizing a Deformable DETR decoder. This fusion method can fully utilize the complementarity between infrared and visible light images to improve the accuracy and robustness of target detection. The model structure of the Deformable DETR decoder is as follows: Figure 9 As shown.

[0070] S4: Input the infrared-visible light image into the target detection model, guide the model to learn small target features through the infrared category feature set and the visible light category feature set, construct a loss function based on the prediction results and annotation information of the target detection model, optimize the parameters of the target detection model, and obtain a trained target detection model.

[0071] Preferably, the construction loss function includes:

[0072] L total =L DETR +λ R-guide L R-guided +λ V-guided L V-guided

[0073] L DETR =λ cls L cls +λ bbox L bbox

[0074]

[0075] Among them, L total L represents the total loss function. DETR This represents DETR Loss, which is responsible for optimizing the classification and bounding boxes of object detection; L R-guided and L V-guided Represent the infrared category-guided loss function and the visible light category-guided loss function, respectively, responsible for optimizing the category prediction accuracy and feature matching of the infrared category feature-guided module and the visible light category feature-guided module; N represents the number of samples; λ cls , λ bbox , λ R-guided , λ V-guided , λ r-feat , λ v-f , λ L1 and λ GIOU To balance the hyperparameters of each loss; L1(·) represents the L1 loss function; GIOU(·) is used to measure the degree of overlap between the predicted bounding box of the object detection model and the ground truth bounding box; b represents the predicted bounding box of the object detection model. i This represents the true bounding box in the annotation information; y Ri y represents the category label of the i-th class in the annotation information. Vi y represents the category label of the i-th class in the annotation information. i This represents the category label of the i-th class in the annotation information, with values ​​of either 0 or 1. This represents the probability of the i-th class predicted by the classifier in the infrared category feature guidance module. This represents the probability of the i-th class predicted by the classifier in the visible light category feature guidance module. This represents the probability of the i-th class predicted by the object detection model; and These represent the output feature maps of the first ResNet50 model and the second ResNet50 model in the i-th feature extraction stage, respectively. and Let represent the enhancement features at the i-th scale of the infrared image and the enhancement features at the i-th scale of the visible light image, respectively, and let ||.|2 represent the L2 loss.

[0076] In this embodiment, the total loss function L total By comprehensively considering DETR loss, infrared category-guided loss, and visible light category-guided loss, the performance of the target detection model can be fully optimized; through classification loss L... cls The model can predict the category of the target more accurately and reduce classification errors; bounding box loss L bboxCombining L1 loss and GIOU loss enables more accurate prediction of target bounding boxes, improving localization accuracy. Infrared category-guided loss L... R-guided and visible light category guidance loss L V-guided The classification loss function optimizes the class prediction capability of the guidance module, improving its accuracy. The feature matching loss function uses L2 loss to measure the difference between the enhanced features output by the guidance module and the feature map output by the ResNet50 model, helping the model learn more robust and accurate feature representations. By utilizing infrared and visible light category feature sets to guide the model in learning small target features, the complementarity of the two modalities is fully utilized, improving the model's ability to detect small targets. In the feature fusion module, multi-scale weighted feature fusion further extracts and utilizes feature information at different scales, helping the model better capture the details and contextual information of small targets. The above loss functions comprehensively optimize and improve the object detection model by considering multiple aspects, including the classification and bounding box prediction capabilities of the object detection model, the effect of infrared and visible light category feature guidance, the impact of different loss terms, and the model's ability to detect small targets. This design makes the model more robust and accurate in complex scenes.

[0077] In summary, this invention extracts multi-scale features from infrared and visible light images using infrared and visible light multi-scale feature extraction modules, respectively. The feature fusion module enables deeper interaction and fusion of features from different modalities, thereby enhancing the model's ability to represent small targets. This method uses InfMAE and McMAE models to extract features from infrared and visible light target region maps, respectively, constructing infrared and visible light category feature sets. By constructing these feature sets, this method can more accurately capture the feature differences between target categories, improving the model's ability to distinguish between different target categories. The infrared and visible light category feature guidance modules utilize the constructed feature sets to guide the model in learning small target features, enabling the model to focus more on small target feature learning during training, thus improving the model's detection accuracy and generalization ability. By combining infrared and visible light images and utilizing dual-modal feature fusion and category feature guidance, this method significantly improves the accuracy and robustness of small target detection. Even in complex application scenarios, such as changes in illumination, target occlusion, and complex backgrounds, this method maintains high detection performance.

[0078] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A dual-modal small target detection method based on category feature guidance, which utilizes a trained target detection model to perform target detection on infrared-visible image pairs to obtain target detection results, characterized in that... The training process of the target detection model includes: S1: Based on the category bounding boxes in the annotation information of the infrared-visible image pair, crop the regions corresponding to the category bounding boxes in the infrared image and the visible image respectively to obtain the corresponding infrared category target region map and the visible light category target region map; S2: Infrared target features are obtained by extracting features from the infrared target area map using the InfMAE model. An infrared category feature set is constructed based on the infrared target features of all categories. Visible light target features are obtained by extracting features from the visible light target area map using the McMAE model. A visible light category feature set is constructed based on the visible light target features of all categories. S3: Construct a target detection model, wherein the target detection model includes: an infrared multi-scale feature extraction module, a visible light multi-scale feature extraction module, an infrared category feature guidance module, a visible light category feature guidance module, a feature fusion module, and a feature prediction module; The infrared multi-scale feature extraction module is used to extract multi-scale features from infrared images to obtain multi-scale features of the infrared images; the visible light multi-scale feature extraction module is used to extract multi-scale features from visible light images to obtain multi-scale features of the visible light images; the infrared category feature guidance module is used to perform cross-attention enhancement on the multi-scale features of the infrared images according to the infrared category feature set to obtain multi-scale enhanced features of the infrared images; the visible light category feature guidance module is used to perform cross-attention enhancement on the multi-scale features of the visible light images according to the visible light category feature set to obtain multi-scale enhanced features of the visible light images; the feature fusion module is used to fuse the multi-scale enhanced features of the infrared images and the multi-scale enhanced features of the visible light images to obtain multi-scale weighted fusion features; the feature prediction module is used to output target detection results based on the multi-scale weighted fusion features. S4: Input the infrared-visible light image into the target detection model, guide the model to learn small target features through the infrared category feature set and the visible light category feature set, construct a loss function based on the prediction results and annotation information of the target detection model, optimize the parameters of the target detection model, and obtain a trained target detection model.

2. The dual-modal small target detection method based on category feature guidance according to claim 1, characterized in that, The construction of the infrared category feature set and the visible light category feature set includes: S21: Scale the size of all cropped infrared category target area maps and visible light category target area maps to the preset size, so that all cropped infrared category target area maps and visible light category target area maps are adjusted to have the same width and height; S22: Feature extraction is performed on the infrared target region map using the InfMAE model. The feature map output from the first encoding layer of the InfMAE model is used as the infrared target feature. The center feature vector of all infrared target features under each category is calculated as the infrared category feature of that category. The infrared category features of all categories are combined into an infrared category feature set. ,in, , Indicates the first Infrared category characteristics of each category, Indicates the total number of categories; S23: Feature extraction is performed on the visible light category target region map using the McMAE model. The feature map output from the first encoding stage of the McMAE model is used as the visible light target feature. The central feature vector of all visible light target features under each category is calculated as the visible light category feature for that category. The visible light category features of all categories are combined into a visible light category feature set. ,in, , Indicates the first Visible light category characteristics of each category, Indicates the total number of categories. and Indicates height and width, Indicates the number of channels.

3. The dual-modal small target detection method based on category feature guidance according to claim 2, characterized in that, The infrared multi-scale feature extraction module adopts the first ResNet50 model, and the visible light multi-scale feature extraction module adopts the second ResNet50 model. The first ResNet50 model is used to extract features from infrared images. The output feature maps of the first ResNet50 model in the first, second, and third feature extraction stages are combined to form multi-scale features of the infrared image. , This indicates that the first ResNet50 model is in the... Output feature maps of each feature extraction stage; The second ResNet50 model is used to extract features from visible light images. The output feature maps of the second ResNet50 model in the first, second, and third feature extraction stages are combined to form a multi-scale feature map of the visible light image. , This indicates that the second ResNet50 model is in the... The output feature map of each feature extraction stage.

4. The dual-modal small target detection method based on category feature guidance according to claim 3, characterized in that, The cross-attention enhancement of multi-scale features of infrared images based on infrared category feature sets includes: S301: Feature maps of each scale of the infrared image Global average pooling is performed separately for each scale. The features from all scales after global average pooling are then concatenated and input into the classifier to predict the probability distribution of each category in the infrared image. ; S302: Targeting the highest probability in infrared images Each category is selected from the infrared category feature set. Infrared category features corresponding to each category; S303: Flatten each selected infrared category feature along the spatial dimension, and then flatten the resulting feature... The feature sequence obtained by concatenating infrared category features is used as the key vector. and numerical vectors ; S304: Feature maps of each scale of the infrared image The image is divided into multiple image blocks along the spatial dimension, ensuring that the width and height of each image block are the same as those of the infrared category features in the infrared category feature set; S305: Flatten each cropped image patch along the spatial dimension, and use the feature sequence of all flattened image patches as the query vector. ; S306: Transfer key vectors Numerical vectors and query vector Perform cross-attention processing, and then process the feature maps after cross-attention. For feature map Enhancement features at each scale of the infrared image obtained at the same size This leads to the construction of multi-scale enhancement features for infrared images. ; in, Indicates the infrared image number 1 Enhanced features at each scale, Represents the reshaping function. Indicates the temperature coefficient. This indicates transpose.

5. The dual-modal small target detection method based on category feature guidance according to claim 4, characterized in that, The method of performing cross-attention enhancement on the multi-scale features of visible light images based on the visible light category feature set includes: S311: Feature maps of each scale of the visible light image Global average pooling is performed separately for each scale. The features from all scales after global average pooling are then concatenated and input into the classifier to predict the probability distribution of each category in the visible light image. ; S312: Targeting the highest probability in visible light images. Categories were selected from the visible light category feature set. Visible light category features corresponding to each category; S313: Flatten each selected visible light category feature along the spatial dimension, and then flatten the resulting... The feature sequence obtained by concatenating visible light category features is used as the key vector. and numerical vectors ; S314: Feature maps of visible light images at each scale The image is divided into multiple image blocks along the spatial dimension, ensuring that the width and height of each image block are the same as those of the visible light category features in the visible light category feature set. S315: Flatten each cropped image patch along the spatial dimension, and use the feature sequence of all flattened image patches as the query vector. ; S316: Transfer key vectors Numerical vectors and query vector Perform cross-attention processing, and then process the feature maps after cross-attention. For feature map Enhanced features at each scale of a visible light image are obtained at the same size. This leads to the construction of multi-scale enhancement features for visible light images. ; in, Represents the visible light image. Enhanced features at each scale, Represents the reshaping function. Indicates the temperature coefficient. This indicates transpose.

6. The dual-modal small target detection method based on category feature guidance according to claim 5, characterized in that, The feature fusion module is used to fuse multi-scale enhancement features of infrared images and multi-scale enhancement features of visible light images to obtain multi-scale weighted fusion features, including: S321: Input the enhanced features of each scale of the infrared image into the first dilated convolutional encoder for encoding processing to obtain the multi-scale mixed features of the infrared image; S322: Input the enhanced features of each scale of the visible light image into the second dilated convolutional encoder for encoding processing to obtain the multi-scale mixed features of the visible light image; S323: Multiscale weighted fusion features are obtained by weighted fusion of multiscale fusion features of infrared images and visible light images at corresponding scales. S324: Multiscale weighted fusion features are obtained by stitching together multiscale weighted fusion features and multiscale fusion features of infrared images.

7. The dual-modal small target detection method based on category feature guidance according to claim 6, characterized in that, The multi-scale weighted fusion features include: in, Indicates the first Weighted fusion features at each scale and Indicates the weighting factor; Indicates the first dilated convolutional encoder; This indicates the second dilated convolutional encoder; Indicates the infrared image number 1 Enhanced features at each scale; Represents the visible light image. Enhanced features at each scale, This represents the concatenation function.

8. The dual-modal small target detection method based on category feature guidance according to claim 7, characterized in that, The feature prediction module is used to output target detection results based on multi-scale weighted fusion features, including: calculating the multi-scale weighted fusion features... The feature sequence is flattened in the spatial dimension and then input into the Deformable DETR decoder to obtain the target detection result.

9. The dual-modal small target detection method based on category feature guidance according to claim 8, characterized in that, The construction loss function includes: in, Represents the total loss function. This represents DETR Loss, which is responsible for optimizing the classification and bounding boxes of object detection. and These represent the infrared category-guided loss function and the visible light category-guided loss function, respectively, which are responsible for optimizing the category prediction accuracy and feature matching of the infrared category feature-guided module and the visible light category feature-guided module. Indicates the number of samples; , , , , , , and To balance the hyperparameters of each loss; Represents the L1 loss function; Used to measure the degree of overlap between the bounding boxes predicted by the object detection model and the ground truth bounding boxes; This represents the predicted bounding box of the object detection model. This represents the actual bounding box in the annotation information; Indicates the first in the annotation information Class category label, Indicates the first in the annotation information Class category label, Indicates the first in the annotation information The category label for a class can take values ​​of 0 or 1. This indicates the first prediction made by the classifier in the infrared category feature guidance module. Probability of class This represents the first prediction made by the classifier in the visible light category feature guidance module. Probability of class The first number predicted by the object detection model is... The probability of a class; These represent the first ResNet50 model at the [missing information] level. The output feature maps of the first feature extraction stage and the second ResNet50 model in the second feature extraction stage. Output feature maps of each feature extraction stage; These represent the infrared images of the first and second generations, respectively. Enhanced features at each scale and visible light image Enhanced features at each scale, This indicates L2 loss.

Images