A small target detection method and system based on multispectral image fusion

Through multi-spectral image fusion and depth feature extraction, combined with attention mechanism, the problem of insufficient accuracy in small object detection in traditional single spectral images is solved, and high accuracy and robust small object detection in complex environments is achieved.

CN118675057BActive Publication Date: 2024-11-19CHANGSHA CHAOCHUANG ELECTRONICS TECH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411171362.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-08-26
Publication Date
2024-11-19
Estimated Expiration
2044-08-26

AI Technical Summary

Technical Problem

Traditional single spectral images have problems with insufficient accuracy and robustness in small-object detection, especially in complex environments that are susceptible to light changes and occlusion, making it difficult to effectively utilize the information in multi-spectral images.

Method used

Using multispectral image fusion method, images are collected in different bands through multispectral cameras, pixel-level fusion is performed, and features are extracted using pre-trained deep convolutional neural networks, and comprehensive loss function is set for training. , improve the accuracy and robustness of small object detection.

Benefits of technology

Through multi-spectral image fusion and depth feature extraction, the detection accuracy and robustness of small targets in complex backgrounds are improved, the retention of spectral features and image clarity are enhanced, and the detection accuracy and stability.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118675057B_ABST
    Figure CN118675057B_ABST
Patent Text Reader

Abstract

The present invention discloses a small target detection method and system based on multispectral image fusion, including: using a multispectral camera to collect images of a target area in different bands to obtain a multispectral image, and preprocessing the images of each band in the multispectral image; fusing the preprocessed multispectral image at the pixel level to obtain a final fused image; using a pretrained deep convolutional neural network to extract the deep features of the final fused image; constructing a target detection network with an attention mechanism based on the deep features and the features of the preprocessed multispectral image; setting the loss function of the target detection network and using the multispectral image with annotated results for training, and using the trained target detection network to perform small target detection on the multispectral image without annotated results. The present invention can utilize the rich information of multispectral images, combine deep features and attention mechanisms, and effectively improve the accuracy and robustness of small target detection.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of target detection, and in particular to a small target detection method and system based on multispectral image fusion. Background Art

[0002] In the development of modern computer vision and remote sensing technology, multispectral images have been widely used in target detection, classification and recognition tasks because they can capture spectral information in different bands. However, how to effectively utilize the information in multispectral images, especially for the detection of small targets, remains a challenging research topic. Traditional target detection methods are mostly based on single spectrum images, which to a certain extent limits the accuracy and robustness of detection. Single spectrum images usually cannot fully reflect the spectral characteristics of the target, especially in complex environments, and are easily affected by factors such as illumination changes and occlusion. Considering that multispectral images can provide richer spectral information by collecting images in different bands, it is helpful to improve the accuracy and stability of target detection. Therefore, it is very urgent to propose an effective multi-source data spectral image processing method in the field of small target detection. Summary of the invention

[0003] In view of this, the present invention provides a small target detection method and system based on multispectral image fusion, aiming to utilize the rich information of multispectral images, combine deep features and attention mechanism, and effectively improve the accuracy and robustness of small target detection.

[0004] In order to achieve the above object, the present invention provides a small target detection method based on multispectral image fusion, comprising the following steps:

[0005] S1. Use a multispectral camera to collect images of the target area in different bands to obtain a multispectral image , and preprocess each band image in the multispectral image;

[0006] S2, fusing the preprocessed multispectral images at the pixel level to obtain a final fused image;

[0007] S21, weighted average

[0008] The weighting coefficient is calculated based on the contrast and clarity of the preprocessed image of each band, wherein the contrast is calculated by the standard deviation of the preprocessed image of each band. The sharpness is measured by the gradient strength of each band of the preprocessed image. to measure;

[0009] The weighting coefficients of the image of each band after preprocessing are calculated by combining contrast and clarity, specifically:

[0010] ;

[0011] in, For multispectral images The number of bands, the image of each band is recorded as and Indicates the band number, After preprocessing The weighting coefficient of the image of each band; ;

[0012] S22. Constructing Laplace Pyramid

[0013] Construct a Laplacian pyramid for each band of the preprocessed image. First, construct a Gaussian pyramid, specifically:

[0014] ;

[0015] in, represents the pixel position of the image, For the Bands in the Gaussian pyramid The image of the layer is located at The value at , , For the The band is The layer Gaussian pyramid is at position The value at

[0016] The Laplacian pyramid is further constructed by the Gaussian pyramid:

[0017] ;

[0018] in, For the The bands in the Laplace pyramid Layer images, is the upsampling operation, and Respectively Bands in the Gaussian pyramid Layer and Layer images;

[0019] S23. Image Fusion

[0020] Perform weighted average fusion on the Laplacian pyramid layers of each band:

[0021] ;

[0022] in, is the fused Laplacian pyramid Layer images, For the The bands in the Laplace pyramid Layer images;

[0023] Reconstruct the final fused image based on the fused Laplacian pyramid , specifically:

[0024] ;

[0025] in, , and They are the fused Laplacian pyramid Layer, Layer and Layer images;

[0026] S3, use the pre-trained deep convolutional neural network to extract the deep features of the final fused image;

[0027] S4, constructing an object detection network with attention mechanism based on deep features and features extracted from pre-processed multispectral images;

[0028] S5. Set the loss function of the target detection network and use the multispectral images with labeled results for training. Use the trained target detection network to detect small targets in the multispectral images without labeled results.

[0029] As a further improvement method of the present invention:

[0030] Optionally, in step S1, a multispectral camera is used to collect images of the target area in different bands to obtain a multispectral image. , and preprocess each band image in the multispectral image, including the following processes:

[0031] For multispectral images The images of each band in are preprocessed, and the preprocessing includes image denoising and image enhancement, specifically:

[0032] S11. Image denoising

[0033] The image of each band is denoised using an improved variable step-size median filtering algorithm. The variable step-size median filtering algorithm first estimates the local standard deviation of the image and defines a local window. , represents the pixel position of the image, and the local window size is , is the local window radius, and the local standard deviation of the image is:

[0034] ;

[0035] in, For the Bands at location The local standard deviation at for The pixel position within for In Location The value at For the Bands at location The mean value at is:

[0036] ;

[0037] according to Dynamically adjust step size and , let the maximum step size be , with a minimum value of 1, and defines the noise level threshold , specifically:

[0038] ;

[0039] Use the dynamically adjusted step size to perform median filtering on the image, and the filter window size is , get the image of each band after filtering , the filtering process is:

[0040] ;

[0041] in, and is the relative displacement within the filter window, for In Location The value at represents the median function, for In Location The value at

[0042] S12. Image Enhancement

[0043] For each local window Calculate the local histogram, specifically:

[0044] ;

[0045] in, represents the gray level, For the The band is The gray level in the window is The value of is a pulse function, when hour is 1, otherwise it is 0;

[0046] Perform contrast limitation on the local histogram and set the contrast limitation threshold to ,but:

[0047] ;

[0048] in, is the first The band is The gray level in the window is The value of is the minimum function;

[0049] The parts exceeding the contrast limit threshold are evenly distributed to other gray levels, specifically:

[0050] ;

[0051] in, After the allocation The band is The gray level in the window is The value of , is the maximum image gray level, is the maximum value function;

[0052] Compute the cumulative distribution function within a local window and normalize it:

[0053] ;

[0054] ;

[0055] in, For the The band is The gray level in the window is The cumulative value of For the The band is The accumulated value of the gray level in the window is 0, After normalization ;

[0056] Map each pixel value to obtain the preprocessed image of each band :

[0057] ;

[0058] in, for In Location The value at Function is used for rounding. For the The band is The gray level in the window is The normalized cumulative value of

[0059] Preprocessed images of each band Together they form a preprocessed multispectral image .

[0060] Optionally, in step S3, a pre-trained deep convolutional neural network is used to extract the deep features of the final fused image, including the following process:

[0061] The pre-trained ResNet50 model is used to extract the deep features of the final fused image. The ResNet50 model consists of several residual blocks, each of which contains a convolutional layer, a normalization layer, an activation layer, and a pooling layer, specifically:

[0062] ;

[0063] in, and ResNet50 model Layer and The output feature map of the layer, , is the number of residual blocks, , For the Layer operations, including convolution, normalization, activation, and pooling;

[0064] go through After residual blocks, the feature map Input into global average pooling to generate deep features .

[0065] Optionally, in step S4, constructing a target detection network with an attention mechanism based on the deep features and the features of the preprocessed multispectral image extracted includes the following processes:

[0066] S41. Feature extraction and fusion

[0067] Extract the features of the preprocessed multispectral image, specifically:

[0068] ;

[0069] in, is the convolutional layer of YOLOv5, is the preprocessed multispectral image, is the extracted features of the preprocessed multispectral image;

[0070] For deep features Remapping is performed, specifically:

[0071] ;

[0072] in, is the fully connected layer, is the deep feature after remapping;

[0073] The extracted pre-processed multispectral image features and the remapped depth features are fused as follows:

[0074] ;

[0075] in, is the fused feature;

[0076] S42. Adding attention mechanism

[0077] Use attention mechanism to process fused features , the attention mechanism includes spatial attention and channel attention, specifically:

[0078] ;

[0079] in, is the attention weight map, is a type B function, is a two-dimensional convolution operation, is a linear rectification function, is the global pooling function, and are the trainable parameters included in the attention mechanism;

[0080] Apply the attention weight map to , and obtain the enhanced features , specifically:

[0081] ;

[0082] S43, set the detection head

[0083] Input the enhanced features into the detection head of YOLOv5 to generate prediction results , specifically:

[0084] ;

[0085] in, is the prediction result of the target detection network, , For the predicted The bounding box of the detected object, including the center coordinates of the bounding box and the width and height of the bounding box, For the predicted The class probability of the detected targets, is the number of targets contained in the prediction results of the target detection network, It is the detection head of YOLOv5.

[0086] Optionally, in step S5, the loss function of the target detection network is set and the multispectral image with the labeled results is used for training, and the trained target detection network is used to perform small target detection on the multispectral image without the labeled results, including the following process:

[0087] S51. Setting loss function

[0088] The loss function consists of bounding box regression loss, category classification loss, and feature enhancement loss with the introduction of the attention mechanism, specifically:

[0089] The bounding box regression loss is formed by using the intersection-over-union loss and the center point loss :

[0090] ;

[0091] in, For the predicted The bounding box coordinates of the detected objects, Marked The bounding box coordinates of the detected objects, and are the center coordinates of the predicted and labeled bounding boxes, respectively. ;

[0092] Use cross entropy loss as the category classification loss :

[0093] ;

[0094] in, Marked The categories of the detected targets are The probability of For the predicted The categories of the detected targets are The probability of is the total number of categories;

[0095] Feature enhancement loss using attention mechanism :

[0096] ;

[0097] in, is the attention weight map for the labeled target;

[0098] The complete loss function is:

[0099] ;

[0100] in, , and is the weight factor;

[0101] S52. Training network

[0102] Use the back-propagation algorithm to calculate the gradient and update the target detection network weights:

[0103] ;

[0104] in, is the target detection network weight, is the learning rate, for about The partial derivative of

[0105] S53, Application Network

[0106] The multispectral image without annotation results is processed in steps S1-S3, and then the trained target detection network is used to detect small targets.

[0107] The present invention also provides a small target detection system based on multispectral image fusion, comprising:

[0108] Preprocessing module: Use a multispectral camera to collect images of the target area in different bands to obtain a multispectral image, and preprocess the images of each band in the multispectral image;

[0109] Fusion module: fuses the preprocessed multispectral images at the pixel level to obtain the final fused image;

[0110] Deep feature extraction module: uses a pre-trained deep convolutional neural network to extract the deep features of the final fused image;

[0111] Network construction module: Build an object detection network with attention mechanism based on deep features and features extracted from pre-processed multispectral images;

[0112] Network training and application module: Set the loss function of the target detection network and use multispectral images with labeled results for training. Use the trained target detection network to detect small targets in multispectral images without labeled results.

[0113] In order to realize the above-mentioned small target detection method based on multispectral image fusion.

[0114] The technical solution of the present invention has at least the following beneficial effects:

[0115] The present invention collects images in different bands through a multispectral camera to obtain rich spectral information of the target area. The advantages of images in each band are combined at the pixel level through a multispectral image fusion method, which improves the retention of target details and spectral features, allowing small targets to be detected more accurately in complex backgrounds.

[0116] The present invention introduces an improved variable step-size median filtering algorithm and a contrast-limited histogram equalization method in the image preprocessing stage to perform image denoising and enhancement processing respectively, which not only effectively removes the noise in the image and improves the clarity and contrast of the image, but also enhances the edge information and details of the image, making the subsequent feature extraction more accurate.

[0117] The present invention uses a pre-trained deep convolutional neural network to extract deep features of the fused image, and utilizes the powerful feature extraction capability of the deep convolutional neural network to extract high-level image features from the fused image, providing a more discriminative feature representation for small target detection.

[0118] The present invention introduces channel attention and spatial attention mechanisms into the target detection network. The attention mechanism dynamically adjusts the feature weights of different channels and spatial positions to highlight key information and suppress irrelevant background, further improving the effect of small target detection.

[0119] The present invention adopts a method combining weighted averaging and Laplacian pyramid for image fusion. The detailed information and structural features of the image are retained through multi-scale fusion, so that small targets at different scales can be effectively detected. Among them, the Laplacian pyramid has significant advantages in multi-scale feature representation, which helps to improve the robustness and accuracy of detection.

[0120] The present invention combines the intersection-over-union loss and the center point distance loss in the design of the loss function, and further introduces the attention feature enhancement loss to deal with the imbalance problem of positive and negative samples, and enhances the detection ability of small targets. The comprehensive loss function is used to better guide the network optimization, thereby improving the detection accuracy and stability. BRIEF DESCRIPTION OF THE DRAWINGS

[0121] Figure 1 A schematic diagram of a small target detection method based on multispectral image fusion according to Embodiment 1 of the present invention;

[0122] Figure 2 Schematic diagram of multispectral image fusion results in Example 1 of the present invention, wherein: Figure 2 (a) is the original image, Figure 2 (b) is the fused image. DETAILED DESCRIPTION

[0123] The present invention is further described below in conjunction with the accompanying drawings, but the present invention is not limited in any way. Any changes or substitutions made based on the teachings of the present invention belong to the protection scope of the present invention.

[0124] Example 1

[0125] A small target detection method based on multispectral image fusion, such as Figure 1 As shown, the following steps are included:

[0126] S1. Use a multispectral camera to collect images of the target area in different bands to obtain a multispectral image , and preprocess the images of each band in the multispectral image.

[0127] Use a multispectral camera to collect images of the target area in different bands to obtain a multispectral image. Assume that the multispectral image collected by the multispectral camera The number of bands is , the image of each band is recorded as ,in Indicates the band number, Represents the pixel position of the image;

[0128] For multispectral images The images of each band in are preprocessed, and the preprocessing includes image denoising and image enhancement, specifically:

[0129] S11. Image denoising

[0130] The image of each band is denoised using an improved variable step-size median filtering algorithm. The variable step-size median filtering algorithm first estimates the local standard deviation of the image and defines a local window. , represents the pixel position of the image, and the local window size is , is the local window radius, and the local standard deviation of the image is:

[0131] ;

[0132] in, For the Bands at location The local standard deviation at for The pixel position within for In Location The value at For the Bands at location The mean value at is:

[0133] ;

[0134] according to Dynamically adjust step size and , let the maximum step size be , with a minimum value of 1, and defines the noise level threshold , specifically:

[0135] ;

[0136] Use the dynamically adjusted step size to perform median filtering on the image, and the filter window size is , get the image of each band after filtering , the filtering process is:

[0137] ;

[0138] in, and is the relative displacement within the filter window, for In Location The value at represents the median function, for In Location The value at

[0139] S12. Image Enhancement

[0140] For each local window Calculate the local histogram, specifically:

[0141] ;

[0142] in, represents the gray level, For the The band is The gray level in the window is The value of is a pulse function, when hour is 1, otherwise it is 0;

[0143] Perform contrast limitation on the local histogram and set the contrast limitation threshold to ,but:

[0144] ;

[0145] ;

[0146] in, is the first The band is The gray level in the window is The value of

[0147] The parts exceeding the contrast limit threshold are evenly distributed to other gray levels, specifically:

[0148] ;

[0149] ;

[0150] in, After the allocation The band is The gray level in the window is The value of , is the maximum image gray level;

[0151] Compute the cumulative distribution function within a local window and normalize it:

[0152] ;

[0153] ;

[0154] in, For the The band is The gray level in the window is The cumulative value of For the The band is The accumulated value of the gray level in the window is 0, After normalization ;

[0155] Map each pixel value to obtain the preprocessed image of each band :

[0156] ;

[0157] in, for In Location The value at Function is used for rounding. For the The band is The gray level in the window is The normalized cumulative value of

[0158] Preprocessed images of each band Together they form a preprocessed multispectral image .

[0159] In preprocessing, the use of an improved variable step-size median filter algorithm can effectively remove noise from the image. This algorithm dynamically adjusts the size of the filter window according to the local noise level of the image, which can remove noise while retaining image details, helping to improve the effect of subsequent feature extraction.

[0160] During the preprocessing process, the contrast-limited histogram equalization method is used to enhance the image contrast. This method improves the local contrast of the image by limiting the contrast of the histogram and redistributing the grayscale values, making the edges and details of the target in the image clearer, which helps to improve the accuracy of target detection.

[0161] The images of each band are preprocessed to make them consistent in terms of noise level, contrast and clarity. This uniform image quality provides a good foundation for subsequent multispectral image fusion and feature extraction, ensuring the quality of the fused image and the reliability of feature extraction.

[0162] S2. Fusing the preprocessed multispectral images at the pixel level to obtain the final fused image.

[0163] The preprocessed multispectral image is fused at the pixel level based on weighted average and Laplace pyramid, specifically:

[0164] S21, weighted average

[0165] The weighting coefficient is calculated based on the contrast and clarity of the preprocessed image of each band, wherein the contrast is calculated by the standard deviation of the preprocessed image of each band. The sharpness is measured by the gradient strength of each band of the preprocessed image. to measure;

[0166] The weighting coefficients of the image of each band after preprocessing are calculated by combining contrast and clarity, specifically:

[0167] ;

[0168] in, After preprocessing The weighting coefficient of the image of each band; ;

[0169] S22. Constructing Laplace Pyramid

[0170] Construct a Laplacian pyramid for each band of the preprocessed image. First, construct a Gaussian pyramid, specifically:

[0171] ;

[0172] in, For the Bands in the Gaussian pyramid The image of the layer is located at The value at , , For the The band is The layer Gaussian pyramid is at position The value at

[0173] The Laplacian pyramid is further constructed by the Gaussian pyramid:

[0174] ;

[0175] in, For the The bands in the Laplace pyramid Layer images, is the upsampling operation, and Respectively Bands in the Gaussian pyramid Layer and Layer images;

[0176] S23. Image Fusion

[0177] Perform weighted average fusion on the Laplacian pyramid layers of each band:

[0178] ;

[0179] in, is the fused Laplacian pyramid Layer images, For the The bands in the Laplace pyramid Layer images;

[0180] Reconstruct the final fused image based on the fused Laplacian pyramid ,like Figure 2 As shown, specifically:

[0181] ;

[0182] in, , and They are the fused Laplacian pyramid Layer, Layer and Layer of image.

[0183] Each band image contains different spectral information. By fusing at the pixel level, these spectral information can be integrated to more comprehensively reflect the characteristics of the target. The fused image not only retains the advantages of each band, but also improves the overall quality of the image, which helps to detect and identify targets more accurately.

[0184] Through pixel-level fusion, the detail information in multispectral images can be preserved and enhanced. For example, weighted average and Laplacian pyramid fusion methods can integrate image details at different scales, making small targets and edge information clearer and more prominent, thus improving detection accuracy.

[0185] The pixel-level fusion method can effectively improve the contrast and clarity of the image. By weighted averaging and multi-scale fusion of images in each band, the visual effect of the image can be improved, making the target more recognizable in a complex background, which is conducive to subsequent feature extraction and detection.

[0186] In the process of pixel-level fusion, the noise characteristics of each band image can be comprehensively utilized to perform effective noise reduction processing and reduce the impact of artifacts and noise on the detection results. In particular, the weighted average method can be used to dynamically adjust the weight according to the signal-to-noise ratio of each band to optimize the fusion effect.

[0187] The Laplacian pyramid method can be used to fuse images at multiple scales, which not only retains the global information of the image but also enhances the local details. This multi-scale feature fusion method is particularly effective for detecting objects of different scales and helps improve the performance of small object detection.

[0188] S3. Use a pre-trained deep convolutional neural network to extract deep features of the final fused image.

[0189] The pre-trained ResNet50 model is used to extract the deep features of the final fused image. The ResNet50 model consists of several residual blocks, each of which contains a convolutional layer, a normalization layer, an activation layer, and a pooling layer, specifically:

[0190] ;

[0191] in, and ResNet50 model Layer and The output feature map of the layer, , is the number of residual blocks, , For the Layer operations, including convolution, normalization, activation, and pooling;

[0192] go through After residual blocks, the feature map Input into global average pooling to generate deep features .

[0193] Pre-trained deep convolutional neural networks have been trained on large-scale datasets and have learned rich feature representation capabilities. Using these pre-trained networks, high-level features of images, including edges, textures, shapes, etc., can be effectively extracted, significantly improving detection accuracy.

[0194] The multi-layer structure of deep convolutional neural networks can extract low-level features of images to high-level semantic features layer by layer. This layer-by-layer feature extraction method enables the model to better understand and capture complex patterns and details in images, especially the characteristics of small objects.

[0195] Small objects usually occupy a small area in the image and are easily overlooked. The multi-layer feature extraction mechanism of deep convolutional neural networks enables it to capture subtle image details and edge features, enhancing the detection ability of small objects.

[0196] S4. Construct an object detection network with attention mechanism based on deep features and features extracted from preprocessed multispectral images.

[0197] S41. Feature extraction and fusion

[0198] Extract the features of the preprocessed multispectral image, specifically:

[0199] ;

[0200] in, is the convolutional layer of YOLOv5, is the preprocessed multispectral image, is the extracted features of the preprocessed multispectral image;

[0201] For deep features Remapping is performed, specifically:

[0202] ;

[0203] in, is the fully connected layer, is the deep feature after remapping;

[0204] The extracted pre-processed multispectral image features and the remapped depth features are fused as follows:

[0205] ;

[0206] in, is the fused feature;

[0207] S42. Adding attention mechanism

[0208] Use attention mechanism to process fused features , the attention mechanism includes spatial attention and channel attention, specifically:

[0209] ;

[0210] in, is the attention weight map, is a type B function, is a two-dimensional convolution operation, is a linear rectification function, is the global pooling function, and are the trainable parameters included in the attention mechanism;

[0211] Apply the attention weight map to , and obtain the enhanced features , specifically:

[0212] ;

[0213] S43, set the detection head

[0214] Input the enhanced features into the detection head of YOLOv5 to generate prediction results , specifically:

[0215] ;

[0216] in, is the prediction result of the target detection network, , For the predicted The bounding box of the detected object, including the center coordinates of the bounding box and the width and height of the bounding box, For the predicted The class probability of the detected targets, is the number of targets contained in the prediction results of the target detection network, It is the detection head of YOLOv5.

[0217] Multispectral images and deep features provide rich spectral and image information. Combining the two can comprehensively utilize the detailed features of images in each band and the advanced image features extracted by deep convolutional neural networks to improve the overall performance of target detection.

[0218] The attention mechanism can dynamically adjust the weights of features according to the characteristics of the input data, highlight key features, and suppress irrelevant or redundant information. Through channel attention and spatial attention mechanisms, it can enhance the feature expression of the target area and improve the detection accuracy of small targets.

[0219] Small targets in images often have low contrast and small size and are easily ignored. By introducing the attention mechanism, we can significantly improve the sensitivity to small targets, enhance the feature expression of small targets, and reduce missed detections and false detections.

[0220] S5. Set the loss function of the target detection network and use the multispectral images with labeled results for training. Use the trained target detection network to detect small targets in the multispectral images without labeled results.

[0221] S51. Setting loss function

[0222] The loss function consists of bounding box regression loss, category classification loss, and feature enhancement loss with the introduction of the attention mechanism, specifically:

[0223] The bounding box regression loss is formed by using the intersection-over-union loss and the center point loss :

[0224] ;

[0225] in, For the predicted The bounding box coordinates of the detected objects, Marked The bounding box coordinates of the detected objects, and are the center coordinates of the predicted and labeled bounding boxes, respectively. ;

[0226] Use cross entropy loss as the category classification loss :

[0227] ;

[0228] in, Marked The categories of the detected targets are The probability of For the predicted The categories of the detected targets are The probability of is the total number of categories;

[0229] Feature enhancement loss using attention mechanism :

[0230] ;

[0231] in, is the attention weight map for the labeled target;

[0232] The complete loss function is:

[0233] ;

[0234] in, , and is the weight factor. In this embodiment , , .

[0235] S52. Training network

[0236] Use the back-propagation algorithm to calculate the gradient and update the target detection network weights:

[0237] ;

[0238] in, is the target detection network weight, is the learning rate. In this embodiment Take 0.001, for about The partial derivative of

[0239] S53, Application Network

[0240] The multispectral image without annotation results is processed in steps S1-S3, and then the trained target detection network is used to detect small targets.

[0241] Setting a comprehensive loss function, including bounding box regression loss, category classification loss, and feature enhancement loss with the introduction of an attention mechanism, can more comprehensively measure the detection performance. By adaptively adjusting the weights of different loss items and optimizing the overall loss function, it helps to improve the detection accuracy and robustness of the model.

[0242] Introducing attention feature enhancement loss into the loss function can effectively deal with the imbalance problem of positive and negative samples and enhance the detection ability of small targets.

[0243] The introduction of the attention mechanism in the loss function can dynamically adjust feature weights, highlight key features, suppress irrelevant information, and enhance the loss by optimizing features, thereby increasing the model's attention to important features and improving the detection effect and stability of small targets.

[0244] Example 2

[0245] A small target detection system based on multispectral image fusion, comprising:

[0246] Preprocessing module: Use a multispectral camera to collect images of the target area in different bands to obtain a multispectral image, and preprocess the images of each band in the multispectral image;

[0247] Fusion module: fuses the preprocessed multispectral images at the pixel level to obtain the final fused image;

[0248] Deep feature extraction module: uses a pre-trained deep convolutional neural network to extract the deep features of the final fused image;

[0249] Network construction module: Build an object detection network with attention mechanism based on deep features and features extracted from pre-processed multispectral images;

[0250] Network training and application module: Set the loss function of the target detection network and use multispectral images with labeled results for training. Use the trained target detection network to detect small targets in multispectral images without labeled results.

[0251] The small target detection method based on multispectral image fusion in Example 1 is implemented by applying the system in this embodiment.

[0252] It should be noted that the serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages and disadvantages of the embodiments. And the terms "including", "comprising" or any other variants thereof in this article are intended to cover non-exclusive inclusion, so that a process, device, article or method including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, device, article or method. In the absence of further restrictions, an element defined by the sentence "including a ..." does not exclude the presence of other identical elements in the process, device, article or method including the element.

[0253] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0254] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A small target detection method based on multispectral image fusion, characterized in that: The following steps are involved: S1. Use a multispectral camera to collect images of the target area in different bands to obtain a multispectral image , and preprocess each band image in the multispectral image; S2, fusing the preprocessed multispectral images at the pixel level to obtain a final fused image; S21, weighted average The weighting coefficient is calculated based on the contrast and clarity of the preprocessed image of each band, wherein the contrast is calculated by the standard deviation of the preprocessed image of each band. The sharpness is measured by the gradient strength of each band of the preprocessed image. to measure; The weighting coefficients of the image of each band after preprocessing are calculated by combining contrast and clarity, specifically: ; in, For multispectral images The number of bands, the image of each band is recorded as and Indicates the band number, After preprocessing The weighting coefficient of the image of each band; ; S22. Constructing Laplace Pyramid Construct a Laplacian pyramid for each band of the preprocessed image. First, construct a Gaussian pyramid, specifically: ; in, represents the pixel position of the image, For the Bands in the Gaussian pyramid The image of the layer is located at The value at , , For the The band is The layer Gaussian pyramid is at position The value at The Laplacian pyramid is further constructed by the Gaussian pyramid: ; in, For the The bands in the Laplace pyramid Layer images, is the upsampling operation, and Respectively Bands in the Gaussian pyramid Layer and Layer images; S23. Image Fusion Perform weighted average fusion on the Laplacian pyramid layers of each band: ; in, The fused Laplacian pyramid Layer images, For the The bands in the Laplace pyramid Layer images; Reconstruct the final fused image based on the fused Laplacian pyramid , specifically: ; in, , and They are the fused Laplacian pyramid Layer, Layer and Layer images; S3, use the pre-trained deep convolutional neural network to extract the deep features of the final fused image; S4, constructing an object detection network with attention mechanism based on deep features and features extracted from pre-processed multispectral images; S5. Set the loss function of the target detection network and use the multispectral images with labeled results for training. Use the trained target detection network to detect small targets in the multispectral images without labeled results.

2. The small target detection method based on multispectral image fusion according to claim 1 is characterized in that: Step S1 The process includes: For multispectral images The images of each band in are preprocessed, and the preprocessing includes image denoising and image enhancement, specifically: S11. Image denoising The image of each band is denoised using an improved variable step-size median filtering algorithm. The variable step-size median filtering algorithm first estimates the local standard deviation of the image and defines a local window. , represents the pixel position of the image, and the local window size is , is the local window radius, and the local standard deviation of the image is: ; in, For the Bands at location The local standard deviation at for The pixel position within for In Location The value at For the Bands at location The mean value at is: ; according to Dynamically adjust step size and , let the maximum step size be , with a minimum value of 1, and defines the noise level threshold , specifically: ; Use the dynamically adjusted step size to perform median filtering on the image, and the filter window size is , get the image of each band after filtering , the filtering process is: ; in, and is the relative displacement within the filter window, for In Location The value at represents the median function, for In Location The value at S12. Image Enhancement For each local window Calculate the local histogram, specifically: ; in, represents the gray level, For the The band is The gray level in the window is The value of is a pulse function, when hour is 1, otherwise it is 0; Perform contrast limitation on the local histogram and set the contrast limitation threshold to ,but: ; in, is the first The band is The gray level in the window is The value of is the minimum function; The parts exceeding the contrast limit threshold are evenly distributed to other gray levels, specifically: ; in, After the allocation The band is The gray level in the window is The value of , is the maximum image gray level, is the maximum value function; Compute the cumulative distribution function within a local window and normalize it: ; ; in, For the The band is The gray level in the window is The cumulative value of For the The band is The accumulated value of the gray level in the window is 0, After normalization ; Map each pixel value to obtain the preprocessed image of each band : ; in, for In Location The value at Function is used for rounding. For the The band is The gray level in the window is The normalized cumulative value of Preprocessed images of each band Together they form a preprocessed multispectral image .

3. The small target detection method based on multispectral image fusion according to claim 2 is characterized in that: Step S3 The process includes: The pre-trained ResNet50 model is used to extract the deep features of the final fused image. The ResNet50 model consists of several residual blocks, each of which contains a convolutional layer, a normalization layer, an activation layer, and a pooling layer, specifically: ; in, and ResNet50 model Layer and The output feature map of the layer, , is the number of residual blocks, , For the Layer operations, including convolution, normalization, activation, and pooling; go through After residual blocks, the feature map Input into global average pooling to generate deep features .

4. The small target detection method based on multispectral image fusion according to claim 3, characterized in that: The step S4 includes the following process: S41. Feature extraction and fusion Extract the features of the preprocessed multispectral image, specifically: ; in, is the convolutional layer of YOLOv5, is the preprocessed multispectral image, is the extracted features of the preprocessed multispectral image; For deep features Remapping is performed, specifically: ; in, is the fully connected layer, is the deep feature after remapping; The extracted pre-processed multispectral image features and the remapped depth features are fused as follows: ; in, is the fused feature; S42. Adding attention mechanism Use attention mechanism to process fused features , the attention mechanism includes spatial attention and channel attention, specifically: ; in, is the attention weight map, is a type B function, is a two-dimensional convolution operation, is a linear rectification function, is the global pooling function, and are the trainable parameters included in the attention mechanism; Apply the attention weight map to , and obtain the enhanced features , specifically: ; S43, set the detection head Input the enhanced features into the detection head of YOLOv5 to generate prediction results , specifically: ; in, is the prediction result of the target detection network, , For the predicted The bounding box of the detected object, including the center coordinates of the bounding box and the width and height of the bounding box, For the predicted The class probability of the detected targets, is the number of targets contained in the prediction results of the target detection network, It is the detection head of YOLOv5.

5. The small target detection method based on multispectral image fusion according to claim 4, characterized in that: Step S5 The process includes: S51. Setting loss function The loss function consists of bounding box regression loss, category classification loss, and feature enhancement loss with the introduction of the attention mechanism, specifically: The bounding box regression loss is formed by using the intersection-over-union loss and the center point loss : ; in, For the predicted The bounding box coordinates of the detected objects, Marked The bounding box coordinates of the detected objects, and are the center coordinates of the predicted and labeled bounding boxes, respectively. ; Use cross entropy loss as the category classification loss : ; in, Marked The categories of the detected targets are The probability of For the predicted The categories of the detected targets are The probability of is the total number of categories; Feature enhancement loss with attention mechanism : ; in, Attention feature map for annotated targets; The complete loss function is: ; in, , and is the weight factor; S52. Training network Use the back-propagation algorithm to calculate the gradient and update the target detection network weights: ; in, is the target detection network weight, is the learning rate, for about The partial derivative of S53, Application Network The multispectral image without annotation results is processed in steps S1-S3, and then the trained target detection network is used to detect small targets.

6. A small target detection system based on multispectral image fusion, characterized in that: include: Preprocessing module: Use a multispectral camera to collect images of the target area in different bands to obtain a multispectral image, and preprocess the images of each band in the multispectral image; Fusion module: fuses the preprocessed multispectral images at the pixel level to obtain the final fused image; Deep feature extraction module: uses a pre-trained deep convolutional neural network to extract the deep features of the final fused image; Network construction module: Build an object detection network with attention mechanism based on deep features and features extracted from pre-processed multispectral images; Network training and application module: Set the loss function of the target detection network and use multispectral images with labeled results for training. Use the trained target detection network to detect small targets in multispectral images without labeled results. To realize the small target detection method based on multispectral image fusion as described in any one of claims 1 to 5.