Small target detection method based on Mama feature fusion
By using Transformer-Mamba hybrid encoder and small target IoU overlap loss function in small target detection, the shortcomings of the RT-DETR detection algorithm in model parameter quantity and global context information extraction ability are solved, and small target detection is realized Performance improvement.
Patent Information
- Application Number
- CN202411011393.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-26
- Publication Date
- 2024-11-15
AI Technical Summary
The existing RT-DETR detection algorithm is not compatible with small object detection to reduce the amount of model parameters, while maintaining the ability to extract global context information, and the loss function design is not enough to optimize small object detection tasks.
Using a small object detection method based on Mamba feature fusion, the Transformer-Mamba hybrid encoder combines the cross-scale Mamba feature fusion module and the small object IoU overlap loss function to realize the remote dependency modeling and control of model parameter quantity, and improve the Loss function to meet the needs of small object detection.
It improves the accuracy and efficiency of small object detection, significantly improves the ability to extract global context information, and reduces the amount of model parameters, and enhances the detection performance of small objects.
Smart Images

Figure CN118968019A_ABST
Abstract
Description
Technical Field
[0001] The invention belongs to the technical field of image target detection, and in particular relates to a small target detection method based on Mamba feature fusion. Background Art
[0002] Object detection is an important research direction in the field of computer vision, which can classify and locate target objects at the same time. Small target detection is a difficult point in object detection, which aims to accurately detect small target objects (Microsoft COCO dataset defines small targets as targets with a resolution less than 32 pixels × 32 pixels). In real scenes, small targets exist in large numbers, so small targets have broad application prospects and play an important role in many fields such as autonomous driving, intelligent monitoring and security, smart medical care, industrial automation, aerospace, etc.
[0003] At present, there are two main architectures for target detection algorithms: detection algorithms based on convolutional neural networks (CNN) and detection algorithms based on attention mechanisms (Transformer). In recent years, researchers have conducted extensive research on CNN-based target detection algorithms, which has enabled target detection to develop from the initial two-stage to a single-stage, and two detection paradigms have emerged: anchor-based and anchor-free. These studies have made significant progress in both detection speed and accuracy. Although CNN-based methods already have excellent feature representation capabilities, due to the limited receptive field of the convolution kernel, the network will focus more on local features and ignore too many global context features. This local feature induction bias limits the ability of small target features to perceive global context information, which is unfavorable for small target detection algorithms.
[0004] Transformer-based object detection algorithms (DETRs) are one of the most popular detection algorithms. Since they eliminate various handcrafted components, such as non-maximum suppression (NMS), this structure greatly simplifies the detection process and realizes end-to-end object detection. The RT-DETR algorithm in this series of algorithms is one of the representative works. This algorithm is improved on the basis of the DETR architecture and further balances the accuracy and real-time performance of the algorithm. Although the use of Transformer can improve the extraction of global context features to a certain extent and enhance the modeling of long-range dependencies by the network model, its high computational cost has not yet been effectively solved.
[0005] The current RT-DETR (Real-Time Detection Transformer) detection algorithm reduces the multi-layer encoder to one layer, and only uses one Transformer module (AIFI) to model the long-range dependency of high-level semantic features, reducing the number of Transformer modules to achieve the purpose of reducing the number of model parameters. Although this operation increases the detection speed of the RT-DETR algorithm, it reduces its encoder's ability to fuse features and reduce its ability to extract global context information, which in turn affects the detection effect of small objects.
[0006] In addition, most of the current mainstream object detection algorithms focus on optimizing the detection performance of large-scale or medium-scale objects when designing their loss functions. However, when these algorithms are directly applied to small target detection tasks, their effectiveness is often limited. The main reason is that small targets occupy a small pixel area in the image, resulting in a relatively scarce amount of effective feature information that can be extracted, and are easily affected by background noise and neighboring objects. Therefore, the loss function needs to be able to handle the subtle differences between these features more finely.
[0007] Although the loss function of the RT-DETR detection algorithm is designed for general object detection datasets such as Microsoft COCO, it is not optimized enough for small object detection tasks. This is mainly because although the COCO dataset contains objects of various scales, the design of the loss function often focuses more on the overall average performance and fails to fully consider the particularity of small object detection.
[0008] In recent years, state-space models (SSM) have become a hot topic of research. The Mamba model built on SSM exhibits linear complexity in input size, solving the computational efficiency problem of Transformer in the state space of long sequence modeling. Subsequently, in the field of general visual backbone networks, Vision Mamba proposed a pure visual backbone network model based on SSM, marking the first time that Mamba was applied to the visual field. VMamba introduced a cross-scanning module, which enables the model to selectively scan 2D images, enhances visual processing capabilities, and shows superiority in image classification tasks. The Mamba model is being increasingly widely used in various visual tasks.
[0009] The present invention proposes a small target detection method based on Mamba feature fusion, establishes a small target detection model based on Mamba feature fusion, adopts a Transformer-Mamba hybrid encoder based on a cross-scale Mamba feature fusion module, realizes the attention mechanism's ability to model long-range dependencies and effectively controls the amount of model parameters, and proposes a small target IoU overlap loss function. Summary of the invention
[0010] The purpose of the present invention is to provide a small target detection method based on Mamba feature fusion, so as to solve the problems that the RT-DETR detection algorithm in the prior art proposed in the above background technology is not compatible with reducing the amount of model parameters and ensuring the ability to extract global context information, and no loss function is proposed for small target detection.
[0011] To achieve the above object, the present invention adopts the following technical solutions:
[0012] The present invention proposes a small target detection method based on Mamba feature fusion, comprising the following steps:
[0013] S1. Data preprocessing: collecting image data and performing data preprocessing on the image data;
[0014] S2, feature extraction: the backbone network ResNet network is used to extract features from the preprocessed image to obtain feature maps of multiple scales;
[0015] S3, feature fusion: Transformer-Mamba hybrid encoder is used to fuse the shallow semantic feature information and deep semantic feature information of the feature map;
[0016] S4, feature decoding: use stacked multiple decoders to decode the features output by the Transformer-Mamba hybrid encoder to obtain a feature sequence, input the feature sequence into the prediction head for prediction, and output the predicted box coordinates and predicted category;
[0017] S5, loss function correction: the outputs of multiple stacked decoders are collected, and then the gradient of the output head of each decoder is calculated and the parameters are adjusted through the loss function;
[0018] S6. Small target detection: Use the calibrated small target detection model based on Mamba feature fusion to perform small target detection on the image to be detected.
[0019] Preferably, the Transformer-Mamba hybrid encoder in S3 is as follows:
[0020] The Transformer-Mamba hybrid encoder includes an intra-level feature interaction module and a cross-scale Mamba feature fusion module;
[0021] The intra-level feature interaction module adopts a multi-head attention mechanism to process the extracted partial feature graphs to obtain deep semantic feature information;
[0022] The cross-scale Mamba feature fusion module includes several SACMamba modules, and the SACMamba module includes a SACM module and a FFM module;
[0023] The SACM module processes the extracted unprocessed remaining feature maps to obtain enhanced shallow semantic feature information;
[0024] The FFM module performs feature fusion of deep semantic feature information and enhanced shallow semantic feature information.
[0025] Preferably, the SACM module processes the input feature map information through multiple series-connected dilated convolution branches, and the expansion rate is continuously increasing, and the processed information of the multiple branches is spliced, and then the information is fused and the channel is compressed through convolution;
[0026] The FFM module, which uses the SS2D module internally, can be used to capture global context information.
[0027] Preferably, the specific process in S3 is as follows:
[0028] The Transformer-Mamba hybrid encoder processes the feature maps S3, S4, and S5 of multiple scales output from the backbone network ResNet network; first, S5 is input into the intra-level feature interaction module to extract deep semantic information to obtain the feature map F5. The process is described as follows:
[0029] Q=K=V=Flatten(S5)
[0030] F5=Reshape(Attn(Q,K,V))
[0031] Output = CSMFM ({S3, S4, F5})
[0032] Among them, Attn represents multi-head attention, Reshape means reshaping the shape of the data into the same shape as S5, which is the inverse of the operation of Flatten;
[0033] Then S3, S4, and F5 are input into the cross-scale Mamba feature fusion module for further processing. The process is described as follows:
[0034] Fl '=SACM(F l )
[0035] F lh =FFM(F l ',F h )
[0036] Among them, F l represents the shallow semantic feature map, F h represents the deep semantic feature map, F l ' represents the enhanced shallow semantic feature map, F lh Represents the output result after feature fusion.
[0037] Preferably, the decoder in S4 includes a multi-head attention module and a deformable attention module.
[0038] Preferably, the specific process in S4 is as follows:
[0039] S3"=Flatten(S3')
[0040] S4”=Flatten(S4')
[0041] S5"=Flatten(S5')
[0042] SS=QuerySelection(Concat(S3”,S4”,S5”))
[0043] SS”=MHSA(SS',SS',SS)
[0044] SS”'=DeformableAttention(SS”)
[0045]
[0046] Among them, S3', S4', S5' represent feature maps, S3", S4", S5" represent feature maps after shape expansion; QuerySelection represents the IoU-aware query module; MHSA and DeformableAttention represent the multi-head attention module and the deformable attention module; SS' represents the feature sequence after adding the position encoding; Boxhead and Classhead represent the coordinate prediction head and the category prediction head.
[0047] Preferably, the loss function in S5 is as follows:
[0048] The classification loss of the predicted category adopts the VFL loss of the RT-DETR framework, and the positioning loss of the predicted box coordinates adopts the STIoU loss function;
[0049] The STIoU loss function is expressed as:
[0050]
[0051] Among them, ω represents the penalty coefficient, γ, λ, β represent custom parameters; Δ represents the sum of the bounding box center distance loss and the length and width loss, Represents the center coordinates of the prediction box, Indicates the center coordinates of the real box, w prd 、h prd Represents the length and width of the prediction box, w gt 、h gt Indicates the length and width of the real box.
[0052] Preferably, the backbone network ResNet network in S2 is formed by stacking multiple residual blocks.
[0053] Preferably, the specific process in S1 is as follows:
[0054] First, the image data is normalized, and then the image data is randomly transformed to achieve data enhancement. The random transformation includes random color perturbation, random expansion, random cropping and random horizontal flipping; finally, the image data is adjusted to a uniform size.
[0055] Preferably, the small target detection model based on Mamba feature fusion in S6 includes a backbone network ResNet network, a Transformer-Mamba hybrid encoder, and a decoding module composed of multiple stacked decoders.
[0056] Compared with the prior art, the present invention has the following beneficial effects:
[0057] (1) The Transformer-Mamba hybrid encoder based on the cross-scale Mamba feature fusion module (CSMFM) and the small target IoU overlap loss function adopted in the present invention are the key technologies of the present invention. The Transformer-Mamba hybrid encoder based on the cross-scale Mamba feature fusion module is composed of an AIFI module and multiple CSMFM modules. The AIFI is implemented by a multi-head attention mechanism. The CSMFM module is based on the architecture of the feature pyramid and can fuse multi-scale features. The SACMamba module in the CSMFM module is responsible for fusing shallow and deep semantic feature information. The SACMamba module is composed of a SACM module and an FFM module. The SACM module enhances shallow features by connecting hollow convolutions in series. FFM relies on the selective scanning mechanism of Mamba to fuse deep features with enhanced shallow features, which is comparable to the attention mechanism's ability to model long-range dependencies while greatly reducing the number of model parameters. Finally, the result after the fusion of features of multiple scales is output.
[0058] (2) The small target IoU overlap loss function (STIoU) of the present invention takes into account that the current GIoU loss function is sensitive to subtle changes in features and IoU values in small target detection, which affects the small target detection effect. In order to adapt to small target detection, STIoU improves GIoU and uses a penalty term that can perceive IoU value changes to scale the overall loss value. At the same time, the bounding box center distance loss and length and width loss are added to increase the diversity and generalization of the loss function. BRIEF DESCRIPTION OF THE DRAWINGS
[0059] Figure 1 It is a structural block diagram of the small target detection model based on Mamba feature fusion in the present invention;
[0060] Figure 2 It is a schematic diagram of the structure of the SACMamba module in the present invention;
[0061] Figure 3 It is a structural schematic diagram of the SACM module in the present invention;
[0062] Figure 4 It is a structural schematic diagram of the FFM module in the present invention. DETAILED DESCRIPTION
[0063] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0064] Embodiment 1:
[0065] The small target detection method based on Mamba feature fusion is as follows:
[0066] Step 1: data preprocessing.
[0067] In order to enable the network to better utilize the input image information, the image data is preprocessed.
[0068] First, normalization is performed to scale the data range of the input image to a standard scale (such as 0 to 1 or -1 to 1), which helps accelerate the convergence process of the model. Secondly, the training data is randomly transformed, and several classic data enhancement methods are randomly selected, including random color perturbation (RandomDistort), random expansion (RandomExpand), random cropping (RandomCrop), and random horizontal flip (RandomHFlip), which can increase the robustness of the network model under different conditions. Finally, the training data is resized to a uniform size. This operation is not only to adapt to the model input requirements, but also to reduce the amount of calculation and enhance generalization capabilities.
[0069] Step 2: feature extraction.
[0070] In this embodiment, a backbone network ResNet18 network composed of multiple residual blocks (Residual Block) is used as the backbone network; it is used to extract features and generate feature maps (S3, S4, S5) of multiple scales. These feature maps represent the information of the image at different levels.
[0071] This step extracts features from the input image information and obtains three feature maps of different sizes.
[0072] C is the number of channels, C1 is 128, C2 is 256, and C3 is 512.
[0073] Step three: feature fusion.
[0074] In this embodiment, a Transformer-Mamba hybrid encoder is used, which mainly includes two modules: intra-level feature interaction (AIFI) and cross-scale Mamba feature fusion module (CSMFM). Multiple scale feature maps extracted by the backbone network are further fused to extract deeper semantic information. Multi-scale features are converted into image feature sequences.
[0075] The encoder processes the multi-layer feature maps S3, S4, and S5 output from the backbone network. The AIFI module (multi-head attention mechanism) is used to process S5 to obtain a high-level semantic feature map F5, and then S3, S4, and F5 are input into the Cross-Scale Mamba Feature Fusion Module (CSMFM) for further processing.
[0076] The main body of the CSMFM module is the structure of the feature pyramid, which contains multiple SACMamba modules (Series Atrous Convolutional Mamba), which is responsible for fusing shallow and deep semantic feature information. After receiving the shallow feature information (low-level) and deep feature information (high-level), the SACMamba module inputs the shallow features into the SACM (Series Atrous Convolutional Module) module for processing to obtain enhanced shallow features, and then inputs the deep features and enhanced shallow features into the FFM (Fusion Feature Mamba Module) module for feature fusion, such as Figure 2 The figure shows the structure diagram of the SACMamba module.
[0077] The SACM module processes the input feature map information through multiple series-connected dilated convolution branches with increasing expansion rate, concatenates the processed information of multiple branches, and finally performs information fusion and channel compression through convolution. The number of series-connected dilated convolution branches in the SACM module increases with the increase of feature map size. When the feature map is F5 size ( Where H is the length of the input image after resizing) is 1, when the feature map is of size S4 When the feature map is of size S3, When , the dilation rate of the dilated convolution branch is 3, and the dilation rate of the dilated convolution branch is selected from (1, 3, 5, 7). The reason for this is that in the shallow part of the network, the feature map with a larger size can often extract more global context information. The structure diagram of the SACM module is shown in the figure. Figure 3 shown.
[0078] FFM fuses the shallow feature information obtained with the deep feature information, and mainly uses the SS2D module (2D-Selective-Scan for Vision Data) to capture global context information. The structure diagram of FFM is shown in the figure. Figure 4 shown.
[0079] Before feature fusion, this step needs to adjust the number of channels of multiple feature maps of different scales passed from the backbone network to a unified 256. This is done to facilitate the calculation of the number of channels in the subsequent feature fusion module. The adjusted multi-scale feature maps are input into the Transformer-Mamba hybrid encoder for feature fusion operation.
[0080] Specifically, S5 is input into the AIFI module (multi-head attention mechanism) to extract high-level semantic information to obtain feature map F5. Then S3, S4, and F5 are input into the cross-scale Mamba feature fusion module (CSMFM) for further processing. This process can be described as follows:
[0081] Q=K=V=Flatten(S5)
[0082] F5=Reshape(Attn(Q,K,V))
[0083] Output = CSMFM ({S3, S4, F5})
[0084] Among them, Attn represents multi-head self-attention, and Reshape means reshaping the shape of the data into the same shape as S5, which is the inverse of the Flatten operation.
[0085] The CSMFM module contains multiple SACMamba modules, which are responsible for fusing shallow and deep semantic feature information. After receiving the shallow feature information and deep feature information, the SACMamba module inputs the shallow features into the SACM module for processing to obtain enhanced shallow features, and then inputs the deep features and enhanced shallow features into the FFM module for feature fusion. This process can be described as follows:
[0086] F l '=SACM(F l )
[0087] F lh =FFM(F l ',F h )
[0088] Among them, F l represents the shallow semantic feature map, F h represents the deep semantic feature map, F l ' represents the enhanced shallow semantic feature map, F lh Represents the output result after feature fusion.
[0089] The feature pyramid structure of the CSMFM module passes the multi-scale feature maps S3', S4', and S5' after feature fusion to the decoder for decoding.
[0090] Step 4: Feature decoding.
[0091] In this embodiment, the decoding stage is composed of a plurality of decoders stacked together. The decoder uses a multi-head self-attention module and a deformable attention module to further decode the features output by the encoder to obtain a feature sequence. Finally, the obtained feature sequence is input into the prediction head for prediction processing, and the predicted box coordinates and predicted categories are output.
[0092] This step needs to reshape the multi-scale features into (N, L, C) before decoding, where N represents the batch size, L represents the sequence length, and C represents the number of channels. The multi-scale features are concatenated according to the sequence dimension to obtain (N, L1, C), where L1 represents the sum of the lengths of the multi-scale feature sequences. The feature sequence is then filtered to retain only the num_query sequences with the highest quality (num_query defaults to 300). The decoder uses a multi-head self-attention module and a deformable attention module to decode the concatenated feature sequence to obtain a feature sequence, and finally inputs the obtained feature sequence into the prediction head for prediction processing, outputting the predicted box coordinates and predicted categories. The process can be defined as:
[0093] S3"=Flatten(S3')
[0094] S4”=Flatten(S4')
[0095] S5"=Flatten(S5')
[0096] SS=QuerySelection(Concat(S3”,S4”,S5”))
[0097] SS”=MHSA(SS',SS',SS)
[0098] SS”'=DeformableAttention(SS”)
[0099]
[0100] Among them, S3', S4', S5' represent feature maps, S3", S4", S5" represent feature maps after shape expansion. QuerySelection represents the IoU-aware query module (filtering the feature sequence). MHSA and DeformableAttention represent the multi-head attention module and the deformable attention module. SS' represents the feature sequence after adding the position encoding. Boxhead and Classhead represent the coordinate prediction head and the category prediction head, which are specifically implemented by multi-layer fully connected layers MLP. Finally, the output is assigned one-to-one labels through Hungarian matching to obtain the final prediction result.
[0101] Step 5: Loss function correction.
[0102] When training the network model, the outputs of multiple stacked decoders are collected, and then the gradient of each decoder output head is calculated and the parameters are adjusted through the loss function. The classification loss uses the VFL loss of the RT-DETR framework, and the positioning loss uses the Small Target IoU Loss to calculate the loss, and start calculating the gradient, and start learning and adjusting the parameters of the positioning detection head of the network model.
[0103] The small target IoU overlap loss function can handle small differences between features more finely, reduce the sensitivity of small targets to changes in IoU values, and accelerate the convergence of the model in small target detection. This loss function is obtained by improving the GIoU loss function. When optimizing the positioning of small targets, a small offset of the bounding box of the small target by the GIoU function may have a significant impact on its positioning accuracy. The GIoU loss function can be expressed as:
[0104]
[0105] L GIoU =1-GIoU
[0106] Among them, A and B are the two bounding boxes that need to be compared, and C is the minimum closure area that can contain both A and B. |C\(AUB)| represents the area left after removing the union area of A and B from C. From the formula, we can see that the final L GIoU It is quite sensitive to changes in IoU values, especially in small object detection. A small deviation of the bounding box may lead to overcorrection of the bounding box.
[0107] Therefore, based on the above loss function, a new improvement scheme is proposed: STIoU (Small Target IoU Loss). The STIoU loss function can be expressed as:
[0108]
[0109] Among them, ω represents the penalty coefficient. As the network is continuously optimized, the IoU value of the bounding box tends to 1. In order to make the main part of the loss function decay, but also to control it not to decay too fast (if the loss value is too small, the network will converge), the penalty coefficient ω is defined to control the decay of the loss value. γ represents a custom parameter, the default is 2.0. λ represents a custom parameter, the default is 2, which controls L STIOU The weight of the second half of the loss. β represents a custom parameter, which defaults to 1.0. Δ represents the sum of the bounding box center distance loss and the length and width loss. Represents the center coordinates of the prediction box, Indicates the center coordinates of the real box, w prd 、h prd Represents the length and width of the prediction box, w gt 、h gt Indicates the length and width of the real box.
[0110] The present invention adopts the above method to train a small target detection model based on Mamba feature fusion, and the model includes a backbone network ResNet network, a Transformer-Mamba hybrid encoder, and a decoding module composed of multiple decoders stacked; Figure 1 shown.
[0111] The entire process of the small target detection model is divided into three stages: feature extraction, feature fusion, feature decoding and prediction. In the feature extraction stage, the image to be detected is input into the backbone network ResNet network to obtain feature maps of multiple scales. In the feature fusion stage, the feature maps obtained from the backbone network are fused through the CNN-Mamba hybrid encoder. In the feature decoding stage, the feature sequence after feature fusion is input into the decoder Decoder for decoding, and finally the decoded information is input into the detection head for prediction processing to generate the final output.
[0112] Experimental verification:
[0113] Based on the content of the above invention, an evaluation of the technical effect is proposed:
[0114] Performance evaluation: The present invention is experimented and evaluated on the VisDrone dataset. The VisDrone dataset contains 8,599 high-resolution images, including 6471 images in the training set, 548 images in the validation set, and 1580 images in the test set. The dataset covers ten object categories: pedestrians, people, bicycles, cars, vans, trucks, tricycles, awning tricycles, buses, and motors. The present invention uses 6471 images in the training set and 1580 images in the test set for training and testing, respectively.
[0115] The invention is compared with the benchmark RT-DETR algorithm. Table 1 shows the detection results on the VisDrone test set.
[0116] Table 1 Comparative experimental results on the VisDrone dataset
[0117]
[0118] Compared with the benchmark RT-DETR algorithm, the accuracy of the method of the present invention is significantly improved under the condition of small increase in parameter quantity and calculation amount, and the mAP is improved by 0.63%. 50 The above description is only used to help understand the method of the present invention and its core essence, but the protection scope of the present invention is not limited thereto. For those skilled in the art in this technical field, according to the technical solution and inventive concept of the present invention, equivalent replacement or change shall be included in the protection scope of the present invention within the technical scope disclosed by the present invention. In summary, the content of this specification should not be understood as limiting the present invention.
Claims
1. A small target detection method based on Mamba feature fusion, characterized in that: The steps include: S1. Data preprocessing: collecting image data and performing data preprocessing on the image data; S2, feature extraction: the backbone network ResNet network is used to extract features from the preprocessed image to obtain feature maps of multiple scales; S3, feature fusion: Transformer-Mamba hybrid encoder is used to fuse the shallow semantic feature information and deep semantic feature information of the feature map; S4, feature decoding: use stacked multiple decoders to decode the features output by the Transformer-Mamba hybrid encoder to obtain a feature sequence, input the feature sequence into the prediction head for prediction, and output the predicted box coordinates and predicted category; S5, loss function correction: the outputs of multiple stacked decoders are collected, and then the gradient of the output head of each decoder is calculated and the parameters are adjusted through the loss function; S6. Small target detection: Use the calibrated small target detection model based on Mamba feature fusion to perform small target detection on the image to be detected.
2. According to claim 1, a small target detection method based on Mamba feature fusion is characterized in that: The Transformer-Mamba hybrid encoder in S3 is as follows: The Transformer-Mamba hybrid encoder includes an intra-level feature interaction module and a cross-scale Mamba feature fusion module; The intra-level feature interaction module adopts a multi-head attention mechanism to process the extracted partial feature graphs to obtain deep semantic feature information; The cross-scale Mamba feature fusion module includes several SACMamba modules, and the SACMamba module includes a SACM module and a FFM module; The SACM module processes the extracted unprocessed remaining feature maps to obtain enhanced shallow semantic feature information; The FFM module performs feature fusion of deep semantic feature information and enhanced shallow semantic feature information.
3. A small target detection model based on Mamba feature fusion according to claim 2, characterized in that: The SACM module processes the input feature map information through multiple series-connected dilated convolution branches with increasing expansion rate, concatenates the processed information of multiple branches, and then performs information fusion and channel compression through convolution; The FFM module, which uses the SS2D module internally, can be used to capture global context information.
4. A small target detection method based on Mamba feature fusion according to claim 2 or 3, characterized in that: The specific process in S3 is as follows: The Transformer-Mamba hybrid encoder processes the feature maps S3, S4, and S5 of multiple scales output from the backbone network ResNet network; first, S5 is input into the intra-level feature interaction module to extract deep semantic information to obtain the feature map F5. The process is described as follows: Q=K=V=Flatten(S5) F5=Reshape(Attn(Q,K,V)) Output = CSMFM ({S3, S4, F5}) Among them, Attn represents multi-head attention, Reshape means reshaping the shape of the data into the same shape as S5, which is the inverse of the operation of Flatten; Then S3, S4, and F5 are input into the cross-scale Mamba feature fusion module for further processing. The process is described as follows: F l '=SACM(F l ) F lh =FFM(F l ',F h ) Among them, F l represents the shallow semantic feature map, F h represents the deep semantic feature map, F l ' represents the enhanced shallow semantic feature map, F lh Represents the output result after feature fusion.
5. The small target detection method based on Mamba feature fusion according to claim 1 is characterized in that: The decoder in S4 includes a multi-head attention module and a deformable attention module.
6. A small target detection method based on Mamba feature fusion according to claim 5, characterized in that: The specific process in S4 is as follows: S3"=Flatten(S3') S4”=Flatten(S4') S5"=Flatten(S5') SS=QuerySelection(Concat(S3”,S4”,S5”)) SS”=MHSA(SS',SS',SS) SS”'=DeformableAttention(SS”) Among them, S3', S4', S5' represent feature maps, S3", S4", S5" represent feature maps after shape expansion; QuerySelection represents the IoU-aware query module; MHSA and DeformableAttention represent the multi-head attention module and the deformable attention module; SS' represents the feature sequence after adding the position encoding; Boxhead and Classhead represent the coordinate prediction head and the category prediction head.
7. The small target detection method based on Mamba feature fusion according to claim 1 is characterized in that: The loss function in S5 is as follows: The classification loss of the predicted category adopts the VFL loss of the RT-DETR framework, and the positioning loss of the predicted box coordinates adopts the STIoU loss function; The STIoU loss function is expressed as: Among them, ω represents the penalty coefficient, γ, λ, β represent custom parameters; Δ represents the sum of the bounding box center distance loss and the length and width loss, Represents the center coordinates of the prediction box, Indicates the center coordinates of the real box, w prd 、h prd Represents the length and width of the prediction box, w gt 、h gt Indicates the length and width of the real box.
8. The small target detection method based on Mamba feature fusion according to claim 1 is characterized in that: The backbone network ResNet in S2 is composed of a plurality of stacked residual blocks.
9. The small target detection method based on Mamba feature fusion according to claim 1 is characterized in that: The specific process in S1 is as follows: First, the image data is normalized, and then the image data is randomly transformed to achieve data enhancement. The random transformation includes random color perturbation, random expansion, random cropping and random horizontal flipping; finally, the image data is adjusted to a uniform size.
10. The small target detection method based on Mamba feature fusion according to claim 1, characterized in that: The small target detection model based on Mamba feature fusion in S6 includes a backbone network ResNet network, a Transformer-Mamba hybrid encoder, and a decoding module composed of multiple stacked decoders.
Citation Information
Cited By
Target vehicle rotating frame detection and identification method and system
CN120014224A
Label-free cell activity detection method and device, computer equipment and storage medium
CN120014633A
Remote sensing image road segmentation method combining road detail feature extraction and context information perception
CN120107575A
A remote sensing image road segmentation method combining road detail feature extraction and context information perception
CN120107575B
General salient target detection method based on Mama
CN120635490A
