A method and device for fine-grained target detection in SAR images with instance feature enhancement

Abstract

The present invention belongs to the field of artificial intelligence and target detection technology, and in particular relates to a method and device for fine-grained target detection in SAR images with instance feature enhancement. The method comprises the following steps: target detection model construction: selecting the YOLOv8 network as the basic network for fine-grained target detection in SAR images, constructing a fine-grained feature enhancement module based on dynamic convolution to replace the feature fusion module in the basic network and generate a target detection model; the feature fusion module is used to dynamically learn spatial input-related convolution kernels from multi-scale features; training sample construction: annotating a preset SAR image and balancing the number of instances using a category-balanced copy-and-paste data enhancement strategy; detection model training: using the training samples to train the target detection model; target detection: using the trained target detection model to perform fine-grained target detection in SAR images.

Description

Technical Field

[0001] The present invention belongs to the technical field of artificial intelligence and target detection, and in particular relates to a method and device for fine-grained target detection in SAR images with instance feature enhancement. Background Art

[0002] Synthetic Aperture Radar (SAR) is an active Earth observation system that enables all-day, all-weather Earth observation. It offers unique advantages in disaster monitoring, environmental monitoring, oceanographic monitoring, surveying and mapping, and military applications. Target detection, which involves determining the coordinates of all targets in an image and identifying their specific categories, is a fundamental task in SAR image processing. In recent years, with the rapid development of artificial intelligence (AI), deep learning-based target detection in SAR images has garnered widespread attention.

[0003] Unlike general SAR image target detection tasks, fine-grained target detection in SAR images requires models to accurately identify targets of similar but different classes within SAR images. Currently, most mainstream SAR image target detection algorithms focus primarily on accurately localizing targets and fail to effectively address the fine-grained classification and recognition required for detection, resulting in accurate localization but misclassification. A major challenge in fine-grained aircraft detection in SAR images lies in the small inter-class differences and large intra-class differences in complex scenes. Due to the unique imaging method, SAR images lack target texture information and are susceptible to noise and redundant background interference, resulting in small inter-class differences and large intra-class differences. The small inter-class differences arise from the similar scattering mechanisms of certain regions within different classes, while the large intra-class differences arise from the different scattering power distributions of different instances within the same class due to varying orientations, scales, and appearances. Another challenge in fine-grained target detection in SAR images is class imbalance, particularly between sub-classes. Conventional object detection model training methods typically use class-balanced datasets. Directly adopting these training methods can lead to suboptimal model training and significant performance degradation.

[0004] Therefore, to achieve accurate fine-grained target detection in SAR images, it is necessary to propose a fine-grained feature enhancement method to improve the fine-grained feature extraction capability of the target detection model, as well as a set of instance-level data enhancement strategies to balance the optimization of the target detection model. This has also become a key issue in current research. Summary of the Invention

[0005] In order to solve the technical problems existing in the above-mentioned prior art, the purpose of the present invention is to provide a method and device for fine-grained target detection in SAR images with instance feature enhancement, which can utilize instance-level data enhancement and fine-grained feature enhancement to improve the fine-grained feature extraction capability of the target detection model and achieve better fine-grained target detection performance.

[0006] The technical solutions for implementing the present invention are as follows:

[0007] In a first aspect, an embodiment of the present application provides a method for fine-grained target detection in SAR images with instance feature enhancement, comprising the following steps:

[0008] Target detection model construction: The YOLOv8 network is selected as the basic network for fine-grained target detection in SAR images. A fine-grained feature enhancement module based on dynamic convolution is constructed to replace the feature fusion module in the basic network to generate a target detection model. The feature fusion module is used to dynamically learn spatial input-related convolution kernels from multi-scale features.

[0009] Training sample construction: The preset SAR images are annotated and the number of instances is balanced using the category-balanced copy-and-paste data augmentation strategy;

[0010] Detection model training: using the training samples to train the target detection model;

[0011] Target detection: Use the trained target detection model to perform fine-grained target detection in SAR images.

[0012] Furthermore, the fine-grained feature enhancement module described in the present invention extracts static fine-grained features from high-level features through static convolution, and learns input-related convolution kernels from static fine-grained features and low-level feature maps through dynamic convolution, and decomposes them into channel-related convolution kernels and space-related convolution kernels, so as to reduce the amount of dynamic convolution parameters while extracting target dynamic fine-grained features in low-level feature maps, and finally fuses the static fine-grained features with the dynamic fine-grained features to output the final fine-grained features.

[0013] Furthermore, the input of the fine-grained feature enhancement module of the present invention is the low-level feature map Y and the high-level feature map X, and the output is the fine-grained feature map The specific processing process is as follows:

[0014] First, static convolution is used to process the low-level feature map Y to generate static fine-grained features Y with enhanced spatial position. static ; Secondly, Y static Splice with X to obtain the cross-scale context feature Z; again, generate the spatial correlation convolution kernel K for the context feature Z spatial and channel-dependent convolution kernel K channel; Then, the spatial correlation convolution kernel K spatial and channel-dependent convolution kernel K channel , convolution is performed on the low-level feature map Y to obtain dynamic fine-grained features Y dynamic ; Finally, the static fine-grained feature Y static and dynamic fine-grained features Y dynamic Connect and fuse along the channel dimension to obtain fine-grained enhanced features

[0015] Furthermore, the algorithm of the fine-grained feature enhancement module of the present invention is:

[0016]

[0017] Z=Concat(Y static , X)

[0018]

[0019] in, It is a convolution layer with a convolution kernel size of k*k, the number of input channels is C, and the number of output channels is C. Concat is a concatenation operation, Avg is an average pooling operation, ReLU is a relu activation function, and F is a convolution operator. Represents broadcasted element-wise addition.

[0020] Furthermore, the number of balanced instances of the category-balanced copy-paste data augmentation strategy described in the present invention is:

[0021] For category i, randomly copy an object from the entire category i and with probability prob i Paste this onto the current image:

[0022]

[0023] Among them, f i represents the frequency of category i targets in the entire training dataset, and t is a hyperparameter.

[0024] Furthermore, when training the detection model, the present invention designs a reasonable positive and negative sample allocation strategy, fine-grained classification and positioning loss function, adopts a stochastic gradient descent optimization algorithm, sets a learning rate decay strategy, and updates the parameters of the SAR image fine-grained target detection network until the network performance converges, thereby obtaining a SAR image fine-grained target detection model with instance feature enhancement.

[0025] Furthermore, the present invention further includes the following steps:

[0026] Construct a fine-grained target classification system and sample annotation specifications based on the preset SAR remote sensing image data and preset expert experience knowledge;

[0027] Based on the fine-grained target classification system and sample annotation specifications, the target location and category in the image are annotated with horizontal boxes;

[0028] The labeled images are sliced ​​to form a fine-grained target detection dataset, which is divided into training set, validation set and test set.

[0029] In a second aspect, an embodiment of the present application provides a SAR image fine-grained target detection device with enhanced instance features, including a target detection model. The model is implemented by using a YOLOv8 network as a base network and replacing a feature fusion module in the base network with a fine-grained feature enhancement module. The algorithm of the fine-grained feature enhancement module is:

[0030]

[0031] Z=Concat(Y static ,X)

[0032]

[0033] Among them, Y is the low-level feature map, X is the high-level feature map, is a fine-grained feature map, It is a convolution layer with a convolution kernel size of k*k, the number of input channels is C, and the number of output channels is C. Concat is a concatenation operation, Avg is an average pooling operation, ReLU is a relu activation function, and F is a convolution operator. Represents broadcasted element-wise addition.

[0034] Compared with the prior art, the present invention has the following beneficial effects:

[0035] First, the present invention adopts a fine-grained feature enhancement method. Through dynamic convolution, when fusing features of adjacent levels, it learns input-related convolution kernels, captures the local information of the target scattered between different feature scale levels from the two dimensions of space and channel, and highlights the fine-grained features of the target.

[0036] Second, the present invention improves the existing image-level category-balanced data enhancement method through instance-level sample enhancement, balances the optimization of the fine-grained target detection network, contributes to better fine-grained feature expression, and comprehensively improves the fine-grained target detection capability of SAR images. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] To more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be derived from these drawings without inventive effort.

[0038] Figure 1 A schematic diagram schematically illustrates a fine-grained target detection network for SAR images based on feature enhancement according to one embodiment of the present invention;

[0039] Figure 2 Schematically showing a fine-grained feature enhancement module based on dynamic convolution according to one embodiment of the present invention;

[0040] Figure 3 Schematically showing a schematic diagram of category-balanced copy-paste data augmentation according to one embodiment of the present invention; DETAILED DESCRIPTION

[0041] The embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0042] It should be noted that, in the absence of conflict, the following embodiments and features in the embodiments may be combined with each other; and, based on the embodiments in this disclosure, all other embodiments obtained by persons of ordinary skill in the art without creative work are within the scope of protection of this disclosure.

[0043] It should be noted that various aspects of the embodiments within the scope of the appended claims are described below. It should be apparent that the aspects described herein can be embodied in a wide variety of forms, and any specific structure and / or function described herein is merely illustrative. Based on this disclosure, it should be understood by those skilled in the art that an aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number of aspects described herein can be used to implement an apparatus and / or practice a method. In addition, other structures and / or functionalities other than one or more of the aspects described herein can be used to implement this apparatus and / or practice this method.

[0044] The present invention provides a method for fine-grained target detection in SAR images with enhanced instance features, comprising the following steps:

[0045] Target detection model construction: The YOLOv8 network is selected as the basic network for fine-grained target detection in SAR images. A fine-grained feature enhancement module based on dynamic convolution is constructed to replace the feature fusion module in the basic network to generate a target detection model. The feature fusion module is used to dynamically learn spatial input-related convolution kernels from multi-scale features.

[0046] Training sample construction: The preset SAR images are annotated and the number of instances is balanced using the category-balanced copy-and-paste data augmentation strategy;

[0047] Detection model training: using the training samples to train the target detection model;

[0048] Target detection: Use the trained target detection model to perform fine-grained target detection in SAR images.

[0049] like Figures 1 to 3 As shown, a method for fine-grained target detection in SAR images with enhanced example features of an embodiment of the present application includes the following steps S1-S6. Steps S1-S6 are not the only execution order of the present invention, but are only one of many execution orders of the embodiment:

[0050] Step S1: Annotate the preset SAR image to obtain a SAR image fine-grained target detection dataset. The specific process of this step is as follows:

[0051] The preset SAR remote sensing image data and the preset expert experience knowledge are used to construct a fine-grained target classification system and sample annotation specifications; based on the fine-grained target classification system and sample annotation specifications, the target positions and categories in the image are horizontally annotated; the annotated image is sliced, the slice size is set to 640*640, and the overlap coefficient is set to 20% to form a fine-grained target detection dataset, which is divided into training set, validation set and test set in a ratio of 7:1:2.

[0052] Step S2: Constructing a basic network for fine-grained target detection in SAR images

[0053] like Figure 1 As shown in the figure, yolov8 is used as the basic network for fine-grained target detection in SAR images. Yolov8 consists of three parts: backbone network, feature pyramid network and detection network.

[0054] The backbone network is mainly used to extract multi-scale feature maps of the input image. It consists of Conv, C2f and SPPF modules. Conv is a convolutional layer. C2f contains two convolutional layers and multiple Bottleneck blocks. The Bottleneck block is a standard bottleneck structure used to improve model performance. The SPPF module is a spatial pyramid pooling module that generates multiple fixed-length feature vectors by performing pooling operations on the input feature maps at different scales, thereby capturing contextual information at different scales.

[0055] The feature pyramid network fuses high-level feature maps with rich semantic information and low-level feature maps with positioning information through C2f, upsample and Concat operations, thereby increasing the multi-scale expression capability of the model.

[0056] The detection module is responsible for obtaining the target fine-grained classification and positioning results of the pixel points on each feature map on the three-level feature map.

[0057] Step S3: Construct a fine-grained feature enhancement module based on dynamic convolution and replace the feature fusion module in the basic network

[0058] The basic network for fine-grained target detection in SAR images uses a backbone network to extract multi-scale feature maps, and fine-grained local features with discriminative capabilities are scattered in the multi-scale features. However, the traditional feature pyramid network uses conventional convolution operators to directly fuse high- and low-level feature maps. The feature extraction and fusion methods are single, and the network's fine-grained feature expression ability is insufficient, making it difficult to effectively distinguish fine-grained targets. Figure 1 As shown in Figure 5, this step constructs the FGFA fine-grained feature enhancement module to replace the Concat splicing submodule in the feature pyramid network to enhance the network's fine-grained feature extraction and expression capabilities.

[0059] In order to alleviate the single feature extraction capability of traditional convolution operators, such as Figure 2 As shown in the figure, this step constructs a fine-grained feature enhancement module based on dynamic convolution. Through dynamic convolution, a convolution kernel related to the input is generated from a high-level feature map with rich semantic information to dynamically capture the complex local semantic information of the target on a low-level feature map with positioning information, and then fuse to generate a fine-grained feature map with more detailed discrimination capabilities. Assume that X represents a high-level feature map and Y represents a low-level feature map. Representing the output fine-grained feature map, the process of the fine-grained feature enhancement module based on dynamic convolution can be expressed as follows:

[0060]

[0061] Z=Concat(Y static ,X)

[0062]

[0063] in, It is a convolution layer with a convolution kernel size of k*k, the number of input channels is C, and the number of output channels is C. Concat is a concatenation operation, Avg is an average pooling operation, ReLU is a relu activation function, and F is a convolution operator. Represents broadcasted element-wise addition.

[0064] Specifically, the FAFG module uses static convolution to process low-level feature maps and generate static feature maps Y with enhanced spatial position. static Subsequently, FAFG will static Splicing with X to achieve simple cross-scale interaction, we get cross-scale context features Z, which reflects the context information between local neighborhoods. These context features Z, through input-related dynamic convolution, capture fine-grained features to exploit the relationship between the two levels. The generation of dynamic convolution kernels is decomposed into the generation of spatial-related convolution kernels and channel-related convolution kernels to minimize parameters and computational costs. Here, K spatial and K channel Represent the generated spatial correlation convolution kernel and channel correlation convolution kernel respectively. Then, the overall dynamic convolution kernel is convolved on the low-level feature map Y to capture the dynamic local semantic information of the target with spatial variation and obtain dynamic fine-grained features. Finally, the static fine-grained features Y are static and dynamic fine-grained features Y dynamc Connect along the channel dimension and obtain enhanced features through a simple fusion layer

[0065] The fine-grained enhancement module based on dynamic convolution can enhance the model's fine-grained feature extraction and expression capabilities in the feature pyramid, allowing the model to extract fine-grained local features of the target under the complex background of SAR images, thereby alleviating the problems of differences between small categories and differences within large categories.

[0066] Step S4: Construct a category-balanced copy-paste data enhancement strategy to balance the number of instances

[0067] The dynamic convolution-based fine-grained feature enhancement module can increase the expressive power of fine-grained features through feature enhancement. However, due to the instance category imbalance problem of the SAR image fine-grained target detection task, the SAR image fine-grained target detection network is prone to under-optimization. The general category-balanced data enhancement strategy adopts a resampling-based method to increase the frequency of observation of the tail category by oversampling images containing the tail category. However, when an image containing the tail category appears multiple times, the head category on the image may also appear multiple times, resulting in poor balancing effect. In addition, the image-level sampling strategy will also significantly increase the training time.

[0068] like Figure 3 As shown, this step constructs a category-balanced copy-paste data augmentation strategy to balance the number of instances at the data level. The basic idea of ​​the category-balanced copy-paste data augmentation strategy is to oversample the limited instances of the minority category at the instance level rather than the image level. More specifically: for category i, the proposed category-balanced copy-paste data augmentation strategy randomly copies an object from the entire category i and oversamples it with probability prob i Paste this onto the current image:

[0069]

[0070] Among them, f i represents the frequency of objects of category i in the entire training dataset, and t is a hyperparameter. If the number of objects of category i is small, the probability of random copy-paste operations is greater. Therefore, the number of aircraft in different categories can be balanced.

[0071] The category-balanced copy-and-paste data augmentation method can directly alleviate the category imbalance problem faced by the fine-grained target detection task in SAR images at the data level. In addition, model training under category balance can also produce a better model optimization process, further improving the fine-grained feature extraction and expression capabilities, and alleviating the problems of differences between small categories and differences within large categories.

[0072] Step S5: Perform model training based on the training set in S1 to obtain a detection model

[0073] The training set in step S1 was used to train the fine-grained object detection network for SAR images with instance feature enhancement. The validation set in step S1 was used to observe the detection performance of the fine-grained object detection network for SAR images. The TaskAlign positive and negative sample allocation strategy was adopted, with the cross-entropy loss function used as the fine-grained classification loss function. CIoU and distributed focal loss were used as the localization loss functions. The backbone network was initialized using the CSPDarknet-53 pre-trained on the COCO dataset. Stochastic gradient descent was used for 300 epochs of training with a batch size of 64 and an initial learning rate of 0.01. The parameters of the fine-grained object detection network for SAR images were updated until the network performance converged. A fine-grained object detection model for SAR images with instance feature enhancement was obtained. The intersection-over-union ratio threshold between target boxes in the post-processing algorithm was set to 0.5 to filter out redundant boxes.

[0074] Step S6: Input the test set in S1 into the trained detection model to obtain fine-grained detection results including

[0075] The test set in S1 is fed into the trained instance-based feature-enhanced fine-grained object detection model to obtain the detection results. The mean average precision (mAP) is used as the evaluation metric to compare the numerical results of the base network and the instance-based feature-enhanced fine-grained object detection method. Furthermore, the visualization results of the base network and the instance-based feature-enhanced fine-grained object detection method are compared.

[0076] The present invention designs a fine-grained enhancement module based on dynamic convolution, which can fully mine the local information of the target scattered between different feature scale levels when fusing high-level feature maps with rich semantic information and low-level feature maps with positioning information, highlight the fine-grained features of the target, and realize fine-grained high-semantic features with high-precision positioning information.

[0077] The present invention adopts a category-balanced copy-and-paste data augmentation method to balance the number of categories at the data instance level, thereby improving the optimization process of the fine-grained target detection network, contributing to better fine-grained feature expression, and comprehensively improving the fine-grained target detection capability of SAR images.

[0078] The present application provides a device for fine-grained target detection in SAR images with enhanced instance features, including a target detection model. The model is implemented by using a YOLOv8 network as a base network and replacing a feature fusion module in the base network with a fine-grained feature enhancement module. The algorithm of the fine-grained feature enhancement module is:

[0079]

[0080] Z=Concat(Ystattc ,X)

[0081]

[0082] Among them, Y is the low-level feature map, X is the high-level feature map, is a fine-grained feature map, It is a convolution layer with a convolution kernel size of k*k, the number of input channels is C, and the number of output channels is C. Concat is a concatenation operation, Avg is an average pooling operation, ReLU is a relu activation function, and F is a convolution operator. Represents broadcasted element-wise addition.

[0083] It should be noted that although the embodiments of the present invention described above are illustrative, they are not intended to limit the present invention. Therefore, the present invention is not limited to the above-mentioned specific embodiments. Without departing from the principles of the present invention, any other embodiments obtained by those skilled in the art under the guidance of the present invention are deemed to be within the protection of the present invention.

Claims

1. A method for fine-grained target detection in SAR images with instance feature enhancement, characterized in that: The following steps are involved: Target detection model construction: The YOLOv8 network is selected as the basic network for fine-grained target detection in SAR images. A fine-grained feature enhancement module based on dynamic convolution is constructed to replace the feature fusion module in the basic network to generate a target detection model. The feature fusion module is used to dynamically learn spatial input-related convolution kernels from multi-scale features. Training sample construction: The preset SAR images are annotated and the number of instances is balanced using the category-balanced copy-and-paste data augmentation strategy; Detection model training: using the training samples to train the target detection model; Target detection: Use the trained target detection model to perform fine-grained target detection in SAR images; The input of the fine-grained feature enhancement module is the low-level feature map and high-level feature maps , the output is a fine-grained feature map ; The specific processing process is: First, static convolution is used to process low-level feature maps , generating static fine-grained features with enhanced spatial position Secondly, and Splicing to obtain cross-scale context features ; Again, for the context features , generating spatially correlated convolution kernels and channel-dependent convolution kernels ; Then, the spatial correlation convolution kernel and channel-dependent convolution kernels , in the low-level feature map Convolution is performed on the dynamic fine-grained features ;Finally, the static fine-grained features and dynamic fine-grained features Connect and fuse along the channel dimension to obtain fine-grained enhanced features ; The algorithm of the fine-grained feature enhancement module is: in, The convolution kernel size is ， The number of input channels is C, and the number of output channels is C. It is a splicing operation. is the average pooling operation, is the relu activation function, is the convolution operator and ⊕ represents broadcasted element-wise addition.

2. The method for fine-grained target detection in SAR images with instance feature enhancement according to claim 1, characterized in that: The fine-grained feature enhancement module extracts static fine-grained features from high-level features through static convolution, learns input-related convolution kernels from static fine-grained features and low-level feature maps through dynamic convolution, and decomposes them into channel-related convolution kernels and space-related convolution kernels to reduce the amount of dynamic convolution parameters while extracting target dynamic fine-grained features in low-level feature maps, and finally fuses the static fine-grained features with the dynamic fine-grained features to output the final fine-grained features.

3. The method for fine-grained target detection in SAR images with instance feature enhancement according to claim 1, characterized in that: The number of balanced instances using the category-balanced copy-paste data augmentation strategy is: For categories , randomly copy the entire category A target in the Paste this onto the current image: in, Indicates the category in the entire training dataset The frequency of the target, is a hyperparameter.

4. The method for fine-grained target detection in SAR images with instance feature enhancement according to claim 1, characterized in that: When training the detection model, a reasonable positive and negative sample allocation strategy, fine-grained classification and positioning loss functions are designed, a stochastic gradient descent optimization algorithm is adopted, a learning rate decay strategy is set, and the parameters of the SAR image fine-grained target detection network are updated until the network performance converges, thus obtaining a SAR image fine-grained target detection model with instance feature enhancement.

5. The method for fine-grained target detection in SAR images with instance feature enhancement according to claim 1, characterized in that: The labeling process for the preset SAR image specifically includes: Construct a fine-grained target classification system and sample annotation specifications based on the preset SAR remote sensing image data and preset expert experience knowledge; Based on the fine-grained target classification system and sample annotation specifications, the target location and category in the image are annotated with horizontal boxes; The labeled images are sliced ​​to form a fine-grained target detection dataset, which is divided into training set, validation set and test set.

6. A fine-grained target detection device for SAR images with instance feature enhancement, for executing the target detection method according to claim 1, comprising a target detection model, characterized in that: The model is implemented by taking the yolov8 network as the base network and replacing the feature fusion module in the base network with a fine-grained feature enhancement module.

Images