Method and device for detecting fine-grained target of SAR (Synthetic Aperture Radar) image based on instance feature enhancement

Abstract

The invention belongs to the technical field of artificial intelligence and target detection, and particularly relates to an SAR image fine-grained target detection method and device based on instance feature enhancement. The method comprises the following steps: constructing a target detection model: selecting a yov8 network as a SAR image fine-grained target detection basic network, constructing a fine-grained feature enhancement module based on dynamic convolution to replace a feature fusion module in the basic network, and generating the target detection model; the feature fusion module is used for dynamically learning space input related convolution kernels from multi-scale features; training sample construction: performing annotation processing on a preset SAR image, and balancing the number of instances by using a category balance copy and paste data enhancement strategy; training a detection model: training a target detection model by using the training sample; and target detection: carrying out SAR image fine-grained target detection by using the trained target detection model.

Description

Technical Field

[0001] The present invention belongs to the technical field of artificial intelligence and target detection, and in particular relates to a method and a device for fine-grained target detection in SAR images with instance feature enhancement. Background Art

[0002] Synthetic Aperture Radar (SAR) is an active earth observation system that can achieve all-day and all-weather earth observation. It has unique advantages in disaster monitoring, environmental monitoring, ocean monitoring, surveying and mapping, and military applications. The target detection task is to give the coordinates of all targets in the image and identify the specific category to which the target belongs. It is one of the important basic tasks of SAR image processing. In recent years, with the rapid development of artificial intelligence technology, SAR image target detection technology based on deep learning has received widespread attention.

[0003] Different from the general SAR image target detection task, the SAR image fine-grained target detection task requires the model to accurately identify similar but different types of targets in the SAR image. Most of the current mainstream SAR image target detection algorithms focus more on how to accurately locate the target, and cannot effectively deal with the fine-grained classification and recognition problem in the detection, resulting in accurate positioning but incorrect classification. A major challenge for fine-grained aircraft detection in SAR images lies in the small inter-class differences and large intra-class differences in complex scenes. Due to the unique imaging method, SAR images lack target texture information and are easily disturbed by noise and redundant background, which can easily lead to problems such as small inter-class differences and large intra-class differences. The small inter-class differences are because some parts of different categories have similar scattering mechanisms, while the large intra-class differences are because different instances of the same category have different scattering power distributions due to different directions, scales and appearances. Another challenge for fine-grained target detection in SAR images comes from class imbalance, especially the imbalance between different subcategories. The general target detection model training method is usually designed with a class-balanced dataset. When these training methods are directly used, it will lead to sub-optimization of model training and a significant performance degradation.

[0004] Therefore, to achieve accurate fine-grained target detection in SAR images, it is necessary to propose a fine-grained feature enhancement method to improve the fine-grained feature extraction capability of the target detection model, as well as a set of instance-level data enhancement strategies to balance the optimization of the target detection model, which has also become a key issue in current research. Summary of the invention

[0005] In order to solve the technical problems existing in the above-mentioned prior art, the purpose of the present invention is to provide a method and device for fine-grained target detection in SAR images with instance feature enhancement, which can utilize instance-level data enhancement and fine-grained feature enhancement to improve the fine-grained feature extraction capability of the target detection model and achieve better fine-grained target detection performance.

[0006] The technical solution for implementing the present invention is as follows:

[0007] In a first aspect, an embodiment of the present application provides a method for fine-grained target detection in a SAR image with enhanced instance features, comprising the following steps:

[0008] Target detection model construction: The yolov8 network is selected as the basic network for fine-grained target detection in SAR images, and a fine-grained feature enhancement module based on dynamic convolution is constructed to replace the feature fusion module in the basic network and generate a target detection model; the feature fusion module is used to dynamically learn the convolution kernel related to the spatial input from multi-scale features;

[0009] Training sample construction: Annotate the preset SAR images and use the category-balanced copy-and-paste data augmentation strategy to balance the number of instances;

[0010] Detection model training: using the training samples to train the target detection model;

[0011] Target detection: Use the trained target detection model to perform fine-grained target detection in SAR images.

[0012] Furthermore, the fine-grained feature enhancement module described in the present invention extracts static fine-grained features from high-level features through static convolution, learns input-related convolution kernels from static fine-grained features and low-level feature maps through dynamic convolution, and decomposes them into channel-related convolution kernels and space-related convolution kernels to reduce the amount of dynamic convolution parameters while extracting target dynamic fine-grained features in low-level feature maps, and finally fuses the static fine-grained features with the dynamic fine-grained features to output the final fine-grained features.

[0013] Furthermore, the input of the fine-grained feature enhancement module of the present invention is the low-level feature map Y and the high-level feature map X, and the output is the fine-grained feature map The specific processing process is as follows:

[0014] First, static convolution is used to process the low-level feature map Y to generate static fine-grained features Y with enhanced spatial position. static ; Secondly, Y static Concatenate with X to obtain the cross-scale context feature Z; again, generate the spatial correlation convolution kernel K for the context feature Z spatial and channel-dependent convolution kernel K channel; Then, the spatial correlation convolution kernel K spatial and channel-dependent convolution kernel K channel , convolution is performed on the low-level feature map Y to obtain the dynamic fine-grained feature Y dynamic ; Finally, the static fine-grained feature Y static and dynamic fine-grained features Y dynamic Connect and fuse along the channel dimension to obtain fine-grained enhanced features

[0015] Furthermore, the algorithm of the fine-grained feature enhancement module of the present invention is:

[0016]

[0017] Z = Concat(Y static , X)

[0018]

[0019] in, It is a convolution layer with a convolution kernel size of k*k, the number of input channels is C, and the number of output channels is C. Concat is a concatenation operation, Avg is an average pooling operation, ReLU is a relu activation function, and F is a convolution operator. Represents broadcasted element-wise addition.

[0020] Furthermore, the number of balanced instances of the category-balanced copy-paste data enhancement strategy described in the present invention is:

[0021] For category i, randomly copy an object from the entire category i and with probability prob i Paste this onto the current image:

[0022]

[0023] Among them, f i represents the frequency of category i targets in the entire training dataset, and t is a hyperparameter.

[0024] Furthermore, when training the detection model, the present invention designs a reasonable positive and negative sample allocation strategy, fine-grained classification and positioning loss function, adopts a stochastic gradient descent optimization algorithm, sets a learning rate decay strategy, and updates the parameters of the SAR image fine-grained target detection network until the network performance converges, thereby obtaining a SAR image fine-grained target detection model with instance feature enhancement.

[0025] Furthermore, the present invention further includes the following steps of labeling a preset SAR image:

[0026] Construct a fine-grained target classification system and sample annotation specifications based on preset SAR remote sensing image data and preset expert experience knowledge;

[0027] Based on the fine-grained target classification system and sample annotation specifications, the target position and category in the image are annotated with horizontal boxes;

[0028] The annotated images are sliced ​​to form a fine-grained object detection dataset, which is divided into training set, validation set and test set.

[0029] In the second aspect, the embodiment of the present application provides a SAR image fine-grained target detection device with instance feature enhancement, including a target detection model, which is: based on the yolov8 network and using a fine-grained feature enhancement module to replace the feature fusion module in the basic network; wherein the algorithm of the fine-grained feature enhancement module is:

[0030]

[0031] Z = Concat(Y static ,X)

[0032]

[0033] Among them, Y is the low-level feature map, X is the high-level feature map, is a fine-grained feature map, It is a convolution layer with a convolution kernel size of k*k, the number of input channels is C, and the number of output channels is C. Concat is a concatenation operation, Avg is an average pooling operation, ReLU is a relu activation function, and F is a convolution operator. Represents broadcasted element-wise addition.

[0034] Compared with the prior art, the present invention has the following beneficial effects:

[0035] First, the present invention adopts a fine-grained feature enhancement method. Through dynamic convolution, when fusing features of adjacent levels, it learns input-related convolution kernels, captures local information of the target scattered between different feature scale levels from two dimensions: space and channel, and highlights the fine-grained features of the target.

[0036] Second, the present invention improves the existing image-level category balanced data enhancement method through instance-level sample enhancement, balances the optimization of the fine-grained target detection network, contributes to better fine-grained feature expression, and comprehensively improves the fine-grained target detection capability of SAR images. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0038] Figure 1 A schematic diagram schematically shows a fine-grained target detection network for SAR images based on feature enhancement according to one embodiment of the present invention;

[0039] Figure 2 A schematic diagram of a fine-grained feature enhancement module based on dynamic convolution according to an embodiment of the present invention is schematically shown;

[0040] Figure 3 Schematically showing a schematic diagram of category balanced copy-paste data enhancement according to an embodiment of the present invention; DETAILED DESCRIPTION

[0041] The embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0042] It should be noted that the following embodiments and features in the embodiments may be combined with each other in the absence of conflict; and, based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in the field without making any creative work are within the scope of protection of the present disclosure.

[0043] It should be noted that various aspects of the embodiments within the scope of the appended claims are described below. It should be apparent that the aspects described herein may be embodied in a wide variety of forms, and any specific structure and / or function described herein is merely illustrative. Based on the present disclosure, it should be understood by those skilled in the art that an aspect described herein may be implemented independently of any other aspect, and two or more of these aspects may be combined in various ways. For example, any number of aspects described herein may be used to implement the device and / or practice the method. In addition, other structures and / or functionalities other than one or more of the aspects described herein may be used to implement this device and / or practice this method.

[0044] The present application provides a method for fine-grained target detection in SAR images with enhanced instance features, comprising the following steps:

[0045] Target detection model construction: The yolov8 network is selected as the basic network for fine-grained target detection in SAR images, and a fine-grained feature enhancement module based on dynamic convolution is constructed to replace the feature fusion module in the basic network and generate a target detection model; the feature fusion module is used to dynamically learn the convolution kernel related to the spatial input from multi-scale features;

[0046] Training sample construction: Annotate the preset SAR images and use the category-balanced copy-and-paste data augmentation strategy to balance the number of instances;

[0047] Detection model training: using the training samples to train the target detection model;

[0048] Target detection: Use the trained target detection model to perform fine-grained target detection in SAR images.

[0049] like Figures 1 to 3 As shown, an example feature-enhanced SAR image fine-grained target detection method of an embodiment of the present application includes the following steps S1-S6. Steps S1-S6 are not the only execution order of the present invention, but only one of many embodiment orders:

[0050] Step S1: annotate the preset SAR image to obtain a SAR image fine-grained target detection dataset; the specific process of this step is:

[0051] The preset SAR remote sensing image data and the preset expert experience knowledge are used to construct a fine-grained target category system and sample annotation specifications. Based on the fine-grained target category system and sample annotation specifications, the target position and category in the image are annotated with horizontal boxes. The annotated image is sliced ​​with the slice size set to 640*640 and the overlap coefficient set to 20% to form a fine-grained target detection dataset, which is divided into training set, validation set and test set in a ratio of 7:1:2.

[0052] Step S2: Constructing a basic network for fine-grained target detection in SAR images

[0053] like Figure 1 As shown in the figure, yolov8 is used as the basic network for fine-grained target detection in SAR images. Yolov8 consists of three parts: backbone network, feature pyramid network and detection network.

[0054] The backbone network is mainly used to extract multi-scale feature maps of the input image. It consists of Conv, C2f and SPPF modules. Conv is a convolutional layer. C2f contains two convolutional layers and multiple Bottleneck blocks. The Bottleneck block is a standard bottleneck structure used to improve model performance. The SPPF module is a spatial pyramid pooling module. It generates multiple fixed-length feature vectors by performing pooling operations on the input feature map at different scales, thereby capturing contextual information of different scales.

[0055] The feature pyramid network fuses high-level feature maps with rich semantic information and low-level feature maps with positioning information through C2f, upsample and Concat operations, thereby increasing the multi-scale expression capability of the model.

[0056] The detection module is responsible for obtaining the fine-grained classification and positioning results of the target pixels on each feature map on the three-level feature maps.

[0057] Step S3: Construct a fine-grained feature enhancement module based on dynamic convolution and replace the feature fusion module in the basic network

[0058] The basic network for fine-grained target detection in SAR images uses a backbone network to extract multi-scale feature maps, and fine-grained local features with discriminative capabilities are scattered in the multi-scale features. However, the traditional feature pyramid network uses conventional convolution operators to directly fuse high- and low-level feature maps. The feature extraction and fusion methods are single, and the network's fine-grained feature expression ability is insufficient, making it difficult to effectively distinguish fine-grained targets. Figure 1 As shown, this step constructs the FGFA fine-grained feature enhancement module to replace the Concat splicing submodule in the feature pyramid network to enhance the fine-grained feature extraction and expression capabilities of the network.

[0059] In order to alleviate the single feature extraction capability of traditional convolution operators, such as Figure 2 As shown in the figure, this step constructs a fine-grained feature enhancement module based on dynamic convolution. Through dynamic convolution, a convolution kernel related to the input is generated from a high-level feature map with rich semantic information to dynamically capture the complex local semantic information of the target on a low-level feature map with positioning information, and then fuse to generate a fine-grained feature map with more detailed discrimination capabilities. Assume that X represents a high-level feature map and Y represents a low-level feature map. Represents the output fine-grained feature map, then the process of the fine-grained feature enhancement module based on dynamic convolution can be expressed as follows:

[0060]

[0061] Z = Concat(Y static ,X)

[0062]

[0063] in, It is a convolution layer with a convolution kernel size of k*k, the number of input channels is C, and the number of output channels is C. Concat is a concatenation operation, Avg is an average pooling operation, ReLU is a relu activation function, and F is a convolution operator. Represents broadcasted element-wise addition.

[0064] Specifically, the FAFG module uses static convolution to process low-level feature maps and generate static feature maps Y with enhanced spatial position. static Subsequently, FAFG will static Concatenate with X to achieve simple cross-scale interaction and obtain cross-scale context features Z, which reflect the context information between local neighborhoods. These context features Z are captured by input-dependent dynamic convolution to exploit the relationship between the two levels. The generation of dynamic convolution kernels is decomposed into the generation of spatial-dependent convolution kernels and channel-dependent convolution kernels to minimize parameters and computational costs. Here, K spatial and K channel Represent the generated spatial correlation convolution kernel and channel correlation convolution kernel respectively. Then, the overall dynamic convolution kernel is convolved on the low-level feature map Y to capture the dynamic local semantic information of the spatially varying target and obtain dynamic fine-grained features. Finally, the static fine-grained features Y static and dynamic fine-grained features Y dynamc Connect along the channel dimension and obtain enhanced features through a simple fusion layer

[0065] The fine-grained enhancement module based on dynamic convolution can enhance the model's fine-grained feature extraction and expression capabilities in the feature pyramid, allowing the model to extract fine-grained local features of the target under the complex background of SAR images, thereby alleviating the problems of differences between small categories and differences within large categories.

[0066] Step S4: Construct a category-balanced copy-paste data enhancement strategy to balance the number of instances

[0067] The fine-grained feature enhancement module based on dynamic convolution can increase the fine-grained feature expression ability by feature enhancement. However, it is limited by the instance category imbalance problem of the SAR image fine-grained target detection task, and the SAR image fine-grained target detection network is prone to under-optimization problems. The general category-balanced data enhancement strategy adopts a resampling-based method to increase the frequency of observing the tail category by oversampling the images containing the tail category. However, when the image containing the tail category appears multiple times, the head category on the image can also appear multiple times, and the balancing effect is not good. In addition, the image-level sampling strategy will also significantly increase the training time.

[0068] like Figure 3 As shown in , this step constructs a category-balanced copy-paste data augmentation strategy to balance the number of instances at the data level. The basic idea of ​​the category-balanced copy-paste data augmentation strategy is to oversample the limited instances of the small sample category at the instance level rather than the image level. More specifically: for category i, the proposed category-balanced copy-paste data augmentation strategy randomly copies an object in the entire category i and oversamples it with probability prob i Paste this onto the current image:

[0069]

[0070] Among them, f i represents the frequency of class i objects in the entire training dataset, and t is a hyperparameter. If the number of class i objects is small, the probability of random copy-paste operations is greater. Therefore, the number of aircraft in different classes can be balanced.

[0071] The category-balanced copy-and-paste data augmentation method can directly alleviate the category imbalance problem faced by the fine-grained target detection task in SAR images at the data level. In addition, model training under category balance can also produce a better model optimization process, further improve the fine-grained feature extraction and expression capabilities, and alleviate the problems of differences between small categories and within large categories.

[0072] Step S5: Perform model training based on the training set in S1 to obtain a detection model

[0073] The training set in step S1 is used to train the SAR image fine-grained target detection network with instance feature enhancement, and the validation set in step S1 is used to observe the detection performance of the SAR image fine-grained target detection network. The TaskAlign positive and negative sample allocation strategy is adopted, the cross entropy loss function is used as the fine-grained classification loss function, CIoU and distributed focalloss are used as the positioning loss function, the CSPDarknet-53 pre-trained with the COCO dataset is used to initialize the backbone network, the stochastic gradient descent method is iteratively trained for 300 rounds, the batch size is 64, the initial learning rate is set to 0.01, and the parameters of the SAR image fine-grained target detection network are updated until the network performance converges, and the SAR image fine-grained target detection model with instance feature enhancement is obtained. The intersection-over-union ratio threshold between target boxes in the post-processing algorithm is set to 0.5 to filter out redundant boxes.

[0074] Step S6: Input the test set in S1 into the trained detection model to obtain fine-grained detection results including

[0075] The test set in S1 is input into the trained instance feature enhanced SAR image fine-grained target detection model to obtain the detection results. On the one hand, the mean average precision mAP is used as the evaluation index to compare the numerical results of the basic network and the instance feature enhanced SAR image fine-grained target detection method, and on the other hand, the visualization results of the basic network and the instance feature enhanced SAR image fine-grained target detection method are compared.

[0076] The present invention designs a fine-grained enhancement module based on dynamic convolution, which can fully mine the local information of the target scattered among different feature scale levels when the high-level feature map with rich semantic information and the low-level feature map with positioning information are fused, highlight the fine-grained features of the target, and realize fine-grained high semantic features with high-precision positioning information.

[0077] The present invention adopts a category-balanced copy-and-paste data enhancement method to balance the number of categories at the data instance level, thereby improving the optimization process of the fine-grained target detection network, contributing to better fine-grained feature expression, and comprehensively improving the fine-grained target detection capability of SAR images.

[0078] The present application discloses a SAR image fine-grained target detection device with enhanced instance features, including a target detection model, which is implemented by using a yolov8 network as a basic network and replacing a feature fusion module in the basic network with a fine-grained feature enhancement module; wherein the algorithm of the fine-grained feature enhancement module is:

[0079]

[0080] Z = Concat(Ystattc ,X)

[0081]

[0082] Among them, Y is the low-level feature map, X is the high-level feature map, is a fine-grained feature map, It is a convolution layer with a convolution kernel size of k*k, the number of input channels is C, and the number of output channels is C. Concat is a concatenation operation, Avg is an average pooling operation, ReLU is a relu activation function, and F is a convolution operator. Represents broadcasted element-wise addition.

[0083] It should be noted that although the embodiments of the present invention described above are illustrative, they are not intended to limit the present invention, and therefore the present invention is not limited to the above specific embodiments. Without departing from the principles of the present invention, any other embodiments obtained by those skilled in the art under the guidance of the present invention are deemed to be within the protection of the present invention.

Claims

1. A method for fine-grained target detection in SAR images with instance feature enhancement, characterized in that: The following steps are involved: Target detection model construction: The yolov8 network is selected as the basic network for fine-grained target detection in SAR images, and a fine-grained feature enhancement module based on dynamic convolution is constructed to replace the feature fusion module in the basic network and generate a target detection model; the feature fusion module is used to dynamically learn the convolution kernel related to the spatial input from multi-scale features; Training sample construction: Annotate the preset SAR images and use the category-balanced copy-and-paste data augmentation strategy to balance the number of instances; Detection model training: using the training samples to train the target detection model; Target detection: Use the trained target detection model to perform fine-grained target detection in SAR images.

2. The method for fine-grained target detection in SAR images with instance feature enhancement according to claim 1, characterized in that: The fine-grained feature enhancement module extracts static fine-grained features from high-level features through static convolution, learns input-related convolution kernels from static fine-grained features and low-level feature maps through dynamic convolution, and decomposes them into channel-related convolution kernels and space-related convolution kernels to reduce the amount of dynamic convolution parameters while extracting target dynamic fine-grained features in low-level feature maps, and finally fuses the static fine-grained features with the dynamic fine-grained features to output the final fine-grained features.

3. The method for fine-grained target detection in SAR images with instance feature enhancement according to claim 2, characterized in that: The input of the fine-grained feature enhancement module is the low-level feature map Y and the high-level feature map X, and the output is the fine-grained feature map The specific processing process is as follows: First, static convolution is used to process the low-level feature map Y to generate static fine-grained features Y with enhanced spatial position. static ; Secondly, Y static Concatenate with X to obtain the cross-scale context feature Z; Again, for the context feature Z, generate a spatial correlation convolution kernel K spatial and channel-dependent convolution kernel K channel ; Then, the spatial correlation convolution kernel K spatial And channel-dependent convolution kernel K channel , convolution is performed on the low-level feature map Y to obtain the dynamic fine-grained feature Y dynamic ; Finally, the static fine-grained feature Y static and dynamic fine-grained features Y dynamic Connect and fuse along the channel dimension to obtain fine-grained enhanced features 4. The method for fine-grained target detection in SAR images with instance feature enhancement according to claim 3, characterized in that: The algorithm of the fine-grained feature enhancement module is: Z=Concat(Y static ,X) in, It is a convolution layer with a convolution kernel size of k*k, the number of input channels is C, and the number of output channels is C. Concat is a concatenation operation, Avg is an average pooling operation, ReLU is a relu activation function, and F is a convolution operator. Represents broadcasted element-wise addition.

5. The method for fine-grained target detection in SAR images with instance feature enhancement according to claim 1, characterized in that: The number of balanced instances using the category-balanced copy-paste data augmentation strategy is: For category i, randomly copy an object from the entire category i and with probability prob i Paste this onto the current image: Among them, f i represents the frequency of category i targets in the entire training dataset, and t is a hyperparameter.

6. The method for fine-grained target detection in SAR images with instance feature enhancement according to claim 1, characterized in that: When training the detection model, a reasonable positive and negative sample allocation strategy and fine-grained classification and positioning loss functions are designed. The stochastic gradient descent optimization algorithm is adopted, and the learning rate decay strategy is set. The parameters of the SAR image fine-grained target detection network are updated until the network performance converges, and a SAR image fine-grained target detection model with instance feature enhancement is obtained.

7. The method for fine-grained target detection in SAR images with instance feature enhancement according to claim 1, characterized in that: The labeling process of the preset SAR image specifically includes: Construct a fine-grained target classification system and sample annotation specifications based on the preset SAR remote sensing image data and the preset expert experience knowledge; Based on the fine-grained target classification system and sample annotation specifications, the target position and category in the image are annotated with horizontal boxes; The labeled images are sliced ​​to form a fine-grained object detection dataset, which is divided into training set, validation set and test set.

8. A SAR image fine-grained target detection device with instance feature enhancement, comprising a target detection model, characterized in that: The model is implemented by taking the yolov8 network as the basic network and replacing the feature fusion module in the basic network with a fine-grained feature enhancement module; wherein the algorithm of the fine-grained feature enhancement module is: Z=Concat(Y static ,X) Among them, Y is the low-level feature map, X is the high-level feature map, is a fine-grained feature map, It is a convolution layer with a convolution kernel size of k*k, the number of input channels is C, and the number of output channels is C. Concat is a concatenation operation, avg is an average pooling operation, ReLU is a relu activation function, and F is a convolution operator. Represents broadcasted element-wise addition.

Images