SAR (Synthetic Aperture Radar) image rotating target detection method and system based on scattering key point guided diffusion model
By using a diffusion model guided by scattering key points, the problems of target outline blurring and false alarms in SAR images are solved, achieving higher detection accuracy and false alarm suppression effect.
Patent Information
- Application Number
- CN202411103236.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-13
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2044-08-13
AI Technical Summary
Existing SAR image rotation target detection methods are insufficient in fitting the blurred outer contour of the target and suppressing false alarms, especially in complex backgrounds where it is difficult to accurately identify and distinguish the target from the background.
A diffusion model guided by scattering key points is adopted. By designing a fine-grained convex polygon representation and a multi-scale scattering key point feature fusion module, the spatial distribution characteristics of the target region are learned by combining the diffusion model, thereby enhancing the distinguishability between the target and the background and reducing false alarms.
It improves the accuracy and recognition capability of rotating target detection in SAR images, reduces the occurrence of false alarms, and adapts to target recognition and localization in complex backgrounds.
Smart Images

Figure CN119007030A_ABST
Abstract
Description
Technical Field
[0001] The invention belongs to the field of remote sensing image processing, and relates to a detection method for a rotating target in a SAR image, and in particular to a detection method and system for a rotating target in a SAR image based on a scattering key point guided diffusion model. Background Art
[0002] Synthetic aperture radar (SAR) is an all-day, all-weather imaging sensor that has been widely used in various fields such as environmental monitoring, military reconnaissance, post-disaster assessment and urban planning. With the rapid development of spaceborne SAR, the automated detection of SAR image targets has attracted widespread attention. Rotating target detection is a challenging image interpretation task that has recently emerged. It is particularly suitable for SAR targets with arbitrary orientation, aspect ratio range and irregular shape characteristics. Due to the unique imaging mechanism, sparsely distributed non-Gaussian and nonlinear coherent speckle noise usually appears in SAR images, causing discontinuity of object texture features in SAR images. Therefore, the outer contour of the target in the SAR image is usually blurred, and there is no clear boundary between the target and the background, which poses a challenge to target recognition; in addition, the complex background in the SAR image will have similar visual attributes to the target, resulting in a large number of false alarms. Traditional SAR target detection methods represented by constant false alarm rate can no longer meet the needs of target detection in these complex scenes, and the gradual rise of deep learning methods with stronger representation capabilities has made it possible to achieve this demand. How to design a deep learning detection method suitable for the target characteristics of SAR images has become a major challenge for rotating target detection in SAR images.
[0003] Early studies usually directly used the horizontal box in general target detection to represent SAR targets, but this representation did not take into account the characteristics of SAR targets and could not meet the needs of accurate detection of SAR targets. Therefore, RoI-Transformer proposed to use a rotation box representation with additional angle information to replace the horizontal box. Compared with the horizontal box, the rotation box can provide accurate direction and scale information of the target, which is more suitable for the detection of rotating targets in SAR images, and has achieved good results. It has been widely used in subsequent SAR target detection methods. However, the rotation box is still based on the representation of regular rectangles, which cannot fit the fuzzy boundaries of the target in the SAR image well, nor can it accurately separate the target area from the background area. With the increasing requirements for the detection of rotating targets in SAR images in recent years, some researchers have gradually realized this problem. SPAN and SFP-Net combined the characteristics of SAR targets and proposed that a set of strong scattering points can be directly used to represent the target. By directly predicting all the strong scattering points in the SAR image and classifying these strong scattering points, the set of strong scattering points belonging to the target can be directly found, and then the detection result of the target can be obtained. This method solves the problems existing in the representation based on rectangular boxes to a certain extent and achieves good detection results.
[0004] In addition to the problem of SAR target characterization methods, the false alarm problem in SAR target detection is also a key research direction. Post-processing algorithm is the simplest method to suppress false alarms. It achieves the purpose of suppressing false alarms by calculating the saliency of the prediction results relative to the background and eliminating the predictions with weaker saliency. However, the robustness of this method is very limited, so it is only used in early research work. With the rise of deep learning methods, more and more researchers have begun to explore methods to suppress false alarms based on deep learning networks. S-SSD directly trains deep neural networks to autonomously learn the saliency information in SAR images, so that it can enhance target features in various complex scenes and suppress the generation of false alarms. However, this method requires high-quality saliency information map annotations to train the network, and it is generally difficult to obtain high-quality saliency information maps. Therefore, with the widespread application of attention mechanisms in recent years, researchers have begun to consider using attention mechanisms to adaptively enhance target features. Both DPAN and AFRAN use attention mechanisms in network structures to find and enhance target features from multi-scale features, suppress background features, and achieve good results in suppressing false alarms. SKG-Net goes a step further and uses the attention mechanism to enhance the features of the target's strong scattering points, so as to more specifically find feature information that can distinguish the target from the complex background, and enhance the network's ability to recognize the target and suppress false alarms.
[0005] Although the deep learning-based rotation target detection method for SAR images has developed very rapidly, there are still some problems with the current detection methods. First, the representation based on the rectangular box cannot fit the fuzzy outer contour of the target well, especially for irregular and densely arranged targets. Although some methods use a set of scattering points to represent the target, which alleviates this problem to a certain extent, it usually requires dense prediction to obtain the scattering points on the entire image, which is computationally intensive and has great difficulty in distinguishing the scattering points of the target and the background. Secondly, most of the existing false alarm suppression methods based on the attention mechanism do not combine the characteristics of the SAR target, but simply use the attention mechanism to fuse multi-scale features. Although SKG-Net specifically uses the attention mechanism to fuse the features of the target's strong scattering points, it is mainly designed for aircraft targets and requires additional auxiliary algorithms. It is not generalizable to other SAR targets. Summary of the invention
[0006] In order to solve the problem of blurred boundary between target and background caused by coherent speckle noise in SAR images and easy false alarm under complex background, a more fine-grained representation is designed to better model the outer contour of the target, and combined with the characteristics that most of the strong scattering areas in SAR images are target areas, a diffusion model is introduced to learn the statistical characteristics of the spatial distribution of the target area, so as to minimize the interference of complex background on target recognition and positioning; and according to the characteristics that SAR target scattering key points contain rich context information, false alarms are further suppressed and the recognition accuracy of the target is improved. The present invention provides a SAR image rotating target detection method based on a scattering key point guided diffusion model.
[0007] The technical solution of the present invention is a SAR image rotating target detection method based on a scattering key point guided diffusion model, comprising the following steps:
[0008] Since the claims are pending finalization, they will not be processed for the time being.
[0009] The present invention aims at the problem of blurred outer contour of target and easy false alarm in complex background in rotating target detection of SAR image. The diffusion model is used to learn the statistical characteristics of spatial distribution of strong scattering areas in images, and the characteristics that most of the strong scattering areas in SAR images are target areas. By designing a target area diffusion module and a denoising decoder, the interference of complex background on target recognition and positioning is minimized, and a multi-scale scattering key point feature fusion module is designed in the denoising decoder to further enhance the distinguishable features of the target relative to the background and reduce the generation of false alarms. The network is based on the Detection Transformer architecture, and end-to-end rotating target detection of SAR images is realized, and the detection accuracy is improved. BRIEF DESCRIPTION OF THE DRAWINGS
[0010] Figure 1 The figure is a flow chart of a method for detecting rotating targets in SAR images according to an embodiment of the present invention.
[0011] Figure 2 It is a schematic diagram of the structure of a feature extraction module according to an embodiment of the present invention.
[0012] Figure 3 It is a schematic diagram of the structure of a multi-scale feature mapping module according to an embodiment of the present invention.
[0013] Figure 4 It is a schematic diagram of the structure of an image feature encoder according to an embodiment of the present invention.
[0014] Figure 5 It is a schematic diagram of the structure of a denoising decoder according to an embodiment of the present invention.
[0015] Figure 6 It is a schematic diagram of the structure of a multi-scale scattering key point feature fusion module according to an embodiment of the present invention.
[0016] Figure 7 It is a schematic diagram of the structure of a convex polygon prediction module according to an embodiment of the present invention.
[0017] Figure 8 It is a flow chart of network training according to an embodiment of the present invention. DETAILED DESCRIPTION
[0018] The present invention is mainly based on a deep learning neural network, and considers the characteristics of the blurred outer contour of the target in the detection of rotating targets in SAR images and the problem of false alarms that are easy to generate in complex scenes, and proposes a SAR image rotating target detection method and system based on a scattering key point guided diffusion model. The method fully considers the problem of blurred outer contour of the target caused by coherent speckle noise in the SAR image and the problem of false alarms that are easy to generate in complex scenes, and better fits the outer contour of the target by designing a fine-grained convex polygon representation to better distinguish the target and background areas, and introduces a diffusion model to learn the statistical characteristics of the strong scattering area in the image space distribution to directly predict the target area, and fully reduce the interference of complex background on target recognition and positioning; in addition, the scattering key points of the target are dynamically captured by designing a multi-scale scattering key point feature fusion module, so as to enhance the distinguishable features of the target and the background based on these scattering key point features, improve the network's recognition ability of the target, and reduce the generation of false alarms. The results obtained by the present invention are more scientific and more accurate.
[0019] The method provided by the present invention can be implemented using computer software technology. Figure 1 , the process of the present invention is specifically described below, as follows:
[0020] A SAR image rotating target detection method based on a scattering key point guided diffusion model, comprising: constructing a SAR image rotating target detection network based on a scattering key point guided diffusion model, wherein the SAR image rotating target detection network comprises a feature extraction module, a multi-scale feature mapping module, an image feature encoder, a target region diffusion module, a target feature initialization module and a time step encoding module, a denoising decoder module and a multi-scale scattering key point feature fusion module therein, and a target convex polygon prediction module;
[0021] The feature extraction module is used to extract feature maps of different scales, input them into the multi-scale feature mapping module, obtain multi-scale feature vectors, input them into the image feature encoder, and fully fuse the features of different scales in the multi-scale feature vectors to obtain the feature vector after multi-scale feature fusion as output; the target area diffusion module is used to obtain the center point coordinates, width and height of the initial target area according to the rotation box annotation of the target in each image through a diffusion process based on Gaussian distribution, and extract the features of the corresponding position of each feature scale from the feature vector output by the image feature encoder according to the center point coordinates of the initialization area, and fuse the extracted features of different feature scales, so as to obtain a feature vector for each initial target area, which is used as the initial target feature vector; the time step encoding module is used to encode the sampling time step t in the target area diffusion module for input The denoising decoder module first uses the initial target area and the initial target feature vector as references, and based on the multi-scale scattering key point feature fusion module, dynamically predicts the scattering key points at different feature scales near the target area, and adaptively extracts the features at the corresponding feature scales from the feature vector obtained from the image feature encoder according to the predicted scattering key points as the key features of the target, and then fuses these extracted key features with the target feature vector to update the target feature vector, and then updates the target area parameters based on the updated target feature vector to obtain a more accurate target area; the target convex polygon prediction module predicts the confidence score of each category for each target and a convex polygon point set consisting of 9 points based on the target feature vector and target area parameters finally updated by the denoising decoder module, thereby obtaining the classification result of the target and the convex polygon prediction result;
[0022] The loss function used in the training process is: the category prediction loss function is calculated by the predicted target category, confidence and target true category labeling; the convex polygon prediction loss function is calculated by the predicted target convex polygon and the target true rotation box labeling parameters; the target area regression loss function is calculated by the final updated target area and the true target area, and then all loss functions are added together to get the total loss function; the trained network is used to detect the data to be detected to obtain the required detection results.
[0023] The following is a detailed description of each module:
[0024] Step 1: construct a feature extraction module to process the input image and obtain input image feature maps of different scales. Figure 2 .
[0025] The feature extraction module consists of 5 parts. The first part consists of a 7*7 convolutional layer with 64 channels, a stride of 2, and a padding of 3, a batch normalization layer, and a 3*3 maximum pooling layer with a stride of 2 and a padding of 1; the second part consists of 1 convolutional block and 2 residual blocks; the third part consists of 1 convolutional block and 3 residual blocks; the fourth part consists of 1 convolutional block and 5 residual blocks; the fifth part consists of 1 convolutional block and 2 residual blocks. The convolution block consists of two parallel branches, one of which consists of a 1*1 convolution layer, ReLU activation function, batch normalization layer, 3*3 convolution layer, ReLU activation function, batch normalization layer, 1*1 convolution layer and batch normalization layer, and the other branch consists of a 1*1 convolution layer and batch normalization layer. The outputs of these two branches are added together and then activated by the ReLU function to obtain the final convolution block output; the residual block consists of a 1*1 convolution layer, ReLU activation function, batch normalization layer, 3*3 convolution layer, ReLU activation function, batch normalization layer, and a 1*1 convolution layer and batch normalization layer. The output is added to the input of the residual block and then activated by the ReLU function to obtain the final residual block output. In each convolution block, the number of channels of the last 1*1 convolution layer of the two branches is twice the number of channels of the input feature map. The number of channels of the last convolution layer in the residual block is the number of channels of the input feature map, and the number of channels of the remaining convolution layers is 1 / 4 of the number of channels of the input feature map. Except for the second part, all 3*3 convolution layers in the remaining convolution blocks and residual blocks have a step size of 2 and a padding of 1, a step size of 1*1 convolution layers has a step size of 1 and a padding of 0, and a step size of all pooling layers has a step size of 2 and a padding of 1. All 3*3 convolution layers in the second part of the convolution blocks and residual blocks have a step size of 1 and a padding of 1, a step size of 1*1 convolution layers has a step size of 1 and a padding of 0, and a step size of 2 and a padding of 1.
[0026] The specific implementation process is described as follows:
[0027] In the first part, the number of channels of the convolution layer is 64. After the input image of 800*800 size and 3 channels is convolved by 7*7, a 64-channel feature map of 400*400 size is obtained. After the 3*3 maximum pooling operation, a 64-channel feature map of 200*200 size is generated. In the second part, the last 1*1 convolution layer of the two branches of the convolution block has 256 channels, and the number of channels of the remaining convolution layers is 64. The last convolution in the residual block The number of channels of the first convolution layer is 256, and the number of channels of the remaining convolution layers is 64. The 200*200 size and 64-channel feature map obtained in the first part is passed through the second part to obtain a 200*200 size and 256-channel output feature map; in the third part, the last 1*1 convolution layer of the two branches of the convolution block has 512 channels, and the number of channels of the remaining convolution layers is 256. The number of channels of the last convolution layer in the residual block is 512, and the number of channels of the remaining convolution layers is 128. The 200*200 size, 256 channel feature map obtained in the second part is converted into a 100*100 size, 512 channel feature map after the third part; in the fourth part, the last 1*1 convolution layer of the two branches of the convolution block has 1024 channels, and the number of channels of the remaining convolution layers is 512. The number of channels of the last convolution layer in the residual block is 1024, and the number of channels of the remaining convolution layers is 256. The 100*100 size, 512 channel feature map obtained in the third part After the feature map passes through the fourth part, a 50*50, 1024 channel feature map is generated; in the fifth part, the number of channels of the last 1*1 convolution layer of the two branches of the convolution block is 2048, and the number of channels of the remaining convolution layers is 1024. The number of channels of the last convolution layer in the residual block is 2048, and the number of channels of the remaining convolution layers is 512. After the 50*50, 1024 channel feature map obtained in the fourth part passes through the fifth part, a 25*25, 2048 channel feature map is generated.
[0028] For the convolutional layer, define I c is the size of the input feature map, O c is the size of the output feature map, k c is the convolution kernel size, p c is the filling parameter, s c is the step size parameter, f c is the number of filters, n c is the number of convolution kernels in the filter (number of output channels).
[0029] The convolution layer performs a two-dimensional convolution operation on each channel dimension of the feature map, by sliding the convolution kernel on the two-dimensional features and performing matrix multiplication on the features within the current convolution kernel range to obtain a single output feature value. The output feature is the weighted sum of the input features at the same position, and the weight is the parameter of the convolution kernel. In order to ensure that the output feature map is the same size as the input feature map, the present invention uses a padding parameter, that is, to fill the edge of the input feature map with zero-value features of the corresponding parameter size, to ensure that when the convolution kernel slides on the input feature map, it can allow the original edge features to be located at the center of its kernel and extend to the padded features, thereby generating an output feature map of the same size as the input feature map. The number of padded features is determined by the padding parameter p. c Determine that the size of the input feature map becomes I after padding c +2*p c , and the output feature map size after the convolution layer is When the technical solution of the present invention is implemented, the step size of the 1*1 convolution layer is 1, the padding parameter is 0, and the step size of the 3*3 convolution layer is 2, and the padding parameter is 1. In each convolution layer, there are filters equal to the number of channels of the output feature map, and in each filter, there are convolution kernels equal to the number of channels of the input feature map. Each convolution kernel in each filter performs a two-dimensional convolution operation on its corresponding channel. The obtained feature maps are added at the corresponding positions, and the bias is added to obtain the final output feature map.
[0030] In order to enhance the robustness of the features extracted by the feature extraction module, the present invention makes extensive use of ReLU activation functions to improve the feature expression capability. The calculation formula of the ReLU activation function is f(x)=max(0,x), where x is the input feature value of the function and max(·) is the function for finding the maximum value. At the same time, in order to prevent the gradient vanishing and gradient exploding problems in the network during training, the feature extraction network also makes extensive use of batch normalization layers, which introduce learnable parameters. β=E(x), where x is the output feature map of the convolutional layer, Var(·) is the function for finding the variance, E(·) is the function for finding the mean, and the batch normalization output is The pooling layer performs a pooling operation on each channel of the feature map. Generally speaking, the feature map of each channel will be divided into multiple areas of pooling size without overlap. The maximum pooling operation will select the largest eigenvalue in each area as the only output eigenvalue, while the average pooling operation will take the average of all eigenvalues in each area as the only output eigenvalue. The feature map obtained after the above operation is the output feature map of the pooling layer;
[0031] Step 2: construct a multi-scale feature mapping module, process the feature maps of different scales obtained by different parts of the feature extraction module in step 1, and obtain a multi-scale feature vector, see Figure 3 .
[0032] When building this module, feature maps with different numbers of channels need to be aligned. Specifically, the feature maps obtained from the third, fourth, and fifth parts of the feature extraction module in step 1 are first extracted and processed as follows: the feature map with 512 channels and a scale of 100*100 obtained from the third part is processed through a 3*3 convolution layer with a step size of 1, a padding of 1, and a channel number of 256 to obtain a feature map with 256 channels and a scale of 100*100; the feature map with 1024 channels and a scale of 50*50 obtained from the fourth part is processed through a 3*3 convolution layer with a step size of 1, a padding of 1, and a channel number of 256 to obtain a feature map with 256 channels and a scale of 100*100; After the 3*3 convolution with stride 1 and 256 channels, the feature map with 256 channels and scale 50*50 is obtained; and the feature map with 2048 channels and scale 25*25 obtained from the fifth part is respectively subjected to the 3*3 convolution layer with stride 1, padding 1, and 256 channels and the 3*3 convolution layer with stride 2, padding 1, and 256 channels, the feature map with 256 channels and scale 25*25 and the feature map with 256 channels and scale 13*13 are obtained. The four feature maps obtained above are transformed into one-dimensional vectors in the spatial dimension to obtain feature vectors with 256 channels and lengths of 10000, 2500, 625 and 169, respectively. These feature vectors are then concatenated in the spatial dimension to obtain a multi-scale feature vector output with a length of 13294 and 256 channels.
[0033] Step 3: construct an image feature encoder, perform multi-scale feature fusion on the multi-scale feature vector obtained by the multi-scale feature mapping module in step 2, and obtain an output feature vector after feature fusion, see Figure 4 .
[0034] The image feature encoder consists of 6 encoder layers, each of which consists of 1 multi-head deformable self-attention layer, 1 batch normalization layer, 1 feedforward neural network and 1 batch normalization layer. Among them, the feedforward neural network consists of 1 linear layer, 1 ReLU activation function and 1 linear layer. Specifically, in the image feature encoder, the first encoder layer will take the multi-scale feature vector with a length of 13294 and 256 channels obtained by the multi-scale feature mapping module in step 2 as input, and the subsequent encoder layers will take the output feature vector of the previous encoder layer as input. In each encoder layer, the multi-head deformable self-attention layer uses the feature vector of the input encoder layer as the query, key, and value at the same time, and uses the position information of each element in the feature vector on the original two-dimensional feature map as a reference point to adaptively capture the relevant feature points that need to calculate the attention matrix, and fuse the features of the corresponding positions according to the calculated attention matrix to achieve the fusion of features at different positions of the input feature vector; the feedforward neural network will first increase the number of channels of the input feature vector, and then reduce it to more fully fuse the features of different channel dimensions. Specifically, the input parameter of the first linear layer in the feedforward neural network is 256, and the output parameter is 1024, while the input parameter of the second linear layer is 1024 and the output parameter is 256. The image feature encoder can fully fuse the features of different scales in the multi-scale feature vector obtained in step 2 to obtain the feature vector F after feature fusion. v as output.
[0035] Step 4: construct a target area diffusion module, obtain the real target area based on the marked target rotation box, and then obtain the noisy target area through the diffusion process as the initial target area.
[0036] When constructing the target area diffusion module, firstly, the rotation box of the target is annotated Get its minimum circumscribed horizontal rectangle, and then expand the width and height of the rectangle by 1.2 times to get the real target area R g ,Right now Where MinAreaRect(·) is a function for calculating the minimum horizontal circumscribed rectangle of a quadrilateral. The diffusion model is a likelihood-based model inspired by non-equilibrium thermodynamics. It defines a Markov chain of the diffusion process by gradually adding noise to the sample data. Therefore, in the iterative process of training, the present invention constructs a Markov chain from the real target area (strong scattering area) R g To the noise area R n The diffusion process, after a random sampling time step t, t∈[0,T], can be expressed as To realize this diffusion process, x t =R g , x0=Rg , is the noise variance scheduling parameter, β s is the noise variance scheduling hyperparameter. In the present invention, the noise variance scheduling hyperparameter of the existing diffusion model is adopted. Through the above formula, we can obtain N g noisy target regions from N g real target regions. Since the Detection Transformer architecture adopted in the present invention is an end-to-end architecture, which requires the lengths of the input vector and the output vector to be the same. Therefore, considering that the present invention needs to predict N results (outputs), after obtaining N n noisy target regions, it is necessary to fill or delete them to obtain the initial target regions. Specifically, when N g < N, N - N g regions are randomly sampled based on the Gaussian distribution and concatenated behind the N g noisy target regions to obtain N initial target regions R g ; when N i > N, the first N of the N g noisy target regions are selected as the initial target regions R g ; when N i = N, the N g noisy target regions are directly used as the initial target regions. In addition, the construction of this module is slightly different in the inference process and the training process. In the inference process, this module directly generates N regions based on the Gaussian distribution as the initial target regions R g , without the need to obtain the noisy target regions based on the real target regions. i
[0037] Step 5: Construct the target feature initialization module and the time step encoding module.
[0038] When constructing the target feature initialization module, first, according to the center point position information of the initial target regions R i obtained by the target region diffusion module in Step 4, the features corresponding to the positions of the feature vectors F v obtained by the image feature encoder in Step 3 are extracted, and the extracted features are concatenated in the channel dimension. Then, through a linear layer, the channel dimension is transformed into 256, so as to obtain a feature vector for each initial target region. These feature vectors together constitute the target feature vector. Specifically, for the j-th target initialization region the extracted feature vector is where is the center point coordinate of The number of channels is 1024 (the number of channels of each scale feature map is 256. After extracting features from the corresponding positions of the four scale feature vectors and concatenating them in the channel dimension, the number of channels becomes 1024). i , we can get a feature vector with a length of N and 1024 channels. After passing through a linear layer with an input parameter of 1024 and an output parameter of 256, we can get the final initial target feature vector with a length of N and 256 channels.
[0039] When constructing the time step encoding module, first use the cosine position encoding method to encode the time step t sampled in step 4 to obtain a time code t with 256 channels. emb = SinePositionalEncoding(t), where SinePostionalEncoding(·) is the cosine position encoding function, and then after a linear layer with an input parameter of 256 and an output parameter of 1024, the time encoding t is obtained after the number of channels is expanded by 4 times emb , and then pass through a SiLU activation function and a linear layer with input and output parameters of 1024, the final time encoding output t with a length of 1 and a channel number of 1024 is obtained emb .
[0040] Step 6: Construct a denoising decoder and a multi-scale scattering key point feature fusion module, see Figure 5 and Figure 6 .
[0041] The denoising decoder consists of 6 denoising decoder layers and corresponding 6 target region update modules. The denoising decoder layer consists of 1 multi-head self-attention layer, 1 batch normalization layer, 1 multi-scale scattering key point feature fusion module, 1 batch normalization layer, 1 feedforward neural network and 1 adaptive batch normalization layer; the multi-scale scattering key point feature fusion module consists of 4 parallel linear layers, 1 linear layer and 1 cross attention layer; the feedforward neural network consists of 1 linear layer, 1 ReLU activation function and 1 linear layer; the target region update module consists of 1 linear layer, ReLU activation function, 1 linear layer, ReLU activation function and 1 linear layer.
[0042] Specifically, the first denoising decoder layer in the denoising decoder takes the initial target feature vector with a length of N and a channel number of 256 obtained by the target feature initialization module in step 5 as input, and the subsequent denoising decoder layers will take the target feature vector updated by the previous denoising decoder layer as input. In each denoising decoder layer, the multi-head self-attention layer uses the target feature vector of the input denoising decoder layer as query, key and value at the same time to calculate the attention matrix between the N target feature vectors, and fuses the features at different positions according to the calculated attention matrix to obtain an output feature vector with a length of N and a number of channels of 256; in the multi-scale scattering key point feature fusion module, the feature vector with a length of N and a number of channels of 256 obtained by the multi-head self-attention layer will pass through 4 parallel linear layers with an input parameter of 256 and an output parameter of 18 to obtain 4 prediction results with a length of N and a number of channels of 18, which correspond to 18 horizontal and vertical coordinate offsets of 9 points of N target areas on 4 feature scales, respectively. Based on the predicted offsets and the coordinates of the center point of the target area, 9 sampling points can be obtained for each target area on 4 feature scales, which are the scattering key points dynamically predicted in the target area. According to these predicted scattering key points, features can be extracted from the corresponding positions of the feature vector obtained in step 3. The features of the same point at different scales are concatenated in the channel dimension, so that a feature vector with a length of 36 and a number of channels of 1024 is obtained for each target area. After a linear layer with an input parameter of 1024 and an output parameter of 256, a scattering key point feature vector with a length of 36 and a number of channels of 256 is obtained. The feature vector is used as the key and value, and the input target feature vector of the module is used as the query and sent to the cross attention layer to obtain a target feature vector with a length of N and a number of channels of 256 after fusion of the scattering key point features; in the feedforward neural network, the multi-scale scattering key point feature fusion module obtains a target feature vector with a length of N and a number of channels of 256. After a linear layer with an input parameter of 256 and an output parameter of 1024, a feature vector with a length of N and a number of channels of 1024 is obtained. After the ReLU activation function obtains nonlinear learning ability, it passes through a linear layer with an input parameter of 1024 and an output parameter of 256, and a target feature vector with a length of N and a number of channels of 256 is obtained as the output.
[0043] In addition, since the diffusion process constructed by the target area diffusion module in step 4 of the present invention needs to be guided by the generated Gaussian noise according to the sampling time step t to obtain the noisy target area, the denoising decoder also needs to guide the denoising process (update process) of the initial target area according to the sampling time step t. In order to achieve this function, the present invention introduces an adaptive batch normalization layer after the feedforward neural network of the denoising decoder layer. Based on the ordinary batch normalization layer, the adaptive batch normalization layer introduces scale scaling and offset parameters related to the time step t. Specifically, the time step encoding module in step 5 obtains a time step encoding t with a length of 1 and a channel number of 1024. emb , after the linear layer with an input parameter of 1024 and an output parameter of 512 and the SiLU activation function, an output vector with a length of 1 and a number of channels of 512 is obtained. The vector is divided into two vectors with a length of 1 and a number of channels of 256 in the channel dimension, and N copies are made. The scale scaling parameter scale and the offset parameter shift with a length of N and a number of channels of 256 can be obtained respectively. scale, shift = expand(split(Linear(t emb ))), where split(·) is the channel splitting function, expand(·) is the replication function, and Linear(·) is the linear layer. In the adaptive batch normalization layer, the feedforward neural network obtains a feature vector F with a length of N and a number of channels of 256. FFN_o , first passes through the ordinary batch normalization layer, and then uses the scale scaling and offset parameters obtained according to the sampling time step t to obtain the final output feature vector F with a length of N and a number of channels of 256 AdaNorm_o , where norm(·) is a normal batch normalization layer. AdaNorm_o That is, the output target feature vector updated for each denoising decoder layer.
[0044] The target feature vector output by each denoising decoder layer is divided into two branches. One branch is directly input into the next decoder layer as the input target feature vector, and the other branch is input into the target region update module to update the target region according to the updated target feature vector, and use the updated target region in the multi-scale scattering key point module in the next decoder layer, so that it can predict more accurate scattering key points. In the target region update module, the input target feature vector with a length of N and a number of channels of 256 is passed through a linear layer with an input parameter of 256 and an output parameter of 4, and a prediction result with a length of N and a number of channels of 4 is obtained, which corresponds to the horizontal and vertical coordinates, width and height offsets of the center points of the N target regions. The parameters of the target region can be updated according to these predicted offsets. Define the original target area as (x0, y0, w0, h0), and the predicted offset as (Δx, Δy, Δw, Δh), then the updated target area (x1, y1, w1, h1) = (x0 + Δx, y0 + Δy, w0 + Δw, h0 + Δh).
[0045] Step 7: construct a target convex polygon prediction module, which consists of a classification branch and a regression branch. The target feature vector and target area finally updated by the denoising decoder in step 6 are used as input to obtain the classification and convex polygon prediction results of N targets. Figure 7 .
[0046] In the classification branch, the denoising decoder in step 6 finally updates the target feature vector with a length of N and a number of channels of 256. The input parameter is 256 and the output parameter is N. c After the linear layer, we get a length of N and a number of channels of N c The prediction results, where N c The number of categories in the data set is marked, and the prediction result on each channel is the confidence prediction score of a category;
[0047] In the regression branch, the target feature vector with a length of N and a number of channels of 256 obtained by the denoising decoder in step 6 is finally updated. After a linear layer with an input and output parameter of 256, a ReLU activation function, a linear layer with an input and output parameter of 256, and a ReLU activation function, the position features in the vector are enhanced, and finally, after a linear layer with an input parameter of 256 and an output parameter of 18, a prediction result with a length of N and a number of channels of 18 is obtained, corresponding to 18 horizontal and vertical coordinate offsets of 9 points in N target areas. According to these predicted offsets and the horizontal and vertical coordinates of the corresponding target area center point, 9 predicted points can be obtained for each target area to form the target convex polygon prediction of the area. By calculating the minimum circumscribed horizontal rectangle of the predicted target convex polygon, the predicted target convex polygon can be converted into a rotation box prediction result.
[0048] Step 8: Train the SAR image rotation target detection network based on the scattering key point guided diffusion model constructed in the above steps on the data set, see Figure 8 The network parameter optimization process is based on the total network loss value obtained during training. When the total loss value converges to a certain value as the number of iterations of network training decreases and no longer fluctuates over a large range, it can be judged that the network has completed convergence. The network weight parameter value at this time is extracted and saved as the final network weight obtained by training. Figure 4 As shown, the total loss function of the present invention is divided into three parts, and the specific calculation method is as follows:
[0049] The first part is the category prediction loss cls , you can use Calculated, where P i is the predicted category probability of the i-th prediction result, is the category probability of the annotation box corresponding to the i-th prediction result. The loss function is essentially the difference between the category prediction probability of the i-th prediction result and the corresponding true category probability.
[0050] The second part is the convex polygon prediction loss loss convex ,use The formula is calculated, where is the prediction target convex polygon of the i-th prediction result, Annotate the target true rotation box corresponding to the i-th prediction result, is and The minimum circumscribed polygon of the i-th prediction result is essentially the fit between the predicted convex polygon of the i-th prediction result and the corresponding target real rotation frame;
[0051] The third part is the target area regression loss loss region ,use The formula is calculated, where is the regression target area corresponding to the i-th prediction result, is the target true area corresponding to the i-th prediction result, C is the minimum enclosing area between the regression target area and the target true area, and the loss function is essentially the gap between the predicted target area and the true target area corresponding to the i-th prediction result.
[0052] The parameters set during the training process of the embodiment of the present invention are as follows in Table 1:
[0053] Table 1 Network training parameter settings
[0054]
[0055] Among them, the Batch parameter controls the number of images sent to the network for training each time; Img width and Img height are the sizes that the input images need to maintain during training. When the input image size is different from the size, it can be changed to the set size through proportional scaling and filling 0 value strategy; Learning rate is the learning rate of training, which determines the update amplitude of network parameters and the convergence speed of the network; Optimizer is the parameter optimizer of the network, and its parameters β1=0.9, β2=0.999 are set; Param scheduler is the learning rate reduction strategy during network training. This embodiment selects the MultiStepLR strategy, that is, every time the training reaches the specified number of iterations, the learning rate is reduced to one tenth of the original, and the learning rate is set to decrease once when the iteration reaches the 175000th and 215000th times; Warm up is the learning rate setting strategy in the initial stage of training, that is, the learning rate is changed to 1% of the set learning rate at the beginning of training, and the learning rate is linearly increased to the set learning rate in the next 2000 steps of iterations to stabilize the parameter optimization process of the network in the initial stage of training; Num Steps is the total number of iterations of training, and this embodiment selects 225000 as the total number of iterations. In actual operation, relevant technicians can change the above training parameters or add additional training parameters according to specific implementation conditions to achieve the best training effect on the network.
[0056] The specific embodiments described herein are merely examples of the spirit of the present invention. Those skilled in the art may make various modifications or additions to the specific embodiments described or replace them in similar ways, but they will not deviate from the spirit of the present invention or exceed the scope defined by the appended claims.
Claims
1. A method for detecting rotating targets in SAR images based on a scattering key point guided diffusion model, characterized in that: include: Constructing and training a SAR image rotating target detection network based on a scattering key point guided diffusion model, the SAR image rotating target detection network includes a feature extraction module, a multi-scale feature mapping module, an image feature encoder, a target region diffusion module, a target feature initialization module and a time step encoding module, a denoising decoder module and a multi-scale scattering key point feature fusion module, and a target convex polygon prediction module; The feature extraction module is used to extract feature maps of different scales, input them into the multi-scale feature mapping module, obtain a multi-scale feature vector, input them into the image feature encoder, and fully fuse the features of different scales in the multi-scale feature vector to obtain a feature vector after multi-scale feature fusion as output; The target area diffusion module is used to obtain the center point coordinates, width and height of the initial target area according to the rotation box annotation of the target in each image through a diffusion process based on Gaussian distribution, and extract the features of the corresponding position of each feature scale from the feature vector output by the image feature encoder according to the center point coordinates of the initialization area, and fuse the extracted features of different feature scales, so as to obtain a feature vector for each initial target area, which is used as the initial target feature vector; The time step encoding module is used to encode and output the sampling time step t in the target area diffusion module; the denoising decoder module first uses the initial target area and the initial target feature vector as a reference, and based on the multi-scale scattering key point feature fusion module, dynamically predicts the scattering key points at different feature scales near the target area, and adaptively extracts the features at the corresponding feature scales from the feature vector obtained from the image feature encoder based on the predicted scattering key points as the key features of the target, and then fuses these extracted key features with the target feature vector to update the target feature vector, and then updates the target area parameters based on the updated target feature vector to obtain a more accurate target area; the target convex polygon prediction module will predict the confidence score of each category for each target and a convex polygon point set consisting of 9 points based on the target feature vector and target area parameters finally updated by the denoising decoder module, thereby obtaining the classification result of the target and the convex polygon prediction result; The loss functions used in the training process are: the category prediction loss function is calculated by the predicted target category, confidence and target true category label; the convex polygon prediction loss function is calculated by the predicted target convex polygon and the target true rotation box label parameters; the target area regression loss function is calculated by the final updated target area and the true target area, and then all loss functions are added together to get the total loss function; Finally, the trained target detection network is used to detect the data to be detected and obtain the required detection results.
2. The SAR image rotating target detection method based on the scattering key point guided diffusion model as claimed in claim 1, characterized in that: The feature extraction module consists of 5 parts. The first part consists of a 7*7 convolutional layer with 64 channels, a stride of 2, and a padding of 3, a batch normalization layer, and a 3*3 maximum pooling layer with a stride of 2 and a padding of 1; the second part consists of 1 convolutional block and 2 residual blocks; the third part consists of 1 convolutional block and 3 residual blocks; the fourth part consists of 1 convolutional block and 5 residual blocks; the fifth part consists of 1 convolutional block and 2 residual blocks; the convolutional block consists of two parallel The outputs of the two branches are added together and then passed through the ReLU activation function to obtain the final convolution block output; the residual block is composed of a 1*1 convolution layer, a ReLU activation function, a batch normalization layer, a 3*3 convolution layer, a ReLU activation function, a batch normalization layer, a 1*1 convolution layer and a batch normalization layer. The output is added to the input of the residual block and then passed through the ReLU activation function to obtain the final residual block output; in each convolution block, the number of channels of the last 1*1 convolution layer of the two branches is twice the number of channels of the input feature map; the number of channels of the last convolution layer in the residual block is the number of channels of the input feature map, and the number of channels of the remaining convolution layers is 1 / 4 input feature map channels; except for the second part, all 3*3 convolutional layers in the remaining convolutional blocks and residual blocks have a step size of 2 and a padding of 1, a step size of 1*1 convolutional layers has a step size of 1 and a padding of 0, and a step size of all pooling layers has a step size of 2 and a padding of 1; while all 3*3 convolutional layers in the second part of the convolutional blocks and residual blocks have a step size of 1 and a padding of 1, a step size of 1*1 convolutional layers has a step size of 1 and a padding of 0, and a step size of 2 and a padding of 1.
3. The SAR image rotating target detection method based on the scattering key point guided diffusion model as claimed in claim 2, characterized in that: The specific processing process of the multi-scale feature mapping module is as follows: The input feature map is the feature map obtained by the third, fourth and fifth parts of the feature extraction module in step 1; The above three input feature maps are respectively subjected to a 3*3 convolution layer with a stride of 1 and a channel of 256 to obtain three output feature maps; then, the last output feature map will be further subjected to another 3*3 convolution layer with a stride of 2 and a channel of 256 to obtain the fourth output feature map; the above four output feature maps will be transformed into one-dimensional vectors in the spatial dimension and concatenated in the dimension to finally obtain a multi-scale feature vector.
4. The method for detecting rotating targets in SAR images based on a scattering key point guided diffusion model as claimed in claim 1, characterized in that: The specific processing process of the image feature encoder is as follows: The image feature encoder consists of several encoder layers, each of which consists of a multi-head deformable self-attention layer, a batch normalization layer, a feedforward neural network, and a batch normalization layer; the feedforward neural network consists of a linear layer, a ReLU activation function, and a linear layer; specifically, in the image feature encoder, the first encoder layer takes the multi-scale feature vector obtained by the multi-scale feature mapping module as input, and the subsequent encoder layers take the output feature vector of the previous encoder layer as input; In each encoder layer, the multi-head deformable self-attention layer uses the feature vector of the input encoder layer as the query, key and value at the same time, and uses the position information of each element in the feature vector on the original two-dimensional feature map as the reference point to adaptively capture the relevant feature points that need to calculate the attention matrix, and fuses the features of the corresponding positions according to the calculated attention matrix to achieve the fusion of features at different positions of the input feature vector; the feedforward neural network will first increase the number of channels of the input feature vector, and then reduce it to more fully fuse the features of different channel dimensions; the image feature encoder fully fuses the features of different scales in the obtained multi-scale feature vector to obtain the feature vector F after feature fusion. v as output.
5. The method for detecting rotating targets in SAR images based on a scattering key point guided diffusion model as claimed in claim 1, characterized in that: The specific implementation process of the target area diffusion module is as follows: First, mark B according to the target's rotation box g Get its minimum circumscribed horizontal rectangle, and then expand the width and height of the rectangle by b times to get the real target area R g , that is, R g = b*MinAreaRect(B g ), where MinAreaRect(·) is a function for calculating the minimum horizontal circumscribed rectangle of a quadrilateral; During the iterative process of training, a diffusion process is constructed from the real target region R g to the noise region R n . After randomly sampling the time step t, t ∈ [0, T], is used to implement this diffusion process, where x t = R g , x0 = R g , is the noise variance scheduling parameter, and β s is the noise variance scheduling hyperparameter; through the above formula, N g noisy target regions R g can be obtained from B n real target regions; in addition, since N results need to be predicted, after obtaining N g target noise regions, they need to be padded or trimmed to obtain the initial target regions; specifically, when N g < N, N - N g regions are randomly sampled based on the Gaussian distribution and concatenated behind the N g noisy target regions to obtain N initial target regions R i ; when N g > N, the first N of the N g noisy target regions are selected as the initial target regions R i ; when N g = N, the N g noisy target regions are directly used as the initial target regions; and during the inference process, the target region diffusion module directly generates N regions based on the Gaussian distribution as the initial target regions R i , without the need to obtain the noisy target regions based on the real target regions.
6. The method for detecting rotating targets in SAR images based on a scattering key point guided diffusion model as claimed in claim 1, characterized in that: The specific processing process of the target feature initialization module and the time step encoding module is as follows: When constructing the target feature initialization module, first obtain the initial target area R according to the target area diffusion module i The center point position information is extracted, and the feature vector F is obtained by the image feature encoder v The features of the corresponding positions are concatenated in the channel dimension, and then transformed into 256 through a linear layer, so that a feature vector is obtained for each initial target area, and these feature vectors together constitute the target feature vector; specifically, for the jth target initialization area The extracted feature vector is in for After the above process, the center point coordinates of N initial target areas R i , we can get a feature vector of length N, and then after one linear layer, we can get the final initial target feature vector; When constructing the time step encoding module, the sampled time step t is first encoded using cosine position encoding to obtain a time code with 256 channels. Then, after passing through a linear layer, the time code with the number of channels expanded by 4 times is obtained. After passing through a SiLU activation function and a linear layer, the final time code output t is obtained. emb .
7. The method for detecting rotating targets in SAR images based on a scattering key point guided diffusion model as claimed in claim 1, characterized in that: The specific processing process of the denoising decoder module and the multi-scale scattering key point feature fusion module is as follows: The denoising decoder module consists of several denoising decoder layers and corresponding target region update modules, where the denoising decoder layer consists of 1 multi-head self-attention layer, 1 batch normalization layer, 1 multi-scale scattering key point feature fusion module, 1 batch normalization layer, 1 feedforward neural network and 1 adaptive batch normalization layer; the multi-scale scattering key point feature fusion module consists of 4 parallel linear layers, 1 linear layer and 1 cross attention layer; the feedforward neural network consists of 1 linear layer, 1 ReLU activation function and 1 linear layer; the target region update module consists of 1 linear layer, ReLU activation function, 1 linear layer, ReLU activation function and 1 linear layer; Specifically, the first denoising decoder layer in the denoising decoder takes the initial target feature vector obtained by the target feature initialization module as input, and the subsequent denoising decoder layers take the target feature vector updated by the previous denoising decoder layer as input; in each denoising decoder layer, the multi-head self-attention layer uses the target feature vector of the input denoising decoder layer as query, key and value at the same time to calculate the attention matrix between N target feature vectors, and fuses the features at different positions according to the calculated attention matrix to obtain an output feature vector with a length of N and a number of channels of 256; The specific processing process of the multi-scale scattering key point feature fusion module is as follows: The feature vector with a length of N and a number of channels of 256 obtained by the multi-head self-attention layer passes through 4 parallel linear layers to obtain 4 prediction results with a length of N and a number of channels of 18, which correspond to the coordinate offsets of 9 points of N target areas on 4 feature scales. According to these predicted offsets and the coordinates of the center points of the N target areas, 9 points are obtained for each target area on 4 feature scales. These points are the scattering key points dynamically predicted in the target area. According to the position information of these points, features are extracted from the corresponding positions on the obtained multi-scale feature vector, and the features of the same point at different scales are spliced together in the channel dimension to obtain a scattering key point feature vector for each target area. After a linear layer, a feature vector with a length of 36 and a number of channels of 256 is obtained. The feature vector is used as the key and value, and the input feature vector of the multi-scale scattering key point feature fusion module is sent to the multi-head self-attention layer as a query to obtain the target feature vector after fusion of the scattering key point features, which has a length of N and a number of channels of 256.
8. The method for detecting rotating targets in SAR images based on a scattering key point guided diffusion model as claimed in claim 7, characterized in that: The adaptive batch normalization layer is based on the ordinary batch normalization layer and introduces the scale scaling and offset parameters related to the time step t; specifically, the final time encoding output t obtained by the time step encoding module emb After the linear layer and SiLU activation function, the output vector is obtained. The output vector is divided into two vectors in the channel dimension and copied N times to obtain the scale scaling parameter scale and the offset parameter shift, respectively. scale, shift = expand (split (Linear (t emb ))), where split(·) is the channel splitting function, expand(·) is the replication function, and Linear(·) is the linear layer; in the adaptive batch normalization layer, the feature vector F obtained by the feedforward neural network FFN_o , first passes through the ordinary batch normalization layer, and then uses the scale scaling and offset parameters obtained according to the sampling time step t to obtain the final output feature vector F AdaNorm_o , F AdaNorm_o =norm(F FFNo )*scale+shift, where norm(·) is a normal batch normalization layer, F AdaNorm_0 That is, the output target feature vector updated by each denoising decoder layer; In addition, the target feature vector output by each denoising decoder layer is divided into two branches. One branch is directly input into the next decoder layer as the input target feature vector, and the other branch is input into the target region update module to update the target region according to the updated target feature vector and use the updated target region in the multi-scale scattered key point module in the next decoder layer, so that it can predict more accurate scattered key points. In the target area update module, after the input target feature vector passes through the linear layer, a prediction result with a length of N and a number of channels of 4 is obtained, which corresponds to the horizontal and vertical coordinates, width and height offsets of the center points of the N target areas. The parameters of the target area can be updated according to these predicted offsets; the original target area is defined as (x0, y0, w0, h0), and the predicted offset is (Δx, Δy, Δw, Δh), then the updated target area (x1, y1, w1, h1) = (x0 + Δx, y0 + Δy, w0 + Δw, h0 + Δh).
9. The method for detecting rotating targets in SAR images based on a scattering key point guided diffusion model as claimed in claim 1, characterized in that: The target convex polygon prediction module includes classification branch and regression branch. The specific processing process is as follows: In the classification branch, the denoising decoder finally updates the target feature vector, which has a length of N and a number of channels N after a linear layer. c The prediction results, where N c The number of categories in the data set is marked, and the prediction result on each channel is the confidence prediction score of a category; In the regression branch, the denoising decoder finally updates the target feature vector, which is enhanced by the linear layer, ReLU activation function, linear layer and ReLU activation function. Finally, it passes through a linear layer to obtain a prediction result with a length of N and 18 channels, corresponding to 18 horizontal and vertical coordinate offsets of 9 points in N target areas. According to these predicted offsets and the horizontal and vertical coordinates of the center point of the corresponding target area, 9 predicted points can be obtained for each target area to form the target convex polygon prediction of the target area. The predicted target convex polygon is converted into a rotated box prediction result by calculating the minimum circumscribed horizontal rectangle of the predicted target convex polygon.
10. A method for detecting rotating targets in SAR images based on a scattering key point guided diffusion model, characterized in that: The method comprises a processor and a memory, wherein the memory is used to store program instructions, and the processor is used to call the stored instructions in the memory to execute the SAR image rotating target detection method based on the scattering key point guided diffusion model as claimed in any one of claims 1 to 9.
Citation Information
Patent Citations
Image deblurring method based on diffusion model
CN116645287A
Target detection method based on random sampling diffusion strategy combined with key point scale modeling
CN117292334A
Magnetic resonance image translation method, system and equipment guided by tumor segmentation and medium
CN118037615A
Target detection method and system in combination with diffusion model and deformable convolution in complex weather
CN118298160A
Cited By
Cross-visual-angle geographic positioning method based on feature center region diffusion and edge radiation
CN119723020A
3D-TEE image left auricle opening plane positioning method based on multi-scale information guide diffusion model
CN119784713A
Cross-sensor SAR (Synthetic Aperture Radar) image target detection method, device, equipment and medium
CN120071024A
Cross-Sensor SAR Image Target Detection Method, Device, Equipment and Medium
CN120071024B
SAR target identification method based on echo domain scattering point collaborative learning
CN120539695A
