A video SAR moving target detection method and device

Abstract

The application provides a video SAR moving target detection method and device, the method comprises the following steps: labeling the video SAR to be trained after frame division, and expanding the data set through data enhancement; performing preliminary feature extraction, inputting the features after preliminary feature extraction into a BiFPN for further feature fusion and extraction; inputting the shallow features output by the BiFPN into a CA, and outputting features that pay more attention to spatial coordinates; inputting the high-level features output by the BiFPN and the features output by the CA into an adaptive feature fusion module after feature fusion, performing adaptive feature fusion on the input features by the adaptive feature fusion module, and performing classification and regression by a detection head; iteratively training a deep neural network to obtain optimal weights; and inputting a video SAR to be detected into the trained deep neural network to output detected moving targets. The application can improve the detection efficiency and accuracy of video SAR moving targets.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of radar image processing, in particular to a video SAR moving target detection method and device. BACKGROUND

[0002] Synthetic Aperture Radar (SAR) is an active earth observation system that can perform high-resolution imaging of various targets at all times and in all weather. Video SAR can continuously observe and image the region of interest, and thus can realize continuous tracking and monitoring of the target.

[0003] For moving targets, Doppler modulation will cause them to shift and defocus in imaging, so that they present irregular shadows in the image. Therefore, moving target detection can be realized by detecting the shadows in the video SAR image. Traditional SAR image processing algorithms usually need to preprocess the image, such as registration, segmentation and extraction, etc. Applying deep neural networks to moving target shadow detection can realize end-to-end shadow detection without complex preprocessing.

[0004] Target detection algorithms based on deep learning mainly include two categories, namely one-stage and two-stage methods. The one-stage method divides the image into SxS grids and calculates the target probability of the grid center falling within each grid. The two-stage method divides the entire detection process into two stages. First, the candidate frame is extracted according to the position of the target in the image, and then classification and regression are performed. The one-stage method is much faster than the two-stage algorithm, and YOLO algorithm is one of the more classic algorithms in this category. However, the existing neural network model has the problem of low precision in moving target detection due to low contrast of video SAR image, coherent speckle noise and other factors. SUMMARY

[0005] In view of the defects in the prior art, the purpose of the present application is to provide a video SAR moving target detection method and device, which can improve the detection efficiency of video SAR moving targets and has higher detection accuracy for moving targets.

[0006] To solve the above problems, the technical scheme of the present application is as follows:

[0007] A video SAR moving target detection method, comprising the following steps:

[0008] Frame the video SAR to be trained and label each frame, and expand the data set by data enhancement;

[0009] Perform preliminary feature extraction, and input the extracted features to BiFPN for further feature fusion and extraction.

[0010] The shallow features output by the BiFPN are input into the CA to output features that pay more attention to spatial coordinates;

[0011] The high-level features output by the BiFPN and the features output by the CA are fused and then input into the adaptive feature fusion module, the adaptive feature fusion module performs adaptive fusion of the input features, and the detection head performs classification and regression;

[0012] The deep neural network is iteratively trained to obtain optimal weights;

[0013] The SAR to be detected is input into the trained deep neural network, and the detected moving target is output.

[0014] Preferably, the step of labeling the frames of the video SAR to be trained and expanding the data set by data augmentation specifically includes: reading the video SAR image to be trained, obtaining the frame rate, width and height of the video, and labeling each frame of the video SAR after framing; using cropping, mirroring, rotation and other augmentation functions to enhance the labeled data set; renaming and storing the augmented images and labels in sequence one by one, and distributing the enhanced data set into a training set and a test set according to a certain proportion.

[0015] Preferably, in the step of performing preliminary feature extraction and inputting the preliminary feature-extracted features into the BiFPN for further feature fusion and extraction, CSPDarknet is used for preliminary feature extraction.

[0016] Preferably, the step of inputting the shallow features output by the BiFPN into the CA to output features that pay more attention to spatial coordinates specifically includes: the CA attention separates the height and width of the input image to encode them respectively, and performs global average pooling on the width and height of the input feature map respectively to obtain 1D feature maps in two directions:

[0017]

[0018] For the input x, use H*1 and 1*W size pooling kernels to perform channel-by-channel encoding along the horizontal coordinate direction and the vertical coordinate direction respectively, superimpose the feature maps in the two directions, and then perform convolution, batch normalization and nonlinearity, and then respectively convolve and activate to multiply with the input x, obtain the attention weight map, up-sample the small scale to ensure that the scales of the two features are the same, and then superimpose them in the channel to output features that pay more attention to spatial coordinates.

[0019] Preferably, the high-level features output by the BiFPN and the features output by the CA are input into the adaptive feature fusion module after feature fusion, and the adaptive feature fusion module performs adaptive fusion of the input features, and the steps of classification and regression of the detection head specifically include: using three decoupled head structures receiving different scale feature layers as the network detection head, using a 1*1 convolution kernel to reduce the channel number in the decoupled head structure, then using convolution, batch normalization and activation blocks, and finally superimposing the obtained values to calculate the coordinates of each grid on the corresponding feature map, creating network coordinate points of the feature map, and projecting the prediction frame obtained by the neural network forward inference onto the original image to obtain the prediction frame.

[0020] Preferably, in the step of iteratively training the deep neural network to obtain optimal weights, before the neural network starts training, a loss function is defined: wherein i is an index value of the training data set, y i is label data, is prediction data, and there are M training data sets.

[0021] Preferably, in the step of iteratively training the deep neural network to obtain optimal weights, the optimizer of the training loss is a function optimization algorithm based on stochastic gradient descent, the gradient corresponding to the weight is calculated for the loss function, the weight value is changed in the opposite direction until the loss function converges to a local minimum, the weight value is updated in each iteration, and the weight calculation formula is: wherein w j is the weight of the jth iteration, w j+1 is the weight of the (j+1)th iteration, lr is the learning rate, L is the loss function, and the weight obtained in each iteration is calculated from the weight of the previous iteration.

[0022] Preferably, in the step of iteratively training the deep neural network to obtain optimal weights, the loss type is the intersection over union (IoU), the IoU is the intersection rate of the generated prediction frame and the real frame, and the IoU formula is: The weight value is saved once in each iteration, and the optimal weight value of the deep neural network is obtained through multiple iterations of training.

[0023] Further, the present application also provides a video SAR moving target detection device, characterized in that the device comprises a processor and a memory for storing executable instructions of the processor, and the processor is configured to execute the video SAR moving target detection method as described above by executing the executable instructions.

[0024] Compared with the prior art, the method of the present application takes YOLOX main network CSPDarknet as the baseline of the network of the present application, and uses BiFPN for further fusion and extraction of features, and part of the output feature layer uses the coordination attention mechanism CA for attention enhancement, and then is fused with the output of BiFPN and input into the adaptive feature fusion module ASFF, and finally three feature layers output from the adaptive feature fusion module are used for classification and regression. The deep neural network designed by the present application is used in video SAR moving target detection, and good results are also obtained in relatively fuzzy video SAR moving target detection; compared with the traditional image processing method, the present application does not need to pre-process the video SAR image, improves the detection efficiency, and compared with the classic deep neural network such as YOLOX and Faster-RCNN, the moving target detection accuracy of the present application is higher. BRIEF DESCRIPTION OF DRAWINGS

[0025] Other features, objects and advantages of the present application will become more apparent from the following detailed description of non-limiting embodiments, made with reference to the accompanying drawings:

[0026] Figure 1 A flow chart of a video SAR moving target detection method provided for an embodiment of the present application is shown in the figure;

[0027] Figure 2 A deep neural network structure diagram in the video SAR moving target detection method provided for an embodiment of the present application is shown in the figure;

[0028] Figure 3 A Fusion structure diagram in the deep neural network structure provided for an embodiment of the present application is shown in the figure;

[0029] Figure 4 A CBS structure diagram in the deep neural network structure provided for an embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0030] The present application will be described in detail below with specific embodiments. The following embodiments will help those skilled in the art to further understand the present application, but do not limit the present application in any form. It should be pointed out that, for those skilled in the art, without departing from the concept of the present application, a number of changes and improvements can be made. These all belong to the protection scope of the present application.

[0031] Specifically, the present application provides a video SAR moving target detection method, as shown in the figure, the method comprises the following steps: Figure 1

[0032] S1: frame the video SAR to be trained and label respectively, and expand the data set by data enhancement;

[0033] ​Specifically, in step S1, the video SAR image to be trained is read, the frame rate, width and height of the video are obtained, and each frame of the video SAR image after framing is labeled; the labeled data set is enhanced using augmentation functions such as cropping, mirroring and rotating; the enhanced images and labels are renamed and stored in sequence one by one, and the enhanced data set is distributed into a training set and a test set according to a certain proportion.

[0034] S2: preliminary feature extraction is performed, and the features after preliminary feature extraction are input into a BiFPN for further feature fusion extraction;

[0035] Specifically, as shown in Figure 2 、 Figure 3 、 Figure 4 The training or test image is input into a CSPDarknet of a deep neural network for preliminary feature extraction, and the output three feature layers are named as “dark3”, “dark4” and “dark5”.

[0036] Further, to improve the detection accuracy of the network, the feature layer after preliminary feature extraction is input into a weighted feature bidirectional pyramid network (BiFPN) for multi-scale feature fusion extraction.

[0037] S3: shallow features output by the BiFPN are input into a CA attention mechanism, and features paying more attention to spatial coordinates are output;

[0038] Specifically, in step S3, the shallow feature layers P3_out and P4_out obtained from the BiFPN are input into the CA attention mechanism.

[0039] The CA attention converts the input 2D coordinates into 1D, that is, the height and width of the image are separated to encode them respectively, and the width and height of the input feature map are globally averaged to obtain 1D feature maps in two directions:

[0040]

[0041] For the input x, the pooling kernels of H*1 and 1*W sizes are used to encode the channels along the horizontal coordinate direction and the vertical coordinate direction respectively, where formula (1) is the output representation of the cth channel with a height of h, and formula (2) is the representation of the cth channel with a width of w. After the superposition of the feature maps in the two directions, convolution, batch normalization and nonlinearity are performed, and then convolution and activation are performed respectively and multiplied with the input x.

[0042] After the attention weight map is obtained by inputting the two feature maps of different sizes into the CA attention mechanism, the smaller one is upsampled to ensure that the scales of the two features are the same, and then the two features are superimposed in the channel to output features paying more attention to spatial coordinates.

[0043] S4: The high-level features output by the BiFPN and the features output by the CA are fused and input to an adaptive feature fusion module, the adaptive feature fusion module performs adaptive fusion of the input features, and then the detection head is used for classification and regression;

[0044] Specifically, in step S4, the high-level feature layer output in step S2 and the feature layer output in step S3 are fused and input to an adaptive feature fusion module (ASFF).

[0045] Specifically, three decoupled head structures receiving different scale feature layers are used as the detection head of the network. In the decoupled head structure, a 1*1 convolution kernel is first used to reduce the channel number, then convolution, batch normalization and activation blocks are used, and finally the obtained values are superimposed to calculate the coordinates of each grid on the corresponding feature map, create the network coordinate points of the feature map, and project the prediction box obtained by the forward inference of the neural network onto the original image to obtain the prediction box.

[0046] When screening the prediction box, it is divided into two steps:

[0047] The first step is the preliminary screening of positive sample prediction boxes, which screens all prediction boxes whose center points are in the real boxes, and then screens the prediction boxes within the square whose real boxes are enlarged by 2.5 times the step size.

[0048] The second step is to use a simplified optimal transport assignment (OTA) algorithm to further screen the prediction boxes.

[0049] S5: Iteratively train the deep neural network to obtain the optimal weight;

[0050] Specifically, in step S5, before training the neural network, a loss function is defined:

[0051]

[0052] where i is the index value of the training data set, y i is the label data, is the prediction data, and there are M training data sets. The training loss optimizer is stochastic gradient descent (SGD), the loss type is Intersection over Union (IoU), the weight is saved once per iteration, and the optimal weight of the deep neural network is obtained after multiple iterations of training.

[0053] SGD is a function optimization algorithm based on stochastic gradient descent, which calculates the gradient corresponding to the weight of the loss function, changes the weight value in the opposite direction until the loss function converges to a local minimum. The weight value is updated in each iteration training, and the weight calculation formula is as follows:

[0054]

[0055] where w j is the weight of the jth iteration, w j+1 is the weight of the j+1th iteration, lr is the learning rate, L is the loss function, and the weight obtained in each iteration training is calculated from the weight of the last iteration.

[0056] IoU is a standard used to measure the accuracy of detecting objects in the dataset when calculating the loss. The IoU formula is as follows:

[0057]

[0058] IoU is the overlap rate of the generated prediction box and the real box, that is, the ratio of their intersection (Area of Overlap) to union (Area of Union).

[0059] S6: input the SAR video to be detected into the trained deep neural network, and output the detected moving target.

[0060] Specifically, in step S6, each frame image of the SAR video to be detected is input into the trained deep neural network, and the detected moving target is obtained.

[0061] Compared with the prior art, the method of the present application uses YOLOX main network CSPDarknet as the baseline of the network of the present application, and uses BiFPN for further fusion and extraction of features. Part of the output feature layer uses the coordination attention mechanism CA for attention enhancement, and then the output of BiFPN is fused and input into the adaptive feature fusion module ASFF. Finally, three feature layers output from the adaptive feature fusion module are used for classification and regression. The deep neural network designed in the present application is used in video SAR moving target detection, and good results are achieved in relatively fuzzy video SAR moving target detection. Compared with the traditional image processing method, the present application does not need to pre-process the video SAR image, improves the detection efficiency, and compared with the classic deep neural network such as YOLOX and Faster-RCNN, the moving target detection accuracy of the present application is higher.

[0062] The specific embodiments of the present application are described above. It needs to be understood that the present application is not limited to the specific embodiments described above, and various changes or modifications can be made by those skilled in the art within the scope of the claims, which does not affect the essential content of the present application. The embodiments of the present application and the features in the embodiments can be combined with each other at will without conflict.

Claims

1. A video SAR moving target detection method, characterized in that: The method comprises the following steps: The video SAR to be trained is divided into frames and labeled separately, and the dataset is expanded through data enhancement; Perform preliminary feature extraction and input the features after preliminary feature extraction into BiFPN for further feature fusion extraction; The shallow features output by BiFPN are input into CA, and the output focuses more on the features of spatial coordinates; The high-level features output by BiFPN and the features output by CA are fused and then input into the adaptive feature fusion module. The adaptive feature fusion module performs adaptive fusion on the input features, and the detection head performs classification and regression. Specifically, it includes: using three decoupling head structures that receive feature layers of different scales as the detection head of the network, first using a 1*1 convolution kernel in the decoupling head structure to reduce the number of channels, then using convolution, batch normalization and activation blocks, and finally superimposing the obtained values ​​to calculate the coordinates of each grid on the corresponding feature map, creating the network coordinate points of the feature map, and projecting the prediction box obtained by the forward reasoning of the neural network onto the original image to obtain the prediction box; Iteratively train the deep neural network to obtain the optimal weights; The video SAR to be detected is input into the trained deep neural network, and the detected moving target is output.

2. The video SAR moving target detection method according to claim 1, characterized in that: The steps of dividing the video SAR to be trained into frames and labeling them separately, and expanding the data set by data augmentation specifically include: reading the video SAR image to be trained, obtaining the frame rate, width and height of the video, and labeling each frame of the video SAR after framing; enhancing the labeled data set using augmentation functions such as cropping, mirroring, and rotation; renaming and storing the augmented images and labels in a one-to-one correspondence in sequence, and allocating the augmented data set into a training set and a test set according to a certain ratio.

3. The video SAR moving target detection method according to claim 1, characterized in that: The preliminary feature extraction is performed, and the features after the preliminary feature extraction are input into the BiFPN for further feature fusion extraction, and the preliminary features are performed using CSPDarknet.

4. The video SAR moving target detection method according to claim 1, wherein: The steps of inputting the shallow features output by BiFPN into CA and outputting features that focus more on spatial coordinates specifically include: CA attention separates the height and width of the input image and encodes them separately, and globally average pools the width and height of the input feature map to obtain 1D feature maps in two directions: For the input x, use H*1 and 1*W pooling kernels to encode each channel along the width and height directions respectively. After the feature maps in the two directions are superimposed, they are convolved, batch normalized and nonlinear, and then convolved and activated respectively before multiplying with the input x. After obtaining the attention weight map, the smaller scale is upsampled to ensure that the scales of the two features are the same and then superimposed on the channel to output features that focus more on spatial coordinates.

5. The video SAR moving target detection method according to claim 1, characterized in that: In the step of iteratively training the deep neural network to obtain the optimal weight, before starting to train the neural network, a loss function is defined: Among them, i is the index value of the training data set, y i is the label data, Is the prediction data, there are M training data sets.

6. The video SAR moving target detection method according to claim 5, characterized in that: In the step of iteratively training the deep neural network to obtain the optimal weight, the optimizer of the training loss is a function optimization algorithm based on stochastic gradient descent, which calculates the gradient corresponding to the weight of the loss function and changes the weight in the opposite direction until the loss function converges to a local minimum. The weight value is updated in each iterative training. The weight calculation formula is: Among them, w j is the weight of the jth iteration, w j+1 is the weight of the j+1th iteration, lr is the learning rate, L is the loss function, and the weight obtained in each iterative training is calculated based on the weight of the previous one.

7. The video SAR moving target detection method according to claim 6, characterized in that: In the step of iteratively training the deep neural network to obtain the optimal weight, the loss type is the intersection over union (IoU), where IoU is the overlap ratio between the generated prediction box and the true box. The IoU formula is: The weights are saved once in each iteration, and the training iterations are repeated multiple times to obtain the optimal weights of the deep neural network.

8. A video SAR moving target detection device, characterized in that: The apparatus includes a processor and a memory for storing executable instructions of the processor, wherein the processor is configured to perform the video SAR moving target detection method according to any one of claims 1 to 7 by executing the executable instructions.

Images