A video SAR combined motion target detection method

Abstract

The application discloses a video SAR combined moving target detection method, solves the problem of low contrast and fuzzy boundary of the shadow of the fast-moving target and the background in the prior art, and comprises the following steps: decomposing a moving target in a video SAR image from the background to obtain an initial moving target image and a sparse image; using a double-flow feature extraction network to perform feature extraction and fusion on the video SAR image and the sparse image to obtain a classification probability of the moving target; the double-flow feature extraction network comprises two single-image feature extraction modules, a double-image feature fusion module and a multi-feature output module; each of the two single-image feature extraction modules comprises a first network block, a second network block, a third network block and a fourth network block connected in sequence; the double-image feature fusion module comprises a first fusion network block, a second fusion network block and a third fusion network block; and the multi-feature output module comprises an FPN block; and the method effectively suppresses background noise.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of radar detection technology, and particularly relates to a video SAR joint moving target detection method. BACKGROUND

[0002] Video SAR is a special synthetic aperture radar system, which can generate a series of synthetic aperture radar images like a video. It can not only work in all-weather and all-day conditions like traditional synthetic aperture radar systems, but also is not affected by the adverse effects caused by the imaging scene. In addition, it can monitor the region of interest for a long time at a high frame rate. Therefore, video synthetic aperture radar has been applied to many fields, including scene monitoring, ground moving target indication (GMTI) and coherent change detection (CCD). At present, video SAR has become a research hotspot in the field of radar moving target detection at home and abroad.

[0003] In "Video SAR moving target indication using deep neural network" (IEEE Transactions on Geoscience and Remote Sensing, vol. 58, pp. 7194-7204, Jun. 2017), Ding Jinshan and others of Xi'an University of Electronic Science and Technology proposed a shadow-assisted moving target detection framework based on a convolutional neural network. In the framework, Faster R-CNN is used for moving target detection, and a density-based clustering algorithm and a bidirectional long short-term memory (Bi-LSTM) network are used to reduce false alarms and missed detections.

[0004] Huang introduced the low-rank and sparse decomposition (LSD) technique into the multichannel SAR ground moving target indication (SAR GMTI) in "Moving target detection via efficient ATI-GoDec approach for multichannel SAR system" (IEEE Geoscience and Remote Sensing Letters, vol. 13, pp. 1320-1324, Sep. 2016) and "GMTI and parameter estimation for MIMO SAR system via fast interferometry RPCA method" (IEEE Transactions on Geoscience and Remote Sensing, vol. 56, no. 9, pp. 1774-1787, Mar. 2018). Compared with the traditional ground moving target indication algorithm, the low-rank and sparse decomposition (LSD) technique can separate the target from the background, that is, the target is enhanced while the background is suppressed, thereby significantly improving the detection performance of the dim target. Therefore, the low-rank and sparse decomposition has been applied to the detection and tracking tasks in the video synthetic aperture radar image.

[0005] However, the moving target shadow is usually a dark target with low intensity and simple features, which is not obviously distinguished from other weak reflectivity areas in the video SAR image. In addition, the generation of the shadow is highly dependent on the movement of the moving target, and in particular for the fast-moving target, the contrast of the shadow with the background is low and the boundary is blurred. Therefore, these CNN-based methods still produce a large number of missed detections and false alarms by using only the target shadow in the video SAR image. SUMMARY

[0006] The present application solves the problem of low contrast and blurred boundary of the shadow with the background for the fast-moving target in the prior art by providing a video SAR joint moving target detection method, and effectively suppresses the background noise.

[0007] The present application provides a video SAR joint moving target detection method, which comprises:

[0008] The moving target in the video SAR image is decomposed from the background by using a Faster RCNN method to obtain an initial moving target image, and a sparse image is obtained according to the initial moving target image;

[0009] A dual-stream feature extraction network is used to perform feature extraction and fusion on the video SAR image and the sparse image to obtain a classification probability of the moving target; wherein the dual-stream feature extraction network comprises two single-image feature extraction modules, a dual-image feature fusion module and a multi-feature output module; the two single-image feature extraction modules are used to perform feature extraction on the video SAR image and the sparse image respectively to obtain a second video SAR feature map and a second target sparse feature map, a third video SAR feature map and a third target sparse feature map, and a fourth video SAR feature map and a fourth target sparse feature map; the dual-image feature fusion module is used to perform feature fusion on the second video SAR feature map and the second target sparse feature map, the third video SAR feature map and the third target sparse feature map, and the fourth video SAR feature map and the fourth target sparse feature map respectively to obtain a first fusion feature, a second fusion feature and a third fusion feature; and the multi-feature output module is used to use the FPN block to predict the first fusion feature, the second fusion feature and the third fusion feature to obtain a classification probability of the moving target.

[0010] In a possible implementation, the Faster RCNN method is used to decompose the moving target in the video SAR image from the background to obtain an initial moving target image, comprising:

[0011] A SAR image sequence corresponding to the video SAR image is determined, and each frame of SAR image in the SAR image sequence is divided into a background and a target shadow;

[0012] An optimization constraint model of the video SAR image is constructed according to the background and the target shadow of each frame of SAR image, and the optimization constraint model is converged to obtain an initial moving target image.

[0013] In a possible implementation, each frame of SAR image in the SAR image sequence is divided into a background and a target shadow, and is expressed as:

[0014] ;

[0015] wherein, represents the background in the SAR image sequence; represents the target shadow in the SAR image sequence.

[0016] In a possible implementation, the optimization constraint model is expressed as:

[0017] ;

[0018] wherein, denotes the rank of a matrix; denotes a normalization factor for balancing the two terms of the objective function; denotes a sequence of SAR images; denotes a background in the sequence of SAR images; denotes a target shadow in the sequence of SAR images; denotes a norm.

[0019] In a possible implementation, the two single-image feature extraction modules are configured to perform feature extraction on the video SAR image and the sparse image respectively, to obtain a second video SAR feature map and a second target sparse feature map, a third video SAR feature map and a third target sparse feature map, and a fourth video SAR feature map and a fourth target sparse feature map, including:

[0020] The two single-image feature extraction modules each include a first network block, a second network block, a third network block, and a fourth network block connected in sequence.

[0021] The first network block includes a spatial contraction layer, a first convolutional layer, and a second convolutional layer connected in sequence, and is configured to convert the video SAR image and the sparse image into a first video SAR feature map and a first target sparse feature map respectively by using the first network block.

[0022] The second network block includes a third convolutional layer and a fourth convolutional layer connected in sequence, and is configured to convert the channel dimension numbers of the first video SAR feature map and the first target sparse feature map respectively by using the second network block, to obtain a second video SAR feature map and a second target sparse feature map; wherein the channel dimension numbers of the second video SAR feature map and the second target sparse feature map are half of the channel dimension numbers of the first video SAR feature map and the first target sparse feature map.

[0023] The third network block includes a fifth convolutional layer and a sixth convolutional layer connected in sequence, and is configured to convert the channel dimension numbers of the second video SAR feature map and the second target sparse feature map respectively by using the third network block, to obtain a third video SAR feature map and a third target sparse feature map; wherein the channel dimension numbers of the third video SAR feature map and the third target sparse feature map are half of the channel dimension numbers of the second video SAR feature map and the second target sparse feature map.

[0024] The fourth network block comprises a seventh convolutional layer, a spatial pyramid pooling layer and an eighth convolutional layer connected in sequence; the channel dimension numbers of the third video SAR feature map and the third target sparse feature map are converted respectively by using the fourth network block to obtain a fourth video SAR feature map and a fourth target sparse feature map; wherein the channel dimension numbers of the fourth video SAR feature map and the fourth target sparse feature map are half of the channel dimension numbers of the third video SAR feature map and the third target sparse feature map.

[0025] In a possible implementation, the dual-image feature fusion module comprises a first fusion network block, a second fusion network block and a third fusion network block; the first fusion network block is arranged between the second network block and the third network block; the second fusion network block is arranged between the third network block and the fourth network block; and the third fusion network block is arranged after the fourth network block.

[0026] In a possible implementation, the first fusion network block comprises a first channel adding unit and a first FFM unit connected in sequence;

[0027] The second fusion network block comprises a second channel adding unit and a second FFM unit connected in sequence;

[0028] The third fusion network block comprises a third channel adding unit and a third FFM unit connected in sequence.

[0029] In a possible implementation, the first fusion network block is configured to respectively flatten the second video SAR feature map and the second target sparse feature map by the first channel adding unit and add position encoding to obtain a second video SAR encoded feature map and a second target sparse encoded feature map respectively; the second video SAR encoded feature map and the second target sparse encoded feature map are input into the first FFM unit to obtain a first projected video SAR feature map and a first projected target sparse feature map correspondingly; and a first fusion feature is obtained according to the first projected video SAR feature map and the first projected target sparse feature map.

[0030] The second fusion network block inputs the first projected video SAR feature map and the first projected target sparse feature map and outputs a second projected video SAR feature map and a second projected target sparse feature map; and a second fusion feature is obtained according to the second projected video SAR feature map and the second projected target sparse feature map.

[0031] The third fusion network block inputs the second projected video SAR feature map and the second projected target sparse feature map, and outputs a third projected video SAR feature map and a third projected target sparse feature map; and third fusion features are obtained according to the third projected video SAR feature map and the third projected target sparse feature map.

[0032] The one or more technical solutions provided in the application have at least the following technical effects or advantages:

[0033] The application first separates the moving target from the background by using low-rank sparse decomposition to generate a sparse image containing the moving target, effectively suppresses the background noise, and uses a double-flow feature extraction network to fuse the video SAR image and the sparse image, enhances the feature extraction capability, and thus improves the detection performance of the moving target. BRIEF DESCRIPTION OF DRAWINGS

[0034] Figure 1 A step flowchart of the video SAR joint moving target detection method provided for the embodiment of the application is provided.

[0035] Figure 2 A statistical result schematic diagram of the video SAR image provided for the embodiment of the application is provided.

[0036] Figure 3 A target shadow map in the video SAR image provided for the embodiment of the application is provided.

[0037] Figure 4 A feature fusion module framework diagram provided for the embodiment of the application is provided.

[0038] Figure 5 A structure diagram of a spatial pyramid pooling layer provided for the embodiment of the application is provided.

[0039] Figure 6 A structure diagram of a feature fusion module provided for the embodiment of the application is provided.

[0040] Figure 7 A multi-feature output module structure diagram provided for the embodiment of the application is provided.

[0041] Figure 8 A change curve schematic diagram of a training loss with an epoch provided for the embodiment of the application is provided.

[0042] Figure 9 A detection result schematic diagram of the method proposed in the application and other methods on the data set at this time provided for the embodiment of the application is provided.

[0043] Figure 10 An mAP schematic diagram of all methods in each target category of the method proposed in the application provided for the embodiment of the application is provided.

[0044] Figure 11 The detection result diagram of five representative test samples provided for the embodiment of the present application;

[0045] Figure 12 The indicative result of each test sample in a plurality of methods provided for the embodiment of the present application is intended to;

[0046] Figure 13 The feature fusion result diagram of four representative test samples provided for the embodiment of the present application;

[0047] Figure 14 The comparison result diagram of the detection result and the threshold of the prediction score provided for the embodiment of the present application;

[0048] Figure 15 The real-time performance comparison result diagram of four methods provided for the embodiment of the present application. DETAILED DESCRIPTION

[0049] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the protection scope of the present application.

[0050] The present application provides a video SAR joint moving target detection method, as shown in the figure, the method comprises the following steps S101-S102. Figure 1

[0051] S101, the moving target in the video SAR image is decomposed from the background by using the Faster RCNN method, and the initial moving target image is obtained, and the sparse image is obtained according to the initial moving target image;

[0052] Specifically, in step S101, the moving target in the video SAR image is decomposed from the background by using the Faster RCNN method, and the initial moving target image is obtained, comprising the following steps S1011-S1012.

[0053] S1011, determine the SAR image sequence corresponding to the video SAR image, and perform format conversion according to the SAR image sequence to obtain that each frame SAR image in the SAR image sequence is divided into background and target shadow;

[0054] Here, the SAR image sequence in the video SAR image can be expressed as:

[0055] (1);

[0056] wherein,​ Represents a SAR image sequence. Indicates the first Frame SAR image, This represents the width and height of each SAR image frame. This indicates the length of the SAR image sequence, i.e., the number of frames. Each SAR image frame can be divided into two parts: background and target shadow, i.e.:

[0057] (2);

[0058] in, This represents the background in the k-th frame of the SAR image. Let represent the target shadow in the k-th frame of the SAR image. Then the SAR image sequence can be re-represented as:

[0059] (3);

[0060] In SAR image sequences, Represents the background in a SAR image sequence; This represents the target shadow in a SAR image sequence.

[0061] For example, compared to the background, the target shadow occupies only a few pixels in a SAR image because it is limited in number and small in size. Figure 2 As shown, Figure 2 In the upper left part of the image, the pixel value of the target shadow is approximately 27. The target shadow only occupies a small portion of the entire SAR image, such as... Figure 3 As shown. Therefore, It exhibits sparsity, while This attribute is even more pronounced in the middle.

[0062] Because video SAR systems typically monitor a specific region of interest for extended periods, the background in a SAR image sequence is often static, meaning the backgrounds in the sequence are almost identical. In other words, if the frames in a SAR image sequence have been pre-registered, the backgrounds will exhibit high similarity and correlation. From the matrix... From this perspective, due to the high similarity between the backgrounds of multiple frames, they can be projected into the same feature subspace. Therefore, the matrix It possesses a low-rank property. To visualize this property, singular value decomposition is used to analyze the distribution of the rank, and the low rank is evaluated using the cumulative distribution function (CDF) to analyze spatial structure information.

[0063] (4);

[0064] in, denotes a diagonal matrix composed of all singular values. denotes the large singular value, denotes the number of singular values, denotes the number of singular values. Figure 2 (c) shows the CDF of the matrix with different rank distribution. Using multi-frame image for CDF analysis, it can be seen that the CDF of video SAR data is more similar to rank-one matrix, as shown in (c). In addition, the distribution of singular values is concentrated, rather than uniform distribution in full rank matrix, and the above results show that the background in SAR image sequence is low rank. Figure 2

[0065] S1012, an optimization constraint model of the video SAR image is constructed, and the optimization constraint model is converged to obtain an initial moving target image.

[0066] Here, in order to obtain the target shadow, a constraint optimization model as shown below is constructed:

[0067] (5);

[0068] wherein, denotes the rank of the matrix; denotes a normalization factor, used to balance the two terms of the objective function; denotes a SAR image sequence; denotes the background in the SAR image sequence; denotes the target shadow in the SAR image sequence; denotes the norm.

[0069] Generally, it is solved by thresholding and Alternating Direction Method of Multipliers (ADMM). However, the optimization process of the former is not smooth, and the convergence speed is relatively slow, so the latter is selected as the solving method in the embodiment of the application. Figure 3 is the decomposition result of the video SAR image. It can be seen that the target shadow in the sparse image is enhanced, and the shadow of non-target is eliminated. Obviously, if the target shadow is detected in the sparse image instead of the SAR image, the false alarm will be reduced. However, the LSD may cause the size distortion of the target shadow in the sparse image, resulting in false size detection results.

[0070] S102, using a double-flow feature extraction network, feature extraction and fusion are performed on the video SAR image and the sparse image to obtain a classification probability of the moving target.

[0071] ​As shown in Figure 4 , the dual-flow feature extraction network comprises two single-image feature extraction modules, a dual-image feature fusion module and a multi-feature output module. Specifically, the parameters of the dual-flow feature extraction network are shown in Table 1:

[0072] Table 1 Structure parameter table of dual-flow feature extraction network

[0073]

[0074] Both single-image feature extraction modules comprise a first network block (L1), a second network block (L2), a third network block (L3) and a fourth network block (L4) connected in sequence.

[0075] Here, the first fusion network block (T1) is arranged between the second network block (L2) and the third network block (L3); the second fusion network block (T2) is arranged between the third network block (L3) and the fourth network block (L4); and the third fusion network block (T3) is arranged after the fourth network block (L4).

[0076] Both single-image feature extraction modules are used to extract features from the video SAR image and the sparse image respectively, to obtain the second video SAR feature map and the second target sparse feature map, the third video SAR feature map and the third target sparse feature map, and the fourth video SAR feature map and the fourth target sparse feature map, respectively.

[0077] Specifically, both single-image feature extraction modules are used to extract features from the video SAR image and the sparse image respectively, to obtain the second video SAR feature map and the second target sparse feature map, the third video SAR feature map and the third target sparse feature map, and the fourth video SAR feature map and the fourth target sparse feature map, respectively, including:

[0078] The first network block uses a spatial shrinkage layer SSL, a first convolutional layer Conv1 and a second convolutional layer Conv3 connected in sequence to convert the video SAR image and the sparse image into a first video SAR feature map and a first target sparse feature map , respectively.

[0079] The second network block uses a third convolutional layer Conv1 and a fourth convolutional layer Conv3 connected in sequence to convert the channel dimension number of the first video SAR feature map and the first target sparse feature map , respectively, to obtain a second video SAR feature map and a second target sparse feature map ; wherein the second video SAR feature map and the second target sparse feature map a half of a channel dimension number of the second video SAR feature map and the second target sparse feature map ; wherein a channel dimension number of the third video SAR feature map and the third target sparse feature map

[0080] is a half of a channel dimension number of the second video SAR feature map and the second target sparse feature map . a half of a channel dimension number of the second video SAR feature map and the second target sparse feature map .

[0081] the fourth video SAR feature map and the fourth target sparse feature map ; wherein a channel dimension number of the fourth video SAR feature map and the fourth target sparse feature map is a half of a channel dimension number of the third video SAR feature map and the third target sparse feature map .

[0082] The dual-image feature fusion module comprises a first fusion network block (T1), a second fusion network block (T2), and a third fusion network block (T3).

[0083] Here, the first fusion network block (T1) comprises a first channel adding unit Add1 and a first FFM unit FFM1 connected in sequence.

[0084] The second fusion network block (T2) comprises a second channel adding unit Add2 and a second FFM unit FFM2 connected in sequence.

[0085] The third fusion network block (T3) comprises a third channel adding unit Add3 and a third FFM unit FFM3 connected in sequence.

[0086] ​​The dual-image feature fusion module is configured to fuse the second video SAR feature map and the second target sparse feature map, the third video SAR feature map and the third target sparse feature map, and the fourth video SAR feature map and the fourth target sparse feature map, to obtain a first fusion feature F1, a second fusion feature F2 and a third fusion feature F3.

[0087] Specifically, the first fusion network block (T1) is configured to respectively flatten the second video SAR feature map and the second target sparse feature map by the first channel adding unit Add1, and add position encoding, to obtain a second video SAR encoded feature map and a second target sparse encoded feature map respectively; the second video SAR encoded feature map and the second target sparse encoded feature map are input into the first FFM unit FFM1, to obtain a first projected video SAR feature map and a first projected target sparse feature map correspondingly; the first fusion feature F1 is obtained according to the first projected video SAR feature map and the first projected target sparse feature map.

[0088] Similarly to the first fusion network block (T1): the input of the second fusion network block (T1) is the first projected video SAR feature map and the first projected target sparse feature map, and the output is a second projected video SAR feature map and a second projected target sparse feature map; the second fusion feature F2 is obtained according to the second projected video SAR feature map and the second projected target sparse feature map; the input of the third fusion network block (T1) is the second projected video SAR feature map and the second projected target sparse feature map, and the output is a third projected video SAR feature map and a third projected target sparse feature map; the third fusion feature F3 is obtained according to the third projected video SAR feature map and the third projected target sparse feature map.

[0089] The multi-feature output module comprises an FPN block; the multi-feature output module is configured to predict the first fusion feature F1, the second fusion feature F2 and the third fusion feature F3 by using the FPN block, to obtain a classification probability of the moving target.

[0090] Exemplarily, the dual-stream feature extraction network is specifically divided into three steps: single-image feature extraction, dual-image feature fusion and multi-feature output. In the proposed dual-stream feature extraction network, single-image feature extraction is first realized by two shallow-to-deep network branches, and features are extracted from video SAR images and sparse images independently. Then, after each feature scale is reduced, the dual-image feature fusion module utilizes the proposed FFM to interact and aggregate features from different branches. Finally, in the multi-feature output module, a feature pyramid network (FPN) is introduced to generate multi-level feature maps. The three steps will be described in detail below.

[0091] (1) Single-image feature extraction module

[0092] The implementation of single-image feature extraction is the same as other CNN-based feature extraction networks such as VGGnet and ResNet, that is, two branches from shallow to deep are used for feature extraction of video SAR images and sparse images respectively. However, compared with traditional feature extraction networks, this method first designs a spatial shrinking level (SSL) to reduce the dimension of the feature map in each network branch. Specifically, the spatial information of the feature map is converted into the channel dimension by reshaping the dimension of the feature map, and then the channel dimension is compressed using a 3x3 convolution layer.

[0093] (6);

[0094] wherein, represents the feature map, represents the output feature map compressed by SSL, represents the reshaping operation, represents the 3x3 convolution layer.

[0095] In addition, in order to improve the feature extraction capability of each network branch for video SAR images and sparse images, a spatial pyramid pooling network (SPP) is introduced. As shown in Figure 5 The structure of SPP is shown in FIG. 4.

[0096] SPP consists of four parallel branches: maximum pooling layers with window sizes of 5x5, 9x9 and 13x13, and a skip connection.

[0097] Due to the stride of 1, SPP performs sliding pooling operation on the input feature map by using multiple maximum pooling layers with different window sizes, and then integrates the pooled features by channel concatenation and a 1x1 convolution layer. SPP extracts and fuses features of different scales, thereby enriching the representation capability of the feature map.

[0098] (7);

[0099] wherein, represents the input feature map, represents the output feature map compressed by SPP, represents the channel concatenation operation, represents the 1x1 convolution layer, represents the maximum pooling layer with a pooling window size of . Wherein represents that the feature map does not perform maximum pooling operation, but performs skip connection.

[0100] (2) Double-image feature fusion module

[0101] Due to the significant performance of the Transformer in various types of tasks, a Transformer-based module—FFM is proposed in this paper to fuse the target features extracted from video SAR images and sparse images.

[0102] As Figure 6 shown, denotes the second video SAR feature map, and denotes the second target sparse feature map, where denotes the length of the feature map, denotes the width of the feature map, denotes the number of channels.

[0103] First, the second video SAR feature map and the second target sparse feature map are flattened, and the and order of the feature matrix are arranged, and then and are obtained, which solves the problem that the FFM based on the Transformer cannot process two-dimensional data.

[0104] Then, the and are spliced, and a position encoding with a size of 2HW×C trainable parameters is added to generate the input of the FFM. The position encoding allows the FFM to capture the spatial information of each token in the sentence. Thereafter, is projected into three separate weight matrices to generate a set of tensors , and .

[0105] (8);

[0106] where denotes and , , and are the projection weight matrices, in the FFM, , and are equal and . The attention weight is calculated by the normalized dot product between and :

[0107] (9);

[0108] where, denotes the transpose transformation, denotes the softmax function that maps each attention weight to [0, 1], denotes the scaling factor used in the training phase to reduce the variance of dot product size and improve the stability of gradient update, and are block matrices. The refined results are obtained by multiplying the attention weights and values.

[0109] (10);

[0110] In addition, the multi-head attention mechanism is adopted to better capture multiple relationships and subtle differences between each token of the sentence.

[0111] (11);

[0112] where, denotes the number of multi-head, in FFM, and denote the weight matrix of projection . Then, a Multilayer Perceptron (MLP) with two fully connected layers and GELU activation function is used to obtain an output with the same size as the input .

[0113] (12);

[0114] where, , denotes the Multilayer Perceptron, denotes the fully connected layer, denotes the GELU activation function.

[0115] Finally, the output is reconstructed according to the shape of and , is further divided into and , the shape is reconstructed from to , and the final outputs and are obtained. When using FFM, it can be seen that the attention weight can be naturally divided into four matrix blocks as shown in equation (10), where and are intra-image attention weights, and ​​is the inter-image attention weight. Therefore, the FFM proposed in the embodiments of the present application can not only pay attention to the position of the target in the feature map, but also interact and fuse the feature maps extracted from the video SAR image and the sparse image.

[0116] The same method as described above is adopted for the second fusion network block (T2) and the third fusion network block (T3) to obtain the corresponding .

[0117] (3) Multi-feature output module

[0118] As shown in Figure 7 , the target shadow has different scales and can be divided into three categories: small targets with a minimum bounding rectangle (MBR) area less than 64, intermediate targets with an MBR area less than 256 and greater than 64, and large targets with an MBR greater than 256. As is known, smaller targets are particularly prone to be lost in deeper feature maps, so using only the deepest feature map to predict targets can lead to missed detection of smaller targets.

[0119] Therefore, a FPN is introduced to address this problem, in which different size target shadows are detected on different levels of feature maps.

[0120] Specifically, in the RPN, a plurality of anchors of appropriate size are first set, and then these anchors are assigned to different feature maps according to their MBR areas.

[0121] For example, if the MBR area of an anchor is less than 64, it will be used for feature map P1 and classified and regressed by using the ground truth with the same MBR area range. Through this setting, target shadows of different sizes can be predicted by different feature maps.

[0122] The dual-stream feature extraction network effectively utilizes and fuses target information from the two images to produce more discriminative target features. Therefore, in the subsequent detection process, the target shadow can be more effectively predicted and classified.

[0123] The following is a simulation embodiment provided by the present application.

[0124] 1. Simulation conditions:

[0125] The simulation of the present application is carried out in the software environment of MATLAB R2020a and VScode.

[0126] 2. Simulation content:

[0127] Real data collected by the lynx video SAR system at Sandia National Laboratories is used to validate the proposed method. The real data is currently the only publicly available video SAR data containing targets. Table 1 is the parameters of the video SAR system. The video SAR images in the real data rotate with the rotation of the imaging coordinate system as the video SAR system adopts spotlight imaging mode and moves along a circular trajectory. The real data consists of 899 consecutive SAR images, which are resized to 256 x 256 pixels. In this paper, each SAR image is resized to 128 x 128 pixels. In addition, the images are first filtered using a mean filter.

[0128] Table 1. Parameters of the video SAR system

[0129]

[0130] (1) Training dataset: The last 700 SAR images in the real data are selected to build the training set. To expand the training set, each SAR image is rotated by 0°, 45°, 90°, 135°, 180°, 225°, 270°, and 315° and randomly cropped. The targets are labeled in each SAR image to generate the ground truth labels. There are 10 4 training samples in total for training the proposed method.

[0131] (2) Testing dataset: The first 199 SAR images in the real data are selected to build the testing dataset. Similarly, to expand the testing dataset, each SAR image is rotated by 0°, 45°, 90°, 135°, 180°, 225°, 270°, and 315° but not randomly cropped. Similarly, there are 10 3 testing samples in total for testing the proposed method. Each testing sample consists of a SAR image and its artificially made ground truth. There are 1447 targets in the testing dataset.

[0132] In the training stage, the combination of Stochastic Gradient Descent (SGD) and 0.9 Momentum is adopted as the optimization strategy. The number of epochs is set to 100 and the batch size is selected to be 8 to prevent running out of memory. The proposed method follows the classic Faster RCNN framework, so it also adopts the same training loss consisting of RPN loss and RCNN loss. The change of the training loss with the epoch is shown as Figure 8

[0133] ​​​In this paper, three evaluation indicators are adopted to verify the detection performance of the proposed method, including False Positive (FP), False Negative (FN) and mean Average Precision (mAP). Among them, false positive refers to the error identification of non-target as target, i.e. false alarm; false negative refers to the target that actually exists but is not detected, i.e. missed detection; the average precision mean is a comprehensive index, which can be defined as:

[0134] (13)

[0135] wherein, is the number of target categories, represents the average precision of the target in the category , TP represents the true positive that meets the Intersection Over Union (IOU) threshold of the detection box and the real label, in this paper, the detection threshold is 0.6 and the IOU threshold is 0.5. Note that the detection performance is negatively correlated with FP and FN, but positively correlated with mAP.

[0136] In order to verify the detection performance of the proposed method, we compared it with four classical detection methods, SSD, YOLOv3-SPP, Faster RCNN and Faster RCNN+FPN, among which SSD and YOLOv3-SPP belong to one-stage detection method, and the other methods are two-stage detection methods. As Figure 9 shown in the detection results of the proposed method and other comparison methods on the test data set.

[0137] In the proposed method, FFM interacts and fuses the features extracted from the original video SAR image with the sparse image, which improves the final detection performance. In order to intuitively explain the working mechanism of FFM to enhance the detection performance, four test samples are selected to show the changes of their features after using FFM. As Figure 11 shown, first, we can see that all the attention weights of the four test samples are non-zero matrices and four block matrices, and they are also non-zero matrices in each attention weight, which means not only the features within the image are interacted and fused (and), but also the features between the images are also done (and). In addition, it can be seen that the target features (yellow box) in the feature map are blurred, and worse, the background similar to the target (green box) shows more obvious features before FFM is involved in feature extraction (see Figure 11a(3)-d(3)). It is foreseeable that if these features are only used for RPN and RCNN, the final detection performance will certainly be unsatisfactory. Nevertheless, the target features exhibit the most discriminative representation in the feature maps while suppressing features of the background similar to the target after FFM is used for feature extraction (see Figure 11 a(4)-d(4)). Therefore, the proposed method can achieve excellent detection performance.

[0138] 3. Simulation effect analysis:

[0139] As shown in Figure 8 , the curve of the training loss of the present application with epoch is shown, where is the RPN loss, is the RCNN loss, is the total loss, it can be seen that and in the initial epoch sharply decrease, while at the end of training tend to be stable, so that the total loss gradually converges;

[0140] As shown in Figure 9 , the detection results of the proposed method and other comparative methods on the test data set are shown. And for each indicator, the best result is in bold, the lower arrow indicates that the detection performance is negatively correlated with the score, and the upper arrow is the opposite. Obviously, the proposed method obtains the optimal effect in all indicators, among which the highest score is obtained in mAP, and the lowest score is obtained in FN and FP. Compared with one-stage methods (SSD and YOLOv3-SPP), two-stage methods (Faster RCNN and Faster RCNN+FPN) show lower FN but higher FP. The reason is that the two-stage method uses a large number of grids generated by other networks to further classify and regress the target, which ensures fewer missed detections but increases FP. By observing the detection indicators of the last three methods, we can see that FPN not only greatly reduces FN and FP, but also improves mAP, which illustrates the effectiveness of FPN. In addition, the proposed method further improves the detection performance, indicating that the dual-stream structure equipped with FFM helps target detection.

[0141] As shown in Figure 10The mAP of all methods in each target category is shown, and the proposed method still achieves the best score in all target categories, obtaining 0.952 in the car category and 0.707 in the truck category. In addition, the scores of almost all methods in the car category are higher than those in the truck category. There are two reasons for this phenomenon. One is that the training samples of the car and truck categories in the training set are imbalanced. The fewer training samples of the truck category lead to the fact that the detection network is not well trained, resulting in relatively poor performance of the truck category. The other is that these detection networks have a large number of network parameters, so a large number of training samples must be used for optimization. Compared with other methods, the proposed method obtains similar detection performance in the two target categories because SSD is equipped with fewer network parameters and predicts targets in multiple feature layers of different scales. Although the proposed method uses FPN to predict targets of different scales, the detection performance is still imbalanced in the two categories due to the large number of network parameters of the Transformer structure, but the proposed method still shows the best detection performance.

[0142] As shown in Figure 11 , the detection results of all methods for five representative test samples are shown, so that the detection performance of the proposed method can be observed intuitively. In each subgraph, the detection results obtained by SSD, YOLOv3-SPP, Faster RCNN, Faster RCNN+FPN and the proposed method are displayed from left to right. Obviously, compared with other methods, the proposed method detects more targets and has no false positives for each test sample. By comparing the detection results of the last three methods with those of the first two methods, it can be seen that the two-stage method obviously misses less but produces more false alarms, which intuitively proves the conclusion obtained from the quantitative detection results.

[0143] As shown in Figure 12 , the detection results of two indicators obtained by all methods for each test sample are given. By observing the detection results of Faster RCNN and Faster RCNN+FPN, the use of FPN significantly reduces false alarms while ensuring a low miss rate, which shows that FPN indeed helps target detection in video SAR images. In addition, compared with the results obtained by Faster RCNN+FPN, the proposed method has fewer misses and false alarms, which shows the effectiveness of the dual-stream feature extraction network equipped with FFM.

[0144] As shown in Figure 13 , the feature fusion results of four representative test samples are shown. In the proposed method, FFM interacts and fuses the features extracted from the original video SAR image with the sparse image, improving the final detection performance. As shown in Figure 13 , first of all, we can see all the attention weights of the four test samples are non-zero matrices and four block matrices , , , and are non-zero matrices in each attention weight, which indicates that not only the features within the image are interactive and fused and , but also the features between the images and . In addition, it can be seen that the target features (yellow boxes) in the feature maps are blurred, and the background similar to the target (green boxes) shows more obvious features before the FFM is involved in feature extraction. It can be predicted that if only these features are used for RPN and RCNN, the final detection performance will be poor. Nevertheless, the target features show the most distinguishable representation in the feature maps while suppressing the features of the background similar to the target after the FFM is used for feature extraction. Therefore, the proposed method can achieve excellent detection performance.

[0145] As shown in Figure 14 , the comparison results of the detection results and the threshold of the prediction score. Among them, the threshold of the prediction score is an important parameter that affects the final detection performance, and an appropriate threshold can produce better detection results. As shown in Figure 14 , the detection results are shown as the prediction score threshold changes, FN increases with the increase of the threshold, while FP is the opposite. If there is a good method to suppress false alarms in the subsequent process, a smaller prediction score threshold should be selected. Similarly, when there is a good method to eliminate missed detections in the subsequent process, a larger prediction score threshold should be selected. In the absence of subsequent processes, an appropriate prediction score threshold should be selected to balance FN and FP, so that both larger and smaller thresholds should be discarded. In addition, it can be seen that when the threshold changes from 0.5 to 0.6, the ratio of FN to FP changes significantly smaller, which indicates that as FP increases, the FN increases less. Therefore, 0.6 is selected as the prediction score threshold of the proposed method.

[0146] As shown in Figure 15 , the frame per second (FPS) of the proposed method and other detection methods is shown. It can be seen that the proposed method scores the lowest, and SSD and YOLOv3-SPP show relatively high FPS on both hardware. This is due to the proposed method being a two-stage detection method and the Transformer framework having a large number of network parameters. With the adoption of GPU, all detection methods have been significantly improved, although the proposed method may not be able to achieve the highest FPS, but it can meet the real-time requirements.

[0147] The various embodiments in this specification are described in a progressive manner, and the same or similar parts among the various embodiments can be referred to each other, and each embodiment focuses on the difference from other embodiments. The whole or part of the present application can be used in a variety of general or special computer system environments or configurations. For example: personal computers, server computers, handheld devices or portable devices, tablet devices, mobile communication terminals, multi-processor systems, microprocessor-based systems, programmable electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and the like.

[0148] The above examples are only used to illustrate the technical solutions of the present application, and are not limited to the present application; although the present application has been described in detail with reference to the foregoing examples, those skilled in the art should understand that the technical solutions recorded in the foregoing examples can still be modified, or part or all of the technical features can be replaced by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the present application.

Claims

1. A method for video SAR joint moving target detection, characterized in that, The method comprises the steps of: using a Faster RCNN method to separate moving targets in a video SAR image from the background to obtain an initial moving target image, and obtaining a sparse image according to the initial moving target image; using a double-flow feature extraction network to perform feature extraction and fusion on the video SAR image and the sparse image to obtain a classification probability of the moving target; wherein the double-flow feature extraction network comprises two single-image feature extraction modules, a double-image feature fusion module, and a multi-feature output module; the two single-image feature extraction modules are used to respectively perform feature extraction on the video SAR image and the sparse image to respectively obtain second video SAR feature maps and second target sparse feature maps, third video SAR feature maps and third target sparse feature maps, and fourth video SAR feature maps and fourth target sparse feature maps; the double-image feature fusion module is used to perform feature fusion on the second video SAR feature maps and the second target sparse feature maps, the third video SAR feature maps and the third target sparse feature maps, and the fourth video SAR feature maps and the fourth target sparse feature maps to obtain first fusion features, second fusion features, and third fusion features; and the multi-feature output module is used to use an FPN block to predict the first fusion features, the second fusion features, and the third fusion features to obtain the classification probability of the moving target.

2. The method of claim 1, wherein, The method of using the Faster RCNN method to separate the moving targets in the video SAR image from the background to obtain the initial moving target image comprises the steps of: determining a SAR image sequence corresponding to the video SAR image, and dividing each SAR image in the SAR image sequence into a background and a target shadow; constructing an optimization constraint model of the video SAR image according to the background and the target shadow of each SAR image, and making the optimization constraint model converge to obtain the initial moving target image.

3. The method of claim 2, wherein, The division of each SAR image in the SAR image sequence into the background and the target shadow is represented as: D = B + T; wherein B represents the background in the SAR image sequence, and T represents the target shadow in the SAR image sequence.

4. The method of claim 2, wherein, The optimization constraint model is represented as: wherein rank(·) represents the rank of a matrix, λ represents a regularization factor for balancing two terms of the objective function, D represents the SAR image sequence, B represents the background in the SAR image sequence, T represents the target shadow in the SAR image sequence, and ||·||0 represents an l0 norm.

5. The method of claim 1, wherein, The two single-image feature extraction modules are used to respectively perform feature extraction on the video SAR image and the sparse image to respectively obtain second video SAR feature maps and second target sparse feature maps, third video SAR feature maps and third target sparse feature maps, and fourth video SAR feature maps and fourth target sparse feature maps, and the method comprises the steps of: The two single-image feature extraction modules each comprise a first network block, a second network block, a third network block, and a fourth network block connected in sequence. The first network block comprises a spatial contraction layer, a first convolutional layer and a second convolutional layer connected in sequence, and the first network block is used for converting the video SAR image and the sparse image into a first video SAR feature map and a first target sparse feature map, respectively; The second network block comprises a third convolutional layer and a fourth convolutional layer connected in sequence; the second network block is used for converting the channel dimension numbers of the first video SAR feature map and the first target sparse feature map, respectively, to obtain a second video SAR feature map and a second target sparse feature map; wherein the channel dimension numbers of the second video SAR feature map and the second target sparse feature map are half of the channel dimension numbers of the first video SAR feature map and the first target sparse feature map; The third network block comprises a fifth convolutional layer and a sixth convolutional layer connected in sequence; the third network block is used for converting the channel dimension numbers of the second video SAR feature map and the second target sparse feature map, respectively, to obtain a third video SAR feature map and a third target sparse feature map; wherein the channel dimension numbers of the third video SAR feature map and the third target sparse feature map are half of the channel dimension numbers of the second video SAR feature map and the second target sparse feature map; The fourth network block comprises a seventh convolutional layer, a spatial pyramid pooling layer and an eighth convolutional layer connected in sequence; the fourth network block is used for converting the channel dimension numbers of the third video SAR feature map and the third target sparse feature map, respectively, to obtain a fourth video SAR feature map and a fourth target sparse feature map; wherein the channel dimension numbers of the fourth video SAR feature map and the fourth target sparse feature map are half of the channel dimension numbers of the third video SAR feature map and the third target sparse feature map.

6. The method of claim 5, wherein, The dual-image feature fusion module comprises a first fusion network block, a second fusion network block and a third fusion network block; the first fusion network block is arranged between the second network block and the third network block; the second fusion network block is arranged between the third network block and the fourth network block; and the third fusion network block is arranged after the fourth network block.

7. The method of claim 6, wherein, The first fusion network block comprises a first channel adding unit and a first FFM unit connected in sequence; The second fusion network block comprises a second channel adding unit and a second FFM unit connected in sequence; The third fusion network block comprises a third channel adding unit and a third FFM unit connected in sequence.

8. The method of claim 7, wherein, The first fusion network block is used for flattening the second video SAR feature map and the second target sparse feature map through the first channel adding unit and adding position encoding to obtain a second video SAR encoded feature map and a second target sparse encoded feature map, respectively; The second video SAR encoding feature map and the second target sparse encoding feature map are input into the first FFM unit, and a first projected video SAR feature map and a first projected target sparse feature map are correspondingly obtained; and first fusion features are obtained according to the first projected video SAR feature map and the first projected target sparse feature map. The second fusion network block inputs are respectively the first projected video SAR feature map and the first projected target sparse feature map, and outputs are a second projected video SAR feature map and a second projected target sparse feature map; and second fusion features are obtained according to the second projected video SAR feature map and the second projected target sparse feature map. The third fusion network block inputs are the second projected video SAR feature map and the second projected target sparse feature map, and outputs are a third projected video SAR feature map and a third projected target sparse feature map; and third fusion features are obtained according to the third projected video SAR feature map and the third projected target sparse feature map.

Images