Video SAR joint moving target detection method

Abstract

The invention discloses a video SAR (Synthetic Aperture Radar) joint moving target detection method, which solves the problems in the prior art that the contrast ratio of a fast moving target shadow and a background is low and the boundary is fuzzy, and comprises the following steps: decomposing a moving target in a video SAR image from the background to obtain an initial moving target image and a sparse image; performing feature extraction fusion on the video SAR image and the sparse image by using a double-flow feature extraction network to obtain a classification probability of a moving target; the double-flow feature extraction network comprises two single-image feature extraction modules, a double-image feature fusion module and a multi-feature output module; each single image feature extraction module comprises a first network block, a second network block, a third network block and a fourth network block which are connected in sequence; the dual-image feature fusion module comprises a first fusion network block, a second fusion network block and a third fusion network block; the multi-feature output module comprises an FPN block; the method effectively suppresses background noise.

Description

Technical Field

[0001] The present invention relates to the field of radar detection technology, and in particular to a video SAR combined moving target detection method. Background Art

[0002] As a special synthetic aperture radar system, video SAR can generate a series of synthetic aperture radar images similar to video. It can not only work under all-day and all-weather conditions like traditional synthetic aperture radar systems, and is not adversely affected by the imaging scene, but also monitor the area of ​​interest for a long time at a high frame rate. Therefore, video synthetic aperture radar has been applied to many fields including scene monitoring, ground moving target indication (GMTI) and coherent change detection (CCD). Nowadays, video SAR has become a research hotspot in radar moving target detection at home and abroad.

[0003] Ding Jinshan et al. from Xidian University proposed a shadow-assisted moving target detection framework based on convolutional neural network in “Video SAR moving target indication using deep neural network” (IEEE Transactions on Geoscience and Remote Sensing, vol. 58, pp. 7194-7204, Jun. 2017). Faster R-CNN was used for moving target detection, and a density-based clustering algorithm and a bidirectional long short-term memory (Bi-LSTM) network were used to reduce the occurrence of false alarms and missed detections.

[0004] Huang introduced the low-rank and sparse decomposition (LSD) technology into multi-channel SAR ground moving target indication (SAR GMTI) in "Moving target detection via efficient ATI-GoDec approach for multichannel SAR system" (IEEE Geoscience and Remote Sensing Letters, vol.13, pp.1320-1324, Sep.2016) and "GMTI and parameter estimation for MIMO SAR system via fast interferometry RPCA method" (IEEE Transactions on Geoscience and Remote Sensing, vol.56, no.9, pp.1774-1787, Mar.2018). Compared with the traditional ground moving target indication algorithm, the low-rank and sparse decomposition (LSD) technology can separate the target from the background, that is, the background is suppressed while the target is enhanced, thereby significantly improving the detection performance of dim targets. Therefore, low-rank sparse decomposition has been applied to detection and tracking tasks in video synthetic aperture radar images.

[0005] However, moving target shadows are usually dim targets with low intensity and simple features, and are not obviously different from other weak reflectivity areas in video SAR images. In addition, the generation of shadows is highly dependent on the motion of moving targets, especially for fast-moving targets, whose shadows have low contrast with the background and blurred boundaries. Therefore, these CNN-based methods only use target shadows in video SAR images, which still produce a large number of missed detections and false positives. Summary of the invention

[0006] The present invention provides a video SAR joint moving target detection method, which solves the problem in the prior art that for fast-moving targets, the contrast between their shadows and backgrounds is low and the boundaries are blurred, thereby achieving effective suppression of background noise.

[0007] The present invention provides a video SAR joint moving target detection method, the method comprising:

[0008] Decomposing a moving target from a background in a video SAR image by using a Faster RCNN method to obtain an initial moving target image, and obtaining a sparse image based on the initial moving target image;

[0009] The dual-stream feature extraction network is used to perform feature extraction and fusion on the video SAR image and the sparse image to obtain the classification probability of the moving target; wherein the dual-stream feature extraction network includes: two single-image feature extraction modules, a dual-image feature fusion module and a multi-feature output module; the two single-image feature extraction modules are used to perform feature extraction on the video SAR image and the sparse image respectively, and obtain the second video SAR feature map and the second target sparse feature map, the third video SAR feature map and the third target sparse feature map, and the fourth video SAR feature map and the fourth target sparse feature map respectively; the dual-image feature fusion module is used to perform feature fusion on the second video SAR feature map and the second target sparse feature map, the third video SAR feature map and the third target sparse feature map, and the fourth video SAR feature map and the fourth target sparse feature map respectively, to obtain the first fused feature, the second fused feature and the third fused feature; the multi-feature output module is used to predict the first fused feature, the second fused feature and the third fused feature by using the FPN block to obtain the classification probability of the moving target.

[0010] In a possible implementation, the method of using the Faster RCNN method to decompose the moving target in the video SAR image from the background to obtain an initial moving target image includes:

[0011] Determine a SAR image sequence corresponding to the video SAR image, and divide each SAR image frame in the SAR image sequence into a background and a target shadow;

[0012] An optimization constraint model of the video SAR image is constructed according to the background and target shadow of each frame of the SAR image, and the optimization constraint model is converged to obtain an initial moving target image.

[0013] In a possible implementation, each frame of the SAR image in the SAR image sequence is divided into background and target shadow and represented as:

[0014] D = B + T;

[0015] Wherein, B represents the background in the SAR image sequence; T represents the target shadow in the SAR image sequence.

[0016] In a possible implementation, the optimization constraint model is expressed as:

[0017]

[0018] Where rank(·) represents the rank of the matrix; λ represents the regularization factor used to balance the two terms of the objective function; D represents the SAR image sequence; B represents the background in the SAR image sequence; T represents the target shadow in the SAR image sequence; ||·||0 represents the l0 norm.

[0019] In a possible implementation, the two single image feature extraction modules are used to perform feature extraction on the video SAR image and the sparse image, respectively, to obtain a second video SAR feature map and a second target sparse feature map, a third video SAR feature map and a third target sparse feature map, and a fourth video SAR feature map and a fourth target sparse feature map, respectively, including:

[0020] The two single image feature extraction modules each include: a first network block, a second network block, a third network block and a fourth network block connected in sequence;

[0021] The first network block includes a spatial contraction layer, a first convolutional layer, and a second convolutional layer connected in sequence, and the video SAR image and the sparse image are converted into a first video SAR feature map and a first target sparse feature map respectively by using the first network block;

[0022] The second network block includes a third convolution layer and a fourth convolution layer connected in sequence; the channel dimensions of the first video SAR feature map and the first target sparse feature map are respectively converted by using the second network block to obtain a second video SAR feature map and a second target sparse feature map; wherein the channel dimensions of the second video SAR feature map and the second target sparse feature map are half of the channel dimensions of the first video SAR feature map and the first target sparse feature map;

[0023] The third network block includes a fifth convolutional layer and a sixth convolutional layer connected in sequence; the channel dimensions of the second video SAR feature map and the second target sparse feature map are respectively converted by the third network block to obtain a third video SAR feature map and a third target sparse feature map; wherein the channel dimensions of the third video SAR feature map and the third target sparse feature map are half of the channel dimensions of the second video SAR feature map and the second target sparse feature map;

[0024] The fourth network block includes a seventh convolutional layer, a spatial pyramid pooling layer and an eighth convolutional layer connected in sequence; the channel dimensions of the third video SAR feature map and the third target sparse feature map are respectively converted by the fourth network block to obtain a fourth video SAR feature map and a fourth target sparse feature map; wherein the channel dimensions of the fourth video SAR feature map and the fourth target sparse feature map are half of the channel dimensions of the third video SAR feature map and the third target sparse feature map.

[0025] In a possible implementation, the dual image feature fusion module includes: a first fusion network block, a second fusion network block and a third fusion network block; the first fusion network block is arranged between the second network block and the third network block; the second fusion network block is arranged between the third network block and the fourth network block; the third fusion network block is arranged after the fourth network block.

[0026] In a possible implementation, the first fusion network block includes a first channel adding unit and a first FFM unit connected in sequence;

[0027] The second fusion network block includes a second channel adding unit and a second FFM unit connected in sequence;

[0028] The third fusion network block includes a third channel adding unit and a third FFM unit connected in sequence.

[0029] In a possible implementation, the first fusion network block is used to flatten the second video SAR feature map and the second target sparse feature map through the first channel adding unit, and add position coding to obtain the second video SAR coded feature map and the second target sparse coded feature map respectively; input the second video SAR coded feature map and the second target sparse coded feature map into the first FFM unit to obtain the first projected video SAR feature map and the first projected target sparse feature map respectively; obtain the first fusion feature according to the first projected video SAR feature map and the first projected target sparse feature map;

[0030] The second fusion network block inputs the first projection video SAR feature map and the first projection target sparse feature map, and outputs the second projection video SAR feature map and the second projection target sparse feature map; the second fusion feature is obtained according to the second projection video SAR feature map and the second projection target sparse feature map;

[0031] The third fusion network block inputs the second projection video SAR feature map and the second projection target sparse feature map, and outputs the third projection video SAR feature map and the third projection target sparse feature map; and obtains the third fusion feature according to the third projection video SAR feature map and the third projection target sparse feature map.

[0032] One or more technical solutions provided in the present invention have at least the following technical effects or advantages:

[0033] The present invention firstly uses low-rank sparse decomposition to separate the moving target from the background and generate a sparse image containing the moving target, which effectively suppresses the background noise. It also uses a dual-stream feature extraction network to fuse the video SAR image and the sparse image to enhance the feature extraction capability, thereby improving the detection performance of the moving target. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 A flowchart of the steps of the video SAR joint moving target detection method provided by an embodiment of the present invention;

[0035] Figure 2 A schematic diagram of statistical results of a video SAR image provided by an embodiment of the present invention;

[0036] Figure 3 A target shadow map in a video SAR image provided by an embodiment of the present invention;

[0037] Figure 4 A feature fusion module framework diagram provided by an embodiment of the present invention;

[0038] Figure 5 A structural diagram of a spatial pyramid pooling layer provided by an embodiment of the present invention;

[0039] Figure 6 A structural diagram of a feature fusion module provided in an embodiment of the present invention;

[0040] Figure 7 A structural diagram of a multi-feature output module provided in an embodiment of the present invention;

[0041] Figure 8 A schematic diagram of a curve showing the change of training loss with epoch provided in an embodiment of the present invention;

[0042] Fig. 9 A schematic diagram of detection results of the method proposed by the present invention and other methods on the data set at this time provided by the embodiment of the present invention;

[0043] Fig.10 A schematic diagram of mAP of all methods in each target category of the method proposed by the present invention provided in an embodiment of the present invention;

[0044] Fig.11 The test result diagram of five representative test samples provided by the embodiment of the present invention;

[0045] Fig.12 Indicative result intentions for each test sample in the multiple methods provided in the embodiments of the present invention;

[0046] Fig.13 A schematic diagram of feature fusion results of four representative test samples provided in an embodiment of the present invention;

[0047] Fig.14 A schematic diagram of the comparison result between the detection result and the threshold value of the prediction score provided by the embodiment of the present invention;

[0048] Fig.15 A schematic diagram of the real-time performance comparison results of four methods provided in embodiments of the present invention. DETAILED DESCRIPTION

[0049] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0050] The present invention provides a video SAR joint moving target detection method, such as Figure 1 As shown, the method includes the following steps S101 to S102.

[0051] S101, using the Faster RCNN method to decompose the moving target in the video SAR image from the background to obtain an initial moving target image, and obtaining a sparse image based on the initial moving target image;

[0052] Specifically, in step S101, the Faster RCNN method is used to decompose the moving target in the video SAR image from the background to obtain an initial moving target image, including the following steps S1011 to S1012.

[0053] S1011, determining a SAR image sequence corresponding to the video SAR image, and performing format conversion according to the SAR image sequence, so that each frame of the SAR image in the SAR image sequence is divided into a background and a target shadow;

[0054] Here, the SAR image sequence in the video SAR image can be expressed as:

[0055]

[0056] Where D represents the SAR image sequence, represents the kth frame of SAR image, (M, N) represents the width and height of each frame of SAR image, and K represents the length of the SAR image sequence, that is, the number of frames. Each frame of SAR image can be divided into two parts: background and target shadow, namely:

[0057] F k =B k +T k (2);

[0058] Among them, B k represents the background in the k-th frame SAR image, T k represents the target shadow in the k-th frame SAR image. Then the SAR image sequence is re-expressed as:

[0059] D = B + T (3);

[0060] In the SAR image sequence, B represents the background in the SAR image sequence; T represents the target shadow in the SAR image sequence.

[0061] For example, compared to the background, target shadows only occupy a few pixels in the SAR image because of their limited number and small size. Figure 2 As shown, Figure 2 The pixel value of the target shadow in the upper left part of the image is basically around 27. The target shadow only occupies a small part of the entire SAR image, such as Figure 3 Therefore, T k exhibits sparsity, and this property is more obvious in T.

[0062] Since video SAR systems usually monitor a specific region of interest for a long time, the background in the SAR image sequence is often static, that is, the background in the SAR image sequence is almost the same. In other words, if the frame images in the SAR image sequence have been pre-registered, the background will show high similarity and correlation. From the perspective of matrix D, due to the high similarity between the backgrounds of multiple frames, they can be projected into the same feature subspace. Therefore, matrix B k It has a low rank property. To visualize this property, singular value decomposition is used to analyze the distribution of ranks, and the low rank is evaluated by the Cumulative Distribution Function (CDF) to analyze the spatial structure information.

[0063]

[0064] Where Λ represents a diagonal matrix consisting of all singular values. Λ(i,i) represents the i-th largest singular value, S represents the number of singular values, and k represents the number of the first k singular values. Figure 2(c) shows the matrix B with different rank distributions. k CDF of SAR data. Using CDF analysis of multiple frames, we can see that the CDF of video SAR data is more similar to a rank-one matrix, such as Figure 2 (c) In addition, the distribution of singular values ​​is concentrated, rather than uniformly distributed in the full-rank matrix. The above results indicate that the background in the SAR image sequence is low-rank.

[0065] S1012, constructing an optimization constraint model of the video SAR image, and making the optimization constraint model converge to obtain an initial moving target image.

[0066] Here, in order to obtain the target shadow, a constrained optimization model is constructed as follows:

[0067]

[0068] Where rank(·) represents the rank of the matrix; λ represents the regularization factor used to balance the two terms of the objective function; D represents the SAR image sequence; B represents the background in the SAR image sequence; T represents the target shadow in the SAR image sequence; ||·||0 represents the l0 norm.

[0069] Generally, the solution is obtained by using threshold generation and the Alternating Direction Method of Multipliers (ADMM). However, the optimization process of the former is not smooth and the convergence speed is relatively slow, so the latter is selected as the solution method in the embodiment of the present invention. Figure 3 is the decomposition result of the video SAR image. It can be seen that the target shadow is enhanced in the sparse image, while the non-target shadow is eliminated. Obviously, if the target shadow is detected in the sparse image instead of the SAR image, the false alarm will be reduced. However, LSD may cause the size distortion of the target shadow in the sparse image, resulting in wrong size detection results.

[0070] S102, using a dual-stream feature extraction network, performing feature extraction and fusion on the video SAR image and the sparse image to obtain the classification probability of the moving target.

[0071] like Figure 4 As shown in Figure 1, the dual-stream feature extraction network includes: two single-image feature extraction modules, a dual-image feature fusion module and a multi-feature output module. Specifically, the parameters of the dual-stream feature extraction network are shown in Table 1:

[0072] Table 1 Structural parameters of the dual-stream feature extraction network

[0073]

[0074] The two single image feature extraction modules each include: a first network block (L1), a second network block (L2), a third network block (L3) and a fourth network block (L4) which are connected in sequence.

[0075] Here, the first fused network block (T1) is arranged between the second network block (L2) and the third network block (L3); the second fused network block (T2) is arranged between the third network block (L3) and the fourth network block (L4); and the third fused network block (T3) is arranged after the fourth network block (L4).

[0076] The two single image feature extraction modules are used to extract features from the video SAR image and the sparse image respectively, and obtain the second video SAR feature map and the second target sparse feature map, the third video SAR feature map and the third target sparse feature map, and the fourth video SAR feature map and the fourth target sparse feature map respectively;

[0077] Specifically, the two single image feature extraction modules are used to extract features from the video SAR image and the sparse image respectively, and obtain the second video SAR feature map and the second target sparse feature map, the third video SAR feature map and the third target sparse feature map, and the fourth video SAR feature map and the fourth target sparse feature map respectively, including:

[0078] The first network block converts the video SAR image and the sparse image into the first video SAR feature map F respectively by using the sequentially connected spatial shrinkage layer SSL, the first convolutional layer Conv1 and the second convolutional layer Conv3. o 1 and the first target sparse feature map F S 1 ;

[0079] The second network block uses the third convolutional layer Conv1 and the fourth convolutional layer Conv3 connected in sequence to transform the first video SAR feature map F o 1 and the first target sparse feature map F S 1 The channel dimension number is converted to obtain the second video SAR feature map F o 2 and the second target sparse feature map F S 2 ; Among them, the second video SAR feature map F o 2 and the second target sparse feature map F S 2 The channel dimension of the first video SAR feature map F o 1 and the first target sparse feature map F S 1Half the number of channel dimensions of ;

[0080] The third network block uses the fifth convolutional layer and the sixth convolutional layer connected in sequence to convert the channel dimensions of the second video SAR feature map and the second target sparse feature map to obtain the third video SAR feature map F o 3 And the third target sparse feature map F S 3 ; Among them, the third video SAR feature map F o 3 And the third target sparse feature map F S 3 The channel dimension number is the second video SAR feature map F o 2 and the second target sparse feature map F S 2 Half the number of channel dimensions of ;

[0081] The fourth network block uses the seventh convolutional layer, the spatial pyramid pooling layer and the eighth convolutional layer connected in sequence to transform the third video SAR feature map F o 3 And the third target sparse feature map F S 3 The channel dimension number is converted to obtain the fourth video SAR feature map F o 4 And the fourth target sparse feature map F S 4 ; Among them, the fourth video SAR feature map F o 4 And the fourth target sparse feature map F S 4 The channel dimension number is the third video SAR feature map F o 3 And the third target sparse feature map F S 3 Half the number of channel dimensions.

[0082] The dual image feature fusion module includes: a first fusion network block (T1), a second fusion network block (T2) and a third fusion network block (T3);

[0083] Here, the first fusion network block (T1) includes a first channel adding unit Add1 and a first FFM unit FFM1 connected in sequence;

[0084] The second fusion network block (T2) comprises a second channel adding unit Add2 and a second FFM unit FFM2 connected in sequence;

[0085] The third fusion network block (T3) includes a third channel adding unit Add3 and a third FFM unit FFM3 which are connected in sequence.

[0086] The dual image feature fusion module is used to perform feature fusion on the second video SAR feature map and the second target sparse feature map, the third video SAR feature map and the third target sparse feature map, and the fourth video SAR feature map and the fourth target sparse feature map to obtain a first fused feature F1, a second fused feature F2, and a third fused feature F3;

[0087] Specifically, the first fusion network block (T1) is used to flatten the second video SAR feature map and the second target sparse feature map through the first channel adding unit Add1, and add position coding to obtain the second video SAR coded feature map and the second target sparse coded feature map respectively; input the second video SAR coded feature map and the second target sparse coded feature map into the first FFM unit FFM1, and obtain the first projection video SAR feature map and the first projection target sparse feature map accordingly; obtain the first fusion feature F1 according to the first projection video SAR feature map and the first projection target sparse feature map;

[0088] Similar to the first fusion network block (T1): the second fusion network block (T1) inputs the first projection video SAR feature map and the first projection target sparse feature map respectively, and outputs the second projection video SAR feature map and the second projection target sparse feature map; according to the second projection video SAR feature map and the second projection target sparse feature map, the second fusion feature F2 is obtained; the third fusion network block (T1) inputs the second projection video SAR feature map and the second projection target sparse feature map, and outputs the third projection video SAR feature map and the third projection target sparse feature map; according to the third projection video SAR feature map and the third projection target sparse feature map, the third fusion feature F3 is obtained.

[0089] The multi-feature output module includes: an FPN block; the multi-feature output module is used to use the FPN block to predict the first fusion feature F1, the second fusion feature F2 and the third fusion feature F3 to obtain the classification probability of the moving target.

[0090] Exemplarily, the dual-stream feature extraction network is specifically divided into three steps: single image feature extraction, dual-image feature fusion, and multi-feature output. In the proposed dual-stream feature extraction network, single image feature extraction is first implemented through two shallow-to-deep network branches to independently extract features from video SAR images and sparse images. Then, after each feature scale is reduced, the dual-image feature fusion module uses the proposed FFM to interact and aggregate features from different branches. Finally, the Feature Pyramid Network (FPN) is introduced in the multi-feature output module to generate a multi-level feature map. These three steps will be described in detail below.

[0091] (1) Single image feature extraction module

[0092] The implementation of single image feature extraction is the same as other CNN-based feature extraction networks such as VGGnet and ResNet, that is, two shallow-to-deep network branches are used for feature extraction of video SAR images and sparse images respectively. However, compared with the traditional feature extraction network, this method first designs a spatial shrinking level (SSL) to reduce the dimension of the feature map in each network branch. Specifically, the spatial information of the feature map is converted into the channel dimension by reshaping the dimension of the feature map, and then the channel dimension is compressed using a 3×3 convolutional layer.

[0093] F SSL =Conv 3×3 (Reshape(F)) (6);

[0094] Among them, F represents the feature map, F SSL represents the output feature map of SSL compression, Reshape(·) represents the reshaping operation, Conv 3×3 (·) indicates a 3×3 convolutional layer.

[0095] In addition, in order to improve the feature extraction capabilities of each network branch for video SAR images and sparse images, a spatial pyramid pooling network (SPP) is introduced. Figure 5 Shown is the structure of SPP.

[0096] SPP consists of four parallel branches: max pooling layers with window sizes of 5×5, 9×9, and 13×13, respectively, and a skip connection.

[0097] Since the stride is 1, SPP performs sliding pooling on the input feature map using multiple maximum pooling layers with different window sizes, and then integrates the pooled features through channel cascading and 1×1 convolutional layers. SPP extracts and fuses features of different scales, thereby enriching the representation capability of the feature map.

[0098] F SPP =Conv 1×1 (Cat(Pooling k′ (F),k′=[0,5,9,13])) (7);

[0099] Among them, F represents the input feature map, F SPP represents the feature map of the output after SPP compression, Cat(·) represents the channel cascade operation, Conv 1×1 (·) represents a 1×1 convolutional layer, Pooling k′ (·) indicates a maximum pooling layer with a pooling window size of k′×k′. Where k′=0 means that the feature map does not perform a maximum pooling operation, but performs a skip connection.

[0100] (2) Dual image feature fusion module

[0101] Since Transformer has significant performance in various types of tasks, a Transformer-based module, FFM, is proposed in dual-image feature fusion to fuse the target features extracted from video SAR images and sparse images.

[0102] like Figure 6 As shown, represents the second video SAR feature map, and Represents the second target sparse feature map, where W represents the length of the feature map, H represents the width of the feature map, and C represents the number of channels.

[0103] First, the second video SAR feature map and the second target sparse feature map are flattened, and F o With F s The order of the feature matrix is ​​arranged, and then we get as well as The problem that Transformer-based FFM cannot process two-dimensional data is solved.

[0104] Then splice I O 2 and I S 2 , and add a positional encoding of trainable parameters of size 2HW×C to generate the input of FFM Positional encoding allows FFM to capture the spatial information of each token in a sentence.2 Projected into three separate weight matrices, generating a set of tensors Q, K, and V.

[0105]

[0106] Where I represents I O 2 and I S 2 , and is the projection weight matrix. In FFM, C Q , C K and C V Equal and C Q =C K =C V = C. The attention weight A is calculated by the normalized dot product between Q and K:

[0107]

[0108] Where T represents the transpose transformation, softmax(·) represents the softmax function mapping each attention weight to [0, 1], A represents the scaling factor used to reduce the variance of dot product size and improve the stability of gradient update during the training phase. OO , A OS , A SO and A SS is a block matrix. The refinement result is obtained by multiplying the attention weights and the values.

[0109]

[0110] In addition, a multi-head attention mechanism is adopted to better capture the various relationships and subtle differences between each token in the sentence.

[0111]

[0112] Where h represents the number of long positions. In FFM, h = 8 and The weight matrix Cat(Z1,Z2,...,Z i ,...,Z h After that, a multilayer perceptron (MLP) with two fully connected layers and GELU activation function is used to obtain an output I′ of the same size as the input I.

[0113]

[0114] Where Z″=Z′+I, MLP(·) represents a multilayer perceptron, FC(·) represents a fully connected layer, and GELU(·) represents a GELU activation function.

[0115] Finally, follow F O 2 and F S 2 The shape reconstruction output I′ is further divided into I o ′ and I s ′, reconstruct the shape from WH*C to W*H*C, and obtain the final output and When using FFM, it can be seen that the attention weight A can be naturally divided into four matrix blocks, as shown in formula (10), where A OO and A SS is the attention weight within the image, A SO and A OS is the attention weight between images. Therefore, the FFM proposed in the embodiment of the present invention can not only focus on the position of the target in the feature map, but also interact and fuse the feature maps extracted from the video SAR image and the sparse image.

[0116] The same method as above is used to obtain the corresponding I′ for the second fused network block (T2) and the third fused network block (T3).

[0117] (3) Multi-feature output module

[0118] like Figure 7 As shown in Figure 1, target shadows have different scales and can be divided into three categories: small targets with a minimum bounding rectangle (MBR) area below 64, intermediate targets with an MBR area below 256 and above 64, and large targets with an MBR area above 256. It is well known that smaller targets are particularly easy to be lost in deeper feature maps, so using only the deepest feature map to predict targets may lead to missed detections of smaller targets.

[0119] Therefore, an FPN is introduced to deal with this problem, in which object shadows of different sizes are detected on feature maps at different levels.

[0120] Specifically, in RPN, multiple anchors of appropriate sizes are first set, and then these anchors are assigned to different feature maps according to their MBR regions.

[0121] For example, if the MBR area of ​​an anchor is lower than 64, it will be used in feature map P1 and classified and regressed by using the ground truth with the same MBR area range. With this setting, object shadows of different sizes can be predicted by different feature maps.

[0122] The two-stream feature extraction network effectively utilizes and fuses the target information from the two images to produce more discriminative target features. Therefore, in the subsequent detection process, the target shadow can be more effectively predicted and classified.

[0123] The following is a simulation embodiment provided by the present invention.

[0124] 1. Simulation conditions:

[0125] The simulation of the present invention is carried out in the software environment of MATLAB R2020a and VScode.

[0126] 2. Simulation content:

[0127] Real data collected by the Lynx video SAR system of Sandia National Laboratories are used to validate the proposed method. The real data are the only publicly available video SAR data containing targets. Table 1 shows the parameters of the video SAR system. The video SAR images in the real data rotate in response to the rotation of the imaging coordinate system as the video SAR system adopts spotlight imaging mode and moves along a circular trajectory. The real data consists of 899 consecutive SAR images with a size of 660×720 pixels. In this paper, each SAR image is resized to 512×512 pixels. In addition, the image is first filtered using a mean filter.

[0128] Table 1 Video SAR system parameters

[0129] System Parameters value Platform height 2km PRF 1kHz frequency 16.7GHz Maximum speed 250m / s Distance resolution 0.1m Total time 232

[0130] (1) Training dataset: The last 700 SAR images in the real data are selected to construct the training set. To expand the training set, each SAR image is rotated by 0°, 45°, 90°, 135°, 180°, 225°, 270°, and 315° and randomly cropped. The target is annotated in each SAR image to generate a true label. A training sample contains a SAR image and its corresponding true label. There are 10 4 training samples are used to train the proposed method.

[0131] (2) Test Dataset: The first 199 SAR images in the real data are selected to construct the test dataset. Similarly, in order to expand the test dataset, each SAR image is rotated by 0°, 45°, 90°, 135°, 180°, 225°, 270°, and 315° without random cropping. Similarly, there are 10 3 test samples are used to test the proposed method. Each test sample consists of a SAR image and its artificially crafted facts. There are 1447 objects in the test dataset.

[0132] In the training phase, a combination of Stochastic Gradient Descent (SGD) and 0.9 Momentum is used as the optimization strategy. The number of epochs is set to 100 and the batch size is selected as 8 to prevent running memory overflow. The proposed method follows the classic Faster RCNN framework, so it also adopts the same training loss composed of RPN loss and RCNN loss. Figure 8 The figure shows how the training loss changes with epoch.

[0133] In this paper, three evaluation indicators are used to verify the detection performance of the proposed method, including false positive (FP), false negative (FN) and mean average precision (mAP). Among them, false alarm refers to the misidentification of non-targets as targets, that is, false alarm; false negative refers to targets that actually exist but are not detected, that is, missed detection; mean average precision is a comprehensive indicator, which can be defined as:

[0134]

[0135] Where n is the number of target categories, z is the average precision of targets in category z, and TP is the true positive example whose detection box and true label meet the Intersection Over Union (IOU) threshold. In this paper, the detection threshold is 0.6 and the IOU threshold is 0.5. Note that the detection performance is negatively correlated with FP and FN, but positively correlated with mAP.

[0136] In order to verify the detection performance of the proposed method, we compared it with four classic detection methods: SSD, YOLOv3-SPP, Faster RCNN, and Faster RCNN+FPN. Among these methods, SSD and YOLOv3-SPP are one-stage detection methods, while the other methods are two-stage detection methods. Fig. 9 Shown are the detection results of the proposed method and other comparison methods on the test dataset.

[0137] In the proposed method, FFM interacts and fuses the features extracted from the raw video SAR image with the sparse image, improving the final detection performance. In order to intuitively explain the working mechanism that allows FFM to enhance the detection performance, four test samples are selected to show the changes in their features after using FFM. Fig.11As shown, first, we can see that all the attention weights of the four test samples are non-zero matrices and four block matrices, and in each attention weight is also a non-zero matrix, which shows that not only the features within the image are interacting and fused (and), but also the features between images are doing so (and). In addition, it can be seen that the target features (yellow box) in the feature map are blurred. What's worse, before FFM does not participate in feature extraction, the background similar to the target (green box) shows more obvious features (see Fig.11 a(3)-d(3)). It can be expected that if only these features are used for RPN and RCNN, the final detection performance will definitely be unsatisfactory. Nevertheless, the target features show the most distinguishable representation in the feature map, while suppressing the features of the background that is similar to the target after feature extraction by FFM (see Fig.11 a(4)-d(4)). Therefore, the proposed method can achieve excellent detection performance.

[0138] 3. Simulation effect analysis:

[0139] like Figure 8 As shown in FIG. , it is the curve of the change of training loss with epoch of the present invention, where is the RPN loss, is the RCNN loss, is the total loss, and we can see that and It drops sharply in the initial epoch and stabilizes at the end of training, making the total loss gradually converge;

[0140] like Fig. 9 As shown in the figure, the detection results of the proposed method and other comparison methods on the test dataset. And for each indicator, the best result is indicated in bold, the down arrow indicates that the detection performance is negatively correlated with the score, and the up arrow is the opposite. It is obvious that the proposed method has achieved the best results in all indicators, among which the highest score is obtained in mAP and the lowest score is obtained in FN and FP. Compared with the one-stage method (SSD and YOLOv3-SPP), the two-stage method (FasterRCNN and Faster RCNN+FPN) shows lower FN but higher FP. The reason is that the two-stage method uses the numerous grids generated by other networks to further classify and regress the target, which ensures less missed detection but increases FP. By observing the detection indicators of the latter three methods, we can see that FPN not only greatly reduces FN and FP, but also improves mAP, which illustrates the effectiveness of FPN. In addition, the proposed method further improves the detection performance, indicating that the two-stream structure equipped with FFM is helpful for target detection.

[0141] like Fig.10As shown in the figure, it is the mAP of all methods in each target category of the present invention. The proposed method still achieves the best score in all target categories, with a score of 0.952 in the car class and a score of 0.707 in the truck class. In addition, almost all methods have higher scores in the car class than in the truck class. There are two reasons for this phenomenon. One is that the training samples of the car class and the truck class in the training set are unbalanced. The small number of training samples in the truck class leads to the detection network not being well trained, resulting in relatively poor performance in the truck class. The other is that these detection networks have a large number of network parameters, so a large number of training samples must be used for optimization. Compared with other methods, since SSD is equipped with fewer network parameters and predicts targets in multiple feature layers at different scales, similar detection performance is obtained in the two target classes. Although the proposed method uses FPN to predict targets of different scales, due to the large number of network parameters of the Transformer structure, its detection performance is still unbalanced in the two categories, but the proposed method still shows the best detection performance.

[0142] like Fig.11 As shown in the figure, the detection results of all methods for five representative test samples are used to visually observe the detection performance of the proposed method. In each sub-figure, the detection results obtained by SSD, YOLOv3-SPP, Faster RCNN, Faster RCNN+FPN and the proposed method are shown from left to right. Obviously, compared with other methods, the proposed method detects more targets and has no false detections for each test sample. Comparing the detection results of the latter three methods with those of the first two methods, it can be seen that the two-stage method obviously misses fewer targets but produces more false alarms, which intuitively proves the conclusions obtained from the quantitative detection results.

[0143] like Fig.12 As shown in the figure, the detection results of the two indicators obtained by all methods for each test sample are given. By observing the detection results of Faster RCNN and Faster RCNN+FPN, the use of FPN significantly reduces false alarms while ensuring a low missed detection rate, which shows that FPN is indeed helpful for target detection in video SAR images. In addition, compared with the results obtained by Faster RCNN+FPN, the proposed method obtains fewer missed detections and false alarms, indicating the effectiveness of the two-stream feature extraction network equipped with FFM.

[0144] like Fig.13 As shown in Figure 2, the feature fusion results of four representative test samples are shown. In the proposed method, FFM interacts and fuses the features extracted from the original video SAR image with the sparse image, which improves the final detection performance. Fig.13As shown, first, we can see that all the attention weights A of the four test samples are non-zero matrices and the four block matrices A OO , A OS , A SO , A SS , and it is also a non-zero matrix in each attention weight, which shows that not only the features within the image are interactive and fused (A OO and A SS ), and the same is true for the features between images (A OS and A SO ). In addition, it can be seen that the target feature (yellow box) in the feature map is blurred, and before FFM is not involved in feature extraction, the background similar to the target (green box) shows more obvious features (see Fig.13 a(3)-d(3)). As can be expected, if only these features are used for RPN and RCNN, the final detection performance is poor. Nevertheless, the target features show the most distinguishable representation in the feature map, while suppressing the features of the background that are similar to the target after feature extraction by FFM (see Fig.13 a(4)-d(4)). Therefore, the proposed method can achieve excellent detection performance.

[0145] like Fig.14 As shown in , the comparison result between the detection result and the threshold of the prediction score is shown in . Among them, the threshold of the prediction score is an important parameter that affects the final detection performance. An appropriate threshold can produce better detection results. Fig.14 As shown in Figure 1, the change of detection results with the prediction score threshold is shown. FN increases with the increase of the threshold, while FP is the opposite. If there is a good method to suppress false alarms in the subsequent process, a smaller prediction score threshold should be selected. Similarly, when there is a good missed detection elimination method in the subsequent process, a larger prediction score threshold should be selected. In the absence of a subsequent process, an appropriate prediction score threshold should be selected to balance FN and FP, so that both larger and smaller thresholds should be discarded. In addition, it can be seen that when the threshold is from 0.5 to 0.6, the ratio of FN to FP changes significantly less, indicating that as FP increases, FN increases less. Therefore, 0.6 is selected as the prediction score threshold for the proposed method.

[0146] like Fig.15As shown in the figure, the frames per second (FPS) of the proposed method and other detection methods are shown. It can be seen that the proposed method has the lowest score, and SSD and YOLOv3-SPP show relatively high FPS on both hardware. This is because the proposed method is a two-stage detection method and has a large number of network parameters due to the Transformer framework. With the adoption of GPUs, all detection methods have been significantly improved. Although the proposed method may not achieve the highest FPS, it can meet the real-time requirements.

[0147] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. All or part of the present invention can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld devices or portable devices, tablet devices, mobile communication terminals, multi-processor systems, microprocessor-based systems, programmable electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc.

[0148] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some or all of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the present invention.

Claims

1. A video SAR joint moving target detection method, characterized in that: include: Decomposing a moving target from a background in a video SAR image by using a Faster RCNN method to obtain an initial moving target image, and obtaining a sparse image based on the initial moving target image; The dual-stream feature extraction network is used to perform feature extraction and fusion on the video SAR image and the sparse image to obtain the classification probability of the moving target; wherein the dual-stream feature extraction network includes: two single-image feature extraction modules, a dual-image feature fusion module and a multi-feature output module; the two single-image feature extraction modules are used to perform feature extraction on the video SAR image and the sparse image respectively, and obtain the second video SAR feature map and the second target sparse feature map, the third video SAR feature map and the third target sparse feature map, and the fourth video SAR feature map and the fourth target sparse feature map respectively; the dual-image feature fusion module is used to perform feature fusion on the second video SAR feature map and the second target sparse feature map, the third video SAR feature map and the third target sparse feature map, and the fourth video SAR feature map and the fourth target sparse feature map respectively, to obtain the first fused feature, the second fused feature and the third fused feature; the multi-feature output module is used to predict the first fused feature, the second fused feature and the third fused feature using the FPN block to obtain the classification probability of the moving target.

2. The video SAR joint moving target detection method according to claim 1, characterized in that: The method of using the Faster RCNN method to decompose the moving target in the video SAR image from the background to obtain an initial moving target image includes: Determine a SAR image sequence corresponding to the video SAR image, and divide each SAR image frame in the SAR image sequence into a background and a target shadow; An optimization constraint model of the video SAR image is constructed according to the background and target shadow of each frame of the SAR image, and the optimization constraint model is converged to obtain an initial moving target image.

3. The video SAR joint moving target detection method according to claim 2, characterized in that: Each frame of the SAR image in the SAR image sequence is divided into background and target shadow, which is expressed as: D = B + T; Wherein, B represents the background in the SAR image sequence; T represents the target shadow in the SAR image sequence.

4. The video SAR joint moving target detection method according to claim 2, characterized in that: The optimization constraint model is expressed as: Where rank(·) represents the rank of the matrix; λ represents the regularization factor used to balance the two terms of the objective function; D represents the SAR image sequence; B represents the background in the SAR image sequence; T represents the target shadow in the SAR image sequence; ||·||0 represents the l0 norm.

5. The video SAR joint moving target detection method according to claim 1, characterized in that: The two single image feature extraction modules are used to respectively extract features from the video SAR image and the sparse image, and obtain a second video SAR feature map and a second target sparse feature map, a third video SAR feature map and a third target sparse feature map, and a fourth video SAR feature map and a fourth target sparse feature map, respectively, including: The two single image feature extraction modules each include: a first network block, a second network block, a third network block and a fourth network block connected in sequence; The first network block includes a spatial contraction layer, a first convolutional layer, and a second convolutional layer connected in sequence, and the video SAR image and the sparse image are converted into a first video SAR feature map and a first target sparse feature map respectively by using the first network block; The second network block includes a third convolution layer and a fourth convolution layer connected in sequence; the channel dimensions of the first video SAR feature map and the first target sparse feature map are respectively converted by using the second network block to obtain a second video SAR feature map and a second target sparse feature map; wherein the channel dimensions of the second video SAR feature map and the second target sparse feature map are half of the channel dimensions of the first video SAR feature map and the first target sparse feature map; The third network block includes a fifth convolutional layer and a sixth convolutional layer connected in sequence; the channel dimensions of the second video SAR feature map and the second target sparse feature map are respectively converted by the third network block to obtain a third video SAR feature map and a third target sparse feature map; wherein the channel dimensions of the third video SAR feature map and the third target sparse feature map are half of the channel dimensions of the second video SAR feature map and the second target sparse feature map; The fourth network block includes a seventh convolutional layer, a spatial pyramid pooling layer and an eighth convolutional layer connected in sequence; the channel dimensions of the third video SAR feature map and the third target sparse feature map are respectively converted by the fourth network block to obtain a fourth video SAR feature map and a fourth target sparse feature map; wherein the channel dimensions of the fourth video SAR feature map and the fourth target sparse feature map are half of the channel dimensions of the third video SAR feature map and the third target sparse feature map.

6. The video SAR joint moving target detection method according to claim 5, characterized in that: The dual image feature fusion module includes: a first fusion network block, a second fusion network block and a third fusion network block; the first fusion network block is arranged between the second network block and the third network block; the second fusion network block is arranged between the third network block and the fourth network block; the third fusion network block is arranged after the fourth network block.

7. The video SAR joint moving target detection method according to claim 6, characterized in that: The first fusion network block includes a first channel adding unit and a first FFM unit connected in sequence; The second fusion network block includes a second channel adding unit and a second FFM unit connected in sequence; The third fusion network block includes a third channel adding unit and a third FFM unit connected in sequence.

8. The video SAR joint moving target detection method according to claim 7, characterized in that: The first fusion network block is used to flatten the second video SAR feature map and the second target sparse feature map through the first channel adding unit, and add position coding to obtain the second video SAR coded feature map and the second target sparse coded feature map respectively; Inputting the second video SAR coded feature map and the second target sparse coded feature map into the first FFM unit, and correspondingly obtaining a first projected video SAR feature map and a first projected target sparse feature map; obtaining a first fusion feature according to the first projected video SAR feature map and the first projected target sparse feature map; The second fusion network block inputs the first projection video SAR feature map and the first projection target sparse feature map, and outputs the second projection video SAR feature map and the second projection target sparse feature map; the second fusion feature is obtained according to the second projection video SAR feature map and the second projection target sparse feature map; The third fusion network block inputs the second projection video SAR feature map and the second projection target sparse feature map, and outputs the third projection video SAR feature map and the third projection target sparse feature map; and obtains the third fusion feature according to the third projection video SAR feature map and the third projection target sparse feature map.

Images