A semantic segmentation method for polarimetric SAR images based on complex U-Net and capsule network

Abstract

The present invention relates to a polarimetric SAR image semantic segmentation method based on a complex U-Net and a capsule network. The invention comprises the following steps: step S1: a complex U-Net encoding network sequentially extracts shallow features F1, middle features F2, high-level features F3, and encoding output features F4 by using a complex convolution layer; step S2: the complex U-Net encoding output features F4 are input into a complex capsule network to obtain high-level features F5 containing target posture information; step S3: the complex capsule network output features F5 are input into a complex U-Net decoding network, and after a complex upsampling-convolution layer and a complex convolution layer, decoding features F′3, F′2, F′1, and a decoding network output feature F′0 are sequentially obtained. During the decoding process, F′3 is spliced ​​with F3, F′2 is spliced ​​with F2, and F′1 is spliced ​​with F1; step S4: the complex U-Net decoding network output feature F′0 is subjected to a modulo operation and a softmax classifier to obtain a polarimetric SAR image semantic segmentation result.

Description

Technical Field

[0001] The present invention relates to the field of radar signal processing, and in particular to a polarimetric SAR image semantic segmentation method based on complex U-Net and capsule network, which is used to extract features containing target posture information in polarimetric SAR complex images, thereby obtaining high polarimetric SAR image semantic segmentation performance. Background Art

[0002] Polarimetric SAR image semantic segmentation is one of the research hotspots in the field of SAR image interpretation. It can not only achieve pixel-by-pixel target classification, but also obtain the location of the target in the image, which is of great significance in both military and civilian fields. In recent years, with the development of deep learning, polarimetric SAR image semantic segmentation based on deep learning has been deeply studied.

[0003] U-Net (Ronneberger O, Fischer P, Brox TU-Net: Convolutional Networks for Biomedical Image Segmentation [J]. Medical Image Computing and Computer-Assisted Intervention, 2015, 56 (9): 234-241.) is a convolutional neural network that includes an encoder and a decoder and has a symmetrical U-shaped structure. It has been widely used in the semantic segmentation of optical images. However, this network is a real network and is difficult to be directly applied to the semantic segmentation of complex polarimetric SAR images. Moreover, since the network parameters are trained on an optical data set, it is difficult to achieve good segmentation performance by directly applying it to polarimetric SAR images with a small data set. Therefore, it is necessary to generalize the real U-Net to the complex domain and adjust its network model parameters to be suitable for the semantic segmentation of complex polarimetric SAR images. In addition, since U-Net is a convolutional neural network, it cannot effectively extract the posture (size, shape, position, etc.) information of the target in the image, which makes the semantic segmentation performance of polarimetric SAR images still need to be improved.

[0004] Capsule network is a neural network that can extract information about the target's posture in a small sample set (Sabour S, Frosst N, Hinton G E. Dynamic Routing between Capsules [C]. Advances in Neural Information Processing Systems. California, USA: NIPS, 2017: 3856-3866.). Extending capsule network to the complex domain and combining it with complex U-Net for polarimetric SAR image semantic segmentation can further extract information such as the size and shape of targets in polarimetric SAR images, as well as the relative position between targets of different categories, which helps to improve the performance of polarimetric SAR image semantic segmentation. Summary of the invention

[0005] The purpose of the present invention is to provide a polarimetric SAR image semantic segmentation method based on complex U-Net and capsule network, which can obtain high semantic segmentation performance.

[0006] To achieve the above object, the present invention provides a method for semantic segmentation of polarimetric SAR images based on complex U-Net and capsule network, the steps of which include:

[0007] Step S1: The complex U-Net encoding network uses complex convolutional layers to sequentially extract shallow features F1, middle features F2, high-level features F3, and encoded output features F4;

[0008] Step S2: Input the complex U-Net encoded output feature F4 into the complex capsule network to obtain a high-level feature F5 containing the target posture information;

[0009] Step S3: Input the complex capsule network output feature F5 into the complex U-Net decoding network, and after the complex upsampling-convolution layer and the complex convolution layer, obtain the decoding features F′3, F′2, F1′, and the decoding network output feature F′0 in turn. During the decoding process, F′3 is concatenated with F3, F′2 is concatenated with F2, and F1′ is concatenated with F1;

[0010] Step S4: The complex U-Net decoding network output feature F′0 is subjected to modulo operation and softmax classifier to obtain the semantic segmentation result of the polarimetric SAR image.

[0011] The beneficial effects of the present invention are as follows: in view of the problem that real U-Net is difficult to directly apply to complex polarization SAR image semantic segmentation and cannot effectively extract the posture information of the target in the polarization SAR image, a method combining complex U-Net and capsule network is adopted to obtain the posture information of the target in the complex image, thereby improving the semantic segmentation performance of polarization SAR image. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 It is a general flow chart of the polarimetric SAR image semantic segmentation method based on complex U-Net and capsule network in the present invention;

[0013] Figure 2 It is a network structure and parameter setting diagram of the combination of complex U-Net and capsule network applied to polarimetric SAR image semantic segmentation in the present invention;

[0014] Figure 3 The results of the mean intersection over union (MIOU), overall accuracy (OA) and mean pixel accuracy (MPA) obtained by applying the present invention and three methods based on U-Net and complex U-Net to the Flevoland dataset are compared;

[0015] Figure 4 It is a comparison of the results of MIOU, OA and MPA obtained by applying the present invention and three methods based on U-Net and complex U-Net to the San Francisco dataset. DETAILED DESCRIPTION

[0016] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with specific embodiments and with reference to the accompanying drawings.

[0017] Figure 1 The overall flow chart of a polarimetric SAR image semantic segmentation method based on complex U-Net and capsule network in the present invention is shown. The specific implementation steps of the method are as follows:

[0018] Step S1: The complex U-Net encoding network uses complex convolutional layers to sequentially extract shallow features F1, middle features F2, high-level features F3, and encoded output features F4;

[0019] Step S11: The structure of the complex U-Net encoding network is as follows Figure 2 As shown in the figure, the input is 6-channel 64×64 polarization SAR data. After two complex convolutions with a step size of 1 and a channel number of 32, the feature F1 with a 32-channel 64×64 is obtained. After another complex convolution with a step size of 2 and a channel number of 32, the feature F1 with a 32-channel 32×32 is obtained.

[0020] Step S12: The 32-channel 32×32 feature is subjected to two complex convolutions with a length of 1 and a number of channels of 64 to obtain a 64-channel 32×32 feature F2, and then subjected to one complex convolution with a step size of 2 and a number of channels of 128 to obtain a 128-channel 16×16 feature;

[0021] Step S13: The 128-channel 16×16 feature is subjected to three complex convolutions with a length of 1 and a number of channels of 128 to obtain a 128-channel 16×16 feature F3, and then subjected to one complex convolution with a step size of 2 and a number of channels of 128 to obtain a 128-channel 8×8 feature F4;

[0022] Step S2: Input the complex U-Net encoded output feature F4 into the complex capsule network to obtain a high-level feature F5 containing the target posture information;

[0023] Step S21: The structure of the complex capsule network is as follows Figure 2 As shown, the complex U-Net encoding output feature F4 is dimensionally upgraded to obtain F′4, that is, the size is increased from 8×8×128 to 8×8×1×128;

[0024] Step S22: The feature Perform local restricted complex dynamic routing to obtain the initial capsule The process is:

[0025] Take feature F′4 as the child capsule and the initial capsule F″4 as the parent capsule; let the number of the layer where the child capsule is located be l (l is an integer greater than 1), and its capsule type is That is, there is only one type of capsule; the layer where the parent capsule is located is numbered l+1, and its capsule type is There are 8 capsule types; A total of 8×8 128-dimensional capsules are included; for any type It contains 8×8 32-dimensional capsules, represented as

[0026] for Define a complex window Its center is (x0, y0) and its size is 3×3×128, where x0 is between 1 and 8 and y0 is between 1 and 8; the complex matrix is Its size is 3×3×128×8×32; complex matrices have the same type The complex window is shared between the complex windows; the complex window and the complex matrix are multiplied to obtain the predicted complex vector Right now

[0027]

[0028] Predict all types of child capsules and perform weighted summation to obtain plural parent capsules Right now

[0029]

[0030] in, is the routing coefficient, and the calculation formula is:

[0031]

[0032] in, is the logarithmic prior probability;

[0033] Squeeze the parent capsule to get the complex vector for,

[0034]

[0035] in,

[0036] Coefficient To update,

[0037]

[0038] in,

[0039]

[0040] Step S23: Initial capsule Perform local restricted complex dynamic routing to obtain complex segmentation capsules The process is:

[0041] Take feature F″4 as the child capsule and the initial capsule F4″′ as the parent capsule; let the number of the layer where the child capsule is located be l (l is an integer greater than 1), and its capsule type is That is, there are 8 types of capsules; the layer where the parent capsule is located is numbered l+1, and its capsule type is That is, there is only one type of capsule; for any type It contains 8×8 32-dimensional capsules in total; for It contains 8×8 16-dimensional capsules, represented as

[0042] For any type Define a complex window Its center is (x0, y0) and its size is 1×1×32, where x0 is between 1 and 8 and y0 is between 1 and 8; the complex matrix is Its size is 1×1×32×1×16; complex matrices have the same type The complex window is shared between the complex windows; the complex window and the complex matrix are multiplied to obtain the predicted complex vector Right now

[0043]

[0044] Predict all types of child capsules and perform weighted summation to obtain plural parent capsules Right now

[0045]

[0046] in, is the routing coefficient, and the calculation formula is:

[0047]

[0048] in, is the logarithmic prior probability;

[0049] Squeeze the parent capsule to get the complex vector for,

[0050]

[0051] in,

[0052] Coefficient To update,

[0053]

[0054] in,

[0055]

[0056] Step S24: reduce the dimension of the complex segmentation capsule F4″′ to obtain high-level features F5;

[0057] Step S3: Input the complex capsule network output feature F5 into the complex U-Net decoding network, and after the complex upsampling-convolution layer and the complex convolution layer, obtain the decoding features F3′, F2′, F1′, and the decoding network output feature F′0 in turn. During the decoding process, F′3 is concatenated with F3, F′2 is concatenated with F2, and F1′ is concatenated with F1;

[0058] Step S31: The structure of the complex U-Net decoding network is as follows Figure 2 As shown in the figure, the input is the 16-channel 8×8 feature F5, and after one complex upsampling-convolution operation, the 128-channel 16×16 feature F′3 is obtained. F′3 is concatenated with the encoding network feature, and then three complex convolutions with a step size of 1 and a channel number of 128 are performed to obtain the 128-channel 16×16 feature;

[0059] Step S32: The 128-channel 16×16 feature is subjected to a complex upsampling-convolution operation to obtain a 64-channel 32×32 feature F2′, which is concatenated with the encoding network feature F2, and then subjected to two complex convolutions with a step size of 1 and a channel number of 64 to obtain a 64-channel 32×32 feature;

[0060] Step S33: The 64-channel 32×32 feature is subjected to a complex upsampling-convolution operation to obtain a 32-channel 64×64 feature F1′, which is concatenated with the encoding network feature F1, and then subjected to two complex convolutions with a step size of 1 and a channel number of 32 to obtain a 32-channel 64×64 feature;

[0061] Step S34: The 32-channel 64×64 features are subjected to 1×1 complex convolution to obtain N-channel 64×64 features F′0, where N is the total number of target categories in the polarimetric SAR image;

[0062] Step S4: The complex U-Net decoding network output feature F′0 is subjected to modulo operation and softmax classifier to obtain the semantic segmentation result of the polarimetric SAR image;

[0063] Step S41: The complex U-Net decoding network output feature F′0 is modulo-ed to convert the complex feature map into a real number O′0, that is,

[0064]

[0065] in, and Respectively represent the real part and imaginary part operation;

[0066] Step S42: For the real feature map O′0={O′ 01 ,O′ 02 ,...,O′ 0n ,...O′ 0N} performs softmax classification, where N is the total number of target categories in the polarimetric SAR image, and obtains the probability p at the pixel position (x′, y′) in the nth feature map. n (x,y), that is

[0067]

[0068] The effect of the present invention can be further illustrated by the following simulation.

[0069] 1. Experimental conditions and methods:

[0070] Experimental simulation environment: Keras, Windows 10

[0071] Experimental methods: real number U-Net method, complex number U-Net method and the present invention.

[0072] 2. Experimental content and results analysis:

[0073] Experimental content: The present invention uses the Flevoland dataset and the San Francisco dataset. The present invention, the complex U-Net network method and the real U-Net method are used to perform image semantic segmentation experiments.

[0074] Experimental results analysis: The experimental results obtained by this invention on the Flevoland dataset are as follows: Figure 3 As shown, the results obtained on the SanFrancisco dataset are as follows Figure 4 As shown. Figure 3 , 4 It can be seen that the classification result of the present invention has higher image semantic segmentation performance than the methods of real U-Net and complex U-Net networks.

[0075] The above description is only a specific implementation of the present invention, but the protection scope of the present invention is not limited thereto. Any person familiar with the technology can understand and think of any changes or substitutions within the technical scope disclosed by the present invention, which should be included in the scope of the present invention.

Claims

1. A polarimetric SAR image semantic segmentation method based on complex U-Net and capsule network, the method comprising the following steps: Step S1: The complex U-Net encoding network uses complex convolutional layers to sequentially extract shallow features F1, middle features F2, high-level features F3, and encoded output features F4; Step S2: Input the complex U-Net encoded output feature F4 into the complex capsule network to obtain a high-level feature F5 containing the target posture information; Step S3: Input the complex capsule network output feature F5 into the complex U-Net decoding network, and after the complex upsampling-convolution layer and the complex convolution layer, obtain the decoding features F′3, F′2, F′1, and the decoding network output feature F′0 in turn. During the decoding process, F′3 is concatenated with F3, F′2 is concatenated with F2, and F′1 is concatenated with F1; Step S4: The complex U-Net decoding network output feature F′0 is subjected to modulo operation and softmax classifier to obtain the semantic segmentation result of the polarimetric SAR image.

2. The polarimetric SAR image semantic segmentation method based on complex U-Net and capsule network according to claim 1, characterized in that: The steps of inputting the complex U-Net encoded output feature F4 into the complex capsule network to obtain the high-level feature F5 containing the target posture information are as follows: Step S21: Upscaling the complex U-Net encoded output feature F4 to obtain F′4; Step S22: Perform local restricted complex dynamic routing on the feature F′4 to obtain the initial capsule F″4; Step S23: Perform local restricted complex dynamic routing on the initial capsule F″4 to obtain a complex segmented capsule F4″′; Step S24: Reduce the dimension of the complex segmented capsule F4″′ to obtain a high-level feature F5 containing target posture information.

3. The polarimetric SAR image semantic segmentation method based on complex U-Net and capsule network according to claim 2, characterized in that: The method of locally restricted complex dynamic routing described in step S22 and step S23 is as follows: The capsule before the local restricted complex dynamic routing is taken as the child capsule, and the capsule after the routing is taken as the parent capsule; the number of the layer where the child capsule is located is recorded as l (l is an integer greater than 1), and its capsule type is Among them, N c Represents the total number of child capsule types, and its value is an integer greater than or equal to 1; the layer where the parent capsule is located is numbered l+1, and its capsule type is Among them, N p Indicates the total number of parent capsule types, and its value is an integer greater than or equal to 1; for any type Total includes h l × l z l dimensional capsule; for any type Total includes h l+1 × l+1 z l+1 The capsule of dimension is represented by For any type Define a complex window Its center is (x0, y0) and its size is k h ×k w × l , where x0 is between 1 and h l Between, y0 is between 1 and w l Between h and k w Less than or equal to h l and w l ;Remember the complex matrix Its size is k h ×k w × l ×N p × l+1 ; complex matrices have the same type The complex window is shared between the complex windows; the complex window and the complex matrix are multiplied to obtain the predicted complex vector Right now Predict all types of child capsules and perform weighted summation to obtain plural parent capsules Right now in, is the routing coefficient, and the calculation formula is: in, is the logarithmic prior probability; Squeeze the parent capsule to get the complex vector for, in, Coefficient To update, in, 4. The polarimetric SAR image semantic segmentation method based on complex U-Net and capsule network according to claim 1, characterized in that: The steps of obtaining the semantic segmentation result of polarimetric SAR image by taking the complex U-Net decoding network output feature F0′ and performing modulo operation and softmax classifier are as follows: Step S41: The complex U-Net decoding network output feature F0′ is modulo-ed to convert the complex feature map into a real number O′0, that is, in, and Respectively represent the real part and imaginary part operation; Step S42: For the real feature map O′0={O′ 01 ,O′ 02 ,...,O′ 0n ,...O′ 0N } performs softmax classification, where N is the total number of target categories in the polarimetric SAR image, and obtains the probability p at the pixel position (x′, y′) in the nth feature map. n (x,y), that is

Images