A method for generating ship targets in arbitrary directions in SAR images based on generative adversarial networks

Abstract

The invention discloses a method for generating ship targets in arbitrary directions of SAR images based on a generative adversarial network, comprising the following steps: preprocessing a small number of SAR images containing ship targets in arbitrary directions; introducing position constraints of ship targets in arbitrary directions and introducing pixel value constraints of ship targets in arbitrary directions, extracting features of the SAR images containing ship targets in arbitrary directions and the position constraints and pixel value constraints; fusing the extracted features of the position constraints, the pixel value constraints and the features of the feature vectors to obtain a fused feature map; inputting the fused feature map into a SAR image ship target generation network to generate SAR images of ship targets in arbitrary directions and their labels; designing a loss function in combination with the characteristics of ship targets in SAR images; and using the loss function to train the SAR image ship target generation network. The method provides data support for detecting ship targets in arbitrary directions in SAR images, and has important research value and significance.

Description

Technical Field

[0001] The present invention relates to the field of SAR image processing, and in particular to a method for generating ship targets in arbitrary directions and position labels thereof in SAR images based on a generative adversarial network. Background Art

[0002] Compared with optical and infrared images, SAR images are acquired through active imaging, have good penetration, are not affected by time or climate, and can achieve all-day and all-weather imaging and reconnaissance. SAR images have become an important data source for ship target detection.

[0003] However, the background environment of SAR images containing ship targets is very complex, the shapes of ship targets are irregular, and the boundary features are fuzzy. In order to accurately locate ship targets in SAR images, a large number of SAR images of ship targets containing rotation position labels are needed to train the detector, which takes a lot of time and manpower.

[0004] In recent years, with the rapid development of artificial intelligence and computer vision, image generation methods based on generative adversarial networks have been widely used in image generation tasks. However, most of the existing SAR image generation methods based on generative adversarial networks are [1-5] , neither can generate SAR images and their location labels at the same time. Summary of the invention

[0005] In order to solve the above problems, the present invention provides a technical solution adopted by the present invention: a method for generating ship targets in any direction of a SAR image based on a generative adversarial network, comprising the following steps:

[0006] A small amount of SAR images containing ship targets in any direction are used for preprocessing; the preprocessing process is as follows:

[0007] The position constraint of ship targets in any direction and the pixel value constraint of ship targets in any direction are introduced, and the position constraint, pixel value constraint and feature vector feature extraction are performed on the SAR images containing ship targets in any direction.

[0008] The extracted position constraint features, pixel value constraint features and feature vector features are fused to obtain a fused feature map;

[0009] The fused feature map is input into the SAR image ship target generation network to generate SAR images and labels of ship targets in any direction.

[0010] Based on the characteristics of ship targets in SAR images, a loss function is designed; the loss function is used to train the network for generating ship targets in any direction in SAR images;

[0011] Further: the cascaded SAR image ship target generation adversarial network includes

[0012] A generator for generating SAR images containing ship targets in any direction and a discriminator for performing authenticity identification of the ship targets on the SAR images containing ship targets in any direction generated by the generator;

[0013] The generator includes a low-resolution reconstruction module for reconstructing a low-resolution SAR image of a ship target and a high-resolution reconstruction module for reconstructing a high-resolution SAR image of a ship target;

[0014] The low-resolution reconstruction module and the high-resolution reconstruction module are cascaded.

[0015] Further: the generation method of introducing the constraint of the ship target position in any direction is as follows:

[0016] The design is based on the position labels of the existing SAR images containing ship targets. Based on the position labels of the SAR images containing ship targets, a mask is constructed as a position constraint. In the mask, the pixel values ​​of the ship target and the background are set to 255 and 0 respectively. Through the position constraint, the generated ship target is restricted to a fixed bounding box. Therefore, the bounding box in the position constraint is used as the position label for generating the SAR ship target.

[0017] Further: the method for generating the constraint of the pixel value of the ship in any direction is as follows:

[0018] The design is based on the position label of the existing SAR image ship target and the pixel mean of the target area and the background area. In order to reflect the pixel values ​​of the SAR image ship target and its background, the pixel value constraint is designed using the average value of the pixels. According to the target position label, the pixel values ​​of the ship target and the background are set to the pixel mean of the corresponding area respectively. Through the pixel value constraint of the ship in any direction, the SAR image ship target with the same rotation angle and different pixel values ​​is generated.

[0019] Furthermore: the loss function of the high-resolution reconstruction module is as follows:

[0020]

[0021] Where: I G Represents the real SAR image ship target, I lc and I pvc Represent position constraints and pixel value constraints respectively; Loss GDL Indicates: gradient loss;

[0022]

[0023] Where: (i, j) is the pixel position in the image.

[0024] A device for generating ship targets in any direction of a SAR image based on a generative adversarial network, comprising:

[0025] Preprocessing module: used to perform preprocessing using a small amount of SAR images containing ship targets in any direction; the preprocessing process is as follows:

[0026] The position constraint of ship targets in any direction and the pixel value constraint of ship targets in any direction are introduced, and the position constraint, pixel value constraint and feature vector feature extraction are performed on the SAR image containing ship targets in any direction.

[0027] Fusion module: used to fuse the extracted position constraint features, pixel value constraint features and feature vector features to obtain a fused feature map;

[0028] Generation module: used to input the fused feature map into the SAR image ship target generation network to generate SAR images and labels of ship targets in any direction;

[0029] Training module: used to design loss function based on the characteristics of ship targets in SAR images; use loss function to train the network for generating ship targets in any direction in SAR images;

[0030] Identification module: to identify the authenticity of ship targets in any direction of SAR images generated by the generator

[0031] The present invention provides a method for generating ship targets in arbitrary directions in SAR images based on a generative adversarial network. The present invention is based on a generative adversarial network and limits the position, rotation angle and pixel value of generated ship targets by introducing position constraints and pixel value constraints of ship targets in arbitrary directions, so that the generated SAR image ship targets contain rotation position labels, providing data support for ship target detection in arbitrary directions in SAR images, and has important research value and research significance. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0033] Figure 1 is a flow chart of the method;

[0034] Figure 2 Generate a network structure diagram for the ship target in cascaded SAR images;

[0035] Figure 3 (a) is the structure diagram of the upsampling module, (b) is the structure diagram of the downsampling module, and (c) is the structure diagram of the output module;

[0036] Figure 4 (a) is the position annotation map, (b) is the position constraint map constructed based on the position labels, and (c) is the pixel value constraint map constructed based on the position labels;

[0037] Figure 5 (a) is the constraint diagram under the first pixel value; (b) is the SAR image generated based on the first pixel value constraint; (c) is the constraint diagram under the second pixel value; (d) is the SAR image generated based on the second pixel value constraint. DETAILED DESCRIPTION

[0038] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0039] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. The following description of at least one exemplary embodiment is actually only illustrative and is by no means intended to limit the present invention and its application or use. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0040] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0041] Unless otherwise specifically stated, the relative arrangement of the parts and steps described in these embodiments, the numerical expressions and numerical values ​​do not limit the scope of the present invention. At the same time, it should be clear that, for ease of description, the sizes of the various parts shown in the drawings are not drawn according to the actual proportional relationship. The technology, methods and equipment known to ordinary technicians in the relevant field may not be discussed in detail, but in appropriate cases, the technology, methods and equipment should be regarded as part of the authorization specification. In all examples shown and discussed here, any specific value should be interpreted as merely exemplary, rather than as a limitation. Therefore, other examples of exemplary embodiments may have different values. It should be noted that similar numbers and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further discussed in subsequent drawings.

[0042] In the description of the present invention, it is necessary to understand that the directions or positional relationships indicated by directional words such as "front, back, up, down, left, right", "lateral, vertical, perpendicular, horizontal" and "top, bottom" are usually based on the directions or positional relationships shown in the drawings. They are only for the convenience of describing the present invention and simplifying the description. Unless otherwise specified, these directional words do not indicate or imply that the device or element referred to must have a specific direction or be constructed and operated in a specific direction. Therefore, they cannot be understood as limiting the scope of protection of the present invention: the directional words "inside and outside" refer to the inside and outside relative to the contours of each component itself.

[0043] For ease of description, spatially relative terms such as "above", "above", "on the upper surface of", "above", etc. may be used here to describe the spatial positional relationship between a device or feature and other devices or features as shown in the figure. It should be understood that spatially relative terms are intended to include different orientations of the device in use or operation in addition to the orientation described in the figure. For example, if the device in the accompanying drawings is inverted, the device described as "above other devices or structures" or "above other devices or structures" will be positioned as "below other devices or structures" or "below their position devices or structures". Thus, the exemplary term "above" can include both "above" and "below". The device can also be positioned in other different ways (rotated 90 degrees or in other orientations), and the spatially relative descriptions used here are interpreted accordingly.

[0044] In addition, it should be noted that the use of terms such as "first" and "second" to limit components is only for the convenience of distinguishing the corresponding components. If not otherwise stated, the above terms have no special meaning and therefore cannot be understood as limiting the scope of protection of the present invention.

[0045] Figure 1is a flow chart of the method;

[0046] A method for generating ship targets in arbitrary directions of SAR images based on a generative adversarial network comprises the following steps:

[0047] S1: Acquire a SAR image containing ship targets in any direction;

[0048] S2: Preprocessing the SAR image containing ship targets in any direction; the preprocessing process is as follows:

[0049] The position constraint of ship targets in any direction and the pixel value constraint of ship targets in any direction are introduced, and the position constraint, pixel value constraint and feature vector feature extraction are performed on the SAR image containing ship targets in any direction.

[0050] S3: Fusing the extracted position constraint features, pixel value constraint features and feature vector features to obtain a fused feature map;

[0051] S4: input the fused feature map into the cascaded SAR image ship target generation adversarial network, and train the cascaded SAR image ship target generation adversarial network to generate a SAR image containing ship targets in any direction;

[0052] S5: Design a loss function based on the characteristics of ship targets in SAR images; use the loss function to construct a cascaded SAR image ship target generative adversarial network based on the generative adversarial network.

[0053] The steps S1 / S2 / S3 / S4 / S5 are performed in order;

[0054] Figure 2 Generate a network structure diagram for the ship target in cascaded SAR images;

[0055] Figure 3 (a) is the structure diagram of the upsampling module, (b) is the structure diagram of the downsampling module, and (c) is the structure diagram of the output module;

[0056] Figure 4 (a) is the position annotation map, (b) is the position constraint map constructed based on the position labels, and (c) is the pixel value constraint map constructed based on the position labels;

[0057] Furthermore: the eigenvector z conforms to the standard normal distribution and has a dimension of 100×1×1.

[0058] The preprocessing is performed by an upsampling module and a downsampling module;

[0059] In order to reconstruct the feature vector into a feature map of size 8×8, two Figure 3The upsampling module shown in (a) performs upsampling operations on the feature vector. The upsampling module is mainly composed of transposed convolution and 1×1 convolution, and adopts a residual structure, which can bring more information to deeper layers and effectively alleviate the gradient vanishing problem. Figure 2 In the upsampling module, (4, 1, 0) indicates that the transposed convolution size, stride, and padding value are set to 4, 1, and 0, respectively.

[0060] Figure 4 (a) is the position annotation map, (b) is the position constraint map constructed based on the position labels, and (c) is the pixel value constraint map constructed based on the position labels;

[0061] Further: the generation method of introducing the constraint of the ship target position in any direction is as follows:

[0062] The design is based on the location tags of existing SAR images containing ship targets, such as Figure 4 As shown in (a), based on the position label of the SAR image containing the ship target, a mask is constructed as a position constraint. In the mask, the pixel values ​​of the ship target and the background are set to 255 and 0, respectively, as shown in Figure 4 As shown in (b), the generated ship target is restricted to a fixed bounding box through position constraints. Therefore, the bounding box in the position constraint is used as the position label for generating the SAR ship target.

[0063] In order to extract the features of position constraints, the output feature map is 8×8, and three Figure 3 The downsampling module shown in (b) performs downsampling operations on the position constraints. The downsampling module is mainly composed of convolutions with a step size of 2 and 1×1 convolutions, and adopts a residual structure design. Figure 2 In the figure, (4, 2, 1) in the downsampling module indicates that the convolution size, stride, and padding value are set to 4, 2, and 1, respectively.

[0064] Further: the method for generating the constraint of the pixel value of the ship in any direction is as follows:

[0065] The design is based on the existing SAR image ship target position label and the pixel mean of the target area and the background area; in order to reflect the pixel value of the SAR image ship target and its background, the pixel value constraint is designed using the pixel mean value. According to the target position label, the pixel values ​​of the ship target and the background are set to the pixel mean of the corresponding area, such as Figure 3 As shown in (c), by constraining the pixel value of the ship in any direction, a SAR image of a ship target with the same rotation angle and different pixel values ​​is generated.

[0066] In order to extract the features of pixel value constraints, the output feature map is 8×8, and three Figure 3The downsampling module shown in (b) performs downsampling operations on the position constraints.

[0067] The feature vector, position constraint and pixel value constraint feature map are extracted and fused using concatenation operation as the input of the SAR image ship target generation network.

[0068] The present invention adopts an iterative minimum-maximum game method and utilizes a generative adversarial network (GAN) to construct a SAR image ship target generation network, including a SAR image arbitrary direction ship target generator and a discriminator. Figure 2 shown.

[0069] Further: the cascaded SAR image ship target generation adversarial network includes a generator for generating SAR images containing ship targets in any direction and a discriminator for performing authenticity identification of ship targets on the SAR images containing ship targets in any direction generated by the generator;

[0070] The generator includes a low-resolution reconstruction module for reconstructing a low-resolution SAR image of a ship target and a high-resolution reconstruction module for reconstructing a high-resolution SAR image of a ship target;

[0071] The low-resolution reconstruction module and the high-resolution reconstruction module are cascaded to gradually improve the quality of the generated SAR image ship target, such as Figure 2 As shown in Figure 2, the low-resolution part consists of two upsampling modules and an output module (such as Figure 3 The high-resolution part consists of an upsampling module and an output module.

[0072] For the discriminator, this application uses 4 downsampling modules and 1 convolution to output a 4×4 feature map matrix.

[0073] Furthermore, for the loss function, the generator G attempts to maximize Loss GAN , the discriminator D tries to minimize Loss GAN , as shown in formula (1).

[0074]

[0075] Among them, I G represents the real SAR image ship target, z represents the input feature vector, I lc and I pvc Represent position constraints and pixel value constraints respectively.

[0076] In the low-resolution reconstruction part, the L1 loss of formula (2) is used to learn the global similarity of ship targets in real SAR images;

[0077]

[0078] Among them, G lr-part (·) represents the low-resolution reconstruction part in the generator.

[0079] The shape of the ship target in the image is irregular and the boundary is blurred. In the high-resolution reconstruction part, in addition to the L1 loss, the present invention adds a gradient loss to sharpen the output reconstructed SAR image.

[0080]

[0081] Among them, Loss GDL Indicates: gradient loss; I P =G hr-part (z,I lc ,I pvc ). hr-part (·) represents the high-resolution reconstruction part of the generator, where (i, j) is the pixel position in the image.

[0082] Therefore, the loss function of the high-resolution reconstruction part can be expressed as:

[0083]

[0084] The total loss function can be expressed as:

[0085]

[0086] Among them, λ=10.

[0087] A device for generating ship targets in any direction of a SAR image based on a generative adversarial network, comprising:

[0088] Preprocessing module: used to perform preprocessing using a small amount of SAR images containing ship targets in any direction; the preprocessing process is as follows:

[0089] Introducing position constraints and pixel value constraints of ship targets in any direction, and extracting features of SAR images containing ship targets in any direction with position constraints and pixel value constraints;

[0090] And extract the feature vector of the SAR image containing the ship target in any direction;

[0091] Fusion module: used to fuse the extracted position constraint features, pixel value constraint features and feature vector features to obtain a fused feature map;

[0092] Training module: used to design loss function based on the characteristics of ship targets in SAR images; use loss function to train the network for generating ship targets in any direction in SAR images;

[0093] Generation module: used to input the fused feature map into the SAR image ship target generation network to generate SAR images and labels of ship targets in any direction;

[0094] Identification module: It is used to identify the authenticity of ship targets in any direction of the SAR images generated by the generator.

[0095] The SAR image output by the SAR image arbitrary direction ship target generation method based on generative adversarial network of the present invention can be directly input into the SAR image arbitrary direction ship target detector. In order to prove that the generated SAR ship image is beneficial to the arbitrary direction SAR image ship target detection task, the generated image and the real image are respectively sent to the SAR image arbitrary direction ship target detector (single-stage detector Rotated RetinaNet and double-stage detector Oriented RCNN), and the detection results are compared. The results are as follows:

[0096] method Input size AP Oriented RCNN 64×64 81.74％ The present invention + Oriented RCNN 64×64 90.76％ Rotated RetinaNet 64×64 90.21％ Invention + Rotated RetinaNet 64×64 93.89％

[0097] Experimental results show that, no matter it is a single-stage target detector or a multi-stage target detector, the present invention can improve the detection performance of ship targets in SAR images by generating ship targets in any direction in the SAR image. Figure 5 (a) is the constraint diagram under the first pixel value; (b) is the SAR image generated based on the first pixel value constraint; (c) is the constraint diagram under the second pixel value; (d) is the SAR image generated based on the second pixel value constraint. Figure 5 As shown, the pixel values ​​of the generated SAR image ship targets can be changed by changing the input pixel value constraints, so as to diversify the generated SAR image ship targets.

[0098] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

[0099] [1]Z.Lu,X.Jiang,A.Kot.Enhance deep learning performance in facerecognition[C] / / Proceedings of 2017 2ndInternational Conference on Image,Vision and Computing,2017:244-248.

[0100] [2]H.Zhang,M.Cisse,Y.N.Dauphin,D.Lopez-Paz.Mixup:beyond empiricalriskminimization[C] / / Proceedings of International Conference on LearningRepresentations,2018.

[0101] [3]Z.Zhong,L.Zheng,G.Kang,S.Li,Y.Yang.Random erasing dataaugmentation[C] / / Proceedings of theAAAI Conference on ArtificialIntelligence,2020,34:13001-13008.

[0102] [4]Q.Lu,H.Jiang,G.Li,W.Ye.Data augmentation method of SAR imagedataset based on Wassersteingenerative adversarial networks[C] / / 2019International Conference on Electronic Engineering andInformatics,2019:488-490.

[0103] [5]K.Zhao,Y.Zhou,X.Chen,H.Zhang.A generative adversarial networkbased image augmentation methodfor ship segmentation in SAR images[C] / / 2020IEEE 9th Joint International Information Technology andArtificialIntelligence Conference,2020,9:1285-1289.

Claims

1. A method for generating ship targets in arbitrary directions in SAR images based on generative adversarial networks. Features: The following steps are involved: A small amount of SAR images containing ship targets in any direction are used for preprocessing; the preprocessing process is as follows: The position constraint of ship targets in any direction and the pixel value constraint of ship targets in any direction are introduced, and the position constraint, pixel value constraint and feature vector feature extraction are performed on the SAR image containing ship targets in any direction. The extracted position constraint features, pixel value constraint features and feature vector features are fused to obtain a fused feature map; The fused feature map is input into the SAR image arbitrary direction ship target generation adversarial network to generate SAR images of ship targets in arbitrary directions and their labels; Based on the characteristics of ship targets in SAR images, a loss function is designed; the loss function is used to train the generative adversarial network for ship targets in arbitrary directions in SAR images.

2. According to the method for generating ship targets in any direction in SAR images based on generative adversarial networks in claim 1, Features: The SAR image arbitrary direction ship target generation adversarial network includes A generator for generating SAR images containing ship targets in any direction and a discriminator for performing authenticity identification of the ship targets on the SAR images containing ship targets in any direction generated by the generator; The generator includes a low-resolution reconstruction module for reconstructing a low-resolution SAR image of a ship target and a high-resolution reconstruction module for reconstructing a high-resolution SAR image of a ship target; The low-resolution reconstruction module and the high-resolution reconstruction module are cascaded.

3. According to the method for generating ship targets in any direction in SAR images based on generative adversarial networks in claim 1, Features: The generation method of introducing the constraint of the ship target position in any direction is as follows: The design is based on the position labels of the existing SAR images containing ship targets. Based on the position labels of the SAR images containing ship targets, a mask is constructed as a position constraint. In the mask, the pixel values ​​of the ship target and the background are set to 255 and 0 respectively. Through the position constraint, the generated ship target is restricted to a fixed bounding box. Therefore, the bounding box in the position constraint is used as the position label for generating the SAR ship target.

4. According to the method for generating ship targets in any direction in SAR images based on generative adversarial networks in claim 1, Features: The method for generating the pixel value constraint of the ship target in any direction is as follows: The design is based on the position label of the existing SAR image ship target and the pixel mean of the target area and the background area. In order to reflect the pixel values ​​of the SAR image ship target and its background, the pixel value constraint is designed using the average value of the pixels. According to the target position label, the pixel values ​​of the ship target and the background are set to the pixel mean of the corresponding area respectively. Through the pixel value constraint of the ship in any direction, the SAR image ship target with the same rotation angle and different pixel values ​​is generated.

5. According to claim 2, a method for generating ship targets in any direction in SAR images based on a generative adversarial network, Features: The loss function of the high-resolution reconstruction module is as follows: （4） in: Represents the real SAR image ship target, and Represent position constraints and pixel value constraints respectively; Indicates: gradient loss; （3） Where: (i, j) is the pixel position in the image.

6. A device for generating ship targets in any direction in SAR images based on generative adversarial networks. Features: include: Preprocessing module: used to perform preprocessing using a small amount of SAR images containing ship targets in any direction; the preprocessing process is as follows: The position constraint of ship targets in any direction and the pixel value constraint of ship targets in any direction are introduced, and the position constraint, pixel value constraint and feature vector feature extraction are performed on the SAR image containing ship targets in any direction. Fusion module: used to fuse the extracted position constraint features, pixel value constraint features and feature vector features to obtain a fused feature map; Generation module: used to input the fused feature map into the SAR image arbitrary direction ship target generation adversarial network to generate SAR images of ship targets in arbitrary directions and their labels; Training module: used to design loss function based on the characteristics of ship targets in SAR images; use loss function to train the generative adversarial network of ship targets in any direction in SAR images; Identification module: It is used to identify the authenticity of ship targets in any direction of the SAR images generated by the generator.

Images