Training sample generation method applied to ship satellite image recognition

Abstract

The embodiment of the invention provides a training sample generation method applied to ship satellite image recognition. The method aims at solving the problem that in the prior art, the performance of a ship satellite image recognition model is reduced due to domain offset and spatial disturbance. According to the method, a novel sample generation network is constructed, and a large number of samples with different styles and spatial disturbances can be generated on the basis of limited samples, so that the recognition capability and robustness of a ship satellite image recognition model are effectively improved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a method for generating training samples for ship satellite image recognition. Background Art

[0002] As deep neural networks demonstrate powerful data representation and data processing capabilities, ship target detection and recognition based on deep neural networks has made breakthroughs and has become an important research direction in both civil and military fields. In practical applications, samples to be recognized often have the characteristics of domain offset and spatial disturbance. The recognition ability of the ship satellite image recognition model depends on the diversity of training samples.

[0003] The existing technical solution is to use the acquired ship synthetic aperture radar (SAR) images as training data, construct the objective function of the deep learning model, and maximize the loss function of the constructed objective function to generate adversarial samples and perform data enhancement on the training samples. This technical solution slightly perturbs the image to generate adversarial samples, but the perturbation of the deep learning model to the image is random and uncontrollable, and it is difficult to clearly determine the form of the perturbation. Summary of the invention

[0004] The training sample generation method proposed in this patent can generate a large number of samples with different characteristics based on limited samples for training recognition models, which can effectively improve the recognition ability of the ship satellite image recognition model.

[0005] An embodiment of the present invention provides a method for generating training samples for ship satellite image recognition, the method comprising:

[0006] Step 1: divide the collected satellite images of ships into different styles according to the background color;

[0007] Step 2: Divide the image sets of different styles into groups of two to form a source domain image set and a target domain image set;

[0008] Step 3: Construct a sample generation network combining Transformer, CNN and RNN;

[0009] Step 4, training the sample generation network, using the generator to transform the image style, and using the transformed image for style transfer;

[0010] Step 5, applying a spatial transformation operation to the intermediate image and the original image to generate new samples with different spatial perturbations;

[0011] Step 6, using the trained sample generation network to generate samples of different styles;

[0012] Step 7: Annotate the generated samples and expand them into the original training dataset.

[0013] In some embodiments, the style division in step 1 adopts K-means clustering algorithm combined with manual screening.

[0014] In some embodiments, the sample generation network proposed in step 3 is based on the idea of ​​CycleGAN, combining three network structures: Transformer, CNN and RNN.

[0015] In some embodiments, the training process in step 4 includes utilizing a generator and a discriminator, and optimizing the network through adversarial loss, cycle consistency loss, identity mapping loss, style consistency loss, and maximum mean difference loss.

[0016] In some embodiments, the intermediate image in step 5 is obtained when calculating the cycle consistency loss and the identity mapping loss; the spatial transformation operations include adding noise, image rotation, image scaling, image translation, image shearing, image affine transformation and image multi-angle rotation.

[0017] In some embodiments, the samples generated in step 6 are used to expand the original training data set to improve the recognition ability and robustness of the ship satellite image recognition model.

[0018] Compared with the prior art, the present invention has the following advantages:

[0019] The generated samples are highly diverse, effectively improving the model's recognition capabilities.

[0020] The style transferred images are of good quality and rich in details.

[0021] Generate a large number of samples to solve the finite sample problem. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The drawings illustrate generally, by way of example and not limitation, various embodiments discussed herein.

[0023] Figure 1 A flow chart of the present invention is shown;

[0024] Figure 2 Demonstrated the process of image segmentation and dataset construction;

[0025] Figure 3 The sample generation network framework diagram is shown;

[0026] Figure 4 The generator structure in the sample generation network is shown;

[0027] Figure 5 The discriminator structure in the sample generation network is shown;

[0028] Figure 6 The Transformer encoder structure is shown;

[0029] Figure 7 A simplified flow chart of the present invention is presented. DETAILED DESCRIPTION

[0030] In order to enable a more detailed understanding of the features and technical contents of the embodiments of the present application, the implementation of the embodiments of the present application is described in detail below in conjunction with the accompanying drawings. The attached drawings are for reference only and are not used to limit the embodiments of the present application.

[0031] In the embodiments of the present application, it should be noted that, unless otherwise specified and limited, the term "connection" should be understood in a broad sense. For example, it can be an electrical connection or a connection between two components. It can be a direct connection or an indirect connection through an intermediate medium. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to the specific circumstances.

[0032] It should be noted that the terms "first\second\third" involved in the embodiments of the present application are only used to distinguish similar objects, and do not represent a specific order for the objects. It is understandable that the specific order or sequence of "first\second\third" can be interchanged where permitted. It should be understood that the objects distinguished by "first\second\third" can be interchanged where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein.

[0033] Domain shift, in the present invention, refers to the different feature distributions of source domain (training) images and target domain (test) images. The deep neural network trained and fitted on the source domain image set cannot adapt to the characteristics of the target domain images, resulting in a decrease in network performance.

[0034] Spatial perturbation, in the present invention, refers to performing various spatial transformation operations on an image to simulate various changes that may occur in actual application scenarios.

[0035] Spatial transformation, through adding noise, image rotation, image scaling, image translation, image shearing, image affine transformation and other spatial transformation operations of training samples, achieves spatial perturbation of the image.

[0036] See also Figure 1 The present invention includes six parts: dividing ship satellite images, constructing a data set, constructing a sample generation network combining Transformer, CNN and RNN, training a sample generation network combining Transformer, CNN and RNN, style transfer and spatial transformation of ship satellite images, and expanding training data using generated samples.

[0037] See also Figure 1 The present invention includes six parts: dividing ship satellite images, constructing a data set, constructing a sample generation network combining Transformer, CNN and RNN, training a sample generation network combining Transformer, CNN and RNN, style transfer and spatial transformation of ship satellite images, and expanding training data using generated samples.

[0038] The specific steps for image segmentation and dataset construction are as follows:

[0039] Step 1: Ship satellite image segmentation

[0040] Combining intelligent methods and manual screening methods, the collected satellite images of ships are classified into different styles according to the background color. Figure 2 As shown in the figure, we first analyze the number of different background colors in the ship satellite images, use this number as the K value of the K-means clustering algorithm, and perform K-means clustering on all ship satellite images. Then, we manually check the clustering results and use the manual screening method to put the images with incorrect clustering into the correct classification, and obtain several image sets divided according to different background styles.

[0041] Step 2: Build the dataset

[0042] After completing the division of the ship satellite image, Figure 2 As shown, the image sets of different styles are divided into two groups, and the two types of images in each group are used as the source domain image set and the target domain image set Where N s and N t are the number of source domain images and target domain images respectively. In each group, there is no restriction on which category is used as the source domain and which category is used as the target domain. In the network training process, the present invention simultaneously inputs the source domain images and the target domain images.

[0043] After each group completes the generation of training samples based on steps 3 to 6 of the present invention, the next group of source domain image data sets and target domain image data sets are input until all groups complete the generation of training samples.

[0044] Step 3: Build a sample generation network combining Transformer, CNN and RNN

[0045] The proposed sample generation network combining Transformer, CNN and RNN is based on the idea of ​​CycleGAN, and combines the three network structures of Transformer, CNN and RNN in the network architecture to make them complement each other. Transformer uses the attention mechanism to capture global features and can obtain more representative image features; CNN has a small receptive field and is easy to obtain local features; and RNN regards multi-scale features as sequence data and can effectively fuse multi-scale features in combination with contextual information. Therefore, the sample generation network proposed in the present invention combines Transformer, CNN and RNN, which can more effectively capture and process global feature information and local feature information.

[0046] like Figure 3 As shown, the proposed network framework includes: Generator G s2t and G t2s , respectively used to transform the style of the image, that is, G s2t : X→Y and G t2s : Y→X; Discriminator D s and D t , which are used to distinguish the style of the source domain image and the target domain image respectively; the intermediate image generated when calculating the loss function, the present invention will apply spatial transformation operations such as translation, distortion, noise, multi-angle rotation, etc. to the intermediate image and the original image.

[0047] Generator G s2t and G t2s The network structure is as follows Figure 4 As shown in Figure 2, the network structures of the two generators are the same, but the parameters are different. In the feature encoding stage, the image is input into L Transformer blocks and a CNN with L encoding layers, and L feature maps of different sizes are obtained. and Among them, V i Dimensions and C i The size of is the same, by fusing the feature V i and C i get In the feature decoding stage, based on the feature and The features of different scales are regarded as sequences, and the ConvGRU algorithm combining RNN and CNN is used to fuse the multi-scale feature information of the context, and finally the generated image P is obtained. L .

[0048] Discriminator D s and D t The network structure is as follows Figure 5As shown in the figure, the size of the output feature map is marked, and the final output is a probability value with a discrimination result range of [0, 1]. The discriminator is used to judge the authenticity of the input image. The discrimination result of the real original image should be a higher probability value close to 1, and the discrimination result of the generated image should be a lower probability value close to 0. During the training process of the discriminator, the difference between the generated image and the real image is minimized to improve the discriminant ability of the discriminator.

[0049] The network structures of the two discriminators are the same, but the parameters are different. The image is input into CNN and Transformer respectively, and the output features of the two networks are fused to finally obtain the discrimination result of the image style. Inspired by the idea of ​​PatchGAN, the CNN of the discriminator of the present invention is based on the structure of PatchGAN, and its output size is 30×30×1. Each element of the matrix corresponds to a small block (pathch) of the input image, and the element value is the probability of the authenticity of the block; the Transformer of the discriminator of the present invention divides the input image into 16 small blocks for feature analysis, and uses a convolution block to make its output size the same as that of CNN. Therefore, the discriminator can focus on each local area of ​​the image, and the output of CNN and Transformer combined can fully cover the image area and improve the discrimination accuracy.

[0050] Step 4: Train a sample generation network combining Transformer, CNN and RNN

[0051] Step 4.1: Use the generator to transfer image style

[0052] like Figure 3 As shown in the figure, the source domain image X and the target domain image Y are respectively input into the proposed sample generation network combining Transformer, CNN and RNN, and the generator G is used to generate s2t Convert X into the target domain style image G s2t (X), using the generator G t2s Convert Y into the target domain style image G t2s (Y), the specific process of image generation is as follows Figure 4 shown.

[0053] In the encoding stage, the image is input into a CNN with L encoding layers and L Transformer blocks respectively, and L RNN-based ConvGRU modules are used for decoding. According to computing resources and actual needs, a suitable CNN-based backbone network can be selected as the encoder, such as the widely used U-Net, VggNet and ResNet, etc., to effectively obtain the local feature information of the image. The parameters of the L Transformer blocks are different, and the output feature sizes are different, which is designed to effectively obtain the global feature information of the image and merge it with the features extracted by CNN. The existing typical Transformer encoder structure is as follows: Figure 6 shown.

[0054] In each Transformer block, the input image is first divided into 16 small blocks, input to the embedding layer, and each small block is mapped into a one-dimensional vector through linear mapping, and then input to the Transformer encoder composed of 6 stacked Transformer encoding blocks, and finally the feature map of the image is output. The tth (t∈{2, 3, 4, 5, 6}) encoding block receives the output feature v of the previous encoding block t-1 As input, after layer normalization (LN) and multi-head self-attention (MSA), it is combined with the input v in the residual connection t-1 Add them together to get v' t Then the input layer is normalized and the multi-layer perceptron (MLP) is connected with v' in the residual connection t Add to get v t The output feature v t The calculation formula is as follows:

[0055] v' t =MSA(LN(v t-1 ))+v t-1

[0056] v t =MLP(LN(v' t ))+v' t

[0057] The feature output of the last encoding block in the i-th Transformer block is V i , the feature output by the i-th encoding layer of CNN is C i The present invention converts V i and C iThe concatenated features are processed using a network structure consisting of a convolutional layer, batch normalization, and a rectified linear unit activation function (ReLU) to obtain the same features as C i Same size F i .

[0058] In the feature decoding stage, the present invention targets feature mapping and L improved ConvGRU modules are used to effectively obtain spatial features and fuse multi-scale feature information. The original ConvGRU module accepts two input features, while the input received by the i-th (i>1) ConvGRU module in the present invention is the feature P output by the i-1th ConvGRU module. i-1 , the fusion feature F of the encoder L-i and the encoder’s CNN features C L-i , where C L-i It helps the network process deep features and improve the network's representation ability. The input P0 received by the first ConvGRU module is a vector initialized to all zeros. The network's final output P L is the generated image after style transfer. i The calculation formula is as follows:

[0059]

[0060]

[0061]

[0062]

[0063] Among them, G r Represents the reset gate, G z represents the update gate, σ(·) and δ(·) represent two activation functions, W represents the weight, f(·) represents two times upsampling, represents the convolution operation, represents a concatenation operation, and ⊙ represents a point-by-point multiplication operation.

[0064] Step 4.2: Use the discriminator to identify the image style

[0065] like Figure 5 As shown, the discriminator identification process of the present invention is exemplified as follows:

[0066] (1) Input the real original image into the corresponding discriminator, and the discrimination result should be a high probability value close to 1.

[0067] (2) The image generated by the generator is input into the corresponding discriminator, and the discrimination result should be a low probability value close to 0. The closer the discrimination result is to the discrimination result of the real original image, the more realistic the generated image is; if there is a gap between the discrimination result and the discrimination result of the real image, it means that the authenticity of the generated image is not good.

[0068] (3) The specific process of using the discriminator to obtain the image discrimination result is as follows: input the ship satellite RGB image of size 256×256×3 into CNN to obtain a 30×30×1 feature map; input it into Transformer to obtain a 64×1024×1 feature map, and then obtain a 30×30×1 feature map after convolution operation. Add the two 30×30×1 feature maps, pass through the classification layer, and obtain the final discrimination result. The feature fusion of CNN and Transformer effectively improves the discrimination accuracy. Among them, the structure of Transformer is as follows: Figure 6 As shown, the structure is the same as that of the Transformer in the generator, but the parameters are different.

[0069] Step 4.3: Calculate the loss function

[0070] The loss function reflects the difference between the network output and the actual result. During the training process, the gradient information of the loss function can be used to continuously optimize the network parameters through the back propagation algorithm to gradually reduce the loss function.

[0071] In order to combine the advantages of different loss functions and achieve more comprehensive network optimization during the training process, the present invention targets the generator and uses adversarial loss function to optimize the network. Cycle consistency loss Identity Mapping Loss Based on this, we propose to combine style consistency loss and maximum mean difference loss The loss function can ensure the stability and style transfer ability of the generator and further improve the quality of image generation.

[0072] When training the proposed sample generation network, the loss function calculated by the present invention is specifically described as follows.

[0073] Introducing the existing adversarial loss in the sample generation network combining Transformer and CNN The calculation formula is as follows:

[0074]

[0075]

[0076] The style-transferred G s2t (X) Input Gt2s , G t2s (Y) Input G s2t , calculate the cycle consistency loss To constrain cycle consistency between generators:

[0077]

[0078] Among them, P X and P Y are the distributions of generated images in the target domain and the source domain, respectively.

[0079] Enter X into G t2s ,Y input G s2t , calculate the identity mapping loss To ensure the robustness of the generator:

[0080]

[0081] In order to further ensure the stability of the generator, this paper proposes a style consistency loss based entirely on the images generated by the generator. The calculation formula is as follows:

[0082]

[0083] Obviously, the feature distribution of the new image obtained after the style transfer of the source domain image should be the same as that of the target domain image; conversely, the feature distribution of the new image obtained after the style transfer of the target domain image should be the same as that of the source domain image. To minimize the difference in feature distribution between the generated image and the original image:

[0084]

[0085] Where F(·) represents a feature extractor composed of a convolutional neural network, φ(·) represents a Gaussian kernel function, the superscript H represents mapping the kernel function to the reproducing kernel Hilbert space, and ||·|||2 represents the calculation of the bi-norm.

[0086] Combining the above loss functions, calculate the total loss function of the generator: Update G t2s and G s2t Parameters, keep D s and D t The parameters remain unchanged.

[0087] Calculate the discriminator’s D s Loss Function Update D s Parameters, keep D t , Gt2s and G s2t The parameters remain unchanged.

[0088] Calculate the discriminator’s D t The loss function Update D t Parameters, keep D s , G t2s and G s2t The parameters remain unchanged.

[0089] Repeat the training process in step 4 above until the iteration of network training is completed.

[0090] Step 5: Generate samples with different spatial perturbations

[0091] The proposed sample generation network combining Transformer, CNN and RNN has been fully trained in the early stage, and the generator has a relatively mature image style transfer capability. In the last iteration of the network training process in step 4, the cycle consistency loss is calculated When storing the intermediate image G t2s (G s2t (X)) and G s2t (G t2s (Y)); Compute the identity mapping loss When storing the intermediate image G t2s (X) and G s2t (Y). Since these intermediate images have undergone a series of transformations by the generator, they are not completely consistent with the original images, but they still maintain the style of the original images and can be used well to generate training samples.

[0092] For the intermediate image and the original image, random spatial transformations such as adding noise, image rotation, image scaling, image translation, image shearing, image affine transformation, and image multi-angle rotation are performed to generate new training samples.

[0093] Step 6: Generate samples of different styles

[0094] The source domain image set X and the target domain image set Y are input into a trained sample generation network combining Transformer, CNN and RNN, respectively using G s2t and G t2s Output style transferred G s2t (X) and G t2s (Y) as training samples of different styles.

[0095] Step 7: Expand training samples

[0096] The training samples generated in step 5 and step 6 are labeled and put into the original training data set to complete the training sample expansion.

[0097] For the entire invention, the preparatory steps for image data preprocessing are step 1 and step 2.

[0098] When performing style transfer, the network training needs to be completed, and when performing spatial transformation, the network training needs to be completed and the image style transfer needs to be completed. Therefore, after step 4 is completed, the network training is completed. After saving the network, it is not necessary to retrain the network every time. The source domain and target domain images can be directly input to achieve the style transfer and spatial transformation of ship satellite images and obtain training samples.

[0099] The simplified process of the present invention is as follows Figure 7 As shown. The core module of the present invention is a proposed sample generation network combining Transformer, CNN and RNN. Compared with the existing methods, on the one hand, it can make the three networks of Transformer, CNN and RNN complement each other, effectively capture the global and local features of the image, and fully combine the multi-scale feature information of the image in the encoding and decoding stages; on the other hand, it optimizes the calculation formula of the generator loss function, improves the network performance, and improves the quality of generated samples. Therefore, the sample generation network proposed in the present invention can better realize image style transfer and generate samples with different styles. Compared with the existing methods that only analyze and use the original input image, the present invention also innovatively uses the intermediate image to generate samples with different spatial perturbations.

[0100] In step 1, before image style classification, it is necessary to screen out satellite images that do not contain ship targets. Combining algorithms and manual work, clustering algorithms are first used to quickly classify images according to their background styles, and then manual screening methods are used to check for misclassifications and fill in the gaps.

[0101] In step 2, the images of different styles are divided into groups of two, and the sample generation network combining Transformer, CNN and RNN does not limit which group of images is used as source domain or target domain input, and each group generates a batch of samples for expanding training data.

[0102] The source domain image and target domain image input in step 6 are consistent with the source domain image and target domain image input in step 4.

[0103] The perturbation forms of image samples are diversified. According to the actual characteristics of ship satellite images, the present invention realizes the style transfer and spatial transformation of images, generates training samples with different styles and spatial perturbations, enriches the diversity of training samples, and effectively improves the recognition ability of the ship satellite image recognition model applied to different actual scenes.

[0104] The style-transferred image has high quality. The present invention aims to generate training samples with rich details and better quality by proposing a sample generation network combining Transformer, CNN and RNN, taking advantage of the advantages of different network structures to fully capture the global and local features of the image, and optimizing the loss function of the generator.

[0105] A large number of training samples are generated. The present invention aims to solve the problem that limited samples limit the training and recognition capabilities of the recognition model. The spatial transformation method is used to quickly and massively generate training samples with different spatial perturbations, and the proposed sample generation network is used to efficiently generate training samples with different styles. Using the expanded sample set to train the ship satellite image recognition model can effectively improve the stability and robustness of the recognition model.

[0106] The technical solutions described in the embodiments of the present application can be combined arbitrarily without conflict.

[0107] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A training sample generation method for ship satellite image recognition, characterized in that: The method comprises: Step 1: divide the collected satellite images of ships into different styles according to the background color; Step 2: Divide the image sets of different styles into groups of two to form a source domain image set and a target domain image set; Step 3: Construct a sample generation network combining Transformer, CNN and RNN; Step 4, training the sample generation network, using the generator to transform the image style, and using the transformed image for style transfer; Step 5, applying a spatial transformation operation to the intermediate image and the original image to generate new samples with different spatial perturbations; Step 6, using the trained sample generation network to generate samples of different styles; Step 7: Annotate the generated samples and expand them into the original training dataset.

2. The method according to claim 1, characterized in that: The style division in step 1 adopts K-means clustering algorithm combined with manual screening.

3. The method according to claim 1, characterized in that: The sample generation network proposed in step 3 is based on the idea of ​​CycleGAN and combines the three network structures of Transformer, CNN and RNN.

4. The method according to claim 1, characterized in that: The training process in step 4 includes utilizing the generator and the discriminator and optimizing the network through adversarial loss, cycle consistency loss, identity mapping loss, style consistency loss, and maximum mean difference loss.

5. The method according to claim 1, characterized in that The intermediate image in step 5 is obtained when calculating the cycle consistency loss and the identity mapping loss; Spatial transformation operations include adding noise, image rotation, image scaling, image translation, image shearing, image affine transformation, and image multi-angle rotation.

6. The method according to claim 1, characterized in that The samples generated in step 6 are used to expand the original training data set to improve the recognition ability and robustness of the ship satellite image recognition model.

Images