SAR ship few-sample multi-task learning method, device, equipment and medium
By employing a multi-task learning approach, and utilizing the alternating updates of the image generation network and the projection discriminator, combined with the perceptual network and the classification network, the problem of image classification and generation under the condition of few ship samples is solved, achieving efficient multi-task training and generation.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- XIAN UNIV OF POSTS & TELECOMM
- Filing Date
- 2025-07-14
- Publication Date
- 2026-06-09
Smart Images
Figure CN121904419B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of computer technology, specifically to a multi-task learning method, apparatus, device, and medium for SAR ships with few samples. Background Technology
[0002] The successful launch of Synthetic Aperture Radar (SAR) satellites has greatly promoted the development of SAR remote sensing technology. In the field of marine monitoring, using SAR imagery for ship classification is a more advanced, specific, and challenging task, and has received increasing attention in recent years.
[0003] Existing classification algorithms mainly rely on a large number of training samples. However, in the field of ship classification, it is difficult to obtain a large number of SAR images. Therefore, existing technologies usually use SAR image generation model algorithms to supplement SAR ship sample data.
[0004] However, existing technologies do not take into account multi-task training of SAR ship sample data, namely ship image classification and image generation. Summary of the Invention
[0005] The purpose of this invention is to address the shortcomings of the prior art by providing a multi-task learning method, apparatus, device, and medium for SAR ships with few samples, so as to both classify ship images and generate ship images of a specified category.
[0006] To achieve the above objectives, the technical solutions adopted in the embodiments of the present invention are as follows:
[0007] First, embodiments of the present invention provide a multi-task learning method for SAR ships with few samples, the method comprising:
[0008] Based on the first category of conditions for ships, the initial image generation network outputs sample images of ships.
[0009] Based on the first category condition and the second category condition, the generated sample image of the ship and the real sample image of the ship are judged by the initial projection discriminator respectively, and the first discrimination result of the generated sample image of the ship and the second discrimination result of the real sample image of the ship are determined. The second category condition is the category condition of the real sample image of the ship.
[0010] Based on the first discrimination result and the second discrimination result, the initial image generation network and the initial projection discriminator are alternately updated to obtain the target image generation network and the target projection discriminator;
[0011] An initial classification network is used to classify the generated sample images of the ship and the real sample images of the ship, respectively, to determine the first classification result and the second classification result;
[0012] The initial classification network is updated based on the loss between the first classification result and the first category condition, and the loss between the second classification result and the second category condition, to obtain the target classification network.
[0013] The multi-task model includes: the target image generation network, the target projection discriminator, and the target classification network.
[0014] Optionally, before alternately updating the initial image generation network and the initial projection discriminator based on the first discrimination result and the second discrimination result, the method further includes:
[0015] Based on the reconstruction code output by the initial projection discriminator for the real sample image of the ship and the second category condition, the initial image generation network outputs a reconstructed sample image of the ship.
[0016] A perceptual network is used to calculate the error between the reconstructed sample images of the ship and the real sample images of the ship to obtain the perceptual loss function.
[0017] The reconstruction loss function is calculated based on the error between the reconstructed ship sample image and the actual ship sample image.
[0018] The step of alternately updating the initial image generation network and the initial projection discriminator based on the first discrimination result and the second discrimination result includes:
[0019] Based on the first discrimination result, the perception loss function, and the reconstruction loss function, the generation loss function is obtained;
[0020] Based on the first discrimination result and the second discrimination result, the discrimination loss function is obtained;
[0021] The initial image generation network and the initial projection discriminator are alternately updated based on the discriminant loss function and the generation loss function.
[0022] Optionally, the step of outputting a sample image of the ship using an initial image generation network based on a first category condition for the ship includes:
[0023] The first category condition and the noise vector are concatenated to obtain a concatenated vector.
[0024] The stitched vector is upsampled using multiple sequentially connected upsampling modules in the initial image generation network to output the generated sample image of the ship.
[0025] Optionally, the step of using an initial projection discriminator to distinguish between the generated ship sample image and the real ship sample image based on the first category condition and the second category condition, and determining the first discrimination result of the generated ship sample image and the second discrimination result of the real ship sample image, includes:
[0026] The generated sample image of the ship and the real sample image of the ship are downsampled by multiple downsampling modules in the initial projection discriminator to obtain two sampled feature vectors.
[0027] The first category condition and the second category condition are processed by the embedding layer in the initial projection discriminator to obtain the category condition encoding vector;
[0028] The two sampled feature vectors are processed by multiple fully connected layers in the initial projection discriminator to obtain two fully connected vectors;
[0029] The inner product module in the initial projection discriminator is used to perform an inner product on the two sampled feature vectors and the two category conditional coding vectors respectively, to obtain two inner product vectors;
[0030] The first discrimination result and the second discrimination result are determined based on the two fully connected vectors and the two inner product vectors, respectively.
[0031] Optionally, the step of classifying the generated sample images of the ship and the real sample images of the ship using an initial classification network includes:
[0032] The generated sample image of the ship and the real sample image of the ship are extracted by using multiple convolutional layers, multiple dense layers and the transition layer between the multiple dense layers in the initial classification network, respectively, to obtain two output feature maps;
[0033] The two output feature maps are classified and predicted using the global average pooling layer in the initial classification network.
[0034] Optionally, the step of using a perceptual network to calculate the error between the reconstructed ship sample image and the real ship sample image to obtain a perceptual loss function includes:
[0035] Multiple residual modules in the perception network are used to extract features from the reconstructed ship sample image and the real ship sample image, respectively, to obtain multiple sets of feature maps at different scales;
[0036] The initial projection discriminator is used to discriminate the multiple sets of feature maps at different scales to determine multiple sets of discrimination results;
[0037] Based on the multiple sets of discrimination results, the perceptual loss is calculated.
[0038] Optionally, the step of alternately updating the initial image generation network and the initial projection discriminator according to the discriminant loss function and the generation loss function includes:
[0039] If the initial image generation network is fixed, the projection discriminator is updated according to the discriminant loss function;
[0040] If the initial projection discriminator is fixed, the initial image generation network is updated according to the generation loss function.
[0041] Second, embodiments of the present invention also provide a multi-task learning device for SAR ships with few samples, the device comprising:
[0042] The image generation module is used to output a sample image of the ship using an initial image generation network based on the first category conditions for the ship.
[0043] The projection discrimination module is used to discriminate the generated sample image of the ship and the real sample image of the ship respectively using an initial projection discriminator according to the first category condition and the second category condition, and to determine the first discrimination result of the generated sample image of the ship and the second discrimination result of the real sample image of the ship, wherein the second category condition is the category condition of the real sample image of the ship;
[0044] The network update module is used to alternately update the initial image generation network and the initial projection discriminator according to the first discrimination result and the second discrimination result to obtain the target image generation network and the target projection discriminator;
[0045] The image classification module is used to classify the generated sample image of the ship and the real sample image of the ship using an initial classification network, and to determine the first classification result and the second classification result.
[0046] The network update module is further configured to update the initial classification network based on the loss between the first classification result and the first category condition, and the loss between the second classification result and the second category condition, to obtain the target classification network;
[0047] The multi-task model includes: the target image generation network, the target projection discriminator, and the target classification network.
[0048] Optionally, the image generation module is further configured to output a reconstructed sample image of the ship using the initial image generation network based on the reconstruction encoding output by the initial projection discriminator for the real sample image of the ship and the second category condition;
[0049] The perceptual network module is used to perform error calculation on the reconstructed sample image of the ship and the real sample image of the ship using a perceptual network to obtain a perceptual loss function.
[0050] The reconstruction module is used to calculate the reconstruction loss function based on the error between the reconstructed sample image of the ship and the real sample image of the ship;
[0051] The network update module is further configured to obtain a generation loss function based on the first discrimination result, the perceptual loss function, and the reconstruction loss function; obtain a discrimination loss function based on the first discrimination result and the second discrimination result; and alternately update the initial image generation network and the initial projection discriminator based on the discrimination loss function and the generation loss function.
[0052] Optionally, the image generation module is further configured to concatenate the first category condition and the noise vector to obtain a concatenated vector; and to upsample the concatenated vector using multiple sequentially connected upsampling modules in the initial image generation network to output the generated sample image of the ship.
[0053] Optionally, the projection discrimination module is further configured to: use multiple downsampling modules in the initial projection discriminator to downsample the generated sample image of the ship and the real sample image of the ship respectively, to obtain two sampled feature vectors; use the embedding layer in the initial projection discriminator to process the first category condition and the second category condition respectively, to obtain two category condition encoding vectors; use multiple fully connected layers in the initial projection discriminator to process the two sampled feature vectors, to obtain two fully connected vectors; use the inner product module in the initial projection discriminator to perform inner product on the two sampled feature vectors and the two category condition encoding vectors respectively, to obtain two inner product vectors; and determine the first discrimination result and the second discrimination result based on the two fully connected vectors and the two inner product vectors respectively.
[0054] Optionally, the image classification module is specifically used to extract features from the generated sample image of the ship and the real sample image of the ship using multiple convolutional layers, multiple dense layers and transition layers between the multiple dense layers in the initial classification network, respectively, to obtain two output feature maps; and to perform classification prediction on the two output feature maps using the global average pooling layer in the initial classification network.
[0055] Optionally, the perception network module is specifically used to extract features from the reconstructed ship sample image and the real ship sample image using multiple residual modules in the perception network, respectively, to obtain multiple sets of feature maps at different scales; to use the initial projection discriminator to discriminate the multiple sets of feature maps at different scales, and to determine multiple sets of discrimination results; and to calculate the perception loss function based on the multiple sets of discrimination results.
[0056] Optionally, the network update module is specifically used to update the projection discriminator according to the discriminant loss function if the initial image generation network is fixed; and to update the initial image generation network according to the generation loss function if the initial projection discriminator is fixed.
[0057] Third, embodiments of the present invention also provide an electronic device, including a processor, a memory, and a communication bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor communicates with the memory through the communication bus, and the processor executes the machine-readable instructions to implement the method described in any of the above embodiments.
[0058] Fourth, embodiments of the present invention also provide a storage medium storing a computer program, wherein the computer program is executed by a processor to perform any of the methods described above.
[0059] The beneficial effects of this invention are:
[0060] The present invention provides a multi-task learning method, apparatus, device, and medium for SAR ships with few sample images. When the number of ship sample images is very small, ship sample images are generated by an image generation network, and then the generated ship sample images are classified by a classification network. This allows for multi-task synchronous training of the image generation network and the classification network. Through training, a multi-task model for ships is obtained, which can both classify ship images and generate images of a specified ship category. This not only improves the training efficiency of the multi-task model, but also allows the training of the multi-task model to be completed with a small number of samples. Attached Figure Description
[0061] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.
[0062] Figure 1 The flowchart of the multi-task learning method provided in the embodiments of the present invention Figure 1 ;
[0063] Figure 2 The flowchart of the multi-task learning method provided in the embodiments of this application Figure 2 ;
[0064] Figure 3 This is a training architecture diagram of the multi-task model provided in an embodiment of the present invention;
[0065] Figure 4 This is a structural diagram of an image generation network provided in an embodiment of the present invention;
[0066] Figure 5 The flowchart of the multi-task learning method provided in the embodiments of the present invention Figure 3 ;
[0067] Figure 6 This is a structural diagram of the projection discriminator provided in an embodiment of the present invention;
[0068] Figure 7 A structural diagram of the classification network provided in an embodiment of the present invention;
[0069] Figure 8 This is a structural diagram of the dense layer and transition layer provided in an embodiment of the present invention;
[0070] Figure 9 This is a structural diagram of the sensing network provided in an embodiment of this application;
[0071] Figure 10 Ship sample images provided for embodiments of the present invention;
[0072] Figure 11 A schematic diagram of a multi-task learning device provided in an embodiment of the present invention;
[0073] Figure 12 This is a schematic diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation
[0074] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.
[0075] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments. Various details in this specification can also be modified and changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Unless otherwise specified, the following embodiments and features can be combined with each other.
[0076] The accompanying drawings are for illustrative purposes only and are schematic diagrams, not actual images, and should not be construed as limiting the invention. It is understandable that some well-known structures and their descriptions may be omitted in the drawings for those skilled in the art.
[0077] Figure 1 The flowchart of the multi-task learning method provided in the embodiments of the present invention Figure 1 ,like Figure 1 As shown, the method may include:
[0078] Step 101: Based on the first category conditions for ships, the initial image generation network outputs sample images of ships.
[0079] Specifically, the initial image generation network is used to generate sample images. The input to the initial image generation network is a first category condition for ships, which indicates the type of ship contained in the ship image to be generated.
[0080] The first category condition is encoded, and the encoded category vector is input into the initial image generation network. The initial image generation network then generates ship sample images corresponding to the first category condition based on the category vector.
[0081] In some embodiments, the concatenated vector of a preset random noise vector and a one-hot vector of a first category condition is input into an initial image generation network, which then generates a ship generation sample image corresponding to the first category condition based on the concatenated vector.
[0082] Step 102: Based on the first category condition and the second category condition, use the initial projection discriminator to discriminate the generated sample image of the ship and the real sample image of the ship respectively, and determine the first discrimination result of the generated sample image of the ship and the second discrimination result of the real sample image of the ship.
[0083] Specifically, the second category condition is used to indicate the type of ship contained in the real sample image of the ship. The projection discriminator is used to output a true or false score result for whether the category of the sample image is consistent with the category condition. If the output result of the projection discriminator is higher, it indicates that the category of the sample image is more consistent with the category condition and that it is more likely to be a real sample image. If the output result of the projection discriminator is lower, it indicates that the category of the sample image is less consistent with the category condition and that it is more likely to be a generated image.
[0084] The projection discriminator and the image generation network form a generative adversarial network. Essentially, the game between the projection discriminator and the image generation network aims to reach an equilibrium state where the generated sample images are indistinguishable from the real sample images. The projection discriminator lowers the score of the generated sample images and raises the score of the real sample images. In this way, the image generation network can be improved based on the score results, thereby improving the performance of the image generation network in an adversarial manner and thus improving the score of the generated sample images.
[0085] Specifically, the first category condition and the generated ship sample image are input into the initial projection discriminator. The initial projection discriminator judges the similarity between the first category condition and the generated ship sample image, and outputs the first discrimination result. The second category condition and the real ship sample image are input into the initial projection discriminator. The initial projection discriminator judges the similarity between the second category condition and the real ship sample image, and outputs the second discrimination result.
[0086] Step 106: Based on the first and second discrimination results, the initial image generation network and the initial projection discriminator are updated alternately to obtain the target image generation network and the target projection discriminator.
[0087] Specifically, the generation loss function of the initial image generation network is determined based on the first discrimination result, the discrimination loss function of the initial projection discriminator is determined based on the first and second discrimination results, and the initial image generation network and the initial projection discriminator are alternately updated based on the generation loss function and the discrimination loss function to obtain the target image generation network and the target projection discriminator.
[0088] In some embodiments, step 106 may include: if the initial image generation network is fixed, updating the projection discriminator according to the discriminant loss function; if the initial projection discriminator is fixed, updating the initial image generation network according to the generation loss function.
[0089] Specifically, the method for alternately updating the initial image generation network and the initial projection discriminator can be as follows:
[0090] In one training round, in the first sub-round, the parameters of the initial image generation network are fixed, and the parameters of the initial projection discriminator are updated according to the discriminant loss function. In the second sub-round, the parameters of the initial projection discriminator are fixed, and the parameters of the initial image generation network are updated according to the generation loss function. In this way, one round of training of the generative adversarial network is completed.
[0091] Alternatively, in the first sub-round, the parameters of the initial projection discriminator are fixed, and the parameters of the initial image generation network are updated according to the generation loss function. In the second sub-round, the parameters of the initial image generation network are fixed, and the parameters of the initial projection discriminator are updated according to the discrimination loss function. In this way, one round of training of the generative adversarial network is completed.
[0092] Example, generating loss function L G and discriminant loss function L D The calculation formulas can be expressed as follows:
[0093]
[0094]
[0095] in, P represents the generated distribution, and P represents the true distribution. Represents the generated image. Represents a real image.
[0096] When the parameters of the initial image generation network are fixed, it is necessary to maximize the discrimination loss function of the initial projection discriminator. L D The expected output of the initial projection discriminator >1, <-1, when the parameters of the initial projection discriminator are fixed, it is necessary to minimize the generation loss function of the initial image generation network. L G The goal of the initial image generation network is to make the discrimination result of the projection discriminator on the generated sample image as high as possible, that is, as close to 1 as possible.
[0097] Step 107: Use the initial classification network to classify the generated sample images of ships and the real sample images of ships respectively, and determine the first classification result and the second classification result.
[0098] Specifically, after completing one training iteration for the image generation network and the projection discriminator, the classification network is trained. The generated ship sample images are input into the initial classification network, which identifies the types of ships contained in the generated ship sample images to determine the first classification result. The real ship sample images are then input into the initial classification network, which identifies the types of ships contained in the real ship sample images to determine the second classification result.
[0099] Step 108: Update the initial classification network based on the loss of the first classification result and the first category condition, and the loss of the second classification result and the second category condition, to obtain the target classification network.
[0100] In this embodiment, a first standard cross-entropy loss is calculated based on the first classification result and the first category condition. A second standard cross-entropy loss is calculated based on the second classification result and the second category condition. The classification loss is determined based on the weighted value of the first standard cross-entropy loss and the second standard cross-entropy loss. The initial classification network is updated based on the classification loss to obtain the target classification network.
[0101] Example, classification loss L C The calculation formula can be expressed as:
[0102]
[0103] in, y The category of the ship in the real sample images of the ship is the second category condition. y’ For the first category of conditions, For classification networks, For image generation networks, Let cross-entropy be the loss function. These are adversarial weights, which are used to control the extent to which generated sample images are used to train the classification network.
[0104] After training, the multi-task model includes: a target image generation network, a target projection discriminator, and a target classification network. The target image generation network generates images containing ships of the corresponding category based on the input ship category conditions. The target projection discriminator scores the ship images generated by the target image generation network, and the target classification network identifies the ship categories contained in the input ship images.
[0105] The multi-task learning method for SAR ships with few samples provided by the present invention generates ship sample images through an image generation network when the number of ship sample images is very small, and then classifies the generated ship sample images through a classification network. This allows for multi-task synchronous training of the image generation network and the classification network, resulting in a multi-task model for ships. This model can both classify ship images and generate images of a specified ship category, which not only improves the training efficiency of the multi-task model but also allows for training of the multi-task model with a small number of samples.
[0106] In one possible implementation, Figure 2 The flowchart of the multi-task learning method provided in the embodiments of this application Figure 2 ,like Figure 2 As shown, prior to step 106 above, the method may further include:
[0107] Step 103: Based on the reconstruction encoding and second category conditions output by the initial projection discriminator for the real sample image of the ship, the initial image generation network outputs the reconstructed sample image of the ship.
[0108] Specifically, in the process of classifying the real sample images of ships, the initial projection discriminator downsamples the real sample images of ships and outputs a feature vector of a preset dimension through a fully connected layer, and uses this feature vector as the reconstruction code.
[0109] The reconstruction code and the second category condition are concatenated and input into the initial image generation network. The initial image generation network generates a ship reconstruction sample image based on the reconstruction concatenation vector. The ship reconstruction sample image is an image similar to the ship real sample image reconstructed by the initial image generation network based on the downsampled features extracted from the ship real sample image by the initial projection discriminator.
[0110] Step 104: Use a perceptual network to calculate the error between the reconstructed ship sample image and the real ship sample image to obtain the perceptual loss function.
[0111] Specifically, since generative adversarial networks (GANs) are also highly dependent on the amount of data, training a GAN when the amount of SAR ship data is small and unbalanced can easily lead to instability in the trained image generation network and pattern collapse. Therefore, it is necessary to use a perceptual network to further stabilize the learning of the GAN, thereby promoting the generation of the image generation network.
[0112] The perceptual network is used to evaluate the reconstruction capability of the initial image generation network. The perceptual network is a pre-trained network. It extracts features from the reconstructed ship sample image and the real ship sample image, respectively, and compares the features of the two to determine the perceptual loss function of the initial image generation network for image reconstruction. The perceptual loss function is used to participate in the update of the initial image generation network.
[0113] Step 105: Calculate the reconstruction loss function based on the error between the reconstructed ship sample image and the actual ship sample image.
[0114] Specifically, to further constrain the image generation network, image information is reused at the pixel level. A pixel-level error loss is constructed between the reconstructed ship sample image and the real ship sample image as the reconstruction loss function. L rec .
[0115] Example, reconstruction loss function L rec The calculation formula can be expressed as: .
[0116] in, Represents the reconstructed vector. Represents real sample images of ships. Represents an image generation network. Representative ship reconstruction sample image.
[0117] Step 106 above may include: obtaining a generation loss function based on the first discrimination result, the perceptual loss function, and the reconstruction loss function; obtaining a discrimination loss function based on the first discrimination result and the second discrimination result; and alternately updating the initial image generation network and the initial projection discriminator based on the discrimination loss function and the generation loss function.
[0118] Specifically, the expression for the discriminant loss function is as shown above, and will not be repeated here.
[0119] Based on the first discrimination result of the initial projection discriminator on the ship-generated sample image, the loss function of the initial image generation network for generating ship-generated sample images is calculated. Based on the sum of this loss function, the perception loss function, and the reconstruction loss function, the generation loss function is determined.
[0120] The specific method for alternately updating the initial image generation network and the initial projection discriminator can be as follows:
[0121] In one training round, in the first sub-round, the parameters of the initial image generation network are fixed, and the parameters of the initial projection discriminator are updated according to the discriminant loss function. In the second sub-round, the parameters of the initial projection discriminator are fixed, and the parameters of the initial image generation network are updated according to the generation loss function. In this way, one round of training of the generative adversarial network is completed.
[0122] Alternatively, in the first sub-round, the parameters of the initial projection discriminator are fixed, and the parameters of the initial image generation network are updated according to the generation loss function. In the second sub-round, the parameters of the initial image generation network are fixed, and the parameters of the initial projection discriminator are updated according to the discrimination loss function. In this way, one round of training of the generative adversarial network is completed.
[0123] In some embodiments, if the perceptual network is a pre-trained network, the parameters of the perceptual network are fixed during the training of the initial image generation network. If the perceptual network is an untrained network, the perceptual network is also trained during the training of the initial image generation network. The multi-task model may also include the perceptual network.
[0124] The multi-task learning method for SAR ships with few samples provided by the present invention introduces a perceptual network during the training of the multi-task model, which can stabilize the learning of the generative adversarial network and thus promote the training effect of the image generation network.
[0125] In one possible implementation, Figure 3 The training architecture diagram of the multi-task model provided in the embodiments of the present invention is as follows: Figure 3 As shown, the real sample images of ships are obtained by sampling ship samples, for example, by acquiring images of ships to obtain real sample images of ships.
[0126] The image generation network generates ship sample images based on category conditions and random noise. The image generation network generates ship reconstructed sample images based on category conditions and reconstruction codes. The projection discriminator generates reconstruction codes based on real sample images. The projection discriminator is also used to determine the discrimination loss function based on the ship generated sample images, real sample images of ships, and category conditions. The perception network is used to calculate the perception loss and reconstruction loss based on the real sample images and reconstructed sample images of ships. The perception loss function includes the perception loss, reconstruction loss, and loss determined based on the first discrimination result. The classification network calculates the classification loss based on the ship generated sample images and real sample images of ships.
[0127] Among them, the ship reconstruction sample images are only used for training the generative adversarial network and are not used for training the classification network.
[0128] In one possible implementation, step 101 above may include:
[0129] The first category condition and noise vector are concatenated to obtain a concatenated vector; multiple sequentially connected upsampling modules in the initial image generation network are used to upsample the concatenated vector to output the ship generated sample image.
[0130] Specifically, Figure 4 The structural diagram of the image generation network provided in the embodiments of the present invention is as follows: Figure 4 As shown, a 128-dimensional random noise vector and a one-hot vector with the first class condition are concatenated as the input to the image generation network. The image generation network includes multiple upsampling modules GBlock, for example, five. Each upsampling module GBlock consists of two BN layers, two Leaky ReLU activation functions, a transposed convolution DeConV for upsampling, and a regular convolution ConV. The transposed convolution DeConV can improve the details and sharpness of the generated image.
[0131] Transposed convolution DeConV and ordinary convolution ConV can also help transfer the detailed texture and shape features of SAR ships to deeper layers of the network through residual connections, thereby alleviating the gradient vanishing problem.
[0132] The kernel size of the transposed convolution in each upsampling module GBlock is 4×4, and the kernel size of the ordinary convolution is set to 3×3.
[0133] It should be noted that, except for GBlock1's transpose convolution which uses a stride of 1, all others use a stride of 2, and the final generated sample image output by the image generation network is 64×64.
[0134] In some implementations, the standard adversarial loss function of the projection discriminator can be expressed as:
[0135] The output of the optimal projection discriminator can be decomposed into the sum of log-likelihood ratios:
[0136] In the task of generating images based on category conditions, it is assumed that... y Given discrete class conditions {1,2,…,c}, the conditional probability is... and Using a log-linear model, then ,in, It is a distinguishing function. Therefore, the log-likelihood ratio can be simplified to the following formula:
[0137] Among them, let The first term is simplified to . This is the normalization constant part related to the partitioning function, which ensures that the probability values are within a reasonable range. Combining the normalization constant and... Integrate into The final result was:
[0138]
[0139] in, It is a category condition. It is the input image. and The main network of the projection discriminator is collectively represented. From a computational perspective, in reality... Is Under the influence of the one-hot vector, from the matrix Select the row vector corresponding to the current category. And then with Perform the inner product operation. In the above formula... The inner product is used to measure the degree of matching between the label and the image. Evaluate whether the input image conforms to the overall distribution of the category conditions.
[0140] In one possible implementation, Figure 5 The flowchart of the multi-task learning method provided in the embodiments of the present invention Figure 3 ,like Figure 5 As shown, step 102 above may include:
[0141] Step 201: Use multiple downsampling modules in the initial projection discriminator to downsample the generated sample image and the real sample image of the ship respectively, and obtain two sampled feature vectors.
[0142] Step 202: The embedding layer in the initial projection discriminator is used to process the first category condition and the second category condition respectively to obtain two category condition encoding vectors.
[0143] Step 203: Process the two sampled feature vectors using multiple fully connected layers in the initial projection discriminator to obtain two fully connected vectors.
[0144] Step 204: Using the inner product module in the initial projection discriminator, perform inner product on the two sampled feature vectors and the two class conditional coding vectors respectively to obtain two inner product vectors.
[0145] Step 205: Determine the first discrimination result and the second discrimination result based on the two fully connected vectors and the two inner product vectors.
[0146] Specifically, Figure 6 A structural diagram of the projection discriminator provided in an embodiment of the present invention is shown below. Figure 6As shown, the projection discriminator comprises a main network and a category-conditional projection part. The main network includes multiple downsampling modules (DBlocks) and multiple fully connected linear layers. For example, there can be four downsampling modules (DBlocks) and two fully connected layers. Each downsampling module (DBlock) consists of a Convolutional layer (ConV), a Batch Normalization (BN) layer, a Down ConV convolutional layer for downsampling, and a Leaky ReLU layer stacked together. All convolutional layers have a kernel size of 3×3. The Down ConV layer in each downsampling module (DBlock) of the main network uses a stride of 2 for downsampling. The category-conditional projection part consists of an embedding layer (Embedding) and inner product modules.
[0147] The ship can be generated as a sample image or a real sample image of a ship as the input image. The main network of the projection discriminator passes through multiple downsampling modules (DBlock) to obtain a feature vector. The last two dimensions of this feature vector are summed to obtain a two-dimensional sampled feature vector, corresponding to the formula above. .
[0148] Then, it splits into two branches, one of which performs a two-layer fully connected operation to obtain the result in the formula above. Another path enters the category-conditional projection section. The category-conditional projection section first applies the first category condition... y’ Or second category conditions Embedding is used to embed the feature vector into the latent feature space, and then it is combined with the two-dimensional sampled feature vector extracted by the main network. By performing the inner product, we obtain the formula above. Finally, the inner product vector After passing through two fully connected layers from the main network The sum is used as the final judgment result.
[0149] The 128-dimensional feature vector obtained from the first fully connected layer, Linear1, is used as a reconstruction code and, together with the category conditions, enters the image generation network to generate ship reconstruction sample images.
[0150] The projection discriminator provided by the present invention improves the image generation quality for specified category conditions by using inner product projection. It integrates category conditions and image features, thereby promoting the learning of different categories by the projection discriminator, improving the performance of the image generation network, and making the generated image not only conform to the target distribution of the image generation network, but also better reflect the input category conditions.
[0151] In one possible implementation, step 108 above may include:
[0152] The initial classification network uses multiple convolutional layers, multiple dense layers, and transition layers between the dense layers to extract features from generated ship sample images and real ship sample images, respectively, to obtain two output feature maps. The initial classification network uses a global average pooling layer to perform classification prediction on the two output feature maps.
[0153] Specifically, Figure 7 A structural diagram of the classification network provided in the embodiments of the present invention, such as... Figure 7 As shown, the classification network of the present invention includes multiple convolutional layers, multiple dense layers, and transition layers between the multiple dense layers. For example, there may be 2 convolutional layers, 5 dense layers, and 4 transition layers.
[0154] Figure 8 The structural diagram of the dense layer and transition layer provided in the embodiments of the present invention is as follows: Figure 8 As shown, each dense layer consists of multiple densely connected convolutional blocks, for example, three. These three densely connected convolutional blocks improve feature extraction. Each convolutional block includes a cascaded batch normalization (BN) layer, a rectified linear function (ReLU) layer, a 3×3 kernel convolutional layer, and a regularization layer. The regularization layer, for example, can use Dropout to reduce overfitting and improve the model's generalization ability.
[0155] The transition layer can include a batch normalization (BN) layer, a ReLU activation function, a 3×3 convolutional layer, and an average pooling layer. Its main function is to reduce dimensionality, further optimize computation, and prevent overfitting.
[0156] In some embodiments, the growth rates of the five dense layers are set to 3, 6, 9, 12, and 15, respectively. Figure 7 The number of channels in each green feature map. The kernel size of all convolutional layers is set to 3×3. The output feature map of the last dense layer is passed through a global average pooling layer and then passed to a fully connected layer for final classification prediction.
[0157] In one possible implementation, step 104 above may include:
[0158] Multiple residual modules in the perceptual network are used to extract features from the reconstructed ship sample images and the real ship sample images, respectively, to obtain multiple sets of feature maps at different scales. An initial projection discriminator is used to discriminate the multiple sets of feature maps at different scales to determine multiple sets of discrimination results. Based on the multiple sets of discrimination results, the perceptual loss function is calculated.
[0159] Specifically, Figure 9 A structural diagram of the perception network provided in the embodiments of this application, such as Figure 9As shown, the perceptual network includes an upsampled transposed convolutional layer (DeConV), multiple residual blocks (Res Blocks), a global average pooling layer (Avg Pool), and a fully connected layer (FC). The number of residual blocks (Res Blocks) can be up to four. The perceptual loss between the reconstructed ship sample image and the real ship sample image is calculated layer by layer using feature maps of different scales output by each residual block (Res Block). L feati .
[0160] Specifically, based on the mean square error of the feature maps of the reconstructed ship sample images at a specified scale and the feature maps of the real ship sample images at a specified scale output by each Res Block, the perceptual loss of the reconstructed ship sample images and the real ship sample images at each scale is calculated. L feati .
[0161] Example, perceptual loss function L feat The calculation formula can be expressed as:
[0162]
[0163] in, Represents the reconstructed vector. y Represents the second category of conditions. Represents real sample images of ships. Represents an image generation network. Represents the perceptual network. Representative ship reconstruction sample image.
[0164] The perceptual network can be a pre-trained neural network with fixed parameters or a neural network under training. In this embodiment, ResNet18 is selected as the perceptual network.
[0165] Based on the aforementioned perceptual loss function, reconstruction loss function, and first discrimination result, a loss function is generated. L G It can be represented as:
[0166]
[0167] Based on the multi-task learning method for SAR ships with few samples provided by the present invention, the following is an experimental analysis of the effect of using the multi-task learning method for SAR ships with few samples of the present invention.
[0168] Specifically, three and five categories of SAR ship data collected from the OpenSARShip database were used to validate the algorithm proposed in this chapter. Each ship target was represented by a single-polarization SAR data sample (VH or VV). The ship data for each polarization channel was randomly divided into five training and test sets. Table 1 lists the number of ship samples for each category, with five sets of data for each polarization channel.
[0169] Table 1. Statistics on the number of SAR ship samples in five categories
[0170]
[0171] The SAR ship data needs to be normalized for mean and variance. The specific method is as follows, where... Indicates the input image. express The mean, express The variance is scaled to a range with a mean of 0 and a variance of 1, normalizing the data to the same scale helps eliminate dimensional differences between different features, making different features comparable and accelerating model convergence.
[0172]
[0173] The network model parameter settings are shown in Table 2. All networks were trained using the Adam optimizer. The learning rate of each parameter was adjusted by calculating the estimates of the first and second moments of the gradient, resulting in faster and more stable model convergence during training. The initial learning rate was set to 0.0003, the Adam optimizer was used, the batch size was set to 64, and the network was trained for a total of 800 iterations. When the number of iterations reached 400 and 650, the learning rate of only the classification network was decayed by a factor of 10.
[0174] Table 2 Parameter settings for the network model
[0175]
[0176] The effectiveness of the method of the present invention is evaluated using four metrics: overall accuracy (OA), precision (P), recall (R), F1 score, and confusion matrix.
[0177] Overall accuracy (OA) measures the proportion of correctly classified samples by the classification network on all test samples; precision (P) represents the proportion of samples predicted as positive by the network that are actually positive, reflecting the false alarm rate of the classification network; recall (R) represents the proportion of correctly identified positive samples out of all actual positive samples, reflecting the false negative rate of the classification network; the F1 score is the harmonic mean of precision and recall, and a higher F1 score indicates better overall performance of the classification network. The specific calculation formulas are shown below:
[0178]
[0179]
[0180]
[0181]
[0182] As shown in Table 3, true positives (1 Positives, TP) are predicted to be positive and are actually positive; false positives (0 Positives, FP) are predicted to be positive but are actually negative; false negatives (false Negatives, FN) are predicted to be negative but are actually positive; and true negatives (1 Negatives, TN) are predicted to be negative and are actually negative.
[0183] Table 3. Confusion Matrix for Binary Classification
[0184]
[0185] The mean and standard deviation are calculated by selecting the best accuracy after the first reduction of the learning rate in each experiment.
[0186] FID (Frechet Inception Distance) is the most commonly used evaluation metric for generated images. It comprehensively represents the Fréchet distance between the feature vectors of the real image and the generated image. The smaller the value, the better the quality of the generated image and the closer it is to the real image. The specific calculation method is as follows:
[0187]
[0188] in, and These represent the generated image and the real image, respectively. This represents the mean. The trace of the matrix is used to select the experimental round with the highest classification accuracy, and the FID is calculated using the images saved in the experiment to generate the network.
[0189] To comprehensively evaluate the effectiveness of the proposed method, comparative experiments were first conducted. Comparative experiments were carried out on five sets of data for VV and VH polarization of SAR ships of categories 3 and 5, respectively. The accuracy of all experimental results was averaged as the overall accuracy, as shown in Tables 4 and 5.
[0190] Table 4 Comparative Experiment of Data from Three Types of Single-Polarization Ships
[0191]
[0192] Table 5. Comparative Experiment of Data from Five Types of Single-Polarization Ships
[0193]
[0194] Compared with state-of-the-art methods ACGAN, EC-GAN, LST-ACGAN, and GAN-CLA, the experimental results show that our proposed method achieves classification accuracies of 87.23% and 66.20% for classes 3 and 5 ships, respectively, which are 3.62% and 2.59% higher than the second-best method, EC-GAN. In terms of F1 score, precision, and recall, our method is the best among the other methods. For the FID index, our method achieves 62.25 for class 3 and 130.25 for class 5, reaching the lowest values compared to other methods, indicating the best generation quality and demonstrating the superiority of our proposed method.
[0195] Figure 10 Ship sample images provided for embodiments of the present invention, such as Figure 10 As shown, the target image generation network trained using this invention generates five categories of ship images, with each column corresponding to one category and five images generated for each category. The generation results demonstrate that the target image generation network trained using this method can generate ship sample images of the specified categories, exhibiting a certain degree of diversity and without pattern collapse, further illustrating the effectiveness of the method.
[0196] Figure 11 A schematic diagram of a multi-task learning device provided in an embodiment of the present invention, such as... Figure 11 As shown, the device may include:
[0197] Image generation module 301 is used to output ship generation sample images using an initial image generation network based on the first category conditions for ships.
[0198] The projection discrimination module 302 is used to discriminate the generated sample image of the ship and the real sample image of the ship respectively by using an initial projection discriminator according to the first category condition and the second category condition, and to determine the first discrimination result of the generated sample image of the ship and the second discrimination result of the real sample image of the ship. The second category condition is the category condition of the real sample image of the ship.
[0199] The network update module 305 is used to alternately update the initial image generation network and the initial projection discriminator according to the first discrimination result and the second discrimination result to obtain the target image generation network and the target projection discriminator.
[0200] The image classification module 306 is used to classify the generated sample image of the ship and the real sample image of the ship using an initial classification network, and to determine the first classification result and the second classification result.
[0201] The network update module 305 is also used to update the initial classification network based on the loss of the first classification result and the first category condition, and the loss of the second classification result and the second category condition, to obtain the target classification network;
[0202] The multi-task model includes: a target image generation network, a target projection discriminator, and a target classification network.
[0203] Optionally, the image generation module 301 is further configured to output a reconstructed sample image of the ship using the initial image generation network based on the reconstruction encoding and second category conditions output by the initial projection discriminator for the real sample image of the ship.
[0204] The perceptual network module 303 is used to calculate the error between the reconstructed sample image of the ship and the real sample image of the ship using the perceptual network, and obtain the perceptual loss function.
[0205] The reconstruction module 304 is used to calculate the reconstruction loss function based on the error between the reconstructed ship sample image and the real ship sample image;
[0206] The network update module 305 is further configured to obtain a generation loss function based on the first discrimination result and the perceptual loss function. A discrimination loss function is obtained based on the first and second discrimination results; and the initial image generation network and the initial projection discriminator are alternately updated based on the discrimination loss function and the generation loss function.
[0207] Optionally, the image generation module 301 is also used to concatenate the first category condition and the noise vector to obtain a concatenated vector; and to upsample the concatenated vector using multiple sequentially connected upsampling modules in the initial image generation network to output a ship-generated sample image.
[0208] Optionally, the projection discrimination module 302 is further configured to: use multiple downsampling modules in the initial projection discriminator to downsample the generated sample image of the ship and the real sample image of the ship respectively, to obtain two sampled feature vectors; use the embedding layer in the initial projection discriminator to process the first category condition and the second category condition respectively, to obtain two category condition encoding vectors; use multiple fully connected layers in the initial projection discriminator to process the two sampled feature vectors, to obtain two fully connected vectors; use the inner product module in the initial projection discriminator to perform inner product on the two sampled feature vectors and the two category condition encoding vectors respectively, to obtain two inner product vectors; and determine the first discrimination result and the second discrimination result based on the two fully connected vectors and the two inner product vectors respectively.
[0209] Optionally, the image classification module 306 is specifically used to extract features from the generated sample image of the ship and the real sample image of the ship using multiple convolutional layers, multiple dense layers and transition layers between multiple dense layers in the initial classification network, respectively, to obtain two output feature maps; and to perform classification prediction on the two output feature maps using the global average pooling layer in the initial classification network.
[0210] Optionally, the perceptual network module 303 is specifically used to extract features from the reconstructed ship sample image and the real ship sample image using multiple residual modules in the perceptual network, respectively, to obtain multiple sets of feature maps at different scales; to use an initial projection discriminator to discriminate the multiple sets of feature maps at different scales, and to determine multiple sets of discrimination results; to calculate the perceptual loss based on the multiple sets of discrimination results; and to calculate the reconstruction loss based on the error between the reconstructed ship sample image and the real ship sample image; wherein, the perceptual loss function includes: perceptual loss and reconstruction loss.
[0211] Optionally, the network update module 305 is specifically used to update the projection discriminator according to the discrimination loss function if the initial image generation network is fixed; and to update the initial image generation network according to the generation loss function if the initial projection discriminator is fixed.
[0212] In one possible implementation, embodiments of the present invention also provide an electronic device. Figure 12 A schematic diagram of an electronic device provided in an embodiment of the present invention, such as... Figure 12 As shown, the electronic device may include a processor 401, a memory 402, and a communication bus 403. The memory 402 stores machine-readable instructions that can be executed by the processor 401. When the electronic device is running, the processor 401 communicates with the memory 402 through the communication bus 403, and the processor 401 executes the machine-readable instructions to implement the SAR ship few-sample multi-task learning method of the above embodiment.
[0213] In one possible implementation, the present invention also provides a storage medium storing a computer program, which, when executed by a processor, performs the SAR ship few-sample multi-task learning method described above.
[0214] The above are merely specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.
Claims
1. A multi-task learning method for SAR ships with few samples, characterized in that, The method includes: Based on the first category of conditions for ships, the initial image generation network outputs sample images of ships. Based on the first category condition and the second category condition, the generated sample image of the ship and the real sample image of the ship are judged by the initial projection discriminator respectively, and the first discrimination result of the generated sample image of the ship and the second discrimination result of the real sample image of the ship are determined. The second category condition is the category condition of the real sample image of the ship. Based on the first discrimination result and the second discrimination result, the initial image generation network and the initial projection discriminator are alternately updated to obtain the target image generation network and the target projection discriminator; An initial classification network is used to classify the generated sample images of the ship and the real sample images of the ship, respectively, to determine the first classification result and the second classification result; The initial classification network is updated based on the loss between the first classification result and the first category condition, and the loss between the second classification result and the second category condition, to obtain the target classification network. The multi-task model includes: the target image generation network, the target projection discriminator, and the target classification network; Before alternately updating the initial image generation network and the initial projection discriminator based on the first discrimination result and the second discrimination result, the method further includes: Based on the reconstruction code output by the initial projection discriminator for the real sample image of the ship and the second category condition, the initial image generation network outputs a reconstructed sample image of the ship. A perceptual network is used to calculate the error between the reconstructed sample images of the ship and the real sample images of the ship to obtain the perceptual loss function. The reconstruction loss function is calculated based on the error between the reconstructed ship sample image and the actual ship sample image. The step of alternately updating the initial image generation network and the initial projection discriminator based on the first discrimination result and the second discrimination result includes: Based on the first discrimination result, the perception loss function, and the reconstruction loss function, the generation loss function is obtained; Based on the first discrimination result and the second discrimination result, the discrimination loss function is obtained; The initial image generation network and the initial projection discriminator are alternately updated based on the discriminant loss function and the generation loss function.
2. The method as described in claim 1, characterized in that, The step of generating sample images of ships using an initial image generation network based on the first category conditions for ships includes: The first category condition and the noise vector are concatenated to obtain a concatenated vector. The stitched vector is upsampled using multiple sequentially connected upsampling modules in the initial image generation network to output the generated sample image of the ship.
3. The method as described in claim 1, characterized in that, The step of using an initial projection discriminator to distinguish between the generated ship sample image and the real ship sample image based on the first category condition and the second category condition, and determining the first discrimination result of the generated ship sample image and the second discrimination result of the real ship sample image, includes: The generated sample image of the ship and the real sample image of the ship are downsampled by multiple downsampling modules in the initial projection discriminator to obtain two sampled feature vectors. The first category condition and the second category condition are processed by the embedding layer in the initial projection discriminator to obtain two category condition encoding vectors; The two sampled feature vectors are processed by multiple fully connected layers in the initial projection discriminator to obtain two fully connected vectors; The inner product module in the initial projection discriminator is used to perform an inner product on the two sampled feature vectors and the two category conditional coding vectors respectively, to obtain two inner product vectors; The first discrimination result and the second discrimination result are determined based on the two fully connected vectors and the two inner product vectors, respectively.
4. The method as described in claim 1, characterized in that, The initial classification network is used to classify the generated sample images of the ship and the real sample images of the ship, respectively, including: The generated sample image of the ship and the real sample image of the ship are extracted by using multiple convolutional layers, multiple dense layers and the transition layer between the multiple dense layers in the initial classification network, respectively, to obtain two output feature maps; The two output feature maps are classified and predicted using the global average pooling layer in the initial classification network.
5. The method as described in claim 1, characterized in that, The step of using a perceptual network to calculate the error between the reconstructed ship sample image and the real ship sample image to obtain a perceptual loss function includes: Multiple residual modules in the perception network are used to extract features from the reconstructed ship sample image and the real ship sample image, respectively, to obtain multiple sets of feature maps at different scales; The perceptual loss function is calculated based on the multiple sets of feature maps at different scales.
6. The method as described in claim 1, characterized in that, The step of alternately updating the initial image generation network and the initial projection discriminator according to the discriminant loss function and the generation loss function includes: If the initial image generation network is fixed, the projection discriminator is updated according to the discriminant loss function; If the initial projection discriminator is fixed, the initial image generation network is updated according to the generation loss function.
7. A multi-task learning device for SAR ships with few samples, characterized in that, The apparatus is used to perform the method as described in any one of claims 1 to 6, the apparatus comprising: The image generation module is used to output a sample image of the ship using an initial image generation network based on the first category conditions for the ship. The projection discrimination module is used to discriminate the generated sample image of the ship and the real sample image of the ship respectively using an initial projection discriminator according to the first category condition and the second category condition, and to determine the first discrimination result of the generated sample image of the ship and the second discrimination result of the real sample image of the ship, wherein the second category condition is the category condition of the real sample image of the ship; The network update module is used to alternately update the initial image generation network and the initial projection discriminator according to the first discrimination result and the second discrimination result to obtain the target image generation network and the target projection discriminator; The image classification module is used to classify the generated sample image of the ship and the real sample image of the ship using an initial classification network, and to determine the first classification result and the second classification result. The network update module is further configured to update the initial classification network based on the loss between the first classification result and the first category condition, and the loss between the second classification result and the second category condition, to obtain the target classification network; The multi-task model includes: the target image generation network, the target projection discriminator, and the target classification network.
8. An electronic device, characterized in that, The device includes a processor, a memory, and a communication bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor communicates with the memory through the communication bus, and the processor executes the machine-readable instructions to implement the method according to any one of claims 1 to 6.
9. A storage medium storing a computer program, wherein the computer program, when executed by a processor, performs the method according to any one of claims 1 to 6.