A few-shot power scene augmentation and classification method based on image generation

Abstract

This invention discloses a few-sample augmentation and classification method for power scenes based on image generation. This method can complete the task of recognizing new categories of objects in power scenes even with extremely limited training data for the new categories. The technical solution of this application mainly consists of the following three parts: To fully explore the relationships between images of different categories, a generative adversarial network (GAN) conditioned on word vectors is first trained on known class data. Secondly, with only a few images of the new category, the GAN is fine-tuned to generate a large number of images for the new category. Finally, an image quality evaluation technique is used to select images with realism and diversity for the task of recognizing new categories of objects in power scenes. The technical effect of this application is that this method can effectively achieve the task of recognizing new categories of objects in power scenes under few-sample conditions.

Description

Technical Field

[0001] This invention relates to a few-sample power scene augmentation and classification method based on image generation, belonging to the field of computer vision technology. Background Technology

[0002] Most existing deep learning-based power scene recognition technologies employ fully supervised methods, which are trained on large-scale power scene datasets. While these fully supervised methods perform well on existing datasets, the cost of data collection and annotation limits their practical applications.

[0003] To address the aforementioned problems, few-shot recognition algorithms have been proposed, aiming to identify new categories of images with a limited number of labeled samples. Most existing few-shot recognition methods employ meta-learning training, learning a category-independent model to quickly adapt to new category recognition. These methods often utilize prototype-based or parameter-generation-based approaches, employing contextual training and dual-branch models to design various modules that better extract prior knowledge from labeled data and enhance the guiding role of labeled data on unlabeled data, thereby improving generalization ability.

[0004] While the aforementioned meta-learning-based methods offer some performance improvements in few-shot recognition tasks, contextual training treats categories in the training set as independent, which does not align with real-world power scenarios. Therefore, there is an urgent need for methods that, under supervision with a very small number of new category images (1-5 images), fully consider the relationships between existing and new categories, and effectively improve few-shot recognition performance in power scenarios. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of the prior art and provide a few-sample power scene augmentation and classification method based on image generation. This method fully utilizes existing data and provides semantic relationships between a small number of samples of a new category. It uses image generation technology to generate more pseudo-samples for the new category to complete the few-sample recognition task of power scenes.

[0006] To achieve the above objectives, the present invention is implemented using the following technical solution:

[0007] In a first aspect, the present invention provides a few-sample power scene augmentation method based on image generation, comprising the following steps:

[0008] Train a generative adversarial network conditioned on known class data;

[0009] With a small number of images for the new category, the generative adversarial network is fine-tuned to generate a large number of images for the new category, thus forming pseudo-samples.

[0010] Based on image quality assessment techniques, images with realism and diversity are selected from pseudo-samples to obtain an augmented dataset for the recognition task of new categories of objects in power scenarios.

[0011] Furthermore, a generative adversarial network conditioned on word vectors is trained on known class data, including:

[0012] A generative adversarial network (GAN) conditioned on base class data is trained. A mapping network M is used to map base class word vectors c and random noise vectors z into latent vectors w. w is then fed into a hierarchical generative network S to generate fake images corresponding to the base class categories.

[0013] The mapping network M can be used to decouple normally distributed noise z to obtain hidden features, thereby enabling better control over the generation details; the mapping network M uses word vector c as the category condition; the generation network S has a hierarchical structure and can generate fake images layer by layer.

[0014] Using the generative adversarial loss function adv1 loss The generative adversarial network (GAN) on the base class is optimized. On one hand, the discriminator is fixed, and the generator (mapping network M and generator network S) is trained to ensure that the generated fake base class images have a consistent distribution with the images in the base class dataset, thus preventing the discriminator D from distinguishing between the two types of images. On the other hand, the generator is fixed, and the discriminator D is trained to distinguish these fake images as much as possible. The training stops when the losses of both the generator network S and the discriminator network D fluctuate within a certain range without showing a significant, continuous upward or downward trend.

[0015] Furthermore, under the condition of a small number of new images, the generative adversarial network is fine-tuned, including:

[0016] Generating fake samples for new categories with few samples. Guided by a small number of images of new categories, the Adversarial Generative Loss Function (Adv2) is used. loss The generative adversarial network (GAN) is fine-tuned to generate a large number of images for new categories. On one hand, the discriminator is fixed while the generator (mapping network M and generator network S) is trained, ensuring that the generated fake images of the new category are distributed consistently with the images in the new category dataset, making it impossible for the discriminator D to distinguish between the two types of images. On the other hand, the generator is fixed while the discriminator D is trained to distinguish these images as much as possible. The training stops when the losses of both the generator network S and the discriminator network D fluctuate within a certain range without showing a significant, continuous upward or downward trend.

[0017] Furthermore, the adversarial loss function Adv1 loss and Adv2 loss The expression is as follows:

[0018]

[0019]

[0020] Where z is random noise sampled from a Gaussian distribution, and c b and c n These are the word vectors of the base class and the new class, respectively, x b and x n These represent real samples of the base class and the new class, respectively, while D and G represent the discrimination part and the generation part, respectively.

[0021] Furthermore, under the condition of a small number of new images, fine-tuning the generative adversarial network also includes:

[0022] In fine-tuning generative adversarial networks, semantic visual alignment loss is also used to prevent the pattern collapse problem caused by conditional GAN ​​networks remembering a small number of samples.

[0023] Semantic visual alignment loss refers to the fact that, for the same noise vector, the similarity distribution between samples generated by the new category and samples generated by the selected base class should be consistent with the similarity distribution between word vectors.

[0024] Semantic visual alignment loss specifically refers to the following: for the same noise vector, the similarity distribution between samples generated by the new category and samples generated by the selected base class should be consistent with the similarity distribution between word vectors, as expressed below:

[0025]

[0026]

[0027]

[0028] Where P visual and P semantic These represent the distributions of visual similarity and word vector similarity, respectively. z is random noise sampled from a Gaussian distribution, and c... i and c j These are the word vectors of the categories selected from the new class and the base class, respectively, and sim is the cosine similarity.

[0029] Furthermore, under the condition of a small number of new images, fine-tuning the generative adversarial network also includes:

[0030] In fine-tuning generative adversarial networks, cross-class similarity alignment loss is also used to prevent the pattern collapse problem caused by conditional GAN ​​networks remembering a small number of samples.

[0031] Cross-class similarity alignment loss refers to the consistency between the similarity distribution of the base class under the same noise distribution when multiple noise samples are randomly sampled, these noises generate multiple samples under the same new class, and the similarity between each pair of samples is calculated.

[0032] Cross-class similarity alignment loss specifically refers to: randomly sampling N noises, which generate N samples within the same class. The similarity between each pair of these samples is calculated, and then normalized using Softmax to obtain the similarity distribution for the current class.

[0033]

[0034] Where z i and z j These are two random noise samples taken from a Gaussian distribution, c n These are the word vectors of the new class.

[0035] Calculate P in the same way. base Then, the cross-category similarity ranking loss is as follows:

[0036]

[0037] Where D KL The similarity between the two distributions is calculated using the Kullback-Leibler divergence.

[0038] Furthermore, image quality assessment techniques are used to filter out images with realism and diversity from the fake samples, including:

[0039] Given a small number of images and generated pseudo-samples, we aim to evaluate and filter pseudo-samples through three aspects: sample quality assessment, sample similarity assessment, and sample diversity assessment, to obtain an augmented dataset X for the task of recognizing new categories of objects in the power scenario.

[0040] X={x|α1Score IQA +α2Score similarity +α3Score diversity >σ}

[0041] Where σ is the given screening threshold, set to 0.6, and α1, α2, and α3 are the weights of the three quality assessment indicators, all of which are set to 1 by default.

[0042] Further, the sample quality assessment includes:

[0043] A quality assessment network is used to extract image features and output a quality score for pseudo-samples. IQA= RanIQA(x), where x is the generated pseudo sample. RankIQA is an existing image quality assessment method. The higher the quality score, the less noise the generated image has.

[0044] Sample similarity assessment includes:

[0045] Calculate the cosine similarity between the pseudo-sample and the given image of the new category, and convert it into a similarity score. The higher the similarity, the higher the similarity score. The calculation formula is as follows:

[0046] Score similarity =cos(x, x) support )

[0047] Where x is the generated sample, x support The new category has a small number of known real samples;

[0048] Sample diversity assessment includes:

[0049] The generated samples are clustered into several classes. A score is awarded based on the distance between the current sample and all cluster centers. The higher the minimum distance between the current sample and all cluster centers, the higher the diversity score. The calculation formula is as follows:

[0050] Score diversity =min(d(x, x) cluster_center ))

[0051] Where x cluster_center These are the sample centers of the new category after clustering.

[0052] Secondly, the present invention provides a method for augmenting and classifying few-sample power scenes based on image generation, comprising the following steps:

[0053] The image-based few-sample power scene generation method described in the first aspect yields an augmented dataset for the task of recognizing new categories of objects in power scenes;

[0054] A fully supervised network was trained using the augmented dataset to identify samples of the new class.

[0055] Compared with the prior art, the beneficial effects achieved by the present invention are as follows:

[0056] 1. The method of the present invention can train a few-sample recognition model for power scenarios under the condition of lack of new samples (only 1-5 images).

[0057] 2. To fully explore the relationships between different image categories, this invention first trains a generative adversarial network (GAN) conditioned on word vectors on base class data. Secondly, with only a few images of the new category, the GAN is fine-tuned to generate a large number of images for the new category. Finally, an image quality assessment technique is used to select images with realism and diversity for the task of recognizing new object categories in power scene data. This method can effectively achieve the task of recognizing new object categories under limited sample conditions. Attached Figure Description

[0058] Figure 1 This is a schematic diagram of the method flow of the present invention;

[0059] Figure 2 This is a schematic diagram of the semantic visual alignment loss and cross-category similarity alignment loss proposed in this invention. Detailed Implementation

[0060] The present invention will be further described below with reference to the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and should not be used to limit the scope of protection of the present invention.

[0061] Example 1:

[0062] This embodiment provides a few-sample image augmentation method based on image generation, the overall process of which is as follows: Figure 1 As shown, it includes the following steps:

[0063] Step 1: Train a generative adversarial network (GAN) conditioned on word vectors based on base class data. A mapping network M maps base class word vectors c and random noise vectors z to latent vectors w. w is then fed into a hierarchical generative network S to generate fake images corresponding to the base class categories. The mapping network M consists of multiple fully connected layers to decouple the normally distributed noise z to obtain hidden features, thus enabling better control over the generation details. The generative network S generates fake images layer by layer. The word vectors are divided into base class word vectors c... b and the new class word vector c n This technology maps category names to vectors, with more similar category names resulting in vectors of higher similarity. This invention employs word vectors based on the CLIP model.

[0064] Step 2: Use the generative adversarial loss function Adv1 loss This ensures that the generated fake base class images are distributed consistently with the images in the base class dataset. Then, a discriminator D is used to determine whether the generated base class images belong to the real or fake base class images. The optimization process can be represented as follows:

[0065]

[0066] Where, x b ~p data (x b ) is an image sampled from a known class distribution, c b These are the word vectors corresponding to known classes, and D represents the discriminator, which can determine whether an image is a real image from a known class or a generated fake image. Ideally, if it is a real image from a known class, the discriminator outputs 1, i.e., D(x) = 1. b ) = 1. If it is a fake image G(z, c) = 1. b If ), then the discriminator output is 0, that is, D(G(z, c)). b )) = 0.

[0067] Step 3: Generating fake samples under the condition of few samples for new categories. Guided by a small number of images of new categories, the generative adversarial loss function Adv2 is used. loss To fine-tune a generative adversarial network with known categories, thereby enabling the generation of a large number of images with new categories, the optimization process can be represented as follows:

[0068]

[0069] Where, x n ~p data (x n (c) is an image sampled from a small number of new category distributions. n Here, D represents the word vector corresponding to the new category, and D denotes the discriminator for the new category, which can determine whether an image is a real image from the new category or a generated fake image. Since the number of images in the new category is limited, various data augmentation techniques are used, including cropping, brightness enhancement, and contrast enhancement. Because the generator and discriminator for the new category are fine-tuned from the generator and discriminator for known categories, they follow a conditional distribution. Ideally, if it is a real image of the new category, the discriminator outputs 1, i.e., D(x) = 1. n |D b ) = 1. If it is a fake image G(z, c) = 1. n |G b If ), then the discriminator output is 0, that is, D(G(z, c)). n |G b )) = 0.

[0070] Step 4, Fine-tuning the Generative Adversarial Network (GAN), also utilizes semantic visual alignment loss to prevent pattern collapse caused by the conditional GAN ​​network remembering a small number of samples. Semantic visual alignment loss refers to the principle that, for the same noise vector, the similarity distribution between samples generated by the new class and samples generated by the selected base class should be consistent with the similarity distribution between word vectors. Its formula is as follows:

[0071]

[0072]

[0073]

[0074] Among them, P visual It is the similarity distribution between samples generated from the new category and samples generated from the selected base class; P sematic This represents the similarity distribution between the word vectors of the new class and the base class. Both distributions are normalized using Softmax, and cosine similarity is used as the similarity metric. Finally, KL divergence is used to constrain the consistency between the two distributions. Here, similarity is used as the loss function, which is optimized through gradient backpropagation. The greater the inconsistency, the larger the loss function, and the greater the magnitude of backpropagation updates the network parameters.

[0075] Step 5, Fine-tuning the Generative Adversarial Network, also utilizes cross-class similarity alignment loss to prevent the pattern collapse problem caused by the conditional GAN ​​network remembering a small number of samples. Cross-class similarity alignment loss refers to the consistency between the similarity distribution of the base class under the same noise distribution when multiple noises are randomly sampled, these noises generate multiple samples under the same new class, and the similarity between each pair is calculated.

[0076]

[0077] Calculate P in the same way. base Then, the cross-category similarity ranking loss is as follows:

[0078]

[0079] Both distributions are normalized using Softmax, and cosine similarity is used as the sim metric. Finally, KL divergence is used to constrain the consistency between the two distributions.

[0080] Step 6: Due to the large number of fake samples generated under limited sample conditions, the realism may be insufficient and there may be a large number of similar fake samples, which will lead to poor performance in new class recognition. Given the supporting images and the generated fake samples, we plan to evaluate and screen fake samples through three aspects: sample quality assessment, sample similarity assessment, and sample diversity assessment.

[0081] Specifically, the sample quality assessment uses a quality assessment network to extract image features and outputs a quality score for the pseudo-samples. IQA = RankIQA(x), where x is the generated pseudo sample. RankIQA is an existing image quality assessment method. The higher the quality score, the less noise such as artifacts in the generated image.

[0082] Sample similarity evaluation calculates the cosine similarity between pseudo-samples and a given image of the new category, and converts this into a similarity score. A higher similarity score results in a higher similarity score. The calculation formula is as follows: Score similarity =cos(x, x) support ), where x is the generated sample, x support The new category has a small number of known real samples;

[0083] Sample diversity assessment generates clusters of samples into several classes and scores them based on the distances of the current sample to all cluster centers. The greater the minimum distance from the current sample to all cluster centers, the higher the diversity score. The calculation formula is as follows: Score diversity =min(d(x, x) cluster_center )), where x cluster_center These are the sample centers of the new category after clustering.

[0084] The final sample score is obtained by averaging the quality score, similarity score, and diversity score. Based on this score, pseudo-samples are filtered to obtain high-quality pseudo-samples. Here, σ is a given filtering threshold, set to 0.6.

[0085] X={x|α1Score IQA +α2Score similarity +α3Score diversity >σ}

[0086] Where α1, α2, and α3 are the weights of the sample quality score, similarity score, and diversity score, respectively. X represents the final selected generated samples, i.e., the augmented dataset.

[0087] Example 2:

[0088] This embodiment provides a few-sample image recognition method based on image generation, including:

[0089] As described in Example 1, the few-sample image augmentation method yields the selected samples, namely...

[0090] The selected samples were then used to train a fully supervised network to identify new categories of samples.

[0091] Finally, a fully supervised network is trained based on the selected samples to complete the recognition of new category samples, using the normal cross-entropy loss function for training. Fully supervised networks that can be used for recognition models in this application include, but are not limited to, ResNet, DenseNet, and Swin-Transformer models.

[0092] This invention enables the training of a few-sample recognition model for power scenarios even with limited new samples (only 1-5 images). To fully explore the relationships between different image categories, a generative adversarial network (GAN) conditioned on word vectors is first trained on the base class data. Secondly, with only a few images of the new category, the GAN is fine-tuned to generate a large number of images for the new category. Finally, an image quality assessment technique is used to select images with realism and diversity for the recognition of new object categories in power scenarios. This method effectively achieves the task of recognizing new object categories under limited sample conditions.

[0093] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0094] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0095] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0096] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0097] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A few-sample power scene augmentation method based on image generation, characterized in that, Includes the following steps: Train a generative adversarial network conditioned on known class data; Under the condition of a small number of images for the new category, the generative adversarial network is fine-tuned so that the fine-tuned generative adversarial network generates a large number of images for the new category, forming pseudo samples; Based on image quality assessment techniques, images with realism and diversity are selected from pseudo-samples to obtain an augmented dataset for the recognition task of new categories of objects in power scenarios. Training a generative adversarial network conditioned on known class data, including: A generative adversarial network conditioned on word vectors is trained based on base class data; a mapping network is then used. base class word vectors and random noise vector Mapping to latent vectors , and then Feed into a hierarchical generative network In this process, fake images corresponding to the base class category are generated; The mapping network Noise used for normal distribution Decoupling is performed to obtain hidden features; mapping network It is based on word vectors As a category condition; Generating Network It has a hierarchical structure and can generate fake images layer by layer; Using a generative adversarial loss function Train a generative adversarial network on a base class; the training method includes: on the one hand, fixing the discriminator part and training the mapping network. and generative networks This ensures that the generated base class fake images are distributed in a consistent manner with the images in the base class dataset, thereby improving the discriminator's performance. The two types of images cannot be distinguished; on the other hand, the generated part is fixed, and the discriminator is trained. This allows the discriminator to distinguish such images as much as possible; Training stopping condition is the generator network and identification network The losses fluctuate within a certain range and do not show a clear trend of continuous increase or decrease.

2. The few-sample power scene augmentation method based on image generation according to claim 1, characterized in that, Under the condition of a small number of new images, fine-tuning the generative adversarial network includes: Generating fake samples for new categories with few samples; using a generative adversarial loss function guided by a given small number of new category images. To fine-tune the generative adversarial network, it is possible to generate a large number of images of new categories. The fine-tuning method includes: on the one hand, fixing the discriminator part and training the mapping network M and the generator network S so that the distribution of the generated fake images of the new category is consistent with the distribution of images in the new category dataset, so that the discriminator D cannot distinguish between the two types of images; on the other hand, fixing the generator part and training the discriminator D so that the discriminator can distinguish such images as much as possible. Training stopping condition is the generator network and identification network The losses fluctuate within a certain range and do not show a clear trend of continuous increase or decrease.

3. The few-sample power scene augmentation method based on image generation according to claim 2, characterized in that, The adversarial loss function and The expression is as follows: ； ； in, It is random noise sampled from a Gaussian distribution. and These are the word vectors of the base class and the new class, respectively. and These represent real samples from the base class and the new class, respectively. and These represent the identification part and the generation part, respectively, and E represents the expectation; These are images sampled from known class distributions. These are images sampled from the new category distribution.

4. The few-sample power scene augmentation method based on image generation according to claim 2, characterized in that, Fine-tuning the generative adversarial network under the condition of a small number of new images also includes: In fine-tuning generative adversarial networks, semantic visual alignment loss is also used to prevent the pattern collapse problem caused by conditional GAN ​​networks remembering a small number of samples. Semantic visual alignment loss refers to the following: for the same noise vector, the similarity distribution between samples generated by the new category and samples generated by the selected base class should be consistent with the similarity distribution between word vectors. The expression is as follows: ； ； ； in and These are the distributions of visual similarity and word vector similarity, respectively. It is random noise sampled from a Gaussian distribution. and These are the word vectors for the categories selected from the new class and the base class, respectively. It is cosine similarity. The similarity between the two distributions is calculated using the Kullback-Leibler divergence. This represents the semantic visual alignment loss.

5. The few-sample power scene augmentation method based on image generation according to claim 2, characterized in that, Fine-tuning the generative adversarial network under the condition of a small number of new images also includes: In fine-tuning generative adversarial networks, cross-class similarity alignment loss is also used to prevent the pattern collapse problem caused by conditional GAN ​​networks remembering a small number of samples. Cross-class similarity alignment loss refers to: Cross-class similarity alignment loss refers to the loss in random sampling This noise, The noise is generated within the same category. There are 10 samples, and the similarity between each pair of these samples is calculated. Normalization yields the similarity distribution for the current category: ； in and These are two random noise samples taken from a Gaussian distribution. These are the word vectors of the new class; Calculate in the same way Then, cross-category similarity ranking loss. as follows: ； in The similarity between the two distributions is calculated using the Kullback-Leibler divergence.

6. The few-sample power scene augmentation method based on image generation according to claim 1, characterized in that, Image quality-based evaluation techniques are used to filter out realistic and diverse images from pseudo-samples, including: Given a small number of images and generated pseudo samples, we aim to evaluate and filter pseudo samples through three aspects: sample quality assessment, sample similarity assessment, and sample diversity assessment, to obtain an augmented dataset X for the task of recognizing new categories of objects in the power scenario. ; in, For quality score, For similarity scores, To score for diversity, This is a given filtering threshold, set to 0.

6. , as well as These are the weights of the three evaluation scores, all of which are set to 1 by default.

7. The few-sample power scene augmentation method based on image generation according to claim 6, characterized in that, Sample quality assessment includes: A quality assessment network is used to extract image features and output quality scores for pseudo-samples. ,in These are generated pseudo-samples. It is an existing method for image quality evaluation. The higher the quality score, the less noise such as artifacts in the generated image. Sample similarity assessment includes: Calculate the cosine similarity between the pseudo-sample and the given image of the new category, and convert it into a similarity score. The higher the similarity, the higher the similarity score. The calculation formula is as follows: ； in These are the generated samples. The new category has a small number of known real samples; Sample diversity assessment includes: The generated samples are clustered into several classes, and a score is given based on the distance between the current sample and all cluster centers. The greater the minimum distance between the current sample and all cluster centers, the higher the diversity score. The calculation formula is as follows: )； in, These are the sample centers of the new category after clustering.

8. A few-sample power scene augmentation and classification method based on image generation, comprising the following steps: The image-based few-sample power scene generation method according to any one of claims 1-7 obtains an augmented dataset for the task of recognizing new categories of objects in power scenes; A fully supervised network was trained using the augmented dataset to identify samples of the new class.

Images