A small sample scene data augmentation method based on a generative model

Abstract

The application discloses a small sample scene data expansion method based on a generative model and belongs to the technical field of image data expansion. Scene image samples with category identification are acquired, a minority sample set is determined according to the sample quantity of each category and is subjected to feature coding to obtain a sample feature vector set; the difficulty of the samples is calculated according to the category scarcity degree, the boundary proximity degree, the recognition confidence fluctuation and the historical misclassification situation to determine a preferential expansion sample set; the same-class near-neighbor interpolation is performed on the preferential expansion samples to obtain a first expansion sample set; the public features of each minority category are extracted to form a category prototype vector, and the category prototype vector is input into a WGAN network generator together with a random noise vector to generate a second expansion sample set under the category prototype constraint; the two types of expansion samples are subjected to quality screening to form a target expansion sample set, and the target expansion sample set is used to expand a training data set. The application can improve the pertinence of the expansion of the minority class samples.

Description

Technical Field

[0001] This invention belongs to the field of image data augmentation technology, and particularly relates to a method for augmenting small sample scene data based on a generative model. Background Technology

[0002] In intelligent analysis tasks such as image recognition, object detection, scene classification, and defect identification, the class distribution of training data directly impacts model training performance. When there are classes with significantly insufficient sample sizes in the training data, the model is prone to being dominated by the majority class during training, leading to insufficient feature representation of the minority class samples. This results in problems such as recognition bias, classification confusion, and decreased generalization ability. Especially in complex scene images, different classes may exhibit small texture differences, weak boundary features, and significant intra-class variations, making it even more difficult to accurately learn small sample classes.

[0003] Existing technologies typically employ methods such as data augmentation, oversampling, or generative models to augment minority class samples. Among these, the traditional SMOTE (Synthetic Minority Over-sampling Technique) method can generate augmented samples through neighborhood interpolation between minority class samples. However, it usually relies primarily on neighborhood relationships between samples to determine the interpolation targets, making it difficult to consider factors such as the scarcity of different minority class samples, the location of class boundaries, and the difficulty of recognition to determine which samples need to be prioritized for augmentation. This results in a lack of specificity in the selection of augmentation targets. Furthermore, the interpolated samples may deviate from the original feature distribution of the minority class samples.

[0004] Generative models can generate new samples by learning the data distribution of training samples. However, in small sample scenarios, the number of minority class samples available for generative models to learn is limited. When the generation process lacks constraints related to specific class features, the generated samples are prone to unstable class features, deviation from the corresponding class distribution, and fluctuations in generation quality.

[0005] Therefore, how to improve the targeting of minority class sample augmentation objects in small sample scenarios and ensure that the generated augmented samples maintain consistency with the feature distribution of the corresponding class is a problem that needs to be solved in the process of data augmentation in small sample scenarios. Summary of the Invention

[0006] To address the problems existing in the prior art, this invention proposes a method for augmenting small sample scene data based on a generative model.

[0007] The technical solution of the present invention is as follows:

[0008] A method for augmenting small sample scene data based on a generative model includes:

[0009] Obtain scene image samples with category labels, determine the minority class sample set according to the number of samples in each category, and perform feature encoding on the minority class sample set to obtain a sample feature vector set;

[0010] The sample difficulty of each minority class sample is calculated based on the scarcity of the class, the proximity of the boundary, the fluctuation of the identification confidence and the history of misclassification, and the sample set is prioritized for expansion based on the sample difficulty.

[0011] Based on the sample feature vector set, perform similar nearest neighbor interpolation on the priority expanded sample set to obtain the first expanded sample set;

[0012] Based on the sample feature vector set, common features of each minority class are extracted to form a class prototype vector set. The class prototype vector and random noise vector are input into the WGAN network generator to generate a second expanded sample set under the class prototype constraint.

[0013] The first and second expanded sample sets are subjected to quality screening to obtain a target expanded sample set, which is then used to expand the training dataset.

[0014] Furthermore, when performing feature encoding on the minority class sample set, the scene image samples are subjected to size unification and pixel normalization processing. Color distribution data, texture change data, edge response data, and region structure data are extracted to form a basic feature map set. The basic feature map set is input into the feature encoding network, and multi-layer intermediate feature maps are obtained through convolutional mapping, downsampling, and channel expansion. The feature maps at different levels are aggregated to obtain a fused feature map. The fused feature map is subjected to global average convergence, linear mapping, and vector normalization to obtain the sample feature vector corresponding to each minority class sample.

[0015] Further, when calculating the sample difficulty, the category scarcity value is determined based on the number of samples in the current minority class and the maximum number of samples in all classes; the boundary proximity value is determined based on the average distance between the current minority class sample and its nearest neighbor sample of the same class and the minimum distance between it and its nearest neighbor sample of a different class; the recognition confidence fluctuation value is determined based on the multi-round recognition results of the current minority class sample; and the historical misclassification value is determined based on the historical misclassification count, consecutive misclassification count, and total number of discrimination counts of the current minority class sample. The category scarcity value, boundary proximity value, recognition confidence fluctuation value, and historical misclassification value are normalized and weighted to obtain the sample difficulty. The samples are then sorted according to the sample difficulty, and minority class samples whose ranking is within a preset proportion range and whose sample difficulty is higher than a preset difficulty threshold are extracted to form the priority expansion sample set.

[0016] Further, when obtaining the first expanded sample set, for each priority expanded sample, candidate samples of the same category with consistent category identifiers are determined from the sample feature vector set. Nearest neighbor samples of the same category are determined according to the feature distance between the priority expanded sample and each candidate sample of the same category, and target nearest neighbor samples are selected from them. Feature interpolation is performed between the feature vectors of the priority expanded sample and the target nearest neighbor sample according to the interpolation coefficient constrained by the preset interpolation range. The obtained interpolated feature vector is subjected to boundary validity verification and truncation correction according to the feature distribution interval corresponding to the candidate samples of the same category, and the first expanded sample is obtained based on the corrected interpolated feature vector.

[0017] Furthermore, when forming the category prototype vector set, the sample feature vectors are merged according to the category and the sample feature vectors in the same category are dimensionally aligned. The mean, median, variance, and non-zero response frequency of each feature dimension are statistically analyzed. Stable common feature components are selected based on the variance and non-zero response frequency. The component weights are determined based on the non-zero response frequency and variance corresponding to each common feature component. After weighting the common feature components, they are assembled according to a preset dimension order. Preset compensation values ​​are written for feature dimensions that do not meet the retention conditions, thus obtaining the category prototype vectors corresponding to each minority class.

[0018] Further, when obtaining the second expanded sample set, the category prototype vector corresponding to the minority class and the random noise vector are dimension-matched and concatenated, and then input into the WGAN network generator to generate candidate expanded samples; the candidate expanded samples and the real minority class samples are input into the discriminator, the Wasserstein distance used to characterize the difference in their distributions is calculated, and the consistency error between the feature vector corresponding to the candidate expanded sample and the category prototype vector is calculated. The joint loss is formed based on the Wasserstein distance and the consistency error, and the generator parameters are updated. The candidate expanded samples that meet the preset convergence conditions are determined as the second expanded samples.

[0019] Furthermore, when performing quality screening on the first and second expanded sample sets, the expanded samples in both sets are classified, and expanded samples whose classified category matches the corresponding category identifier and whose identification confidence is higher than a preset threshold are retained. The retained expanded samples are feature-encoded to obtain expanded sample feature vectors, and the feature distance between the expanded sample feature vectors and the corresponding category prototype vectors is calculated. Expanded samples whose feature distance is within a preset retention interval are retained. The similarity between expanded samples and between expanded samples and the original minority class samples is calculated, and expanded samples with similarity higher than a preset duplication threshold are deleted to obtain the target expanded sample set.

[0020] Furthermore, when expanding the training dataset using the target expanded sample set, the category identifier, source identifier, and screening identifier of the target expanded samples are verified, and the verified target expanded samples are written into the corresponding category positions of the original training dataset according to the category. During the writing process, the number of samples in each category is counted, and it is determined whether to continue writing expanded samples of the corresponding category based on the number of samples and the preset balance condition, thus forming the expanded training dataset, and the sample number, category number, source number, and writing batch number corresponding to each sample are recorded.

[0021] Compared with the prior art, the present invention has the following beneficial effects:

[0022] This invention calculates the sample difficulty of minority class samples based on category scarcity, boundary proximity, identification confidence fluctuations, and historical misclassification, and determines the priority sample set to be expanded based on the sample difficulty. By comprehensively considering the scarcity of minority class samples, category boundary location, and actual identification difficulty, it solves the problem of insufficient targeting when determining expansion targets solely based on random selection or a single neighborhood relationship. This allows subsequent data expansion to prioritize minority class samples that are more in need of expansion, improving the targeting of sample expansion.

[0023] This invention performs a nearest neighbor search on priority expanded samples based on sample feature vectors, and performs feature interpolation between the priority expanded samples and their nearest neighbors within a preset interpolation range. Simultaneously, it verifies and corrects the boundary validity of the interpolation results based on the feature distribution of the same type of samples. This reduces the deviation of the interpolation results from the corresponding minority class feature distribution, ensuring that the resulting first expanded sample maintains good distribution consistency with the original minority class samples and increases the effective coverage of minority class samples in the feature space.

[0024] This invention generates class prototype vectors based on the common features of minority class samples. These class prototype vectors, along with a random noise vector, are input into a WGAN (Wasserstein Generative Adversarial Network) generator to generate second expanded samples under class prototype constraints. By utilizing class prototype vectors to represent the common features of the corresponding minority classes, this invention addresses the problem of class drift caused by the lack of class feature constraints during the generation process. It ensures that the generation process is constrained by the common features of the corresponding classes, thereby improving the class consistency and generation stability of the second expanded samples.

[0025] This invention performs a unified quality screening on the first expanded samples obtained by nearest neighbor interpolation and the second expanded samples generated by category prototype constraints. It also filters out expanded samples that do not meet the retention criteria by combining the category recognition results, the feature distance between the expanded sample feature vector and the corresponding category prototype vector, and the similarity between samples. This reduces the number of expanded samples with inconsistent category recognition results, deviations from the corresponding category feature distribution, or high repetition with existing samples included in the training dataset. The final target expanded samples meet the corresponding screening requirements in terms of category consistency, reasonable distribution, and sample diversity, thus improving the usability of the expanded training dataset. Attached Figure Description

[0026] Figure 1 A flowchart illustrating the method for augmenting small sample scenario data based on generative models. Detailed Implementation

[0027] The present invention will be further illustrated below with reference to the accompanying drawings and specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. After reading this invention, any modifications of the invention in various equivalent forms by those skilled in the art will fall within the scope defined by the appended claims.

[0028] Example 1:

[0029] This invention provides a method for augmenting small sample scene data based on a generative model, such as... Figure 1 As shown, it includes:

[0030] Obtain scene image samples with category labels, determine the minority class sample set according to the number of samples in each category, and perform feature encoding on the minority class sample set to obtain the sample feature vector set;

[0031] Specifically, the category identifiers corresponding to each scene image sample are read, and the number of samples in each category is counted according to the category identifiers. Categories with a sample number lower than a preset value are identified as minority categories. Scene image samples belonging to minority categories are aggregated to form a minority category sample set.

[0032] The sample difficulty of each minority class is calculated based on the scarcity of the class, the proximity of the boundary, the fluctuation of the identification confidence, and the historical misclassification. The sample set is then prioritized for expansion based on the sample difficulty.

[0033] Based on the sample feature vector set, perform similar nearest neighbor interpolation on the priority expanded sample set to obtain the first expanded sample set;

[0034] Based on the sample feature vector set, common features of each minority class are extracted to form a class prototype vector set. The class prototype vector and random noise vector are input into the WGAN network generator to generate a second expanded sample set under the class prototype constraint.

[0035] The first and second expanded sample sets are quality-screened to obtain a target expanded sample set, which is then used to expand the training dataset. Exemplarily, this invention can be applied to scenarios such as power equipment defect detection, industrial product quality inspection, and security monitoring anomaly identification. In these scenarios, collected and labeled equipment defect images, product quality inspection images, or security monitoring scene images can be used as scene image samples. When the number of minority class samples (such as defective or anomaly categories) is significantly less than the number of normal class samples, the method described in this invention can be used to expand the minority class scene images to form a more balanced training dataset for subsequent image recognition model training. It should be noted that the above application scenarios are merely illustrative, and this invention is not limited to these scenarios. The method described in this invention can also be used for data expansion in remote sensing image analysis, medical image classification, and other image recognition tasks with unbalanced class distribution or insufficient minority class image samples.

[0036] Example 2:

[0037] This embodiment further designs, based on Embodiment 1, by performing feature encoding on the minority class sample set, uniformizing the size and normalizing the pixels of the scene image samples, and extracting color distribution data, texture change data, edge response data and region structure data to form a basic feature map set.

[0038] Specifically, connected component identification is performed on the scene image, and a region structure feature map with the same size as the scene image is established for each connected component. The pixel position belonging to the current connected component is determined as the region response position. An area normalization value is written to the region response position according to the area of ​​the connected component. The boundary response position is determined according to the coverage of the outer boundary and written as the boundary response value. The center response position is determined according to the center position of the connected component and written as the center position response value according to the distance from the center position to each pixel position in the region. The obtained area response map, boundary response map and center position response map are used as the feature channels corresponding to the region structure data, and are arranged with color distribution data, texture change data and edge response data according to the preset channel order to form a basic feature map set.

[0039] The basic feature map set is input into the feature encoding network, and multi-layer intermediate feature maps are obtained through convolutional mapping, downsampling and channel expansion.

[0040] Specifically, the basic feature map set is input into the feature encoding network according to a preset channel order. Local receptive fields are scanned on each input channel using convolutional kernels, and corresponding data is mapped and calculated at each window position to form a convolutional feature map. Subsequently, the convolutional feature map is downsampled to reduce its spatial size, and the number of output channels is increased during subsequent mapping processes, so that different mapping results are written to different channel positions. The output results of each layer form multiple layers of intermediate feature maps. For example, in this embodiment, the scene image is uniformly 224×224 pixels, and the pixels are normalized to the range [0, 1]. The basic feature channels include RGB 3 channels and 9 channels for texture variation, edge response, area response map, boundary response map, and center position response map. The feature encoding network adopts a four-layer convolutional structure with a 3×3 kernel size. The number of output channels for each layer is 64, 128, 256, and 512 respectively. Each convolutional layer is followed by 2×2 max pooling for downsampling. After global average pooling and linear mapping, a 128-dimensional sample feature vector is output and L2 vector normalization is performed.

[0041] Feature maps from different levels are aggregated to obtain a fused feature map.

[0042] For example, when aggregating feature maps of different levels, the spatial size of each feature map is adjusted to a uniform size, and the edge position response and local detail response in the shallow feature map, as well as the category-related response and overall region response in the deep feature map, are extracted respectively. They are then spliced ​​together according to a preset channel order, and the spliced ​​feature map is compressed by channel to obtain a fused feature map.

[0043] Global average convergence, linear mapping, and vector normalization are performed on the fused feature maps to obtain sample feature vectors corresponding to each minority class sample.

[0044] Specifically, when performing global averaging convergence on the fused feature map, the average value of all spatial locations within each channel is calculated, and the converged feature vector is formed according to the channel order. During the linear mapping process, the converged feature vector is mapped to the weight parameters of the linear mapping layer and combined with the bias parameters to obtain a fixed-length encoding result. Then, the fixed-length encoding result is vector normalized to obtain the sample feature vector. For example, this embodiment also establishes a feature decoding network corresponding to the feature encoding network to restore the interpolated feature vector to a scene image sample. The feature decoding network restores the 128-dimensional feature vector to a 7×7×256 initial decoded feature map through linear mapping, then restores the spatial size step by step through 5 transposed convolution upsampling operations, finally outputting a 224×224×3 pixel matrix, and performs inverse normalization processing corresponding to the aforementioned pixel normalization to obtain a restored image with the same size and number of color channels as the original scene image. The feature decoding network is optimized using mean squared error loss during the training phase.

[0045] Example 3:

[0046] This embodiment is further designed based on Embodiment 1 in that: when calculating the sample difficulty, the category scarcity value is determined based on the current number of samples in the minority class and the maximum number of samples in all categories.

[0047] Specifically, the total number of samples corresponding to the current minority class is counted, and the maximum number of samples in all classes is read. The difference between the maximum number of samples and the total number of samples in the current minority class is calculated, and the difference is proportionally converted to obtain a class scarcity value used to characterize the scarcity of samples in the current class.

[0048] The boundary proximity value is determined based on the average distance between the current minority class sample and its nearest neighbor sample of the same class, and the minimum distance between the current minority class sample and its nearest neighbor sample of the opposite class.

[0049] Specifically, the distance values ​​between the current minority class sample and a preset number of similar neighboring samples are calculated, and the average distance value is obtained by averaging the distance values; the distance values ​​between the current minority class sample and a preset number of dissimilar neighboring samples are calculated, and the minimum distance value is extracted from them; the distance difference between the average distance value and the minimum distance value is calculated, and the distance difference is proportionally converted to the average distance value to obtain a boundary proximity value used to characterize the degree of boundary proximity of the current minority class sample.

[0050] The identification confidence fluctuation value is determined based on the multi-round identification results of the current minority class samples.

[0051] Specifically, in the continuous training rounds of the scene image recognition model, after each preset training round, the current minority class sample is input into the scene image recognition model under the parameter state of the corresponding training round. The recognition confidence of the current minority class sample corresponding to the true category is recorded, and a category confidence value sequence is formed according to the training round order. The maximum and minimum confidence in the category confidence value sequence are extracted to obtain the confidence range. The average confidence is calculated by the average of the recognition confidence in each round. The confidence fluctuation amplitude is obtained according to the deviation of the recognition confidence in each round from the average confidence. The confidence range, average confidence, and confidence fluctuation amplitude are normalized respectively. The normalized average confidence is converted into an underconfidence value. The confidence range, underconfidence value, and confidence fluctuation amplitude are then combined according to preset weights to obtain the recognition confidence fluctuation value.

[0052] The historical misclassification value is determined based on the number of historical misclassifications, the number of consecutive misclassifications, and the total number of discriminations for the current minority class samples.

[0053] Specifically, the category discrimination results generated by the scene image recognition model for the current minority class sample in the aforementioned continuous training rounds and the corresponding training rounds are written into the historical classification record; the historical classification record of the current minority class sample is read in the order of training rounds, and the discrimination result where the predicted category is inconsistent with the true category of the current minority class sample is recorded as a misclassification. The cumulative number of misclassifications in the historical classification record is counted, and the maximum number of consecutive misclassifications in the continuous training rounds is counted. The total number of category discriminations corresponding to the historical classification record is determined as the total number of discriminations; the misclassification ratio is determined according to the proportion of misclassifications to the total number of discriminations, and the consecutive misclassification ratio is determined according to the proportion of consecutive misclassifications to the total number of discriminations. The misclassification ratio and the consecutive misclassification ratio are combined according to a preset weight to obtain the historical misclassification value used to characterize the historical misclassification situation of the current minority class sample.

[0054] The sample difficulty is calculated by normalizing and weighting the values ​​of category scarcity, boundary proximity, identification confidence fluctuation, and historical misclassification. The samples are then sorted according to their difficulty, and minority class samples whose ranking falls within a preset proportion range and whose difficulty exceeds a preset difficulty threshold are extracted to form a priority expansion sample set. For example, in this embodiment, the four components of sample difficulty are normalized and weighted as follows: category scarcity weight 0.30, boundary proximity weight 0.30, identification confidence fluctuation weight 0.20, and historical misclassification weight 0.20. The selection criteria for the priority expansion sample set are: ranking within the top 50% and sample difficulty not lower than 0.45. The search count for both similar and dissimilar nearest neighbor samples is 5. For example, in this embodiment, the scene image recognition model used to calculate identification confidence fluctuation and historical misclassification adopts a convolutional neural network structure, with a total training epoch of 100 epochs, recording the identification confidence of each minority class sample every 10 epochs. During training, the batch size was 32, the SGD optimizer was used, the learning rate was 0.01, and the momentum was 0.9.

[0055] Example 4:

[0056] This embodiment is further designed based on Embodiment 1 in that: when the first expanded sample set is obtained, for each priority expanded sample, candidate samples of the same type with consistent category identifiers are determined from the sample feature vector set, and the nearest neighbor samples of the same type are determined according to the feature distance between the priority expanded sample and each candidate sample of the same type.

[0057] For example, the feature distance is determined as follows: read the feature vector of the current priority expanded sample and the feature vector of the candidate sample, calculate the difference between the corresponding components in each feature dimension, square and sum the differences in each dimension, and then take the square root of the sum to obtain the feature distance between the current priority expanded sample and the candidate sample; sort the obtained feature distances from smallest to largest, and determine the nearest neighbor samples of the same type based on the sorting results.

[0058] The target nearest neighbor sample is selected from similar nearest neighbor samples. Specifically, during the similar nearest neighbor search, the current priority expansion sample itself is excluded, and a predetermined number of similar nearest neighbor samples are selected from other samples with the same category label as the current priority expansion sample, in ascending order of feature distance. The selection weight is determined based on the feature distance between each similar nearest neighbor sample and the current priority expansion sample; the smaller the feature distance, the greater the corresponding selection weight. When determining the selection weight, a predetermined positive number is added to each feature distance, the reciprocal is taken, and the resulting value is normalized so that the sum of the selection weights corresponding to each similar nearest neighbor sample is one. The target nearest neighbor sample is selected from the similar nearest neighbor samples according to the normalized selection weight.

[0059] Feature interpolation is performed between the feature vectors of the priority expanded samples and the target nearest neighbor samples according to interpolation coefficients constrained by a preset interpolation range, resulting in an interpolated feature vector. The boundary validity of the interpolated feature vector is checked based on the value range of each feature dimension for similar candidate samples. When the interpolation result of any feature dimension exceeds the corresponding feature distribution interval, the value of that dimension is corrected to the boundary position of the corresponding feature distribution interval, resulting in a corrected interpolated feature vector. For example, in this embodiment, the number of similar nearest neighbor samples searched is 5, and the constraint range of the interpolation coefficients is [0.2, 0.8]. The weights are calculated by adding a preset positive number to each feature distance, taking the reciprocal, and then normalizing the sum of the weights to 1.

[0060] The corrected interpolated feature vector is input into a pre-established feature decoding network. Through linear mapping, the corrected interpolated feature vector is converted into an initial decoded feature map with a preset spatial size and number of channels. The initial decoded feature map is then sequentially upsampled and convolutionally mapped to gradually restore the image spatial size. The output mapping yields a pixel matrix with the same size and number of color channels as the original scene image. The pixel matrix undergoes inverse normalization processing, corresponding to the aforementioned pixel normalization, to obtain the first augmented sample. The first augmented sample is presented as a scene image sample, recording the corresponding source sample identifier, target nearest neighbor sample identifier, and interpolation coefficients. This allows the first augmented sample to undergo subsequent quality screening in the same image sample data format as the second augmented sample, and it is then written into the original training dataset.

[0061] Example 5:

[0062] This embodiment is further designed based on Embodiment 1 in that: when forming the category prototype vector set, the sample feature vectors are merged according to the category and the sample feature vectors in the same category are dimensionally aligned.

[0063] Specifically, when aligning the feature vectors of samples within the same category, the dimension, component order, and value range of each sample feature vector are checked. Under the condition that the dimension and component correspondence of each sample feature vector are consistent, a feature sequence aligned within the category is formed.

[0064] Calculate the mean, median, variance, and non-zero response frequency for each feature dimension, and then select stable common feature components based on variance and non-zero response frequency.

[0065] Specifically, the variance values ​​corresponding to each feature dimension are compared with a preset fluctuation threshold, and the non-zero response frequency is compared with a preset frequency threshold. Feature components with variance values ​​lower than the preset fluctuation threshold and non-zero response frequencies higher than the preset frequency threshold are determined as common feature components.

[0066] The component weights are determined based on the non-zero response frequency and variance of each common feature component. Specifically, for feature dimensions that meet the retention criteria, the mean and median of that feature dimension are compared. When the difference between the mean and median does not exceed a preset difference threshold, the mean is determined as the basic feature value of that common feature component; when the difference between the mean and median exceeds the preset difference threshold, the median is determined as the basic feature value of that common feature component. The non-zero response frequency is converted into a non-zero response proportion, and the variance is increased by a preset stability constant and then the reciprocal is taken to obtain the feature stability. The product of the non-zero response proportion and the feature stability is used as the initial component weights. Each initial component weight is normalized to obtain the component weights corresponding to each common feature component, where the higher the non-zero response frequency and the smaller the variance, the larger the corresponding component weight. The basic feature values ​​of each common feature component are combined with the corresponding component weights to obtain weighted common feature components, which are then assembled according to a preset dimension order. Preset compensation values ​​are written for feature dimensions that do not meet the retention criteria to obtain the category prototype vectors corresponding to each minority class. For example, in this embodiment, the fluctuation threshold for stability screening is set to 1.0 times the median variance of each feature dimension, and the non-zero response frequency threshold is set to 0.5. That is, feature dimensions with variances lower than the median and non-zero response frequencies higher than 50% are identified as common feature components.

[0067] For example, when assembling the category prototype vector, a vector position corresponding to the original feature dimension order is established according to the dimension number corresponding to each retained feature component, and the retained feature components are written into the corresponding position; for feature dimensions that do not meet the retention conditions, compensation values ​​can be written according to preset compensation rules, and the compensation values ​​can be zero values, the mean value or median value of the dimension in the corresponding category feature subset; after completing the data writing of each vector position, the category identifier is bound to the obtained vector to form the category prototype vector corresponding to the minority class.

[0068] Example 6:

[0069] This embodiment, based on Embodiment 1, is further designed as follows: when obtaining the second expanded sample set, the category prototype vector corresponding to the minority class and the random noise vector are dimension-matched and concatenated, and then input into the WGAN network generator to generate candidate expanded samples; the candidate expanded samples and the real minority class samples are input into the discriminator, the Wasserstein distance used to characterize the difference in their distributions is calculated, and the consistency error between the feature vector corresponding to the candidate expanded sample and the category prototype vector is calculated. The joint loss is formed based on the Wasserstein distance and the consistency error, and the generator parameters are updated. The candidate expanded samples that meet the preset convergence conditions are determined as the second expanded samples.

[0070] Specifically, the prototype vectors of the corresponding minority classes are dimension-matched and concatenated with random noise vectors and then input into the WGAN network generator. The generator performs feature mapping and spatial size restoration, and outputs candidate augmented samples with the same image size and number of color channels as the original scene image.

[0071] The candidate augmented samples are input into the feature encoding network according to the feature encoding method of the aforementioned minority class samples. After convolutional mapping, downsampling, channel expansion, feature aggregation, global averaging convergence, linear mapping and vector normalization, the feature vector corresponding to the candidate augmented samples is obtained. The feature vector is compared with the corresponding minority class class prototype vector according to the feature dimension, and the class prototype consistency error is determined according to the difference between the feature values ​​of each dimension.

[0072] The candidate augmented samples and the real minority class samples are input into the WGAN discriminator. The discriminator scores the candidate augmented samples and the real minority class samples are obtained respectively. The Wasserstein distance, which is used to characterize the difference between the distribution of the candidate augmented samples and the distribution of the real minority class samples, is determined based on the difference in the average discriminant scores of the two classes. The Wasserstein distance and the class prototype consistency error are normalized and weighted according to the preset loss weight to obtain the joint loss. The joint loss is used as the basis for updating the generator parameters.

[0073] During the iterative training of the generator, the joint loss and Wasserstein distance are continuously recorded. When the change in joint loss within a preset training epoch is lower than a preset loss change threshold, and the Wasserstein distance is lower than a preset distribution difference threshold, the preset convergence condition is met, and the candidate augmented sample output by the current generator is determined as the second augmented sample. For example, in this embodiment, the dimension of the random noise vector is 64, following a standard normal distribution, and the dimension of the class prototype vector is 128. The WGAN network generator adopts a fully connected layer structure, with a layer structure of 64→256→512→128; the discriminator adopts a fully connected layer structure, with a layer structure of 128→256→64→1. During training, the batch size is 32, the RMSProp optimizer is used, the learning rate is 0.0001, and the discriminator weight pruning range is [-0.01, 0.01]. In the joint loss, the distribution error weight corresponding to the Wasserstein distance is 0.4, and the class prototype consistency error weight is 0.6. The maximum number of iterations is 200. The convergence criteria are that the change in the joint loss is less than 0.001 for 10 consecutive iterations and the Wasserstein distance is less than 0.1.

[0074] Example 7:

[0075] This embodiment, based on Embodiment 1, further designs the following: When performing quality screening on the first and second expanded sample sets, the expanded samples in both sets are classified, and expanded samples whose classified category matches the corresponding category identifier and whose identification confidence is higher than a preset threshold are retained; the retained expanded samples are feature-encoded to obtain expanded sample feature vectors, the feature distance between the expanded sample feature vectors and the corresponding category prototype vectors is calculated, and expanded samples whose feature distance is within a preset retention interval are retained; the similarity between expanded samples and between expanded samples and the original minority class samples is calculated, and expanded samples with similarity higher than a preset duplication threshold are deleted to obtain the target expanded sample set.

[0076] When calculating the similarity between expanded samples and between expanded samples and the original minority class samples, feature encoding is performed on the expanded samples and the original minority class samples based on the aforementioned feature encoding process, resulting in corresponding expanded sample feature vectors and original minority class sample feature vectors, respectively. For any two samples to be compared, cosine similarity is calculated based on the feature vectors corresponding to the two samples, and the cosine similarity is used as a similarity value characterizing the closeness of the feature spaces of the two samples. The larger the cosine similarity, the closer the directions of the two samples are in the feature space, and the more similar the feature information they contain. When the cosine similarity between an expanded sample and other expanded samples or the original minority class samples is higher than a preset repetition threshold, the corresponding expanded sample is identified as a sample with a high degree of repetition and deleted, resulting in the target expanded sample set.

[0077] Specifically, for each expanded sample retained after quality screening, its source identifier, category identifier, and screening identifier are recorded. The source identifier indicates whether the corresponding expanded sample originates from a first or second expanded sample set, and the screening identifier records whether the corresponding expanded sample meets the retention criteria, thus forming a screening record corresponding to the target expanded sample. For example, in this embodiment, the confidence threshold for category identification is set to 0.60, the distance retention interval between the category prototype vector and the target vector is set to [0.1, 2.0], and the cosine similarity duplication threshold is set to 0.985. That is, when the cosine similarity between two samples exceeds 0.985, they are determined to have a high degree of duplication, and one of them is deleted.

[0078] Example 8:

[0079] This embodiment is further designed based on Embodiment 1 in that: when expanding the training dataset using the target expanded sample set, the category identifier, source identifier, and screening identifier of the target expanded sample are verified.

[0080] Specifically, the category identifier, source identifier, and filter identifier corresponding to each target augmented sample are read sequentially. A consistency check is performed on the category identifier to determine whether it is consistent with the corresponding category identifier recorded in the augmentation generation stage. A source correspondence check is performed on the source identifier to determine whether the augmented sample comes from the first augmented sample set or the second augmented sample set. A retention status check is performed on the filter identifier, and augmented samples with consistent category identifiers, valid source identifiers, and filter identifiers that meet the retention conditions are identified as samples to be written.

[0081] According to the category, the verified target augmented samples are written into the corresponding category positions of the original training dataset. During the writing process, the number of samples in each category is counted, and it is determined whether to continue writing augmented samples of the corresponding category based on the number of samples and the preset balance condition, so as to form the augmented training dataset and record the sample number, category number, source number and writing batch number of each sample.

[0082] For example, the expanded training dataset is traversed according to the sample arrangement order, and a corresponding sample number is assigned to each sample. The category identifier, source identifier, and write batch identifier corresponding to the sample are recorded as category number, source number, and write batch number, respectively. A sample index record corresponding one-to-one with the expanded training dataset is established so as to distinguish and trace the original samples and expanded samples from different sources.

[0083] Application Examples:

[0084] To verify the feasibility of the generative model-based small-sample scene data augmentation method described in this invention in training data of scene images with unbalanced class distribution, this example applies the method of this invention to the training data augmentation process of a complex scene image recognition model to verify whether the method can augment classes with a small number of samples and make the number of samples in each class tend to be balanced after augmentation.

[0085] The scene image data used in this application embodiment comes from a pre-established scene image sample library. Each scene image in the sample library has been categorized and manually verified, and then grouped according to category. The categorized scene images are divided into a training set and a test set according to a preset ratio. The training set is used to perform data augmentation and scene image recognition model training of this invention. The test set maintains the original number of samples and category distribution, and is only used to evaluate the model's recognition performance, avoiding the inclusion of augmented samples in the test set and thus affecting the evaluation results.

[0086] This application example uses a convolutional neural network (CNN) as a unified scene image recognition model. During training before and after data augmentation, the model structure, parameter initialization method, number of training epochs, batch size, optimization method, and learning rate settings remain consistent. Only the sample composition in the training dataset is changed to compare recognition results under different training data conditions. Model training and testing are performed on a data processing device equipped with a multi-core processor, graphics processor, and ample memory. A program runtime environment supporting deep learning model training is used for training and inference, and all experiments are conducted under the same software version and hardware conditions. Specifically, the CNN used in this application example contains four convolutional layers, each with a 3×3 kernel size and output channels of 64, 128, 256, and 512 respectively. Each convolutional layer is followed by 2×2 max pooling and ReLU activation. The training epochs are 100, the batch size is 32, the SGD optimizer is used, the learning rate is 0.01, and the momentum is 0.9. The model structure and hyperparameters remain consistent throughout the training process before and after data augmentation.

[0087] This application example uses a training dataset containing image samples from five categories: A, B, C, D, and E. Before expansion, category A contained 120 samples, category B contained 95 samples, category C contained 80 samples, category D contained 60 samples, and category E contained 50 samples. There was a significant difference in the number of samples in each category. Category A, which had the most samples, had 2.4 times more samples than category E, which had the fewest samples, resulting in an application scenario with an imbalanced distribution of small sample categories.

[0088] First, the category identifiers of the image samples in each scene are read and the number of samples in each category is counted. Based on the number of samples, the minority class samples that need to be expanded are determined, and the minority class samples are feature-encoded according to the aforementioned embodiment to obtain the corresponding sample feature vectors.

[0089] Subsequently, the sample difficulty of minority class samples is calculated based on category scarcity, boundary proximity, identification confidence fluctuations, and historical misclassifications, and priority samples for expansion are determined according to sample difficulty. For priority expansion samples, a nearest neighbor search of the same class is performed in the feature space, and the first expansion sample is obtained through feature interpolation constrained by a preset interpolation range and boundary validity verification.

[0090] Meanwhile, the common features of minority class samples in the same category are statistically analyzed and filtered to form the category prototype vector of the corresponding category. The category prototype vector and the random noise vector are input into the WGAN network generator to generate the second expanded sample under the category prototype constraint.

[0091] After generating two types of augmented samples, a unified quality screening was performed on the first and second augmented samples. Based on the category identification results, the feature distance between the feature vector of the augmented sample and the corresponding category prototype vector, and the similarity with other samples, augmented samples that did not meet the retention criteria were removed. The selected target augmented samples were then written into the original training dataset according to their categories. The writing of augmented samples was controlled according to the number of samples in each category and the balance condition, resulting in the augmented training dataset. In the above augmentation process, the difficulty assessment stage selected 198 priority augmented samples from 405 minority class samples; the same-category nearest neighbor interpolation stage generated 198 first-stage augmented samples; and the category prototype constraint generation stage output approximately 2700 candidate augmented samples. The quality screening stage processed approximately 2900 augmented samples. After category identification verification, approximately 1500 were retained; after distribution distance screening, approximately 1500 were retained; after repeatability verification, approximately 1490 were retained; and finally, 1005 target augmented samples were selected according to the balance condition of each category and written into the training dataset. During the training of the WGAN network generator, the joint loss for each class showed a stable decreasing trend. Specifically, the joint loss for class A decreased from an initial 0.72 to 0.11 after 35 training epochs and then converged; for class B, it decreased from 0.75 to 0.09 after 42 epochs; for class C, from 0.68 to 0.10 after 30 epochs; for class D, from 0.80 to 0.12 after 48 epochs; and for class E, from 0.95 to 0.10 after 56 epochs. The Wasserstein distance for each class decreased to below 0.1 in the later stages of training, and the consistency error decreased to below 0.05, satisfying the preset convergence condition.

[0092] After the data augmentation, the number of samples in class A increased from 120 to 320, class B from 95 to 300, class C from 80 to 290, class D from 60 to 260, and class E from 50 to 240. This shows that the number of samples in each class was increased, and classes with fewer samples were expanded accordingly, thus narrowing the difference in sample size between the expanded classes compared to the original training data.

[0093] When using the augmented training data for scene image recognition, the recognition accuracies for classes A, B, C, D, and E were 78.5%, 74.3%, 72.1%, 69.8%, and 67.5%, respectively; the recall rates were 75.2%, 72.6%, 70.4%, 68.2%, and 65.9%, respectively; and the F1 scores were 76.8%, 73.4%, 71.2%, 69.0%, and 66.7%, respectively. These results demonstrate that the data augmentation process of this invention can generate an augmented training dataset for subsequent scene image recognition training.

[0094] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for augmenting small sample scene data based on a generative model, characterized in that, include: Obtain scene image samples with category labels, determine the minority class sample set according to the number of samples in each category, and perform feature encoding on the minority class sample set to obtain a sample feature vector set; The sample difficulty of each minority class sample is calculated based on the scarcity of the class, the proximity of the boundary, the fluctuation of the identification confidence and the history of misclassification, and the sample set is prioritized for expansion based on the sample difficulty. Based on the sample feature vector set, perform similar nearest neighbor interpolation on the priority expanded sample set to obtain the first expanded sample set; Based on the sample feature vector set, common features of each minority class are extracted to form a class prototype vector set. The class prototype vector and random noise vector are input into the WGAN network generator to generate a second expanded sample set under the class prototype constraint. The first and second expanded sample sets are subjected to quality screening to obtain a target expanded sample set, which is then used to expand the training dataset.

2. The method for augmenting small sample scene data based on a generative model according to claim 1, characterized in that, When performing feature encoding on the minority class sample set, the scene image samples are subjected to size unification and pixel normalization processing. Color distribution data, texture change data, edge response data, and region structure data are extracted to form a basic feature map set. The basic feature map set is input into the feature encoding network, and multi-layer intermediate feature maps are obtained through convolutional mapping, downsampling, and channel expansion. The feature maps at different levels are aggregated to obtain a fused feature map. The fused feature map is subjected to global average convergence, linear mapping, and vector normalization to obtain the sample feature vector corresponding to each minority class sample.

3. The method for augmenting small sample scene data based on a generative model according to claim 1, characterized in that, When calculating the sample difficulty, the category scarcity value is determined based on the number of samples in the current minority class and the maximum number of samples in all classes; the boundary proximity value is determined based on the average distance between the current minority class sample and its nearest neighbor sample of the same class and the minimum distance between the current minority class sample and its nearest neighbor sample of a different class; the recognition confidence fluctuation value is determined based on the multi-round recognition results of the current minority class sample; and the historical misclassification value is determined based on the historical misclassification count, consecutive misclassification count, and total number of discrimination counts of the current minority class sample. The category scarcity value, boundary proximity value, recognition confidence fluctuation value, and historical misclassification value are normalized and weighted to obtain the sample difficulty. The samples are then sorted according to the sample difficulty, and minority class samples whose ranking is within a preset proportion range and whose sample difficulty is higher than a preset difficulty threshold are extracted to form the priority expansion sample set.

4. The method for augmenting small sample scene data based on a generative model according to claim 1, characterized in that, When the first expanded sample set is obtained, for each priority expanded sample, candidate samples with the same category identifier are determined from the sample feature vector set. The nearest neighbor samples of the same category are determined according to the feature distance between the priority expanded sample and each candidate sample of the same category, and target nearest neighbor samples are selected from them. Feature interpolation is performed between the feature vectors of the priority expanded sample and the target nearest neighbor sample according to the interpolation coefficient constrained by the preset interpolation range. The obtained interpolated feature vector is subjected to boundary validity verification and truncation correction according to the feature distribution interval corresponding to the candidate samples of the same category, and the first expanded sample is obtained based on the corrected interpolated feature vector.

5. The method for augmenting small sample scene data based on a generative model according to claim 1, characterized in that, When forming the category prototype vector set, the sample feature vectors are merged according to the category and the sample feature vectors in the same category are dimensionally aligned. The mean, median, variance and non-zero response frequency of each feature dimension are counted. Stable common feature components are selected based on the variance and non-zero response frequency. The component weights are determined based on the non-zero response frequency and variance of each common feature component. After weighting the common feature components, they are assembled according to a preset dimension order. Preset compensation values ​​are written for feature dimensions that do not meet the retention conditions to obtain the category prototype vectors corresponding to each minority class.

6. The method for augmenting small sample scene data based on a generative model according to claim 1, characterized in that, When the second expanded sample set is obtained, the category prototype vector of the corresponding minority class and the random noise vector are dimension-matched and concatenated and then input into the WGAN network generator to generate candidate expanded samples. The candidate expanded samples and the real minority class samples are input into the discriminator to calculate the Wasserstein distance used to characterize the difference in their distributions, and the consistency error between the feature vector corresponding to the candidate expanded sample and the category prototype vector is calculated. The joint loss is formed based on the Wasserstein distance and the consistency error and the generator parameters are updated. The candidate expanded samples that meet the preset convergence conditions are determined as the second expanded samples.

7. The method for augmenting small sample scene data based on a generative model according to claim 1, characterized in that, When performing quality screening on the first and second expanded sample sets, the expanded samples in both sets are classified, and expanded samples whose classified category matches the corresponding category identifier and whose identification confidence is higher than a preset threshold are retained. The retained expanded samples are feature-encoded to obtain expanded sample feature vectors. The feature distance between the expanded sample feature vector and the corresponding category prototype vector is calculated, and expanded samples whose feature distance is within a preset retention interval are retained. The similarity between expanded samples and between expanded samples and the original minority class samples is calculated, and expanded samples with similarity higher than a preset duplication threshold are deleted to obtain the target expanded sample set.

8. The method for augmenting small sample scene data based on a generative model according to claim 1, characterized in that, When expanding the training dataset using the target expanded sample set, the category identifier, source identifier, and screening identifier of the target expanded samples are verified, and the verified target expanded samples are written into the corresponding category positions of the original training dataset according to the category. During the writing process, the number of samples in each category is counted, and it is determined whether to continue writing expanded samples of the corresponding category based on the number of samples and the preset balance condition, so as to form the expanded training dataset, and the sample number, category number, source number, and writing batch number of each sample are recorded.

Images