A few-shot image recognition method with semantically guided augmented data generation

Abstract

The present invention discloses a few-shot image recognition method for semantically guided augmented data generation, which comprises the following steps: constructing a semantically guided augmented data generation model; an instance-level data generation module, for each training data in the support set data, using a category activation mapping method to perform local semantic attention features, and simultaneously using semantic embedding fusion to generate global semantic attention features, thereby generating enhanced support set data; a prototype-level data generation module, for each category in the support set, guiding the few-shot class to combine with the relevant basic class to perform information migration based on a distance metric in a semantic space, estimating the type prototype and related distribution of the few-shot class, and sampling on the related distribution to obtain augmented sample data; and combining the enhanced support set data and the augmented sample data with the original support set as a new support set to train an image recognition model.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and more specifically, to a few-sample image recognition method for generating semantically guided augmented data. Background Art

[0002] The development of deep learning has led to significant progress in the field of computer vision. However, the massive amount of annotated data required to train deep learning models has hindered its widespread application. Traditional deep learning image recognition tasks often require a large amount of data for each category. This leads to overfitting when applied to few-shot learning scenarios with limited training data, severely impacting model robustness. Few-shot learning, inspired by how humans rapidly learn the world using prior knowledge, addresses this data scarcity issue from four perspectives: metric learning, training optimization, model structure, and data generation.

[0003] (1) Metric learning methods consider designing different metric methods based on the trained feature extractor to overcome the disadvantage of deep learning methods that require a large amount of training data to fit. The Prototypical Network is the most widely used method. It directly calculates the average of the features of a few samples of each class to obtain the class prototype, and then uses the cosine similarity measurement between the test data and each class prototype to obtain the classification result.

[0004] (2) Training optimization methods attempt to reduce the model's reliance on data volume through training methods. They typically utilize a meta-learning framework to train a few-shot model by learning a meta-learner that controls the learning process. MAML (Model-Agnostic Meta-Learning) utilizes a meta-learning framework to train the meta-learner through individual few-shot tasks, obtaining an initialization that can quickly fit a small number of samples. LSTM-based models can capture long-term knowledge across tasks and control the model's optimization process.

[0005] (3) Model-based methods are used to design the model structure so that it can overcome the problem of data scarcity, such as generating model parameters specifically for each few-sample task, or using Bayesian models to update robust predictions.

[0006] (4) Data generation methods attempt to use prior knowledge to generate data for few-sample data, which is the most direct way to solve the few-sample problem. Using unlabeled data or other basic data, data generation is performed on the few-sample data to increase the diversity of the few-sample data.

[0007] The deficiencies and defects of the above prior art are as follows:

[0008] Metric learning and model-based methods often require adjustments to the model structure itself, making them less versatile and incompatible with other model structures. Meta-learning frameworks for training optimization methods require high-level optimization of the meta-learner, which increases training costs. Furthermore, since the training objective can only indirectly affect model performance, it is not as effective as directly training the model. Regarding data generation methods, existing techniques only consider visual information transfer. When the data in the few-shot dataset is atypical and of unstable quality, this can easily affect the data generation of the entire category, leading to unstable model performance. Summary of the Invention

[0009] In order to solve the problems of the above-mentioned deficiencies and defects in the prior art, the present invention provides a few-shot image recognition method for semantically guided augmented data generation, which can generate rich, high-quality training data for few-shot class data.

[0010] In order to achieve the above-mentioned purpose of the present invention, the technical solutions adopted are as follows:

[0011] A few-shot image recognition method using semantically guided augmented data generation, wherein the method is as follows:

[0012] Construct a semantic-guided augmented data generation model, including instance-level data generation modules and prototype-level data generation modules;

[0013] The instance-level data generation module uses the instance-level data to generate support set data in the few-shot class data. For each training data in the support set data, the class activation mapping method is used to perform local semantic attention features, and semantic embedding fusion is used to generate global semantic attention features to generate enhanced support set data.

[0014] The prototype-level data generation module uses the prototype-level data to generate support set data in the minority class data. For each category in the support set, based on the distance metric in the semantic space, it guides the minority class to combine with the related basic class to perform information migration, thereby estimating the type prototype and related distribution of the minority class, and sampling on the related distribution to obtain augmented sample data;

[0015] Finally, the enhanced support set data and augmented sample data are combined with the original support set as a new support set to train the image recognition model.

[0016] Preferably, instance-level data is used to generate support set data in the few-shot class data. For each training data in the support set data, the category activation mapping method is used to guide local semantic attention, as follows:

[0017] For a few-sample class data in the support set data (x, y), where x represents the image data and y represents the category corresponding to the image data, first calculate the semantic distance between the category of the support set data and the category of the base class:

[0018] d s (y, c) = || v y -v c || 2 , c∈C base

[0019] Among them, v y Represents the semantic features corresponding to the few-sample category y, v c Represents the semantic features corresponding to the basic class category c, c represents the category of the basic class data, C base Represents the basic class category set;

[0020] Select the k most similar basic categories:

[0021] S y =top k (c∈C base |-d s (y,c))

[0022] According to the k basic class categories, the category activation mapping method is used to activate the pre-trained feature extractor Φ, and the local spatial attention map on the k basic class dataset is obtained:

[0023]

[0024] Among them, CAM represents the category activation mapping method; Φ represents the feature extractor, which is pre-trained on the basic class data; S y represents the set formed by k basic categories, x represents image data, and c′ represents the set S y Category in;

[0025] Use the obtained local spatial attention map to enhance the original support set data:

[0026]

[0027] Where ⊙ represents the element-wise product, (·) t Represents the element-wise index and smoothes the attention map with parameter t.

[0028] Preferably, the generation of global semantic attention features by semantic embedding fusion is as follows:

[0029] By training on basic category data, a global attention fusion model of semantic features and visual features is obtained, so that the global attention fusion model can generate global attention and make the visual features close to the overall category prototype.

[0030] Furthermore, during training, for category y i ∈C base and the corresponding visual features f i , generate global semantic attention feature h i ∈R d , d represents the dimension of the feature;

[0031]

[0032] in, Represents category y i The semantic features corresponding to the semantic data; f i Represents image data x i Corresponding visual features; Represents the global attention fusion model, which is a fully connected layer; C base Represents basic class data; It is the feature splicing operation;

[0033] Then the global semantic attention feature h i The visual features f used to correct the original support set data i , to enhance the original support set data:

[0034]

[0035] Among them, the ReLU function keeps the features positive and aligned with the feature extractor; σ is the sine activation function; λ is the hyperparameter that controls the correction strength.

[0036] Furthermore, during training, the mean square error is used as the loss function, with the category y i The prototype features are used as optimization targets:

[0037]

[0038] Among them, n is the number of training data, y i is the category of the i-th image data, is the corresponding category y i Category prototype features.

[0039] Preferably, the prototype-level data is used to generate the support set data in the minority class data. For each category in the support set, the distance metric in the semantic space is used to guide the minority class to combine with the related basic class to perform information migration, thereby estimating the type prototype of the minority class. Specifically: given the minority class support set image data xi and the corresponding category y;

[0040] First, the image data x i The visual feature f is transformed into Turkey power step:

[0041] f i ′=f i τ

[0042] Where τ represents power;

[0043] Then calculate the data f i ′ and the visual distance between the category prototypes of each basic class:

[0044] d v (x i ,c)=||f i ′-μ c || 2 , c∈C base

[0045] Among them, μ c is the prototype feature of the corresponding basic class c, C base Represents the basic class category set;

[0046] At the same time, calculate the data f i ′ and the semantic distance d between each basic category s (y, c).

[0047] d s (y, c) = || v y -v c || 2 , c∈C base

[0048] Among them, v y Represents the semantic features corresponding to the few-sample category y, v c Indicates the semantic features corresponding to the basic class category c, c represents the category of the basic class, C base Represents the basic class category set;

[0049] Next, the most relevant p basic classes are selected by semantic distance, and then the closest q basic classes are selected from the p basic classes according to visual distance:

[0050] T y =top p (c∈C base |-d s (c, y))

[0051] B i =top q(c∈T y |-d v (x i , c))

[0052] Among them, p and q are hyperparameters used to control the range of semantic selection, and p>q; T y represents a category set consisting of p basic categories, B i Represents a category set consisting of q basic categories;

[0053] According to the category set B i , combined with the current few-sample class data f i 'Generate new few-sample data:

[0054]

[0055] Among them, μ c represents the category prototype feature of the basic category c; α represents the weight of controlling the category prototype feature of the basic category;

[0056] For the K training data of the support set of the few-sample class y, calculate the mean of its feature prototype:

[0057]

[0058] Furthermore, the relevant distribution of the minority sample class is estimated, and sampling is performed on the relevant distribution to obtain augmented samples, as follows:

[0059] Through the covariance matrix of the basic category, the covariance matrix of the few-sample class y is fused and calculated as the data distribution:

[0060]

[0061]

[0062] Among them, Σ c Represents the covariance matrix of the base class c, which is used to represent the Gaussian distribution of class c; 1 represents an all-one matrix, which is the same size as the covariance matrix; β is a hyperparameter that provides the basic variance in each dimension;

[0063] At this point, guided by the semantic and visual fusion metrics, the prototype-level correlation distribution for the few-sample class y is estimated, and Gaussian sampling is performed on the correlation distribution to generate feature-level augmented sample data.

[0064] Preferably, when testing and constructing a semantically guided augmented data generation model, repeatedly sample the few-shot class data to obtain multiple sets of independent few-shot tasks T = (S, Q); where, It is called the support set, which includes N different categories and K training data for each category; It is called the query set, which includes N different categories consistent with the support set S and M test data for each category that is disjoint from S;

[0065] Training is performed on the support set S of each few-shot task T and testing is performed on the query set Q to obtain the accuracy of each task. The mean of the accuracy of multiple task groups and the confidence interval with 95% confidence level are used as training indicators for the semantically guided augmented data generation model.

[0066] Furthermore, the few-shot class data is sampled according to the N-way K-shot setting to obtain multiple groups of independent few-shot tasks.

[0067] A computer system includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the few-sample image recognition method for generating semantically guided augmented data are implemented.

[0068] The beneficial effects of the present invention are as follows:

[0069] Compared with traditional approaches, this method fully considers the guidance of semantic information on visual information. It associates visual features with conceptual semantic information, generating reliable and rich augmented data at both the instance and prototype levels, thus avoiding the impact of fluctuations in visual information quality on model performance. This method achieves excellent performance on miniImageNet, tieredImageNet, and CUB. Furthermore, the method is highly versatile, with experiments demonstrating significant performance improvements when applied to feature extractors of varying architectures. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] Figure 1 This is a principle block diagram of the few-shot image recognition method for semantically guided augmented data generation according to the present invention. DETAILED DESCRIPTION

[0071] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0072] Example 1

[0073] Before introducing the method described in this embodiment, we first define various parameters that may be involved in the method described in this embodiment. For a given data set D = {(x i ,y i )}, where x i Represents image data, y i ∈C represents the corresponding image category, C is the category set; the dataset D is divided into two disjoint sets according to the category, namely D = D base ∪D novel, where D base Known as the base class data and D novel It is called the few-sample data, the basic data D base The corresponding image category set is C base , few-sample data D novel The corresponding image category set is C novel , respectively called the base class and the few-sample class, and both meet C=C base ∪C novel and principles;

[0074] The basic class data D base It is the basic class data containing rich training data, which is used as a part of prior knowledge for reference when training the small sample class data; the small sample class data D novel When used, sampling is performed according to the N-way K-shot setting, that is, for each independent few-sample task T = (S, Q), It is called the support set, which is the training data part of the few-shot task, containing N different categories and K training data for each category; It is called the query set, which is the test part of the few-shot task and contains N different categories consistent with S and M test data in each category that do not intersect with S.

[0075] The definitions of visual features, semantic features, and category prototypes involved in this embodiment are as follows:

[0076] The visual feature is the feature vector extracted by the feature extractor Φ, that is, for the image x i There is f i =Φ(x i ), where Φ is the base class data D base The pre-trained one is f i ∈R d , d represents the dimension of the feature, that is, the length of the feature vector.

[0077] The semantic features are the feature vectors extracted by the semantic model DistillBERT by combining the data category with its corresponding description sentence on WordNet, that is, for category c, its corresponding description sentence is S c , then the corresponding semantic feature is v c =Φ db (S c ), where Φ db is a pre-trained semantic model, v c ∈R m, m represents the dimension of the semantic feature, that is, the length of the semantic feature.

[0078] The category prototype is the center of each type of visual feature, that is, for category c there is a category prototype Among them, f j is the jth visual feature of class c, n c is the amount of data for this class; assuming that each class obeys a Gaussian distribution, then for class c, there is a covariance matrix Using the covariance matrix ∑ c Represents its Gaussian distribution.

[0079] This embodiment starts from the augmentation stage of image recognition model training data, and uses semantic information and prior information to generate rich, high-quality training data for small sample class data.

[0080] like Figure 1 As shown in FIG, a few-shot image recognition method with semantically guided augmented data generation is described, wherein the method is specifically as follows:

[0081] Construct a semantic-guided augmented data generation model, including instance-level data generation modules and prototype-level data generation modules;

[0082] The instance-level data generation module uses the instance-level data to generate support set data in the few-shot class data. For each training data in the support set data, the class activation mapping method is used to perform local semantic attention features, and semantic embedding fusion is used to generate global semantic attention features to generate enhanced support set data.

[0083] The prototype-level data generation module uses the prototype-level data to generate support set data in the minority class data. For each category in the support set, based on the distance metric in the semantic space, it guides the minority class to combine with the related basic class to perform information migration, thereby estimating the type prototype and related distribution of the minority class, and sampling on the related distribution to obtain augmented sample data;

[0084] Finally, the enhanced support set data and augmented sample data are combined with the original support set as a new support set to train the image recognition model.

[0085] In a specific embodiment, instance-level data is used to generate support set data in the few-shot class data. For each training data in the support set data, the category activation mapping method is used to guide local semantic attention, as follows:

[0086] For a few-sample class data in the support set data (x, y), where x represents the image data and y represents the category corresponding to the image data, first calculate the semantic distance between the category of the support set data and the category of the base class:

[0087] d s (y, c) = || v y -v c || 2 , c∈C base

[0088] Among them, v y Represents the semantic features corresponding to the few-sample category y, v c Represents the semantic features corresponding to the basic class category c, c represents the category of the basic class data, C base Represents the basic class category set;

[0089] Select the k most similar basic categories:

[0090] S y =top k (c∈C base |-d s (y,c))

[0091] According to the k basic class categories, the category activation mapping method is used to activate the pre-trained feature extractor Φ, and the local spatial attention map on the k basic class dataset is obtained:

[0092]

[0093] Among them, CAM represents the category activation mapping method; Φ represents the feature extractor, which is pre-trained on the basic class data; S y represents the set formed by k basic categories, x represents image data, and c′ represents the set S y Category in;

[0094] The feature extractor used in this embodiment is WideResNet28, with a visual feature dimension of 640 and a semantic feature dimension of 768.

[0095] Use the obtained local spatial attention map to enhance the original support set data:

[0096]

[0097] Where ⊙ represents the element-wise product, (·) t Represents the element-level index, and the attention map is smoothed with parameter t. In the specific operation, parameter t is set to 0.5 and k is set to 1.

[0098] Because semantically similar concepts often share similar visual features, this embodiment locally enhances support set samples by using spatial attention maps activated by base classifiers that are semantically highly correlated with the corresponding minority class. This uses the CAM technique, which generates a spatial attention map of the image classifier for each class on the corresponding image.

[0099] In a specific embodiment, the generation of global semantic attention features by semantic embedding fusion is as follows:

[0100] By training on basic category data, a global attention fusion model of semantic features and visual features is obtained, so that the global attention fusion model can generate global attention and make the visual features close to the overall category prototype.

[0101] In a specific embodiment, during training, for category y i ∈C base and the corresponding visual features f i , generate global semantic attention feature h i ∈R d , d represents the dimension of the feature;

[0102]

[0103] in, Represents category y i The semantic features corresponding to the semantic data; f i Represents image data x i Corresponding visual features; Represents the global attention fusion model, which is a fully connected layer; C base Represents basic class data; It is the feature splicing operation;

[0104] Then the global semantic attention feature h i The visual features f used to correct the original support set data i , to enhance the original support set data:

[0105]

[0106] Among them, the ReLU function keeps the features positive and aligned with the feature extractor; σ is the sine activation function; λ is the hyperparameter that controls the correction strength.

[0107] In a specific embodiment, during training, the mean square error is used as the loss function, and the category y i The prototype features are used as optimization targets:

[0108]

[0109] Among them, n is the number of training data, y i is the category of the i-th image data, is the corresponding category y i Category prototype features.

[0110] In this embodiment, 100,000 iterations were performed when training the global attention fusion model. The SGD optimizer was used, the batch data for each iteration was 5, the learning rate was 0.01, and the fusion parameter λ was set to 0.3.

[0111] In a specific embodiment, prototype-level data is used to generate a prototype for each category. According to the distance metric in the semantic space, the few-sample class is guided to combine with the related basic class to perform information migration, thereby estimating the type prototype of the few-sample class. Specifically: Given the image data x of the few-sample class support set i and the corresponding category y;

[0112] First, the image data x i The visual feature f is transformed into Turkey power step:

[0113] f i ′=f i τ

[0114] Wherein, τ represents power, τ = 0.5;

[0115] Then calculate the data f i ′ and the visual distance between the category prototypes of each basic class:

[0116] d v (x i ,c)=||f i ′-μ c || 2 , c∈C base

[0117] Among them, μ c is the prototype feature of the corresponding basic class c, C base Represents the basic class category set;

[0118] At the same time, calculate the data f i ′ and the semantic distance d between each basic category s (y, c).

[0119] d s (y, c) = || v y -v c || 2 , c∈C base

[0120] Among them, vy Represents the semantic features corresponding to the few-sample category y, v c Indicates the semantic features corresponding to the basic class category c, c represents the category of the basic class, C base Represents the basic class category set;

[0121] Next, the most relevant p basic classes are selected by semantic distance, and then the closest q basic classes are selected from the p basic classes according to visual distance:

[0122] T y =top p (c∈C base |-d s (c, y))

[0123] B i =top q (c∈T y |-d v (x i , c))

[0124] Among them, p and q are hyperparameters used to control the range of semantic selection, and p>q; T y represents a category set consisting of p basic categories, B i Represents a category set consisting of q basic categories;

[0125] According to the category set B i , combined with the current few-sample class data f i 'Generate new few-sample data:

[0126]

[0127] Among them, μ c represents the category prototype feature of the basic category c, and α represents the weight of controlling the category prototype feature of the basic category;

[0128] For the K training data of the support set of the few-sample class y, calculate the mean of its feature prototype:

[0129]

[0130] In a specific embodiment, the relevant distribution of the minority sample class is estimated, and sampling is performed on the relevant distribution to obtain augmented samples, as follows:

[0131] Through the covariance matrix of the basic category, the covariance matrix of the few-sample class y is fused and calculated as the data distribution:

[0132]

[0133]

[0134] Among them, ∑ c Represents the covariance matrix of the base class c, which is used to represent the Gaussian distribution of class c; 1 represents an all-one matrix, which is the same size as the covariance matrix; β is a hyperparameter that provides the basic variance in each dimension;

[0135] At this point, guided by the semantic and visual fusion metrics, the prototype-level correlation distribution for the few-sample class y is estimated, and Gaussian sampling is performed on the correlation distribution to generate feature-level augmented sample data.

[0136] In this embodiment, for prototype-level data, q=2, for task settings of K=1 and K=5, p is 10 and 32 respectively, α=0.6, β=0.2, and the number of Gaussian sampling data points for each category is 200.

[0137] In a specific embodiment, when testing the semantically guided augmented data generation model, the few-shot class data is repeatedly sampled to obtain multiple sets of independent few-shot tasks T = (S, Q); wherein, It is called the support set, which includes N different categories and K training data for each category; It is called the query set, which includes N different categories consistent with the support set S and M test data for each category that is disjoint from S.

[0138] Training is performed on the support set S of each few-shot task T and testing is performed on the query set Q to obtain the accuracy of each task. The mean of the accuracy of multiple task groups and the confidence interval with 95% confidence level are used as training indicators for the semantically guided augmented data generation model.

[0139] In a specific implementation, the few-shot class data is sampled according to an N-way K-shot setting to obtain multiple groups of independent few-shot tasks.

[0140] The semantically guided augmented data generation model described in this embodiment uses a local and global attention mechanism to enhance the original data through semantic guidance and embedding, thereby enhancing the original support set data and obtaining enhanced support set data.

[0141] A prototype-level data generation method that uses semantic and visual fusion metrics to transfer data distribution based on prior information.

[0142] Example 2

[0143] A computer system includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the few-sample image recognition method for generating semantically guided augmented data as described in Example 1 are implemented.

[0144] Example 3

[0145] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the few-shot image recognition method for generating semantically guided augmented data as described in Example 1.

[0146] Obviously, the above embodiments of the present invention are merely examples for the purpose of illustrating the present invention, and are not intended to limit the embodiments of the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention shall be included within the scope of protection of the claims of the present invention.

Claims

1. A few-shot image recognition method with semantically guided augmented data generation, characterized by: The method described is as follows: Construct a semantic-guided augmented data generation model, including instance-level data generation modules and prototype-level data generation modules; The instance-level data generation module uses the instance-level data to generate support set data in the few-shot class data. For each training data in the support set data, the class activation mapping method is used to perform local semantic attention features, and semantic embedding fusion is used to generate global semantic attention features to generate enhanced support set data. The prototype-level data generation module uses the prototype-level data to generate support set data in the minority class data. For each category in the support set, based on the distance metric in the semantic space, it guides the minority class to combine with the related basic class to perform information migration, thereby estimating the type prototype and related distribution of the minority class, and sampling on the related distribution to obtain augmented sample data; Finally, the enhanced support set data, augmented sample data and the original support set are combined as a new support set to train the image recognition model; During training, for the category and the corresponding visual features , generating global semantic attention features , Represents the dimension of the feature; in, Representation category Semantic features corresponding to semantic data; Represents image data x i Corresponding visual features; Represents the global attention fusion model, which is a fully connected layer; Represents basic class data; It is the feature splicing operation; Then the global semantic attention feature Visual features used to correct the original support set data , to enhance the original support set data: Among them, the ReLU function keeps the features positive and aligned with the feature extractor; is the sinusoidal activation function; is a hyperparameter that controls the strength of the correction; Use prototype-level data to generate support set data in the few-sample class data. For each category in the support set, guide the few-sample class to combine with the relevant basic class to perform information migration based on the distance metric in the semantic space, thereby estimating the type prototype of the few-sample class. Specifically: Given the few-sample class support set image data and the corresponding categories ; First, image data Visual features Perform Turkey power step conversion; Then calculate the data The visual distance from the category prototype of each base class; At the same time, the calculation data Semantic distance from each basic class category ; Next, the most relevant p basic classes are selected by semantic distance, and then the closest q basic classes are selected from the p basic classes according to visual distance; According to the category set , combined with the current few-sample class data Generate new few-sample data; For the few-sample class Support set training data, and calculate the mean of its feature prototype; Estimate the relevant distribution of the minority class and sample on the relevant distribution to obtain augmented samples, as follows: Through the covariance matrix of the basic category, the few-sample category is fused and calculated The covariance matrix of , as the data distribution; At this point, according to the guidance of semantic and visual fusion metrics, the estimated The prototype-level correlation distribution of is obtained, and Gaussian sampling is performed on the correlation distribution to generate feature-level augmented sample data.

2. The method for few-shot image recognition with semantically guided augmented data generation according to claim 1, characterized in that: Use instance-level data to generate support set data in the few-shot class data. For each training data in the support set data, use the category activation mapping method to guide local semantic attention, as follows: For a support set data in a few sample class data ,in, , first calculate the semantic distance between the categories of the support set data and the basic category: in, Represents the semantic features corresponding to the few-sample category y, Represents the semantic features corresponding to the basic class c, Represents the category of the basic class data, Represents the basic class category set; Select the closest Basic categories: according to Basic class categories, using the category activation mapping method to activate the pre-trained feature extractor , get in Local spatial attention map on the base class dataset: in, represents the category activation mapping method; Represents a feature extractor, which is pre-trained on the basic class data; express A collection of basic categories, represents image data, Representing a collection Category in; Use the obtained local spatial attention map to enhance the original support set data: in, represents the element-wise product, Represents element-level index, using parameters Smoothed attention map.

3. The method for few-shot image recognition with semantically guided augmented data generation according to claim 1, characterized in that: The above-mentioned method of generating global semantic attention features by using semantic embedding fusion is as follows: By training on basic category data, a global attention fusion model of semantic features and visual features is obtained, so that the global attention fusion model can generate global attention and make the visual features close to the overall category prototype.

4. The method for few-shot image recognition with semantically guided augmented data generation according to claim 3, characterized in that: During training, the mean square error is used as the loss function, and the category The prototype features are used as optimization targets: in, n is the number of training data, For the i categories of image data, For the corresponding category Category prototype features.

5. The method for few-shot image recognition with semantically guided augmented data generation according to claim 1, characterized in that: The method uses prototype-level data to generate support set data in the minority class data. For each category in the support set, according to the distance metric in the semantic space, the minority class is guided to combine with the relevant basic class to perform information migration, thereby estimating the type prototype of the minority class. Specifically: given the minority class support set image data and the corresponding categories ; First, image data Visual features Perform Turkey power step transformation: in, represents power; Then calculate the data The visual distance from the category prototype of each base class: in, For the corresponding basic class The prototype characteristics of the category, Represents the basic class category set; At the same time, the calculation data Semantic distance from each basic class category : in, Represents the semantic features corresponding to the few-sample category y, Represents the semantic features corresponding to the basic class c, Represents the category of the base class, Represents the basic class category set; Next, the most relevant p basic classes are selected by semantic distance, and then the closest q basic classes are selected from the p basic classes according to visual distance: Among them, p and q are hyperparameters used to control the range of semantic selection, and ; represents a category set consisting of p basic categories, Represents a category set consisting of q basic categories; According to the category set , combined with the current few-sample class data Generate new few-sample data: in, Indicates the base class category Category prototype characteristics; Represents the weight of the prototype feature that controls the basic category; For the few-sample class Support set training data, calculate the mean of its feature prototype: 。 6. The method for few-shot image recognition with semantically guided augmented data generation according to claim 5, characterized in that: The relevant distribution of the minority sample class is estimated, and sampling is performed on the relevant distribution to obtain augmented samples, as follows: Through the covariance matrix of the basic category, the few-sample category is fused and calculated The covariance matrix of , as the data distribution: in, Represents the covariance matrix of the base class category c, which is used to represent the Gaussian distribution of category c; represents an all-1 matrix, which is the same size as the covariance matrix; For hyperparameters, provide the basic variance in each dimension; At this point, according to the guidance of semantic and visual fusion metrics, the estimated The prototype-level correlation distribution of is obtained, and Gaussian sampling is performed on the correlation distribution to generate feature-level augmented sample data.

7. The method for few-shot image recognition with semantically guided augmented data generation according to claim 1, characterized in that: When testing and building a semantically guided augmented data generation model, repeatedly sample the few-shot class data to obtain multiple independent few-shot tasks. ;in, It is called the support set, which includes N different categories and K training data for each category; It is called the query set, which includes N different categories consistent with the support set S and M test data of each category that is disjoint from S; Training is performed on the support set S of each few-shot task T and testing is performed on the query set Q to obtain the accuracy of each task. The mean of the accuracy of multiple task groups and the confidence interval with a 95% confidence level are used as training indicators for the semantically guided augmented data generation model.

8. The method for few-shot image recognition with semantically guided augmented data generation according to claim 7, characterized in that: The few-shot class data is sampled according to the N-way K-shot setting to obtain multiple groups of independent few-shot tasks.

9. A computer system comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the few-shot image recognition method for generating semantically guided augmented data are implemented as claimed in any one of claims 1 to 8.

Images