A small sample learning method and device based on sample pair relation propagation

Abstract

The application discloses a kind of small sample learning method and device based on sample pair relation propagation.The method can obtain better relation coding by explicitly modeling and propagating the relation between the sample pairs of support set-query set.Through the introduction of pseudo-relation node, the characteristic information of the query set sample itself can be effectively retained. In addition, the application further provides an effective transduction learning strategy, which can better mine the relationship information between the query set samples, thereby obtaining more accurate classification results. Compared with the prior art, the present application better mines the potential information contained in the sample pairs of support set-query set in each task, and has higher accuracy and better generalization ability when processing a completely new task.

Description

Technical Field

[0001] This invention relates to a few-shot learning method, and more particularly to a few-shot learning method based on the propagation of sample pair relations between support set and query set. It also relates to a corresponding few-shot learning device, and belongs to the field of machine learning technology. Background Technology

[0002] Data is a crucial resource in machine learning. But how do we train models when data is scarce? Few-shot learning is one effective solution. It involves studying how to extract effective concepts from one or a few limited samples, enabling the model to quickly adapt to new, unseen categories, under conditions of sparse data (where each category may have only one or a few samples).

[0003] In recent years, various few-shot learning methods have been proposed. These methods can be broadly categorized into optimization-based, generative, and metric-based few-shot learning methods. Optimization-based few-shot learning attempts to capture the relationships between different tasks to balance the model's prior knowledge with the information of the current task. However, because this method requires iterating over samples from all categories in the task before updating parameters, it cannot learn effective feature encodings. Generative few-shot learning methods employ data augmentation or directly generate parameters for unseen categories to train the meta-learner. Metric-based few-shot learning maps support set and query set sample features to a shared encoding space, then calculates the similarity between samples to complete the classification task.

[0004] However, existing few-shot learning methods generally fail to model the relationship between support-query sample pairs, neglecting this potential information resource and still facing problems such as low classification accuracy and poor generalization ability when facing new tasks. In Chinese invention patent ZL201510801292.3, Zhejiang University disclosed a few-shot set (target domain) classification algorithm based on improved MFA and transfer learning. First, a homogeneous dataset (source domain) with a large number of labeled samples and the marginal Fisher criterion (inner product distance) are used to map the features of the source and target domain samples to a low-dimensional space. Then, singular points at the source domain boundary are removed, and sample points from dense regions of the source domain are selected and combined with a small number of labeled sample points in the target domain to form transfer learning training sample pairs. Next, a nonlinear feature transformation matrix from the target domain to the source domain is learned in the kernel space. Finally, a classifier is trained using a large number of labeled samples from the source domain, and the target domain samples are mapped back to the source domain through the nonlinear feature transformation matrix, and then input into the trained classifier to obtain the object classification results on the few-shot set. Summary of the Invention

[0005] The primary technical problem to be solved by this invention is to provide a few-shot learning method based on sample-pair relation propagation.

[0006] Another technical problem to be solved by the present invention is to provide a few-shot learning device based on sample pair relation propagation.

[0007] To achieve the above objectives, the present invention adopts the following technical solution:

[0008] According to a first aspect of the present invention, a few-shot learning method based on sample-pair relation propagation is provided, comprising the following steps:

[0009] S1, Randomly shuffle the dataset;

[0010] S2, load the pre-trained parameters of the multi-layer convolutional neural network and initialize other parameters;

[0011] S3, randomly sample a task from the dataset;

[0012] S4, use a feature extraction network to extract the global features of each sample in the task;

[0013] S5. Construct a set of relation nodes based on the obtained global features, generate a relation matrix using weight matrix loss, and initialize the graph structure.

[0014] S6. The graph convolution operation is used to update the relation nodes to obtain the final relation code, and the prediction score is calculated based on the relation code and the cross-entropy classification loss.

[0015] S7, return to step S3, iterate until step S7 until the overall loss function converges, save the parameters of each layer of the neural network, and complete the training process.

[0016] Preferably, in step S3, a task τ = S∪Q is randomly sampled from the dataset; wherein, The support set for the task contains N different categories, and each category contains K samples; Where K and N are both positive integers.

[0017] Preferably, in step S4, a multi-layer convolutional neural network is used to extract features from each sample in the sampled task to obtain the three-dimensional local features of each sample. Then, the local features are converted into one-dimensional global features through a fully connected layer or global pooling.

[0018] Preferably, step S5 includes the following sub-steps:

[0019] S51, construct a set of relation nodes V = {V} based on the obtained global feature set. i};

[0020] S52, introduce pseudo-relation node V into the set of relation nodes. q ;

[0021] S53, the weight matrix A of the graph structure is calculated based on the dominant node in the relation nodes and the weight matrix loss.

[0022] Preferably, the relation node V i There are two construction strategies: non-transduction learning settings and transduction learning settings.

[0023] In the non-transduction learning setting, the relation node V i for:

[0024] V i =[g i g q ]

[0025] Under the transduction learning setting, the relation node V i for:

[0026]

[0027] Where g i i = 1, 2, ..., T S G represents the global features of the support set samples. q and i = 1, 2, ..., T Q This represents the global vector of the query set samples, and [·] indicates the concatenation operation.

[0028] Preferably, the pseudo-relation node V q It includes two construction strategies: non-transduction learning settings and transduction learning settings.

[0029] In the non-transduction learning setting, the pseudo-relation node V q for:

[0030] V q =[g q g q ]

[0031] Under the transduction learning setting, the pseudo-relation node V q for:

[0032]

[0033] Among them, g q and i = 1, 2, ..., T QThis represents the global vector of the query set samples, and [·] indicates the concatenation operation.

[0034] Preferably, the weight matrix A incorporates two construction strategies: non-transductive learning setting and transductive learning setting.

[0035] Under the non-transductive learning setting, the weight matrix A is:

[0036]

[0037] Under the transduction learning setting, the weight matrix A is:

[0038]

[0039] Among them, l ij The formula for calculating the collaborative labels of the support set samples is as follows:

[0040]

[0041] in, This is a function for calculating the adjacency matrix.

[0042] Preferably, step S6 includes the following sub-steps:

[0043] S61, a highly discriminative relational encoding is obtained through graph convolution operations;

[0044] S62 calculates the prediction score based on relational encoding and cross-entropy classification loss.

[0045] Preferably, step S7 includes the following sub-steps:

[0046] Calculate the weight matrix loss for weight matrix A, and score the prediction result S. q Calculate the cross-entropy classification loss; weight the weight matrix loss and the cross-entropy classification loss, and construct the overall loss function. Perform iterative training until the overall loss function is reached. Convergence; the parameters of each layer of the neural network that performs best on the validation set are saved, completing the training process;

[0047] Wherein, the overall loss function for:

[0048]

[0049] In the above formula, Represents the cross-entropy classification loss. Let represent the weight matrix loss, and λ represent the parameter used to scale the weight matrix loss.

[0050] According to a second aspect of the present invention, a few-shot learning apparatus based on sample-pair relation propagation is provided, comprising a processor and a memory, wherein the processor reads a computer program in the memory for performing the following operations:

[0051] S1, Randomly shuffle the dataset;

[0052] S2, load the pre-trained parameters of the multi-layer convolutional neural network and initialize other parameters;

[0053] S3, randomly sample a task from the dataset;

[0054] S4, use a feature extraction network to extract the global features of each sample in the task;

[0055] S5. Construct a set of relation nodes based on the obtained global features, generate a relation matrix using weight matrix loss, and initialize the graph structure.

[0056] S6. The graph convolution operation is used to update the relation nodes to obtain the final relation code, and the prediction score is calculated based on the relation code and the cross-entropy classification loss.

[0057] S7, return to step S3, iterate until step S7 until the overall loss function converges, save the parameters of each layer of the neural network, and complete the training process.

[0058] This invention achieves more discriminative relation encoding by explicitly modeling and propagating the relationships between support set and query set sample pairs. Introducing pseudo-relation nodes effectively preserves the feature information of the query set samples themselves. Furthermore, this invention provides an effective transduction learning strategy that can better mine the relationship information between query set samples, resulting in more accurate classification results. Compared with existing technologies, this invention better mines the potential information contained in the support set and query set sample pairs in each task, exhibiting higher accuracy and better generalization ability when handling novel tasks. Attached Figure Description

[0059] Figure 1 Flowchart of the few-shot learning method based on sample-pair relation propagation provided by this invention;

[0060] Figure 2 This is a schematic diagram of the few-shot learning device based on sample-pair relation propagation provided by the present invention. Detailed Implementation

[0061] The technical content of the present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0062] Currently, deep neural networks have important applications in various fields such as image recognition, speech recognition, and natural language processing. However, deep neural network models typically have millions of parameters and require supervised training with a large amount of labeled data to achieve relatively good results. In practice, it is often difficult to provide deep neural network models with sufficient labeled data.

[0063] To address this, this invention provides a few-shot learning method based on sample-pair relation propagation. The method aims to fully exploit the latent information contained in the support-query-set sample pairs in each task. By modeling and propagating the relationships between these pairs, it obtains more discriminative relation encodings and introduces pseudo-relationship nodes to preserve the feature information of the query-set samples themselves. Furthermore, this method provides an effective transduction learning strategy that can better mine the relational information between query-set samples, thereby obtaining more accurate classification results. The following describes the method in conjunction with... Figure 1 Please provide a detailed explanation.

[0064] like Figure 1 As shown, the few-shot learning method provided in this embodiment of the invention mainly includes the following steps:

[0065] S1, Randomly shuffle the dataset X;

[0066] S2, load the pre-trained parameters of the multi-layer convolutional neural network and initialize other parameters;

[0067] S3, randomly sample a task τ from dataset X;

[0068] S4, use a feature extraction network to extract the global features g of each sample in the task. i ;

[0069] S5, based on the obtained global feature g i Construct a set of relation nodes V, generate a relation matrix A using weight matrix loss, and initialize the graph structure;

[0070] S6. Graph convolution operations are used to update the relation nodes to obtain the final relation encoding. The prediction score S is calculated based on relation encoding and cross-entropy classification loss. q ;

[0071] S7, return to step S3, iterate until step S7 until the overall loss function converges, save the parameters of each layer of the neural network, and complete the training process.

[0072] In one embodiment of the present invention, step S3 specifically includes the following sub-steps:

[0073] A task τ = S∪Q is randomly sampled from the dataset. The support set for the task contains N different categories, and each category contains K samples. Where K and N are both positive integers. For example, in dataset X, tasks are sampled and trained using two experimental settings: 5-way 1-shot and 5-way 5-shot. During testing, one sample is sampled for each category in the task as a query sample, and a total of 10,000 tasks are sampled for testing.

[0074] In one embodiment of the present invention, step S4 specifically includes the following sub-steps:

[0075] Multi-layer convolutional neural networks, such as WideResNet or ResNet, are used to extract features from each sample in the sampled task (the corresponding multi-layer convolutional neural network is called a feature extraction network), obtaining the three-dimensional local features of each sample. Then, the local features are transformed into one-dimensional global features g through fully connected layers or global pooling. i This yields a new set of global features.

[0076] It should be noted that WideResNet and ResNet in the above embodiments are both typical multi-layer convolutional neural networks. WideResNet was published by Sergey Zagoruyko in 2016, aiming to improve ResNet by increasing network width, thereby enhancing performance and training speed. In one embodiment of the present invention, the WideResNet used during training is a WRN-28-10 architecture, and its parameters are obtained by pre-training the cross-entropy classification loss on the entire training set, and its parameters remain unchanged during the training of the few-shot learning model.

[0077] In this invention, relational nodes are constructed by directly concatenating the global features of the support set and query set samples, and the relationship between query set samples is propagated by concatenating the global features of all query set samples at once under the transduction learning setting.

[0078] In one embodiment of the present invention, step S5 specifically includes the following sub-steps:

[0079] S51, construct a set of relation nodes V = {V} based on the obtained global feature set. i};

[0080] S52, introduce pseudo-relation node V into the set of relation nodes. q ;

[0081] S53, the weight matrix A of the graph structure is calculated based on the dominant node in the relation nodes and the weight matrix loss.

[0082] In a preferred embodiment of the present invention, the relation node V in step S51 i Specifically, this includes two construction strategies: non-transduction learning settings and transduction learning settings.

[0083] In the non-transfer learning setting, relation node V i Specifically:

[0084] V i =[g i g q ]

[0085] Under the transduction learning setting, relation node V i Specifically:

[0086]

[0087] Where g i i = 1, 2, ..., T S G represents the global features of the support set samples. q and i = 1, 2, ..., T Q This represents the global vector of the query set samples. [·] indicates a concatenation operation.

[0088] In a preferred embodiment of the present invention, the pseudo-relation node V in step S52 q Specifically, it includes two construction strategies: non-transduction learning settings and transduction learning settings.

[0089] In the non-transduction learning setting, the pseudo-relation node V q Specifically:

[0090] V q =[g q g q ]

[0091] Under the transduction learning setting, pseudo-relation node V q Specifically:

[0092]

[0093] Among them, g q and i = 1, 2, ..., T Q This represents the global vector of the query set samples. [·] indicates a concatenation operation.

[0094] In one embodiment of the present invention, it is assumed that during training, the dimension of the global vector of each sample is [1, 640]. Under the non-transfer learning setting, the dimension of each relation node and pseudo-relation node is [1, 1440]. Under the transfer learning setting, the dimension is [1, 640*(T)]. Q +1)], where T Q +1 refers to the number of samples in the query set.

[0095] In a preferred embodiment of the present invention, the weight matrix A in step S53 specifically includes two construction strategies: non-transductive learning setting and transductive learning setting.

[0096] Under the non-transductive learning setting, the weight matrix A is specifically as follows:

[0097]

[0098] Under the transduction learning setting, the weight matrix A is specifically as follows:

[0099]

[0100] Among them, l ij The specific formula for calculating the collaborative labels of the support set samples is as follows:

[0101]

[0102] in, This is a function for calculating the adjacency matrix. In practice, various distance metrics are applicable, such as the cosine function, Euclidean distance, and neural networks. During training, A three-layer fully connected layer is preferred.

[0103] Accordingly, the weight matrix loss in S53 is as follows:

[0104]

[0105] In one embodiment of the present invention, step S6 specifically includes the following sub-steps:

[0106] S61, a highly discriminative relational encoding is obtained through graph convolution operations;

[0107] S62 calculates the prediction score based on relational encoding and cross-entropy classification loss.

[0108] In a preferred embodiment of the present invention, the graph convolution in step S61 specifically involves:

[0109]

[0110] in, It is constructed by adding a self-loop matrix to the weight matrix. The matrix is ​​the degree matrix of the weight matrix, and the specific construction method is as follows: W represents the learnable parameters, and σ represents the activation function. This is a highly discriminative relation encoding obtained after graph convolution.

[0111] In a preferred embodiment of the present invention, the cross-entropy classification loss in step S62 is specifically as follows:

[0112]

[0113] in, This represents the prediction score of the query set sample belonging to each category of the support set, as predicted by the few-shot learning method provided in this embodiment of the invention.

[0114] In one embodiment of the present invention, step S7 specifically includes the following sub-steps:

[0115] Calculate the weight matrix loss for weight matrix A, and score the prediction result S. q Calculate the cross-entropy classification loss; weight the weight matrix loss and the cross-entropy classification loss, and construct the overall loss function. Perform iterative training until the overall loss function is reached. Convergence; the parameters of each layer of the neural network that performs best on the validation set are saved, completing the training process.

[0116] Among them, the overall loss function Specifically:

[0117]

[0118] In the above formula, Represents the cross-entropy classification loss. Let represent the weight matrix loss, and λ represent the parameter used to scale the weight matrix loss.

[0119] To implement the few-shot learning method based on sample-pair relation propagation provided by this invention, this invention also provides a few-shot learning device based on sample-pair relation propagation. For example... Figure 2As shown, the device includes a memory 21 and a processor 22, and may further include communication components, sensor components, power supply components, multimedia components, and input / output interfaces as needed. The memory, communication components, sensor components, power supply components, multimedia components, and input / output interfaces are all connected to the processor 22. The memory 21 can be a static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, etc. The processor can be a central processing unit (CPU), graphics processing unit (GPU), field-programmable gate array (FPGA), application-specific integrated circuit (ASIC), digital signal processing (DSP) chip, etc. Other communication components, sensor components, power supply components, multimedia components, etc., can all be implemented using common components found in existing smartphones, and will not be specifically described here.

[0120] On the other hand, in the aforementioned few-shot learning device based on sample-pair relation propagation, the processor 22 reads the computer program in the memory 21 to perform the following operations:

[0121] S1, Randomly shuffle the dataset X;

[0122] S2, load the pre-trained parameters of the multi-layer convolutional neural network and initialize other parameters;

[0123] S3, randomly sample a task τ from dataset X;

[0124] S4, use a feature extraction network to extract the global features g of each sample in the task. i ;

[0125] S5, based on the obtained global feature g i Construct a set of relation nodes V, generate a relation matrix A using weight matrix loss, and initialize the graph structure;

[0126] S6. Graph convolution operations are used to update the relation nodes to obtain the final relation encoding. The prediction score S is calculated based on relation encoding and cross-entropy classification loss. q ;

[0127] S7, return to step S3, iterate until step S7 until the overall loss function converges, save the parameters of each layer of the neural network, and complete the training process.

[0128] The few-shot learning method and apparatus provided by this invention employs random sampling to obtain training and testing tasks during the training process. The corresponding neural network architecture can consist of WideResNet, a graph convolutional neural network, and two three-layer fully connected layers. Specifically, a multi-layer convolutional neural network is used to extract global features of the samples, a graph convolutional neural network is used to propagate the relationships between sample pairs, and two three-layer fully connected layers are used as the weight calculation function and the final score calculation function, respectively. The weight matrix loss is calculated on the weight matrix obtained from the first fully connected layer, and the cross-entropy classification loss is calculated on the score obtained from the second fully connected layer. The graph convolutional neural network contains only one layer of graph convolution.

[0129] In one embodiment of the present invention, steps S3 to S7 are executed iteratively. The selected activation function σ is ReLU. The calculation function for the prediction result score is... This also uses three fully connected layers. During training, the Adam optimizer can be used, with an initial learning rate of 0.001 and λ of 1. The input image size is set to 84×84, and 20 images are trained per batch. Gradient decay is incorporated during training, with a decay rate of 0.000001. Decay occurs every 15,000 generations on the miniImageNet dataset and every 30,000 generations on the tieredImageNet dataset. Training is iterated until the overall loss function of the deep neural network converges. The parameters of each layer of the best-performing neural network on the validation set are saved, completing the training of the deep neural network.

[0130] The few-shot learning method and apparatus provided by this invention executes the above steps S1 to S7 sequentially during the testing process to obtain the prediction result of the category to which each image in the query set belongs.

[0131] This invention achieves more discriminative relation encoding by explicitly modeling and propagating the relationships between support set and query set sample pairs. Introducing pseudo-relation nodes effectively preserves the feature information of the query set samples themselves. Furthermore, this invention provides an effective transduction learning strategy that can better mine the relationship information between query set samples, resulting in more accurate classification results. Compared with existing technologies, this invention better mines the potential information contained in the support set and query set sample pairs in each task, exhibiting higher accuracy and better generalization ability when handling novel tasks.

[0132] The foregoing has provided a detailed description of the few-shot learning method and apparatus based on sample-pair relation propagation provided by this invention. Any obvious modifications made by those skilled in the art without departing from the essence of this invention will constitute an infringement of the patent rights of this invention and will incur corresponding legal liability.

Claims

1. A few-shot learning method based on sample-pair relation propagation for training a deep neural network for image recognition, characterized in that... Includes the following steps: S1, Randomly shuffle the dataset; S2, load the pre-trained parameters of the multi-layer convolutional neural network and initialize other parameters; S3, randomly sample a task from the dataset; S4, use a feature extraction network to extract the global features of each sample in the task; Relationship nodes are constructed by directly concatenating the global features of the support set and query set samples, and the propagation of relationships between query set samples is accomplished by concatenating the global features of all query set samples at once under the transduction learning setting. S5, construct a set of relation nodes based on the obtained global feature set, generate a relation matrix using weight matrix loss, and initialize the graph structure; introduce pseudo-relation nodes into the set of relation nodes, and calculate the weight matrix of the graph structure based on the dominant node in the relation nodes and the weight matrix loss; wherein, the pseudo-relation nodes It includes two construction strategies: non-transduction learning settings and transduction learning settings. In the non-transduction learning setting, the pseudo-relation node for: Under the transduction learning setting, the pseudo-relation node for: in, and , , This represents the global vector of the query set samples. Indicates a splicing operation; S6. The graph convolution operation is used to update the relation nodes to obtain the final relation code, and the prediction score is calculated based on the relation code and the cross-entropy classification loss. S7, return to step S3, iterate until step S7 until the overall loss function converges, save the parameters of each layer of the neural network, and complete the training process; S8. During the test, steps S1 to S7 are executed sequentially to obtain the prediction results of the category to which each image in the query set belongs.

2. The few-shot learning method as described in claim 1, characterized in that: In step S3, a task is randomly sampled from the dataset. ;in, The support set for the task contains N different categories, and each category contains K samples; , where K and N are both positive integers.

3. The few-shot learning method as described in claim 1, characterized in that: In step S4, a multi-layer convolutional neural network is used to extract features from each sample in the sampled task to obtain the three-dimensional local features of each sample. Then, the local features are converted into one-dimensional global features through a fully connected layer or global pooling.

4. The few-shot learning method as described in claim 1, characterized in that: The relation node There are two construction strategies: non-transduction learning settings and transduction learning settings. In the non-transduction learning setting, the relation nodes for: Under the transduction learning setting, the relationship nodes for: in , This represents the global features of the support set samples. and , , This represents the global vector of the query set samples. This indicates a splicing operation.

5. The few-shot learning method as described in claim 1, characterized in that: The weight matrix A It includes two construction strategies: non-transduction learning settings and transduction learning settings. In the non-transduction learning setting, the weight matrix A for: Under the transduction learning setting, the weight matrix A for: in, The formula for calculating the collaborative labels of the support set samples is as follows: in, This is a function for calculating the adjacency matrix.

6. The few-shot learning method as described in claim 1, characterized in that... Step S6 includes the following sub-steps: S61, a highly discriminative relational encoding is obtained through graph convolution operations; S62 calculates the prediction score based on relational encoding and cross-entropy classification loss.

7. The few-shot learning method as described in claim 1, characterized in that... Step S7 includes the following sub-steps: For the weight matrix Calculate the weight matrix loss and score the prediction results. Calculate the cross-entropy classification loss; weight the weight matrix loss and the cross-entropy classification loss, and construct the overall loss function. Iterative training is performed until the overall loss function is reached. Convergence; the parameters of each layer of the neural network that performs best on the validation set are saved, completing the training process; Wherein, the overall loss function for: In the above formula, Represents the cross-entropy classification loss. Represents the loss of the weight matrix. This represents the parameter used to scale the loss of the weight matrix.

8. A few-shot learning device based on sample-pair relation propagation, characterized in that... It includes a processor and a memory, wherein the processor reads a computer program from the memory for executing the few-shot learning method according to any one of claims 1 to 7.

Images