Dual-branch multi-scale branch tutoring learning method based on relational network metric

Abstract

The present invention discloses a dual-branch multi-scale branch tutoring learning method based on relational network measurement, and relates to the field of small sample image recognition and classification. The present invention uses a relational network to measure the similarity of sample pairs on the basis of extracting multi-level and multi-scale features, and uses dual-branch tutoring learning to improve the recognition ability of the model, wherein a multi-scale feature generator is used to generate deep multi-scale features in branch one, and features of different levels are extracted in branch two. Then, both branches use a learnable relational network to measure the similarity of features at each scale or each level. In order to play a leading role in the final classification, a tutoring learning module is designed, and branch one is used to tutor branch two for learning, and the deep features are used to finally integrate the relationship scores at all scales and levels to obtain the classification results, which can improve the image recognition and classification accuracy in the case of small samples as a whole.

Description

Technical Field

[0001] The present invention relates to the field of small sample image recognition and classification, and in particular to a dual-branch multi-scale branch tutoring learning method based on relational network measurement. Background Art

[0002] In recent years, deep learning has developed rapidly due to the support of massive data, especially in the field of computer vision. However, the training of deep learning models often relies on a large amount of labeled data. When there is a small amount of data and scarce samples, the model is prone to overfitting and its generalization ability will be relatively poor. In practical problems, there are indeed situations where sample data is small or scarce, such as image data of rare species, medical images, etc. In contrast to deep learning models, humans have powerful learning and cognitive abilities, which enable humans to establish cognition of new things by learning only a small amount of data or even concepts. Therefore, inspired by human cognitive abilities, how to enable deep learning models to establish effective cognition and generalize to new categories through training with a small number of samples is one of the problems with great research value in the field of deep learning, known as the small sample learning problem.

[0003] Small sample learning refers to training a model with limited sample data, so that the model can generalize to new things and establish good cognition, simulating the process of humans associating new things with prior knowledge and cognizing them. For small sample classification problems, a classification model is trained with limited data, and then it is asked to classify new categories that have not been encountered during training, so the model is required to have good generalization ability. In order to solve this kind of problem, many excellent methods have been proposed.

[0004] At present, Zhang et al. use a Kronecker product module to capture the spatial correlation between pairs of image features and measure them using a relational network. Qin et al. divide the original image into salient patches and combine them with an attention mechanism to enhance the model's target recognition ability. Wang et al. improve the classification accuracy in small sample cases by weighted fusion of multi-level image features.

[0005] However, during the specific implementation process, the applicant found that there are other solutions to the problems of insufficient utilization of data in small sample cases and low recognition and classification accuracy of image recognition models. Therefore, it is necessary to propose a new solution to the above problems. Summary of the invention

[0006] The object of the present invention is to provide a dual-branch multi-scale branch tutoring learning method based on relational network measurement.

[0007] To achieve the above object, the present invention provides the following technical solution: a dual-branch multi-scale branch tutoring learning method based on relational network measurement, comprising at least the following steps:

[0008] Step 1: construct a multi-level and multi-scale feature extraction network;

[0009] Step 2, constructing a similarity measurement network using relational network measurement and a dual-branch tutoring learning module;

[0010] Step 3, input the image into the dual-branch multi-scale tutored learning metric network for training, and save the optimal model;

[0011] Step 4: Use the optimal model to perform classification prediction and obtain the classification accuracy.

[0012] Preferably, the step 1 at least comprises the following steps:

[0013] After the standard 4-layer convolutional network, a multi-scale feature generator is input to generate deep multi-scale image features;

[0014] That is, we get (W1×H1×C), (W2×H2×C), (W3×H3×C), (W4×H4×C), (W5×H5×C), where C=64 means 64 channels and a total of 5 scale feature maps;

[0015] Through the standard 4-layer convolutional network and the fifth branch of the multi-scale feature generator, feature maps of different depth levels are obtained;

[0016] That is, we obtain feature maps of three different depth levels: (W1×H1×C), (W2×H2×C), and (W3×H3×C).

[0017] Preferably, the formulas of the feature maps of the five scales are:

[0018]

[0019] where F θ represents the feature extractor, G φ represents a multi-scale generator, z represents different scales, x s and x q represent support and query samples respectively.

[0020] Preferably, the formulas of the feature maps of the three different depth levels are:

[0021]

[0022] Where d represents different depth levels.

[0023] Preferably, the dual-branch tutoring learning module in step 2 includes at least branch one and branch two;

[0024] The application of branch one comprises at least the following steps:

[0025] Output the 5-scale feature maps of the multi-scale feature generator, and form a query-support feature pair based on the query and support samples at each scale;

[0026] That is, the features of the two samples are concatenated in the channel dimension and then input into the relational network, which includes a Kronecker product module to capture the spatial correlation mapping between feature pairs;

[0027] The original feature map is used to generate a pair of feature space correlation maps, which are then concatenated with the original image to learn the similarity of the samples and measure the relationship between the sample pairs, obtaining the relationship (classification) score at each scale (a total of 5 scales). Where z∈{1,…,5}, q represents the query sample, s represents the support sample, and the relationship score is expressed as:

[0028]

[0029] Among them, z represents different scales, and concate means concatenating features in the channel dimension;

[0030] Therefore, for a C-way K-shot classification task, the total loss of branch 1 is:

[0031]

[0032] Among them l CE represents the cross entropy loss function, n = C × K represents the number of support images in each episode, and m = C × M represents the number of query images;

[0033] Indicates the support image x at the zth scale si With the query image x qi The relationship score of , 1(·) is equal to 1 when the condition is met, and equal to 0 when the condition is not met.

[0034] Preferably, the application of the second branch includes at least the following steps:

[0035] Image features at different depths are extracted, and at each depth, query-support feature pairs are formed according to query and support samples;

[0036] That is, the features of the two samples are concatenated in the channel dimension and then input into the relational network. At the same time, the low-level relational information is passed to the deeper relational network and concatenated with the deeper feature pairs to learn the similarity of the samples and measure the relationship between the samples, and the classification results at different depths (3 depths in total) are obtained. Where d∈{1,…,3}, q represents the query sample, s represents the support sample, and the relationship score is expressed as:

[0037]

[0038] d∈{1, 2, 3}, i∈{1,…,n}, j∈{1,…m}

[0039] Where d represents different levels, Concate means concatenating features in the channel dimension. represents the relationship information passed from the previous layer, and the first layer does not have this item, so the total loss of branch two is:

[0040] Preferably, in step 2, the relationship score result of the relationship network measurement of branch one is used to construct a soft label of similarity knowledge, and the branch two measurement network learning similarity is coached in the coaching learning module T, and the learning summary r of branch one is used Z And the final learning summary of branch 2 D , the constructed soft label loss is expressed as:

[0041] L T = l KL (softmax(r z / t), softmax(r D / t))

[0042] Among them l KL represents the KL divergence loss, softmmax(·) represents normalization, so the total loss function of the model is expressed as:

[0043]

[0044] Where α is the weight parameter, taking a = 0.3, and the final classification result is determined by the sum of the relationship scores of all scales and levels, specifically expressed as: r final =r Z +r D / 3.

[0045] Compared with the prior art, the present invention has the following beneficial effects:

[0046] The present invention uses a relational network to measure the similarity of sample pairs on the basis of extracting multi-level and multi-scale features, and uses dual-branch tutoring learning to improve the recognition ability of the model, wherein deep multi-scale features are preferentially generated by a multi-scale feature generator in branch one, and features of different levels are extracted in branch two. Then, both branches use a learnable relational network to measure the similarity of features at each scale or each level. In order to give full play to the dominant role of deep features in the final classification, a tutoring learning module is designed, and branch one is used to tutor branch two for learning. Finally, the relationship scores at all scales and levels are integrated to obtain the classification result, which can improve the image recognition classification accuracy in the case of small samples as a whole. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for describing the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0048] Figure 1 is a flow chart of the present invention;

[0049] Figure 2 It is a structural diagram of the multi-scale generator module of the present invention;

[0050] Figure 3 This is a relationship network structure diagram of branch 2 of the present invention;

[0051] Figure 4 It is a structural schematic diagram of the dual-branch multi-scale branch tutoring learning model based on relational network measurement of the present invention. DETAILED DESCRIPTION

[0052] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0053] Embodiment 1:

[0054] This embodiment is used to disclose a dual-branch multi-scale branch tutored learning method based on relational network metric. By designing a multi-level multi-scale feature extraction network, that is, using a standard 4-layer convolutional neural network (CNN) and a multi-scale feature generator, limited samples are fully utilized without using additional sample data and information, thereby reducing the cost of obtaining additional labeled information and improving the accuracy of recognition and classification.

[0055] The dual-branch multi-scale branch tutoring learning method based on relational network measurement at least includes the following steps:

[0056] Step 1: construct a multi-level and multi-scale feature extraction network;

[0057] Step 2, constructing a similarity measurement network using relational network measurement and a dual-branch tutoring learning module;

[0058] Step 3, input the image into the dual-branch multi-scale tutored learning metric network for training, and save the optimal model;

[0059] Step 4: Use the optimal model to perform classification prediction and obtain the classification accuracy.

[0060] Step 1 includes at least the following steps:

[0061] After the standard 4-layer convolutional network, a multi-scale feature generator is input to generate deep multi-scale image features;

[0062] That is, we get (W1×H1×C), (W2×H2×C), (W3×H3×C), (W4×H4×C), (W5×H5×C), where C=64 means 64 channels and a total of 5 scale feature maps;

[0063] Through the standard 4-layer convolutional network and the fifth branch of the multi-scale feature generator, feature maps of different depth levels are obtained;

[0064] That is, we obtain feature maps of three different depth levels: (W1×H1×C), (W2×H2×C), and (W3×H3×C).

[0065] Formula for the feature map of 5 scales:

[0066]

[0067] where F θ represents the feature extractor, G φ represents a multi-scale generator, z represents different scales, x s and x q represent support and query samples respectively.

[0068] Formulas for feature maps at 3 different depth levels:

[0069]

[0070] Where d represents different depth levels.

[0071] Through the multi-scale feature generator, image sample features of multiple scales are generated, which increases the number of high-level feature representations of the samples and improves the accuracy of the model; by obtaining the feature information of the image at different levels of the convolutional network, image information of different granularities (such as lines, local shapes, global semantic information, etc.) is extracted, thereby improving the model's ability to measure the similarity of samples in all aspects.

[0072] In step 2, the dual-branch tutoring learning module includes at least branch 1 and branch 2, where deep multi-scale image features are used for branch 1 and image features of different depth levels are used for branch 2. The feature maps of the dual branches are spliced ​​on the channel in the form of query-support sample pairs, and then sent to their respective relationship network measurement modules for similarity measurement. The relationship networks of the two branches are slightly different. Specifically, the relationship network of branch 1 uses the Kronecker product module to obtain the spatial correlation mapping of deep multi-scale image features, while the relationship network of branch 2 passes shallow relationship information to a deeper level to fuse deep features for similarity measurement.

[0073] By constructing a unique dual-branch relational network measurement module, the classification accuracy of the model can be improved based on the original data alone without obtaining additional feature information in small sample situations with limited data.

[0074] The application of branch 1 includes at least the following steps:

[0075] Output the 5-scale feature maps of the multi-scale feature generator, and form query-support feature pairs based on the query and support samples at each scale;

[0076] That is, the features of the two samples are concatenated in the channel dimension and then input into the relational network, which includes a Kronecker product module to capture the spatial correlation mapping between feature pairs;

[0077] The original feature map is used to generate a pair of feature space correlation maps, which are then concatenated with the original image to learn the similarity of the samples and measure the relationship between the sample pairs, obtaining the relationship (classification) score at each scale (a total of 5 scales). Where z∈{1,…,5}, q represents the query sample, s represents the support sample, and the relationship score is expressed as:

[0078]

[0079] Among them, z represents different scales, and concate means concatenating features in the channel dimension;

[0080] Therefore, for a C-way K-shot classification task, the total loss of branch 1 is:

[0081]

[0082] Among them l CE represents the cross entropy loss function, n = C × K represents the number of support images in each episode, and m = C × M represents the number of query images;

[0083] Indicates the support image x at the zth scale si With the query image x qi The relationship score of , 1(·) is equal to 1 when the condition is met, and equal to 0 when the condition is not met.

[0084] The application of branch 2 includes at least the following steps:

[0085] Image features at different depths will be extracted, and at each depth, query-support feature pairs will be formed according to the query and support samples;

[0086] That is, the features of the two samples are concatenated in the channel dimension and then input into the relational network. At the same time, the low-level relational information is passed to the deeper relational network and concatenated with the deeper feature pairs to learn the similarity of the samples and measure the relationship between the samples, and the classification results at different depths (3 depths in total) are obtained. Where d∈{1,…,3}, q represents the query sample, s represents the support sample, and the relationship score is expressed as:

[0087]

[0088] d∈{1, 2, 3},i∈{1,…,n},j∈{1,…m}

[0089] Where d represents different levels, Concate means concatenating features in the channel dimension. represents the relationship information passed from the previous layer, and the first layer does not have this item, so the total loss of branch two is:

[0090] In step 2, the relationship score results of the relationship network measurement of branch one are used to construct soft labels of similarity knowledge, and the branch two measurement network learning similarity is coached in the coaching learning module T, and the learning summary r of branch one is used Z And the final learning summary of branch 2 D , the constructed soft label loss is expressed as:

[0091] L T = l KL (softmax(r z / t), softmax(r D / t))

[0092] Among them l KL represents the KL divergence loss, and softmax(·) represents normalization, so the total loss function of the model is expressed as:

[0093]

[0094] Where α is the weight parameter, α=0.3, and the final classification result is determined by the sum of the relationship scores of all scales and levels, which is specifically expressed as: final =r Z +r D / 3.

[0095] Embodiment 2:

[0096] This embodiment is used to further explain in detail the above embodiments:

[0097] Step 1: Build a multi-level, multi-scale feature extraction network. Considering the actual situation of the small sample problem (small amount of data), a multi-scale feature generator is connected to the standard 4-layer convolutional neural network (the specific structure is as follows Figure 2 As shown in the figure, we extract different levels of deep multi-scale image features, corresponding to the two branches respectively. In Table 1, we designed a set of comparative experiments on the miniImagenet dataset to test the performance of the dual-branch multi-scale model in classification effect. The experimental results show that the feature extraction model using dual branches improves the effect of using a single branch by 0.37% and 1.3% respectively in the 5-way 1-shot setting. After combining the branch tutoring learning module, the dual-branch multi-scale model is 1.34% better than the original model. In the 5-way 5-shot setting, the effect is improved by 1.34% and 4.13% respectively compared with using a single branch alone. After combining the branch tutoring learning module, the dual-branch multi-scale model is 0.43% better than the original model.

[0098] Table 1 Ablation experiment results of different branches on the miniImagenet dataset

[0099]

[0100] First, we combine the idea of ​​multi-level and multi-scale, use a standard 4-layer convolutional neural network module, and then introduce a multi-scale feature generator for deep multi-scale semantic features. Finally, we form a dual-branch model structure to process different feature pair information respectively. The complete dual-branch structure is as follows: Figure 4 shown.

[0101] The multi-scale feature generator first performs multi-scale transformation on the deep feature map (here is branch one) to obtain features at different scales, and then uses the Kronecker product module in the shared weight relational network module of branch one to perform spatial correlation mapping. The two original features and the two spatial correlation maps are then concatenated on the channel to obtain the input features of the relational network metric module.

[0102] For feature graphs at different depth levels (here is branch 2), the query-support sample feature pair concatenation is also performed, and then input into the relationship network measurement module of branch 2. It should be noted that the relationship network in branch 2 is independent (3 relationship sub-networks), and the shallow relationship network will pass the measured feature fusion information to the deeper network (the first layer does not have this item). The complete relationship network module of branch 2 is as follows Figure 3 shown.

[0103] In Table 2, we designed a set of ablation experiments to analyze the impact of information transfer between layers. The baseline model represents a dual-branch structure with a tutoring learning module. Among them, the shallowest layer of branch two is obtained from the output of the first layer of the feature extraction module, and * indicates that the second layer of the feature extraction module is selected to constitute the shallowest layer of branch two features, and the maxpooling layer of the second layer is delayed. Compared with our baseline model, when the second layer features are selected, the classification effect of our model is improved by 0.94% and 0.44% in the 5-way 1-shot and 5-shot scenarios respectively. In addition, it can be found from Table 2 that in the case of 5-way 1-shot, the knowledge transfer between the internal layers of the relationship network module of branch two will also help improve the accuracy (0.41%). However, in the case of 5-way 5-shot, the addition of knowledge transfer between layers has a negative impact in this scenario. Analysis shows that this is because in this scenario setting, the prototype learned by the model is relatively robust, and the higher-level features with semantic information will have a greater impact on the final classification. Therefore, the information transfer from the shallow layer will affect the final decision of the model.

[0104] Table 2. Ablation experimental results of the position of the group adaptation module in the model scaling network with residual

[0105]

[0106] Step 2, considering that the branch coaching module uses the relationship network of branch one to coach the relationship network of branch two to learn the similarity between image pairs, specifically, the knowledge learned by branch one is used to make soft labels, the learned knowledge is passed to branch two, and the hyperparameter t is introduced to measure its softening degree. The experimental results are shown in Table 3. In Table 3, we designed a set of comparative experiments to analyze the impact of the softening hyperparameter t on the classification effect of the model. When t=1, it means no softening. The larger t is, the greater the softening degree is, which makes the difference between categories smaller. From the results, it can be found that in the case of 5-way 1-shot, the classification effect is best when t=20, but in the case of 5-shot, the classification effect is best when t=7.

[0107] Table 3 Experiment on the impact of the softening degree hyperparameter t on the model classification effect

[0108]

[0109] Step 3: Use the training set images for training, perform random flipping, cropping, and erasing data enhancement on the original image samples, and input them into the dual-branch multi-scale tutor learning metric network for training. The model structure is as follows: Figure 4 As shown. Check the classification effect of the model and save the model weight.

[0110] In Table 4, we designed comparative experiments on two standard small sample datasets to compare the effectiveness of our dual-branch multi-scale branch tutoring learning method based on relational network metrics compared with other advanced models. The experimental results show that our model has better recognition and classification effects than advanced methods.

[0111] Table 4 Comparative experimental results on three small sample standard datasets

[0112]

[0113]

[0114] Step 4: Use the optimal model to perform classification prediction and obtain the classification accuracy.

Claims

1. A dual-branch multi-scale branch tutoring learning method based on relational network metrics, The method comprises the following steps: Step 1: construct a multi-level and multi-scale feature extraction network; Step 2, constructing a similarity measurement network using relational network measurement and a dual-branch tutoring learning module; The dual-branch tutoring learning module includes at least branch one and branch two; The first branch outputs the feature graphs of the five scales of the multi-scale feature generator, and forms a query-support feature pair based on the query and support samples at each scale; The second branch is to extract image features at different depths, and at each depth, form a query-support feature pair according to the query and support samples; Use the relationship score results of the relationship network measurement of branch one to construct the soft label of similarity knowledge, and provide guidance for the branch two measurement network learning similarity in the guidance learning module T. Use the learning summary r of branch one Z And the final learning summary of branch 2 D , construct soft label loss; Step 3, input the image into the dual-branch multi-scale tutored learning metric network for training, and save the optimal model; Step 4: Use the optimal model to perform classification prediction and obtain the classification accuracy.

2. The dual-branch multi-scale branch tutoring learning method based on relational network metric according to claim 1 is characterized by: The step 1 at least comprises the following steps: After the standard 4-layer convolutional network, a multi-scale feature generator is input to generate deep multi-scale image features; That is, we get (W1×H1×C), (W2×H2×C), (W3×H3×C), (W4×H4×C), (W5×H5×C), where C=64 means 64 channels and a total of 5 scale feature maps; Through the standard 4-layer convolutional network and the fifth branch of the multi-scale feature generator, feature maps of different depth levels are obtained; That is, we obtain feature maps of three different depth levels: (W1×H1×C), (W2×H2×C), and (W3×H3×C).

3. The dual-branch multi-scale branch tutoring learning method based on relational network metric according to claim 2 is characterized in that: The formula of the feature map of the five scales is: where F θ represents the feature extractor, G φ represents a multi-scale generator, z represents different scales, x s and x q denote support and query samples respectively.

4. The dual-branch multi-scale branch tutoring learning method based on relational network metric according to claim 3 is characterized by: The formulas for the feature maps of the three different depth levels are: Where d represents different depth levels.

5. The dual-branch multi-scale branch tutoring learning method based on relational network metric according to claim 4 is characterized in that: The feature maps of the five scales of the multi-scale feature generator are outputted, and at each scale, query-support feature pairs are formed according to the query and support samples, that is, the features of the two samples are concatenated in the channel dimension, and then input into the relational network, which includes a Kronecker product module for capturing the spatial correlation mapping between feature pairs; The original feature map is used to generate a pair of feature space correlation maps, which are then concatenated with the original image to learn the similarity of the samples and measure the relationship between the sample pairs to obtain the relationship score at each scale. Where z∈{1,…,5}, q represents the query sample, s represents the support sample, and the relationship score is expressed as: Among them, z represents different scales, and concate means concatenating features in the channel dimension. represents the spatial correlation mapping between the query and the support image obtained by the Kronecker product; Therefore, for a C-way K-shot classification task, the total loss of branch 1 is: Among them l CE represents the cross entropy loss function, n = C × K represents the number of support images in each episode, and m = C × M represents the number of query images; Indicates the support image x at the zth scale si With the query image x qi The relationship score of , 1(·) is equal to 1 when the condition is met, and equal to 0 when the condition is not met.

6. The dual-branch multi-scale branch tutoring learning method based on relational network metrics according to claim 5 is characterized by: The image features of different depths are extracted. At each depth, query-support feature pairs are formed according to the query and support samples, that is, the features of the two samples are spliced ​​in the channel dimension and then input into the relationship network. At the same time, the low-level relationship information is passed to the deeper relationship network and spliced ​​with the deeper feature pairs to learn the similarity of the samples and measure the relationship between the samples, so as to obtain the classification results at different depths. Where d∈{1,…,3}, q represents the query sample, s represents the support sample, and the relationship score is expressed as: Where d represents different levels, Concate means concatenating features in the channel dimension. represents the relationship information passed from the previous layer, and the first layer does not have this item, so the total loss of branch two is:

7. The dual-branch multi-scale branch tutoring learning method based on relational network metrics according to claim 6 is characterized in that: The constructed soft label loss is expressed as: L T =l KL (softmax(r Z / t),softmax(r D / t)) Among them l KL represents the KL divergence loss, and softmax(·) represents normalization, so the total loss function of the model is expressed as: Where α is the weight parameter, α=0.3, and the final classification result is determined by the sum of the relationship scores of all scales and levels, which is specifically expressed as: final =r Z +r D / 3.

Images