Cross-domain few-shot image recognition system based on target-guided dynamic mixup

Abstract

The application belongs to the technical field of computers, and particularly relates to a cross-domain small sample image recognition system based on target guidance dynamic Mixup. The application comprises a dynamic Mixup mixing ratio generation module, a Mixup-3T image classification module and a double-layer meta-learning training module. The application generates a target-guided Mixup mixing ratio through the dynamic Mixup mixing ratio generation module, and then inputs the Mixup mixing ratio and an image to be classified into the Mixup-3T image classification module for classification. The dynamic Mixup mixing ratio generation module and the Mixup-3T image classification module are trained through the double-layer meta-learning training module. The application can generate an optimal Mixup mixing ratio for a target domain and a current model, and improve the accuracy of cross-domain small sample image classification by using the mixing ratio.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of computer, and particularly relates to a cross-domain small sample image recognition system. BACKGROUND

[0002] Humans can learn the features of objects from very few samples to identify and classify the objects. Small sample image recognition helps deep learning systems to more effectively use training samples, thereby greatly reducing the number of training samples and the cost of human annotation. Cross-domain small sample image recognition is a special case in small sample image recognition. This method studies the case where training samples and test samples come from different distributions (domains) in the field of small sample image recognition. Since the deployment environment of the system is often very different from the training environment, the research on the cross-domain small sample image recognition system has more practical significance and broad research value.

[0003] Although this task has important research significance, there are still few directly related research works. Literatures [1, 2, 3] study to improve the generalization of features to improve the transferability of the model, but the effect of this kind of algorithm is not ideal due to the lack of target domain images. Literatures [4, 5] enhance knowledge transfer by fine-tuning on the test set, but the fine-tuning operation increases the time cost.

[0004] In order to improve the accuracy and efficiency of the algorithm, we advocate collecting a very small number of target domain samples into the training set during actual training, and this part of data is called extra target domain dataset [6].

[0005] The present application introduces the Mixup method [7] into the cross-domain small sample image recognition system. Unlike the existing Mixup method, the present application proposes a dynamic Mixup proportion generation module and a double-layer meta-learning generation module that can dynamically generate the mixing proportion of Mixup for different target domain datasets and the current model state. At the same time, the present application proposes a three-task image classification network Mixup-3T containing source domain classification tasks, target domain classification tasks and mixed image classification tasks to fully utilize the pictures generated by the dynamic Mixup algorithm for knowledge transfer. SUMMARY

[0006] The present application aims to provide a cross-domain small sample image recognition system based on target guided dynamic Mixup, which can improve the accuracy and efficiency of the algorithm.

[0007] The application provides a cross-domain small sample image recognition system based on target guidance dynamic Mixup, which claims to train on a source domain dataset and a small amount of images from a target domain (an additional target domain dataset), and test on a target domain dataset to improve the accuracy and efficiency of the cross-domain small sample image recognition system; specifically comprising the following three modules: (1) a Mixup-3T image classification module; (2) a dynamic Mixup mixing ratio generation module; and (3) a double-layer meta-learning training module; the dynamic Mixup mixing ratio generation module is used to generate a target-guided Mixup mixing ratio, and then the Mixup mixing ratio and the image to be classified are input into the Mixup-3T image classification module for classification; the double-layer meta-learning training module is used to train the dynamic Mixup mixing ratio generation module and the Mixup-3T image classification module.

[0008] The dynamic Mixup mixing ratio generation module is composed of three fully connected networks; the target-guided Mixup mixing ratio is generated by the dynamic Mixup mixing ratio generation module.

[0009] The Mixup-3T image classification module comprises a feature extraction network and a classifier; the Mixup mixing ratio obtained by the dynamic Mixup mixing ratio generation module and the image to be classified are classified.

[0010] The double-layer meta-learning training module is used to train the Mixup-3T image classification module and the dynamic Mixup mixing ratio generation module.

[0011] In the application, the Mixup-3T image classification module receives different inputs and obtains corresponding outputs during the training, verification and testing processes, and the details are as follows:

[0012] During the training, the input is an image from the source domain, an image from the additional target domain, a label corresponding to the input image, and a Mixup mixing ratio, and the output is a three-task classification loss The specific steps are as follows:

[0013] (1) According to the input Mixup mixing ratio, the Mixup [1] The input source domain and target domain images are mixed to generate mixed images and mixed image labels.

[0014] (2) The feature extraction network is used to extract the feature representation of the source domain image, the additional target domain image and the mixed image, respectively; the feature extraction network can be any feature extraction network, for example, ResNet10 in document [9];

[0015] (3) input the feature representation of the extracted source domain image, the additional target domain image and the mixed image into the classifier respectively to obtain classification probability of the image on each category, and according to the label corresponding to the image, calculate the classification loss of the source domain image, the additional target domain image and the mixed image by using a Cross-Entorpy algorithm, and add the classification loss according to a fixed proportion to obtain a three-task classification loss L FSL The classifier can be any classifier, for example, GNN in [3].

[0016] During verification, the input is an additional target domain image and a label corresponding to the input image, and the output is a classification verification loss on the additional target domain data, and the specific steps are as follows:

[0017] (1) the same feature extraction network described above is used to extract the feature representation of the additional target domain image;

[0018] (2) input the extracted additional target domain image into the classifier to obtain classification probability of the image on each category, and calculate the classification loss of the three kinds of images by using a Cross-Entorpy algorithm according to the label corresponding to the image.

[0019] During testing, the input is a target domain image for testing, and the output is a predicted label corresponding to the input image, and the specific steps are as follows:

[0020] (1) the same feature extraction network described above is used to extract the feature representation of the additional target domain image;

[0021] (2) input the extracted additional target domain image into the classifier to obtain classification probability of the image on each category, and the category with the maximum classification probability is taken as the predicted label of the image.

[0022] In the application, the dynamic Mixup mixing ratio generation module adopts a three-layer fully connected network, the input of which is the classification verification loss of the Mixup-3T network on the additional target domain data, and the output is the Mixup mixing ratio.

[0023] In the application, the double-layer meta-learning training module inputs a source domain data set and an additional target domain data set, and outputs trained Mixup-3T network parameters, and the specific steps are as follows:

[0024] (1) initialize the network parameters of the Mixup-3T image classification module and the dynamic Mixup mixing ratio generation module;

[0025] (2) randomly sample an N*(k+q) picture data set from the additional target domain data set Where N is the total number of categories, k is the training samples for each category, and q is the number of test samples;

[0026] (3) Enter f m3t Perform verification and calculate verification loss;

[0027] (4) Input the validation loss into the dynamic Mixup mixing ratio generation module g G To obtain the Mixup mixing ratio

[0028] (5) Randomly sample a set E containing N*(k+q) images from the source dataset. S A dataset E containing N*(k+q) images is randomly sampled from the additional target domain dataset and the additional target domain dataset. T A dataset E containing N*(k+q) images is randomly sampled from the additional target domain dataset and the additional target domain dataset. T_val ;

[0029] (6) Copy Mixup-3T to obtain a copy model ′f of Mixup-3T. m3t ;

[0030] (7) E S E T and Mixup mixing ratio Input f′ m3t During training, the three-task classification loss is obtained.

[0031] (8)Use Perform backpropagation to update f′ m3t Parameters;

[0032] (9) E T_val Input f′ m3t The verification loss is calculated and the parameters of the dynamic Mixup mixing ratio generation module are updated by backpropagation through the verification loss.

[0033] (10) Input the updated dynamic Mixup ratio generation module to obtain the Mixup ratio;

[0034] (11) E S E T Mixup mixing ratio input f m3t ,get use Backpropagation update f m3t ;

[0035] (12) E T_val Enter fm3t performing verification calculation verification loss;

[0036] (13) repeating steps (5)-(12) until the number of repetitions reaches the maximum number of repetition rounds.

[0037] The innovation of the present application is:

[0038] (1) The present application introduces the Mixup method into the cross-domain small sample learning field. Unlike existing Mixup methods, the method proposed in the present application can dynamically generate effective Mixup mixing ratios for different target domain data sets and current model states to help network learning.

[0039] (2) The present application proposes a three-task image classification network Mixup-3T containing source domain classification tasks, target domain classification tasks, and mixed image classification tasks, which can fully utilize the pictures generated by the dynamic Mixup algorithm for knowledge transfer from the source domain to the target domain.

[0040] Therefore, the present application can greatly improve the accuracy and efficiency of the cross-domain small sample image recognition system. BRIEF DESCRIPTION OF DRAWINGS

[0041] Figure 1 is a training flowchart schematic diagram of the present application.

[0042] Figure 2 is a flowchart of the Mixup-3T image classification module during training. DETAILED DESCRIPTION

[0043] The present application will be further described below in conjunction with the drawings and examples.

[0044] Figure 1 is a training flowchart schematic diagram of the present application. The present application adopts a double-layer meta-learning training framework. In the first layer, data subsets E S , E S , and E T_val containing N*(k+q) pictures are sampled from the source domain and the target domain and the additional target domain set, respectively. T_val The E G is input into the dynamic Mixup mixing ratio generation network g TG to output the Mixup mixing ratio λ S . Then E S , E TG , and λ TG are input into Mixup-3T for pseudo-back propagation; pseudo-back propagation means first copying Mixup-3T, updating the gradient on the copied network, calculating the verification loss of the copied network on the additional target domain data set, and finally updating the dynamic Mixup mixing ratio generation network gG . In the second stage, use the updated g G to regenerate λ TG . Input E S , E S and λ TG into Mixup-3T for backpropagation. "BP" is the abbreviation of "Backward Propagation", and the character with a caret (^) indicates that the character is generated in the first layer.

[0045] Figure 2 is the flow chart of the Mixup-3T image classification module in the training process of the present application. The module is composed of feature extraction and GNN classifier. In the training process, E S , E S and λ TG are input, and L S , L T and L mix are output.

[0046] The specific working steps of the whole system are as follows:

[0047] Step 1. Initialize the Mixup-3T image classification module f m3t , and the network parameters of the dynamic Mixup mixing ratio generation module;

[0048] Step 2. Randomly sample a data set containing N*(k+q) pictures from the additional target domain data set where N is the total number of categories, k is the training sample of each category, and q is the number of test samples. N, k, q are selected according to the actual data, and the N categories are rewritten as [0, 1, …, N-1];

[0049] Step 3. Input and the corresponding image label into f m3t for verification, and calculate the verification loss where f m3t is composed of a cascaded ResNet10 network and a GNN classification network. The specific process of step 3 is as follows:

[0050] (1) Use the ResNet10[9] feature extraction network to extract the features of

[0051] (2) Use GNN

[10] to calculate the classification probability of N categories based on the extracted features

[0052] (3) Use the Cross-Entropy algorithm to calculate​​ The classification loss is as follows:

[0053]

[0054] Step 4. The verification loss is input into the dynamic Mixup mixing ratio generation module g G to obtain the Mixup mixing ratio where g G is composed of three fully connected layers, and the specific mathematical formula is as follows:

[0055]

[0056] where t is the cycle number minus 1, for example, the mixing t is 0 when the first cycle.

[0057] Step 5. Randomly sample a set E S containing N*(k+q) pictures from the source data set, randomly sample a set E T containing N*(k+q) pictures from the additional target domain data set, and randomly sample a set E T_val containing N*(k+q) pictures from the additional target domain data set. Rewrite the N class labels in E T , E S and E T_val to [0, 1, …, N-1].

[0058] Step 6. Copy Mixup-3T to obtain a copy model f′ m3t of Mixup-3T.

[0059] Step 7. Input E S , E T and Mixup mixing ratio into f′ m3t for training to obtain three-task classification loss The specific process of step 7 is as follows:

[0060] (1) Combine E S , E T and the corresponding labels with the Mixup mixing ratio to obtain the mixed data set E mix and its corresponding label y mix . When mixing, first select the corresponding class pair according to the class name, for example, mix class 0 in E S with class 0 in E T ; mix class 1 in E S with class 1 in E T . The specific mathematical formula for mixing is as follows:

[0061]

[0062]

[0063] When the first cycle, the above formula can be taken as 1;

[0064] (2) using ResNet10 to extract features from E S , E T and E mix respectively;

[0065] (3) using GNN to calculate the classification probability of the extracted features for N categories and

[0066] (4) using Cross-Entropy algorithm to calculate the classification loss of and and and add them up in the ratio of 0.25, 0.25, 0.5 to get The specific mathematical formula is as follows:

[0067]

[0068]

[0069]

[0070]

[0071] Step 8. Use to update the parameters of f′ m3t

[0072] Step 9. Input E T_val to f′ m3t to calculate the validation loss. The specific process of step 9 is as follows:

[0073] (1) using ResNet10 feature extraction network to extract features from E T_val

[0074] (2) using GNN to calculate the classification probability of the extracted features for N categories.

[0075] (3) using Cross-Entropy algorithm to calculate the classification loss, the specific mathematical formula is as follows:

[0076] ​​​

[0077] Step 10. Update the parameters ω of the dynamic Mixup mixing ratio generation module by backpropagation t+1 .

[0078] Step 11. Input the validation loss and the updated dynamic Mixup mixing ratio generation module to obtain the Mixup mixing ratio, with the specific formula as follows:

[0079]

[0080] Step 12. Input E S , E T , and the Mixup mixing ratio into f m3t to obtain The specific process of Step 12 is as follows:

[0081] (1) Combine the Mixup mixing ratio E S and E T , and the corresponding labels to obtain the mixed dataset E mix and its corresponding labels y mix . When mixing, first select the corresponding class pair according to the class name, for example, mix class 0 in E S with class 0 in E T ; mix class 1 in E S with class 1 in E T . The specific mathematical formula for mixing is as follows:

[0082]

[0083]

[0084] When the first loop is executed, the in the above formula can be taken as 1;

[0085] (2) Use ResNet10 to extract features from E S , E T , and E mix respectively;

[0086] (3) Use GNN to calculate the classification probability of N classes for the extracted features respectively;

[0087] (4) Use the Cross-Entropy algorithm to calculate the classification loss and and add them together with a ratio of 0.25, 0.25, and 0.5 to obtain The specific mathematical formula is as follows:

[0088]

[0089]

[0090]

[0091]

[0092] Step 13. Use backpropagation to update f m3t .

[0093] Step 14. Input E T_val into f m3t to perform validation calculation and validation loss, the specific process of step 14 is:

[0094] (1) Use ResNet10 feature extraction network to extract features from E T_val ;

[0095] (2) Use GNN to calculate the classification probability of N classes for the extracted features

[0096] (3) Use Cross-Entropy algorithm to calculate the classification loss of , the specific mathematical formula is as follows:

[0097]

[0098] Step 15. Repeat steps 4-14 until the number of repetitions reaches 40,000, and store the parameters of the Mixup-3T model at this time for image classification.

[0099] Step 16. Use Mixup-3T for image classification;

[0100] The specific process of step 16 is:

[0101] (1) Randomly sample a data set E test containing N*(k+q) images from the test data set;

[0102] (2) Use ResNet10 feature extraction network to extract features from E test ;

[0103] (3) Use GNN to calculate the classification probability of N classes for the extracted features;

[0104] (4) The class with the highest classification probability is taken as the predicted class of the network.

[0105] Method evaluation. The accuracy of the invention is used as the evaluation index of the method. Specifically, the number of correctly classified samples is divided by the total number of test samples.

[0106] References

[0107] [1] Haoqing Wang and Zhi-Hong Deng. 2021. Cross-domain few-shot classification via adversarial task augmentation. arXiv preprint arXiv:2104.14385 (2021).

[0108] [2] Yuqian Fu, Yu Xie, Yanwei Fu, Jingjing Chen, and Yu-Gang Jiang. 2022. Wave-SAN: Wavelet based Style Augmentation Network for Cross-Domain Few-Shot Learning. arXiv preprint arXiv:2203.07656 (2022).

[0109] [3] Hung-Yu Tseng, Hsin-Ying Lee, Jia-Bin Huang, and Ming-Hsuan Yang. 2020. Cross-domain few-shot classification via learned feature-wise transformation. arXiv preprint arXiv:2001.08735 (2020).

[0110] [4] Hanwen Liang, Qiong Zhang, Peng Dai, and Juwei Lu. 2021. Boosting the Generalization Capability in Cross-Domain Few-shot Learning via Noise-enhanced Supervised Autoencoder. In Proceedings of the IEEE / CVF International Conference on Computer Vision. 9424-9434.

[0111] [5] Rajshekhar Das, Yu-Xiong Wang, and José MF Moura. 2021. On the importance of distractors for few-shot classification. In Proceedings of the IEEE / CVF International Conference on Computer Vision. 9030-9040.

[0112] [6] Yuqian Fu, Yanwei Fu, and Yu-Gang Jiang. 2021. Meta-FDMixup: Cross-Domain Few-Shot Learning Guided by Labeled Target Data. In Proceedings of the 29 th ACM International Conference on Multimedia. 5326-5334.

[0113] [7] Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. 2017. mixup: Beyond empirical risk minimization. arXiv preprint arXiv:1710.09412 (2017)

[0114] [8] The Mathematics of Information Coding, Extraction and Distribution, by George Cybenko, Dianne P. O'Leary, Jorma Rissanen, 1999, p. 82

[0115] [9] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. 770-778.

[0116]

[10] Victor Garcia and Joan Bruna. 2017. Few-shot learning with graph neural networks. arXiv preprint arXiv:1711.04043 (2017).

Claims

1. A cross-domain few-shot image recognition system based on target-guided dynamic Mixup, characterized in that, The source domain dataset is trained on a small number of images from the target domain, and the target domain dataset is tested to improve the accuracy and efficiency of the cross-domain small sample image recognition system, and the system comprises the following three modules: (1) a Mixup-3T image classification module; (2) a dynamic Mixup mixing ratio generation module; and (3) a double-layer meta-learning training module. The dynamic Mixup mixing ratio generation module comprises three fully connected networks, and generates a target-guided Mixup mixing ratio through the dynamic Mixup mixing ratio generation module. The Mixup-3T image classification module comprises a feature extraction network and a classifier, and classifies the Mixup mixing ratio obtained by the dynamic Mixup mixing ratio generation module and the image to be classified. The double-layer meta-learning training module is used for training the Mixup-3T image classification module and the dynamic Mixup mixing ratio generation module. During the training, verification and testing processes, the Mixup-3T image classification module receives different inputs and obtains corresponding outputs, and the specific steps are as follows: In training, the input is an image from the source domain, an image from the additional target domain, the label corresponding to the input image, and the Mixup mixing ratio, and the output is a three-task classification loss The specific steps are as follows: (1) according to the input Mixup mixing ratio, the source domain and target domain images are mixed according to Mixup to generate mixed images and mixed image labels; (2) the feature extraction network is used to extract the feature representation of the source domain image, the additional target domain image and the mixed image; (3) input the feature representation of the extracted source domain image, the additional target domain image and the mixed image into the classifier respectively, obtain the classification probability of the image on each category, according to the label corresponding to the image, calculate the classification loss of the source domain image, the additional target domain image and the mixed image by using the Cross-Entorpy algorithm, and add the classification loss according to a fixed proportion to obtain the three-task classification loss L FSL ; During verification, the input is an additional target domain image and a label corresponding to the input image, and the output is a classification verification loss on the additional target domain data, and the specific steps are as follows: (1) the same feature extraction network is used to extract the feature representation of the additional target domain image; (2) the extracted additional target domain image is input into the classifier to obtain the classification probability of the image in each class, and the Cross-Entorpy algorithm is used to calculate the classification loss of the three types of images according to the corresponding label of the image; During testing, the input is a target domain image for testing, and the output is a predicted label corresponding to the input image, and the specific steps are as follows: (1) the same feature extraction network is used to extract the feature representation of the additional target domain image; (2) the extracted additional target domain image is input into the classifier to obtain the classification probability of the image in each class, and the classification probability of the largest class is taken as the predicted label of the image.

2. The cross-domain few-shot image recognition system of claim 1, wherein, The dynamic Mixup mixing ratio generation module adopts a three-layer fully connected network, and the input of the three-layer fully connected network is the classification verification loss of the Mixup-3T network on the additional target domain data, and the output is the Mixup mixing ratio.

3. The cross-domain few-shot image recognition system of claim 2, wherein, The double-layer meta-learning training module inputs the source domain dataset and the additional target domain dataset, and outputs the trained Mixup-3T network parameters, and the specific steps are as follows: (1) the network parameters of the Mixup-3T image classification module and the dynamic Mixup mixing ratio generation module are initialized; (2) Randomly sample one dataset containing N*(k+q) picture data from the additional target domain dataset where N is the total number of categories, k is the training sample of each category, and q is the number of test samples; (3) to input f m3t verification is performed, and a verification loss is calculated; (4) input the verification loss into the dynamic Mixup mixing ratio generation module g G , to obtain the Mixup mixing ratio (5) Randomly sample a set E of N * (k + q) images from the source dataset S from the additional target domain dataset and randomly sample a set E of N * (k + q) images from the additional target domain dataset T from the additional target domain dataset and randomly sample a set E of N * (k + q) images from the additional target domain dataset T_val ; (6) Duplicate Mixup-3T, get a copy model f' of Mixup-3T m3t ; (7) E S , E T and Mixup mixing ratio input f′ m3t training, resulting in three-task classification loss (8)Use Perform backpropagation to update f′ m3t Parameters; (9) will be E T_val Input f' m3t The verification loss is calculated, and the parameters of the dynamic Mixup mixing ratio generation module are updated by back propagation of the verification loss. (10) the Mixup mixing ratio obtained by the updated dynamic Mixup mixing ratio generation module is obtained. (11) E S , E T , Mixup mixing ratio input f m3t , get with backpropagation update f m3t ; (12) to E T_val input f m3t perform validation computation validation loss; (13) Repeat steps (5)-(12) until the number of repetitions reaches the maximum number of repetition rounds.

4. The cross-domain few-shot image recognition system of claim 3, wherein, The feature extraction network adopts a ResNet10 network, and the classifier adopts a GNN classifier; the specific working steps of the system are as follows: Step 1. Initialize the Mixup-3T image classification module f m3t network parameters of the dynamic Mixup mixing ratio generation module; Step 2. Randomly sample one dataset containing N*(k+q) picture data from the additional target domain dataset where N is the total number of categories, k is the training sample of each category, and q is the number of test samples; N, k, and q are selected according to the actual data needs, and the N categories are rewritten as [0, 1, …, N-1]; Step 3. With corresponding image labels Enter f m3t Perform verification and calculate verification loss. Where f m3t It consists of a cascaded ResNet10 network and a GNN classification network, and the specific process is as follows: (1) using a ResNet10 feature extraction network to extract features of the input image extracting features (2) Using GNN to compute on extracted features Classification probabilities for N classes (3) The Cross-Entropy algorithm is used to calculate the classification loss, with the following mathematical formula: ​ Step 4. The verification loss is input into the dynamic Mixup mixing ratio generation module g G , to obtain the Mixup mixing ratio where g G is composed of three fully connected layers, and the specific mathematical formula is as follows: Where t is the number of cycles minus 1; Step 5. Randomly sample a set E of N*(k+q) images from the source dataset S , from the extra target domain dataset and a set E of N*(k+q) images randomly sampled from the extra target domain dataset T , from the extra target domain dataset and a set E of N*(k+q) images randomly sampled from the extra target domain dataset T_val ; rewrite the N class labels in E T , E S , and E T_val to [0, 1, …, N-1] Step 6. Duplicate Mixup-3T to get a copy model f' of Mixup-3T m3t ; Step 7. Train E S , E T and Mixup mixing ratio Input f′ m3t to obtain a three-task classification loss The specific process is as follows: (1) Combine Mixup mixing ratio with E S With E T and the corresponding label mixing, get mixed dataset E mix , and its corresponding label y mix ; when mixing, first select the corresponding class pair according to the class name, including the mixing of class 0 in E S and class 0 in E T , the mixing of class 1 in E S and class 1 in E T ; the specific mathematical formula of mixing is as follows: When the first cycle, the above formula in is 1; (2) Extract features using ResNet10 on E S , E T , and E mix , respectively; (3) using the GNN to calculate classification probabilities of the extracted features for the N classes respectively and (4) Calculate using Cross-Entropy algorithm and classification loss and and add up with a ratio of 0.25, 0.25, 0.5 The specific mathematical formula is as follows: Step 8. Utilize Perform backpropagation update f' = f - a * df / dx m3t of the parameters Step 9. The E T_val Input f' m3t The verification calculation verifies the loss; the specific process is as follows: (1) using ResNet10 feature extraction network to E T_val extract features; (2) Use GNN to calculate the classification probability of the extracted features on N categories; (3) Use the Cross-Entropy algorithm to calculate the classification loss, and the specific mathematical formula is as follows: Step 10. Updating the parameters ω of the dynamic Mixup mixing ratio generation module by backpropagation updating the parameters ω of the dynamic Mixup mixing ratio generation module by backpropagation t+1 ; Step 11. The verification loss The updated dynamic Mixup mixing ratio generation module is obtained by inputting The specific formula is as follows: Step 12. The E S , E T , Input f m3t Get The specific process is as follows: (1) combined E S with E T and the corresponding label mixture, get the mixed dataset E mix , and its corresponding label y mix ; when mixing, first select the corresponding class pair according to the class name, including the mixing of class 0 in E S and class 0 in E T ; the mixing of class 1 in E S and class 1 in E T ; the specific mathematical formula of mixing is as follows: When the first cycle, the above formula in is 1; (2) Extract features from E S , E T and E mix using ResNet10 respectively. (3) Train the model using the features extracted in step (2). (3) Use GNN to calculate the classification probability of the extracted features on N categories respectively; (4) Calculate classification loss using Cross-Entropy algorithm and and adding them up with a ratio of 0.25, 0.25, 0.5 The specific mathematical formula is as follows: Step 13. Use Backpropagation updates f m3t ; Step 14. The E T_val Input f m3t The verification calculation verifies the loss, and the specific process is as follows: (1) using ResNet10 feature extraction network to E T_val extract features; (2) using the GNN to calculate classification probabilities for the extracted features over N classes (3) The Cross-Entropy algorithm is used to calculate the classification loss, with the following mathematical formula: ​ Step 15. Repeat steps 4-14 until the number of repetitions reaches 40000, and store the parameters of the Mixup-3T model at this time for image classification; Step 16. Image classification with Mixup-3T; the specific process is as follows: (1) randomly sample a data set E containing N*(k+q) picture data from the test data set test ; (2) using ResNet10 feature extraction network to E test extract features; (3) Use GNN to calculate the classification probability of the extracted features on N categories; (4) The category with the highest classification probability is taken as the category predicted by the network.

Images