A combined incremental learning method for image classification task

Abstract

The present application relates to the field of computer vision image classification, and provides a combined incremental learning method for image classification tasks, comprising: extracting image features through an image feature extractor; selecting appropriate prompts in an object prompt pool and a combined prompt pool and fusing them respectively; using the object prompt to guide the attribute prompt selection and fusing them; obtaining the feature sequence of the training image; obtaining the attribute feature, the object feature and the combined feature of the training image through a Transformer Encoder respectively; using an attribute classifier, an object classifier and a combined classifier to classify the attribute, the object and the combination respectively to obtain the prediction score of the attribute, the object and the combination; finally, testing the test image to obtain the final combined prediction score, and the combination with the highest score is the prediction classification result. The present application can greatly improve the classification accuracy of combined incremental images and make the image classification more accurate.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer vision image classification, and particularly relates to a combination incremental learning method for image classification tasks. BACKGROUND

[0002] Image classification based on incremental learning is a setting in which image and label data are provided to the model in a streaming fashion when the data annotation cannot be obtained at one time. However, the traditional incremental learning setting strictly limits the data, that is, the old task data cannot appear repeatedly in the new task. In view of the above limitation, combination incremental learning provides a more realistic and flexible scenario. In combination incremental learning, the class label is composed of both attributes and objects. In this way, the model can separate the attributes and objects of known classes to learn a large number of primitive visual features, and then recombine these primitive (including attribute and object concepts) features to obtain new combination visual features, so as to achieve a more fine-grained understanding of images.

[0003] Combination incremental learning can be widely applied to fine-grained clothing image recognition, fine-grained food image recognition and other real applications with combination characteristics. In combination incremental image classification, a major challenge is called catastrophic forgetting, that is, the adaptation of the model to the new data distribution usually leads to a significant reduction in the ability to recognize the old data distribution.

[0004] To overcome this difficulty, many studies have been devoted to preserving the network model or sample data of the old task, and then reusing the knowledge of the old task to constrain the training of the new task. For example, the most classic incremental learning method LwF (Li Z, Hoiem D. Learning without forgetting [J]. IEEE transactions on pattern analysis and machine intelligence, 2017, 40(12): 2935-2947.) proposes a model that preserves the old model after each old task ends, and uses the method of knowledge distillation to preserve the old task knowledge of the model in the subsequent task, but the LwF method maintains the ability of knowledge for the fine-grained image classification task. In addition, iCaRL (Rebuffi S A, Kolesnikov A, Sperl G, et al. icarl: Incremental classifier and representation learning [C] / / Proceedings of the IEEE conference on Computer Vision and Pattern Recognition. 2017: 2001-2010.) saves a small amount of old task samples and repeatedly trains in the subsequent task by adding them to the training data set to remember the knowledge of the old task. In recent years, the new paradigm of incremental learning based on prompt learning has gradually gained widespread recognition in the academic community. L2P (Wang Z, Zhang Z, Lee C Y, et al. Learning to prompt for continual learning [C] / / Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. 2022: 139-149.) first introduced the method of prompt learning into the field of incremental learning, which uses a pre-trained model to maintain the recognition ability of old task images through selective prompting. However, these methods only consider the general image recognition scenario and ignore the more practical fine-grained combined image recognition scenario, and cannot use primitive visual features to guide the learning of combined visual features. SUMMARY

[0005] In order to overcome the shortcomings of the existing incremental learning method in the combined incremental image classification task, the application provides a combined incremental image classification method based on multi-prompt pool cooperation, which can fully utilize the characteristics of repeated appearance of object objects and their attributes in past tasks, and further utilize this part of knowledge to recombine the primitive visual features in subsequent tasks to improve the combined classification ability of the model and alleviate the phenomenon of catastrophic forgetting.

[0006] The technical solutions of the application are as follows:

[0007] A combined incremental learning method for image classification tasks, comprising the following steps:

[0008] Step 100, initialization, randomly initializing the attribute prompt pool Object prompt pool Combined prompt pool

[0009] Step 200, inputting the training image x into the pre-trained image feature extractor f(·) to obtain the image feature f(x);

[0010] Step 201, object prompt and combined prompt selection, using the image feature f(x) to select appropriate prompts in the object prompt pool and the combined prompt pool respectively;

[0011] Step 202, respectively fusing the selected prompts in the object prompt pool and the selected prompts in the combined prompt pool;

[0012] Step 203, attribute prompt selection, using the object prompt to guide the attribute prompt selection and performing fusion;

[0013] Step 204, obtaining the feature sequence of the training image x;

[0014] Step 205, inputting the fused attribute prompt, object prompt, combined prompt and the feature sequence obtained in step 204 into the pre-trained Transformer Encoder to respectively obtain the attribute feature, object feature and combined feature of the training image;

[0015] Step 206, using the attribute classifier, object classifier and combined classifier to respectively classify the attribute, object and combination to obtain the prediction scores of the attribute, object and combination;

[0016] Step 207, iteration, using the classification results to calculate the loss function to constrain the object prompt pool, attribute prompt pool and combined prompt pool, updating the attribute prompt pool, object prompt pool and combined prompt pool, and repeating steps 200-207;

[0017] Step 300, test, input the test image into the model described in steps 200-207, according to the prediction classification score output by the attribute classifier, object classifier, and combination classifier, obtain the final combination prediction score, and the combination with the highest score is the prediction classification result.

[0018] Preferably, in step 200, ViT B / 16 trained on the ImageNet-1K image classification dataset is used as the image feature extractor f(·), where ViT B / 16 refers to Vision Transformer Base16.

[0019] Preferably, in step 201, the selection method of object prompts and combination prompts is as follows:

[0020] The image feature obtained by the image feature extractor from the image x is denoted as f(x), and the cosine similarity between the image feature f(x) and the key values in the object prompt pool is calculated, and the object prompts corresponding to the k key values with the highest scores are selected Similarly, the cosine similarity between the image feature f(x) and the key values in the combination prompt pool is calculated, and the combination prompts corresponding to the k key values with the highest scores are selected

[0021] Preferably, in step 202, the fusion method of object prompts and combination prompts is as follows:

[0022] The selected object prompts in step 201 are subjected to a generalized mean fusion operation, which is as follows:

[0023]

[0024] Where GEM(·) represents the generalized mean fusion operation, η represents an artificially selected hyperparameter, and P o is the object prompt after fusion.

[0025] Similarly, the selected combination prompts in step 201 are subjected to a generalized mean fusion, and the fusion method is as follows:

[0026]

[0027] Where GEM(·) represents the generalized mean fusion operation, η represents an artificially selected hyperparameter, and P c is the combination prompt after fusion.

[0028] Preferably, in step 203, the object prompt guides the selection of the attribute prompt, and the method is as follows:

[0029] The fused object hint P obtained in step 202 o As a guiding signal, it is injected into the image feature f(x) to obtain the attribute cue selection signal f. s (x). The injection method uses a cross-attention mechanism, and the specific operation formula is as follows:

[0030]

[0031] Where CrossAtten(·,·) is the Cross Attention operation, W Q W K W V These represent the learnable mapping matrices in the query vector (Query), the key vector (Key), and the content vector (Value) in the cross-attention mechanism, respectively, where D represents the dimension of the vectors. Since f s (x) incorporates information about object hints P o Information related to f s (x) represents the prompt selection signal following the object prompt guidance, which can be regarded as the attribute prompt selection signal.

[0032] Use the attribute hints to select signal f s (x) Calculate its relationship with the attribute hint pool Cosine similarity between key values, select the attribute hints corresponding to the k key values ​​with the highest scores. Perform generalized mean fusion, using the same fusion method as in step 202, as shown below:

[0033]

[0034] Where GEM(·) represents the generalized mean fusion operation, and η represents the manually selected hyperparameter. P s This is a hint for the combined result after merging.

[0035] Preferably, in step 204, the feature sequence of the training image x is obtained, and the design method is as follows:

[0036] The Embedding Layer of ViT B / 16, trained on the ImageNet-1K image classification dataset, is selected as the feature sequence extractor. After the training image is input into the feature sequence extractor, the output of the feature sequence extractor is the feature sequence.

[0037] Preferably, in step 205, the method for obtaining the attribute features, object features, and combined features of the training image is designed as follows:

[0038] First, the object prompt and the combination prompt obtained in step 202 after fusion, the combination prompt obtained in step 203 after fusion, and the training image feature sequence obtained in step 204 are spliced in the first dimension in the order of "combination-attribute-object-image feature sequence" to obtain a mixed feature sequence; then, the mixed feature sequence is input into the Transformer Encoder in ViT B / 16 trained on the ImageNet-1K image classification dataset; the Token at the first position output by the Transformer Encoder is the combination feature, the Token at the second position output by the Transformer Encoder is the attribute feature, and the Token at the third position output by the Transformer Encoder is the object feature.

[0039] Preferably, in step 206, the prediction scores of the attribute, the object, and the combination are obtained, and the design method is as follows:

[0040] Three parameter-shared linear layers are set as an attribute classifier, an object classifier, and a combination classifier. The attribute prediction score is obtained by inputting the attribute feature obtained in step 205 into the attribute classifier. The object prediction score is obtained by inputting the object feature obtained in step 205 into the object classifier. The combination prediction score is obtained by inputting the combination feature obtained in step 205 into the combination classifier.

[0041] Preferably, in step 207, the loss function is calculated, and the design method is as follows:

[0042] The training of the model described in steps 200-206 is constrained by three loss functions, which are a diversity loss function L div , a proxy loss function L sur , and a classification loss function L Class .

[0043] The diversity loss function L div is obtained by summing the direction decoupling losses of the three prompt pools and between the prompt pools, that is, wherein represents the direction decoupling loss between the attribute pool and the attribute pool, the object pool and the object pool, and the combination pool and the combination pool, represents the direction decoupling loss between the attribute pool and the object pool, the attribute pool and the combination pool, and the combination pool and the object pool. The six loss calculation methods are the same, and represents any one of the six losses, and the calculation formula is:

[0044]

[0045] wherein, M is the size of the prompt pool, and θthre is a hyper-parameter, set by human, the angle; θ nm represents any two prompts and the angle between them, measured as follows:

[0046]

[0047] where ∈ is set to a very small constant to prevent the denominator from being 0. and represent any two prompts in the prompt pool, and || ||2 represents the L2 norm.

[0048] Agent loss function L sur is used to constrain the selection of prompt pool keys, which is obtained by summing the cosine similarity between visual features and key values, calculated as follows:

[0049]

[0050] where, is the attribute prompt selected from the attribute prompt pool corresponding key value; is the object prompt selected from the object prompt pool corresponding key value; is the combination prompt selected from the combination prompt pool corresponding key value, COS(·) represents the cosine similarity.

[0051] Classification loss L Class is composed of traditional cross-entropy loss L CE and inverse cross-entropy loss L RCE , which act on attribute classification, object classification, and combination classification, respectively, with specific formulas as follows:

[0052]

[0053] where, represent attribute classification cross-entropy loss, object classification cross-entropy loss, and combination classification cross-entropy loss, respectively, represent attribute inverse cross-entropy loss, object inverse cross-entropy loss, and combination inverse cross-entropy loss, respectively. α is the weight, used to balance the effects of cross-entropy loss and inverse cross-entropy loss.

[0054] Finally, the loss function of the entire network is calculated as:

[0055] L total = L div + L sur + L Class

[0056] Preferably, in step 300, the calculation of the final combined prediction score is designed as follows:

[0057] The attribute prediction score, the object prediction score and the combined prediction score are marked as p(s|x), p(o|x) and p(c|x) respectively, and the final combined prediction score is the sum of the three prediction scores, which is represented as:

[0058] p final =p(c|x)+β(p(s|x)+p(o|x))

[0059] Wherein, β is a hyperparameter, used to balance the contribution between combination and base element. After obtaining the prediction combined prediction score, the combination with the highest score is the prediction classification result.

[0060] The beneficial effects of the present application are:

[0061] The present application designs two combined incremental image data sets to verify the effectiveness of the method, which are Split-Clothing and Split-UT-Zappos data sets. The present application is compared with the most advanced incremental image classification method based on prompt learning, including L2P, Dual-Prompt and LGCL three methods. In order to compare fairly, the models of these methods are directly provided by the authors or their original training models. Compared with the three most advanced methods, the present application method achieves the most advanced performance on all indicators of the data set, including image classification average accuracy (Avg Acc), forgetting rate (FTT), attribute classification average accuracy (State), object classification average accuracy (Object) and harmonic mean (HM). Especially in the comprehensive index of harmonic mean (HM), compared with the previous method, it has made obvious improvement. BRIEF DESCRIPTION OF DRAWINGS

[0062] Figure 1 is the workflow diagram of the combined incremental image classification method based on multi-prompt pool cooperation provided by the present application;

[0063] Figure 2 is a schematic diagram of the combined incremental image classification network based on multi-prompt pool cooperation provided by the present application;

[0064] Figure 3 is a schematic diagram of the specific operation of object prompt guiding attribute prompt in the network of the present application;

[0065] Figure 4are the quantitative experiment and qualitative experiment results of the method provided in the application, wherein (a), (b) and (c) are respectively the accuracy rates of the combination, the attribute and the object classification in the incremental learning task setting of the application, and (d) is the result of the method provided in the application and the L2P method in the text search based on the image on the Split-UT-Zappos data set;

[0066] Figure 5 are the comparison of the results of the application and other methods on the Split-Clothing data set using the t-SNE method for visualization, wherein (a) is the L2P method, and (b) is the method CompILer provided in the application. DETAILED DESCRIPTION

[0067] The specific embodiments of the application are further described below in combination with the drawings and technical solutions.

[0068] Figure 1 is the work flow chart of the combination incremental image classification method based on the multi-prompt pool cooperation provided in the application. The specific operation is taken as an example as shown in Figure 2 , and includes the following steps:

[0069] Step 100, initialization, randomly initializing the attribute prompt pool the object prompt pool the combination prompt pool Each prompt pool is composed of an index key value and a prompt. The sizes of the three prompt pools are consistent, the index key values are all matrices with a size of 20 and a dimension of 768, and the prompts are all tensors with a size of 20, a length of 5 and a dimension of 768, which are randomly generated in the standard normal distribution.

[0070] Step 200, inputting the training image x into the pre-trained feature extractor f(·), the structure of f(·) is consistent with the Vision Transformer and is pre-trained using the ImageNet 21K data set. The size of all input images is adjusted to 224x224. The random seed of the whole experiment is set to 42, the Adam optimizer (which can be referred to in Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR, 2015) is adopted, and the network learning rate is 0.03. The whole network is experimented on an NVIDIA RTX 3090 GPU, the batch size is set to 16 during training, and the epoch number is set to 20;

[0071] Step 201, object prompt, combination prompt selection, select appropriate prompts in the object prompt pool and the combination prompt pool respectively using visual features, select the prompts corresponding to the top 5 index key values by calculating the cosine similarity between the output of the feature extractor and all index key values in the prompt pool;

[0072] Step 202, respectively, fuse the selected prompts in the object prompt pool and the selected prompts in the combination prompt pool, and fuse the 5 prompts selected from the prompt pool into 1 prompt using the generalized mean fusion operation;

[0073] Step 203, as shown in Figure 3 , attribute prompt selection, use the cross attention mechanism to realize the guidance of the object prompt to the attribute prompt by taking the fused object prompt as Q and the output of the feature extractor f(·) as K and V. The number of attention heads is 1, the Dropout parameter is set to 0, and batch_first is set to True. Calculate the cosine similarity between the features after the guidance and all index key values in the attribute prompt pool, select the prompts corresponding to the top 5 index key values, and fuse them into one prompt using the generalized mean fusion operation;

[0074] Step 204, classification, connect the fused attribute prompt, object prompt, and combination prompt to the feature sequence output by the VisionTransformer, and then input them together into the multi-head attention mechanism of the Vision Transformer. Use the attribute classifier, object classifier, and combination classifier to classify, and calculate the classifier probability from the output of the multi-head attention mechanism;

[0075] Step 205, iteration, use the classification result to calculate the loss function to constrain the model, update the attribute prompt pool, object prompt pool, combination prompt pool, attribute classifier, object classifier, and combination classifier, and repeat steps 200-205. The loss function is composed of three parts: diversity loss function, proxy loss function, and classification loss function;

[0076] Step 300, test, input the test image into the trained model (referred to as CompILer), and output the predicted classification result.

[0077] As shown in Figure 4 , the present application carried out experiments on the Split-Clothing dataset, which contains a total of 5 incremental stages. The experiment was evaluated in stages for attribute accuracy, object accuracy, and combination accuracy. The proposed method achieved satisfactory experimental results, with an average combination accuracy of 88.74%, an average attribute accuracy of 91.61%, and an average object accuracy of 96.34%.

[0078] As Figure 5 shown in the t-SNE experimental results compared with other continuous learning methods, the image feature distribution of the present application makes the inter-class dispersion degree higher and the intra-class aggregation degree more dense, so the classification performance is significantly improved.

Claims

1. A combined incremental learning method for an image classification task, characterized in that, The method comprises the following steps: Step 100, initialization, randomly initialize attribute cue pool Object cue pool Combined cue pool Step 200, inputting a training image x into a pre-trained image feature extractor f(·) to obtain an image feature f(x); Step 201, object prompt selection and combination prompt selection, using the image feature f(x) to select appropriate prompts in an object prompt pool and a combination prompt pool respectively; Step 202, respectively fusing the selected prompts in the object prompt pool and the selected prompts in the combination prompt pool; Step 203, attribute prompt selection, using the object prompt to guide the attribute prompt selection, and performing fusion; Step 204, obtaining a feature sequence of the training image x; Step 205, inputting the fused attribute prompt, object prompt, combination prompt and the feature sequence obtained in step 204 into a pre-trained Transformer Encoder to respectively obtain an attribute feature, an object feature and a combination feature of the training image; Step 206, using an attribute classifier, an object classifier and a combination classifier to respectively classify the attribute, the object and the combination to obtain prediction scores of the attribute, the object and the combination; Step 207, iteration, using the classification results to calculate a loss function to constrain the object prompt pool, the attribute prompt pool and the combination prompt pool, updating the attribute prompt pool, the object prompt pool and the combination prompt pool, and repeating steps 200-207; Step 300, testing, inputting a test image into the model described in steps 200-207, obtaining a final combination prediction score according to prediction classification scores output by the attribute classifier, the object classifier and the combination classifier, and the combination with the highest score being a prediction classification result.

2. The combined incremental learning method for image classification tasks according to claim 1, wherein, In step 200, a ViT B / 16 trained on an ImageNet-1K image classification dataset is used as the image feature extractor f(·), wherein the ViT B / 16 refers to a Vision Transformer Base 16.

3. The combined incremental learning method for image classification tasks according to claim 1, wherein, In step 201, the selection methods of the object prompt and the combination prompt are as follows: The image feature of the image x is denoted as f(x), and the cosine similarity between the image feature f(x) and the object prompt pool is calculated. The object prompt corresponding to the k key values with the highest scores is selected Similarly, the cosine similarity between the image feature f(x) and the combination prompt pool is calculated. The combination prompt corresponding to the k key values with the highest scores is selected 4. The combined incremental learning method for image classification tasks according to claim 1, wherein, In step 202, the fusion methods of the object prompt and the combination prompt are as follows: presenting the selected object to step 201 The generalized mean fusion operation is performed as follows: where GEM(·) represents a generalized mean fusion operation, and η denotes a hyperparameter selected artificially; P o prompting for the object after fusion; By analogy, the combined cues selected in step 201 are prompted Generalized mean fusion is performed, in the following manner: where GEM(·) represents the generalized mean fusion operation, and η denotes a hyperparameter selected artificially; P c is the combined cue after fusion.

5. The combined incremental learning method for image classification tasks according to claim 1, wherein, In step 203, the object prompt guides the selection of the attribute prompt, and the method is as follows: The prompted object after fusion obtained in step 202 is prompted P o As a guide signal, injected into the image feature f(x) to obtain the attribute prompt selection signal f s (x); the injection method uses a cross attention mechanism (Cross Attention), and the specific operation formula is as follows: where CrossAtten(·,·) is a cross attention operation, W Q , W K , W V represent the learnable mapping matrices in the query vector Query, the key vector Key, and the value vector Value of the cross attention mechanism, respectively, and D represents the dimension of the vector; since the relevant information about the object prompt P s is fused in f o (x), f s (x) represents the prompt selection signal after the guidance of the object prompt, which can be regarded as an attribute prompt selection signal; Using attribute prompt selection signal f s (x) Calculate the cosine similarity between the attribute prompt pool and the key value, select the attribute prompt corresponding to the k key value with the highest score Generalized mean fusion is performed, and the fusion method is consistent with that in step 202, as shown below: where GEM(·) represents the generalized mean fusion operation, and η denotes a hyperparameter selected artificially; P s is the combined cue after fusion.

6. The combined incremental learning method for image classification tasks according to claim 1, wherein, In step 204, the feature sequence of the training image x is obtained, and the design method is as follows: The Embedding Layer in the ViT B / 16 trained on the ImageNet-1K image classification dataset is selected as the feature sequence obtainer, and the output of the feature sequence obtainer after inputting the training image into the feature sequence obtainer is the feature sequence.

7. The combined incremental learning method for image classification tasks according to claim 1, wherein, In step 205, the attribute feature, the object feature and the combination feature of the training image are obtained, and the design method is as follows: Firstly, the object prompt and the combination prompt obtained after fusion in step 202, the combination prompt obtained after fusion in step 203, and the training image feature sequence obtained in step 204 are spliced in the first dimension in the order of "combination-attribute-object-image feature sequence", to obtain a mixed feature sequence; then, the mixed feature sequence is input into the Transformer Encoder in ViT B / 16 trained on the ImageNet-1K image classification dataset; the Token in the first position output by the Transformer Encoder is the combination feature, the Token in the second position output by the Transformer Encoder is the attribute feature, and the Token in the third position output by the Transformer Encoder is the object feature.

8. The combined incremental learning method for image classification tasks according to claim 1, wherein, In step 206, the prediction scores of the attribute, the object, and the combination are obtained, and the design method is as follows: Three parameter-shared linear layers are set as an attribute classifier, an object classifier, and a combination classifier respectively; the attribute feature obtained in step 205 is input into the attribute classifier to obtain an attribute prediction score; the object feature obtained in step 205 is input into the object classifier to obtain an object prediction score; and the combination feature obtained in step 205 is input into the combination classifier to obtain a combination prediction score.

9. The combined incremental learning method for image classification tasks according to claim 1, wherein, In step 207, the calculation of the loss function is as follows: The training of the model described in steps 200-206 is subject to three loss functions, namely a diversity loss function L div , an agent loss function L sur , and a classification loss function L Class . Diversity loss function L div The direction decoupling loss summation is obtained by calculating the direction decoupling loss inside and between the three prompt pools, that is, Wherein respectively represent the direction decoupling loss between the attribute pool and the attribute pool, between the object pool and the object pool, and between the combination pool and the combination pool, respectively represent the direction decoupling loss between the attribute pool and the object pool, between the attribute pool and the combination pool, and between the combination pool and the object pool; the calculation methods of the six losses are the same, using represents any one of the six losses, and the calculation formula is: where M is the hint pool size, θ thre is a hyper-parameter, an angle set by human; θ nm represents the angle between any two hints and is measured as follows: Wherein, ∈ is set to be a very small constant, to prevent the denominator from being 0; With Denotes any two prompts in the prompt pool, and || ||2 represents the L2 norm; Proxy loss function L sur To constrain the selection of the prompt pool key-value, the cosine similarity between the visual features and the key-value is summed up, and the calculation method is as follows: wherein, a property hint selected from the property hint pool the corresponding key value; an object hint selected from the object hint pool the corresponding key value; a combination hint selected from the combination hint pool the corresponding key value, COS(·) represents cosine similarity; Classification loss L Class The traditional cross-entropy loss L CE And the reverse cross-entropy loss L RCE Composed of attribute classification, object classification, and combined classification, the specific formulas are as follows: wherein, respectively represent attribute classification cross-entropy loss, object classification cross-entropy loss, and combined classification cross-entropy loss, respectively represent attribute reverse cross-entropy loss, object reverse cross-entropy loss, and combined reverse cross-entropy loss; and a is a weight used to balance the effects of cross-entropy loss and reverse cross-entropy loss. Finally, the loss function of the entire network is calculated as: L total = L div + L sur + L Class .

10. The combined incremental learning method for image classification tasks according to claim 1, wherein, In step 300, the calculation of the final combination prediction score is as follows: The attribute prediction score, the object prediction score, and the combination prediction score are marked as p(s|x), p(o|x), and p(c|x) respectively, and the final combination prediction score is the sum of the three prediction scores, represented as: p final = p(c|x) + β(p(s|x) + p(o|x)) Wherein, β is a hyperparameter for balancing the contribution between the combination and the primitive; after obtaining the prediction combination prediction score, the combination with the highest score is the prediction classification result.

Images