Combined incremental learning method for image classification task

Abstract

The invention relates to the field of computer vision image classification, and provides an image classification task-oriented combined incremental learning method, which comprises the following steps of: extracting image features from an image feature extractor; selecting proper prompts from an object prompt pool and a combined prompt pool for the image features, and fusing the proper prompts; the object prompt is used for guiding attribute prompt selection and fusion; acquiring a feature sequence of the training image; respectively obtaining an attribute feature, an object feature and a combination feature of the training image through Transform Encoder; classifying the attributes, the objects and the combinations by using an attribute classifier, an object classifier and a combination classifier to obtain prediction scores of the attributes, the objects and the combinations; and finally, testing the test image to obtain a final combination prediction score, wherein the combination with the highest score is a prediction classification result. The method can greatly improve the classification precision of the combined incremental image, and enables the image classification to be more accurate.

Description

Technical Field

[0001] The present invention relates to the field of computer vision image classification, and in particular to a combined incremental learning method for image classification tasks. Background Art

[0002] Image classification based on incremental learning is a setting in which image and label data are gradually provided to model training in the form of a stream when data annotation cannot be obtained all at once. However, the traditional incremental learning setting has strict restrictions on the data, that is, old task data will not be repeated in the new task. In view of the above restrictions, combinatorial incremental learning provides a more realistic and flexible scenario. In combinatorial incremental learning, the category label is composed of both attributes and objects. In this way, the model can separate the attributes and objects of known categories to learn a large number of primitive visual features, and then recombine these primitive (including attribute and object concept) features to obtain new combined visual features, achieving a more fine-grained understanding of the image.

[0003] Combinatorial incremental learning can be widely used in many real-world applications with combinatorial characteristics in the future, such as fine-grained clothing image recognition, fine-grained food image recognition, etc. In combinatorial incremental image classification, a major challenge is called catastrophic forgetting, that is, the model's adaptation to the new data distribution usually leads to a significant reduction in the ability to recognize the old data distribution.

[0004] In order to overcome this difficulty, many studies have been devoted to retaining the network model or sample data of the old task, and then reusing the knowledge of the old task for constraints in the training of the new task. For example, the most classic incremental learning method LwF (Li Z, Hoiem D. Learning without forgetting [J]. IEEE transactions on pattern analysis and machine intelligence, 2017, 40 (12): 2935-2947.), the model it proposed retains the old model after each old task, and uses the knowledge distillation method to retain the old task knowledge of the model in the subsequent tasks, but the LwF method has poor ability to maintain knowledge for fine-grained image classification tasks. In addition, iCaRL (Rebuffi SA, Kolesnikov A, Sperl G, et al. iCaRL: Incremental classifier and representation learning [C] / / Proceedings of the IEEE conference on Computer Vision and Pattern Recognition. 2017: 2001-2010.) saves a small number of samples of the old task, and adds them to the training data set in the subsequent tasks for repeated training, so as to memorize the knowledge of the old task. In recent years, the new paradigm of incremental learning based on prompt learning has gradually gained wide recognition in the academic community. L2P (Wang Z, Zhang Z, Lee CY, et al. Learning toprompt for continual learning [C] / / Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. 2022: 139-149.) first introduced the prompt learning method into the field of incremental learning. It uses pre-trained models to maintain the recognition ability of old task images through selective prompts. However, these methods only consider the scenario of general image recognition, while ignoring the more practical scenario of fine-grained combined image recognition, and cannot use primitive visual features to guide the learning of combined visual features. Summary of the invention

[0005] In order to overcome the shortcomings of existing incremental learning methods in combined incremental image classification tasks, the present invention provides a combined incremental image classification method based on multi-cue pool collaboration, which can make full use of the characteristics of objects and their attributes that appear repeatedly in past tasks, and further use this part of knowledge to recombine primitive visual features in subsequent tasks to improve the combined classification ability of the model and alleviate the phenomenon of catastrophic forgetting.

[0006] The technical solution of the present invention is as follows:

[0007] A combined incremental learning method for image classification tasks, comprising the following steps:

[0008] Step 100, initialization, randomly initialize the attribute prompt pool Object Hint Pool Combined Tip Pool

[0009] Step 200, input the training image x into the pre-trained image feature extractor f(·) to obtain the image feature f(x);

[0010] Step 201, object prompt and combination prompt selection, using the image feature f(x) to select appropriate prompts from the object prompt pool and the combination prompt pool respectively;

[0011] Step 202, respectively fusing the prompts selected in the object prompt pool and the prompts selected in the combined prompt pool;

[0012] Step 203, attribute prompt selection, using object prompts to guide attribute prompt selection and perform fusion;

[0013] Step 204, obtaining a feature sequence of the training image x;

[0014] Step 205, inputting the fused attribute prompt, object prompt, combination prompt and the feature sequence obtained in step 204 into the pre-trained Transformer Encoder to obtain the attribute features, object features and combination features of the training image respectively;

[0015] Step 206, using the attribute classifier, the object classifier, and the combination classifier to classify the attribute, the object, and the combination respectively, and obtain the prediction scores of the attribute, the object, and the combination;

[0016] Step 207, iteration, using the classification result to calculate the loss function to constrain the object prompt pool, the attribute prompt pool, and the combination prompt pool, update the attribute prompt pool, the object prompt pool, and the combination prompt pool, and repeat steps 200 to 207;

[0017] Step 300, testing, inputs the test image into the model described in steps 200 to 207, obtains the final combined prediction score based on the predicted classification scores output by the attribute classifier, object classifier, and combined classifier, and the combination with the highest score is the predicted classification result.

[0018] Preferably, in step 200, ViT B / 16 trained on the ImageNet-1K image classification dataset is used as the image feature extractor f(·), where ViT B / 16 refers to Vision Transformer Base16.

[0019] Preferably, the method for selecting the object prompt and the combination prompt in step 201 is as follows:

[0020] The image feature obtained by the image feature extractor is recorded as f(x). By calculating the image feature f(x) and the object prompt pool The cosine similarity between the key values ​​in the object prompt is selected, and the object prompts corresponding to the k key values ​​with the highest scores are selected. Similarly, calculate the image feature f(x) and the combined prompt pool The cosine similarity between the key values ​​in the formula is used to select the combination prompts corresponding to the k key values ​​with the highest scores.

[0021] Preferably, in step 202, the method for fusing the object prompt and the combination prompt is as follows:

[0022] Prompt the object selected in step 201 Perform the generalized mean fusion operation as follows:

[0023]

[0024] where GEM(·) represents the generalized mean fusion operation, η represents a manually selected hyperparameter; P o Hint for the fused object.

[0025] Similarly, the combination selected in step 201 is prompted Perform generalized mean fusion, the fusion method is as follows:

[0026]

[0027] where GEM(·) represents the generalized mean fusion operation, η represents a manually selected hyperparameter; P c This is a hint for the combination after fusion.

[0028] Preferably, in step 203, the object prompt guides the selection of the attribute prompt in the following manner:

[0029] The fused object obtained in step 202 is prompted P o As a guiding signal, it is injected into the image feature f(x) to obtain the attribute prompt selection signal f s (x). The injection method uses the Cross Attention mechanism, and the specific operation formula is as follows:

[0030]

[0031] Among them, CrossAtten(·,·) is the cross attention mechanism (Cross Attention) operation, W Q , W K , W V They represent the learnable mapping matrices in the query vector Query, the queried vector Key, and the content vector Value in the cross-attention mechanism, and D represents the dimension of the vector. s (x) incorporates the object hint P o Therefore, f s (x) represents the cue selection signal after the object cue guidance, which can be regarded as the attribute cue selection signal.

[0032] Use attribute hints to select signal f s (x) Calculate its attribute hint pool The cosine similarity between the key values ​​in the table, select the attribute prompts corresponding to the k key values ​​with the highest scores Perform generalized mean fusion, and the fusion method is consistent with the fusion method in step 202, as shown below:

[0033]

[0034] Here, GEM(·) represents the generalized mean fusion operation and η represents a manually selected hyperparameter. s This is a hint for the combination after fusion.

[0035] Preferably, in step 204, a feature sequence of the training image x is obtained, and the design method is as follows:

[0036] The Embedding Layer in ViT B / 16 trained on the ImageNet-1K image classification dataset is selected as the feature sequence acquirer. After the training image is input into the feature sequence acquirer, the output of the feature sequence acquirer is the feature sequence.

[0037] Preferably, in step 205, the acquisition of attribute features, object features and combined features of the training image is designed as follows:

[0038] First, the fused object prompts and combination prompts obtained in step 202, the fused combination prompts obtained in step 203, and the training image feature sequence obtained in step 204 are spliced ​​together in the order of "combination-attribute-object-image feature sequence" in the first dimension to obtain a mixed feature sequence; then, the mixed feature sequence is input into the Transformer Encoder in ViT B / 16 trained on the ImageNet-1K image classification dataset; the Token at the first position of the Transformer Encoder output is the combination feature, the Token at the second position of the output is the attribute feature, and the Token at the third position of the output is the object feature.

[0039] Preferably, in step 206, the predicted scores of the attributes, objects, and combinations are obtained, and the design method is as follows:

[0040] Three linear layers whose parameters are not shared are set as attribute classifier, object classifier and combination classifier respectively. The attribute feature obtained in step 205 is input into the attribute classifier to obtain the attribute prediction score. The object feature obtained in step 205 is input into the object classifier to obtain the object prediction score. The combination feature obtained in step 205 is input into the combination classifier to obtain the combination prediction score.

[0041] Preferably, in step 207, the loss function is calculated and designed as follows:

[0042] The training of the model described in steps 200 to 206 is subject to three loss functions, namely, the diversity loss function L div , proxy loss function L sur And the classification loss function L Class ;

[0043] Diversity loss function L div It is obtained by calculating the sum of the directional decoupling losses within the three cue pools and between the cue pools, that is, in They represent the directional decoupling losses between attribute pools, between object pools, and between combination pools. They represent the directional decoupling losses between the attribute pool and the object pool, between the attribute pool and the combination pool, and between the combination pool and the object pool. The six losses are calculated in the same way, using represents any one of the six losses, and its calculation formula is:

[0044]

[0045] Where M is the size of the prompt pool, θthre is a hyperparameter, which is a manually set angle; θ nm Represents any two prompts and The angle between them is measured as follows:

[0046]

[0047] Here, ∈ is set to a very small constant to prevent the denominator from being zero. and represents any two prompts in the prompt pool, and ∥∥2 represents the L2 norm.

[0048] Proxy loss function L sur The selection of the key value for the constraint hint pool is obtained by calculating the sum of the cosine similarity between the visual features and the key value. The calculation method is as follows:

[0049]

[0050] in, The attribute hint selected from the attribute hint pool The corresponding key value; A cue for an object selected from the object cue pool The corresponding key value; is a combination prompt selected from the combination prompt pool The corresponding key value, COS(·) represents the cosine similarity.

[0051] Classification loss L Class By the traditional cross entropy loss L CE and the reverse cross entropy loss L RCE Composition, respectively acting on attribute classification, object classification, and combination classification. The specific formula is as follows:

[0052]

[0053] in, They represent attribute classification cross entropy loss, object classification cross entropy loss, and combination classification cross entropy loss, respectively. They represent attribute reverse cross entropy loss, object reverse cross entropy loss, and combination reverse cross entropy loss respectively. α is a weight used to balance the effects of cross entropy loss and reverse cross entropy loss.

[0054] Finally, the loss function of the entire network is calculated as:

[0055] L total =L div +L sur +L Class

[0056] Preferably, in step 300, the calculation of the final combined prediction score is designed as follows:

[0057] The attribute prediction score, object prediction score, and combined prediction score are marked as p(s|x), p(o|x), and p(c|x), respectively. The final combined prediction score is the sum of the three prediction scores, expressed as:

[0058] p final =p(c|x)+β(p(s|x)+p(o|x))

[0059] Among them, β is a hyperparameter used to balance the contribution between the combination and the primitive. After obtaining the prediction score of the prediction combination, the combination with the highest score is the prediction classification result.

[0060] Beneficial effects of the present invention:

[0061] The present invention designs two combined incremental image datasets to verify the effectiveness of the method, namely the Split-Clothing and Split-UT-Zappos datasets. The present invention is compared with the most advanced incremental image classification methods based on prompt learning, including L2P, Dual-Prompt, and LGCL. For fair comparison, the models of these methods are directly provided by their authors or their original training models. Compared with the three most advanced methods, the method of the present invention has achieved the most advanced performance in all indicators on the dataset, including the average accuracy of image classification (Avg Acc), the forgetting rate (FTT), the average accuracy of attribute classification (State), the average accuracy of object classification (Object), and the harmonic mean (HM). Especially in the comprehensive indicator of the harmonic mean (HM), a significant improvement has been achieved compared with the previous methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 It is a workflow diagram of the combined incremental image classification method based on multi-prompt pool collaboration provided by the present invention;

[0063] Figure 2 is a schematic diagram of a combined incremental image classification network based on multi-cue pool collaboration provided by the present invention;

[0064] Figure 3 It is a schematic diagram of the object prompting guidance attribute prompting specific operation in the network of the present invention;

[0065] Figure 4The quantitative and qualitative experimental results of the method proposed in the present invention are shown, where (a), (b), and (c) are the accuracy of the combination, attribute, and object classification in the incremental learning task setting of the present invention, respectively, and (d) is the result of the image-based text search of the present invention and the L2P method on the Split-UT-Zappos dataset;

[0066] Figure 5 The results of the present invention are compared with those of other methods on the Split-Clothing dataset using the t-SNE method for visualization, where (a) is the L2P method and (b) is the CompILer method proposed by the present invention. DETAILED DESCRIPTION

[0067] The specific implementation of the present invention is further described below in conjunction with the accompanying drawings and technical solutions.

[0068] Figure 1 This is a workflow diagram of the combined incremental image classification method based on multi-prompt pool collaboration provided by the present invention. Figure 2 Taking the example shown, the following steps are included:

[0069] Step 100, initialization, randomly initialize the attribute prompt pool Object Hint Pool Combined Tip Pool Each hint pool is composed of an index key value and a hint. The sizes of the three hint pools are consistent. The index key values ​​are all matrices of size 20 and dimension 768. The hints are all tensors of size 20, length 5, and dimension 768. The initialization method follows the standard normal distribution and random generation.

[0070] Step 200, input the training image x into the pre-trained feature extractor f(·), the structure of f(·) is consistent with the Vision Transformer and is pre-trained using the ImageNet 21K dataset. The size of all input images is adjusted to 224×224. The random seed of the entire experiment is set to 42, and the Adam optimizer is used (see Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In3rd International Conference on Learning Representations, ICLR, 2015), and the network learning rate is 0.03. The entire network is experimented on an NVIDIA RTX 3090 GPU, the batch size during training is set to 16, and the number of epochs is set to 20;

[0071] Step 201, object prompt and combination prompt selection, using visual features to select appropriate prompts in the object prompt pool and the combination prompt pool respectively, by calculating the cosine similarity between the output of the feature extractor and all the index key values ​​in the prompt pool, selecting the prompts corresponding to the five index key values ​​with the highest similarity;

[0072] Step 202, respectively fuse the prompts selected from the object prompt pool and the prompts selected from the combination prompt pool, and fuse the five prompts selected from the prompt pool into one prompt using a generalized mean fusion operation;

[0073] Step 203, such as Figure 3 As shown in the figure, attribute prompt selection uses the fused object prompt as Q, the output of the feature extractor f(·) as K and V, and uses the cross attention mechanism to achieve the guidance of the object prompt on the attribute prompt. The number of attention heads is 1, the Dropout parameter is set to 0, and batch_first is set to True. The cosine similarity of the guided features and all index key values ​​in the attribute prompt pool is used to select the prompts corresponding to the 5 index key values ​​with the highest similarity and use the generalized mean fusion operation to fuse them into one prompt;

[0074] Step 204, classification, connect the fused attribute prompts, object prompts, and combination prompts to the feature sequence output by VisionTransformer, and then send them together to the multi-head attention mechanism of Vision Transformer. Use attribute classifiers, object classifiers, and combination classifiers for classification, and calculate the classifier probability based on the output of the multi-head attention mechanism;

[0075] Step 205, iterate, use the classification results to calculate the loss function constraint model, update the attribute prompt pool, object prompt pool, combination prompt pool, attribute classifier, object classifier and combination classifier, and repeat steps 200 to 205. The loss function consists of three parts: diversity loss function, proxy loss function and classification loss function;

[0076] Step 300, testing, inputs the test image into the trained model (referred to as CompILer), and outputs the predicted classification result.

[0077] like Figure 4 As shown in the figure, the present invention was experimented on the Split-Clothing dataset, which contains a total of 5 incremental stages. The experiment evaluated the attribute accuracy, object accuracy, and combination accuracy in stages. The proposed method achieved satisfactory experimental results, with an average combination accuracy of 88.74%, an average attribute accuracy of 91.61%, and an average object accuracy of 96.34%.

[0078] like Figure 5 As shown, the present invention and other continuous learning methods have been compared with t-SNE experimental results. In comparison, the image feature distribution of the present invention makes the inter-class dispersion higher and the intra-class aggregation denser, so the classification performance is significantly improved.

Claims

1. A combined incremental learning method for image classification tasks, characterized in that: The following steps are involved: Step 100, initialization, randomly initialize the attribute prompt pool Object Hint Pool Combined Tip Pool Step 200, input the training image x into the pre-trained image feature extractor f(·) to obtain the image feature f(x); Step 201, object prompt and combination prompt selection, using the image feature f(x) to select appropriate prompts from the object prompt pool and the combination prompt pool respectively; Step 202, respectively fusing the prompts selected in the object prompt pool and the prompts selected in the combined prompt pool; Step 203, attribute prompt selection, using object prompts to guide attribute prompt selection and perform fusion; Step 204, obtaining a feature sequence of the training image x; Step 205, inputting the fused attribute prompt, object prompt, combination prompt and the feature sequence obtained in step 204 into the pre-trained Transformer Encoder to obtain the attribute features, object features and combination features of the training image respectively; Step 206, using the attribute classifier, the object classifier, and the combination classifier to classify the attribute, the object, and the combination respectively, and obtain the prediction scores of the attribute, the object, and the combination; Step 207, iteration, using the classification result to calculate the loss function to constrain the object prompt pool, the attribute prompt pool, and the combination prompt pool, update the attribute prompt pool, the object prompt pool, and the combination prompt pool, and repeat steps 200 to 207; Step 300, testing, inputs the test image into the model described in steps 200 to 207, obtains the final combined prediction score based on the predicted classification scores output by the attribute classifier, object classifier, and combined classifier, and the combination with the highest score is the predicted classification result.

2. The combined incremental learning method for image classification tasks according to claim 1, characterized in that: In step 200 , ViT B / 16 trained on the ImageNet-1K image classification dataset is used as the image feature extractor f(·), where ViT B / 16 refers to Vision Transformer Base 16.

3. The combined incremental learning method for image classification tasks according to claim 1, characterized in that: The method for selecting the object prompt and the combination prompt in step 201 is as follows: The image feature obtained by the image feature extractor is recorded as f(x). By calculating the image feature f(x) and the object prompt pool The cosine similarity between the key values ​​in the object prompt is selected, and the object prompts corresponding to the k key values ​​with the highest scores are selected. Similarly, calculate the image feature f(x) and the combined prompt pool The cosine similarity between the key values ​​in the formula is used to select the combination prompts corresponding to the k key values ​​with the highest scores.

4. The combined incremental learning method for image classification tasks according to claim 1, characterized in that: In step 202, the method for fusing the object prompt and the combined prompt is as follows: Prompt the object selected in step 201 Perform the generalized mean fusion operation as follows: where GEM(·) represents the generalized mean fusion operation, η represents a manually selected hyperparameter; P o Provides hints for objects after fusion; Similarly, the combination selected in step 201 is prompted Perform generalized mean fusion, the fusion method is as follows: where GEM(·) represents the generalized mean fusion operation, η represents a manually selected hyperparameter; P c This is a hint for the combination after fusion.

5. The combined incremental learning method for image classification tasks according to claim 1, characterized in that: In step 203, the object prompt guides the selection of the attribute prompt as follows: The fused object obtained in step 202 is prompted P o As a guiding signal, it is injected into the image feature f(x) to obtain the attribute prompt selection signal f s (x); The injection method uses the cross attention mechanism (Cross Attention), and the specific operation formula is as follows: Among them, CrossAtten(·,·) is the cross attention mechanism (Cross Attention) operation, W Q , W K , W V They represent the learnable mapping matrices in the query vector Query, the queried vector Key, and the content vector Value in the cross-attention mechanism, and D represents the dimension of the vector; since f s (x) incorporates the object hint P o Therefore, f s (x) represents the cue selection signal after the object cue guidance, which can be regarded as the attribute cue selection signal; Use attribute hints to select signal f s (x) Calculate its attribute hint pool The cosine similarity between the key values ​​in the table, select the attribute prompts corresponding to the k key values ​​with the highest scores Perform generalized mean fusion, and the fusion method is consistent with the fusion method in step 202, as shown below: where GEM(·) represents the generalized mean fusion operation, η represents a manually selected hyperparameter; P s This is a hint for the combination after fusion.

6. The combined incremental learning method for image classification tasks according to claim 1, characterized in that: In step 204, a feature sequence of the training image x is obtained, and the design method is as follows: The Embedding Layer in ViT B / 16 trained on the ImageNet-1K image classification dataset is selected as the feature sequence acquirer. After the training image is input into the feature sequence acquirer, the output of the feature sequence acquirer is the feature sequence.

7. The combined incremental learning method for image classification tasks according to claim 1, characterized in that: In step 205, the acquisition of attribute features, object features and combined features of the training image is designed as follows: First, the fused object prompt and combination prompt obtained in step 202, the fused combination prompt obtained in step 203, and the training image feature sequence obtained in step 204 are spliced ​​together in the order of "combination-attribute-object-image feature sequence" in the first dimension to obtain a mixed feature sequence; then, the mixed feature sequence is input into the Transformer Encoder in ViT B / 16 trained on the ImageNet-1K image classification dataset; the Token at the first position output by the Transformer Encoder is the combination feature, the Token at the second position output is the attribute feature, and the Token at the third position output is the object feature.

8. The combined incremental learning method for image classification tasks according to claim 1, characterized in that: In step 206, the predicted scores of the attributes, objects, and combinations are obtained, and the design method is as follows: Three linear layers whose parameters are not shared are set as attribute classifier, object classifier and combination classifier respectively; the attribute feature obtained in step 205 is input into the attribute classifier to obtain the attribute prediction score; the object feature obtained in step 205 is input into the object classifier to obtain the object prediction score; the combination feature obtained in step 205 is input into the combination classifier to obtain the combination prediction score.

9. The combined incremental learning method for image classification tasks according to claim 1, characterized in that: In step 207, the loss function is calculated and designed as follows: The training of the model described in steps 200 to 206 is subject to three loss functions, namely, the diversity loss function L div , proxy loss function L sur And the classification loss function L Class ; Diversity loss function L div It is obtained by calculating the sum of the directional decoupling losses within the three cue pools and between the cue pools, that is, in They represent the directional decoupling losses between attribute pools, between object pools, and between combination pools. They represent the directional decoupling losses between the attribute pool and the object pool, between the attribute pool and the combination pool, and between the combination pool and the object pool. The six losses are calculated in the same way, using represents any one of the six losses, and its calculation formula is: Where M is the size of the prompt pool, θ thre is a hyperparameter, which is a manually set angle; θ nm Represents any two prompts and The angle between them is measured as follows: Among them, ∈ is set to a very small constant to prevent the denominator from being 0; and represents any two prompts in the prompt pool, ∥∥2 represents the L2 norm; Proxy loss function L sur The selection of the key value for the constraint hint pool is obtained by calculating the sum of the cosine similarity between the visual features and the key value. The calculation method is as follows: in, The attribute hint selected from the attribute hint pool The corresponding key value; A cue for an object selected from the object cue pool The corresponding key value; is a combination prompt selected from the combination prompt pool The corresponding key value, COS(·) represents the cosine similarity; Classification loss L Class By the traditional cross entropy loss L CE and the reverse cross entropy loss L RCE Composition, respectively acting on attribute classification, object classification, and combination classification. The specific formula is as follows: in, They represent attribute classification cross entropy loss, object classification cross entropy loss, and combination classification cross entropy loss, respectively. They represent attribute reverse cross entropy loss, object reverse cross entropy loss, and combination reverse cross entropy loss respectively; α is a weight used to balance the effects of cross entropy loss and reverse cross entropy loss; Finally, the loss function of the entire network is calculated as: L total =L div +L sur +L Class .

10. The combined incremental learning method for image classification tasks according to claim 1, characterized in that: In step 300, the calculation of the final combined prediction score is designed as follows: The attribute prediction score, object prediction score, and combined prediction score are marked as p(s|x), p(o|x), and p(c|x), respectively. The final combined prediction score is the sum of the three prediction scores, expressed as: p final =p(c|x)+β(p(s|x)+p(o|x)) Among them, β is a hyperparameter used to balance the contribution between the combination and the primitive; after obtaining the prediction score of the prediction combination, the combination with the highest score is the prediction classification result.

Images