Reweighting method based on multi-granularity margins

Abstract

The invention discloses a reweighting method based on multi-granularity margins, and belongs to the technical field of computer vision questions and answers. The method comprises the following steps: firstly preprocessing training data, calculating the probability value of each answer under two different division granularities of question types and specific questions according to the questions, measuring the uncertainty of margins through the similarity between the probability values and the overall distribution of the answers, and carrying out weighted summation on multi-granularity marginal weights and question type margins, so as to obtain a weighted sum; and obtaining a marginal loss function fused with multi-granularity information. And through gradient guidance, quantizing an image contribution degree by using a gradient value between an answer and an image feature as an adjustment factor to correct margins, and enabling a reweighting model to identify samples with different difficulties by adjusting a sample weight. According to the method, the coarse and fine granularity relationship between the question and the answer under statistical distribution is concerned, language deviation caused by unbalanced data distribution is effectively relieved by using a re-weighting method, and meanwhile, the method concerns to differentially treat samples with different classification difficulties, so that the robustness and accuracy of the model on an out-of-distribution data set are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision question answering, and in particular to a re-weighting method based on multi-granularity margins. Background Art

[0002] Visual question answering aims to enable computers to understand images and answer natural language questions about their contents. However, since the distribution of training datasets and data in actual scenarios is often significantly different, VQA models often rely on the language correlation between questions and answers in training, ignoring the full use of visual information. This makes it possible for the model to give answers based only on clues in the question text, or even without referring to the image content, in actual applications, resulting in incorrect answers. Some researchers have recently discovered that most existing VQA models rely heavily on using only the language correlation between questions and answers to answer most questions. This blind prediction of answers is called language bias.

[0003] Language bias stems from unbalanced data distribution. Take "How many bananas are there" in the VQA-CP v2 dataset as an example. From a coarse-grained perspective, it belongs to the question type "How many", and "two" is a common head data in the answer distribution of this type. Affected by language bias, when the model reads "how many", it often directly gives the common answer "two", while ignoring more text and image information and giving wrong answers. Reweighting can adjust the classification boundary by assigning different weights, focusing on minority class features to improve classification performance. The current popular method is to start from statistical distribution and assign weights to different classes based on the marginal values ​​obtained by the answer distribution probability under the question type, hoping that the model will treat all types of answers equally, avoid language bias interference, and obtain answers based on text and pictures. However, for the above example, this sample also belongs to the specific question "How many bananas are there" category under fine-grained division, and "two" is the tail data in this answer distribution. At this time, when looking at the answer distribution under the question type division, less weight will be given to the head data "two", which will alleviate the language bias at the question type level but aggravate the bias at the specific question level. The reason is that, starting from the specific problem division, the weight of "two" should be increased to correct the language bias at the specific problem level, but the actual operation is contrary to this, resulting in uncertainty in only considering the margin under a certain specific division. Re-adjusting the weight of the sample through this margin cannot enable the model to obtain the answer through real text and images without being disturbed by the bias, and thus the prediction result is also uncertain.

[0004] Secondly, for samples belonging to the same category, more prediction space should be reserved for difficult samples. However, the current method uses the answer prediction probability value to assess sample difficulty, ignoring the fact that the prediction probability is already affected by language bias and cannot accurately reflect sample difficulty. The contribution of visual features to the results affects the difficulty of sample classification. Excessive image noise will make it difficult for the model to obtain effective information, and the answer may be outside the classification boundary. Summary of the invention

[0005] The purpose of the present invention is to address the defects and shortcomings of the above-mentioned prior art and provide a reweighting method based on multi-granularity margins. The method can learn multi-granularity margins under statistical distribution. The adjustment of sample weights prompts the loss function to fully consider the distribution characteristics of answers under different granularities during the training process, which helps the model to better perceive the characteristics of each level of data, fit the distribution of answers under different divisions, and improve the robustness of the model. At the same time, through gradient guidance, the average gradient value is used to present the relationship between the rate of change of answers and image features of samples of different difficulty levels. By considering the difficulty of the samples, the weights of the samples are adjusted. Based on this, the model can pay different attention to samples of different difficulty levels, thereby enhancing the overall performance.

[0006] The technical solution adopted by the present invention to solve the technical problem is: a re-weighting method based on multi-granularity margins, the method comprising the following steps:

[0007] Step 1: Preprocess the training data set, starting from the two division granularities of question type dimension and specific question dimension, count the frequency of each answer in each question type or specific question, and then use the probability calculation formula to calculate the probability distribution value of each answer at the corresponding division granularity.

[0008] Step 2: For the text-image pairs in the training data set, input the visual question-answering model to extract image features and text features, fuse the joint features with the attention mechanism, input the classifier to obtain the output value, and obtain the answer probability distribution p through softmax. The maximum probability value is the final answer.

[0009] Step 3: Calculation of category margins;

[0010] Based on the different distribution of answers in step 1 above at different granularities, the margin has uncertainty. Through the collaborative operation of Euclidean distance and JS divergence, the marginal uncertainty with noise insensitivity based on the degree of numerical deviation is obtained. While retaining the dominance of the question type margin, it can be flexibly adjusted according to the differences in the distribution of answers of samples under multi-granularity division. This uncertainty is used as the weight of the multi-granularity margin and the weighted summation of the marginal value of the question type alone to determine the final category margin, so that the category margin can dynamically adapt to the complex characteristics of different samples.

[0011] Step 4: Calculation of sample difficulty margin;

[0012] For the predicted answer value output by the classifier in step 2 above, the image information in the text image pair will make a certain contribution to the prediction of the answer. The difference in the degree of contribution of different image information to the answer prediction is a key factor in measuring the difficulty of sample classification. Therefore, starting from the contribution sensitivity, the predicted answer value output by the classifier in step 2 above is calculated relative to the gradient average of the image block features after feature extraction, and the sample difficulty margin based on gradient perception is obtained.

[0013] Step 5: Calculation of multi-granularity marginal loss function;

[0014] Based on the sample difficulty margin obtained in step 4 above, the data is scaled to the same calculation layer of the category margin obtained in step 3 by the maximum and minimum scaling method, i.e., the MinMaxScaler method. The multi-granularity marginal value is summed up and substituted into the marginal loss function as the marginal factor to construct a multi-granularity marginal loss function.

[0015] Step 6: Repeat steps 3-5 above to train the visual question answering model. Based on the multi-granularity margin obtained in step 5 above, re-weight different samples. The loss value of the multi-granularity margin loss function is applied to back propagation. After multiple rounds of training, the model converges and the final model parameters are obtained.

[0016] As a further improvement of the present invention, the specific steps of the above step 3 of the present invention include:

[0017] Step S31: The marginal uncertainty caused by the difference in the probability value of the answer obtained in step 1 under different partitioning granularities is measured by Euclidean distance, and the JS divergence is introduced to effectively constrain the uncertainty fluctuation caused by the sensitivity of probability similarity by utilizing the good noise insensitivity of distribution divergence. The marginal uncertainty Unc is calculated as follows:

[0018] Unc=Ed*JS distances

[0019]

[0020]

[0021] Among them, Ed is the linear distance in Euclidean space between the probability values ​​of the answers to the questions at different division granularities, JS distances It is the JS divergence similarity calculation of two answer distributions under different partitioning granularities. are given question types Q in the training data respectively k and specific questions t The following answers a iThe probability of occurrence, n represents the number of different answer options.

[0022] Step S32: The marginal uncertainty Unc of the similarity result measurement obtained in step S31 captures the correlation and difference between the question type granularity and the specific question granularity, and uses it as a dynamic weight factor of the multi-granularity margin. When the answer distribution of the sample under multi-granularity division presents a high uncertainty, the weight factor will correspondingly increase the influence of the multi-granularity margin in the category margin calculation; conversely, when the uncertainty is low, the question type margin can adapt to both divisions at the same time and occupy a more dominant position, then the category margin m cat The calculation process is as follows:

[0023]

[0024]

[0025]

[0026]

[0027] in It is to integrate multi-granularity partition information to obtain the middle value of the margin. The positive or negative value of sign is used to determine the adjustment to enhance or weaken the marginal value of the problem type only. N is a Gaussian distribution. is the standard deviation, through Randomization treatment It can improve the universality of model prediction, and finally the category margin m obtained based on the probability value cat This ensures that frequent answer use cases have smaller margins than rare answer use cases.

[0028] As a further improvement of the present invention, the specific steps of step 4 include:

[0029] Because the classification score layer of the classifier connects the activation of the previous layer of the network with the scores of each category, its gradient relative to the image features can measure the contribution of the image features to the final category prediction. The calculation process is as follows:

[0030]

[0031]

[0032] Among them, v j For a given problem image pair, the image patch in image V of sample i, f d is the classifier output value obtained in step 2, d is the index of the predicted category probability, S iRepresents the gradient matrix of the answer prediction value relative to each image block, that is, for an image of size 36×2048, the dimension of the gradient matrix will be 36×2048, N is the total number of image blocks, m def The average gradient value is the sample difficulty margin.

[0033] As a further improvement of the present invention, the specific steps of the above step 5 of the present invention include:

[0034] Step S51: The sample difficulty margin m obtained in step 4 above is def , scaled to the category margin m obtained in step 3) by the maximum and minimum scaling method cat The same calculation layer of , the formula is as follows:

[0035]

[0036] Among them, m sca_def Indicates the scaled data, min def Indicates m def The minimum value in def Indicates m def The maximum value in , max is the maximum value of the interval to be mapped, and min is the minimum value of the interval to be mapped.

[0037] Step S52: In order to make the model fully consider the multi-granularity information and sample classification difficulty differences when the answer distribution is different at different division granularities, to achieve the model to train all sample classes equally and to adjust the emphasis on samples of different classification difficulties in a targeted manner, the sample difficulty margin m obtained in the above step 51 is sca_def and the category margin m obtained in step 3 above cat Sum to get the multi-granularity margin m comb , in order to re-adjust the weights of different samples based on multi-granularity margins during training. The calculation process is as follows:

[0038] m comb =m cat +m sca_def

[0039] S53: The multi-granularity margin m obtained in step 52 is comb Substituting the marginal loss function into the improved multi-granularity marginal loss function, the formula is as follows:

[0040]

[0041]

[0042] Among them, W i and x are the classifier weight vector and answer feature respectively, is the normalized result, θi For x and W i The angle between the two, A is the answer set, that is, the answer a i ∈A,m comb By category margin m cat and sample difficulty margin m sca_def composition.

[0043] As a further improvement of the present invention, the specific steps of the above step 6 of the present invention include:

[0044] The multi-granularity marginal loss function obtained in step 5 above is applied to the model. During the model training process, back-propagation is performed to optimize the model parameters according to the loss value until the model converges.

[0045] Beneficial effects:

[0046] 1. Compared with the traditional method for the language bias problem caused by unbalanced data distribution, which only considers the distribution of questions in question types or only considers the distribution of answers to questions under specific questions, and reweights the samples by the marginal values ​​obtained by the distribution in an attempt to balance the training process of the model, the method in the present invention considers the different language bias problems caused by the different answer distributions of the questions under two division granularities, and can reasonably adjust the marginal values ​​according to the distribution of different answer categories under multiple division granularities to find a balance point, adjust the weights of different categories according to the marginal values, and make the model train all samples equally, so as to better deal with the language bias problem and alleviate the influence of the language correlation between questions and answers on the model, but instead derive answers based on the question and image information.

[0047] 2. Unlike the traditional method of evaluating the difficulty of sample classification by the result probability value in model training, but not considering that the result probability value may be affected by language bias, the present invention evaluates the sample difficulty by the influence of the image information in the question image pair on the final answer, and the model can adjust the sample weight in real time during the learning process according to the difference in the sample difficulty margin, to distinguish the degree of importance attached to different samples, and further optimize the overall performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 It is an architectural diagram of the re-weighting method based on multi-granularity margins of the present invention. DETAILED DESCRIPTION

[0049] The following is a diagram describing the implementation of the present invention. It should be noted that the embodiments described in the present invention are only a part of the embodiments of the present invention, and other details that are not closely related to the present invention are omitted. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0050] like Figure 1 As shown, the present invention provides a re-weighting method based on multi-granularity margins, which firstly counts the question type granularity and the answer distribution at the specific question granularity in the data set; performs feature extraction and feature fusion on the question and image through the visual question answering model, and then sends the fused features to the classifier to obtain the classification score; measures the uncertainty of the margin through the difference in answer distribution to obtain the category margin that fuses the multi-division granularity information, and calculates the gradient-aware sample difficulty margin based on the contribution of the image information to the answer, and calculates the sum of the margins of the two parts as the multi-granularity margin, and proposes a multi-granularity margin loss function to calculate the loss; finally, trains the visual question answering model and optimizes the parameters of the visual question answering model.

[0051] The specific implementation steps of the present invention include the following:

[0052] Step 1: Preprocess the training data set, starting from the two division granularities of question type dimension and specific question dimension, count the frequency of each answer in each question type or specific question, and then use the probability calculation formula to calculate the probability distribution value of each answer at the corresponding division granularity.

[0053] Step 2: For the text-image pairs in the training data set, input the visual question-answering model to extract image features and text features, fuse the joint features with the attention mechanism, input the classifier to obtain the output value, and obtain the answer probability distribution p through softmax. The maximum probability value is the final answer.

[0054] Step 3: Calculation of category margin: Based on the different distribution of answers in step 1 at different granularities, the margin is uncertain. Through the collaborative operation of Euclidean distance and JS divergence, the marginal uncertainty that is noise-insensitive based on the degree of numerical deviation is obtained. While retaining the dominance of the question type margin, it can be flexibly adjusted according to the differences in the distribution of answers of samples under multi-granularity division. This uncertainty is used as the weight of the multi-granularity margin and is weighted and summed with the question type marginal value alone to determine the final category margin, so that the category margin can dynamically adapt to the complex characteristics of different samples.

[0055] Step 4: Calculation of sample difficulty margin: For the predicted answer value output by the classifier in step 2 above, the image information in the text image pair will make a certain contribution to the prediction of the answer. The difference in the degree of contribution of different image information to the answer prediction is a key factor in measuring the difficulty of sample classification. Therefore, starting from the contribution sensitivity, the predicted answer value output by the classifier in step 2 above is calculated relative to the gradient average of the image block features after feature extraction, and the sample difficulty margin based on gradient perception is obtained.

[0056] Step 5: Calculation of multi-granularity marginal loss function: Based on the sample difficulty margin obtained in step 4 above, the data is scaled to the same calculation layer of the category margin obtained in step 3 above by the maximum and minimum scaling method, i.e., MinMaxScaler method. The multi-granularity marginal value is summed up and substituted into the marginal loss function as the marginal factor to construct a multi-granularity marginal loss function.

[0057] Step 6: Repeat steps 3-5 above to train the visual question answering model. Based on the multi-granularity margin obtained in step 5 above, the different samples are re-weighted. The loss value of the multi-granularity margin loss function is applied to back propagation. After multiple rounds of training, the model converges and the final model parameters are obtained.

[0058] Steps 1-6 are specifically described below using simulated data. The present invention uses the question “How many bananas are there?” in the VQA-CP v2 training set. Figure 1 The use cases in the section are described in detail.

[0059] The dataset in step 1 uses VQA-CP v2, which ensures that the answers to each type of question have significantly different distribution characteristics in the training set and the test set, and is often used to evaluate whether the model has over-correction. For the question "How many bananas are there?" in the dataset, it belongs to both the question type "How many" and the specific question "How many bananas are there". When counting the frequency of answers in these two categories, different distributions will appear. The calculation process of the answer probability values ​​under the two divisions is as follows:

[0060]

[0061]

[0062] in are given question types Q in the training data respectively k That is, "How many" and specific questions q t That is, the answer to "How many bananas are there" i The probability of occurrence, A is the answer set. It's Q k ,q t Answer a iThe frequency of occurrence of “How many” is obtained through calculation, and ε is a hyperparameter to avoid computational overflow. The distribution of answers to the “How many” category is obtained as p1 = (0.131, 0.383…), corresponding to answers 4, 2…, respectively, and the distribution of answers to the “How many bananas are there” category is p2 = (0.207, 0.095…).

[0063] The text-image pair processing in step 2 specifically includes:

[0064] S21: For the visual question answering model, the VQA model UpDn, which is commonly used in debiasing tasks, is used. For the question "How many bananas are there?" in the dataset, the question is first segmented according to spaces, all English words are lowercase and punctuation is deleted, and indexes are designed for all words that appear in the question to build a vocabulary. The processed question is then used as the input of the UpDn model, and the 300-dimensional Glove is used to initialize the question words. The question length is fixed to 14, and the question words that are shorter are used. <pad>The label is filled and then input into a single-layer GRU to obtain a 1024-dimensional problem feature representation.

[0065] S22: The image corresponding to "How many bananas are there" is as follows: Figure 1 As shown in the use case, as the input of the UpDn model, the image features with 36×2048 dimensions are obtained after encoding through the pre-trained Faster R-CNN, where 36 represents the number of image blocks and the feature of each image block is a 2048-dimensional vector.

[0066] S23: The problem features of S21 and the visual features of S22 are fused through a top-down attention mechanism.

[0067] S24: Send the fused features of S23 to the classifier to obtain the classification score f.

[0068] S25: The classification score f of S24 is then processed by softmax to obtain the predicted answer probability value. The answer corresponding to the largest probability value is the final model output answer. When the model is not trained, due to the influence of language bias, for "Ho w many bananas are there”, the answer is probably not Figure 1 The actual answer in the use case is 5, but the most common answer in the "How many" category is 2, or the most common answer in the "How many bananas are there" category is 4 without considering the image information.

[0069] In step 3, for the category margin m cat The calculation process is as follows:

[0070] S3 1: The marginal uncertainty caused by the difference in the probability values ​​of the answers to the question "How many bananas are there" at different partitioning granularities in step 1 is measured by Euclidean distance. The JS divergence is introduced to effectively constrain the uncertainty fluctuation caused by the sensitivity to probability similarity by utilizing the good noise insensitivity of distribution divergence. The marginal uncertainty Unc is calculated as follows:

[0071] Unc=Ed*JS distances

[0072]

[0073]

[0074] Among them, Ed is the linear distance in Euclidean space between the probability values ​​of the answers to the questions at different division granularities, JS distances It is the JS divergence similarity calculation of two answer distributions under different partitioning granularities. They are the answers to the given question type "How many" and the specific question "How many bananas are there" in the training data. i The probability of occurrence, n represents the number of different answer options. According to the various answers to the question "How many bananas are there", the corresponding marginal uncertainty value will be obtained. For example, to calculate the marginal uncertainty corresponding to answer 4, the probability values ​​of answer 4 under the two distributions, 0.131 and 0.207, are substituted into the calculation of the Euclidean distance, while the JS divergence calculates the similarity between the two distributions of p1 and p2.

[0075] S32: The marginal uncertainty Unc of the similarity result measurement for the question "How many bananas are there" obtained in step S31 captures the correlation and difference between the granularity of the question type "How many" and the granularity of the specific question "How many bananas are there", and uses it as a dynamic weight factor for the multi-granularity margin. When the answer distribution of the sample under multi-granularity division presents a high uncertainty, the weight factor will correspondingly increase the influence of the multi-granularity margin in the calculation of the category margin; conversely, when the uncertainty is low, since the answer distribution under the two divisions is similar, that is, there will be no interference from two language biases at the same time, the question type margin can adapt to the two divisions at the same time and occupy a more dominant position, then the category margin m cat The calculation process is as follows:

[0076]

[0077]

[0078]

[0079]

[0080] in It is to integrate multi-granularity information to obtain the middle value of the margin. The sign is used to determine whether to strengthen or weaken the marginal value of the question type only. For example, for the answer 4 under "How many bananas are there?", the marginal value obtained at the question type granularity is based on the probability value of 0.131. The answer probability values ​​under the two granularities are different, so it is necessary to find a marginal value that can adapt to both divisions through adjustment. Then N is a Gaussian distribution, is the standard deviation, through Randomization treatment It can improve the universality of model prediction, and finally the category margin m obtained based on the probability value cat This ensures that frequent answer use cases have smaller margins than rare answer use cases.

[0081] Sample difficulty margin m for step 4 def The calculation process is as follows:

[0082] Because the classification score layer of the classifier connects the activation of the previous layer of the network with the scores of each category, its gradient relative to the image features can measure the contribution of the image features to the final category prediction. The calculation process is as follows:

[0083]

[0084]

[0085] Among them, v j is the image patch in the image Picture corresponding to the question "How many bananas are there?", d is the classifier output value obtained in step 2, d is the index of the predicted category probability, S i The gradient matrix represents the model's predicted answer value for "How many bananas are there?" relative to each image block. This gradient matrix represents the rate of change between the answer and the image features, reflecting the contribution of the image features to the final answer. That is, for an image of size 36×2048, the dimension of the gradient matrix will be 36×2048, N is the total number of image blocks 36, and m is the number of pixels. def The average gradient value is the sample difficulty margin.

[0086] The specific steps of step 5 include:

[0087] S51: Subtract the sample difficulty margin m obtained in step 4 from the def , scaled to the category margin m obtained in step 3 by the maximum and minimum scaling method cat The same calculation layer of , the formula is as follows:

[0088]

[0089] Among them, m sca_def Indicates the scaled data, min def Indicates m def中 The minimum value, max def Indicates m def The maximum value in , max is the maximum value of the interval to be mapped, and min is the minimum value of the interval to be mapped.

[0090] S52: In order to make the model fully consider the multi-granularity information of the question "How many bananas are there?" when the answer distribution p1 and p2 are different at different division granularities and the difference in sample classification difficulty, the model can equally train all sample classes and adjust the emphasis on samples of different classification difficulties in a targeted manner. The sample difficulty margin m obtained in step 51 is sca_def and the category margin m obtained in step 3 cat Sum to get the multi-granularity margin m comb , in order to re-adjust the weights of different samples based on multi-granularity margins during training. The calculation process is as follows:

[0091] m comb =m cat +m sca_def

[0092] S53: The multi-granularity margin m obtained in step 52 is comb Substituting the marginal loss function into the multi-granularity marginal loss function, the formula is as follows:

[0093]

[0094]

[0095] Among them, W i and x are the classifier weight vector and answer feature respectively, is the normalized result, θ i For x and W i The angle between the two, A is the answer set, that is, the answer a i ∈A,L mag That is, the multi-granularity marginal loss, m comb By category margin m cat and sample difficulty margin m sca_def composition.

[0096] In step 6, the multi-granularity margin loss function obtained in step 5 is applied to the visual question answering model UpDn for training on the VQA-CPv2 training set. The multi-granularity margin is used to adjust the weights of different samples. The model parameters are optimized by backpropagation during the model training process by calculating the loss value. The number of training rounds is set to 30 rounds until the model converges and the training ends. At this time, since the model re-weights the samples through the marginal value during training, the model is no longer affected by the language bias caused by the imbalanced data distribution. For the question "How many bananas are there?", the model will be based on Figure 1 The correct answer is 5 based on the actual situation provided by the image information.

[0097] The method proposed in this invention is compared with the method based on reweighting to solve language bias, which also acts on the visual question answering model UpDn. The performance of the method is verified on the VQA-CP v2 validation set. The VQA-CP v2 dataset is divided into three types of data according to the answer type of the question, including Yes / No type, Number type and Other type. Overall represents the overall result performance of the dataset. The results are shown in Table 1. The learning effect of the reweighting method based on multi-granularity margin proposed in this paper is better than other advanced methods. Since the visual question answering task is a classification task, the correctness of the model answer needs to be evaluated, so the accuracy is used as the evaluation criterion of the model performance, that is, the proportion of correctly answered questions to the total number of questions.

[0098] Table 1 Accuracy comparison with other methods on VQA-CP v2 dataset

[0099]

[0100]

[0101] Among them, LRS, AdaVQA, RMLVQA, DFQ, and FAN-VQA are all reweighting methods proposed to solve the language bias problem in visual question answering, while RMMGM is a reweighting method based on multi-granularity margins proposed by the present invention. The best result in each column is marked in bold, and the suboptimal result is underlined. According to Table 1, the method proposed by the present invention performs well on the out-of-distribution dataset VQA-CP v2. Among them, the accuracy on the "Number" class reaches 53.13%, which is a significant improvement of 4.13% over the suboptimal result, which is generally considered to be more challenging in VQA tasks. The results are also suboptimal on the "Yes / No" and "Other" classes, and the overall accuracy has achieved the current optimal level, indicating the effectiveness of the method in alleviating language bias and improving model performance.

[0102] In summary, the present invention can well capture the different language bias problems that exist simultaneously due to different answer distributions when the questions are at different division granularities, as well as the judgment of the difficulty of sample classification by image information through the reweighting method of multi-granularity margins, and dynamically adjust the sample weights of the model during the training process through the calculation of multi-granularity marginal values, so that the model treats each type of data equally during the training process. The method in the present invention makes up for the fact that the current reweighting method does not consider the coarse and fine granularity relationship between questions and answers to avoid the influence of language bias, and the consideration of learning based on sample difficulty. The overall performance on the out-of-distribution dataset VQA-CP v2 is better than other advanced methods, has good use value, effectively improves the robustness of the visual question answering model, and is beneficial to the performance of the model in complex real-world environments.< / pad>

Claims

1. A multi-granularity margin-based reweighting method, characterized in that: The method comprises the following steps: Step 1: Preprocess the training data set, starting from the two granularities of question type and specific question, count the frequency of each answer in each question type or specific question, and then calculate the probability distribution value of each answer at the corresponding granularity through the probability calculation formula; Step 2: For the text-image pairs in the training data set, input the visual question-answering model to extract image features and text features, fuse them with the attention mechanism to obtain joint features, input them into the classifier to obtain the output value, and obtain the answer probability distribution p through softmax. The maximum probability value is the final answer. Step 3: Calculation of category margin: Based on the different distribution of answers in step 1 above at different granularities, the margin is uncertain. Through the collaborative operation of Euclidean distance and JS divergence, the marginal uncertainty that is noise-insensitive based on the degree of numerical deviation is obtained. This method can retain the dominance of the question type margin and can be flexibly adjusted according to the differences in the distribution of answers of samples under multi-granularity division. This uncertainty is used as the weight of the multi-granularity margin and is weighted and summed with the question type marginal value to determine the final category margin, so that the category margin can dynamically adapt to the complex characteristics of different samples; Step 4: Calculation of sample difficulty margin: For the predicted answer value output by the classifier in step 2 above, the image information in the text image pair will make a certain contribution to the prediction of the answer. The difference in the degree of contribution of different image information to the answer prediction is the key factor in measuring the difficulty of sample classification. Therefore, starting from the contribution sensitivity, the predicted answer value output by the classifier in step 2 above is calculated relative to the gradient average of the image block features after feature extraction to obtain the sample difficulty margin based on gradient perception; Step 5: Calculation of multi-granularity marginal loss function; Based on the sample difficulty margin obtained in step 4 above, the data is scaled to the same calculation layer of the category margin obtained in step 3 above by the maximum and minimum scaling, i.e., the MinMaxScaler method. The multi-granularity marginal value is summed up and substituted into the marginal loss function as the marginal factor to construct a multi-granularity marginal loss function. Step 6: Repeat steps 3-5 above to train the visual question answering model. Based on the multi-granularity margin obtained in step 5 above, re-weight different samples. The loss value of the multi-granularity margin loss function is applied to back propagation. After multiple rounds of training, the model converges and the final model parameters are obtained.

2. The multi-granularity margin-based re-weighting method according to claim 1, characterized in that: The specific steps of step 3 include: Step S31: The marginal uncertainty caused by the difference in the probability value of the answer obtained in step 1 under different partitioning granularities is measured by Euclidean distance, and the JS divergence is introduced to effectively constrain the uncertainty fluctuation caused by the sensitivity of probability similarity by utilizing the good noise insensitivity of distribution divergence. The marginal uncertainty Unc is calculated as follows: Unc=Ed*JS distances Among them, Ed is the linear distance in Euclidean space between the probability values ​​of the answers to the questions at different division granularities, JS distances It is the JS divergence similarity calculation of two answer distributions under different partitioning granularities. are given question types Q in the training data respectively k and specific questions t The following answers a i The probability of occurrence, n represents the number of different answer options; S32: The marginal uncertainty Unc of the similarity result measurement obtained in step S31 captures the correlation and difference between the question type granularity and the specific question granularity, and uses it as a dynamic weight factor of the multi-granularity margin. When the answer distribution of the sample under multi-granularity division presents a high uncertainty, the weight factor will correspondingly increase the influence of the multi-granularity margin in the category margin calculation; conversely, when the uncertainty is low, the question type margin can adapt to both divisions at the same time and occupy a more dominant position, then the category margin m cat The calculation process is as follows: in It is to integrate multi-granularity partition information to obtain the middle value of the margin. The positive or negative value of sign is used to determine the adjustment to enhance or weaken the marginal value of the problem type only. N is a Gaussian distribution. is the standard deviation, through Randomization treatment It can improve the universality of model prediction, and finally the category margin m obtained based on the probability value cat This ensures that frequent answer use cases have smaller margins than rare answer use cases.

3. The multi-granularity margin-based re-weighting method according to claim 1, characterized in that: In step 4, because the classification score layer of the classifier connects the activation of the previous layer of the network with the scores of each category, its gradient relative to the image feature can measure the contribution of the image feature to the final category prediction. The calculation process is as follows: Among them, v j For a given problem image pair, the image patch in image V of sample i, f d is the classifier output value obtained in step 2, d is the index of the predicted category probability, S i Represents the gradient matrix of the answer prediction value relative to each image block, that is, for an image of size 36×2048, the dimension of the gradient matrix will be 36×2048, N is the total number of image blocks, m def The average gradient value is the sample difficulty margin.

4. The multi-granularity margin-based re-weighting method according to claim 1, characterized in that: The specific steps of step 5 include: S51: Subtract the sample difficulty margin m obtained in step 4 from the above def , scaled to the category margin m obtained in step 3 above by the maximum and minimum scaling method cat The same calculation layer of , the formula is as follows: Among them, m sca_def Indicates the scaled data, min def Indicates m def The minimum value in def Indicates m def The maximum value in , max is the maximum value of the interval to be mapped, and min is the minimum value of the interval to be mapped; S52: In order to make the model fully consider the multi-granularity information and sample classification difficulty differences when the answer distribution is different at different division granularities, to achieve the model to train all sample classes equally and to adjust the emphasis on samples of different classification difficulties in a targeted manner, the sample difficulty margin m obtained in the above step 51 is sca_def and the category margin m obtained in step 3 cat Sum to get the multi-granularity margin m comb , in order to re-adjust the weights of different samples based on multi-granularity margins during training. The calculation process is as follows: m comb =m cat +m sca_def S53: The multi-granularity margin m obtained in step 52 is comb Substituting the marginal loss function into the improved multi-granularity marginal loss function, the formula is as follows: Among them, W i and x are the classifier weight vector and answer feature respectively, is the normalized result, θ i For x and W i The angle between the two, A is the answer set, that is, the answer a i ∈A,m comb By category margin m cat and sample difficulty margin m sca_def composition.

5. The multi-granularity margin-based re-weighting method according to claim 1, characterized in that: The specific steps of step 6 include: applying the multi-granularity marginal loss function obtained in step 5 to the model, and performing back propagation to optimize the model parameters according to the loss value during the model training process until the model converges.

Images